System, method, and computer program product for generating arbitrary views of a scene

Abstract

The present invention relates to arbitrary view generation. Techniques for generating an arbitrary view based on a plurality of other existing views are disclosed. In some embodiments, the arbitrary view generation includes: retrieving a first image having a first perspective and a second image having a second perspective; transforming each of the first image and the second image into a third perspective; and generating a third image having the third perspective by combining at least some of the pixels from the transformed first image with at least some of the pixels from the transformed second image.

Description

[0001] This application is a divisional application. The name of the invention of the parent application is “Arbitrary View Generation”. The application date is March 24, 2017, and the application number is 201780019122.7. Background Art

[0002] Existing rendering techniques face a trade-off between the competing goals of quality and speed. High-quality rendering requires significant processing resources and time. However, slow rendering techniques are unacceptable in many applications, such as interactive real-time applications. Lower-quality but faster rendering techniques are typically advantageous for such applications. For example, rasterization is commonly employed by real-time graphics applications for relatively fast rendering, but at the expense of quality. Therefore, there is a need for improved techniques that do not significantly compromise quality or speed. Summary of the Invention

[0003] A system for generating arbitrary views of a scene according to the present invention includes: a processor configured to: obtain a corresponding set of one or more views for each of a plurality of different scenes; normalize each of the set of views of each scene having an existing perspective different from a desired perspective to the desired perspective, wherein the normalized view having the desired perspective has at least some missing pixels at locations introduced in the desired perspective that are not present in the existing perspective; and generate an overall view comprising the plurality of different scenes having the desired perspective using, at least in part, pixels from views of the scene having the desired perspective including the normalized view; and a memory coupled to the processor and configured to provide instructions to the processor. Preferably, each scene comprises an object, a plurality of objects, or a rich virtual environment. The one or more views comprising the set of views of the scene are independently captured, rendered, or both. The set of views of the scene comprises photographs, photorealistic renderings, or both. The set of views of at least one scene comprises all available existing views of the scene. The set of views of at least one scene comprises a subset of all available existing views of the scene. The set of views of at least one scene comprises a threshold number of views. The set of views of the at least one scene comprises a single view, the existing viewing angle of the single view comprising the desired viewing angle. Figure 1The processor is further configured to store and know for each obtained view. The camera characteristics include one or more of: position, orientation, rotation, angle, depth, focal length, aperture, zoom level, shutter speed, and exposure. Normalization includes transforming from an existing perspective to a desired perspective based on known camera information associated with each view. Normalization includes transforming depth based on known depth information associated with each view. The processor is further configured to normalize to desired lighting conditions based on known lighting information associated with each view. Generating the overall view includes interpolating one or more pixels of the overall view that are not available from any view of the scene. The multiple scenes are generated separately and independently. The desired perspective is constrained based on existing perspectives of views of the scene that can be used to generate the overall view. The overall view includes one or more of: a still image, a view of a three-dimensional space, and a view of a virtual environment. The overall view includes frames of an animation or video sequence. A method for generating arbitrary views of a scene according to the present invention comprises: for each of a plurality of different scenes, obtaining a corresponding set of one or more views; normalizing each of the set of views of each scene having an existing perspective different from a desired perspective to the desired perspective, wherein the normalized view of the desired perspective has at least some missing pixels at locations introduced in the desired perspective that are not present in the existing perspective; and generating an overall view comprising the plurality of different scenes having the desired perspective using, at least in part, pixels from views of the scene having the desired perspective including the normalized view. A computer program product according to the present invention, comprising computer instructions, when executed on a processor, is configured to perform the following operations: for each of a plurality of different scenes, obtaining a corresponding set of one or more views; normalizing each of the set of views of each scene having an existing perspective different from the desired perspective to the desired perspective, wherein the normalized view of the desired perspective has at least some missing pixels at locations introduced in the desired perspective that are not present in the existing perspective; and generating an overall view comprising the plurality of different scenes having the desired perspective using, at least in part, pixels from views of the scene having the normalized view. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Various embodiments of the invention are disclosed in the following detailed description and accompanying drawings.

[0005] Figure 1 is a high-level block diagram illustrating an embodiment of a system for generating arbitrary views of a scene.

[0006] Figure 2 An example of a database asset is shown.

[0007] Figure 3is a flow chart illustrating an embodiment of a process for generating arbitrary perspectives. DETAILED DESCRIPTION

[0008] The present invention can be implemented in many ways, including as: a process; an apparatus; a system; a composition of matter; a computer program product embodied on a computer-readable storage medium; and / or a processor, such as a processor configured to execute instructions stored on and / or provided by a memory, the memory being coupled to the processor. In this specification, these implementations or any other form that the present invention may take may be referred to as techniques. In general, the order of the steps of the disclosed processes may be changed within the scope of the present invention. Unless otherwise stated, a component such as a processor or memory described as being configured to perform a task may be implemented as: a general component that is temporarily configured to perform the task at a given time; or a specific component that is manufactured to perform the task. As used herein, the term "processor" refers to one or more devices, circuits, and / or processing cores that are configured to process data (such as computer program instructions).

[0009] The following is a diagram illustrating the principles of the present invention. Figure 1 A detailed description of one or more embodiments of the present invention is provided below. The present invention is described in conjunction with such embodiments, but the invention is not limited to any embodiment. The scope of the present invention is limited only by the claims, and the invention encompasses many alternatives, modifications, and equivalents. In the following description, many specific details are set forth in order to provide a thorough understanding of the present invention. These details are provided for illustrative purposes, and the present invention can be practiced according to the claims without some or all of these specific details. For the purpose of clarity, technical material known in the art related to the present invention has not been described in detail so that the present invention is not unnecessarily obscured.

[0010] Techniques for generating arbitrary views of a scene are disclosed. The paradigm described herein necessitates very low processing or computational overhead while still providing high-definition output, effectively eliminating the challenging trade-off between rendering speed and quality. The disclosed techniques are particularly useful for very quickly generating high-quality output for interactive real-time graphics applications. Such applications rely on substantially instantaneous presentation of preferably high-quality output in response to and based on user manipulation of a rendered interactive view or scene.

[0011] Figure 1is a high-level block diagram illustrating an embodiment of a system 100 for generating arbitrary views of a scene. As depicted, an arbitrary view generator 102 receives a request for an arbitrary view as input 104, generates the requested view based on existing database assets 106, and provides the generated view in response to the input request as output 108. In various embodiments, the arbitrary view generator 102 may include a processor such as a central processing unit (CPU) or a graphics processing unit (GPU). Figure 1 The depicted configuration of the system 100 in FIG. 1 is provided for purposes of explanation. Generally, the system 100 may include any other suitable number and / or configuration of interconnected components that provide the described functionality. For example, in other embodiments, any view generator 102 may include internal components 110 - 116 in a different configuration, any view generator 102 may include multiple parallel physical and / or virtual processors, the database 106 may include multiple networked databases or an asset cloud, and the like.

[0012] The arbitrary view request 104 comprises a request for an arbitrary perspective of a scene. In some embodiments, the requested perspective of the scene is not already present in the asset database 106, which includes other perspectives or viewpoints of the scene. In various embodiments, the arbitrary view request 104 may be received from a process or a user. For example, the input 104 may be received from a user interface in response to user manipulation of a rendered scene or portion thereof, such as user manipulation of a camera viewpoint of the rendered scene. As another example, the arbitrary view request 104 may be received in response to specification of a movement or travel path within a virtual environment, such as a virtual fly-through of the scene. In some embodiments, the possible arbitrary views of the scene that may be requested are at least partially constrained. For example, a user may not be able to manipulate the camera viewpoint of the rendered interactive scene to any random position, but rather may be constrained to certain positions or perspectives of the scene.

[0013] Database 106 stores multiple views of each stored asset. In a given scenario, the specific scenes of an asset whose specifications are stored in database 106 are referred to as multiple views. In various embodiments, a scene may include a single object, multiple objects, or a rich virtual environment. Specifically, database 106 stores multiple images corresponding to different perspectives or viewpoints of each asset. The images stored in database 106 include high-quality photographs or photo-realistic renderings. Such high-definition, high-resolution images used to populate database 106 can be captured or rendered during an offline process or obtained from an external source. In some embodiments, corresponding camera characteristics are stored with each image stored in database 106. That is, camera properties such as relative position or location, orientation, rotation, depth information, focal length, aperture, zoom level, etc. are stored with each image. Additionally, camera lighting information such as shutter speed and exposure may be stored with each image stored in database 106.

[0014] In various embodiments, any number of different perspectives of an asset may be stored in database 106. Figure 2 An example of a database asset is shown. In the given example, seventy-three views corresponding to different angles around a chair object are captured or rendered and stored in database 106. The views can be captured, for example, by rotating the camera around the chair or rotating the chair in front of the camera. Relative object and camera position and orientation information is stored with each generated image. Figure 2 Specifically, views of a scene including a single object are illustrated. Database 106 may also store specifications for scenes including multiple objects or rich virtual environments. In this case, multiple views corresponding to different positions or locations in the scene or three-dimensional space are captured or rendered and stored in database 106 along with corresponding camera information. Generally, the images stored in database 106 may include two or three dimensions and may include stills or frames of animations or video sequences.

[0015] In response to a request for an arbitrary view of the scene 104 that is not already present in the database 106, the arbitrary view generator 102 generates the requested arbitrary view based on a plurality of other existing views of the scene stored in the database 106. Figure 1In an example configuration, the asset management engine 110 of the arbitrary view generator 102 manages the database 106. For example, the asset management engine 110 can facilitate the storage and retrieval of data in the database 106. In response to a request for an arbitrary view of the scene 104, the asset management engine 110 identifies and obtains multiple other existing views of the scene from the database 106. In some embodiments, the asset management engine 110 retrieves all existing views of the scene from the database 106. Alternatively, the asset management engine 110 can select and retrieve a subset of existing views that are, for example, closest to the requested arbitrary view. In this case, the asset management engine 110 is configured to intelligently select a subset of existing views from which pixels can be harvested to generate the requested arbitrary view. In various embodiments, multiple existing views can be retrieved by the asset management engine 110 together, or as and when they are needed by other components of the arbitrary view generator 102.

[0016] The perspective of each existing view retrieved by the asset management engine 110 is transformed by the perspective transformation engine 112 of the arbitrary view generator 102 into the perspective of the requested arbitrary view. As previously described, the precise camera information is known and stored with each image stored in the database 106. Therefore, the perspective change from an existing view to the requested arbitrary view comprises a simple geometric mapping or transformation. In various embodiments, the perspective transformation engine 112 may employ any one or more suitable mathematical techniques to transform the perspective of the existing view into the perspective of the arbitrary view. In the case where the requested view includes an arbitrary view that is not identical to any existing view, the transformation of the perspective of the existing view to the arbitrary view will include at least some unmapped or missing pixels, i.e., at angles or positions introduced in the arbitrary view that are not present in the existing views.

[0017] Pixel information from a single perspective-transformed existing view will not be able to populate all the pixels of a different view. However, in many cases, most, if not all, of the pixels that make up the requested arbitrary view can be harvested from multiple perspective-transformed existing views. The merging engine 114 of the arbitrary view generator 102 combines the pixels from multiple perspective-transformed existing views to generate the requested arbitrary view. Ideally, all pixels that make up the arbitrary view are harvested from existing views. This may be possible, for example, if a sufficiently diverse set of existing views or perspectives of the asset under consideration is available and / or if the requested perspective is not too dissimilar to the existing perspectives.

[0018] Any suitable technique can be used to combine or merge pixels from multiple perspective-transformed existing views to generate the requested arbitrary view. In one embodiment, a first existing view that is closest to the requested arbitrary view is selected and retrieved from the database 106 and transformed to the perspective of the requested arbitrary view. Pixels from the perspective-transformed first existing view are then harvested and used to fill in corresponding pixels in the requested arbitrary view. To fill in pixels of the requested arbitrary view that are not available from the first existing view, a second existing view that includes at least some of these remaining pixels is selected and retrieved from the database 106 and transformed to the perspective of the requested arbitrary view. Pixels from the perspective-transformed second existing view that are not available from the first existing view are then harvested and used to fill in corresponding pixels in the requested arbitrary view. This process can be repeated for any number of additional existing views until all pixels of the requested arbitrary view have been filled and / or until all existing views have been exhausted or a specified threshold number of existing views have been used.

[0019] In some embodiments, the requested arbitrary view may include some pixels that are not available from any existing view. In this case, the interpolation engine 116 is configured to fill in any remaining pixels of the requested arbitrary view. In various embodiments, any one or more appropriate interpolation techniques may be employed by the interpolation engine 116 to generate these unfilled pixels in the requested arbitrary view. Examples of employable interpolation techniques include, for example, linear interpolation, nearest neighbor interpolation, etc. Interpolation of pixels introduces averaging or smoothing. Overall image quality may not be significantly affected by certain interpolations, but excessive interpolation may introduce unacceptable blur. Therefore, it is desirable to use interpolation sparingly. As described above, if all pixels of the requested arbitrary view can be obtained from existing views, interpolation is avoided entirely. However, if the requested arbitrary view includes some pixels that are not available from any existing view, interpolation is introduced. Generally, the amount of interpolation required depends on the number of available existing views, the diversity of the perspectives of the existing views, and / or how different the perspective of the arbitrary view is from that of the existing views.

[0020] about Figure 2In the example depicted in , seventy-three views around a chair object are stored as existing views of the chair. A plurality of these existing views, preferably with minimal (if any) interpolation, can be used to generate any view around the chair object that is different or unique from any of the stored views. However, generating and storing such an exhaustive set of existing views may not be efficient or desirable. In some cases, a significantly smaller number of existing views covering a sufficiently diverse set of viewing angles can be generated and stored instead. For example, the seventy-three views of the chair object can be whittled down to a small set of a few views around the chair object.

[0021] As mentioned previously, in some embodiments, the possible arbitrary views that may be requested may be at least partially constrained. For example, a user may be restricted from moving a virtual camera associated with an interactive scene to certain locations. Figure 2 Given the example of , the possible arbitrary views that can be requested can be limited to any location around the chair object, but may not include, for example, any location below the chair object due to insufficient pixel data for the bottom of the chair object. This constraint on the allowed arbitrary views ensures that the requested arbitrary view can be generated by the arbitrary view generator 102 based on the existing data.

[0022] The arbitrary view generator 102 generates and outputs the requested arbitrary view 108 in response to an input arbitrary view request 104. The resolution or quality of the generated arbitrary view 108 is the same or similar to the quality of the existing views used to generate it, because pixels from those views are used to generate the arbitrary view. Therefore, using high-definition existing views results in high-definition output in most cases. In some embodiments, the generated arbitrary view 108 is stored in the database 106 along with other existing views of the associated scene, and can subsequently be used to generate other arbitrary views of the scene in response to further requests for arbitrary views. In the case where the input 104 includes a request for an existing view in the database 106, the requested view need not be generated from the other views as described; instead, the requested view is retrieved via a simple database lookup and presented directly as the output 108.

[0023] Furthermore, the arbitrary view generator 102 can be configured to generate an arbitrary overall view using the described techniques. That is, the input 104 may include a request to combine multiple objects into a single custom view. In this case, the aforementioned techniques are performed for each of the multiple objects and combined to generate a single integrated or overall view including the multiple objects. Specifically, an existing view of each of the multiple objects is selected and retrieved from the database 106 by the asset management engine 110, the existing view is transformed to the perspective of the requested view by the perspective transformation engine 112, pixels from the perspective-transformed existing view are used by the merging engine 114 to fill in corresponding pixels of the requested overall view, and any remaining unfilled pixels in the overall view are interpolated by the interpolation engine 116. In some embodiments, the requested overall view may include perspectives that already exist for one or more objects that make up the overall view. In this case, the existing view of the object asset corresponding to the requested perspective is used to directly fill in the pixels corresponding to the object in the overall view, rather than first generating the requested perspective based on other existing views of the object.

[0024] As an example of an arbitrary overall view including multiple objects, consider Figure 2 The chair object and the table object are captured or rendered separately. The chair object and the table object can be combined using the disclosed techniques to generate a single overall view of both objects. Thus, using the disclosed techniques, the independently captured or rendered images or views of each of a plurality of objects can be consistently combined to generate a scene including the plurality of objects and having a desired perspective. As previously described, the depth information for each of the existing views is known. The perspective transform for each of the existing views includes a depth transform, thereby allowing the plurality of objects to be properly positioned relative to each other in the overall view.

[0025] Generating an arbitrary overall view is not limited to combining multiple individual objects into a customized view. Instead, multiple scenes with multiple objects or multiple rich virtual environments can be similarly combined into a customized overall view. For example, multiple separately and independently generated virtual environments that may come from different content generation sources and may have different existing individual perspectives can be combined into an overall view with a desired perspective. Therefore, in general, the arbitrary view generator 102 can be configured to consistently combine or reconcile multiple independent assets including potentially different existing views into an overall view with a desired, potentially arbitrary perspective. A perfectly harmonious resulting overall view is generated because all combined assets are normalized to the same perspective. The possible arbitrary perspectives of the overall view can be constrained based on the existing views of the individual assets from which the overall view can be generated.

[0026] Figure 3is a flow chart illustrating an embodiment of a process for generating an arbitrary perspective. Process 300 may be performed, for example, by Figure 1 In various embodiments, the process 300 may be employed to generate any view or any overall view of a specified asset.

[0027] Process 300 begins at step 302, where a request for an arbitrary perspective is received. In some embodiments, the request received at step 302 may include a request for an arbitrary perspective of a specified scene that is different from any currently available perspective of the scene. In this case, for example, the arbitrary perspective request may be received in response to a requested change in perspective of a presented view of the scene. Such a change in perspective may be facilitated by changing or manipulating a virtual camera associated with the scene (such as by panning the camera, changing the focal length, changing the zoom level, etc.). Alternatively, in some embodiments, the request received at step 302 may include a request for an arbitrary overall view. As an example, such an arbitrary overall view request may be received with respect to an application that allows multiple independent objects to be selected and provides an integrated, perspective-corrected overall view of the selected objects.

[0028] At step 304, a plurality of existing images from which at least a portion of the requested arbitrary perspective is generated are retrieved from one or more associated asset databases. In the event that the request received at step 302 includes a request for arbitrary perspectives of a specified asset, the plurality of retrieved images may be associated with the specified asset, and in the event that the request received at step 302 includes a request for arbitrary overall views, the plurality of retrieved images may be associated with a plurality of assets.

[0029] At step 306, each of the multiple existing images retrieved at step 304, each having a different perspective, is transformed to the arbitrary perspective requested at step 302. Each of the existing images retrieved at step 304 includes associated perspective information. The perspective of each image is defined by the camera characteristics associated with generating the image (such as relative position, orientation, rotation, angle, depth, focal length, aperture, zoom level, lighting information, etc.). Since the complete camera information is known for each image, the perspective transformation of step 306 comprises a simple mathematical operation. In some embodiments, step 306 may also optionally include an illumination transformation so that all images are uniformly normalized to the same desired lighting conditions.

[0030] At step 308, at least a portion of the image with the arbitrary perspective requested at step 302 is filled with pixels harvested from the perspective-transformed existing image. That is, pixels from multiple perspective-corrected existing images are employed to generate the image with the requested arbitrary perspective.

[0031] At step 310, a determination is made as to whether the generated image with the requested arbitrary viewing angle is complete. If it is determined at step 310 that the generated image with the requested arbitrary viewing angle is incomplete, then at step 312 a determination is made as to whether any more existing images are available from which any remaining unfilled pixels of the generated image may be mined. If it is determined at step 312 that more existing images are available, then at step 314 one or more additional existing images are retrieved, and process 300 continues at step 306.

[0032] If it is determined at step 310 that the generated image having the requested arbitrary viewing angle is incomplete and if it is determined at step 312 that no more existing images are available, any remaining unfilled pixels of the generated image are interpolated at step 316. Any one or more suitable interpolation techniques may be employed at step 316.

[0033] If it is determined at step 310 that the generated image with the requested arbitrary viewing angle is complete or after interpolating any remaining unfilled pixels at step 316, the generated image with the requested arbitrary viewing angle is output at step 318. Process 300 then ends.

[0034] As described, the disclosed techniques can be used to generate arbitrary perspectives based on other existing perspectives. Normalizing different existing perspectives to a common desired perspective is possible because camera information is stored with each existing perspective. A resulting image with the desired perspective can be constructed by mining pixels from the perspective-transformed existing image. The processing associated with generating arbitrary perspectives using the disclosed techniques is not only fast and nearly instantaneous, but also results in high-quality output, making the disclosed techniques particularly powerful for interactive, real-time graphics applications.

[0035] Although the foregoing embodiments have been described in some detail for purposes of clarity of understanding, the invention is not limited to the details provided. There are many alternative ways of implementing the invention. The disclosed embodiments are illustrative and not restrictive.

Claims

1. A system for generating an arbitrary view of a scene, comprising: Processor, configured as: For each of a plurality of different scenes, obtaining a corresponding set of one or more views; normalizing each of a set of views of each scene having an existing viewpoint different from a desired viewpoint to the desired viewpoint, wherein the normalized views having the desired viewpoint have at least some missing pixels at locations introduced in the desired viewpoint that are not present in the existing viewpoint; as well as generating an overall view comprising a plurality of different scenes having the desired perspective using, at least in part, pixels from a view of the scene having the desired perspective comprising the normalized view; as well as A memory is coupled to the processor and configured to provide instructions to the processor.

2. The system of claim 1, wherein each scene comprises an object, a plurality of objects, or a rich virtual environment. 3 . The system of claim 1 , wherein one or more views comprising the set of views of the scene are independently captured, rendered, or both. The system of claim 1 , wherein the set of views of the scene comprises photographs, photorealistic renderings, or both. The system of claim 1 , wherein the set of views of at least one scene comprises all existing views of the scene that are available. The system of claim 1 , wherein the set of views of at least one scene comprises a subset of all existing views of the scene that are available. The system of claim 1 , wherein the set of views of at least one scene comprises a threshold number of views.

8. The system of claim 1, wherein the set of views of at least one scene comprises a single view, the existing viewing angle of the single view comprising the desired viewing angle.

9. The system of claim 1, wherein camera characteristics are stored with and known for each acquired view.

10. The system of claim 9, wherein the camera characteristics include one or more of: position, orientation, rotation, angle, depth, focal length, aperture, zoom level, shutter speed, and exposure.

11. The system of claim 1 , wherein normalizing comprises transforming from an existing perspective to a desired perspective based on known camera information associated with each view. 12 . The system of claim 1 , wherein normalizing comprises transforming depth based on known depth information associated with each view.

13. The system of claim 1, wherein the processor is further configured to normalize to desired lighting conditions based on known lighting information associated with each view.

14. The system of claim 1, wherein generating the overall view comprises interpolating one or more pixels of the overall view that are not available from any view of the scene.

15. The system of claim 1, wherein the plurality of different scenarios are generated separately and independently.

16. The system of claim 1, wherein the desired viewing angle is constrained based on existing viewing angles of views of the scene that can be used to generate the overall view.

17. The system of claim 1, wherein the overall view comprises one or more of: a still image, a view of a three-dimensional space, and a view of a virtual environment.

18. The system of claim 1, wherein the overall view comprises frames of an animation or video sequence.

19. A method for generating an arbitrary view of a scene, comprising: For each of a plurality of different scenes, obtaining a corresponding set of one or more views; normalizing each of a set of views of each scene having an existing viewpoint different from a desired viewpoint to the desired viewpoint, wherein the normalized views having the desired viewpoint have at least some missing pixels at locations introduced in the desired viewpoint that are not present in the existing viewpoint; as well as An overall view comprising a plurality of different scenes having the desired viewing angle is generated using, at least in part, pixels from a view of a scene having a desired viewing angle comprising a normalized view.

20. A computer program product comprising computer instructions which, when executed on a processor, are configured to: For each of a plurality of different scenes, obtaining a corresponding set of one or more views; normalizing each of a set of views of each scene having an existing viewpoint different from a desired viewpoint to the desired viewpoint, wherein the normalized views having the desired viewpoint have at least some missing pixels at locations introduced in the desired viewpoint that were not present in the existing viewpoint; and An overall view comprising a plurality of different scenes having the desired viewing angle is generated using, at least in part, pixels from a view of a scene having a desired viewing angle comprising a normalized view.

Images