System and method for use in photography

Abstract

Examples of systems and methods for filming are disclosed. In the examples, prop spatial data for prop devices and digital asset spatial data for virtual assets may be received. The spatial data of the digital assets may be updated based on the prop data to adjust their position, movement, and orientation in virtual space. In some examples, a scene model, a virtual representation of a scene, may be created using depth and color data. A scaled-down diorama of the scene model may be generated. A virtual camera may be configured to provide a perspective view of the visualized diorama on an output device. In some examples, the scene model may include virtual points corresponding to scene locations, creating a waypoint scene model. Augmented video data may incorporate these virtual points based on the waypoint scene model. The waypoint scene model may also be scaled down to provide a diorama.

Description

Technical Field

[0001] Related Applications This application claims the benefit and priority of U.S. Provisional Application No. 63 / 510,465, filed on June 27, 2023, entitled "Systems and Methods for Use in Scene Previewalization," and U.S. Provisional Application No. 63 / 606,804, filed on December 6, 2023, entitled "Systems and Methods for Digital Prop Tracking," each of which is incorporated herein by reference in its entirety.

[0002] The present disclosure generally relates to video production.

Background Art

[0003] Previewalization, often abbreviated as previz, is a process used in the video production, animation, and other visual media industries to create a preliminary visual representation of a planned scene or sequence. It involves creating a simplified or rough version of the intended shot using storyboards, 3D computer graphics, or other visual aids. Alternative techniques to traditional previz techniques have been developed for scene visualization. Previewalization systems have been developed to enable a director to capture a "good enough" shot of a scene with digital assets. Such systems enable a director to visualize a scene with animated (or static) digital assets using compositing techniques. Before the development of previewalization systems, a director would shoot a scene without digital assets, and a post-production team (or individual) would insert digital assets to enable a producer to visualize a shot of the scene with digital assets.

[0004] Compositing is the process or technique of combining visual elements from separate sources into a single image, often creating the illusion that all those elements are part of the same scene. Today, most, though not all, compositing is achieved through digital image manipulation. All compositing involves replacing selected parts of an image with other material from another image, though not always. In the digital method of compositing, a software command specifies a narrowly defined color as part of the image to be replaced. The software then replaces all pixels within the specified color range with pixels from another image, aligned to appear as part of the original image. Visual effects (sometimes abbreviated as VFX) are the process of creating or manipulating images outside the context of live-action footage shot in filmmaking and video production. VFX involves the integration of live-action footage (which may include in-camera special effects) and other live-action or realistic-looking computer-generated images (CGI) (digital or light, animals, or living things), but is dangerous, expensive, unrealistic, time-consuming, or impossible to shoot and capture on film.

[0005] In film and television production (or video production), prop tracking can refer to the process of tracking the movement, position, and / or orientation of physical props within a video scene (e.g., film footage) so that digital assets can be precisely aligned with them during post-production. Props are currently tracked using marker-based tracking systems. Marker-based tracking uses or requires placing one or more markers on the prop at key points, often high-contrast spheres or cubes. Depending on the marker-based tracking system, one or more markers can be simple geometric shapes or more complex patterns.

[0006] During filming, one or more cameras capture the movement of these markers along with the action. In post-production, tracking software analyzes the footage frame by frame. The software is designed to recognize the markers and calculate their position and movement in three-dimensional (3D) space. This data is then used to create a "skeleton" or framework that matches the movement of the prop. By attaching CGI elements to this skeleton, it is possible to ensure that they move in perfect synchronization with the filmed prop. The advantages of marker-based tracking include its accuracy and reliability. However, a disadvantage is that it is often time-consuming, as the markers must be removed from the final footage through a process called "painting out." [Overview of the project]

[0007] To provide a basic understanding, various details of this disclosure are summarized below. This summary is not intended to be a broad overview of the disclosure, nor is it intended to identify or describe the scope of any particular element of the disclosure. Rather, the main purpose of this summary is to present some of the concepts of the disclosure in a simplified form before the more detailed explanations presented below.

[0008] In the example, the computer implementation method may include the steps of: receiving prop space data for prop devices in physical space; receiving digital asset space data for digital assets in virtual space; updating space data for digital assets based on the prop space data; and updating the position, movement, and / or orientation of digital assets in virtual space based on the updated space data for digital assets.

[0009] In another example, the method may comprise the steps of: receiving prop space data about prop devices in a scene; receiving video data including captured video frames of the scene; generating a scene model of the scene with digital assets; updating the position, orientation, and / or movement of the digital assets in the scene model based on the prop space data; compositing one or more video frames and digital assets from the received video data with the updated position, movement, and / or orientation; and rendering the composite video frames onto an output device.

[0010] In a further example, a computer implementation method may include a step of identifying a video frame from among video frames stored in a cache memory space based on video frame latency data. The video frame latency data may specify the number of the video frame stored in the cache memory space before the video frame is selected. The method may further include a step of identifying a scene-related information frame from among scene-related information frames based on the time code of the video frame, and a step of providing a composite video frame based on the video frame and the scene-related information frame.

[0011] In a further example, a system for providing augmented video data may include memory for storing machine-readable instructions and data, and one or more processors for accessing the memory and executing the machine-readable instructions. The machine-readable instructions may include a video frame retriever for identifying video frames among video frames based on video frame latency data. The video frame latency data may specify the number of a video frame that is stored in memory before the video frame is selected. The machine-readable instructions may further include a scene-related frame retriever for identifying scene-related information frames among scene-related information frames based on the timecode of the video frame, a digital asset retriever for retrieving digital asset data that characterizes the digital asset, and a digital compositor for providing augmented video data with the digital asset based on the digital asset data, scene-related information frames, and video frame.

[0012] In a further example, a computer implementation method may include a step of identifying video frames among video frames provided by a video camera that represent a scene in film production, based on video frame latency data. The video frame latency data may specify the number of a video frame that is stored in the memory of the computing platform before the video frame is selected. The method may further include a step of identifying scene-related information frames among scene-related information frames provided by a scene capture device, based on an evaluation of the timecode of the video frame against the timestamp of the scene-related information frame. The timestamp for the scene-related information frame may be generated based on a frame delta value representing the amount of time between each scene-related information frame provided by each scene capture device of the scene capture device. The method may further include a step of providing augmented video data with digital assets based on the scene-related information frames and video frames.

[0013] In further examples, the computer implementation method may include the steps of: providing a scene model, which is a virtual representation of the scene, based on depth and color data captured for the scene; creating a scaled-down version of the scene model corresponding to a diorama of the scene; setting up a virtual camera for the scene model to provide a perspective view of the diorama; and outputting the diorama of the scene as a perspective view on an output device.

[0014] In another example, a computer implementation method may include the steps of: receiving a waypoint instruction that identifies virtual points for a digital asset for use in a scene; updating the scene model to include virtual points at locations in the scene model corresponding to locations in the scene and providing a waypoint scene model; providing augmented video data that includes one or more composite video frames with virtual points in the scene based on the waypoint scene model; and rendering the augmented video data onto an output device.

[0015] Any combination of the various embodiments and implementations disclosed herein may be used in further embodiments in a manner consistent with this disclosure. These and other embodiments and features can be understood from the following descriptions of the specific embodiments presented herein, in the accompanying drawings and claims. [Brief explanation of the drawing]

[0016] Embodiments will be described with reference to the attached drawings. In the drawings, similar reference numerals may indicate the same or functionally similar elements. The drawing in which an element first appears is generally indicated by the leftmost digit of the corresponding reference numeral.

[0017] [Figure 1] This is a block diagram of a digital prop tracking system.

[0018] [Figure 2]It is a block diagram of an example of a synthesis engine (or system) that can be used for digital synthesis during production (or shooting).

[0019] [Figure 3] It is a block diagram of an example of a computing platform.

[0020] [Figure 4] It is a block diagram of an example of a system for visualizing a video scene with one or more digital assets.

[0021] [Figure 5] It is a block diagram of an example of a waypoint system for animating digital assets in a scene.

[0022] [Figure 6] It is a block diagram of an example of a diorama pipeline for generating a diorama of a scene.

[0023] [Figure 7] It is an example of a method for operating digital assets through prop tracking.

[0024] [Figure 8] It is an example of a method for providing extended video data during production using digital asset prop tracking.

[0025] [Figure 9] It is an example of a method for providing synthesized video frames during production.

[0026] [Figure 10] It is an example of a method for providing extended video data during production.

[0027] [Figure 11]This is an example of a method for waypoint animation of digital assets in a scene.

[0028] [Figure 12] This is an example of a method for outputting a diorama of a scene.

[0029]

[0030] [Figure 13] This is a block diagram of a computing environment that can be used to perform one or more methods according to aspects of this disclosure.

[0031] [Figure 14] This is a block diagram of a cloud computing environment that can be used to perform one or more methods according to the embodiments of this disclosure. [Modes for carrying out the invention]

[0032] Next, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Similar elements in various drawings may be indicated by the same reference numerals for consistency. Furthermore, the following detailed description of embodiments of the present disclosure will mention a number of specific details in order to provide a more thorough understanding of the claimed subject matter. However, it will be apparent to those skilled in the art that embodiments disclosed herein may be practiced without these specific details. In other cases, well-known features are not described in detail to avoid unnecessarily complicating the description. Furthermore, it will be apparent to those skilled in the art that the scale of elements presented in the accompanying drawings may vary without departing from the scope of the present disclosure.

[0033] One or more embodiments of the present disclosure relating to systems and methods that can be used during (or before) the shooting of a scene (also known as production) are disclosed herein. When video data and other relevant scene information (necessary for digital compositing) arrive at a compositing device (also known as a compositor), in some cases the compositor may not use most of the relevant scene data. This may result in visual problems in the composite video data provided by the compositor during shooting. Because the compositor may receive data at different times (as different devices are used to provide data over a network), in some cases the movement of digital assets may not be properly synchronized with the video footage. Multiple visual problems that CGI synchronization problems may result in CGI assets appearing unconvincing or unrealistic during shooting. Such visual problems may include, for example, unnatural movement, timing problems, lighting and shadow problems, and / or scale and perspective problems.

[0034] Visual issues can arise during on-set filming, for example, during complex VFX scenes, because the timing and choreography of the action need to be synchronized to ensure that the director's vision or expectations are met. Additionally, visual issues can also make filming more difficult because they are distracting and can transmit production errors downstream (e.g., post-production), requiring the production team to rapidly expend resources to correct these errors. For example, if a particular VFX shot turns out not to be as planned, or if there are mistakes in the original footage, the VFX team will need to spend resources (e.g., computing resources) in post-production to identify the errors / mistakes and correct these issues.

[0035] Additionally, on-set filming is limited to viewing the scene or set through conventional first-person or third-person perspectives. However, coordinating actors and / or digital assets (e.g., CGI assets) and scene assets (or objects) is difficult and complex because the composition of objects, actors, and / or digital assets and / or interactions between them often involve the interaction of many different variables (e.g., location of objects, placement of actors, actions of actors, etc.). Conventional scene perspectives are limited in terms of scene visibility, which can cause filmmakers to miss errors that could be corrected during filming, further requiring the post-production team to find these mistakes. In some cases, the mistakes can be so significant that they may necessitate reshooting the scene. Examples disclosed herein utilize a data manager for digital compositing that can identify and correct the timing of appropriate scene-related information data, thereby mitigating or eliminating visual problems in composite video data (or augmented video data), such as during on-set filming. By implementing the systems and methods disclosed herein, real-time (or near-real-time (e.g., with a delay of less than a second)) can be achieved from the outset (e.g., during pre-production). The systems and methods disclosed herein combine different shooting techniques with data timing management of various data source devices (or systems) communicating over a communication network, so that they enable (or allow) filmmakers or productions (e.g., teams) to view the shooting results (with a delay of less than a second) and use those results to provide information about the acting and capture.

[0036] In several examples, as disclosed herein, techniques are disclosed for providing non-person viewpoints of a scene, for example, during filming, and by extension, on the set or at the filming location. According to one or more examples disclosed herein, non-person viewpoints of a scene (i.e., non-first-person or third-person viewpoints) can be provided (as a digital representation referred herein to as a diorama) so that the filmmaker (and / or other filmmakers) can visualize shots of the scene with digital asset information and make corrections in real time (e.g., during filming). In several examples, as disclosed herein, techniques for waypoint animation are described. For example, virtual locations defining virtual paths can be rendered in a scene and used to navigate digital assets throughout the scene. Virtual paths in a scene allow the filmmaker to visualize the movement of digital assets before video shooting, so that any scene errors can be corrected during production (rather than during post-production).

[0037] While systems and methods related to production are presented herein, the examples herein should not be construed as being limited solely to production. The systems and methods disclosed herein can be used for any purpose.

[0038] One or more embodiments of this disclosure also relate to digital prop tracking. Digital prop tracking can refer to the process of tracking a digital counterpart (digital asset) in three-dimensional (3D) space for a physical prop so that the position, movement, and / or orientation of the digital asset matches the position, movement, and / or orientation of the physical prop. Thus, digital prop tracking involves tracking a physical object in physical space (e.g., a real-world environment, e.g., a scene) and using the position, movement, and / or orientation of the physical object to control or define the position, movement, and / or orientation of a digital asset in virtual space (e.g., 3D space). The position, movement, and / or orientation of a physical object in physical space may be referred to as object space data, while the position, movement, and / or orientation of a digital asset in digital space may be referred to as digital asset space data. In some examples, digital prop tracking may include digital compositing. For example, a digital asset may be rendered in a shot of a scene (e.g., embedded in or integrated into a video frame of a scene) based on object space data. Therefore, augmented video data can be generated that includes digital assets with position, movement, and / or orientation based on object space data.

[0039] In some examples, the systems and methods disclosed herein may be used during (or before) the shooting (also known as production) of a scene. According to the examples disclosed herein, for example, props may be tracked during production without the use of markers and, consequently, marker-based systems. Through the use of the systems and / or methods disclosed herein, filmmakers may render (or have rendered) digital assets inserted into a scene in real time (i.e., during production) and animate digital assets based on the location and / or orientation of props. By using systems and / or methods such as those disclosed herein, filmmakers may visualize a scene with animations (or static digital assets) during production using compositing techniques (e.g., those disclosed herein) to see the movement and / or orientation of digital assets (e.g., objects in a scene, such as actors, physical objects) relative to the scene. Therefore, post-production costs are reduced through the use of the systems and / or methods disclosed herein because digital assets and / or props in a scene can be adjusted in real time, minimizing shooting errors. The systems and methods disclosed herein may be used during production.

[0040] Figure 1 is a block diagram of a digital prop tracking system 100, referred to herein as the “tracker system.” The tracker system 100 can be used for digital prop tracking during film production, i.e., during the shooting of a scene. Therefore, in some cases, the tracker system 100 can be used during film production (e.g., not after production). While examples of the tracker system 100 being used in film production are presented herein, the tracker system 100 may be used in other examples or applications. The tracker system 100 includes a prop tracker engine 102. The prop tracker engine 102 can be implemented as hardware, software, and / or a combination thereof. Therefore, in some examples, the prop tracker engine 102 can be implemented as machine-readable instructions that can be executed on a computing platform or device, for example, as disclosed herein. The prop tracker engine 102 may be used to calculate or determine digital asset space data for digital assets to be rendered in a scene, based on the position, movement, and / or orientation of the prop device 104, which may be referred to as prop space data 106.

[0041] The prop device 104 may be any device capable of providing its position, movement, and / or orientation. In some examples, the prop device 1204 may be any hardware, apparatus, or device designed with the specific purpose of providing prop space data, such as prop space data 106. In some examples, the prop device 104 is a dedicated device capable of providing prop space data. In some examples, the prop device 104 is a mobile device, such as a tablet or cellular device. Illustrative mobile devices may include, but are not limited to, the iPhone®. In some examples, the prop device 104 may include or be an inertial measurement unit (IMU).

[0042] The prop tracker engine 102 may receive prop space data 106 as shown in Figure 1. The prop tracker engine 102 may also receive digital asset space data 108 that can characterize the position, orientation, and / or movement of digital assets in the 3D model of the scene. In some cases, the 3D model of the scene may be rendered according to the examples disclosed herein. In other cases, different techniques may be used to render the 3D model of the scene.

[0043] The prop tracker engine 102 may update (e.g., change, adjust, set, etc.) the initial position, orientation, and / or movement of the digital asset in the 3D model of the scene based on the prop space data 106. Based on the update, the prop tracker engine 102 may provide updated digital asset space data 110. Thus, the prop tracker engine 102 may update the digital asset to have an orientation based on the prop space data 106. During filming (production), the prop device 104 may provide the prop space data 106, and the prop tracker engine 102 may use that data to manipulate the digital asset in the 3D model of the scene (or 3D space). In this way, the prop device 104 may be manipulated (e.g., by moving it around), and the digital asset in the 3D model will follow the manipulation of the prop device 104. For example, if the prop device 104 moves or is moved, the digital asset in the 3D model will also move based on the movement of the prop device 104.

[0044] In some examples, the tracker system 100 includes a digital compositor 112. The digital compositor 112 may provide augmented video data 114 based on updated digital asset space data 110 and video data 116, which includes video frames of the scene. The video data 116 may be provided from a video source 118 (e.g., a camera, a portable device (e.g., a mobile phone, a tablet, etc.)). The video source 118 may be used to capture the scene. The digital compositor 112 may insert digital assets into one or more video frames of the video frames of the scene in the video data 116 to constitute an augmented video frame corresponding to the augmented video data 114. The augmented video data 114 may be rendered or made to be rendered on an output device 120 (e.g., a display, a heads-up display, a portable device (e.g., a tablet, a mobile phone, etc.), a viewing screen, a television, etc.). In some examples, the video source 118 includes an output device 120.

[0045] Therefore, the tracker system 100 can be used for digital prop tracking through the use of the prop device 104 and the prop tracker engine 102, eliminating the need for marker-based systems and enabling improved film production (for example, by minimizing errors) while minimizing post-production costs.

[0046] Figure 2 is a block diagram of an example of a compositing engine 200 that may be used for digital compositing during production. The compositing engine 200 may run on a computing device such as a computing platform (e.g., computing platform 300 as shown in Figure 3, or computing platform 410 as shown in Figure 4) or on a computing system (e.g., computing system 1300 as shown in Figure 13). The compositing engine 200 may include a data loader 202 for loading input data 206 into the system 200. A data manager 242 may partition a cache memory space 204 for storing input data 206 corresponding to data used by the digital compositor 208 of the compositing engine 200 for generating augmented video data 210. In some examples, the digital compositor 208 is the digital compositor 112 as shown in Figure 1. Thus, in the example of Figure 2, one or more examples of Figure 1 may be referenced. For example, the data manager 242 (or data loader 202) may partition the cache memory space 204 into multiple cache locations (or partitions) to store video frames 212, scene-related information frames 214, and digital asset data 216 for a scene. In some examples, the video frames 212 may be provided as part of the video data 116, as shown in Figure 1. Thus, in some examples, a video source 118, as shown in Figure 1, may provide the video frames 212.

[0047] The data manager 242 (or data loader 202) may store video frames 212 in a first set of partitions in the cache memory space 204, scene-related information frames 214 in a second set of partitions in the cache memory space 204, and digital asset data 216 in a third set of partitions in the cache memory space 204. In some examples, the first set of partitions may include even and odd video frame cache locations, and each video frame of video frames 212 may be stored in one of the even and odd video frame cache locations based on the respective frame number of the video frame. In some examples, video frames 212 are provided during production, and therefore in real time while the scene is being filmed, from a video camera (e.g., a digital video camera). In other examples, video frames 212 are provided from a storage device, for example, during post-production. Therefore, in some cases, the compositing engine 200 may be used during post-production to provide augmented video data 210. In some examples, augmented video data 210 is augmented video data 114, as shown in Figure 1.

[0048] Scene-related information frames 214 may be provided over a network from different types of scene capture devices used to capture information related to the shooting of a scene being captured by a video camera. Video frames 212 are provided by the video camera. The network may be a wired and / or wireless network. Each scene-related information frame 214 may include a timestamp generated based on the local clock of the respective scene capture device. Video frames may be provided by the video camera at a different frequency than that which is provided by the scene capture devices for the scene-related information frames 214.

[0049] As used herein, the term “frame” may represent or refer to a discrete unit of data. Therefore, in some examples, the term “frame” as used herein may refer to a video frame (e.g., one or more still images captured or generated by a video camera) or a data frame (e.g., a pose data frame). For example, each scene-related information frame 214 may contain camera pose data for a video camera providing a video frame 212. Pose data for a video camera typically refers to the camera’s position and orientation at a specific point in time, represented in three-dimensional (3D) space. It typically includes information such as the camera’s location, orientation, and field of view, and other relevant parameters such as focal length, aperture, and shutter speed. In some examples, when the cache memory space 204 is partitioned, the digital compositor 208 may initiate digital compositing, which may include applying VFX to at least one video frame (e.g., inserting a CGI asset). In some examples, the digital compositor 208 may include, or communicate with, a prop tracker engine 102, as shown in Figure 1. The prop tracker engine 102 may be used to provide spatial data (e.g., position, orientation, and / or movement) about digital assets that are synthesized by the digital compositor 208 to provide augmented video data 210.

[0050] For example, the digital compositor 208 may issue a video frame request 218 for a video frame among the video frames 212. The compositing engine 200 includes a data manager 242 which includes a video frame retriever 220 that can process the video frame request 218 to provide a selected video frame 222. The video frame retriever 220 may provide the selected video frame 222 based on video frame latency data 224. The video frame latency data 224 may specify the number of video frames to be stored in the cache memory space 204 for digital compositing such that visual problems (e.g., those described herein) present in the augmented video data 210 are minimized or absent. Thus, the video frame latency data 224 may specify how quickly the composite video frames (which are part of the augmented video data 210) are rendered on the display. For example, if the video frame latency data 224 indicates a 3-frame latency, the video frame retriever 220 may provide the third stored frame to the digital compositor 208 as the selected video frame 222.

[0051] The selected video frame 222 may also be provided to the scene-related frame retriever 226 of the data manager 242. The scene-related frame retriever 226 may extract a timecode from the selected video frame 222. The scene-related frame retriever 226 may use the timecode to identify the scene-related information frame of the scene-related information frame 214. In some examples, the scene-related information frame is identified by evaluating the extracted timecode against the timestamp of the scene-related information frame 214 and selecting the scene-related information frame that is temporally or geographically closest to the timecode of the selected video frame 222. Other techniques for identifying the scene-related information frame of the scene-related information frame 214, such as those disclosed herein, may also be used. The scene-related frame retriever 226 may provide a scene-related information frame as a selected scene-related information frame 228, as shown in Figure 1, which may be received by the digital compositor 208. The compositing engine 200 also includes a digital asset retriever 230 that can retrieve relevant digital asset data about a scene stored in a cache memory space 204 and provide this data to the digital compositor 208 as selected digital asset data 232. In some examples, the data manager 242 includes the digital asset retriever 230.

[0052] In some examples, the digital compositor 208 may receive a virtual environment of a scene and provide a 3D model 234 of the scene. The 3D model 234 may be used by the digital compositor 208 to generate augmented video data 210. In some examples, the 3D model 234 may be generated by a model generator 244 based on environmental data 246. The environmental data 246 may be provided by one or more environmental sensors (not shown in Figure 2). One or more environmental sensors may include, for example, infrared systems, light detection and ranging (LIDAR) systems, thermal imaging systems, ultrasonic systems, stereoscopic systems, RGB cameras, optical systems, or any device / sensor systems currently known in the art, and combinations thereof, that can measure / capture depth and distance data of objects in the environment. Thus, the environmental data 246 may characterize depth and distance data of objects in the environment (e.g., a scene).

[0053] In some cases, a QR code (registered trademark) (or other image mark) may be used to establish a common origin in the coordinate system of a real environment (e.g., a scene). Multiple QR codes (registered trademarks) may be used to define the location of the coordinate system origin for the real environment. The QR codes (registered trademarks) may have different orientations to improve the X, Y, and Z information of the coordinate system. Information characterizing the coordinate system and the common origin may be provided as coordinate system data 248. For example, a device (e.g., a portable device such as a mobile phone or tablet) may be used to record the location / position / orientation of each QR code (registered trademark) in the coordinate system of the real environment and provide this data as coordinate system data 248. The model generator 244 receives the coordinate system data 248 and may register virtual markers in the 3D model 234 as starting locations for digital assets to be inserted into the 3D model 234. Thus, the model generator 244 may insert digital assets into the 3D model 234 at the virtual markers registered in the 3D model 234. In some examples, graphical instructions are provided in the 3D model 234 indicating where the digital assets are to be placed within the 3D model 234. Thus, the coordinate system data 248 can provide the transformation (or registration) of the digital assets into the actual environment.

[0054] The 3D model 234 may include a virtual camera representing a video camera used to capture the scene, and digital assets (e.g., CGI assets) to be inserted into the scene. The digital compositor 208 may output a composite video frame, which may be provided as extended video data 210 based on the 3D model 234, as well as a selected video frame and a scene-related information frame, and in some cases, digital asset data provided from the cache memory space 204. Thus, each composite video frame may be provided based on the selected video frame 222, the selected scene-related information frame 228, and in some cases, the selected digital asset data 232.

[0055] In some cases, the 3D model 234 may be updated by the digital compositor 208 based on a selected scene-related information frame 228. For example, the digital compositor 208 may update the pose of the virtual camera in the 3D model 234 based on the closest pose data (reflected as the selected scene-related information frame 228) captured or measured for the video camera. As a further example, if the position of the video camera changes to a new camera position in the real-world environment (from the previous camera position), the 3D model 234 may be updated to reflect the new position of the video camera, and the position of any CGI assets in the 3D model 234 may be updated to match the new video camera position based on the selected scene-related information frame 228.

[0056] The digital compositor 208 may, for example, adjust the position, orientation, and / or scale of the CGI asset in the 3D model 234 so that the CGI asset appears in the exact location in the 3D model 234 relative to real-world footage (e.g., a real-world environment). The digital compositor 208 may provide augmented video data 210 using the updated 3D model. For example, the digital compositor 208 may overlay the CGI asset on a selected video frame so that the CGI asset appears in the updated position in the scene based on the updated 3D model 234, providing a composite video frame.

[0057] Since the scene-related information frame 214 is provided over a network (e.g., LAN, WAN, and / or the Internet), network latency may occur in the scene-related information frame 214. To compensate for the network latency effect on the scene-related information frame 214, the synthesis engine 200 may include a network analyzer 236. The network analyzer 236 can analyze the network through which the scene capture device provides the scene-related information frame 214 (e.g., network 253 shown in Figure 2, or network 435 shown in Figure 4) to determine the network latency of the network. In some examples, the data manager 242 includes a network analyzer 236. The network analyzer 236 may output network latency data 238 that characterizes the network latency of the network, which can be received by the scene-related frame retriever 226. The scene-related frame retriever 226 may update the timestamp of each scene-related information frame 214 based on the network latency of the network through which the scene-related information frame 214 is provided, based on the network latency of the network through which the scene-related information frame 214 is provided, based on the network latency data 238. Therefore, in some examples, the scene-related information frame identified by the scene-related frame retriever 226 can be obtained based on an evaluation of the adjusted timestamp (with respect to network latency) of the scene-related information frame 214 and the timecode of the video frame.

[0058] In additional or alternative examples, the timestamp of each scene-related information frame 214 may be updated based on a frame delta value 240. The frame delta value 240 may represent the amount of time between each scene-related information frame provided by each scene capture device. Since the internal clocks of each scene capture device are internally synchronized, the delta values ​​between timestamps of adjacent scene-related information frames from the same scene capture device may be approximately the same over a period of time. The scene-related frame retriever 226 may use the frame delta value to adjust the timestamps of the scene-related information frames provided by each scene capture device to compensate for the clock drift of each scene capture device.

[0059] By configuring the compositing engine 200 based on video frame latency data 224, and in some cases further based on network latency data 238 and / or frame delta values ​​240, the compositing engine 200 can provide extended shots of the scene at a given frame latency (e.g., 3 frames latency) sufficient to allow shots of the scene to be visualized during production with little or no visual problems. This is because the compositing engine 200 uses the data manager 242 to identify relevant scene-related information frames and correct the timing of these frames according to a timestamp adjustment scheme (such as those disclosed herein) to enable accurate alignment of non-frame data from various and dissimilar scene sources (e.g., scene capture devices) to video frames.

[0060] Figure 3 is a block diagram of an example computing platform 300 that may be used for digital synthesis. The computing platform 300 described herein may synthesize video frames and digital assets to provide a composite stream of video frames for rendering on a display. The computing platform 300 may include a processor unit 302. The processor unit 302 may be implemented on a single integrated circuit (IC) (e.g., die) or using multiple ICs. The processor unit 302 may be a general-purpose processor, a dedicated processor, an application-specific processor, an embedded processor, or similar. The processor unit 302 may include N processors, which may be collectively referred to as a CPU processing pool 306 in the example of Figure 3. In some cases, each processor may include multiple cores.

[0061] The processor unit 302 may include cache memory, which in the example in Figure 3 may be collectively referred to as the CPU memory space 308. The cache memory may include, for example, L1, L2, and / or L3 caches. The CPU memory space 308 may be a logical representation of the cache memory of the processor unit 302 that can be partitioned, as disclosed herein. In some examples, the CPU memory space 308 is distributed across multiple dies. For the sake of clarity and brevity, not all components of the processor unit 302 (e.g., functional blocks such as the memory controller, system interface, input / output (I / O) device controller, control unit, and arithmetic logic unit (ALU)) are shown in the example in Figure 3.

[0062] In some cases, the CPU memory space 308 includes a cache (or subset of caches) for each processor (or processor core). In further examples, the CPU memory space 308 may include a shared cache (e.g., an L3 cache) that can be shared between processors and / or cores. The CPU memory space 308 may be partitioned by the synthesis engine 316 for the real-time generation of augmented video data 312. The synthesis engine 316 may correspond to the synthesis engine 200 shown in Figure 2. In some examples, the synthesis engine 316 may include or communicate with a prop tracker engine 102, as shown in Figure 1. Thus, in the example in Figure 3, one or more examples from Figures 1-2 may be referenced. The prop tracker engine 102 may be used to provide spatial data (e.g., position, orientation, and / or movement) about the digital assets synthesized by the synthesis engine 316 to provide the augmented video data 312. In some examples, the augmented video data 312 corresponds to augmented video data 114 as shown in Figure 1, or augmented video data 210 as shown in Figure 2. The extended video data 312 may include one or more composite video frames that can be rendered on the display 314.

[0063] In some cases, the real-time generation of the extended video data 312 may correspond to the generation of video frames with a latency of less than or equal to three frames. One or more processors of the CPU processing pool 306 may read and execute program instructions (or parts thereof) representing the synthesis engine 316. The synthesis engine 316 may run on a processor unit 302 to implement at least some of the functions disclosed herein. In some examples, the synthesis engine 316 represents an application (e.g., a software application) that may run on the computing platform 300.

[0064] The computing platform 300 may include a graphics processing unit (GPU) 318 for providing enhanced video data 312. For example, the GPU 318 may include a GPU processing pool 320 that can provide synthetic data that can be transformed to provide enhanced video data 312. For example, the GPU processing pool 320 may include multiple processing units. The GPU 318 may include a GPU memory space 322. For example, the GPU memory space 322 may include similar and / or different types of memory, such as local memory, shared memory, global memory (e.g., DRAM), texture memory, and / or constant memory. Thus, the GPU memory space 322 may include cache memory such as an L2 cache. For the purpose of clarity and brevity, not all components of the GPU 318 (e.g., interconnects, I / O interfaces, registers, ALU, texture units, load / store units, other fixed-function blocks, etc.) are shown in the example in Figure 3. As the CPU memory space 308, the GPU memory space 322 may be a logical representation of the cache memory of a partitionable GPU 318, as disclosed herein. In some examples, the CPU memory space 308 and / or the GPU memory space 322 may correspond to the cache memory space 204, as shown in Figure 2.

[0065] For example, the computing platform 300 may include multiple interfaces, such as a video interface 324 and an I / O interface 326. The video interface 324 may be used to receive video frames 328, which may be provided to the GPU 318 via the bus 330. In some examples, the video frames 328 are provided by a video capture device 332, as shown in Figure 1. The video frames 328 may represent images of a scene. The video frames 328 may also include metadata such as a timecode (or timestamp), frame rate, resolution, and / or other information describing the video frame 328 itself.

[0066] In some examples, the video capture device 332 may be implemented as a video capture card and therefore as part of the computing platform 300. In other examples, the video capture device 332 may be implemented as a standalone device capable of communicating with the computing platform 300 (e.g., using wired and / or wireless media). The video capture device 332 may receive video data (e.g., video data 436 as shown in Figure 4) from a video camera (e.g., video camera 406, or from another video source (e.g., storing recorded video data in the cloud, disk, or another storage location)). In some examples, the video camera is a digital video camera such as a digital motion picture camera (e.g., Arri Alexa developed by Arri). In other examples, the video camera may correspond to the camera of a portable device (e.g., a mobile phone, tablet, etc.). In some examples, video frame 328 may correspond to one of the video frames 212, as shown in Figure 2.

[0067] In some examples, the I / O interface 326 may be used to receive scene-related information frames 334 and digital asset data 336. Scene-related information frames 334 may include, for example, face tracking data, body tracking data, asset pose data, and / or camera pose data. In some examples, scene-related information frames 334 may correspond to one of the scene-related information frames 214, as shown in Figure 2. The type of scene-related information frame 334 may depend on the type of scene capture device used for the scene. For example, the I / O interface 326 may include a network interface that can be used to communicate with external devices, systems, and / or servers to receive scene-related information frames 334 and digital asset data 336. Scene-related information frames 334 and / or digital asset data 336 may be provided via the network 353, as shown in Figure 3.

[0068] Network 353 may include one or more networks and / or the Internet. One or more networks may include, for example, a local area network (LAN) or a wide area network (WAN). In additional or alternative examples, I / O interfaces 326 may include hard disk drive interfaces, magnetic disk drive interfaces, and optical drive interfaces, or input device interfaces, which may be used to enable scene-related information frames 334 and digital asset data 336 to be loaded (e.g., stored) into system memory 339. System memory 339 may represent one or more memory devices, such as random access memory (RAM) devices, which may include static and / or dynamic RAM devices. One or more memory devices may include, for example, double data rate 2 (DDR2) devices, double data rate 3 (DDR3) devices, double data rate 4 (DDR4) devices, low power DDR3 (LPDDR3) devices, low power DDR4 (LPDDR4) devices, wide I / O 2 (WIO2) devices, high bandwidth memory (HBM) dynamic random access memory (DRAM) devices, HBM2 DRAM (HBM2 DRAM) devices, double data rate 5 (DDR5) devices, and low power DDR5 (LPDDR5) devices (e.g., mobile DDR). In some examples, one or more memory devices may include DDR SDRAM type devices.

[0069] The processor unit 302, GPU 318, video interface 324, I / O interface 326, and memory 339 may be connected to a system bus 330. The system bus 330 may represent a communication / transmission medium through which data can be provided between components, as shown in Figure 3. In some cases, the system bus 330 includes any number of buses, such as a backside bus, a frontside (system) bus, a peripheral component interconnect (PCI), or a PCIe bus. The system bus 330 may include corresponding circuits and / or devices that enable components to communicate and exchange data to provide augmented video data 312 based on video frames 328, scene-related information frames 334, and digital asset data 336.

[0070] In some examples, the processor unit 302 and / or the GPU 318 may convert the video frame 328 to a texture file, depending on the implementation and design of the computing platform 300. For example, the video frame 328 may be decoded and decompressed (e.g., by the processor unit 302 or the GPU 318) and then converted to a texture file format that can be loaded and rendered by the GPU 318 along with VFX, according to examples disclosed herein. In examples where the processor unit 302 has sufficient computing power, the processor unit 302 may implement the conversion from video frame to texture file, and the GPU 318 may use the texture file for the video frame 328 for image rendering. In some examples, the conversion from video frame to texture file is performed by both the processor unit 302 and the GPU 318, each handling different operations of the conversion process. For example, the processor unit 302 may be responsible for decoding and decompressing the video frame 328, while the GPU 318 may be responsible for converting the decoded video frame to a texture file format and image rendering.

[0071] Overall, the specific implementation of the video frame to texture file conversion process may depend on the system architecture, hardware capabilities, and software design of the computing platform 300. The compositing engine 316 may be used to implement digital compositing to provide a composite video frame based on the video frame 328, scene-related information frame 334, and digital asset data 336, by controlling the components of the computing platform 300. For example, the compositing engine 316 may be implemented as machine-readable instructions that can be executed by the CPU processing pool 306 to control when data is provided to and from the GPU 318, and / or loaded / retrieved from and processed by the CPU and / or GPU memory spaces 308 and 322.

[0072] During CGI filming, it is desirable to know the location and / or behavior (movement) of CGI assets and other elements in relation to each other in the scene. That is, during production, the director or producer may want to know the position and / or location of CGI assets so that other elements (e.g., actors, props, etc.) or the CGI assets themselves can be adjusted to create a seamless integration of the two. For example, the director may want CGI assets to move and behave in a manner consistent with live-action footage. Knowledge of where CGI assets are located and how they behave is important for the director during filming (e.g., production) because it can minimize retakes and further reduce post-production time and costs. For example, knowledge of how CGI assets behave in a scene can help the director adjust the position or behavior of actors relative to the CGI assets. Generally, to help actors know where to look and how to interact with CGI assets, filmmakers often use on-set visual cues such as reference objects, markers, targets, and verbal cues, and in some cases, show actors a pre-visualization of the scene. Pre-visualization (or "pre-vis") is a technique used in filmmaking to create a rough, animated version of the final sequence, giving actors a general idea of ​​what the CGI assets will look like, where they will be located, and how they will behave before the scene is shot.

[0073] Alternative pre-visualization techniques have been developed that allow directors to view assets in real time on a display during filming. For example, a director may use a pre-visualization system configured to provide a composite video feed with CGI assets incorporated while the scene is being filmed. An example of such a device / system is described in U.S. Patent Application No. 17 / 410,479, filed on 24 August 2021 and issued as U.S. Patent No. 11,682,175, which is incorporated herein by reference in its entirety. The pre-visualization system allows the director to have a "good enough" shot of the scene with the CGI assets, and as a result, production issues (e.g., where actors should actually look, where CGI assets should be positioned, how CGI assets should behave, etc.) can be corrected during filming, and therefore at the production stage. The pre-visualization system may implement digital compositing to provide a composite video frame with the CGI assets incorporated.

[0074] During digital compositing, the movement of CGI assets is synchronized with the video footage in film or video, and as a result, when the composite video frames are played back on the display, the movement of the CGI assets is coordinated with the movement of the live-action footage. A pre-visualization system may be configured to provide CGI asset-live-action footage synchronization. For example, a pre-visualization system may implement compositing techniques / operations for combining different elements such as live-action footage, CGI assets, CGI asset movement, and other special effects into augmented video data (one or more composite video frames). The augmented video data can be rendered and visualized on the display by the user (e.g., the director).

[0075] In some cases, data for generating augmented video data may arrive at the pre-visualization system's compositing device (also known as a compositor) at different times, for example, during filming. Data may arrive at the compositing device at different times via a network (such as network 353 shown in Figure 3). Additionally, data may be generated by different scene capture devices based on internal or local clocks (which may drift over time). Furthermore, in some cases, the scene capture devices may not be time-synchronized (or jammed) with the video camera (e.g., connected to a central timecode generator). In some scenarios, one or more scene capture devices may not support jamming. Therefore, when video frames and other scene data arrive at the compositor, the compositor may not use most of the relevant scene-related information data (frames) for digital compositing, which can result in visual problems in the composite video data. Because the compositor may receive data at different times, in some cases, the movement of CGI assets may not be properly synchronized with the video footage, which can result in several visual problems that make CGI assets appear unconvincing or unrealistic during filming.

[0076] According to examples disclosed herein, the compositing engine 316 may be configured to mitigate or (in some cases) eliminate visual problems in the augmented video data 312. The compositing engine 316 may control which scene-related information frames 334 are used for compositing to provide augmented video data 312 with reduced or absent visual problems. Thus, in some examples, the compositing engine may employ a data manager 242 as shown in Figure 2.

[0077] For example, during filming (production), for digital compositing, a video camera may provide video frames 328 to a computing platform 300, where a video interface 324 may communicate with a GPU 318 via a bus 330. In some examples, video frames 328 are loaded into CPU memory space 308 and / or GPU memory space 322 for VFX processing (e.g., CGI insertion). In some examples, CPU memory space 308 may include even and odd video frame cache locations 338-340, a scene cache location 342, and a digital asset cache location 344.

[0078] The processor unit 302 or GPU 318 may load video frame 328 into one of the even or odd video frame cache locations 338-340 based on the timecode of video frame 328. For example, if the timecode for video frame 328 appears as 01:23:45:12, then since "12" is an even number, the processor unit 302 or GPU 318 may load video frame 328 into the even video frame cache location 338. The timecode for a video frame is a numerical representation of the specific time at which the frame appears in the video, and includes hours, minutes, seconds, and frames. A video frame 328 loaded into an even video frame cache location may be referred to as an even video frame, and therefore a video frame loaded into an odd video frame cache location may be referred to as an odd video frame.

[0079] In some examples, the GPU 318 (or in combination with the processor unit 302) may convert the video frame 328 into a texture file format which may be referred to as texture frame data. The texture frame data may be stored in the GPU memory space 322. For example, the compositing engine 316 may cause the GPU processing pool 320 (or in combination with the CPU processing pool 306) to convert the video frame 328 and provide the texture frame data. The compositing engine 316 may cause threads to be assigned to processing elements of the GPU 318 and / or the processor unit 302 for the conversion of the video frame 328. Once converted, the texture frame data may be stored in an even-numbered video frame cache location 338 (based on the timecode for the video frame 328) and may be referred to as even-numbered texture frame data.

[0080] Subsequently received video frames may be processed in the same or similar manner as disclosed herein and stored in odd video frame cache locations 340 instead of even video frame cache locations 338. In some examples, when subsequent video frames are converted to texture frame data, the texture frame data may be stored in odd video frame cache locations 340 (based on the timecode for the subsequent video frame) and may be referred to as odd texture frame data. For example, if the timecode for the subsequent video frame is "13", since "13" is odd, the processor unit 302 or GPU 318 may load the subsequent video frame (or odd texture frame data) into odd video frame cache location 340.

[0081] In some cases, the compositing engine 316 may receive video frame latency data 317. The video frame latency data 317 may specify a number of video frames to be stored in the CPU or GPU memory space 308 or 322 (before digital compositing) for digital compositing such that there are minimal or no visual problems present in the extended video data 312. Thus, the video frame latency data 317 may specify how quickly the composite video frames are rendered on the display 314. Therefore, the number of frames specified by the video frame latency data 317 may be such that there are few or no visual problems in the extended video data 312. In an unrestricted example, the video frame latency data 317 indicates 3 video frames. Based on the video frame latency data 317, the compositing engine 316 may select or identify video frames (or texture frame data) for digital compositing. For example, if the video frame latency data 317 indicates three video frames, the synthesis engine 316 may select a third video frame (stored in memory space) for digital synthesis.

[0082] The time or amount of time required to generate texture frame data may vary from video frame to video frame due to the processing or loading requirements of the computing platform 300. The amount of processing required may vary based on the complexity of the frame (e.g., if it has more complex visual content). Therefore, the video frame conversion time (the amount of time required to generate texture frame data) may vary or change from video frame to video frame. Additionally, odd and even video frames (or odd and even texture frame data) may be loaded (for further processing) into their corresponding cache memory locations before the scene-related information frame 334 reaches the computing platform 300. In some cases, by the time the third video frame arrives and is stored (in some cases converted to the corresponding texture file and then stored), the scene-related information frame 334 (generated at approximately the same time as the video frame 328) has already reached the computing platform 300. The compositing engine 316 selects the third video frame for digital compositing, for example, thereby providing sufficient time for the scene-related information frame 334 to reach the computing platform 300.

[0083] In some examples, the synthesis engine 316 may partition the GPU memory space 322. For example, if the GPU 318 has sufficient computing power, the GPU memory space 322 may be partitioned in the same or similar manner as the CPU memory space 308. Thus, in some examples, the GPU memory space 322 may include even and odd video frame cache locations 346-348, a scene cache location 350, and a digital asset cache location 352. Video frames (or corresponding texture frame data) may be stored in the even and odd video frame cache locations 346-348 in the same or similar manner as disclosed herein. Thus, in some implementations, the write of even / odd video frames (or corresponding texture files, if converted by the GPU processing pool 320) may be removed, and the video frame (or texture file) data may instead be stored in the GPU memory space 322. In an example where the corresponding video frame needs to be stored on an external device, the GPU318 may store the corresponding video frame in the CPU memory space 308, and as a result, the corresponding video frame can be provided to the external device.

[0084] When the scene-related information frame 334 is received, the compositing engine 316 may cause the scene-related information frame 334 to be stored in memory 339, scene cache location 342, or scene cache location 350 (for later retrieval). Following the example in Figure 3, the compositing engine 316 may cause the processor unit 302 or GPU 318 to select a third video frame (or third texture frame data) from CPU memory space 308 or GPU memory space 322 in response to receiving the scene-related information frame 334. The compositing engine 316 may cause the scene-related information frame 334 to be loaded into either scene cache location 342 or scene cache location 350. The compositing engine 316 may also cause the digital asset data 336 to be loaded into either digital asset cache location 344 or digital asset cache location 352 for digital compositing.

[0085] In some examples, the compositing engine 316 may cause a processor unit 302 or a GPU 318 to generate a virtual environment (3D model) 354 that can represent a scene, and therefore a real-world environment. The compositing engine 316 (or another system) may insert CGI assets into the 3D model 354 based on digital asset data 336. In other examples, a different system may be used to create the virtual environment, which may be provided to a computing platform 300. The process of creating the 3D model 354 may include creating objects and elements in the scene, such as buildings, props, characters, and special effects. The 3D model 354 may include a virtual camera that may correspond to a video camera used to capture the scene. The virtual camera may be adjusted based on camera pose data. The location of the CGI assets is known in the 3D model 354, and since the virtual environment represents a scene, the CGI assets may be inserted into the scene during digital compositing based on digital asset data 336. In some examples, the 3D model 354 corresponds to the 3D model 234 as shown in Figure 2.

[0086] In some examples, a scene capture device providing scene-related information frames 334 may record or measure related data at a different frame rate than that at which the video camera provides video frames. For example, a scene capture device may provide scene-related information frames at a higher frame rate (e.g., about 60 FPS) than a digital camera (e.g., about 30 FPS). Because the scene capture device provides scene-related information frames 334 at a different frequency than that at which the video camera provides video frames 328, the use of inaccurate scene-related information may result in visual problems in the augmented video data 312. According to the examples disclosed herein, the compositing engine 316 may identify most appropriate or accurate scene-related information frames for digital compositing (e.g., by the digital compositor 208, as shown in Figure 2). In each example, the scene-related information frame 334 provided by the corresponding scene capture device may include a timestamp that indicates when the scene-related information frame 334 was generated (based on the local clock of the scene capture device).

[0087] For example, the compositing engine 316 may cause the processor unit 302 or GPU 318 to update the 3D model 354 based on the scene-related information frame 334. The compositing engine 316 causes the processor unit 302 or GPU 318 to evaluate the scene-related information frame 334 provided at different points in time by the corresponding scene capture device and identify the scene-related information frame 334 with the timestamp closest to the timecode of the selected video frame (e.g., the third video frame). In some examples, the compositing engine 316 may cause the processor unit 302 or GPU 318 to convert the timecode for the selected video frame to a timestamp. For example, the compositing engine 316 may cause the processor unit 302 or GPU 318 to convert the timecode based on the frame rate of the video camera and the number of frames in the selected video frame. For example, if the timecode for the 14th frame of the video is provided (by the video camera) at a frame rate of 30 FPS, the calculation would be timestamp = (14-1) / 30 = 0.367 seconds. By using the calculated timestamp for the selected video frame, the synthesis engine 316 can cause the processor unit 302 or GPU 318 to identify a scene-related information frame 334 with a timestamp closest to the timestamp of the selected video frame, which may be referred to as the closest scene-related information frame.

[0088] The synthesis engine 316 may cause the processor unit 302 or GPU 318 to update the 3D model 354 based on the nearest scene-related information frame. For example, the synthesis engine 316 may cause the processor unit 302 or GPU 318 to update the pose of the virtual camera in the 3D model 354 based on the nearest pose data captured or measured for a video camera. When each instance of the scene-related information frame 334 reaches the computing platform 300, the scene-related information frame 334 may be stored in memory 339 or the corresponding cache memory space (e.g., one of the CPU or GPU memory spaces 308 and 322).

[0089] The compositing engine 316 can cause the processor unit 302 or GPU 318 to identify the nearest scene-related information frame for generating a composite video frame. For example, if the position of a video camera in a real-world environment changes (from a previous camera position) to a new camera position, the 3D model 354 may be updated to reflect the new position of the video camera, and the position of any CGI assets in the 3D model 354 may be updated to match the new video camera position based on the nearest scene-related information frame. The compositing engine 316 can cause the processor unit 302 or GPU 318 to adjust, for example, the position, orientation, and / or scale of CGI assets in the 3D model 354 so that the CGI assets appear in the correct location in the 3D model 354 relative to the real-world image (e.g., the real-world environment).

[0090] For example, if the timestamp for a selected video frame (or selected texture frame data) is closer to the first timestamp value of the first scene-related information frame (e.g., in terms of distance, time, percentage, etc.) than the second timestamp value of the second scene-related information frame, the synthesis engine 316 may use the scene-related information frame 334 (e.g., one of the first or second scene-related information frames) with the given timestamp value as the closest scene-related information frame. In some examples, the synthesis engine 316 may cause the processor unit 302 or GPU 318 to use interpolated scene-related data to update the 3D model 354. For example, if the timestamp for a selected video frame is a given distance, time value, or percentage from the first and second timestamp values, the synthesis engine 316 may cause the processor unit 302 or GPU 318 to interpolate scene-related data based on the first and second scene-related data sets. The interpolated scene-related data may be used as the closest scene-related information frame to update the 3D model 354.

[0091] The processor unit 302 or GPU 318 may use the updated 3D model for digital compositing. In some examples, the 3D model is updated as part of the digital compositing. For example, the processor unit 302 or GPU 318 may overlay a CGI asset onto a selected video frame so that the CGI asset appears in the scene at an updated position based on the updated 3D model 354. In some examples, the compositing engine 316 may cause the processor unit 302 or GPU 318 to convert texture file data for the selected video frame back into a video frame format for blending (e.g., overlaying) the CGI on the selected video frame. Regardless of the technique or method used for digital compositing, the processor unit 302 or GPU 318 may provide a composite video frame for the selected video frame for rendering on the display 314. When inserting a CGI asset into a video frame, the compositing engine 316 may use various techniques to ensure that the CGI asset matches the lighting, shadows, reflections, and other visual characteristics of the live-action footage. This may involve adjusting color grading, applying depth of field or motion blur effects, and adjusting the transparency or blend mode of CGI assets to ensure they look like a natural part of the scene.

[0092] Since the scene-related information frame 334 is provided over the network 253 (e.g., LAN, WAN, and / or the Internet), network latency may occur in the scene-related information frame 334. Network latency may add an additional timing delay to the scene-related information frame 334 so that the synthesis engine 316 may select an inaccurate scene-related information frame. To compensate for the network latency effect on the scene-related information frame 334, the synthesis engine 316 may communicate using the network interface (corresponding to one of the I / O interfaces 326) and send a ping to each scene capture device that provided the scene-related information frame 334. The synthesis engine 316 may determine the amount of time required for the ping data to travel from the computing platform 300 to the scene capture device and back. This amount of time may correspond to network latency. The synthesis engine 316 may adjust the timestamp for the scene-related information frame 334 (from a given scene capture device) based on the determined network latency when it reaches the computing platform 300 over the network 353. The adjusted timestamps can then be evaluated in the same or similar manner as disclosed herein to identify the nearest scene-related information frame for the selected video frame (or selected texture frame data) for digital synthesis.

[0093] In some cases, a scene capture device may be time-synchronized with a video camera. For example, a scene capture device and a video camera may be configured using a timecode generator. Each timecode generator may produce a unique timecode signal that can be synchronized with a master timecode source. The master timecode source may be a standalone device such as a timecode generator. Once the timecode generator is synchronized, the scene capture device may provide a scene-related information frame 334 with a timestamp (or timecode) synchronized with the master timecode source. Similarly, the video camera may provide a video frame 328 with a timecode synchronized with the master timecode source. However, since the clock of the scene capture device is synchronized with the clock of its timecode generator, and clocks are known to drift over time, the timestamp (or timecode) of each instance of the scene-related information frame 334 provided by the scene capture device will differ from the expected time (if there is no drift).

[0094] Additionally, network latency may affect the timestamp (or timecode) of the scene-related information frame 334. This occurs as the data travels from the scene capture device to the computing platform 300 via the network 353. Since the compositing engine 316 is configured based on the video frame latency data 317, the compositing engine 316 may identify or select a video frame (or texture frame data) where sufficient time is provided for the scene-related information frame 334 to arrive from the scene capture device to the computing platform 300. According to the examples disclosed herein, the compositing engine 316 may identify the nearest scene-related information frame for the selected video frame (or selected texture frame data) in the same or similar manner as disclosed herein for digital compositing.

[0095] In some cases, the synthesis engine 316 causes the processor unit 302 or GPU 318 to adjust the timestamps of scene-related information frames 334 based on frame delta values. The frame delta value may represent the amount of time between each scene-related information frame provided by the scene capture device. Since the internal clocks of the scene capture device are internally synchronized, the delta values ​​between timestamps of adjacent scene-related information frames may be approximately the same over a period of time. The synthesis engine 316 may use the frame delta value to cause the processor unit 302 or GPU 318 to adjust the timestamps of scene-related information frames 334 to compensate for drift in the scene capture device. If the synthesis engine 316 determines that visual problems are increasing in the augmented video data 312, the synthesis engine 316 may automatically adjust the video frame latency to reduce these visual effects, or in some cases, the user may provide updated video frame latency data that identifies the new video frame latency for the computing platform 300. Therefore, in some cases, the user may vary the amount of visual issues in the augmented video data 312 based on the video frame latency data 317 to the computing platform 300.

[0096] For example, the compositing engine 316 may provide a graphical user interface (GUI), or another module or device may provide the computing platform 300 with a GUI that includes GUI elements for setting video frame latency. The video frame latency set by the user based on the GUI may be provided as video frame latency data 317. Thus, by controlling which video frames (or associated texture files) are used for digital compositing based on the video frame latency data 317, sufficient time is provided so that, as a result, scene-related information frames 334 can arrive from different scene capture devices used in film production. In some cases, a 3-frame latency was determined to provide sufficient time for different types of scene-related information frames to arrive at the computing platform 300 through the network 353. Furthermore, a 3-frame latency was determined to provide sufficiently near real-time composited video that is useful in production for directing live-action footage that supports CGI asset-live-action video synchronization.

[0097] In some examples, the computing platform 300 is located on location, i.e., on the film set. In other examples, the computing platform 300 is located in a cloud environment. When the computing platform 300 is located in a cloud environment, the augmented video data 312 can be transmitted via a network to a display 314 located on the film set. In this example, the components of the computing platform 300 are shown to be implemented on the same system, but in other examples, different components may be distributed across different systems (in some cases, a cloud computing environment) and communicate, for example, via a network.

[0098] In some cases, display 314 is a camera viewfinder, and therefore the augmented video data 312 can be rendered to a camera operator, who may be the director in some cases. In some cases, display 314 is one or more displays, including a camera viewfinder and display of a device, such as a tablet or mobile device. In some cases, one or more displays include displays located in the video village, which is an area on the set, where members of the film crew can observe the augmented and / or (raw) video footage as it is being filmed. Members of the film crew can view the augmented and / or raw video footage to correct any potential identified film production problems (e.g., an actor looking in the wrong direction).

[0099] Since more than one video camera is used during shooting, in some examples the compositing engine 316 may provide corresponding augmented video data (in the perspective of each digital video camera) in the same or similar manner as disclosed herein. In some examples each video camera has its own computing platform, similar to computing platform 300, as shown in Figure 3, to provide corresponding augmented video data. In some examples each video camera is jam-synced. For example, if a scene is shot by more than one video camera, the video cameras may be electronically time-synchronized so that all digital video cameras may provide video frames at a given point in time with the same or similar timecode. For example all video cameras may provide a given video frame at approximately the same time, and therefore with the same or similar timecode. In some examples the timecode is Society of Motion Picture and Television Engineers (SMPTE) timecode.

[0100] In some examples, the computing platform 300 can be dynamically adjusted based on video frame latency data 317 so that when a new scene capture device is used (e.g., on set, during filming, etc.), new scene-related information frames can be synchronized with the video frames to provide enhanced video data 312 with little to no visual problems. Thus, when the frame rate (the rate at which the scene capture device provides data) changes (e.g., becomes higher or lower), the computing platform 300 can be configured to adapt to the new frame rate based on video frame latency data 317 provided by the user.

[0101] In some examples, once video frame 328 is used by the computing platform 300, the synthesis engine 316 may cause the processor unit 302 or GPU 318 to remove video frame 328 from the corresponding cache memory space. Thus, the synthesis engine 316 may expire video frames and associated data (e.g., metadata) that are no longer needed to free up space in the corresponding cache memory space for new data / information that can be processed according to the examples disclosed herein.

[0102] In some cases, for example, if the digital video camera is offline or if video frame 328 cannot be provided to the computing platform 300, the computing platform 300 may not receive video frame 328. In these cases, the computing platform 300 may create an artificial video frame with a timecode that is processed in the same or similar manner as disclosed herein and is therefore temporally aligned with the scene-related information frame 334. For example, if a director wants to adjust a CGI asset but does not need the scene, the computing platform 300 may be used to provide the CGI asset (within the shot) in the extended video data 312, and as a result, the director may use the computing platform 300 to make the appropriate changes.

[0103] Therefore, the computing platform 300 can provide extended shots of a scene with a given frame latency (e.g., 3 frames latency) sufficient to allow the shots of the scene to be visualized during film production with little or no visual problems. The compositing engine 316 uses a data management and timecode scheme that enables precise alignment of non-frame data from various and dissimilar data sources (e.g., scene capture devices) to video frames.

[0104] Figure 4 is a block diagram of an example system 400 for visualizing a CGI video scene. System 400 may be used to provide real-time augmented video data on display 402, such as during filming. The augmented video data may correspond to augmented video data 210, as shown in Figure 2, or augmented video data 312, as shown in Figure 3. Thus, the example in Figure 3 may refer to the examples in Figures 1 to 3. In the example in Figure 4, display 402 shows a composite video frame of augmented video data (at a certain point in time) containing an image of scene 404 captured by video camera 406, with a digital asset 408 (e.g., a CGI asset). A computing platform 410 may be used to provide augmented video data with one or more images incorporating the digital asset. The composite video frame on display 402 is illustrative and may contain any number of digital assets or none at all in other examples (e.g., when the digital asset moves out of the scene or the field of view of video camera 406). In some examples, the computing platform 410 may correspond to the computing platform 300 shown in Figure 3.

[0105] For example, a computing platform 410 may receive scene-related data 412, which may include one or more scene-related information frames, such as those disclosed herein with respect to Figures 2-3. For example, the scene-related data may include rig tracking data 414, which may be provided by a rig tracking system (or device) 416. The rig tracking system 416 may be used to track the rotation and position of a rig on which a video camera 406 is mounted. Examples of rigs may include tripods, handheld rigs, shoulder rigs, overhead rigs, POV rigs, camera dollies, sliders, rigs, camera cranes, camera stabilizers, Snorricams, etc. The rig tracking data 414 may characterize the rotation and position of the rig. The rig tracking data may also include timestamps indicating when the rotation and position of the rig were captured.

[0106] In some examples, scene-related data 412 may include body data 418 that may be provided by a body tracking system (or device) 420. The body tracking system 420 may be used to track the location of a person in scene 404 in a body tracking coordinate system. The body data 418 may specify the location information of the person in the body tracking coordinate system. In some examples, the body data 418 may specify rotation and / or joint information for each joint of the person. The body data 418 may also specify the position and orientation of the person in the body tracking coordinate system. The body tracking system 420 may provide body data 418 with a timestamp indicating the time when the body movement was captured. Examples of body tracking systems may include motion capture suits, video camera systems (e.g., one or more cameras, such as a mobile phone, used to capture body movement), etc. In an example where the body tracking system 420 is implemented as a video camera system, the body movement captured by the video camera system may be extracted from the image and provided as or as part of the body data 418. Furthermore, in an example where the body tracking system 420 is implemented as a motion capture suit, the motion capture suit may be time-synchronized with a video camera 406 or multiple video cameras 406 that capture the scene 404, and the body data 418 may include a timecode indicating the time when the body's movement was captured. In some examples, face data 422 may be used to animate the movement of a digital asset 408, for example, the body of the digital asset 408.

[0107] In some cases, scene-related data 412 includes face data 422 that may be provided by a face tracking system 424. Face data 422 may characterize the movement of a person's face. The face tracking system 424 may be implemented as a face motion capture system. In some cases, the face tracking system 424 includes a mobile device (e.g., a mobile phone) with software for processing images captured by the mobile device's camera to determine facial features and / or the movement of a person. The face tracking system 424 may provide face data 422 with a timestamp indicating the time when the movement of a person's face was captured. Face data 422 may be used to animate the facial expressions of a digital asset 408, for example, the facial expressions of the digital asset 408.

[0108] In some examples, scene-related data 412 includes prop data 426 that may be provided by a prop tracking system (or device) 428. In some examples, the prop tracking system 428 is a digital prop tracking system 100, as shown in Figure 1. The prop tracking system 428 may be used to track one or more props in scene 304. The prop tracking system 428 may provide prop data 426 with a timestamp indicating the time when the prop movement was captured. In some examples, the prop data 426 includes or corresponds to prop space data 106, as shown in Figure 1. In other examples, the prop space data 106 includes a timestamp indicating the time when one or more prop movements of one or more prop devices (e.g., prop device 104, as shown in Figure 1) were captured. In some examples, prop data 426 and / or prop space data 106 may include starting poses for assets such as an animated character, a sword and shield held in someone's hand as they move, or a piece of furniture located in the scene, such as a CGI asset.

[0109] In some examples, the computing platform 410 may receive digital asset data 430 that may correspond to digital asset data 336, as shown in Figure 3. For example, the digital asset data 430 may include CGI asset data and related information for animating one or more CGI assets (e.g., digital asset 408) in a scene such as scene 404. In some examples, the digital asset data 430 may include controller data 432 provided by the asset controller 434. In one example, the asset controller 434 is an input device, such as a controller. The asset controller 434 may be used to control the movement and facial expressions of digital asset 408 in scene 404. For example, the asset controller 434 may output controller data 432 that characterizes the movement of the face and / or body implemented by digital asset 408. The controller data 432 may include a timecode indicating the time when the movement of the face and / or body was captured. The controller data 432 may be used to animate digital asset 408 (e.g., by the computing platform 410). In some examples, controller data 432 may indicate the start and end times for a CGI action (e.g., movement and / or rotation). In some examples, multiple asset controllers may be used. For example, a first asset controller may be used to manipulate the torso of digital asset 408, another asset controller for the arms of digital asset 408, a further asset controller for manipulating the movement of the fingers of digital asset 408, another asset controller device for manipulating the lips of digital asset 408, and so on. Controller data 432 from each of the asset controllers may be provided to the computing platform 410 for processing according to the examples disclosed herein. Thus, multiple different asset controllers may be used to control different parts of digital asset 408, enabling different movements / movements of the digital asset to be superimposed on a single character.

[0110] In the example in Figure 4, the video camera 406 provides video data 436 with one or more video frames containing images of scene 404. Each video frame may contain timecode. If multiple digital video cameras are used to capture scene 404, the digital video cameras may be time-synchronized as described herein. In some examples, if timecode is not present, timecode may be generated from the computing platform 410 clock. This still allows synchronization, but may be less accurate than when using a dedicated timecode device. In some examples, one or more video frames of the video data 436 may include video frame 212 as shown in Figure 2 and / or video frame 328 as shown in Figure 3.

[0111] Computing platform 410 may be implemented in the same or similar manner as computing platform 400 to process video data 436, scene-related data 412, and / or digital asset data 430 to provide augmented video data rendered on display 402 in the example of Figure 4. In some examples, scene-related data 412 includes scene-related information frame 214 shown in Figure 2 and / or scene-related information frame 334 shown in Figure 3. Computing platform 410 may provide augmented shots of a scene with a given frame latency (e.g., 3 frames latency) sufficient so that shots of a scene can be visualized during film production with little or no visual problems. By configuring computing platform 410 with data manager 242 as shown in Figure 2, computing platform 410 is enabled to accurately align non-frame data from various and dissimilar sources, such as rig tracking system 416, body tracking system 420, face tracking system 424, and asset controller 434, to video frames.

[0112] In some examples, rig tracking data 414, body data 418, face data 422, prop data 426, controller data 432, and video data 436 are provided via a network 435, such as network 353, as shown in Figure 2. In other examples, some or all of the data 414, 418, 422, 426, 432, and 436 are provided via network 435. In some examples, the video camera 406 may be connected to a computing platform 410, which may be considered outside or not part of network 435, using physical cables. In some examples, the video camera 406 and systems 416, 420, 424, 428, and asset controller 434 may be connected to the computing platform 410 using a network 435, which may include wired and / or wireless connections. For example, high-bandwidth cables may be used to connect the HDMI® output or SDI output of the video camera 406 to the computing platform 410 in order to transmit high-resolution video data to the computing platform 410 at a high rate. Examples of HDMI® cables include, but are not limited to, HDMI 2.1 cables with a bandwidth of 48 Gbps that carry resolutions up to 10K at frame rates up to 120 fps (8K60 and 4K120). Examples of SDI cables include, but are not limited to, DIN 1.0 / 2.3 to BNC female adapter cables. External or internal hardware capture cards (such as the internal PCI Express Blackmagic Design DeckLink SDI Capture Card or the external unit Matrox MAXO2 Mini Max for Laptops, which support various video inputs such as HDMI® HD 10-bit, Component HD / SD 10-bit, Y / C (S-Video) 10-bit, or Composite 10-bit) may be used to provide video data 436 to the interfaces of the computing platform 310 (for example, one of the I / O interfaces 326 as shown in Figure 3).

[0113] In some examples, system 400 may include a server 437 that can be connected to a network 435, and rig tracking data 414, body data 418, face data 422, prop data 426, controller data 432, and video data 436 may be routed through server 437. For example, server 437 may communicate with each system 416, 428, 420, 424 and / or asset controller 434 and route the data as messages to computing platform 410. Thus, in some examples, the server may use a publisher / subscriber paradigm where each system 416, 428, 420, 424 and / or asset controller 434 is the publisher and the computing platform is the subscriber. In a further example, server 437 may be implemented on a device such as a laptop.

[0114] In some cases, the computing platform 410 may be used for post-production. For example, pre-recorded video data 438 with video frames may be processed in the same or similar manner as video data 436 with other scene-related information (data / frames) to provide augmented video data. For example, the computing platform 410 may be used to re-animate live-action footage and insert new digital assets (not used during production) into video footage (represented by the pre-recorded video data 438). In some cases, the computing platform 410 may be used to modify the actions and / or behavior of inserted digital elements (e.g., digital asset 408) that are first used during production.

[0115] For example, the computing platform 410 may include a virtual environment builder 440 that can recreate or regenerate a 3D model 442 representing a scene captured by one or more video cameras that are the source of pre-recorded video data 438, based on environmental data 444 captured about the scene. The environmental data 444 may be provided by one or more environmental sensors, such as those disclosed herein. An input device 446 may be used to provide commands / actions to the model-up data 448 to adjust one or more virtual features of the 3D model 442. One or more virtual features may include the position / orientation of each virtual camera representing the video cameras that are the source of the pre-recorded video data 438, the position / orientation of digital assets in the 3D model 442, and other types of virtual features. In some cases, the computing platform 410 may include a digital compositor 450 that can be implemented similarly to the digital compositor 208 shown in Figure 2. The digital compositor 450 may provide augmented video data based on the 3D model 442 and pre-recorded video data 438. Thus, in some examples, the computing platform 410 may be used in post-production to allow reshooting of the scene by adjusting the 3D model 442, thereby modifying or adjusting the VFX (if available in the pre-recorded video data 438), or in some cases, to provide augmented video data by inserting it into the pre-recorded video data 438. In some examples, the digital compositor 450 may include, or communicate with, a prop tracker engine 102, as shown in Figure 1. The prop tracker engine 102 may be used to provide spatial data (e.g., position, orientation, and / or movement) about the digital assets that are composited by the digital compositor 450 to provide the scene 404.

[0116] Figure 5 is a block diagram of an example of a waypoint system 500 for animating (e.g., movement, action, sound, etc.) a digital asset (e.g., digital asset 408, as shown in Figure 4) in a scene (e.g., scene 404, as shown in Figure 4). Therefore, the example in Figure 5 may refer to the examples in Figures 1 to 4. The waypoint system 500 may be implemented on a computing platform or cloud computing environment as disclosed herein. The waypoint system 500 may provide a waypoint scene model 502 for identifying multiple virtual points (or locations) for a scene, and a digital asset (e.g., a CGI asset) may be directed thereto. Adjacent virtual points may define virtual intervals, and virtual intervals may be connected to define a virtual route or path for a digital asset. A given virtual point may be identified as a starting virtual point from which the digital asset can be animated. Another virtual point (in some cases, a given virtual point) may identify an ending point from which the animation of the digital asset ends. The waypoint scene model 502 may identify or specify the animation of a digital asset along each virtual segment (at start and end virtual points, in some cases). In some cases, the animation of a digital asset along each virtual segment may differ. For example, a digital asset may be walking along the first virtual segment, and running along the second virtual segment. Other actions / movements are assumed to be within the scope of this disclosure.

[0117] The waypoint system 500 includes a virtual route creator 504. The virtual route creator 504 may provide a waypoint scene model 502 based on depth data 508, animation and / or waypoint instructions 510, and digital asset data 512. The depth data 508 may correspond to depth data (e.g., depth data 508 shown in Figure 5) as disclosed herein. The depth data 508 may be used by the virtual route creator 504 to identify objects and surfaces in the scene for the navigation of digital assets, in accordance with the waypoint and / or animation instructions 510. In some examples, the virtual route creator 504 (or different modules / block diagrams as disclosed herein) may be used to create a scene model 526 of the scene (based on the depth data 508). The scene model 526 may be populated with virtual location (point) information to provide a waypoint scene model 502 based on animation and / or waypoint instructions 510. In some examples, the virtual root creator 504 receives the scene model 526 from the model generator 244 shown in Figure 2. Thus, in some implementations, the scene model 526 may correspond to the 3D model 234, as shown in Figure 2. The virtual root creator 504 may provide the waypoint scene model 502 with virtual points that define virtual paths that may be behind and / or in front of objects in the scene (e.g., props / non-props, and actors and / or animals).

[0118] In further or additional examples, the waypoint scene model 502 may be reduced according to the examples herein to provide a diorama of the scene. Since the waypoint scene model 502 includes virtual points and therefore can define virtual paths for digital asset animation (or visualization), in some cases the diorama may include virtual points and virtual paths. In some examples the diorama of the waypoint scene model 502 may be animated, allowing a user (e.g., a creator) to visualize the animation of digital assets along the virtual paths.

[0119] Animation and / or waypoint instructions 510 may be provided by an input component 514. The input component 514 may include an interface (e.g., in some cases, an application program interface (API)) for interface-connecting with an input device 516 for receiving animation and / or waypoint instructions 510. The input device 516 may correspond to any device that can be used to animate the digital asset. In some examples, the input device 516 is a keyboard or a controller (e.g., a console controller). In some examples, the input device 516 is another system or server. Animation and / or waypoint instructions 510 may indicate the animation of the digital asset along each virtual section or virtual path. Instructions 510 may also specify each virtual point (or waypoint) in the scene.

[0120] The virtual root creator 504 may update the scene model 526 to include virtual points in a model representing a desired location in the scene. The updated scene model 526 may be referred to as the waypoint scene model 502. The virtual points may be connected to define a virtual path or trajectory for the animation of digital assets. A digital asset along one or more virtual segments of a virtual path may, in some embodiments, be unique with respect to other or the remaining virtual segments. In some examples, the waypoint scene model 502 may be used to provide augmented video data 518 without digital assets and to depict a virtual path (or multiple virtual segments of a virtual path) along which digital assets can be animated. By providing augmented video data 518 without digital assets and with only a virtual path (or a set of virtual segments of a virtual path), filmmakers (and / or other filmmakers) can coordinate the movement of digital assets in a more efficient way. In some examples, the virtual root creator 504 may provide the waypoint scene model 502 by inserting digital assets into the scene model 526 based on digital asset data 512. Digital asset data 512 may characterize the digital asset and, in some cases, the actions and / or movements of the digital asset. Waypoint scene model 502 may include data for animating the digital asset, which may be provided based on the digital asset data 512, along a virtual path defined by virtual points. In other cases, the data may be associated with the waypoint scene model 502 (e.g., logically linked in memory, such as those disclosed herein).

[0121] In some examples, the waypoint scene model 502 may be provided to the compositing engine 520. In some cases, the compositing engine 520 may be configured to implement one or more functions (or methods) of the digital compositor 112 as shown in Figure 1, the digital compositor 208 as shown in Figure 2, the compositing engine 316 as shown in Figure 3, and / or the digital compositor 450 as shown in Figure 4. The compositing engine 520 may receive the waypoint scene model 502, which can be used to provide one or more video frames 522 of the scene and augmented video data 518. In some examples, the compositing engine 520 may also receive other information for compositing, e.g., scene-related information / data (e.g., one or more scene-related frames as disclosed herein), and / or, in some cases, digital asset data 512, to provide augmented video data 518. In some examples, the compositing engine 520 may include, or communicate with, a prop tracker engine 102, as shown in Figure 1. The synthesis engine 520 may be used to provide spatial data (e.g., position, orientation, and / or movement) about the digital assets synthesized by the synthesis engine 520 in order to provide the augmented video data 518.

[0122] The extended video data 518 may include one or more composite video frames with virtual locations and virtual intervals between virtual locations depicted in the scene. In some cases, one or more composite video frames depict a virtual path along which a digital asset can be animated. In some cases, the virtual path may be modified based on user input and provided to a virtual route creator 504 to provide an updated waypoint scene model 502. For example, an input device 516 may be used in some cases to modify the virtual path, and user input may be provided as a new or updated waypoint instruction.

[0123] The augmented video data 518 can be visualized on an output device 524, for example, as disclosed herein. In some examples, as described herein, the augmented video data 518 may include one or more composite video frames with only virtual locations and / or virtual paths. In other examples, the augmented video data 518 may include one or more composite video frames with virtual paths and digital assets that can be animated along the virtual paths in the scene. For example, graphical user interface (GUI) elements can be rendered on an output device 524 that can be interacted with (e.g., by a user) to start and stop the animation of digital assets along a virtual path (or a set of virtual segments of a virtual path).

[0124] In some cases, multiple different virtual paths may be defined by the virtual root creator 504 for different digital assets in the scene model 526 based on animation and / or waypoint instructions 510. For example, the virtual root creator 504 may create a first virtual path in the scene model 526 for a first digital asset, and a second virtual path in the scene model 526 for a second digital asset. In some cases, the virtual root creator 504 may evaluate the first and second virtual paths for any intersection (or crossing point) of the virtual paths.

[0125] Since the first and second virtual paths can intersect, the first and second digital assets can intersect, and one of the digital assets can occlude the other. The virtual root creator 504 can determine or identify any virtual path intersection (between any multiple virtual paths in the scene model 526) and use occlusion order data to determine which of the first and second digital assets should be occluded by the other. The occlusion order data may indicate the rendering order of the first and second digital assets and thus specify which of the first and second digital assets should occlude the other. In some examples, the virtual root creator 504 may use depth sorting techniques based on depth or distance from the viewpoint of a camera (e.g., providing one or more video frames 522). The virtual root creator 504 may provide the waypoint scene model 502 with data indicating the order of the first and second digital assets at the virtual path intersection.

[0126] Therefore, the Waypoint System 500 allows, in some cases, to insert a virtual path into a scene along which a digital asset can be animated. This enables filmmakers (and / or other filmmakers) to visualize the virtual path along which the digital asset will be animated, for example, during film production (e.g., shooting on set), and as a result, scene errors can be identified and corrected during production (rather than in post-production).

[0127] Figure 6 is a block diagram of an example of a diorama pipeline 600 that may be used to provide a scene (for example, scene 404 as shown in Figure 4) (shooting set) diorama 602. In some examples, the diorama pipeline 600 is implemented as part of a model generator 244, as shown in Figure 2, and in other examples, it may communicate with the model generator 244 to provide the diorama 602. Thus, in the example of Figure 6, examples of Figures 1-5 may be referenced. The diorama pipeline 600 may be implemented on a computing platform or in a cloud computing environment, as disclosed herein.

[0128] The diorama pipeline 600 may provide a diorama 602 based on depth data 604, color data 606 (e.g., red, green, blue (RGB) information), digital asset data 608, and in some cases, based on body movement data 610. In some implementations, the diorama 602 may be a digital representation of a scaled (reduced) or reduced scene with digital information (e.g., digital assets, e.g., CGI assets embedded therein). A virtual camera may be used to view the diorama 602 from a non-human viewpoint, and this virtual camera may be referred to as a “diorama view camera”. For example, the diorama pipeline 600 may receive perspective data 612 that can specify the perspective of the diorama view camera relative to the diorama 602. The perspective data 612 may show multiple different non-human perspectives of the diorama 602, e.g., god's-eye view (top-down perspective), bird's-eye view, or insect's-eye view.

[0129] For example, the diorama pipeline 600 includes a surface engine 614. The surface engine 614 may receive depth data 604. The surface engine 614 may process the depth data 604 to (digitally) reconstruct one or more surfaces in the scene (e.g., of objects and / or the environment of the scene) to provide a surface model. The surface model may represent the geometric details provided by the depth data 604 (e.g., point cloud data). The surface model may therefore approximate the shapes and geometric structures of objects and / or the environment of the scene.

[0130] For example, the surface engine 614 may convert the point cloud provided as depth data 604 into a surface mesh, or a set of interconnected triangles, to represent one or more surfaces in the scene and provide a surface model. The surface engine 614 may use any type of algorithm to provide the surface. For example, the surface engine 614 may use Poisson reconstruction, marching cubes, Delaunay triangulation, or any other type of algorithm. The point cloud may represent a set of 3D points describing the surfaces in the scene. Each point in the point cloud may correspond to a specific location in space (in the scene) and may contain (or be associated with) information about its position and distance from the camera. The density of points in the point cloud may vary depending on the capture method and the resolution of the sensor or camera.

[0131] In some examples, the diorama pipeline 600 includes a texture map component 616 that can receive a surface model for texture surface mapping. For example, the texture map component 616 may use color data 606 to map textures onto a surface model to provide a texture-mapped surface model. The color data 606 may be provided from a color camera or sensor that can be synchronized with depth sensing technology used in some cases to provide depth data 604. These devices can simultaneously capture depth (distance) and color information, for example, by using stereo vision, structured light, or an RGB-D sensor.

[0132] For example, during the capture process, each point in the point cloud may be associated with a corresponding RGB value representing the color of that point in the scene. These RGB values ​​can be aligned with depth data 604 to ensure that the color information matches the spatial coordinates of the points. When mapping a texture onto a surface mesh, the correspondence between the RGB information and the mesh surface can be established through UV mapping or texture coordinates. UV mapping is a technique used to define a two-dimensional coordinate system (U and V) on a surface mesh. The UV coordinates can then be used by the texture map component 616 to project or map the RGB values ​​(of the color data 606) onto the corresponding vertices of the surface mesh. Various algorithms, including but not limited to nearest neighbor mapping, centroid mapping, or more advanced techniques such as mesh parameterization or surface parameterization, can be used by the texture map component 616 for texture mapping.

[0133] In some examples, the diorama pipeline 600 includes a light and shader component 618. The light and shader component 618 can provide a scene model 620 by applying light and / or shading to a texture-mapping surface model, as shown in Figure 6. For example, the shader component 618 can add lighting and shading effects to a texture-mapping surface model to enhance its realism. In some examples, the shader component 618 may be omitted, and the texture-mapping surface model may be provided as the scene model 620. The scene model 620 may be used to represent a scene and therefore to resemble the appearance and characteristics of the real-world environment on which it is based. In some examples, the scene model 620 may correspond to a 3D model 234, as shown in Figure 2.

[0134] The example in Figure 6 shows the surface engine 614, texture map component 616, and light and shader component 618 as part of the diorama pipeline 600, but in other examples, these block diagrams may be implemented as standalone units or as part of a system (e.g., a compositing system or a different system) that can communicate with the diorama pipeline 600.

[0135] The diorama pipeline 600 may include a diorama generator 622 that can provide a diorama 602 based on a scene model 620, digital asset data 608, body movement data 610, and perspective data 612. In response to providing (or generating) the diorama 602, the diorama generator 622 may create a scaled-down version of the scene (e.g., before and / or during filming). The diorama generator 622 may create a scaled-down version of the scene model 620. A virtual world (or environment) may be defined to provide a digital space in which the digital models and / or data can be located. In some examples, the virtual world includes the diorama 602. The virtual world may be created by a virtual environment engine 624. In some examples, the diorama pipeline 600 includes the virtual environment engine 624. A diorama view camera may be inserted into the digital space. The position and / or orientation of the diorama view camera relative to the diorama 602 may be adjusted based on the perspective data 612. The diorama view camera can be adjusted in digital space to provide a non-human viewpoint (e.g., god's-eye view) of diorama 602.

[0136] In some examples, the diorama generator 622 may receive animation and / or waypoint commands 510, as shown in Figure 5, and use this data to insert waypoints (virtual points) into the scene model 620 and define virtual paths within them. Digital assets may be animated along the virtual paths as disclosed herein. Thus, in some examples, the diorama pipeline 600 includes a waypoint system 500, and in some cases, animation and / or waypoint commands 510 may be used for waypoint insertion and digital asset virtual path animation.

[0137] In some cases, the diorama generator 622 may insert one or more digital assets into the scene model 620 based on digital asset data 608 and provide data over time for animating the inserted digital assets. The data may be provided as part of the diorama 602 or (as disclosed herein) may be logically linked to the diorama 602 in memory. For example, a first digital asset may be inserted into the scene model 620 which may represent a digital asset (e.g., a CGI asset) to be inserted into a scene, which may be referred to as a scene digital asset. In some cases, the scene digital asset may be inserted into the scene (scene shot) using a compositing engine 626. The compositing engine 626 may be configured to implement one or more functions (or methods) of a digital compositor 112 as shown in Figure 1, a digital compositor 208 as shown in Figure 2, a compositing engine 316 as shown in Figure 3, a digital engine 450 as shown in Figure 5, and / or a compositing engine 520 as shown in Figure 5. In some examples, the compositing engine 626 may include or communicate with the prop tracker engine 102, as shown in Figure 1. The compositing engine 626 may be used to provide spatial data (e.g., position, orientation, and / or movement) about the digital assets that are composited by the compositing engine 520 to provide the augmented video data 630. In some examples, a second digital asset may be inserted into a scene model 620 that may represent an actor (or other element in the scene), and the actor's movement may be mapped to the second digital asset using body movement data 610. In some examples, the body movement data 610 corresponds to body data 418, as shown in Figure 4. The diorama 602 may be animated, or still frames of the diorama 602 may be provided, allowing a user (e.g., a director) to visualize the scene (digitally represented) through a non-personal viewpoint based on the scene model 620.

[0138] In some examples, as disclosed herein, the compositing engine 626 may receive one or more video frames 628 for digital compositing according to one or more examples. The compositing engine 626 may provide augmented video data 630 based on the video frames 628 and the diorama 602. One or more video frames may be provided by a camera in some cases, as disclosed herein. For example, the camera may correspond to a camera on a portable device, such as a tablet or mobile phone. For example, one or more video frames may include a captured table, and the compositing engine 626 may render the diorama 602 onto the table. In some examples, the video frames are shots of a captured scene. In other examples, the video frames are non-scene shots of, for example, the director's location (or room, area, etc.) used in production.

[0139] The compositing engine 626 may provide augmented video data 630 with one or more composited video frames. The composited video frames may include a diorama 602 rendered from a resulting perspective view (non-person viewpoint) of a diorama view camera. The augmented video data 630 may be rendered on an output device 632, as shown in Figure 6. The output device 632 may correspond to a viewfinder on a camera, display, television, VR headset, portable device (e.g., portable computer, tablet, mobile phone, etc.), or any other visual output device capable of rendering composited video frames.

[0140] In some cases, the scene model 620 may be simulated, and the augmented video data 630 may provide a composite video frame with a diorama 602 that mimics the simulated scene model 620, enabling a user (e.g., a producer) to visualize shots of the scene from a non-personal viewpoint (e.g., a god's-eye view) with one or more digital assets animated (or simulated) within them. In an example where a second digital asset is used, the augmented video data 630 may include a second digital asset representing an actor in the diorama 602, which may mimic (or represent) the actor's movement while the actor is being filmed.

[0141] In some cases, an actor's voice may be captured while the actor is being filmed and synchronized (by the compositing engine 626 or a different module) with the movement of a second digital asset (corresponding to the actor's movement). In some cases, one or more GUI elements may be rendered on the output device 632, allowing a user (e.g., a director) to start and stop the animation of the diorama 602. By providing GUI elements, it is possible to control the animation of the diorama 602 and allow the user to step through it (e.g., forward and backward in time relative to the diorama 602). Furthermore, GUI elements (e.g., a stop GUI element) may be used to allow the user to stop the animation of the diorama 602 at a specific point in time for further investigation (e.g., correcting digital asset placement, actor actions and / or placement, etc.). In some cases, the diorama 602 may be rendered on the output device 632 without being combined with other data, e.g., video frames 628 and digital asset data 608. Therefore, in some examples, the synthesis engine 626 may provide a frame for the diorama 602 that can be output onto the output device 632 for visualization.

[0142] In some cases, the diorama generator 622 may use camera viewpoint data 634 for the camera that provided the video frame 628 to set (or configure) a diorama view camera for the diorama 602. For example, the camera viewpoint data 634 may also provide position and / or orientation information for the camera, and in some cases, other camera information (e.g., field of view, lens distortion, sensor static, etc.) that may be used to set up the diorama view camera. For example, based on the camera viewpoint data 634, the diorama generator 622 may adjust the diorama view camera, and therefore the profile view of the scene model 620, to provide the diorama 602 in a specific orientation relative to the diorama view camera.

[0143] For example, if an observer (e.g., a supervisor) wants to view the diorama 602 on the output device 632 from the left at a specific angle, the diorama generator 622 can provide the diorama 602 on the output device 632, and as a result, the observer can see the diorama 602 at that specific angle. Thus, the observer can view the diorama 602 in different non-person perspectives (not limited to a top view where the specific angle to the diorama 602 can be zero, for example). The observer can view the diorama 602 in an oblique or inclined view.

[0144] In some examples, multiple cameras are used to provide each video frame in a different recording (or capture) perspective. A compositing engine 626 (or each respective compositing engine) may be used to provide corresponding augmented video data with each diorama 602 in a viewpoint angle (or view profile) obtained based on camera viewpoint data for a given camera. For example, if two observers are looking at diorama 602, one observer may be looking at diorama 602 on output device 632 in one tilt / perspective view, while the other observer may be looking at diorama on another output device in a different tilt / perspective view.

[0145] For example, if diorama 602 is rendered on a table captured by one or more video frames provided by first and second cameras, the first output device may show diorama 602 from one viewpoint (e.g., from the left at an angle to diorama 602), and the second output device may show diorama 602 from a different viewpoint (e.g., from the right at an angle to diorama 602). This allows multiple different filmmakers (e.g., directors, producers, etc.) to visualize the scene from the perspective of their own devices through diorama 602, for example, before filming or while filming is in progress.

[0146] Diorama 602 provides a non-personal viewpoint of a scene with digital asset information (representing actors in some cases), allowing producers (and / or other film industry professionals) to visualize the scene before or during filming. The movement of actors and objects in a scene (e.g., cars, airplanes, etc.) can be viewed through this non-traditional viewpoint. For example, the god's-eye view of a scene provided by Diorama 602 allows producers to determine whether the movement of actors and / or digital assets needs correction or adjustment. Furthermore, the god's-eye view of a scene provided by Diorama 602 allows producers to correct the placement of actors and / or objects in the scene, for example, by adjusting the location of props. Thus, the god's-eye view of a scene provided by Diorama 602 allows producers to consider multiple different actions / elements of a scene simultaneously. This is a feature not found in the conventional first-person or third-person viewpoints of scenes on set.

[0147] For example, a god's-eye view of a scene through Diorama 602 provides the filmmaker with a better overall view of the scene (for example, when an understanding of the scene space is needed or desired), improving scene planning (e.g., decision-making), object placement / management, spatial awareness (e.g., making it easier to understand the relationships between scenes, objects, actors, and / or digital assets), and macro-level control (e.g., enabling the filmmaker to manage multiple actors and / or digital assets simultaneously and coordinate their movement and interaction with greater precision). Furthermore, since Diorama 602 can be simulated or animated, the filmmaker may be provided with a depiction of the scene (in the digital domain) of how the scene was captured. This allows the filmmaker to make corrections during filming and thus reduce post-production costs because errors and mistakes can be caught beforehand, i.e., during filming, rather than after filming / post-production.

[0148] In light of the structural and functional features described above, the method examples are better understood by referring to Figures 7-12. For the sake of brevity, the method examples in Figures 7-12 are shown and described as being performed sequentially, but it is understood and recognized that the examples are not limited by the order in which they are shown. Some actions may occur multiple times and / or simultaneously in other examples, in a different order than that shown and described herein. Furthermore, it is not necessary for all described actions to be performed in order to implement the method.

[0149] Figure 7 shows an example of method 700 for providing a composite video frame. As shown in Figure 1, method 700 may be implemented by a prop tracker engine 102. Thus, in the example of Figure 7, one or more examples of Figures 1-6 may be referenced. Method 700 may begin in 702 by receiving prop space data (e.g., prop space data 106, as shown in Figure 1) about a prop device in physical space (e.g., probe device 104, as shown in Figure 1). In 704, digital asset space data (e.g., digital asset space data 108, as shown in Figure 1) about a digital asset in virtual space. In 706, the digital asset space data about the digital asset may be updated based on the prop space data to provide updated digital asset space data (e.g., updated digital asset space data 110, as shown in Figure 1). In 708, the position, movement and / or orientation of the digital asset in virtual space may be updated based on the updated space data about the digital asset.

[0150] Figure 8 is an example of Method 800 for providing augmented video data during production using digital asset prop tracking. Method 800 may be implemented by components and / or systems as disclosed herein. Thus, in the example of Figure 8, one or more examples of Figures 1-7 may be referenced. Method 800 may begin in 802 by receiving prop space data (e.g., prop space data 106, as shown in Figure 1) about prop devices in a scene (e.g., prop space data 106, as shown in Figure 1) (e.g., in prop tracker engine 102, as shown in Figure 1). In 804, video data (e.g., video data 116, as shown in Figure 1) containing captured video frames of the scene may be received (e.g., by digital compositor 112, as shown in Figure 1). In 806, a scene model of the scene with digital assets (e.g., 3D model 234, as shown in Figure 2) may be generated (e.g., by model generator 244, as shown in Figure 2). In 808, the position, orientation, and / or movement of the digital asset in the scene model may be updated based on prop space data (e.g., by the prop tracker engine 102). In 810, one or more video frames of the received video data, and the digital asset with the updated position, movement, and / or orientation, may be composited (e.g., by the digital compositor 112). In 812, the composited video frames may be rendered on an output device (e.g., output device 120 as shown in Figure 1) (e.g., by the digital compositor 112).

[0151] Figure 9 shows an example of method 900 for providing a composite video frame. Method 900 can be implemented by a composite engine 200 as shown in Figure 1, or a composite engine 316 as shown in Figure 3. Thus, in the example of Figure 9, one or more examples of Figures 1-8 may be referenced. Method 900 may begin with a video frame (e.g., one of the video frames 212 as shown in Figure 2) among video frames stored in a cache memory space (e.g., cache memory space 204 as shown in Figure 2) being identified (e.g., by a video frame retriever 220 as shown in Figure 2) based on video frame latency data (e.g., video frame latency data 224 as shown in Figure 2). The video frame latency data may specify the number of the video frame stored in the cache memory space before the video frame is selected. In 904, a scene-related information frame among the scene-related information frames (for example, one of the scene-related information frames 214 as shown in Figure 2) can be identified based on the timecode of the video frame (for example, by a scene-related frame retriever 226 as shown in Figure 2). In 906, a composite video frame (for example, part of the extended video data 110 as shown in Figure 1) can be provided based on the video frame and the scene-related information frame.

[0152] Figure 10 is an example of method 1000 for providing augmented video data during scene pre-visualization. Method 1000 can be implemented by a compositing engine 200 as shown in Figure 2, or a compositing engine 316 as shown in Figure 3. Thus, in the example of Figure 10, one or more examples of Figures 1-9 may be referenced. Method 1000 may begin in 1002 by identifying a video frame (e.g., one of the video frames 212 as shown in Figure 2) from among the video frames provided by a video camera (e.g., camera 406 as shown in Figure 4) representing a scene in film production (e.g., scene 404 as shown in Figure 4), based on video frame latency data (e.g., video frame latency data 224 as shown in Figure 2). The video frame latency data may specify the number of a video frame that is stored in the memory of the computing platform before the video frame is selected.

[0153] In 1004, a scene-related information frame (for example, one of the scene-related information frames 214, as shown in Figure 2) among the scene-related information frames provided by a scene capture device (for example, one of the systems 416, 420, 324, and / or 428, and / or asset controller 434, as shown in Figure 4) may be identified (for example, by the scene-related frame retriever 226, as shown in Figure 2) based on an evaluation of the timecode of the video frame against the timestamp of the scene-related information frame. The timestamp for the scene-related information frame may be generated based on a frame delta value representing the amount of time between each scene-related information frame provided by each of the scene capture devices. In 1006, extended video data (for example, extended video data 210, as shown in Figure 1) with a digital asset (for example, a digital asset 208, as shown in Figure 2) based on the scene-related information frame and video frame may be provided (for example, for rendering on an output device such as a display 314, as shown in Figure 3, or a display 402, as shown in Figure 4).

[0154] Figure 11 is an example of Method 1100 for waypoint animation of a digital asset in a scene. Method 1100 may be implemented by one or more block diagrams (or modules) relating to, for example, Figure 5. Thus, in the example of Figure 11, one or more examples of Figures 1-10 may be referenced. Method 1100 may be initiated in 1102 by receiving a waypoint instruction (e.g., animation and / or waypoint instruction 510, as shown in Figure 5) (e.g., by a virtual root creator 504, as shown in Figure 5). The waypoint instruction may identify virtual points (waypoints) for a digital asset to be used in a scene (e.g., scene 404, as shown in Figure 4). In 1104, the scene model (e.g., scene model 526, as shown in Figure 5) is updated to include the virtual points at locations in the scene model corresponding to locations in the scene, thereby providing a waypoint scene model (e.g., waypoint scene model 502, as shown in Figure 5). In 1106, the augmented video data (e.g., augmented video data 518, as shown in Figure 5) may include one or more composite video frames (e.g., compositing engine 520, as shown in Figure 5) with virtual points in the scene, which may be provided based on a waypoint scene model. The augmented video data may be rendered or made to be rendered on an output device (e.g., output device 524, as shown in Figure 5) for visualization.

[0155] In some examples, in 1102, an animation instruction (e.g., an animation and / or waypoint instruction 510) may be received that identifies the animation of digital assets between adjacent virtual points of a virtual point. A waypoint scene model with data specifying the animation of digital assets between adjacent virtual points may be provided in step 1104. In 1106, the augmented video data with composite frames may be rendered (or made to be rendered) on an output device to provide visual animation of digital assets at and / or between adjacent points based on the waypoint scene model.

[0156] Figure 12 is an example of method 1200 for outputting augmented video data with a diorama (e.g., diorama 602 as shown in Figure 6). Method 1200 may be implemented, for example, by one or more block diagrams (or modules) relating to Figure 12. Thus, one or more examples of Figures 1-11 may be referenced in the example of Figure 12. Method 1200 may begin in 1202 by receiving depth and color data (e.g., depth and color data 604 and 606 as shown in Figure 6) for a scene (e.g., scene 404 as shown in Figure 4) (e.g., by diorama pipeline 1200 as shown in Figure 12). In 1204, a scene model (e.g., scene model 620 as shown in Figure 6) may be provided based on the depth and color data (e.g., by diorama pipeline 600). The scene model may provide a virtual (or digital) representation of the scene, including objects and / or other elements in the scene. In 1206, a scaled-down version of the scene model corresponding to the scene's diorama can be created (for example, by diorama pipeline 500).

[0157] In 1208, a diorama view camera (digital video camera) may be configured with respect to the diorama (for example, by the diorama pipeline 600, for example, based on perspective data 612, as shown in Figure 6) to provide a perspective view of the diorama. In 1210, augmented video data (for example, augmented video data 630, as shown in Figure 6) is provided, which includes one or more composite video frames with the perspective view of the diorama, for visualization on an output device (for example, output device 632, as shown in Figure 6).

[0158] While this disclosure has described several exemplary embodiments, it will be understood by those skilled in the art that various modifications may be made without departing from the spirit and scope of the invention, and that equivalents may be substituted for those elements. Furthermore, it will be understood by those skilled in the art that many modifications will be made to adapt specific equipment, situations, or materials to the embodiments of this disclosure without departing from their essential scope. Accordingly, the invention is not limited to any particular embodiment disclosed or the best mode intended to carry out the invention, but rather is intended to include all embodiments that fall within the scope set forth in the appended claims. Furthermore, references in the appended claims to any device or system, or component of a device or system, that is adapted, arranged, capable, configured, enabled, operable, or operates to perform a particular function, encompass that device, system, or component insofar as it is adapted, arranged, capable, configured, enabled, operable, or operates in such a way, regardless of whether its or their particular function is activated, turned on, or unlocked or not.

[0159] In light of the above-mentioned structural and functional descriptions, it will be understood by those skilled in the art that some of the embodiments may be embodied as methods, data processing systems, or computer program products. Accordingly, these parts of the embodiments may take the form of hardware embodiments as a whole, software embodiments as a whole, or embodiments combining software and hardware, such as those shown and described with respect to the computer system in Figure 13. Furthermore, some of the embodiments may be computer program products on a computer-readable storage medium having computer-readable program code on such medium. Any non-temporary, tangible storage medium having a structure may be used, including but not limited to static and dynamic storage devices, hard disks, optical storage devices, and magnetic storage devices, but excluding any medium that is not eligible for patent protection under Section 101 of the United States Patent Act (such as the propagating electrical or electromagnetic signals themselves). Examples, but not limited to, computer-readable storage media may include semiconductor-based circuits or devices or other ICs (e.g., field-programmable gate arrays (FPGAs) or ASICs), hard disks, HDDs, hybrid hard drives (HHDs), optical disks, optical disk drives (ODDs), magneto-optical disks, magneto-optical drives, floppy disks, floppy disk drives (FDDs), magnetic tapes, holographic storage media, solid-state drives (SSDs), RAM drives, secure digital cards, secure digital drives, or other suitable computer-readable storage media, or any combination of two or more of these. Computer-readable non-temporary storage media may be volatile, non-volatile, or a combination of volatile and non-volatile, as needed.

[0160] In this regard, Figure 13 shows an example of a computing system 1300 that may be employed to carry out one or more embodiments of the present disclosure. The computing system 1300 may be implemented on one or more general-purpose network computer systems, embedded computer systems, routers, switches, server devices, client devices, various intermediate devices / nodes, or standalone computer systems. In addition, the computing system 1300 may be implemented on various mobile clients, such as personal digital assistants (PDAs®), laptop computers, pagers, and the like, provided that they have sufficient processing power. In other examples, the computing system 1300 may be implemented on dedicated hardware.

[0161] The computing system 1300 includes a processing unit 1302, memory 1304, and a system bus 1306 that connects various system components, including memory 1304, to the processing unit 1302. Dual microprocessor and other multiprocessor architectures can also be used as the processing unit 1302. The system bus 1306 may be one of several types of bus structures, including a memory bus or memory controller, peripheral bus, and local bus, using any of various bus architectures. Memory 1304 includes read-only memory (ROM) 1310 and random access memory (RAM) 1312. A basic input / output system (BIOS) 1314 may reside in the ROM 1310 and contain basic routines that help transfer information between elements within the computing system 1300.

[0162] The computing system 1300 may include a hard disk drive 1316, a magnetic disk drive 1318 for reading from or writing to, for example, a removable disk 1320, and an optical disk drive 1322 for reading from, for example, a CD-ROM disk 1324 or for reading from or writing to other optical media. The hard disk drive 1316, the magnetic disk drive 1318, and the optical disk drive 1322 are connected to the system bus 1306 by a hard disk drive interface 1326, a magnetic disk drive interface 1328, and an optical drive interface 1330, respectively. The drives and associated computer-readable media provide non-volatile storage of data, data structures, and computer-executable instructions for the computing system 1300. While the above description of computer-readable media refers to hard disks, removable magnetic disks, and CDs, other types of computer-readable media, such as various forms of magnetic cassettes, flash memory cards, digital video discs, and similar, may also be used in the operating environment. Furthermore, any such medium may include computer executable instructions for implementing one or more parts of the embodiments shown and described herein. The computing system 1300 may include a GPU interface 1358 which can be used for interface connection with the GPU 1360. In some examples, the GPU 1360 may correspond to a GPU 318 as shown in Figure 3. In further examples, the computing system 1300 includes the GPU 1360.

[0163] Multiple program modules, including an operating system 1332, one or more application programs 1334, other program modules 1336, and program data 1338, may be stored in the drive and RAM 1310. For example, RAM 1310 may include one or more block diagrams (or modules) as disclosed herein. Program data 1332 may include one or more types of data as disclosed herein.

[0164] A user may input commands and information to the computing system 1300 through one or more input devices 1340, such as a pointing device (e.g., mouse, touchscreen), keyboard, microphone, joystick, gamepad, scanner, and the like. For example, one or more input devices 1340 may be used to provide video frame latency data, as disclosed herein. These and other input devices are often connected to the processing unit 1302 through corresponding port interfaces 1342 coupled to the system bus, but may be connected by other interfaces such as parallel ports, serial ports, or Universal Serial Bus (USB). One or more output devices 1344 (e.g., displays, monitors, printers, projectors, or other types of display devices) are also connected to the system bus 1306 through an interface 1346, such as a video adapter.

[0165] The computing system 1300 may operate in a network environment using logical connections to one or more remote computers, such as remote computers 1348. The remote computers 1348 may be workstations, computer systems, routers, peer devices, or other common network nodes, and typically include many or all of the elements described for the computing system 1300. The logical connections schematically shown in 1350 may include local area networks (LANs) and wide area networks (WANs). When used in a LAN network environment, the computing system 1300 may be connected to the local network via a network interface or adapter 1352. When used in a WAN network environment, the computing system 1300 may include a modem or be connected to a communication server on the LAN. A modem, which may be internal or external, may be connected to the system bus 1306 via an appropriate port interface. In a network environment, application programs 1334 or program data 1338 described for the computing system 1300 or a portion thereof may be stored in a remote memory storage device 1354.

[0166] This disclosure includes a detailed description of computing platforms and / or computer implementations, but the teachings enumerated herein are not limited to such computing platforms. Rather, embodiments of this disclosure can be implemented in combination with any other type of computing environment that is currently known or may be developed in the future.

[0167] Cloud computing is a service delivery model that enables convenient on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal administrative effort or interaction with service providers. This cloud model may include at least five features, at least three service models (e.g., Software-as-a-Service (SaaS), Platform-as-a-Service (PaaS), and / or Infrastructure-as-a-Service (IaaS)), and at least four deployment models (e.g., private cloud, community cloud, public cloud, and / or hybrid cloud)). Cloud computing environments can be service-oriented, emphasizing statelessness, low coupling, modularity, and semantic interoperability.

[0168] Figure 14 is an example of a cloud computing environment 1400 that may be used to implement one or more modules and / or systems, as disclosed herein, by one or more examples. Thus, in the example of Figure 14, one or more examples of Figures 1-13 may be referenced. As shown, the cloud computing environment 1400 may include one or more cloud computing nodes 1402 that can communicate with local computing devices used by a cloud consumer (or user), such as a personal digital assistant (PDA®), cellular, or portable device 1404, a desktop computer 1406, and / or a laptop computer 1408. The computing nodes 1402 can communicate with each other. In some examples, the computing nodes 1402 may be physically or virtually grouped in one or more networks, such as a private, community, public, or hybrid cloud, or a combination thereof (not shown). This enables the cloud computing environment 1400 to provide infrastructure, platforms, and / or software as a service, without requiring the cloud consumer to maintain resources on local computing devices. Devices 1404–1408 are illustrative and the computing node 1402 and the cloud computing environment 1400 may communicate with any type of computerized device through any type of network and / or network addressable connection (e.g., using a web browser). In some examples, one or more computing nodes 1402 are used to implement one or more examples disclosed herein for processing and computing data. Thus, in some examples, one or more computing nodes may be used to implement modules, platforms and / or systems as disclosed herein.

[0169] In some embodiments, the cloud computing environment 1400 may provide one or more functional abstraction layers. It is understood that the cloud computing environment 1400 is not required to provide all of the one or more functional abstraction layers (and corresponding functionalities and / or components) as disclosed herein. For example, the cloud computing environment 1400 may provide hardware and software layers that may include hardware and software components. Examples of hardware components include mainframes; RISC (Reduced Instruction Set Computer) architecture-based servers; servers; blade servers; storage devices; and network and networking components. In some embodiments, software components include network application server software and database software.

[0170] In some examples, the cloud computing environment 1400 may provide a virtualization layer that provides an abstraction layer, from which the following examples of virtual entities may be provided: virtual servers; virtual storage; virtual networks including virtual private networks; virtual applications and operating systems; and virtual clients. In some examples, the cloud computing environment 1400 may provide a management layer that may provide the functions described below. For example, the management layer may provide resource provisioning that can provide dynamic procurement of computing resources and other resources used to perform tasks within the cloud computing environment. The management layer may also provide metering and pricing to provide cost tracking when resources are used within the cloud computing environment 1400, and billing or invoicing for the consumption of these resources. In one example, these resources may include application software licenses. Security provides identity verification of cloud consumers and tasks, as well as protection of data and other resources. The management layer may also provide a user portal that provides access to the cloud computing environment 1400 for consumers and system administrators. The management layer may also provide service level management that can provide cloud computing resource allocation and management to ensure that the required service levels are met. Service Level Agreement (SLA) planning and implementation are also provided, and in accordance with the SLA, it may be possible to provide pre-arrangement and procurement of cloud computing resources that are expected to meet future requirements.

[0171] In some examples, the cloud computing environment 1400 may provide a workload layer that provides examples of functions that the cloud computing environment 1400 may utilize for that purpose. Examples of workloads and functions that may be provided from this layer include: mapping and navigation; software development and lifecycle management; virtual classroom education delivery; data analysis processing; and transaction processing. Various embodiments of this disclosure may utilize the cloud computing environment 1400.

[0172] Additional Embodiments

[0173] Embodiments disclosed herein include the following:

[0174] A. A computer implementation method comprising the steps of: providing a scene model, which is a virtual representation of the scene, based on depth and color data captured for the scene; creating a scaled-down version of the scene model corresponding to a diorama of the scene; setting up a virtual camera for the scene model to provide a perspective view of the diorama; and outputting the diorama of the scene as a perspective view on an output device.

[0175] B. A computer implementation method comprising the steps of: receiving a waypoint command to identify virtual points for a digital asset for use in a scene; updating the scene model to include virtual points at locations in the scene model corresponding to locations in the scene and providing a waypoint scene model; providing augmented video data including one or more composite video frames with virtual points in the scene based on the waypoint scene model; and rendering the augmented video data on an output device.

[0176] C. A computer implementation method comprising the steps of receiving prop space data for a prop device in physical space, receiving digital asset space data for a digital asset in virtual space, updating space data for a digital asset based on the prop space data, and updating the position, movement and / or orientation of a digital asset in virtual space based on the updated space data for the digital asset.

[0177] D. A method comprising the steps of receiving prop space data about prop devices in a scene, receiving video data including captured video frames of the scene, generating a scene model of the scene with digital assets, updating the position, orientation, and / or movement of the digital assets in the scene model based on the prop space data, compositing one or more video frames of the received video data and the digital assets with the updated position, movement, and / or orientation, and rendering the composite video frames on an output device.

[0178] Each of embodiments A to D may have one or more of the following additional elements in any combination: Element 1: The above step includes generating extended video data which includes one or more composite video frames with a diorama; Element 2: The above step includes providing extended video data which includes compositing one or more video frames provided by a video camera and a diorama; Element 3: A virtual camera which is further set based on camera viewpoint data for the video camera which specifies a perspective view of the diorama; Element 4: A perspective view which includes a top view, an oblique view, or an inclined view; Element 5: A camera which is a first video camera which is an output device which is a first output device which is a virtual camera which is a first virtual camera which is a perspective view which is a first perspective view which is a method which further includes: setting a second virtual camera in a virtual environment for a scene model which provides a second perspective view of the diorama based on camera viewpoint data for the second video camera which is a Element 6: The first and second output devices are mobile devices; Element 7: Insert one or more digital assets into a scene model representing the digital assets used in the scene; Element 8: The one or more digital assets are the first digital assets, and the method further comprises the step of inserting a second digital asset representing an actor into the scene model, where the movement of the second digital asset in the scene model is synchronized with the movement of the actor; Element 9: The diorama is animated to provide a visual representation of the scene; Element 10: Further comprises the step of receiving animation commands that identify the animation of digital assets between adjacent virtual points of a virtual point, where the waypoint scene model is provided with data specifying the animation of digital assets between adjacent virtual points; Element 11: The above providing step includes the step of generating augmented video data with composite frames, which include the virtual points in the scene and the digital assets between adjacent virtual points;Element 12: The above step includes rendering the extended video onto an output device to provide visual animation of digital assets to and / or between adjacent points based on the waypoint scene model; Element 13: Further comprising the step of creating a scaled-down version of the waypoint scene model corresponding to the scene diorama, setting up a virtual camera for the scene model to provide a perspective view of the diorama; Element 14: The above extended video data is provided with one or more composite video frames accompanied by the diorama; Element 15: The above step of providing the extended video data includes compositing one or more video frames provided by the video camera and the diorama to provide the extended video data; Element 16: The virtual camera is further set based on camera viewpoint data for the video camera to specify a perspective view of the diorama; Element 17: The output device is a portable device, and is one of the following: a mobile phone, tablet, television (TV) device, and laptop computer; Element 18: The diorama is animated to provide a visual representation of the scene; Element 1 9: The prop device is a mobile phone, which provides prop spatial data; Element 20: The step of receiving video data containing captured video frames of a scene, where the prop device is positioned in the scene; and further comprising the step of compositing the received video data and one or more video frames of the digital asset, with position, movement, and / or orientation based on updated spatial data of the digital asset; Element 21: The compositing step provides augmented video data, and the method further comprises the step of rendering the augmented video data onto an output device; Element 22: The output device is one of a portable device and a camera viewfinder; Element 23: The prop device is a first mobile phone, and the video data is provided by a second mobile device; Element 24: The step of receiving video data containing captured video frames of a scene, where the prop device is positioned in the scene; the step of creating a scaled-down version of the scene model corresponding to a diorama of the scene; the step of setting up a virtual camera relative to the scene model to provide a perspective view of the diorama;The method further comprises the step of outputting a diorama of a scene in perspective on an output device, with digital assets having position, movement and / or orientation based on spatial data updated for the digital assets based on prop spatial data; Element 24: The above step includes the step of generating extended video data which includes one or more composite video frames with the diorama; Element 25: The step of generating extended video data which includes the step of compositing one or more video frames provided by a video camera and the diorama to provide extended video data; Element 26: A virtual camera is further set based on camera viewpoint data for the video camera to specify a perspective view of the diorama; Element 27: The camera is a first video camera, the output device is a first output device, the virtual camera is a first virtual camera, the perspective view is a first perspective view, and the method further comprises the step of setting a second virtual camera in a virtual environment for the scene model and providing a second perspective view of the diorama based on camera viewpoint data for the second video camera; and outputting a diorama of a scene, Element 28: The first and second output devices are mobile devices; Element 29: The diorama is animated to provide a visual representation of the scene; Element 30: The step of receiving a waypoint command that identifies virtual points for digital assets about the scene; The step of updating the scene model to include virtual points for locations in the scene that correspond to locations in the scene; The step of providing augmented video data that includes one or more composite video frames with virtual points in the scene, the augmented video data with digital assets with position, movement, and / or orientation based on the updated spatial data; and the step of rendering the augmented video data on the output device; Element 31: The step of receiving an animation command that identifies the animation of digital assets between adjacent virtual points of a virtual point, the digital assets are animated between adjacent virtual points based on the animation command;Element 32: The providing step includes generating augmented video data with a composite frame containing digital assets between virtual points in the scene and adjacent virtual points; Element 33: The above step includes rendering the augmented video onto an output device to provide visual animation of the digital assets to and / or between adjacent points, based at least on the updated spatial data; Element 34: The step further includes creating a scaled-down version of the scene model corresponding to the diorama of the scene; and setting up a virtual camera for the scene model to provide a perspective view of the diorama; and Element 35: The step further includes causing a prop device to provide a prop spatial device, where the prop device is a first mobile phone and the video data is provided by a second mobile device.

[0179] The present invention may be an integrated system, method, and / or computer program product at any possible level of technical detail. The computer program product may include a computer-readable storage medium (or a plurality of computer-readable storage mediums) having computer-readable program instructions for causing a processor to perform aspects of the present invention. The computer-readable storage medium may be a tangible device capable of holding and storing instructions for use by an instruction execution device. The computer-readable storage medium may be, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any preferred combination of those described above. A non-exclusive list of more specific examples of computer-readable storage mediums includes, namely, portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded devices such as punch cards or grooved raised structures on which instructions are recorded, and any preferred combination of those described above. When used herein, computer-readable storage media should not be interpreted as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through optical fiber cables), or electrical signals transmitted through wires.

[0180] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to each computing / processing device, or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface within each computing / processing device receives computer-readable program instructions from the network and transfers them for storage in a computer-readable storage medium within each computing / processing device.

[0181] The computer-readable program instructions for performing the operation of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuits, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk® or C++, and procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or wide area network (WAN), or this connection may be to an external computer (for example, through the Internet using an Internet service provider). In some embodiments, for example, an electronic circuit including a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA) may be personalized by executing computer-readable program instructions by utilizing state information of computer-readable program instructions in order to perform an aspect of the present invention.

[0182] Furthermore, specific embodiments are described herein with reference to block diagrams of methods, systems, and computer program products. It will be understood that the blocks in the diagrams and combinations of blocks within the diagrams can be implemented by computer executable instructions. These computer executable instructions are provided to one or more processors of a general-purpose computer, a dedicated computer, or other programmable data processing device (or combination of devices and circuits) so that instructions executed through the processors can create a machine that implements the functions specified within the blocks or combinations of blocks.

[0183] Furthermore, these computer-executable instructions may be stored in computer-readable memory, and these instructions can instruct a computer or other programmable data processing device to function in a particular manner, so that the instructions stored in computer-readable memory produce a product containing instructions that implement functions specified in a block or multiple blocks of a flowchart. Also, computer program instructions can be loaded into a computer or other programmable data processing device to execute a series of operational steps on the computer or other programmable device, thereby producing a computer implementation process, so that the instructions executed on the computer or other programmable device provide steps for implementing functions specified in a block or multiple blocks of a flowchart.

[0184] These computer-readable program instructions can be provided to the processor of a general-purpose computer, a dedicated computer, or other programmable data processing device to generate a machine, thereby creating means for instructions executed via the processor of the computer or other programmable data processing device to implement functions / operations specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions, which can instruct computers, programmable data processing devices, and / or other devices to function in a particular manner, may be stored in a computer-readable storage medium, so that a computer-readable storage medium storing instructions comprises a product containing instructions that implements modes of functions / operations specified in one or more blocks of a flowchart and / or block diagram.

[0185] Computer-readable program instructions may also be loaded into a computer, other programmable data processing device, or other device to perform a series of operational steps on the computer, other programmable device, or other device, thereby generating a computer implementation process in which the instructions executed on the computer, other programmable device, or other device implement the functions / operations specified in one or more blocks of a flowchart and / or block diagram.

[0186] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions described in a block may occur in an order different from the order shown in the drawing. For example, two blocks shown consecutively may actually be executed substantially simultaneously, or blocks may, in some cases, be executed in reverse order depending on the functions involved. It should also be noted that each block in a block diagram and / or a flowchart diagram, and combinations of blocks in a block diagram and / or a flowchart diagram, may be implemented by a dedicated hardware-based system that performs a specified function or operation, or executes a combination of dedicated hardware and computer instructions.

[0187] The terms used herein are intended to describe only specific embodiments and are not intended to limit the invention. Where used herein, for example, the singular forms "a," "an," and "the" are intended to include the plural form unless otherwise explicitly stated in the context. The terms "include," "include," "equip," "have," and / or "have," and their variations, when used herein, specify the presence of the mentioned feature, integer, step, operation, element, and / or component, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups. In addition, the use of ordinal numbers (e.g., 1st, 2nd, 3rd, etc.) is for distinction, not for counting. For example, the use of "3rd" does not imply that there must be a corresponding "1st" or "2nd." Furthermore, as used herein, the terms “connected” or “connected to,” or “attached” or “attached to,” or “attached” or “attached to,” may indicate the establishment of either a direct or indirect connection, and are not limited to such a connection unless explicitly stated otherwise. Moreover, to the extent that terms such as “include,” “have,” and “own” are used in the Detailed Description, Claims, Appendices, and Drawings, such terms are intended to be inclusive in the same manner as “equipment” when the term “equipment” is adopted as a transitional term in a claim. The term “based on” means “based on at least part of.” The terms “about” and “approximate” may be used to include any numerical value that may vary without changing the fundamental function of the value. When used with a range, “about” and “approximate” also disclose a range defined by the absolute values ​​of the two endpoints; for example, “about 2 to about 4” also discloses the range “2 to 4.” In general, the terms “about” and “approximate” may refer to plus or minus 5 to 10% of the indicated number.

[0188] The above descriptions include only examples of systems, computer program products, and computer implementation methods. Naturally, it is not possible to describe every conceivable combination of components, products, and / or computer implementation methods for the purposes of illustrating this disclosure, and those skilled in the art will recognize that many further combinations and substitutions of the disclosure are possible. While the descriptions of various embodiments have been presented for illustrative purposes, they are not intended to be exhaustive or to limit oneself to the disclosed embodiments. Many modifications and variations will become apparent to those skilled in the art without departing from the scope and spirit of the described embodiments.

Claims

1. A step of providing a scene model, which is a virtual representation of the scene, based on depth and color data captured for the scene; The step of creating a scaled-down version of the scene model corresponding to the diorama of the aforementioned scene; The steps of setting up a virtual camera for the scene model in order to provide a perspective view of the diorama; and The step of outputting the diorama of the aforementioned scene as a perspective view on an output device. A computer implementation method comprising the following:

2. The computer implementation method according to claim 1, wherein the output step includes the step of generating extended video data which includes one or more composite video frames accompanied by the diorama.

3. The computer implementation method according to claim 2, wherein the step of generating augmented video data includes the step of combining one or more video frames provided by a video camera and the diorama to provide the augmented video data.

4. The computer implementation method according to claim 2, wherein the virtual camera is further configured based on camera viewpoint data for the video camera in order to specify the perspective view of the diorama.

5. The computer implementation method according to claim 4, wherein the perspective view includes a top view, an oblique view, or an inclined view.

6. The camera is a first video camera, the output device is a first output device, the virtual camera is a first virtual camera, the perspective view is a first perspective view, and the method further: The steps include setting a second virtual camera in the virtual environment for the scene model and providing a second perspective view of the diorama based on camera viewpoint data for the second video camera; and The step of rendering the diorama of the scene in the second perspective view onto the second output device. The computer implementation method according to claim 4, comprising:

7. The computer implementation method according to claim 6, wherein the first and second output devices are mobile devices.

8. The computer implementation method according to claim 2, further comprising the step of inserting one or more digital assets into the scene model representing the digital assets used in the scene.

9. The computer implementation method according to claim 8, wherein the one or more digital assets are first digital assets, and the method further comprises the step of inserting a second digital asset representing an actor into the scene model, wherein the movement of the second digital asset in the scene model is synchronized with the movement of the actor.

10. The computer implementation method according to claim 1, wherein the diorama is animated in order to provide a visual representation of the scene.

11. The stage of receiving waypoint instructions that identify virtual points for digital assets for use in a scene; A step of providing a waypoint scene model by updating the scene model to include the virtual point at the location in the scene corresponding to the location in the scene; A step of providing augmented video data including one or more composite video frames with the virtual points in the scene based on the waypoint scene model; and The step of rendering the aforementioned extended video data onto the output device. A computer implementation method comprising the following:

12. The computer implementation method according to claim 11, further comprising the step of receiving an animation command that identifies the animation of a digital asset between adjacent virtual points among the virtual points, wherein the waypoint scene model is provided with data specifying the animation of the digital asset between the adjacent virtual points.

13. The computer implementation method according to claim 12, wherein the step of providing includes the step of generating the augmented video data comprising a composite frame with the digital assets between the virtual points and adjacent virtual points in the scene.

14. The computer implementation method according to claim 13, wherein the rendering step includes rendering the extended video onto the output device to provide a visual animation of the digital assets to and / or between the adjacent points based on the waypoint scene model.

15. The step of creating a scaled-down version of the waypoint scene model corresponding to the diorama of the aforementioned scene; and The step of setting up a virtual camera for the scene model and providing a perspective view of the diorama. The computer implementation method according to claim 11, further comprising the above.

16. The computer implementation method according to claim 15, wherein the extended video data is provided with the one or more composite video frames accompanied by the diorama.

17. The computer implementation method according to claim 16, wherein the step of providing augmented video data includes the step of synthesizing one or more video frames provided by a video camera and the diorama to provide the augmented video data.

18. The computer implementation method according to claim 17, wherein the virtual camera is further configured based on camera viewpoint data for the video camera in order to specify the perspective view of the diorama.

19. The computer implementation method according to claim 18, wherein the output device is a portable device, and is one of a mobile phone, a tablet, a television (TV) device, and a laptop computer.

20. The computer implementation method according to claim 19, wherein the diorama is animated in order to provide a visual representation of the scene.

Images