Pre-visualization devices and systems for the film industry

Abstract

The present disclosure relates to a system that captures combined image data and environmental data of an environment. The system uses the environmental data to generate a detailed virtual scan of the environment. From the scan, computer-generated models and image "assets" are inserted into the detailed virtual environment. These assets are scaled and positioned in specific locations and orientations within the virtual environment. The scaled and positioned assets are then composited with a real-time video signal, allowing a user to view the assets in real time on a display.

Description

Technical Field

[0001] This application claims the benefit of priority of U.S. Provisional Patent Application No. 62 / 706,537, filed on August 24, 2020, entitled "PREVISUALIZATION DEVICES AND SYSTEMS FOR THE FILM INDUSTRY", the entire disclosure of which is incorporated herein by reference in its entirety.

Background Art

[0002] In film production, previsualization is the visualization of complex scenes for movies and shows before they are filmed. Previsualization allows directors, cinematographers, or visual effects supervisors to try out all options of various production and art directions, such as lighting, camera placement and movement, staging instructions, and editing, without having to bear the actual production costs. It includes various technologies for the planning and conceptualization of film scenes.

[0003] In television shows and movies, live actors and real environments are combined with computer-generated imagery (CGI). CGI elements include not only moving graphics (vehicles, spaceships, and characters / creatures), but also landscapes and props. Currently, in the preparation and filming of these CGI-driven scenes, green screen technicians physically enter the set with a tall boom with a ball at the tip to simulate the height of specific CGI assets that will later be added to the final video production using computer animation. Other filming location environmental areas may be physically marked with tape to indicate where CGI assets will be placed or pass through. Therefore, it takes a significant amount of time to plan the desired locations for hard-to-visualize CGI assets and place actors there, making preparation and filming difficult.

Summary of the Invention

[0004] The various details of this disclosure are summarized below to provide a basic understanding. This summary is not a comprehensive overview of the disclosure and is not intended to identify or define any specific elements of the disclosure. Rather, its primary purpose is to present some of the concepts of the disclosure in a simplified form prior to the more detailed explanations that follow.

[0005] This disclosure relates to a pre-visualization system that virtually maps an actual film set environment so that filmmakers can see all the elements of a particular scene before filming, and allows scaled 3D CGI assets, such as digital vehicles and creatures, to be inserted into a pre-visualization video feed. This pre-visualization system combines a real-time video signal with at least one CGI asset and provides an augmented video signal to a video display component. The crew members of the filming project, including the director, location scout, and actors, can view the display and adjust acting direction and body movements while reviewing the augmented video signal, thereby making the filming process more efficient.

[0006] Embodiments disclosed herein include a method for generating a pre-visualization video signal for digital imaging. The method includes the steps of generating a raw video signal from an image sensor using a video module, and generating a 3D model of an environment from environmental data collected by an environmental sensor using an environment module. The method further includes the steps of placing a CGI asset at a specific location in the 3D model of the environment using an asset module, and tracking the position and orientation of the image sensor based on data received from a motion sensor associated with the image sensor using a camera tracking module. The method also includes the steps of generating an augmented video signal, which includes a raw video signal containing the placed CGI asset, and displaying the augmented video signal on at least one display. In further embodiments, the environment module is configured to receive a first set of environmental data from a first pre-visualization device and a second set of environmental data from a second pre-visualization device, each of which captures environmental data from different viewpoints of the environment. In another further embodiment, the method further includes the step of occluding features in the raw video signal based on the determined depth of the features using a depth occlusion module. In yet another further embodiment, the method further includes the step of receiving real-time motion capture data from a motion capture system in order to simultaneously animate CGI assets in the augmented video signal using a puppeteer module. In yet another further embodiment, the method includes the step of transmitting the augmented video signal to at least one display. In yet another further embodiment, the method includes the step of recording the raw video signal to a storage device.In another further embodiment, the image sensors are a first image sensor of a first pre-visualization device that generates a first raw video signal, and a second image sensor of a second pre-visualization device that generates a second raw video signal, each of which is augmented with CGI assets based on generated environmental data and a calculated viewpoint of each associated device. In yet another further embodiment, the method includes the step of storing the generated 3D model on a server.

[0007] Embodiments disclosed herein may further include a pre-visualization camera system comprising an image sensor configured to generate a raw video signal and a first environmental sensor configured to acquire environmental measurements of an environment and generate a 3D model of that environment. The system also includes a motion sensor configured to generate camera tracking data associated with the movement of the pre-visualization camera system and a camera viewfinder configured to display the video signal to at least a camera operator. The system further includes a compositor configured to generate an augmented video signal comprising a raw video signal containing a raw video signal that includes a placed CGI asset located within the generated 3D model and the position of the pre-visualization camera system, the augmented video signal being received by the camera viewfinder and displayed to the camera operator. In further embodiments, the system includes a data storage medium configured to record the raw video signal. In yet another further embodiment, the data storage stores the camera tracking data. In another further embodiment, the system includes a camera system interface configured to receive supplemental environmental measurements from a second pre-visualization device having a second environmental sensor that communicates with the camera system interface, the supplemental environmental measurements being used to enhance the fidelity of the 3D model. In yet another further embodiment, the first environmental sensor is of a first type, and the second environmental sensor is of a different second type, the type of environmental sensor being selected from the group including infrared systems, light detection and ranging (LIDAR) systems, thermal imaging systems, ultrasonic systems, stereoscopic systems, and optical systems.

[0008] Embodiments further disclosed herein may further include a pre-visualization system including a camera system comprising a digital processor, a camera image sensor communicating with the digital processor configured to generate a raw video signal from a camera, a camera environment sensor communicating with the digital processor and configured to generate a first set of environmental measurements of the environment, and a camera display communicating with the digital processor and configured to display a first extended video signal. The pre-visualization system also includes a digital pre-visualization device comprising a device environment sensor configured to communicate with the camera system and generate a second set of environmental measurements of the environment. The digital processor generates a 3D model of the environment based on the first set of environmental measurements and the second set of environmental measurements, and places CGI assets in position and orientation within the 3D model. The camera display is configured to display a camera extended video signal including the raw video signal from the camera and the placed CGI assets. In further embodiments, the system further includes a storage device configured to record the raw video signal from the camera. In another further embodiment, the system further includes a motion capture system configured to generate animation data, and a processor in the pre-visualization system animates the placed CGI assets in real time based on the generated animation data. In yet another further embodiment, the system further includes a remote monitor configured to display the extended video signal from the camera.In another further embodiment, the pre-visualization device further includes a device processor, a device image sensor configured to communicate with the processor and generate a raw video signal of the device, and a device display configured to display an augmented video signal of the device, wherein the device processor generates a device 3D model of the environment based on a first set of environmental measurements and a second set of environmental measurements, and places CGI assets in position and orientation within the 3D model, and the device display displays an augmented video signal of the device, including the raw video signal of the device and the placed CGI assets. In another further embodiment, the camera environment sensor is of a first type, and the device environment sensor is of a different second type, the type of environment sensor is selected from the group including infrared systems, light detection and ranging (LIDAR) systems, thermal imaging systems, ultrasonic systems, stereoscopic systems, RGB cameras, and optical systems. In another further embodiment, the system includes a server that communicates with a camera system and a pre-visualization device, and is configured to receive a first set of environmental measurements and a second set of environmental measurements, and to distribute each set of environmental measurements to the connected pre-visualization system and camera system, respectively.

[0009] The following figures are included to illustrate specific aspects of the embodiments and should not be considered exclusive embodiments. The disclosed subject matter is subject to considerable modifications, alterations, combinations, and equivalents in form and function, so as may occur to those skilled in the art and those interested in the disclosure. [Brief explanation of the drawing]

[0010] [Figure 1] This figure shows an electronic device for pre-visualization of CGI assets as disclosed herein. [Figure 2] This figure shows an exemplary pre-visualization camera system as disclosed herein. [Figure 3]This figure shows an exemplary pre-visualization system as described herein. [Figure 4] This figure illustrates an exemplary pre-visualization system where multiple devices view the same environment, each having a different state of the inserted CGI assets. [Modes for carrying out the invention]

[0011] A more complete understanding of the components, processes, and equipment disclosed herein can be obtained by referring to the accompanying drawings. These drawings are merely schematic representations for convenience and ease of illustration of the disclosure and are therefore not intended to show the relative sizes and dimensions of the devices or their components, and / or to define or limit the scope of the exemplary embodiments.

[0012] In the following description, specific terminology is used for clarity, but these terms refer only to specific structures of the embodiments selected for illustration in the drawings and do not define or limit the scope of disclosure. In the drawings and the following description, similar numerical notations should be understood to refer to components of similar function.

[0013] The singular forms "a," "an," and "the" refer to multiple objects unless the context clearly indicates otherwise.

[0014] As used herein, the terms “generally” and “substantially” encompass structural or numerical modifications that do not substantially impair the purpose of the elements or numbers modified by such terms.

[0015] The terms "about" and "approximately" can be used to include any number that may vary without changing the fundamental function of its value. When used with a range, "about" and "approximately" also disclose a range defined by the absolute values ​​of two endpoints; for example, "about 2 to about 4" also discloses the range "2 to 4". In general, the terms "about" and "approximately" can refer to plus or minus 10% of some given number.

[0016] As used herein, the term “CGI Asset” means a digital creation, rendering, or model of an object. CGI assets include, but are not limited to, automobiles, spaceships, monsters, creatures, machines, tables, statues, buildings, animals, and weapons. CGI assets are created by digital artists, graphic designers, and others.

[0017] As used herein, the term “raw video signal” means a video signal obtained directly from the image sensor of a camera, for example, capturing the environment of a filming location and actors performing in a scene. As used herein, the term “extended video signal” means a video signal that includes a combination of a raw video signal and at least one CGI asset placed in the environment, for example, a video signal that includes the environment, actors, and a CGI asset.

[0018] An exemplary embodiment of this disclosure relates to a system for capturing a combination of depth / environmental data of a movie set environment and image data from a video signal. The system uses depth / environmental data of the surrounding movie set environment to generate a detailed virtual scan of the same environment ("Virtual Environment"). Computer-generated models and images, i.e., CGI assets, are inserted into the detailed virtual environment. These CGI assets are scaled and placed at specific locations within the virtual environment, with specific orientations and, in some cases, predetermined animations / movements. The scaled and placed CGI assets are composited with the raw video signal in real time, allowing the system's user to view the CGI assets in the environment on a display in real time. As the user moves through the real environment, the CGI assets may appear stationary relative to the real environment, or, in situations where the CGI assets consist of predetermined animations, they may appear to move relative to the real environment and move within the volume of the CGI assets. The system can accommodate multiple image capture devices and depth capture devices and combine all the collected data to enhance the fidelity of the virtual environment. Each image capture device and depth / environment capture device views the same real-world environment from different reference frames, allowing CGI assets to be seen from its individual perspectives.

[0019] Referring here to Figure 1, an exemplary embodiment of a pre-visualization system 10 is shown, which includes a digital pre-visualization device 100a configured for use in an environment mapping and CGI asset insertion visualization system. The various components shown in Figure 1 are for illustrative purposes only, and it will be understood that other similar components implemented by hardware, software, or a combination thereof can be substituted therewith without departing from the scope of this disclosure.

[0020] Figure 1 shows an exemplary digital pre-visualization device 100a, which includes at least one image sensor 102 for capturing visual data, an environment sensor 103 for capturing environmental data, at least one motion sensor 104 configured to detect the orientation of the digital pre-visualization device 100a, a processor 106, and a storage medium / memory 108.

[0021] The digital processor 106 is configured to control the operation and components of the digital pre-visualization device 100a and can execute applications, apps, and instructions stored in the device memory 108 and / or accessible via the communication device 110. The digital processor 106 can be variously embodied as a single-core processor, a dual-core processor (or more commonly a multi-core processor), a digital processor and cooperating math coprocessor, a digital controller, a graphics processing unit (GPU), and the like. In some embodiments, the digital processor 104 and memory 108 may be combined on a single chip.

[0022] At least one image sensor 102 may be, for example, a charge-coupled device (CCD) sensor or a complementary metal-oxide semiconductor (CMOS) sensor configured to capture visual data and generate a video feed (e.g., a series of images). In other words, the image sensor 102 may be an analog, digital, or a combination thereof camera. The image sensor 102 detects and transmits information used to create images or videos that can be stored in memory 108 or transmitted to other storage media or devices via the onboard communication device 110. For example, the image sensor 102 may generate a video signal that can be transmitted to another device 100n via Wi-Fi.

[0023] Digital previsualization 100a also includes an environmental sensor 103. This sensor is configured to scan the surrounding environment and determine its geometric arrangement and spatial configuration. The environmental sensor 103 can operate by capturing depth points used by the system 10 to construct a virtual environment that includes a scaled virtual 3D model of the environment. The environmental sensor 103 can be embodied in various ways as an infrared system, a light detection and ranging (LIDAR) system, a thermal imaging system, an ultrasonic system, a stereoscopic vision system, an RGB camera, an optical system, or any device / sensor system currently known in the art, and combinations thereof. For example, without limitation, the environmental sensor 103 can be an infrared emitter and sensor. A common infrared emitter projects infrared dots in a known pattern onto the surrounding environment. These infrared dots are not within the visible spectrum of the human eye and typically do not interfere with the capture of raw video signals. The infrared dots are then captured by either an infrared sensor or an image sensor 102 for analysis in determining the geometric arrangement and spatial configuration of the surrounding environment. In other embodiments, the environmental sensor 103 can be a LIDAR-based system. In a LIDAR-type system, a pulsed laser is shone on the surrounding environment, and the time it takes for the laser signal to return is used to generate a 3D model of the environment with high accuracy.

[0024] Since each can have advantages based on the physical mechanism utilized for capture, it will be understood that any sensor system or combination of sensor systems can be utilized as the environmental sensor 103. For example, it may be difficult for an infrared system to capture an outdoor environment during the day because the surrounding environment may be saturated with infrared, and thus it may be difficult for the sensor to accurately capture the emitted pattern and accurately reproduce a virtual environment of the geometric and spatial configuration.

[0025] The motion sensor 104 can be a sensor or a combination of sensors that can detect the motion, orientation, acceleration, and position of the preview visualization device 100a. Thus, the position and orientation of the device 100a with respect to the environment (and the virtual environment) can be determined. The motion sensor 104 can be variously embodied as a gravity sensor, an accelerometer, a gyroscope, a magnetometer, etc., or a combination thereof. For example, without limitation, the motion sensor 104 can be a commercially available inertial measurement unit (IMU) as a sensing unit that includes an accelerometer, a gyroscope, and a magnetometer.

[0026] The gravity sensor is a motion sensor 104 configured to measure the orientation of the digital preview visualization 100a with respect to the direction of gravity and generate orientation data regarding the same. With the gravity sensor, the digital preview visualization device 100a can recognize the direction of gravity with respect to the device 100a (by the processor 106), for example, based on a calculated three-dimensional vector. The gravity sensor can indicate the orientation of the digital preview visualization device 100a, such as the degree of rotation with respect to the direction of gravity.

[0027] The accelerometer is a motion sensor 104 configured to detect a change in velocity during a time period, sense acceleration, and generate orientation data regarding the same. A three-axis accelerometer can include a plurality of motion sensors arranged in the x-axis, y-axis, and z-axis directions. The processor 106 of the digital preview visualization device 100a receives data values measured in multiple axial directions as vector values from the accelerometer. Then, the processor 106 can determine the direction in which the digital preview visualization device 100a is rotated or tilted based on the values obtained for the three axes.

[0028] A gyroscope is a motion sensor 110 configured to calculate the angle at which device 100a is rotating around an axis and to generate orientation data related to that angle. This can be expressed as a numerical value. A 3-axis gyroscope calculates the degree to which device 100a is rotating around three axes. In this way, at least one motion sensor can generate information (data) about the position and orientation of device 100n and generate orientation data about it relative to the environment.

[0029] In some embodiments, the electronic device 100a is equipped with a communication device 110 configured to communicate with other digital pre-visualization devices 100n (equipped similarly to device 100a), a server 160, a network cloud 170, a storage device, and the like. The communication device 110 may include wired communication components, wireless communication components, cellular communication components, near-field communication components, Bluetooth® components, Wi-Fi components, and other communication components for providing communication via other modalities. This list of exemplary communication devices is for illustrative purposes only and does not preclude the use of any alternative or combination of one or more of these components, or the use of any other communication components that perform substantially the same functions in substantially the same manner. Environmental data, orientation data, and image data may be transmitted to other connected devices via the communication device 110.

[0030] The digital pre-visualization device 100a may also include a user interface 112 configured to receive commands from the user of device 100a. The user interface may include, but is not limited to, a touchscreen device, keyboard, mouse, motion sensor, buttons, knobs, voice actuation, headset, hand recognition, gaze recognition, etc. The user interface 112 may present the user with a graphical interface that facilitates the operation of device 100a and various other components of system 10, or components connected thereto.

[0031] The processor 106 can access device memory 108, storage 161 on a remote server 160, cloud-based storage 170 containing a database of CGI assets, or other onboard or remote storage devices. As briefly stated above, a CGI asset is a digital creation, rendering, or model of an object, including a predetermined three-dimensional shape and dimensional scale. CGI assets are defined and generated by the visual effects department / graphic designer and can be uploaded to a database of CGI assets for use by digital pre-visualization devices 100a-100n. In some embodiments, a CGI asset includes "animation data," i.e., joint positions, rotations, translations, scales, motions, and keyframes, where keyframes define the start and end points of any smooth change in the animation. This animation data may be compiled into a CGI asset file or in a separate data file associated with a particular CGI asset. A CGI asset may have multiple associated animation files to accommodate various movements (such as the movement of the asset in various directions and at various speeds). As a non-limiting example, animation data may include the flight path of a landing spaceship, the movement of a destructive monster's swinging arms, and the flying motion of rolling grass. In some embodiments and in the more detailed description below, animation data associated with a CGI asset can be manipulated / modified via the user interface 112, for example, to speed up / slow down the movement of the CGI asset.

[0032] After 3D mapping the surrounding environment, the stored CGI assets can be selected, positioned, and appropriately scaled in the three-dimensional virtual environment by a user operating the user interface 112. A user viewing the display 114 of device 100 sees both a real-time image of the surrounding environment captured by the image sensor 102 and the CGI assets positioned accordingly, scaled and oriented in a specific direction. A user of device 100a can physically move around in the surrounding environment while the environment sensor 103 and motion sensor 104 continuously capture environment data and the relative motion of device 100a, respectively. Digital pre-visualization 100a displays the CGI assets positioned within the captured video signal in their original position and orientation relative to device 100a's new real-time position. That is, the display 114 displays the CGI assets as if they were an actual part of the environment captured by the image sensor 102. As a non-limiting example, a user could stand on the 10-yard line and capture the surrounding environment of a football field while placing an asset on the 50-yard line. The user could then move to the 30-yard line and still see the CGI asset on the 50-yard line, revealing its lateral movement as well.

[0033] In some embodiments, and as briefly described above, a CGI asset may have associated animation data. Here, a starting point for the animated CGI asset may be selected, and the animation associated with that asset may be executed by system 10. For example, but not limited to, a user of device 100 may select a starting and / or ending point in a virtual environment mapped to the surrounding environment, and the CGI asset is configured to move from the starting point to the ending point based on predetermined animation data. In some embodiments, at least one device 100 includes an interface 112 configured to modify the animation data on the fly. The user may be able to adjust the entire animation or a part / section of the animation. That is, the user can select or generate keyframes in the animation data and adjust parameters (e.g., speed) of the animation data before or after the selected / generated keyframes.

[0034] Additional digital pre-visualization devices 100n, which can be equipped in a similar manner to the digital pre-visualization device 100a, can be connected to the pre-visualization device 100a directly or indirectly via a local server 160 or the Internet 170. Although one additional device 100n is illustrated, it should be understood that any number of devices 100a-100n can be connected to it without departing from the scope of this disclosure. These additional digital pre-visualization devices 100n synchronize with the environmental data collected and processed by the electronic device 100a, and further complement the environmental data generated by the first device 100a with additional measurements (data) generated by the additional devices 100n. The additional measurements by the additional devices 100n increase the fidelity of the three-dimensional model of the environment, making the entire system 10 more accurate. Placed and oriented CGI assets can be viewed on the display of the additional device 100n for its particular position and viewing angle. In other words, each additional device 100n may have a different view of the environment and CGI assets than the digital pre-visualization device 100a, but the CGI is viewed by each as if it were in the same location in the real environment.

[0035] A pre-visualization camera system will be described in accordance with another aspect of this disclosure. Pre-visualization camera systems, such as the pre-visualization system 10 described above, have particular applications in the film industry in light of the increasing use of computer-generated visual effects, including computer-generated characters, vehicles, creatures, and environments. A pre-visualization camera system allows those involved in a film project (film, television episode, etc.), such as a director, to review not only what has been captured by cameras on set, but also CGI assets integrated into camera shots, while maintaining a clean video signal for recording. This provides an opportunity to review a comprehensive visualization of a sequence in real time and to immediately adjust camera angles and positions, etc., thereby enabling efficient sequence shooting and direction.

[0036] This camera system also allows crew members to "scout" locations. That is, crew members can virtually lay out sets, scenes, actions, or sequences before filming begins, in order to visualize the workflow once formal production starts at the chosen location. This camera system also takes into account long-distance filmmaking, such as during pandemics or overseas productions. Actors can be captured in real time at a single location somewhere in the world, and their movements can be transformed into virtual puppet-like CGI assets that move in a 3D capture volume exactly as seen by the camera in real time, and can be directed by the director's voice from around the world. The reverse is also possible; while real sets and actors receive instructions from around the world, the director does not need to be on set, and the sets, locations, and actors can be virtually projected to the director. Other aspects and advantages will become clear in the following description.

[0037] Figure 2 shows an exemplary pre-visualization camera system 200 used in the film industry, although camera 201 may be applied to fields other than film. The most advanced systems currently available include digital movie cameras 201 for digital filmmaking, which digitally capture images by rapidly shooting still images with an internal image sensor (e.g., the CMOS sensor mentioned above). This differs from historical movie cameras used for shooting on film stock. Several digital video cameras specifically designed for high-end digital filmmaking applications are available on the market. These cameras typically offer the ability to use relatively large sensors, selectable frame rates, low-compression or sometimes uncompressed recording options, and high-quality optics. These are marketed by vendors including, but not limited to, Sony, Red, and Canon, and include Sony's CineAlta® series, Red's ONE®, and Panavision's Genesis®.

[0038] The image sequence (video) is typically recorded as image and / or video files (e.g., .jpg, .mov, .mpeg, etc.) on a hard drive 204 or flash memory. These files can be easily copied to another storage device connected to the editing system, typically a large RAID (Redundant Array of Inexpensive Disks). The editing system may include a computer and / or additional equipment such as switchers, capture / playback devices, encoding devices, and color correction devices. Once the data is copied from the on-set storage medium (hard drive 204) to the storage array, the on-set storage medium (hard drive 204) is erased and returned to the set for further shooting.

[0039] Currently, the digital camera generates a raw video signal 211 that is recorded onto a storage medium and displayed on both the viewfinder 216 of the camera 201 and at least one monitor 210 in the on-set video village 208. The viewfinder 216 of the digital camera 201 allows the camera operator to accurately view in real time what the image sensor 202 of the camera 201 is capturing and recording. The raw video signal 211 generated by the image sensor of the camera 201 is also transmitted (wired or wirelessly) to the video village 208. The video village 208 is a location on set where at least one large monitor 210 is set up so that key members of the film crew can monitor the footage being shot. These crew members, including the director and cinematographer, watch the raw video signal in real time to notice (and correct) any potential problems.

[0040] As mentioned above, shooting sequences / scenes that include CGI assets presents challenges because these CGI assets do not physically exist on set and are added to the scene considerably later in the shoot. The difficulties relate to directing the actors' gaze (i.e., where the actors should look relative to the CGI assets) and their positioning (i.e., where the actors should stand relative to the CGI assets). The pre-visualization system 200 described herein addresses these and other challenges.

[0041] Generally, the pre-visualization camera system 200 captures environmental data associated with a video signal and can combine the raw video signal with a predetermined scale of CGI assets 250, so that filmmakers can "see" CGI assets in a scene and give instructions accordingly. In exemplary embodiments, it will be understood that the CGI asset 250 is illustrated as a cube. This is simply a simplification for illustrative purposes, and the CGI asset 250 can be any shape and may have associated motion or animation as described above. The pre-visualization camera system 200 includes a digital video camera 201 having at least one image sensor 202 configured to capture a sequence of images. The image sensor 202 of the digital video camera 201 is similar in some respects to the image sensor 102 and is best understood in that respect. The image sensor 202 of the digital camera 201 transmits a sequence of images as a raw video signal 211 to a computer 206 via an input / output interface 215. In some embodiments, the pre-visualization camera system 200 includes an image capture device 205 configured to intercept the raw video signal 211, which is the output of the image sensor 202. The capture device 205 can be implemented in various ways, such as capture card hardware for acquiring signals for operation. The capture device 205 may be located inside the computer system 206 or may be external hardware communicating with the computer system's interface 215. In some embodiments, the raw video signal 211 is sent to both the computer 206 and the storage medium 204 via the interface 215 for recording and storage. In this way, the raw video signal 211 generated by the image sensor 202 of the camera 201 is recorded and stored for later processing, for example, by the visual effects department during post-production. As defined above, the raw video signal 211 means the video signal directly generated from the image sensor 202 of the camera 201 without the addition of the CGI assets described later.

[0042] The computer 206 may be integrated into the digital camera 201 or positioned near the camera 201, for example by custom mounting hardware, brackets, braces, etc. In this way, potential radio interference from Wi-Fi and other signals / frequencies commonly found in shooting locations is reduced. Without departing the scope of this disclosure, the computer 206 can be embodied in various ways, for example, as a personal computer (illustrated), a tablet, a smartphone, or other known device hosting a software platform, operating system, and / or application. The computer system 206 may also be configured to interface with any known camera system 201 and perform synthesis of raw video signals, environmental data, and CGI assets to create an extended video feed 212. That is, the computer system 206 may have plug-and-play functionality to connect to any camera and / or sensor and receive digital signals from there.

[0043] The computer system 206 may be any of various commercially available processors and may be similar in some respects to processor 106 in Figure 1, and therefore may be best understood in relation to it. The computer 206 also includes at least one user interface 207 and / or display 209 configured to present relevant data captured by the pre-visualization camera system 200, including displaying CGI assets 250, environmental data, and / or video signals from the image sensor 202. The user interface 207 also allows the user to input commands into the computer 206 to monitor and control various components of the pre-visualization system 200. In the exemplary embodiment, the user interface 207 is a keyboard, but it will be understood here that it may be replaced by other user interfaces, such as a touchscreen interface, computer mouse, etc. The computer 206 may also host an operating system including, but not limited to, Windows®, Linux®, Apple®, Android®, or a proprietary operating system. In some embodiments, the computer includes a graphics processing unit (GPU) configured to process video signals generated by at least the image sensor 202.

[0044] The pre-visualization camera system 200 also includes at least one environmental sensor 203, which is mounted on either a digital camera 201 or an auxiliary component / mounting bracket, hardware, etc. The environmental sensor 203 is configured to capture the geometric arrangement and spatial configuration of the surrounding environment, for example, by generating depth points, and may be similar in some respects to the environmental sensor 103 in Figure 1, and is therefore best understood in relation thereto. The environmental data captured by the environmental sensor 203 is transmitted to a computer 206 for processing, for example via an interface 215. The computer 206 uses the environmental data continuously and / or periodically generated by the environmental sensor 203 to generate a three-dimensional model / mesh of the surrounding environment ("virtual environment"). As will be explained in more detail below, the computer 206 can access a database of CGI assets 250 (vehicles, characters, creatures, props, scenery, etc.) in which the dimensions of each CGI asset are defined, and can insert the CGI assets 250 into a generated virtual environment so that a user of the system viewing the monitor 210 can see the CGI assets that have been placed in the video since they were captured by the image sensor 202.

[0045] Digital cameras 201 are generally configured to accept a variety of interchangeable optical lenses. In filmmaking, camera lenses have a significant impact on the appearance of the image and the recording of the filmmaker's intended narrative. Lenses include, but are not limited to, wide-angle lenses, fisheye lenses, and zoom lenses. Two basic parameters of an optical lens are focal length and aperture value. The focal length of a lens determines the magnification of the image projected onto the image plane, and the aperture determines the amount of light in that image. For a given photographic system, the focal length determines the angle of view, with shorter focal length lenses providing a wider field of view than longer focal length lenses. Opening the aperture, indicated by a small F-number, allows for the use of faster shutter speeds for the same exposure. A side effect of using lenses with different focal lengths is that the distance at which the subject can be framed differs, resulting in different perspectives. Considering the various perspectives associated with different lenses, it may be necessary to calibrate the environmental sensor 203 for each individual optical lens used in the camera 201. Calibration for focal length, aperture, zoom field of view, lens type, and / or camera film back is calculated to ensure that CGI assets are properly scaled and positioned within the augmented video signal. For example, a wide-angle lens may cause buildings at the edges to appear distorted, and changing lenses alters parallax, so calibration may also include correcting for parallax as an image.

[0046] In some embodiments, a companion application / module containing a database of optical lenses and preset calibration values ​​(focal length, zoom, aperture, lens type, camera film back, camera focal plane, camera sensor size) is configured to run on a computer system 203. When the optical lens of camera 201 is replaced, the user of the companion application can select a new optical lens, and the companion application calibrates the camera system 200 so that the data from the environmental sensor 203 and the raw video signal 211 from camera 201 contain substantially similar scales. This ensures that the captured environmental data matches the ratio of the captured image data, enabling proper placement and observation of the inserted CGI asset 250. Since different cameras may have different image sensor sizes that contribute to the size of the field of view, the size of the image sensor 202 of the digital video camera 201 is also included as a factor in the calibration calculation.

[0047] The pre-visualization camera system 200 also includes a motion sensor 230 attached to the camera 201 so that the position, orientation, and movement of the camera 201 can be monitored, tracked, and stored. In this way, the pre-visualization camera system 200 can calculate the position and orientation of the camera 201 relative to a 3D model of the generated environment, with respect to the origin. Since all camera movements (transformations) are tracked and stored, the camera's position relative to what is captured (environmentally captured) is known. This helps visual effects artists to add the final high-resolution CGI assets to the raw video signal to create a video work, as they have little to no need to think about how to prepare the effects visually. For example, at a particular frame of the video signal, the position and rotation of the CGI asset 250 can be known because the position and rotation of the camera 201 relative to the mapped virtual environment are known. Camera transformation position and rotation are captured and tracked by timestamps, but may not strictly match the timestamp of the current camera frame. The exact position of the virtual camera, and therefore the relative position of the CGI assets, can be interpolated between the nearest recorded camera transformation captures. In this way, the camera image, camera position and rotation, the virtual camera displaying the image on the compositor, and all CGI assets are synchronized to their precise positions.

[0048] In some embodiments, the pre-visualization camera system 200 also includes an input / output interface 215 configured to transmit data collected by camera 201 and various sensors (e.g., sensor 203) and / or a video signal (enhanced video signal 212) enhanced with placed CGI assets. For example, computer 206 enhances the CGI assets into an enhanced video signal 212 by combining the raw video signal 211 from the image sensor 202 of camera 201 with a 3D model of the environment constructed from environmental data. This enhanced video signal 212 may be transmitted to the camera's viewfinder 216 so that the camera operator can see the CGI assets while camera 201 is shooting. Simultaneously, the enhanced video signal 212 may be transmitted to a monitor 210 in a video village 208 so that crew members can view the camera's field of view, including the CGI assets 250, in real time.

[0049] In some embodiments, the augmented video signal 212 is recorded on a storage medium such as a storage medium 204. During filming, the raw video signal 211 (without low-resolution CGI assets) and the augmented video signal 212 (with low-resolution CGI assets) of the same sequence / scene are captured. In the same way that the pre-visualization system 200 allows the director to optimally direct actors to react to CGI assets 250 that are not actually present, the augmented video signal 212 provides guidance to the visual effects department to process the raw video signal 211 and add high-definition visual effects. That is, post-production visual effects artists may have difficulty optimally positioning high-resolution CGI assets. Using the augmented video signal 212, which previously helped actors position themselves and react to CGI assets, the process of placing the final visual effects within the raw video signal to complete the work is facilitated.

[0050] In another aspect of this disclosure, referring to Figure 3, an exemplary pre-visualization system 300 for the film industry is given. While this disclosure describes a pre-visualization system in relation to the filming of movies and television programs, it should be understood that this disclosure may also be applicable to other similar uses. The various components shown in Figure 3 are for illustrative purposes of an exemplary embodiment, and it should be further understood that other similar components implemented by hardware, software, or a combination thereof can be substituted therein. System 300 is configured to combine the raw video signal generated by a digital camera 201 with CGI assets placed in a three-dimensional model of the environment in which the filming took place.

[0051] As shown in Figure 3, System 300 includes a central system, generally referred to as Computer System 206, capable of implementing the exemplary methods described below. As stated above, Computer System 206 can be embodied in various ways without departing from the scope of this disclosure. The exemplary Computer System 206 includes a processor 306, which implements the exemplary methods by executing processing instructions 310 stored in a memory 308 connected to the processor 306, and also controls the overall operation of Computer System 206. In some embodiments, the processor 306 and the memory 308 may be integrated on a single chip.

[0052] All the various components of the computer system 206 can be connected by the data / control bus 320. The processor 306 of the computer system 206 can communicate with associated data storage 301, digital video camera 201 / image sensor 202, environmental sensor 203, motion sensor 230, and other digital pre-visualization devices 100a-100n via communication link 342. The processor can also communicate with other components, including server 160, cloud network 170, video village 208, and viewfinder display 216, via link 343. Although each component is illustrated to connect to the computer via one of the two illustrated links 342, 343, it will be understood that the number of links 342, 343 is not limited, and any component can connect to the processor via any communication link. Appropriate communication links 342, 343 may include, for example, a dedicated communication network, infrared, optical, or other appropriate wired or wireless data communication.

[0053] Instruction 310 includes a video module 330 configured to receive and process a raw video signal (e.g., raw video signal 211) from the image sensor 202 of a digital video camera 201. The raw video signal 211 is an electronically reconstructed moving visual image in the form of encoded digital data. The raw video signal may be characterized by the number of pixels supported horizontally, for example, 1080P, also known as HD, 2K, or BT.709. In prior art digital video cameras, the raw video signal 211 is passed directly to the viewfinder 216, allowing the camera operator to view the raw video signal 211. In some embodiments, the raw video signal 211 from the image sensor 202 is split and sent both to a computer 206, which is received and processed by the video module 230, and to a storage medium such as a storage medium 204 or 301.

[0054] In some embodiments, the video module 330 is configured to change the video encoding format of the raw video signal 211. That is, the encoding format of the raw video signal 211 received from the image sensor 202 of the camera 201 is changed by the video module 330, and the newly formatted raw video signal is transmitted to the viewfinder 216 or the display of the video village 208. Examples of video encoding formats include, but are not limited to, H.262 (MPEG-2 Part 2), MPEG-4 Part 2, H.264 (MPEG-4 Part 10), HEVC (H.265), Theora, RealVideo RV40, VP9, ​​and AV1. In other embodiments, the video module 330 is configured to change the video encoding of the video signal.

[0055] In further embodiments, the video module 330 is configured to modify the compression of the raw video signal 211. That is, the raw video signal 211 can be compressed to make the video file size smaller than the original format. In embodiments where the video module 330 compresses the raw video signal 211, processing of the compressed raw video signal by the processor 306 and other devices / modules may be faster compared to the uncompressed raw video signal 211. The video module 330 may also optimize the performance characteristics of the raw video signal 211, such as, but not limited to, frames per second (FPS), video quality, and resolution. This optimization allows the raw video signal to be processed more efficiently and quickly, resulting in smoother operation of the system 200.

[0056] Instruction 310 also includes an environment module 332 configured to receive environmental data from one or more environment sensors 203 attached to the camera 201, or otherwise provided as a reference thereof, as illustrated with respect to Figure 2. The environment module 332 uses the environmental data obtained by the environment sensors 230 to determine the geometric arrangement and spatial configuration of the surrounding environment by locating the positions of geometric objects and / or points and determining the distances between those objects and / or points. The environment module 332 uses the environmental data continuously generated by the environment sensors 203 to create a three-dimensional model / mesh (virtual environment) of the environment. In some embodiments, the environment module / environment sensors periodically (not continuously) generate and update the environmental data and virtual environment. In some embodiments, the environment module 332 uses a photogrammetry algorithm to generate a virtual environment of the surrounding environment using one or a combination of the video signal generated by the camera and the environmental data generated by the environment sensors. For example, multiple photographs from one or both of the image sensor and data sensor may be stitched together to construct a three-dimensional model of the surrounding environment. In other embodiments, point data generated from points observed in a real environment (either object features or infrared point illumination), such as a point cloud, is converted into a mesh (polygon or triangular mesh) model that represents the actual surrounding environment.

[0057] In some embodiments, the environment module 330 is configured to modify the level of detail (LOD) of the generated 3D model, including but not limited to the geometrical arrangement details and pixel complexity in the virtual environment. Generally, in computer graphics, considering LOD may involve reducing the complexity of the 3D model representation. LOD techniques improve the efficiency of 3D rendering by reducing the load on graphics processing.

[0058] In some embodiments, the environment module 332 is configured to receive environmental data from other devices communicating with the pre-visualization system 300. For example, the environment module 332 may receive environmental data from an environmental sensor 203 attached to the camera 201, and from additional digital pre-visualization devices and environmental sensors 103, such as the digital pre-visualization devices 100a-100n described in more detail above with respect to Figure 1. In some embodiments, the environment module 332 is configured to receive environmental data provided to the server and cloud by the connected devices 100a-100n, 203 from the server 160 and / or the cloud network 170.

[0059] Instruction 310 also includes an asset module 334 configured to retrieve a CGI asset (e.g., CGI asset 250) and insert the selected CGI asset into a 3D modeling environment (virtual environment) generated by the environment module 332. The CGI asset 250 may be stored in a database on storage device 301, or it may be accessible via cloud storage or removable storage 161 that can communicate with server 160 to the asset module 334. The CGI asset 250 in the database is defined as having a three-dimensional shape and dimensions. For example, but not limited to, the CGI asset could be a CGI dragon model having a three-dimensional body and a predetermined dimensional scale, i.e., the dragon could be configured to be approximately 50m in length. Thus, the CGI dragon would appear much larger than a typical human in the image standing next to the creature. In some embodiments, the CGI asset may be uploaded directly or remotely (via cloud 170 or remote server 160) as desired, and immediately accessible to the asset module 334. Animation data can also be embedded in CGI asset files, and / or CGI asset 250 can be associated with one or more animation files containing motion sequences, keyframes, e.g., the position, rotation, translation, and scaling of animation joints. World coordinates and scale of vertices are stored. Texture UV UDIM tile data, timestamps, frame times, animation curve data, and keyframes are stored. For example, if a director on set wants to introduce a new CGI asset 250 into a scene, the director can have the visual effects team upload the desired CGI asset 250 to system 300 via cloud 170. The director then has the ability to instantly place the new CGI asset in the scene and adjust the shooting and actor direction accordingly. This also applies to sets, environments, and set extensions.In some embodiments, the asset module 334 is configured to modify the level of detail (LOD) of the CGI asset 250, including but not limited to the geometrical array details and pixel complexity within the asset. LOD techniques improve the efficiency of 3D rendering by reducing the graphics processing load. In some embodiments, the user may modify the animation data associated with the CGI asset, for example, by speeding up or slowing down all or part of a predetermined motion.

[0060] Instruction 310 also includes a camera tracking module 336 configured to determine the position of camera 201 relative to a 3D model generated by environment module 332. Camera tracking module 336 receives real-time spatial tracking data of camera from camera 201 or motion sensors 230 placed on the camera rig. In this way, as the camera operator moves camera 201 to capture the environment and scene at various angles, the calculated movement of camera 201 by camera tracking module 332 is reported to the display of the CGI asset in the augmented video signal. In other words, camera tracking of camera 201 ensures that although the environment and the camera's field of view are changing, the placed CGI asset 250 remains reliably in its selected position despite being viewed from a changed camera viewpoint. In some embodiments, the camera tracking module 336 triangulates the position of each device (camera 201, pre-visualization devices 100a-100n) based on visual matching from all devices (camera 201, pre-visualization devices 100a-100n) to real-time 3D scans.

[0061] In some embodiments, the lighthouse device 360 ​​is configured to track the location of camera 201 and each pre-visualization device 100a-100n. The lighthouse device 360 ​​can be embodied in various ways. In some embodiments, the lighthouse device 360 ​​includes at least one camera or environmental sensor that detects the presence and location of users or devices in the set. The at least one camera (image sensor) or environmental sensor may be analogous to image sensor 201 and environmental sensor 203, and is best described in that regard. The lighthouse device 360 ​​may be positioned outside the shooting location so that the camera crew and / or their devices 100 can be within the field of view of the camera on the lighthouse device 360. The lighthouse device 360 ​​transmits the captured location data associated with each user and / or device to a camera tracking module 336 in order to process the location of each device. In some embodiments, the lighthouse device 360 ​​includes unique markers, such as QR codes, physical pillars, or objects ("spatial anchors"), that each device (camera 201 and pre-visualization devices 100a-100n) can see with its respective image sensor. In these embodiments, the camera tracking module 336 uses the spatial anchors to triangulate the position of each device, camera 201, and pre-visualization devices 100a-100n in real time based on video signals from multiple devices. In some embodiments, the spatial anchors may be virtual anchors. Without the lighthouse device 360 ​​or a similar system, each device 100a-100n would have its own field of view of where the CGI assets are placed, and changes or placements made on one device may not be similarly reflected in all other devices 100a-100n. In some embodiments, the lighthouse device 360 ​​provides a shared field of view of object position and rotation that underpins the entire system.

[0062] In some embodiments, the camera movement calculated by the camera tracking module 336 is recorded on a storage medium such as the storage medium 161. After the raw video signal of the scene is recorded, the raw video signal and camera movement may be given to the special effects department for final adjustments to the scene, for example, to add high-resolution CGI assets to the raw video signal. Having the coordinates of the camera 201 during filming this sequence is generally helpful to the visual effects team trying to complete the shot, as it allows them to understand the viewpoint and angle of what has been captured in the scene. The positional data may be recorded as an FBX file or a 3D asset file, as described above.

[0063] Instruction 310 also includes a depth occlusion module 338 that can determine whether objects on the set should be placed in front of or behind the CGI asset. Conventional augmented reality systems have difficulty reading the depth of objects relative to the CGI asset. For example, if an actor walks into an area of ​​the set during filming, the actor is usually placed on top of the CGI asset 250 in the composite image by default. As a result, the composite image cannot represent the desired depth of the interaction between the actor and the CGI asset, making directing and filming the actor more difficult. Conventional methods to solve the depth occlusion problem involve the use of body recognition modules. These modules are provided with information about the actor and the CGI graphic. They can calculate where the person is and where the CGI asset is, and position the person and CGI asset accordingly. This method is effective for one device but does not work well for multiple devices. Currently, the compositor 340 generates the augmented video signal based on the data provided by each module. In other words, the compositor receives raw video signals from the camera feed and CGI assets and performs occlusion. Each device may have its own camera feed or system-specific native occlusion capabilities, but the ability to process data in real time places an extremely high load on the processor, as it contains a large amount of data. This may be sufficient for a single device, but data processing becomes difficult for systems without native occlusion capabilities, such as compositors or computers.

[0064] In some embodiments, the depth occlusion module 338 is configured to calculate the depth of pixels in the raw video signal 211 and remove those pixels calculated as being behind the placed CGI asset 250. This makes the CGI asset fully interactive and capable of occluding and / or being occluded by objects and actors in the actual scene. The image sensor 202 of the digital camera 201 may capture an actor walking far across the field of view of the digital camera 201. A CGI asset digitally placed at the forefront of the virtual environment can hide him or her so that the walking actor walks "behind" the CGI asset. If the image sensor 202 of the digital camera 201 captures an actor walking at a distance in front of the placed CGI asset, the pedestrian will occlude at least a portion of the CGI asset for that portion of the walking path. Without the depth occlusion module 338, the CGI asset would simply overlap the video image and always appear at the forefront.

[0065] In some embodiments, the depth occlusion module 338 communicates with the environment sensor 203 and / or a digital 3D model of the environment and receives signals generated therefrom. The depth occlusion module 338 may render the 3D model as a transparent mask that hides virtual objects. Occlusion is generally difficult because typical augmented reality systems do not have the ability to perceive their environment accurately or quickly enough for realistic occlusion. However, in this pre-visualization system 300, depth occlusion is facilitated by having multiple devices that provide environmental data to the system 300, for example, device 100a with the environment sensor 103 complements, in most cases, the environmental data provided by the environment sensor 203 at different viewpoints of the same environment.

[0066] In some embodiments, the depth occlusion module 338 is configured to generate a digital avatar (avatar data) of a person in the field of view of a device, for example, device 100a. That is, the current position of the person (actor) is tracked, and a humanoid avatar is created that tracks the actor's movements at the joint level. The limited avatar data can then be propagated to another device 100n and to the compositor 340 to generate an augmented video signal. The avatar data can be associated with an occlusion mesh, i.e., an instruction for the compositor 340 to create holes in the video feed that are the shape of the avatar. That is, the occlusion mesh can make pixels that have a depth farther from the camera than the object transparent (remove them). In other words, the corresponding avatar created from the avatar data is generated on the compositor 340 along with the occlusion mesh. The position of the occlusion avatar matches the position of the actual actor, but the occlusion mesh cuts out all CGI assets behind it. Therefore, the occlusion avatar becomes simply another CGI object on the compositor, possessing depth and position data. In this way, a portion of the image in the raw video signal 211 is displayed where the avatar data indicates the presence of a hole. In other words, the holes in the avatar data are filled with the raw video feed signal 211 behind the avatar.

[0067] In some implementations, the surrounding environment is scanned, and a spatial map of largely static objects (walls, floors, furniture) is created. Before shooting, the spatial map is saved as an FBx file and imported into Composition 340. Occlusion meshes can be applied to objects in the spatial map, and pixels in the depth range farther from the camera than the object can be removed. An occlusion mesh on an object (tracked down to an actor in the scene, or an avatar on a cube representing a wall) removes all pixels behind it, so when a device renders something with an occlusion mesh, it cuts out all the pixels where the avatar or wall would have been, leaving only empty (black) pixels in the image rendered from that device. When the final image is composited, the compositor efficiently cuts out any CGI assets that would end up behind objects in the camera image.

[0068] As a further example, an actor and their position being tracked on a first device can be pushed to an avatar via an occlusion mesh through a compositor. If the actor is behind a CGI asset, the asset will appear in front of the cinema camera image, hiding the actor, and the occlusion mesh will do nothing. If the actor is in front of the CGI asset, the occlusion mesh will cut out pixels from the CGI asset, making the actor in the cinema camera image appear as if the person is in front of the asset.

[0069] In some embodiments, the depth occlusion module 338 may have submodules to facilitate the processing of depth occlusion features. Each submodule may be associated with specific properties (field of view, focus, etc.) that enable the creation of an image that is the same as (or similar to) what a physical camera captures (raw video feed). A first occlusion submodule may receive a raw video feed from a physical camera and ignore CGI assets. A second submodule includes CGI objects but does not include a camera feed. The images from the first and second submodules are then composited with any effects, including but not limited to occlusion, motion blur, and color correction. The final rendered image may be sent to a third submodule, whose output may be sent to a display. In some further embodiments, the submodules may be embodied as virtual cameras that generate images. In further embodiments, each submodule is embodied as a hardware component of the system.

[0070] In some embodiments, system 300 includes a puppeteer module 342 configured to use real-time motion capture with low latency to animate an inserted CGI asset 250. That is, a human actor 345 (who may not be present on set) is recorded by the motion capture system 344, and the information obtained from recording the human actor is used to animate a digital character model in a two-dimensional or three-dimensional computer animation. Generally, the movements of actor 345 are captured disregarding actor 345's visual appearance, and this animation data of the actor is mapped and / or converted to a three-dimensional CGI asset model or a skeleton associated with CGI asset 250 so that CGI asset 250 performs actor 345's movements. For example, if CGI asset 250 is required to walk from one point to another on set, actor 345, who may or may not be in the field of view of the main recording camera 201, can perform the walking motion captured by the motion capture system 344. The puppeteer module 342 uses walking movements captured from actor 345 to animate the CGI asset 250 in the extended video signal 212. In some embodiments, the motion capture system 344 is configured to manipulate actor 345's face, and the actor's face, representing a non-human character, is modified in real time by the puppeteer module 342 so that the on-set director can properly direct actor 345 and others on set.

[0071] In some embodiments, the puppeteer module 342 may supply puppeteer movements that are distributed to the asset module 334 for animation. That is, the puppeteer movements are modeled on the asset skeleton that coordinates the movements of the 3D CGI asset 250. Similarly, the asset module 334 may modify the LOD of the assets and provide various CGI assets that can be applied to the same skeleton model. For example, if processing power is limited, CGI assets with low LOD may be associated with the skeleton model. If the processing power of the system is sufficient, CGI assets with high LOD may be associated with the skeleton model of the puppeteer module 342, providing higher quality and higher resolution assets in the extended video signal. In other words, the system may swap CGI assets with different resolutions depending on the processing hardware, making video processing more efficient when processing power is low and increasing the resolution of the video signal when processing power is high.

[0072] The video module 330, environment module 332, asset module 334, camera tracking module 336, and depth occlusion module 338 work together to generate a real-time augmented video signal (pre-visualized image) 212, which is provided to the camera's viewfinder 216 and at least one monitor 210 of at least one video village 208. That is, the augmented video signal 212 includes a raw video signal (or a portion thereof, if occlusion is required) with a CGI asset that has a specific object, or a CGI asset 250 in which the CGI asset itself is occluded based on the depth of the object calculated in the real environment. In this way, crew members on set can get an idea of ​​what the final scene will look like when special effects are added to the recorded raw video footage.

[0073] In some embodiments, system 300 includes a compositor 340 configured to communicate with each of modules 330-338 and generate an augmented video signal 212 based on data provided by each module 330-342. The compositor 340 can be variously embodied as part of the hardware that communicates with the modules and / or computer system 206 in instruction 310. In some embodiments, the compositor 340 communicates with a plurality of camera systems 200 and / or digital pre-visualization devices 100a-100n.

[0074] The pre-visualization system 300 enables the use of multiple devices, such as the camera system 200 and digital pre-visualization devices 100a-100n, each contributing to the environmental data (and fidelity) of a 3D model of the environment. In some embodiments, each of the 200 and 100a-100n devices generates its own 3D model from the environmental data captured by that device. In other embodiments, each of the 200 and 100a-100n devices generates its own 3D model (virtual environment) from the environmental data captured by all connected devices. Each of these 3D models is stored in the respective associated device or local server and can be transmitted to other devices connected to the system 300 upon request. In yet another embodiment, each device in the system 300 can access a shared 3D model residing on server 160 or in cloud 170.

[0075] For example, each device accesses the 3D model and the placed CGI assets so that a first device, such as a camera system 200, can view the scene and display a first extended video signal containing CGI assets from a first viewpoint, and a second device, such as a digital pre-visualization device 100a, can view the scene and display a second extended video signal containing CGI assets from a second viewpoint. Each device / system can exchange data and share processing tasks with each other, either directly or through the use of computers, such as a computer system 206, a server 160, a local area network (LAN), a wide area network (WAN), and / or a cloud 170.

[0076] By using multiple systems and devices 200 and 100a-100n, each having its own extended video signal 212, crew members on set performing various tasks can view the scene with their specific goals in mind. For example, but not limited to, camera system 200 could capture a particular scene and transmit the extended video signal to both the viewfinder 216 and the video village 208. Simultaneously, crew members positioned in different locations on set, directing a crowd in the same scene, can use a portable digital pre-visualization device 100 to view different perspectives of the scene with the extended video signal tailored to the viewpoint of that device. In this way, crew members can direct people around a CGI asset 250 as if they were right in front of them, resulting in realistic crowd movements.

[0077] Furthermore, the multi-device system 300 allows any device or subsystem 200, 100 to change the position and orientation of CGI assets in a scene. For example, a camera operator or director using camera system 200 can first roughly position CGI asset 250 in the desired location. Another crew member, such as a cinematographer, can then use another device, such as a digital pre-visualization device 100, to move the CGI asset to a more precise position while viewing the scene and asset from different viewpoints. In this case, the change in the position of the CGI asset in the virtual 3D environment is communicated by system 300 to other devices and subsystems connected to system 300. In some embodiments, the administrator of system 300 can set specific permissions for devices connected to system 300, for example, allowing some users of device 100 to modify CGI assets while others are not.

[0078] In other embodiments, each subsystem / device (200, 100a-100n, respectively) may choose to receive an extended video stream from another device. For example, a user of the remote digital pre-visualization device 100a may choose to view the extended video signal 211 captured by the camera system 200. This allows crew members to use device 100a to access various viewpoints of the scene with CGI assets and, accordingly, direct people in the scene to take pictures.

[0079] In some embodiments, referring to Figure 4, instruction 310 includes a room module 346. The room module 346 is configured to display different states of the same CGI assets 450a and 450b for the same real environment 460. That is, each digital pre-visualization device 110a, 100n scans the same real environment 460 and adds environment data to a 3D model of that environment (improving the fidelity of the 3D model). Digital pre-visualization device 100a displays a first room for environment 460 having a first state of CGI asset 450a, in which case the CGI building 450a is seen between two trees. Digital pre-visualization device 100n displays a second room for environment 460 having a second state of CGI asset 450b, in which case the partially destroyed CGI building 450b is seen between two trees. Room Module 346 may have a specific use for filmmakers to scout film locations, but other uses are possible, and the use of scouting film locations is presented without limitation to illustrate the features of System 300 and Room Module 346. For example, a specific environment 460 (location) may be chosen as the filming location for a movie. The plot of the movie may include a specific building in the environment, with a version 450a where the building is intact at the beginning of the movie and a version 450b where the building is destroyed later in the movie. System 300 may be built on a local server 160 to which digital pre-visualization devices 100a and 100n are connected. Each of the digital pre-visualization devices 100a and 100n simultaneously and continuously scans the local environment 460 and provides data for generating a 3D model (virtual environment) by the environment module 332. The first crew, surveying the location and setup for the earlier scene, including the intact building 450a, may use a digital pre-visualization device 100a to analyze the location and possible positions of the actors in the first virtual room, Room 1.A second crew can simultaneously survey the same location and setup for later scenes, including the destroyed building 450b, and use the pre-visualization device 100b to analyze the actors' locations and possible positions in a second virtual room, Room 2. In other words, the Room Module 346 allows multiple people on set to see different versions of the same space in the real world. This allows crew members to plan in advance so that multiple teams can mark the set and use multiple states of the same asset 450. In other embodiments, the room is configured to use different CGI assets rather than different states of the same CGI asset.

[0080] In some embodiments, data processing and storage are performed in Cloud 170. A local application communicating with Cloud 170 may be configured to track the movements of all devices and subsystems of a pre-visualization system, such as an exemplary pre-visualization system 300. The data collected by the local application may be pushed to a server 160 that distributes the data to all connected devices in real time. Thus, if a particular device 100a has permission to run a CGI asset 250, the movements of the CGI asset 250 in that device will communicate the movements of the CGI asset 250 to all devices connected to device 100a.

[0081] In some embodiments, a pre-visualization system, such as the exemplary pre-visualization system 300, is configured to retain a virtual environment indefinitely. For example, the virtual environment, i.e., the 3D model generated by a combination of devices that collect environment data, may be stored on a server 160 at location, a computer system such as the exemplary computer system 206, the storage of the pre-visualization device 100, or a server in the cloud 170. In this way, even if one device, for example, the digital pre-visualization device 100, crashes, the generated 3D model will not be lost. Furthermore, when the crashed device 100 comes back online and starts scanning the environment, the software hosted on device 100 is configured to recognize its position relative to the reality and the virtual model and to reinsert its CGI assets as if the device had not crashed. Indefinite storage of the 3D model facilitates transitions between multi-day work. For example, if you have a 3D model of a specific outdoor environment on a particular day, and it becomes impossible to continue working in that outdoor environment (for example, if it rains), the placement of the environment data and CGI assets can be preserved until it becomes possible to return to working in the outdoor environment.

[0082] One or more exemplary embodiments incorporating embodiments of the present invention disclosed herein are presented herein. For clarity, this application does not describe or show all features of physical implementations. It is understood that in developing physical embodiments incorporating embodiments of the present invention, numerous implementation-specific decisions will need to be made to achieve the developer's goals, including compliance with system-related, business-related, government-related, and other constraints, and these will change by implementation and from time to time. While the developer's efforts may be time-consuming, such efforts will still be routine work for those skilled in the art and those who benefit from this disclosure.

[0083] The methods illustrated throughout this specification may be implemented in computer program products that can be run on a computer. These computer program products may include non-temporary computer-readable recording media on which control programs are recorded, such as disks and hard drives. Common forms of non-temporary computer-readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tapes and other magnetic storage media, CD-ROMs, DVDs and other optical media, RAM, PROMs, EPROMs, Flash-EPROMs and other memory chips and cartridges, or other tangible media that can be read and used by a computer.

[0084] As used herein, the term “software” is intended to encompass a collection or set of instructions that can be executed by a computer or other digital system to configure the computer or other digital system to perform a task that is the intent of the software. As used herein, the term “software” is intended to encompass such instructions stored in storage media such as RAM, hard disks, and optical disks, and is also intended to encompass so-called “firmware,” which is software stored in ROM and the like. Such software can be configured in a variety of ways and may include software components configured as libraries, internet-based programs stored on remote servers, source code, interpreter code, object code, and directly executable code. Software is intended to be able to call system-level code or other software residing on a server or elsewhere to perform specific functions.

[0085] To assist the Patent Office and readers of this application and the resulting patent in the interpretation of the attached claims, the applicant states that, unless the words “means for” or “steps for” are expressly used in any particular claim, none of the attached claims or claim elements are intended to invoke § 112(f) of the U.S. Patent Act.

Claims

1. A method for generating a video signal during scene pre-visualization, The video module receives a raw video signal containing one or more images representing the environment captured by the image sensor. The environmental module generates a virtual environmental model of the environment based on environmental data that is captured by at least one environmental sensor and characterizes the environment. The asset module inserts a CGI (computer-generated image) asset into a given location within the virtual environment, The camera tracking module determines the position and orientation of the image sensor with respect to the virtual environment based on motion sensor data provided by the motion sensor, The steps of generating an augmented video signal, including the raw video signal containing the CGI asset, based on the given location of the CGI asset in the virtual environment and the determined position and orientation of the image sensor relative to the virtual environment, Methods that include...

2. The aforementioned environmental sensor is the first environmental sensor, The method according to claim 1, wherein the environment module is configured to receive a first set of environment data from a first pre-visualization device having the first environment sensor, which is the environment sensor, and a second set of environment data from a second pre-visualization device having the second environment sensor characterizing the environment, and each of the first pre-visualization device and the second pre-visualization device captures the environment from different viewpoints.

3. The method according to claim 1, further comprising the step of occluding one or more features of the raw video signal by the CGI asset based on a determined depth of the one or more features using a depth occlusion module.

4. The method according to claim 1, wherein a puppeteer module receives motion capture data characterizing human movements captured by a motion capture system, and the CGI assets in the augmented video signal are animated based on the motion capture data.

5. The method according to claim 1, further comprising the step of displaying the extended video signal on at least one display.

6. A pre-visualization camera system used to generate a video signal during scene pre-visualization, A camera configured to output a raw video signal containing one or more images of the environment corresponding to the aforementioned scene, An environmental sensor configured to output environmental data that characterizes the aforementioned environment, An environment module configured to generate a 3D model of the scene based on the aforementioned environment data, and to insert a computer-generated image (CGI) at a given position in the 3D model, A motion sensor configured to generate camera tracking data that characterizes the movement of the aforementioned camera, A compositor configured to generate an extended video signal including the raw video signal which includes the positions of the CGI asset and the pre-visualization camera system based on the camera tracking data and the given positions of the CGI asset in the 3D model. Equipped with, A pre-visualization camera system in which an extended video signal is received by the camera viewfinder of the camera.

7. The aforementioned environmental sensor is the first environmental sensor of the first pre-visualization device, The pre-visualization camera system according to claim 6, further comprising a camera system interface configured to receive supplemental environmental measurement data characterizing the scene from a second environmental sensor of a second pre-visualization device, wherein the environmental module is configured to generate the 3D model of the scene based on the supplemental environmental measurement data.

8. The pre-visualization camera system according to claim 7, wherein the first environmental sensor is of a first type, the second environmental sensor is of a second type different from the first type, and each of the first and second environmental sensors is one of an infrared system, a light detection and ranging (LIDAR) system, a thermal imaging system, an ultrasonic system, a stereoscopic system, and an optical system.

9. A pre-visualization system for use in generating video signals during scene pre-visualization, It is a camera system, Processor and A camera image sensor configured to generate a raw video signal containing one or more images representing the environment, A camera environment sensor configured to generate a first set of environmental measurement data characterizing the aforementioned environment, Camera display and A camera system equipped with, A pre-visualization device comprising a device environment sensor configured to generate a second set of environmental measurement data characterizing the aforementioned environment, and Equipped with, The processor generates a 3D model of the environment based on a first set of environmental measurement data and a second set of environmental measurement data, using a CGI (computer-generated image) asset placed at a given location. The aforementioned camera display displays the extended video signal, The extended video signal includes the raw video signal and the CGI asset, The extended video signal is further generated based on a given position of the CGI asset in the 3D model of the environment. The aforementioned camera image sensor is a pre-visualization system that tracks data characterizing the movement of the camera image sensor.

10. The pre-visualization system according to claim 9, further comprising a motion capture system configured to generate animation data that characterizes human movement, wherein the processor animates the CGI asset based on the animation data.

11. The pre-visualization system according to claim 9, further comprising a remote monitor configured to display the extended video signal.

12. The raw video signal is the first raw video signal, the extended video signal is the first extended video signal, and the 3D model is the first 3D model. The aforementioned pre-visualization device, Device processor and A device image sensor configured to generate a second raw video signal including one or more images from a different viewpoint than the camera image sensor in the aforementioned environment, Device display and The CGI asset comprises a first CGI asset associated with the extended video signal and a second CGI asset associated with the extended video signal. The device processor generates the second 3D model of the environment based on the first set of environment measurement data and the second set of environment measurement data, using CGI assets of position and orientation in the second 3D model of the environment. The pre-visualization system according to claim 9, wherein the device display displays a second enhanced video signal including the second raw video signal and the CGI asset based on the position and orientation of the CGI asset in the second 3D model of the environment.

13. The pre-visualization system according to claim 9, wherein the camera environmental sensor is of a first type, the device environmental sensor is of a second type different from the first type, and each of the first type environmental sensor and the second type environmental sensor is one of an infrared system, a light detection and ranging (LIDAR) system, a thermal imaging system, an ultrasonic system, a stereoscopic system, an RGB camera, and an optical system.

14. The pre-visualization system according to claim 9, further comprising a server configured to provide the first set and the second set of environmental measurement data to the pre-visualization device and the camera system, respectively.

15. The occlusion step described above is: The depth occlusion module includes the step of determining the depth of pixels in the raw video signal based on at least one of the environment data and the virtual environment model, The depth occlusion module identifies a subset of pixels in the raw video signal that corresponds to a portion of one or more features behind the CGI asset, based on the determined depth of the subset of pixels. The steps include removing a subset of pixels in the raw video signal that correspond to one or more features behind the CGI asset, The method according to claim 3, further comprising:

16. The method according to claim 1, wherein the depth occlusion module occludes the CGI asset at least partially using one or more features of the raw video signal based on the determined depth of one or more features.

17. The occlusion step described above is: The depth occlusion module includes the step of determining the depth of pixels in the raw video signal based on at least one of the environment data and the virtual environment model, The depth occlusion module identifies a subset of pixels in the raw video signal that corresponds to a portion of one or more features in front of the CGI asset, based on the determined depth of the subset of pixels. The depth occlusion module identifies the portion of the CGI asset that is in front of the CGI asset, and the step of removing the portion of the CGI asset that is behind the portion of the CGI asset that is in front of the portion of the CGI asset that is in front of the portion of the CGI asset; The method according to claim 16, further comprising the following:

18. The aforementioned environment includes a person, The aforementioned method, The depth occlusion module generates an avatar representing the person, The method according to claim 1, further comprising the step of the depth occlusion module outputting avatar data that characterizes the avatar.

19. The extended video signal is a first extended video signal, and the raw video signal is a first raw video signal of the environment from a first viewpoint. The method according to claim 18, further comprising the step of generating a second raw video signal of the environment from a second viewpoint and a second enhanced video signal including the CGI assets and an avatar, wherein the avatar is generated based on avatar data.

20. The method according to claim 19, wherein the step of generating the second extended video signal includes the steps of removing pixels from the second raw video signal of the environment of the avatar, based on the avatar data, in regions having the shape of the avatar, and inserting the avatar from the second raw video signal into the positions of the removed pixels.

21. The steps include capturing the raw video signal provided by the image sensor, The steps include: saving the raw video signal as raw video signal data on a storage medium in response to the capture of the raw video signal; The method according to claim 1, further comprising the step of providing the raw video signal to the video module in response to the capture of the raw video signal.

22. The method according to claim 21, wherein a capture device is used to store the raw video signal in the storage medium and the raw video signal is provided to the video module.

23. The method according to claim 1, wherein the extended video signal is provided to a camera viewfinder at a first position during the shooting of the environment, and to at least one display at a second position during the shooting of the environment.

Images