Rendering back plate

Abstract

To provide processing and rendering of video data.SOLUTION: Video rendering includes the steps of: tracking spatial coordinates of at least one camera during a video sequence forming a shot having multiple frames, each of the at least one camera having a lens; creating a lens profile storing lens data corresponding to the lens of the at least one camera during the shot; encoding the lens data; sending the lens data to a render engine; retracing the movement of the at least one camera during the shot; recreating the lens and one or more characteristics of the lens during the shot; and replicating the shot in a virtual environment using the retraced camera movement and recreated lens characteristics.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] FIELD OF THE DISCLOSURE This disclosure relates to rendering video, and in particular to rendering backplates. [Background technology]

[0002] In traditional systems for video production, rendering a background "plate" (sometimes called a "backplate") involves filming a background scene without any subjects. However, the use of backplates has the disadvantage that the process requires a large amount of manual labor, which can be expensive and tedious. Summary of the Invention [Problem to be solved by the invention]

[0003] The present disclosure provides for processing and rendering of video data. [Means for solving the problem]

[0004] In one implementation, a method for video rendering is disclosed that includes tracking spatial coordinates of at least one camera in a video sequence forming a shot having a plurality of frames, each of the at least one camera having a lens, creating a lens profile that stores lens data corresponding to the lens of the at least one camera in the shot, encoding the lens data, transmitting the lens data to a render engine, retrace movement of the at least one camera in the shot, recreating the lens and one or more characteristics of the lens in the shot, and replicating the shot in a virtual environment using the retrace camera movement and re-created lens characteristics.

[0005] In one implementation, the spatial coordinates of each of the at least one camera include a position of each camera. In one implementation, the spatial coordinates of each of the at least one camera include , and an orientation of each camera. In one implementation, the orientation of each camera includes pitch, yaw, and roll axes used to track the local rotation of each camera. In one implementation, the lens profile includes a nodal point, which is the point where all light beams intersect within the lens and is projected onto an image plane. In one implementation, the lens profile includes at least one of an image plane distance to the nodal point, a focal length of each camera, a lens distortion profile, an image center offset, a lens aperture, and a focus distance. In one implementation, the method further includes synchronizing the lens data with each frame of the shot. In one implementation, replicating the shot includes virtually replicating the shot by mimicking the lens and the lens characteristics for each frame.

[0006] In another implementation, a system for video rendering is disclosed that includes at least one camera for capturing images of a background scene and outputting the captured images as camera data, each of the at least one camera having a lens, at least one sensor for tracking spatial coordinates of the at least one camera in a video sequence forming a shot having a plurality of frames and outputting the tracked spatial coordinates as sensor data, and a processor coupled to the at least one camera and the at least one sensor for generating a lens profile that stores lens data corresponding to the lens of the at least one camera in the shot, wherein the processor processes the camera data, the sensor data, and the lens data to replicate the shot.

[0007] In one implementation, the lens data is synchronized to each frame of the shot. In one implementation, the lens data is synchronized to a time code. In one implementation, the system further includes a renderer for rendering the replicated shot. In one implementation, the processor encodes the lens data, retraces movement of the at least one camera during the shot, recreates the lens and one or more characteristics of the lens, and replicates the shot in a virtual environment. In one implementation, the system further includes a render engine for retracing movement of the at least one camera and mimicking the lens and its characteristics frame by frame to virtually replicate the shot.

[0008] In another implementation, a non-transitory computer-readable storage medium storing a computer program for rendering video is disclosed, the computer program including executable instructions that cause a computer to: track spatial coordinates of at least one camera in a video sequence forming a shot having a plurality of frames, each of the at least one camera having a lens; generate a lens profile that stores lens data corresponding to the lens of the at least one camera in the shot; encode the lens data; transmit the lens data to a render engine; retrace the movement of the at least one camera in the shot; recreate the lens and one or more characteristics of the lens in the shot; and replicate the shot in a virtual environment using the retrace camera movement and recreated lens characteristics.

[0009] In one implementation, the spatial coordinates of each of the at least one camera include the position of each camera. In one implementation, the lens profile includes a nodal point (located on the optical axis) that is the point where all light beams intersect within the lens and is projected onto an image plane. In one implementation, the lens profile includes at least one of the image plane distance to the nodal point, a focal length of each camera, a lens distortion profile, an image center offset, a lens aperture, and a focus distance. In one implementation, the computer program further includes executable instructions that cause the computer to synchronize the lens data with each frame of the shot. In one implementation, the executable instructions that cause the computer to replicate the shot include executable instructions that cause the computer to virtually replicate the shot by mimicking the lens and the lens characteristics for each frame.

[0010] Other features and advantages will become apparent from the present specification, which illustrates by way of example embodiments of the disclosure.

[0011] Details of the present disclosure, both as to its structure and operation, can be gleaned in part by studying the accompanying drawings, in which like parts are designated with like reference numerals. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a flow diagram of a method for video rendering according to one implementation of the present disclosure. [Figure 2A] FIG. 1 is a block diagram of a video rendering system according to one implementation of the present disclosure. [Figure 2B] FIG. 2 is a block diagram of a video rendering system according to another implementation of the present disclosure. [Figure 3A] 1 is a diagram of a computer system and a user according to an implementation of the present disclosure. [Figure 3B]FIG. 1 is a functional block diagram illustrating a computer system hosting a video rendering application in accordance with an implementation of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0013] As noted above, the use of backplates in rendering backgrounds in video productions has the disadvantage that the process requires a large amount of manual labor, and therefore can be expensive and tedious.

[0014] Certain implementations of the present disclosure provide methods and systems for implementing techniques for processing and rendering video data. In one implementation, a video system renders accurate, high-fidelity backplates for visual effects (VFX), such as those used in movies, TV, and commercials. This method can draw from assets with resolutions above 8K or any other assets, allowing users to freely define the resolution for distribution.

[0015] After reading the following description, it will become apparent how to implement the present disclosure in various implementations and applications. While various implementations of the present disclosure are described herein, it should be understood that these implementations are presented by way of example only, and not by way of limitation. Therefore, detailed descriptions of various implementations should not be construed as limiting the scope or breadth of the present disclosure.

[0016] The features provided in the following implementations may include, but are not limited to, one or more of the following items. For example, one implementation records lens distortion, imager dimensions, lens nodal point, focal length, focus distance, and aperture along with camera telemetry data during video production. Also, timecode and / or frame number are recorded for each take (or throughout the recording). Another implementation uses a tool or script to send the recorded data to a render engine that contains assets (i.e., background assets for plates). Individual frames are then rendered at a desired image quality. Individual frames can also be combined into a video file.

[0017] 1 is a flow diagram of a method 100 for video rendering according to one implementation of the present disclosure. In the implementation shown in FIG. 1, step 110 tracks the spatial coordinates of a camera in a video sequence forming a shot having multiple frames. In one implementation, the spatial coordinates of the camera include the position of the camera. In another implementation, the spatial coordinates of the camera include the orientation of the camera.

[0018] One example of behavior is tracking the position of the main capture camera within the environment in real time. One implementation uses a six-degree-of-freedom tracking system; for example, X, Y, and Z coordinates are used to track the translation of the virtual camera from its origin, and pitch, yaw, and roll axes are used to track the local rotation of the camera. In one implementation, the origin of the virtual camera (i.e., 0,0,0) is the nodal point of the physical lens.

[0019] In one implementation, the camera includes a lens. In step 120, a lens profile is generated that stores lens data corresponding to the camera's lens in the shot. In one implementation, the lens profile includes the following parameters: (1) active sensor dimensions / film plane / imager measurements, (2) image plane distance to nodal point, (3) camera focal length that accounts for zoom and lens breathing, (4) lens distortion profile, (5) image center offset, (6) lens aperture, (7) focus distance, and (8) lens vignetting or shading.

[0020] In one implementation, the lens data is synchronized with each frame of the shot. In step 130, the lens data is encoded and sent to the render engine in step 140. In one implementation, a tool, script, or plug-in (e.g., Unreal Engine 4 or Unity) is used to feed the lens data back to the render engine. Next, in step 150, the camera movement during the shot is retraced. In step 160, the lens and one or more lens properties are recreated during the shot, and in step 170, the shot is replicated in the virtual environment using the retraced camera movement and recreated lens properties.

[0021] In one implementation, the render engine retraces the movement of the main capture camera, mimicking the lens and its characteristics for each frame to virtually replicate the shot. This allows the operator to freely define the resolution for the plate while taking full advantage of all image render quality settings. Depending on the quality of the asset, this also means that there is little or no noise in the digital image. This depends on the resolution quality of the asset and the method of ingestion. With high-quality assets, it is possible to extract backplates with resolutions above 8K.

[0022] FIG. 2A is a block diagram of a video rendering system 200 according to one implementation of the present disclosure. In the implementation shown in FIG. 2A, the video rendering system 200 is used in a video production or studio environment. The system 200 includes one or more cameras 220, 222 for image capture of a background scene 210, one or more sensors / trackers 230 for tracking the spatial coordinates (e.g., position and orientation) of the cameras, one or more processors 240 for processing the camera and sensor data to provide a render engine 242 for rendering frames of video, and a renderer 244. In one implementation, the system 200 renders frames from light detection and ranging (lidar) scanned assets with resolutions exceeding 8K. However, in general, assets of any quality can be used, although asset quality may limit the final achievable resolution.

[0023] 2A, the tracker 230 tracks the spatial coordinates of one or more cameras 220, 222 in a video sequence forming a shot having multiple frames. In some implementations, the tracker 230 can track the spatial coordinates of a single camera 220, in which case tracking of the camera 222 is optional (dotted line). In one implementation, the spatial coordinates of the camera include the camera's position. In another implementation, the spatial coordinates of the camera include the camera's orientation.

[0024] One example of operation is tracking the position of the cameras 220, 222 within the environment in real time. One implementation uses a six-degree-of-freedom tracking system. For example, X, Y, and Z coordinates are used to track the translation of the virtual camera from its origin, and pitch, yaw, and roll axes are used to track the local rotation of the camera. In one implementation, the origin of the virtual camera (i.e., 0,0,0) is the nodal point of the physical lens.

[0025] In the implementation shown in FIG. 2A , processor 240 is coupled to cameras 220, 222. In one implementation, processor 240 generates a lens profile that stores lens data corresponding to the lens of each camera 220 or 222 in a shot. In one implementation, processor 240 synchronizes the lens data to each frame of the shot. In another implementation, processor 240 synchronizes the lens data to a timecode. In one implementation, processor 240 encodes the lens data and sends the encoded lens data to render engine 242. Processor 240 then retraces the camera movement during the shot. Processor 240 also recreates the lens and one or more characteristics of the lens and replicates the shot in the virtual environment using the retrace camera movement and recreated lens characteristics. Processor 240 then sends the replicated shot to renderer 244 to render the shot.

[0026] In one implementation, processor 240 generates a lens profile and encodes lens data in real time during production. In one implementation, the lens profile includes the following parameters: (1) active sensor dimensions / film plane / imager measurements, (2) image plane distance to nodal point, (3) camera focal length accounting for zoom and lens breathing, (4) lens distortion profile, (5) image center offset, (6) lens aperture, and (7) focus distance. In another implementation, the lens profile includes light extinction information (i.e., vignetting).

[0027] In one implementation, the render engine 242 of the processor 240 retraces the movement of the cameras 220, 222, mimicking the lenses and their characteristics for each frame to virtually replicate the shot. This allows the operator to freely define the resolution for the plate while taking full advantage of all image render quality settings. Depending on the quality of the asset, this also means that there is little or no noise in the digital image. This depends on the resolution quality of the asset and the capture method. With high-quality assets, it is possible to extract backplates with resolutions above 8K.

[0028] FIG. 2B is a block diagram of a video rendering system 250 according to another implementation of the present disclosure. In the implementation shown in FIG. 2B, the video rendering system 250 is used in a video production or studio environment. The system 250 includes one or more cameras 220, 222 for image capture of a background scene 210, a tracker 230 for tracking the spatial coordinates (e.g., position and orientation) of the cameras 220, 222, a lens profiler 260, a lens encoder 262, a processor 270, and a renderer 280. The lens profiler 260, lens encoder 262, and processor 270 combine to provide a render engine 272 for processing camera and sensor data and rendering frames of video via the renderer 280. In one implementation, the system 250 renders frames from light detection and ranging (lidar) scanned assets with resolutions exceeding 8K. However, in general, assets of any quality can be used, although asset quality may limit the final achievable resolution.

[0029] In the implementation shown in FIG. 2B , lens profiler 260 is coupled to cameras 220, 222. In one implementation, lens profiler 260 generates lens profiles that store lens data corresponding to the lenses of each camera 220 or 222 in a shot. In one implementation, lens data is synchronized to each frame of the shot. In another implementation, lens data is synchronized to a time code. Lens profiler 260 then sends the lens data to lens encoder 262. In one implementation, lens encoder 262 encodes the lens data and sends the encoded lens data to render engine 272 of processor 270, which then retraces the camera movement during the shot. Processor 270 also recreates the lens and one or more characteristics of the lens and replicates the shot within the virtual environment using the retrace camera movement from tracker 230 and the recreated lens characteristics from lens encoder 262. The processor 270 then sends the duplicated shot to the renderer 280 to render the shot.

[0030] In one implementation, lens profiler 260 and lens encoder 262 generate lens profiles and encode lens data in real time during production. In one implementation, the lens profile includes the following parameters: (1) active sensor dimensions / film plane / imager measurements, (2) image plane distance to nodal point, (3) camera focal length accounting for zoom and lens breathing, (4) lens distortion profile, (5) image center offset, (6) lens aperture, (7) focus distance, and (8) vignetting or shading.

[0031] In one implementation, the render engine 272 of the processor 270 retraces the movement of the cameras 220, 222, mimicking the lenses and their characteristics for each frame to virtually replicate the shot. This allows the operator to freely define the resolution for the plate while taking full advantage of all image render quality settings. Depending on the quality of the asset, this also means that there is no or very little noise in the digital image. This depends on the resolution quality of the asset and the capture method. With high-quality assets, it is possible to extract backplates with resolutions above 8K.

[0032] Modifications to the system are possible; for example, assets of any type and resolution are compatible with this workflow. The quality of the rendered plates depends on the quality of the assets, e.g., 3D assets with resolutions above 8K. The method can also be applied to lower resolution assets and 2D assets.

[0033] 3A is a diagram of a computer system 300 and a user 302, in accordance with an implementation of the present disclosure. User 302 uses computer system 300 to implement an application 390 for video rendering, such as that shown and described with respect to method 100 of FIG. 1 and systems 200, 250 of FIGS. 2A and 2B.

[0034] The computer system 300 stores and executes the video rendering application 390 of Figure 3B. Additionally, the computer system 300 can communicate with a software program 304. The software program 304 can include the software code for the video rendering application 390. The software program 304 can be loaded onto an external medium, such as a CD, DVD, or storage drive, as described further below.

[0035] Additionally, computer system 300 can be connected to network 380. Network 380 can be connected in a variety of different architectures, such as a client-server architecture, a peer-to-peer network architecture, or other types of architectures. For example, network 380 can communicate with server 385, which coordinates engines and data used in video rendering application 390. Network 380 can also be different types of networks. For example, network 380 can be the Internet, a local area network or any variation of a local area network, a wide area network, a metropolitan area network, an intranet or extranet, or a wireless network.

[0036] 3B is a functional block diagram illustrating a computer system 300 hosting a video rendering application 390 according to an implementation of the present disclosure. The controller 310 is a programmable processor that controls the operation of the computer system 300 and its components. The controller 310 loads instructions (e.g., in the form of a computer program) from the memory 320 or an internal controller memory (not shown) and executes these instructions to control the system, for example, to provide data processing and capture camera movement data. In doing so, the controller 310 provides a software system for the video rendering application 390, for example, to render a backplate. Alternatively, this service can be implemented as a separate hardware component in the controller 310 or the computer system 300.

[0037] Memory 320 temporarily stores data for use by other components of computer system 300. In one implementation, memory 320 is implemented as RAM. In another implementation, memory 320 also includes long-term or permanent memory, such as flash memory and / or ROM.

[0038] Storage 330 stores data temporarily or long-term for use by other components of computer system 300. For example, storage 330 stores data used by video rendering application 390. In one implementation, storage 330 is a hard disk drive.

[0039] Media device 340 accepts removable media and reads and / or writes data to the inserted media. In one implementation, for example, media device 340 is an optical disc drive.

[0040] User interface 350 includes components for receiving user input from a user of computer system 300 and presenting information to user 302. In one implementation, user interface 350 includes a keyboard, a mouse, audio speakers, and a display. Controller 310 uses the input from user 302 to coordinate the operation of computer system 300.

[0041] I / O interface 360 ​​includes one or more I / O ports for connecting to corresponding I / O devices, such as external storage or supplemental devices (e.g., printers or PDAs). In one implementation, the ports of I / O interface 360 ​​include ports such as USB ports, PCMCIA ports, serial ports, and / or parallel ports. In another implementation, I / O interface 360 ​​includes a wireless interface for communicating wirelessly with external devices.

[0042] Network interface 370 includes wired and / or wireless network connections, such as an RJ-45 supporting an Ethernet connection or a "Wi-Fi" interface (including but not limited to 802.11).

[0043] Computer system 300 includes additional hardware and software typical of a computer system (e.g., power, cooling, operating system), but these components are not specifically shown in Figure 3B for simplicity. Other implementations may use different configurations of computer systems (e.g., different bus or storage configurations or multiprocessor configurations).

[0044] The description herein of the disclosed implementations is provided to enable any person skilled in the art to make or use the disclosure. Many modifications of these implementations will be readily apparent to those skilled in the art, and the principles defined herein may be applied to other implementations without departing from the spirit or scope of the disclosure.

[0045] Further variations and implementations are possible. For example, in addition to video production for film or television, implementations of the present systems and methods can be adapted and adapted for other applications, such as commercials, web-based or internet content, virtual production (e.g., virtual reality environments), and visual effects. Thus, the present disclosure is not intended to be limited to the implementations shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0046] Not all features of each of the above embodiments are necessarily required in a particular implementation of the present disclosure. Furthermore, it should be understood that the description and drawings presented herein are representative of the subject matter broadly intended by the present disclosure. Furthermore, it should be understood that the scope of the present disclosure fully encompasses other implementations that may become apparent to those skilled in the art, and therefore, the scope of the present disclosure is not limited by anything other than the appended claims. [Explanation of symbols]

[0047] 100...Methods for Video Rendering 110... Tracking the spatial coordinates of a camera in a video sequence forming a shot with multiple frames 120...Create a lens profile that stores lens data corresponding to the camera lens in the shot 130...Encodes lens data 140...Sends lens data to the render engine 150...Retrace the camera movement during the shot 160...Recreate the lens in the shot and one or more lens characteristics 170...Replicate shots in a virtual environment using retraced camera movements and recreated lens properties 200...Video Rendering System 210...Background scene 220,222...camera 230...Sensor / Tracker 240...processor 242...Render engine 244...Renderer 250...Video Rendering System 260...Lens Profiler 262...Lens encoder 270...processor 272...Render engine 280...Renderer 300...Computer Systems 302...User 304...Software Programs 310...Controller 320...Memory 330…Storage 340...Media Devices 350...User Interface 360...I / O interface 370...Network Interface 380…Network 385...server 390...Video rendering application

Claims

1. 1. A method for use in virtual production, comprising: detecting six degrees of freedom of a first camera forming a first shot having a plurality of frames and a second camera forming a second shot having a plurality of frames, and generating first six-degrees-of-freedom data indicative of the six degrees of freedom of the first camera and second six-degrees-of-freedom data indicative of the six degrees of freedom of the second camera in real time; outputting a first instruction to trace the six degrees of freedom and lens characteristics of the first camera onto a first virtual camera based on first lens data indicating lens characteristics of the first camera in the first shot and the first six degrees of freedom data, in order to virtually replicate the first shot; and outputting second instructions to trace the six degrees of freedom and lens characteristics of the second camera to a second virtual camera based on second lens data indicating lens characteristics of the second camera in the second shot and the second six degrees of freedom data, in order to virtually replicate the second shot; Each of the first lens data and the second lens data includes data indicating at least a lens distortion, a focus distance, and an aperture. method.

2. At least one of the first lens data and the second lens data includes data indicating a nodal point, which is a point where all light beams intersect within the lens and is projected onto an image plane. The method of claim 1.

3. At least one of the first lens data and the second lens data includes data indicative of lens breathing. The method of claim 1.

4. At least one of the first lens data and the second lens data includes data indicative of lens shading. The method of claim 1.

5. outputting at least one of the first six degrees of freedom data, the second six degrees of freedom data, the first lens data, or the second lens data via wireless communication. The method of claim 1.

6. A first tracker detects six degrees of freedom of the first camera, and a second tracker detects six degrees of freedom of the second camera. The method of claim 1.

7. synchronizing the first lens data with the first shot; and synchronizing the second lens data with the second shot. The method of claim 1.

8. The method further includes virtually duplicating the first shot and the second shot using the same rendering engine. The method of claim 1.

9. The rendering engine is capable of rendering with assets having a resolution of 8K or higher. The method of claim 8.

10. the step of outputting the first instruction includes the step of generating the first instruction by a plug-in; the step of outputting the second instruction includes the step of generating the second instruction by a plug-in; The method of claim 1.

11. 1. A system for virtual production, comprising: a first camera having a lens and forming a first shot having a plurality of frames; a second camera having a lens and forming a second shot having a plurality of frames; a sensor or tracker that detects six degrees of freedom of the first camera and the second camera, respectively, and generates first six-degrees-of-freedom data indicating the six degrees of freedom of the first camera and second six-degrees-of-freedom data indicating the six degrees of freedom of the second camera in real time, respectively; a render engine plug-in that outputs a first instruction to trace six degrees of freedom and lens characteristics of the first camera onto a first virtual camera based on first lens data indicating lens characteristics of the first camera in the first shot and the first six degrees of freedom data, in order to virtually replicate the first shot; and outputs a second instruction to trace six degrees of freedom and lens characteristics of the second camera onto a second virtual camera based on second lens data indicating lens characteristics of the second camera in the second shot and the second six degrees of freedom data, in order to virtually replicate the second shot; Each of the first lens data and the second lens data includes data indicating at least a lens distortion, a focus distance, and an aperture. system.

12. At least one of the first lens data and the second lens data includes data indicating a nodal point, which is a point where all light beams intersect within the lens and is projected onto an image plane. The system of claim 11.

13. At least one of the first lens data and the second lens data includes data indicative of lens breathing. The system of claim 11.

14. At least one of the first lens data and the second lens data includes data indicative of lens shading. The system of claim 11.

15. The plug-in outputs at least one of the first six degrees of freedom data, the second six degrees of freedom data, the first lens data, or the second lens data via wireless communication. The system of claim 11.

16. The sensors or trackers are a first tracker that detects six degrees of freedom of the first camera and a second tracker that detects six degrees of freedom of the second camera. The system of claim 11.

17. The plug-in synchronizes the first lens data with the first shot and synchronizes the second lens data with the second shot. The system of claim 11.

18. The plug-in virtually replicates the first shot and the second shot using the same rendering engine. The system of claim 11.

19. The rendering engine is capable of rendering with assets having a resolution of 8K or higher.

20. The system of claim 18.

20. The plug-in receives at least one of the first lens data or the second lens data in real time. The system of claim 11.

Images