Backplate Rendering

Abstract

To provide processing and rendering of video data.SOLUTION: Video rendering includes the steps of: tracking spatial coordinates of at least one camera during a video sequence forming a shot having multiple frames, each of the at least one camera having a lens; creating a lens profile storing lens data corresponding to the lens of the at least one camera during the shot; encoding the lens data; sending the lens data to a render engine; retracing the movement of the at least one camera during the shot; recreating the lens and one or more characteristics of the lens during the shot; and replicating the shot in a virtual environment using the retraced camera movement and recreated lens characteristics.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] FIELD OF THE DISCLOSURE

[0001] This disclosure relates to rendering video, and more particularly to rendering backplates. [Background technology]

[0002]

[0002] In conventional systems for video production, rendering a background "plate" (sometimes called a "backplate") involves filming a background scene without any subjects. However, the use of backplates has the disadvantage that the process requires a large amount of manual labor, which can be expensive and tedious. Summary of the Invention [Problem to be solved by the invention]

[0003]

[0003] The present disclosure provides for processing and rendering video data. [Means for solving the problem]

[0004] In one implementation, a method for video rendering is disclosed that includes tracking spatial coordinates of at least one camera in a video sequence forming a shot having a plurality of frames, each of the at least one camera having a lens, creating a lens profile that stores lens data corresponding to the lens of the at least one camera in the shot, encoding the lens data, transmitting the lens data to a render engine, retrace movement of the at least one camera in the shot, recreating the lens and one or more characteristics of the lens in the shot, and replicating the shot in a virtual environment using the retrace camera movement and re-created lens characteristics.

[0005] In one implementation, the spatial coordinates of each of the at least one camera include a position of each camera. In one implementation, the spatial coordinates of each of the at least one camera include an orientation of each camera. In one implementation, the orientation of each camera includes pitch, yaw, and roll axes used to track local rotation of each camera. In one implementation, the lens profile includes a nodal point, which is a point where all light beams intersect within the lens and is projected onto an image plane. In one implementation, the lens profile includes at least one of an image plane distance to the nodal point, a focal length of each camera, a lens distortion profile, an image center offset, a lens aperture, and a focus distance. In one implementation, the method further includes synchronizing the lens data with each frame of the shot. In one implementation, replicating the shot includes virtually replicating the shot by mimicking the lens and the lens characteristics for each frame.

[0006] In another implementation, a system for video rendering is disclosed that includes at least one camera for capturing images of a background scene and outputting the captured images as camera data, each of the at least one camera having a lens, at least one sensor for tracking spatial coordinates of the at least one camera in a video sequence forming a shot having a plurality of frames and outputting the tracked spatial coordinates as sensor data, and a processor coupled to the at least one camera and the at least one sensor for generating a lens profile that stores lens data corresponding to the lens of the at least one camera in the shot, wherein the processor processes the camera data, the sensor data, and the lens data to replicate the shot.

[0007] In one implementation, the lens data is synchronized to each frame of the shot. In one implementation, the lens data is synchronized to a time code. In one implementation, the system further includes a renderer for rendering the replicated shot. In one implementation, the processor encodes the lens data, retraces movement of the at least one camera during the shot, recreates the lens and one or more characteristics of the lens, and replicates the shot in a virtual environment. In one implementation, the system further includes a render engine for virtually replicating the shot by retracing movement of the at least one camera and mimicking the lens and its characteristics frame by frame.

[0008] In another implementation, a non-transitory computer-readable storage medium storing a computer program for rendering video is disclosed, the computer program including executable instructions that cause a computer to: track spatial coordinates of at least one camera in a video sequence forming a shot having a plurality of frames, each of the at least one camera having a lens; generate a lens profile that stores lens data corresponding to the lens of the at least one camera in the shot; encode the lens data; transmit the lens data to a render engine; retrace the movement of the at least one camera in the shot; recreate the lens and one or more characteristics of the lens in the shot; and replicate the shot in a virtual environment using the retrace camera movement and recreated lens characteristics.

[0009]

[0009] In one implementation, the spatial coordinates of each of the at least one camera include a position of each camera. In one implementation, the lens profile includes a nodal point (located on the optical axis) that is the point where all light beams intersect within the lens and is projected onto an image plane. In one implementation, the lens profile includes at least one of an image plane distance to the nodal point, a focal length of each camera, a lens distortion profile, an image center offset, a lens aperture, and a focus distance. In one implementation, the computer program further includes executable instructions that cause the computer to synchronize the lens data with each frame of the shot. In one implementation, the executable instructions that cause the computer to replicate the shot include executable instructions that cause the computer to virtually replicate the shot by mimicking the lens and the lens characteristics for each frame.

[0010]

[0010] Other features and advantages will become apparent from the present specification, which illustrates, by way of example, aspects of the disclosure.

[0011]

[0011] Details of the present disclosure, both as to its structure and operation, can be gleaned in part by studying the accompanying drawings, in which like parts are designated by like reference numerals and in which: [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a flow diagram of a method for video rendering according to one implementation of the present disclosure. [Figure 2A] FIG. 1 is a block diagram of a video rendering system according to one implementation of the present disclosure. [Figure 2B] FIG. 2 is a block diagram of a video rendering system according to another implementation of the present disclosure. [Figure 3A] 1 is a diagram of a computer system and a user according to an implementation of the present disclosure. [Figure 3B]FIG. 1 is a functional block diagram illustrating a computer system hosting a video rendering application in accordance with an implementation of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0013]

[0017] As noted above, the use of backplates in rendering backgrounds in video productions has the disadvantage that the process requires a large amount of manual labor, and therefore can be expensive and tedious.

[0014]

[0018] Certain implementations of the present disclosure provide methods and systems for implementing techniques for processing and rendering video data. In one implementation, a video system renders accurate, high-fidelity backplates for visual effects (VFX), such as those used in movies, TV, and commercials. This method can draw from assets with resolutions above 8K or any other assets, allowing users to freely define the resolution for distribution.

[0015]

[0019] After reading the following description, it will become apparent how to implement the present disclosure in various implementations and applications. While various implementations of the present disclosure are described herein, it should be understood that these implementations are presented by way of example only, and not by way of limitation. Therefore, detailed descriptions of various implementations should not be construed as limiting the scope or breadth of the present disclosure.

[0016]

[0020] The features provided in the following implementations may include, but are not limited to, one or more of the following items. For example, one implementation records lens distortion, imager dimensions, lens nodal point, focal length, focus distance, and aperture along with camera telemetry data during video production. Also, timecode and / or frame number are recorded for each take (or throughout the recording). Another implementation uses a tool or script to send the recorded data to a render engine that contains assets (i.e., background assets for plates). Individual frames are then rendered at a desired image quality. Individual frames can also be combined into a video file.

[0017]

[0021] 1 is a flow diagram of a method 100 for video rendering according to one implementation of the present disclosure. In the implementation shown in FIG. 1, step 110 tracks the spatial coordinates of a camera in a video sequence forming a shot having multiple frames. In one implementation, the spatial coordinates of the camera include the position of the camera. In another implementation, the spatial coordinates of the camera include the orientation of the camera.

[0018]

[0022] One example of behavior is tracking the position of the main capture camera within the environment in real time. One implementation uses a six-degree-of-freedom tracking system; for example, X, Y, and Z coordinates are used to track the translation of the virtual camera from its origin, and pitch, yaw, and roll axes are used to track the local rotation of the camera. In one implementation, the origin of the virtual camera (i.e., 0,0,0) is the nodal point of the physical lens.

[0019]

[0023] In one implementation, the camera includes a lens. In step 120, a lens profile is generated that stores lens data corresponding to the camera's lens in the shot. In one implementation, the lens profile includes the following parameters: (1) active sensor dimensions / film plane / imager measurements, (2) image plane distance to nodal point, (3) camera focal length that accounts for zoom and lens breathing, (4) lens distortion profile, (5) image center offset, (6) lens aperture, (7) focus distance, and (8) lens vignetting or shading.

[0020]

[0024] In one implementation, the lens data is synchronized with each frame of the shot. In step 130, the lens data is encoded and sent to the render engine in step 140. In one implementation, a tool, script, or plug-in (e.g., Unreal Engine 4 or Unity) is used to feed the lens data back to the render engine. Next, in step 150, the camera movement during the shot is retraced. In step 160, the lens and one or more lens properties are recreated during the shot, and in step 170, the shot is replicated in the virtual environment using the retraced camera movement and recreated lens properties.

[0021]

[0025] In one implementation, the render engine retraces the movement of the main capture camera, mimicking the lens and its characteristics for each frame to virtually replicate the shot. This allows the operator to freely define the resolution for the plate while taking full advantage of all image render quality settings. Depending on the quality of the asset, this also means that there is little or no noise in the digital image. This depends on the resolution quality of the asset and the method of ingestion. With high-quality assets, it is possible to extract backplates with resolutions above 8K.

[0022]

[0026] FIG. 2A is a block diagram of a video rendering system 200 according to one implementation of the present disclosure. In the implementation shown in FIG. 2A, the video rendering system 200 is used in a video production or studio environment. The system 200 includes one or more cameras 220, 222 for image capture of a background scene 210, one or more sensors / trackers 230 for tracking the spatial coordinates (e.g., position and orientation) of the cameras, one or more processors 240 for processing the camera and sensor data to provide a render engine 242 for rendering frames of video, and a renderer 244. In one implementation, the system 200 renders frames from light detection and ranging (lidar) scanned assets with resolutions exceeding 8K. However, in general, assets of any quality can be used, although asset quality may limit the final achievable resolution.

[0023]

[0027] 2A, the tracker 230 tracks the spatial coordinates of one or more cameras 220, 222 in a video sequence forming a shot having multiple frames. In some implementations, the tracker 230 can track the spatial coordinates of a single camera 220, in which case tracking of the camera 222 is optional (dotted line). In one implementation, the spatial coordinates of the camera include the camera's position. In another implementation, the spatial coordinates of the camera include the camera's orientation.

[0024]

[0028] One example of operation is tracking the position of the cameras 220, 222 within the environment in real time. One implementation uses a six-degree-of-freedom tracking system. For example, X, Y, and Z coordinates are used to track the translation of the virtual camera from its origin, and pitch, yaw, and roll axes are used to track the local rotation of the camera. In one implementation, the origin of the virtual camera (i.e., 0,0,0) is the nodal point of the physical lens.

[0025]

[0029] In the implementation shown in FIG. 2A , processor 240 is coupled to cameras 220, 222. In one implementation, processor 240 generates a lens profile that stores lens data corresponding to the lens of each camera 220 or 222 in a shot. In one implementation, processor 240 synchronizes the lens data to each frame of the shot. In another implementation, processor 240 synchronizes the lens data to a timecode. In one implementation, processor 240 encodes the lens data and sends the encoded lens data to render engine 242. Processor 240 then retraces the camera movement during the shot. Processor 240 also recreates the lens and one or more characteristics of the lens and replicates the shot in the virtual environment using the retrace camera movement and recreated lens characteristics. Processor 240 then sends the replicated shot to renderer 244 to render the shot.

[0026]

[0030] In one implementation, processor 240 generates a lens profile and encodes lens data in real time during production. In one implementation, the lens profile includes the following parameters: (1) active sensor dimensions / film plane / imager measurements, (2) image plane distance to nodal point, (3) camera focal length accounting for zoom and lens breathing, (4) lens distortion profile, (5) image center offset, (6) lens aperture, and (7) focus distance. In another implementation, the lens profile includes light extinction information (i.e., vignetting).

[0027]

[0031] In one implementation, the render engine 242 of the processor 240 retraces the movement of the cameras 220, 222, mimicking the lenses and their characteristics for each frame to virtually replicate the shot. This allows the operator to freely define the resolution for the plate while taking full advantage of all image render quality settings. Depending on the quality of the asset, this also means that there is little or no noise in the digital image. This depends on the resolution quality of the asset and the capture method. With high-quality assets, it is possible to extract backplates with resolutions above 8K.

[0028]

[0032] FIG. 2B is a block diagram of a video rendering system 250 according to another implementation of the present disclosure. In the implementation shown in FIG. 2B, the video rendering system 250 is used in a video production or studio environment. The system 250 includes one or more cameras 220, 222 for image capture of a background scene 210, a tracker 230 for tracking the spatial coordinates (e.g., position and orientation) of the cameras 220, 222, a lens profiler 260, a lens encoder 262, a processor 270, and a renderer 280. The lens profiler 260, lens encoder 262, and processor 270 combine to provide a render engine 272 for processing camera and sensor data and rendering frames of video via the renderer 280. In one implementation, the system 250 renders frames from light detection and ranging (lidar) scanned assets with resolutions exceeding 8K. However, in general, assets of any quality can be used, although asset quality may limit the final achievable resolution.

[0029]

[0033] In the implementation shown in FIG. 2B , lens profiler 260 is coupled to cameras 220, 222. In one implementation, lens profiler 260 generates lens profiles that store lens data corresponding to the lenses of each camera 220 or 222 in a shot. In one implementation, lens data is synchronized to each frame of the shot. In another implementation, lens data is synchronized to a time code. Lens profiler 260 then sends the lens data to lens encoder 262. In one implementation, lens encoder 262 encodes the lens data and sends the encoded lens data to render engine 272 of processor 270, which then retraces the camera movement during the shot. Processor 270 also recreates the lens and one or more characteristics of the lens and replicates the shot within the virtual environment using the retrace camera movement from tracker 230 and the recreated lens characteristics from lens encoder 262. The processor 270 then sends the duplicated shot to the renderer 280 to render the shot.

[0030]

[0034] In one implementation, lens profiler 260 and lens encoder 262 generate lens profiles and encode lens data in real time during production. In one implementation, the lens profile includes the following parameters: (1) active sensor dimensions / film plane / imager measurements, (2) image plane distance to nodal point, (3) camera focal length accounting for zoom and lens breathing, (4) lens distortion profile, (5) image center offset, (6) lens aperture, (7) focus distance, and (8) vignetting or shading.

[0031]

[0035] In one implementation, the render engine 272 of the processor 270 retraces the movement of the cameras 220, 222, mimicking the lenses and their characteristics for each frame to virtually replicate the shot. This allows the operator to freely define the resolution for the plate while taking full advantage of all image render quality settings. Depending on the quality of the asset, this also means that there is no or very little noise in the digital image. This depends on the resolution quality of the asset and the capture method. With high-quality assets, it is possible to extract backplates with resolutions above 8K.

[0032]

[0036] Modifications to the system are possible; for example, assets of any type and resolution are compatible with this workflow. The quality of the rendered plates depends on the quality of the assets, e.g., 3D assets with resolutions above 8K. The method can also be applied to lower resolution assets and 2D assets.

[0033]

[0037] 3A is a diagram of a computer system 300 and a user 302, in accordance with an implementation of the present disclosure. User 302 uses computer system 300 to implement an application 390 for video rendering, such as that shown and described with respect to method 100 of FIG. 1 and systems 200, 250 of FIGS. 2A and 2B.

[0034]

[0038] The computer system 300 stores and executes the video rendering application 390 of Figure 3B. Additionally, the computer system 300 can communicate with a software program 304. The software program 304 can include the software code for the video rendering application 390. The software program 304 can be loaded onto an external medium, such as a CD, DVD, or storage drive, as described further below.

[0035]

[0039] Additionally, computer system 300 can be connected to network 380. Network 380 can be connected in a variety of different architectures, such as a client-server architecture, a peer-to-peer network architecture, or other types of architectures. For example, network 380 can communicate with server 385, which coordinates engines and data used in video rendering application 390. Network 380 can also be different types of networks. For example, network 380 can be the Internet, a local area network or any variation of a local area network, a wide area network, a metropolitan area network, an intranet or extranet, or a wireless network.

[0036]

[0040] 3B is a functional block diagram illustrating a computer system 300 hosting a video rendering application 390 according to an implementation of the present disclosure. The controller 310 is a programmable processor that controls the operation of the computer system 300 and its components. The controller 310 loads instructions (e.g., in the form of a computer program) from the memory 320 or an internal controller memory (not shown) and executes these instructions to control the system, for example, to provide data processing and capture camera movement data. In doing so, the controller 310 provides a software system for the video rendering application 390, for example, to render a backplate. Alternatively, this service can be implemented as a separate hardware component in the controller 310 or the computer system 300.

[0037]

[0041] Memory 320 temporarily stores data for use by other components of computer system 300. In one implementation, memory 320 is implemented as RAM. In another implementation, memory 320 also includes long-term or permanent memory, such as flash memory and / or ROM.

[0038]

[0042] Storage 330 stores data temporarily or long-term for use by other components of computer system 300. For example, storage 330 stores data used by video rendering application 390. In one implementation, storage 330 is a hard disk drive.

[0039]

[0043] Media device 340 accepts removable media and reads and / or writes data to the inserted media. In one implementation, for example, media device 340 is an optical disc drive.

[0040]

[0044] User interface 350 includes components for receiving user input from a user of computer system 300 and presenting information to user 302. In one implementation, user interface 350 includes a keyboard, a mouse, audio speakers, and a display. Controller 310 uses the input from user 302 to coordinate the operation of computer system 300.

[0041]

[0045] I / O interface 360 ​​includes one or more I / O ports for connecting to corresponding I / O devices, such as external storage or supplemental devices (e.g., printers or PDAs). In one implementation, the ports of I / O interface 360 ​​include ports such as USB ports, PCMCIA ports, serial ports, and / or parallel ports. In another implementation, I / O interface 360 ​​includes a wireless interface for communicating wirelessly with external devices.

[0042]

[0046] Network interface 370 includes wired and / or wireless network connections, such as an RJ-45 supporting an Ethernet connection or a "Wi-Fi" interface (including but not limited to 802.11).

[0043]

[0047] Computer system 300 includes additional hardware and software typical of a computer system (e.g., power, cooling, operating system), but these components are not specifically shown in Figure 3B for simplicity. Other implementations may use different configurations of computer systems (e.g., different bus or storage configurations or multiprocessor configurations).

[0044]

[0048] The description herein of the disclosed implementations is provided to enable any person skilled in the art to make or use the disclosure. Many modifications of these implementations will be readily apparent to those skilled in the art, and the principles defined herein may be applied to other implementations without departing from the spirit or scope of the disclosure.

[0045]

[0049] Further variations and implementations are possible. For example, in addition to video production for film or television, implementations of the present systems and methods can be adapted and adapted for other applications, such as commercials, web-based or internet content, virtual production (e.g., virtual reality environments), and visual effects. Thus, the present disclosure is not intended to be limited to the implementations shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0046]

[0050] Not all features of each of the above embodiments are necessarily required in a particular implementation of the present disclosure. Furthermore, it should be understood that the description and drawings presented herein are representative of the subject matter broadly intended by the present disclosure. Furthermore, it should be understood that the scope of the present disclosure fully encompasses other implementations that may become apparent to those skilled in the art, and therefore, the scope of the present disclosure is not limited by anything other than the appended claims. [Explanation of symbols]

[0047] 100 Methods for Video Rendering Tracking the spatial coordinates of a camera in a video sequence forming a shot with multiple frames 120 Generate a lens profile that stores lens data corresponding to the camera lens in the shot 130 Encode lens data 140 Send lens data to the render engine Retracing the camera movement during 150 shots Recreate the lens and one or more lens characteristics in 160 shots 170 Replicate shots in a virtual environment using retraced camera movements and recreated lens properties 200 Video Rendering System 210 Background Scenes 220,222 cameras 230 Sensors / Trackers 240 processors 242 Render Engine 244 Renderer 250 Video Rendering System 260 Lens Profiler 262 Lens Encoder 270 processor 272 Render Engine 280 Renderer 300 Computer Systems 302 users 304 Software Programs 310 Controller 320 memory 330 Storage 340 Media Devices 350 User Interface 360 I / O Interface 370 Network Interface 380 Network 385 Server 390 Video Rendering Applications

Claims

1. 1. A method for video rendering, comprising: tracking spatial coordinates of a plurality of cameras in a video sequence forming a shot having a plurality of frames, each of the plurality of cameras having a lens; outputting the tracked spatial coordinates as sensor data so that the movements of the multiple cameras can be retraced; creating a lens profile that stores lens data corresponding to the lenses of the cameras in the shot; encoding the lens data; sending the lens data to a render engine; using the sensor data including the tracked spatial coordinates in the render engine to retrace the movements of the multiple cameras during the shot to virtually replicate the shot; using the lens data to recreate the lens and one or more characteristics of the lens in the shot; replicating the shot in a virtual environment using the retrace camera movement and recreated lens properties; A method comprising:

2. The virtual environment is generated based on scan data of the real environment. The method of claim 1.

3. the spatial coordinates of each of the plurality of cameras include a position of each camera; The method of claim 1.

4. the spatial coordinates of each of the plurality of cameras include an orientation of each camera; The method of claim 1.

5. The orientation of each camera includes pitch, yaw, and roll axes used to track the local rotation of each camera. The method of claim 4.

6. the lens profile includes a nodal point, which is a point where all light beams intersect within the lens and is projected onto an image plane; The method of claim 1.

7. the lens profile includes at least one of a distance of the image plane to the nodal point, a focal length of each camera, a lens distortion profile, an image center offset, a lens aperture, a focus distance, and vignetting or lens shading; The method of claim 6.

8. The lens profile is the point where all light beams intersect within the lens and includes at least one of the following: a nodal point projected onto an image plane, a distance of the image plane to the nodal point, an image center shift, a lens aperture, and vignetting or lens shading. The method of claim 1.

9. and synchronizing the lens data with each frame of the shot. The method of claim 1.

10. The step of replicating the shot includes a step of virtually replicating the shot by mimicking the lens and the lens characteristics for each frame. The method of claim 1.

11. 1. A system for video rendering, comprising: a plurality of cameras, each having a lens, for capturing images of a background scene and outputting the captured images as camera data; and a plurality of sensors for tracking spatial coordinates of the plurality of cameras in a video sequence forming a shot having a plurality of frames, and outputting the tracked spatial coordinates as sensor data so that movement of the plurality of cameras can be retraceable; a processor coupled to the plurality of cameras and the plurality of sensors for generating a lens profile storing lens data corresponding to the lenses of the plurality of cameras in the shot; Including, the processor retraces the movements of the multiple cameras during the shot using the camera data and the sensor data including the tracked spatial coordinates, and processes the camera data, the sensor data, and the lens data to replicate the shot in a virtual environment using lens characteristics corresponding to the retrace camera movements and the lens data. system.

12. synchronizing the lens data with each frame of the shot; The video rendering system of claim 11.

13. synchronizing the lens data with a time code; The video rendering system of claim 11.

14. a renderer for rendering the replicated shots; The video rendering system of claim 11.

15. The processor encodes the lens data and generates a signal for the lens and one or more of the lens. recreates two or more properties, The video rendering system of claim 11.

16. a render engine for retracing the movements of the multiple cameras to mimic the lenses and their characteristics for each frame to virtually replicate the shot; Further comprising: The video rendering system of claim 11.

17. On the computer, receiving tracked spatial coordinates of a plurality of cameras in a video sequence forming a shot having a plurality of frames, each of the plurality of cameras having a lens; receiving a lens profile storing lens data corresponding to the lenses of the plurality of cameras; using the tracked spatial coordinates to retrace the movements of the cameras during the shot to virtually replicate the shot; replicating the shot in a virtual environment based on the retrace camera movement and the lens profile; to carry out program.

18. The computer, recreating the lens and one or more properties of the lens; replicating the shot within the virtual environment based on the retrace camera movement and recreated lens properties; to carry out The program according to claim 17.

19. the lens profile includes at least one of the following: a distance of the image plane to a nodal point, a focal length of each camera, a lens distortion profile, an image center offset, a lens aperture, a focus distance, and vignetting or lens shading; The program according to claim 17.

20. The lens data is synchronized with a time code. The program according to claim 17.

21. the lens data is synchronized to each frame of the shot; The program according to claim 17.

22. causing the computer to virtually replicate the shot by mimicking the lens and lens characteristics on a frame-by-frame basis; 19. The program of claim 18.

Images