Rendering backplane

Abstract

The invention relates to rendering a backplane. Video rendering, including: tracking spatial coordinates of at least one camera during formation of a video sequence of a shot having a plurality of frames, where each of the at least one camera has a lens; creating a lens profile that stores lens data corresponding to a lens of the at least one camera during photographing; encoding the lens data; sending the lens data to a rendering engine; tracing the movement of the at least one camera during shooting; recreating the lens and one or more characteristics of the lens during photographing; and replicating the lens in the virtual environment using the traced camera movement and the recreated lens characteristics.

Description

[0001] This application is a divisional application of Chinese patent application 202080048901.1, filed on December 28, 2021, with the invention name “Rendering Backplane”. Technical Field

[0002] The present disclosure relates to rendering video, and more particularly, to rendering backplates. Background Art

[0003] In conventional systems for video production, rendering a background "plate" (sometimes called a "back plate") involves photographing a background scene without the subject. However, the disadvantage of using a back plate is that the process involves a lot of manual work and can therefore be expensive and cumbersome. Summary of the invention

[0004] The present disclosure provides for processing and rendering video data.

[0005] In one embodiment, a method for video rendering is disclosed. The method includes: tracking spatial coordinates of at least one camera during a video sequence forming a shot having a plurality of frames, wherein each of the at least one camera has a lens; creating a lens profile storing lens data corresponding to the lens of the at least one camera during capture; encoding the lens data; sending the lens data to a rendering engine; tracing the movement of the at least one camera during capture; recreating the lens and one or more properties of the lens during capture; and replicating the shot in a virtual environment using the traced camera movement and the recreated lens properties.

[0006] In one embodiment, the spatial coordinates of each camera in at least one camera include the position of each camera. In one embodiment, the spatial coordinates of each camera in at least one camera include the orientation of each camera. In one embodiment, the orientation of each camera includes a pitch axis, a yaw axis, and a roll axis for tracking the local rotation of each camera. In one embodiment, the lens profile includes a node, which is the point where all light beams intersect and cross within the lens, and then the point is projected onto the image plane. In one embodiment, the lens profile includes at least one of the following: an image plane distance to a node; a focal length of each camera; a lens distortion profile; an image center shift; a lens aperture; and a focal distance. In one embodiment, the method further includes synchronizing the lens data with a corresponding frame of the lens. In one embodiment, replicating a lens includes emulating the lens and lens characteristics frame by frame to virtually replicate the lens.

[0007] In another embodiment, a system for video rendering is disclosed. The system includes: at least one camera for capturing an image of a background scene, the at least one camera for outputting the captured image as camera data, wherein each camera of the at least one camera has a lens; at least one sensor for tracking spatial coordinates of at least one camera during a video sequence forming a shot having a plurality of frames, the at least one tracker for outputting the tracked spatial coordinates as sensor data; and a processor coupled to the at least one camera and the at least one sensor, the processor for generating a lens profile storing lens data corresponding to the lens of the at least one camera during capture, wherein the processor processes the camera data, the sensor data, and the lens data to replicate the shot.

[0008] In one embodiment, the lens data is synchronized to the corresponding frame of the shot. In one embodiment, the lens data is synchronized to a time code. In one embodiment, the system further comprises a renderer to render the replicated shot. In one embodiment, the processor encodes the lens data, traces the movement of at least one camera during filming, recreates the lens and one or more characteristics of the lens, and replicates the shot in a virtual environment. In one embodiment, the system further comprises a rendering engine to trace the movement of at least one camera and mimic the lens and its characteristics frame by frame to virtually replicate the shot.

[0009] In another embodiment, a non-transitory computer-readable storage medium storing a computer program for rendering a video is disclosed. The computer program includes executable instructions that cause a computer to: track spatial coordinates of at least one camera during a video sequence forming a shot having a plurality of frames, wherein each of the at least one camera has a lens; generate a lens profile storing lens data corresponding to the lens of the at least one camera during capture; encode the lens data; send the lens data to a rendering engine; retrace the movement of at least one camera during capture; recreate the lens and one or more properties of the lens during capture; and replicate the shot in a virtual environment using the retraceable camera movement and the recreated lens properties.

[0010] In one embodiment, the spatial coordinates of each camera in the at least one camera include the position of each camera. In one embodiment, the lens profile includes a node (which is located on the optical axis), which is the point where all light beams intersect and cross within the lens, and then the point is projected onto the image plane. In one embodiment, the lens profile includes at least one of the following: image plane distance to the node; focal length of each camera; lens distortion profile; image center shift; lens aperture; and focus distance. In one embodiment, the computer program also includes executable instructions that cause the computer to synchronize the lens data with the corresponding frame of the lens. In one embodiment, the executable instructions that cause the computer to replicate the lens include executable instructions that cause the computer to simulate the lens and lens characteristics frame by frame to virtually replicate the lens.

[0011] Other features and advantages should be apparent from the present description, which illustrates by way of example various aspects of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Details as to the structure and operation of the present disclosure may be gleaned in part by examination of the accompanying drawings, in which like reference numerals refer to like parts, and in which:

[0013] Figure 1 is a flowchart of a method for video rendering according to one embodiment of the present disclosure;

[0014] Figure 2A is a block diagram of a video rendering system according to one embodiment of the present disclosure;

[0015] Figure 2B is a block diagram of a video rendering system according to another embodiment of the present disclosure;

[0016] Figure 3A is a representation of a computer system and a user according to an embodiment of the present disclosure; and

[0017] Figure 3B is a functional block diagram illustrating a computer system hosting a video rendering application according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0018] As mentioned above, the disadvantage of using backplates when rendering backgrounds in video production is that the process involves a lot of manual work. Therefore, the process can be expensive and cumbersome.

[0019] Certain embodiments of the present disclosure provide methods and systems for implementing techniques for processing and rendering video data. In one embodiment, a video system renders accurate, high-fidelity backplates for visual effects (VFX) for use in film, television, and commercials. This method can extract from assets with resolutions exceeding 8K or any other asset, allowing users to freely define the resolution for delivery.

[0020] After reading the following description, it will become apparent how to implement the present disclosure in various embodiments and applications. Although various embodiments of the present disclosure will be described herein, it should be understood that these embodiments are presented only by way of example and not limitation. Therefore, the detailed description of various embodiments should not be interpreted as limiting the scope or breadth of the present disclosure.

[0021] Features provided in the following embodiments may include, but are not limited to, one or more of the following items. For example, in one embodiment, camera telemetry is recorded during video production along with lens distortion, imager dimensions, lens nodes, focal length, focal distance, and aperture. The time code and / or frame number of each shot (or recording duration) are also recorded. In another embodiment, the recorded data is fed into a rendering engine containing assets (i.e., background assets of the board) using a tool or script. The individual frames are then rendered with the desired image quality. It is also possible to combine the individual frames into a video file.

[0022] Figure 1 is a flowchart of a method 100 for video rendering according to one embodiment of the present disclosure. Figure 1 In the illustrated embodiment, at step 110, the spatial coordinates of the camera are tracked during the formation of a video sequence of a shot having a plurality of frames. In one embodiment, the spatial coordinates of the camera include the position of the camera. In another embodiment, the spatial coordinates of the camera include the orientation of the camera.

[0023] In one example of operation, the main capture camera position is tracked in real time within the environment. In one embodiment, a six degree of freedom tracking system is used. For example, X, Y, Z coordinates are used to track translation from the virtual camera origin, and the pitch, yaw, roll axes are used to track the local rotation of the camera. In one embodiment, the origin (or 0,0,0) of the virtual camera is the node of the physical lens.

[0024] In one embodiment, the camera includes a lens. At step 120, a lens profile is generated that stores lens data corresponding to the lens of the camera during capture. In one embodiment, the lens profile includes the following parameters: (1) measurement of active sensor dimensions / film plane / imager; (2) image plane distance to the nodal point; (3) focal length of the camera taking into account zoom and lens breathing; (4) lens distortion profile; (5) image center shift; (6) lens aperture; (7) focal distance; and (8) vignetting or shading of the lens.

[0025] In one embodiment, the lens data is synchronized to the corresponding frame of the shot. The lens data is encoded at step 130 and sent to a rendering engine at step 140. In one embodiment, the lens data is fed back into the rendering engine using a tool, script, or plugin such as Unreal Engine 4 or Unity. The movement of the camera during the shoot is then traced back at step 150. At step 160, the lens and one or more characteristics of the lens during the shoot are recreated, and at step 170, the lens is replicated in a virtual environment using the traced camera movement and the recreated lens characteristics.

[0026] In one embodiment, the rendering engine traces the movements of the main capture camera and mimics the lens and its characteristics frame by frame to virtually replicate the shot. This allows the operator to freely define the resolution of the plate while taking full advantage of all image rendering quality settings. Depending on the quality of the asset, this also means that there is little or no noise in the digital image. This depends on the resolution quality of the asset and the capture method. With high-quality assets, it is possible to extract backplates with resolutions exceeding 8K.

[0027] Figure 2A is a block diagram of a video rendering system 200 according to one embodiment of the present disclosure. Figure 2A In the illustrated embodiment, a video rendering system 200 is used in a video production or studio environment. The system 200 includes one or more cameras 220, 222 for image capture of a background scene 210, one or more sensors / trackers 230 for tracking the spatial coordinates (e.g., position and orientation) of the camera, one or more processors 240 for processing camera and sensor data and providing a rendering engine 242 for rendering video frames, and a renderer 244. In one embodiment, the system 200 renders frames derived from light detection and ranging (LIDAR) scanned assets with a resolution exceeding 8K. However, assets of any quality may generally be used, even if the quality of the assets will limit the final achievable solution.

[0028] exist Figure 2AIn the illustrated embodiment, the tracker 230 tracks the spatial coordinates of one or more cameras 220, 222 during a video sequence forming a shot having a plurality of frames. In some embodiments, the tracker 230 may track the spatial coordinates of a single camera 220, in which case the tracking of the camera 222 is optional (dotted line). In one embodiment, the spatial coordinates of the camera include the position of the camera. In another embodiment, the spatial coordinates of the camera include the orientation of the camera.

[0029] In one example of operation, the position of the cameras 220, 222 is tracked in real time within the environment. In one embodiment, a six degree of freedom tracking system is used. For example, X, Y, Z coordinates are used to track translation from the virtual camera origin, and the pitch, yaw, roll axes are used to track the local rotation of the camera. In one embodiment, the origin (or 0,0,0) of the virtual camera is the node of the physical lens.

[0030] exist Figure 2A In the illustrated embodiment, a processor 240 is coupled to the cameras 220, 222. In one embodiment, the processor 240 generates a lens profile that stores lens data corresponding to the lens of each camera 220 or 222 during capture. In one embodiment, the lens data is synchronized to the corresponding frame of the shot. In another embodiment, the lens data is synchronized to a time code. In one embodiment, the processor 240 encodes the lens data and sends the encoded lens data to the rendering engine 242. The processor 240 then traces the movement of the camera during capture. The processor 240 also recreates the lens and one or more characteristics of the lens and replicates the shot in the virtual environment using the traced camera movement and the recreated lens characteristics. The processor 240 then sends the replicated shot to the renderer 244 to render the shot.

[0031] In one embodiment, processor 240 generates a lens profile and encodes the lens data in real time during production. In one embodiment, the lens profile includes the following parameters: (1) measurement of active sensor dimensions / film plane / imager; (2) image plane distance to the nodal point; (3) focal length of the camera taking into account zoom and lens breathing; (4) lens distortion profile; (5) image center shift; (6) lens aperture; and (7) focus distance. In another embodiment, the lens profile includes light falloff information (i.e., vignetting).

[0032] In one embodiment, the rendering engine 242 of the processor 240 traces the movement of the cameras 220, 222 and mimics the lens and its characteristics frame by frame to virtually replicate the shot. This allows the operator to freely define the resolution of the plate while taking full advantage of all image rendering quality settings. Depending on the quality of the asset, this also means that there is no or almost no noise in the digital image. This depends on the resolution quality of the asset and the capture method. With high quality assets, it is possible to extract backplates with resolutions exceeding 8K.

[0033] Figure 2B is a block diagram of a video rendering system 250 according to another embodiment of the present disclosure. Figure 2B In the illustrated embodiment, a video rendering system 250 is used in a video production or studio environment. The system 250 includes one or more cameras 220, 222 for image capture of a background scene 210, a tracker 230 for tracking spatial coordinates (e.g., position and orientation) of the cameras 220, 222, a lens profiler 260, a lens encoder 262, a processor 270, and a renderer 280. The lens profiler 260, the lens encoder 262, and the processor 270 combine to process camera and sensor data and provide a rendering engine 272 for rendering video frames by the renderer 280. In one embodiment, the system 200 renders frames of assets derived from light detection and ranging (LIDAR) scans of more than 8K resolution. However, it is generally possible to use assets of any quality, even if the asset quality will limit the final achievable solution.

[0034] exist Figure 2B In the illustrated embodiment, a lens profiler 260 is coupled to the cameras 220, 222. In one embodiment, the lens profiler 260 generates a lens profile that stores lens data corresponding to the lens of each camera 220 or 222 during capture. In one embodiment, the lens data is synchronized with the corresponding frame of the lens. In another embodiment, the lens data is synchronized to a time code. The lens profiler 260 then sends the lens data to a lens encoder 262. In one embodiment, the lens encoder 262 encodes the lens data and sends the encoded lens data to a rendering engine 272 of a processor 270, which then traces the movement of the camera during capture. The processor 270 also recreates the lens and one or more characteristics of the lens and replicates the lens in a virtual environment using the traced camera movement from the tracker 230 and the recreated lens characteristics from the lens encoder 262. The processor 270 then sends the replicated lens to a renderer 280 to render the lens.

[0035] In one embodiment, the lens profiler 260 and lens encoder 262 generate lens profiles and encode lens data in real time during production. In one embodiment, the lens profile includes the following parameters: (1) measurement of active sensor dimensions / film plane / imager; (2) image plane distance to the nodal point; (3) focal length of the camera taking into account zoom and lens breathing; (4) lens distortion profile; (5) image center shift; (6) lens aperture; and (7) focal distance; and (8) vignetting or lens shading.

[0036] In one embodiment, the rendering engine 272 of the processor 270 traces the movement of the cameras 220, 222 and mimics the lens and its characteristics frame by frame to virtually replicate the shot. This allows the operator to freely define the resolution of the plate while taking full advantage of all image rendering quality settings. Depending on the quality of the asset, this also means that there is no or almost no noise in the digital image. This depends on the resolution quality of the asset and the capture method. With high quality assets, it is possible to extract backplates with resolutions exceeding 8K.

[0037] Variations of the system are also possible. For example, assets of any kind and resolution are compatible with this workflow. The quality of the rendered board depends on the quality of the asset (e.g. the asset is 3D and has over 8K resolution). It is possible to apply this method to assets of lower resolution, or even 2D assets.

[0038] Figure 3A 300 and a user 302 according to an embodiment of the present disclosure. The user 302 uses the computer system 300 to implement an application 390 for video rendering, as described with respect to FIG. Figure 1 Method 100 and Figure 2A and 2B The systems 200, 250 shown and described in FIG.

[0039] The computer system 300 stores and executes Figure 3B The computer system 300 may also include a video rendering application 390. In addition, the computer system 300 may communicate with a software program 304. The software program 304 may include software code for the video rendering application 390. The software program 304 may be loaded on an external medium such as a CD, DVD, or storage drive, as will be further explained below.

[0040] In addition, the computer system 300 can be connected to a network 380. The network 380 can be connected in a variety of different architectures, such as a client-server architecture, a peer-to-peer network architecture, or other types of architectures. For example, the network 380 can communicate with a server 385 that coordinates engines and data used within the video rendering application 390. Moreover, the network can be a different type of network. For example, the network 380 can be the Internet, a local area network or any variation of a local area network, a wide area network, a metropolitan area network, an intranet or an extranet, or a wireless network.

[0041] Figure 3B is a functional block diagram illustrating a computer system 300 hosting a video rendering application 390 according to an embodiment of the present disclosure. The controller 310 is a programmable processor and controls the operation of the computer system 300 and its components. The controller 310 loads instructions (e.g., in the form of a computer program) from a memory 320 or an embedded controller memory (not shown) and executes these instructions to control the system, such as to provide data processing for capturing camera movement data. In its execution, the controller 310 provides a software system to the video rendering application 390, such as to render a backplane. Alternatively, this service can be implemented as a separate hardware component in the controller 310 or computer system 300.

[0042] Memory 320 temporarily stores data for use by other components of computer system 300. In one embodiment, memory 320 is implemented as RAM. In another embodiment, memory 320 also includes long-term or permanent memory, such as flash memory and / or ROM.

[0043] Storage device 330 stores data temporarily or long-term for use by other components of computer system 300. For example, memory 330 stores data used by video rendering application 390. In one implementation, storage device 330 is a hard drive.

[0044] The media device 340 receives removable media and reads and / or writes data to the inserted media. In one implementation, for example, the media device 340 is an optical disk drive.

[0045] User interface 350 includes components for accepting user input from a user of computer system 300 and presenting information to user 302. In one implementation, user interface 350 includes a keyboard, a mouse, audio speakers, and a display. Controller 310 uses input from user 302 to adjust the operation of computer system 300.

[0046] The I / O interface 360 ​​includes one or more I / O ports to connect to corresponding I / O devices, such as external storage devices or supplemental devices (e.g., printers or PDAs). In one embodiment, the ports of the I / O interface 360 ​​include ports such as a USB port, a PCMCIA port, a serial port, and / or a parallel port. In another embodiment, the I / O interface 360 ​​includes a wireless interface for wireless communication with external devices.

[0047] Network interface 370 includes a wired and / or wireless network connection, such as an RJ-45 or "Wi-Fi" interface supporting an Ethernet connection (including but not limited to 802.11).

[0048] Computer system 300 includes additional hardware and software typical of computer systems (e.g., power supply, cooling, operating system), but for simplicity, the following are described in detail. Figure 3B These components are not specifically shown in the drawings. In other implementations, different configurations of the computer system can be used (eg, different bus or storage configurations or multi-processor configurations).

[0049] The description of the disclosed embodiments herein is provided to enable any person skilled in the art to make or use the present disclosure. Many modifications to these embodiments will be apparent to those skilled in the art, and the principles defined herein may be applied to other embodiments without departing from the spirit or scope of the present disclosure.

[0050] Additional variations and implementations are possible. For example, in addition to video production for film or television, implementations of the systems and methods may be applied and adapted to other applications, such as commercials, web or Internet-based content, virtual production (e.g., virtual reality environments), and visual effects. Thus, the present disclosure is not intended to be limited to the implementations shown herein, but is to be accorded the widest scope consistent with the principal and novel features disclosed herein.

[0051] Not all features of each of the examples discussed above are necessarily required in a particular embodiment of the present disclosure. In addition, it should be understood that the description and drawings presented herein represent the broadly contemplated subject matter of the present disclosure. It should also be understood that the scope of the present disclosure fully encompasses other embodiments that may become apparent to those skilled in the art, and the scope of the present disclosure is accordingly limited only by the appended claims.

Claims

1. A method for video rendering, comprising: obtaining spatial coordinates of at least one camera tracked during a video sequence forming a shot having a plurality of frames; receiving a lens profile storing lens data corresponding to a lens of the at least one camera during capture, the lens data comprising one or more lens characteristics, wherein the one or more lens characteristics include: a measure of active sensor dimensions; a distance to an image plane of a node, the node being projected onto the image plane and being the point at which all light beams intersect and cross within the lens; a focal length of each camera; a lens distortion profile; an image center shift; a lens aperture; a focal distance; and vignetting or lens shading; tracing back the movement of the at least one camera during filming; recreating the lens to virtually replicate the lens; and The shot is replicated in a virtual environment using the retroactive camera movement and the recreated lens properties by mimicking the lens and the one or more lens properties of the lens for frames of the shot. 2 . The method of claim 1 , further comprising tracking spatial coordinates of the at least one camera during a video sequence. 3 . The method of claim 1 , further comprising creating a lens profile storing lens data corresponding to a lens of the at least one camera during the capturing.

4. The method according to any one of the preceding claims, further comprising: encoding a lens profile; as well as Send the lens profile to the rendering engine. The method of claim 4 , wherein the lens profile is fed back into the rendering engine using a plug-in.

6. A method according to any one of the preceding claims, wherein: The lens data is synchronized with the corresponding frame of the shot.

7. The method according to any one of claims 1 to 5, wherein the lens data is synchronized with a time code.

8. A system for video rendering, the system comprising a processor configured to: obtaining spatial coordinates of at least one camera tracked during a video sequence forming a shot having a plurality of frames; receiving a lens profile storing lens data corresponding to a lens of the at least one camera during capture, the lens data comprising one or more lens characteristics, wherein: The one or more lens characteristics include: a measure of active sensor dimensions; a distance to the image plane of a node, which is projected onto the image plane and is the point where all beams intersect and cross within the lens; a focal length of each camera; a lens distortion profile; an image center shift; a lens aperture; a focal distance; and vignetting or lens shading; tracing back the movement of the at least one camera during filming; recreating the lens to virtually replicate the lens; and The shot is replicated in a virtual environment using the retroactive camera movement and the recreated lens properties by mimicking the lens and the one or more lens properties of the lens for frames of the shot.

9. The system of claim 8, further comprising a tracker that tracks spatial coordinates of the at least one camera during a video sequence. 10 . The system of claim 8 , further comprising a lens profiler that generates a lens profile storing lens data corresponding to a lens of the at least one camera during the capturing.

11. The system according to any one of claims 8 to 10, further comprising: A lens encoder is used to encode the lens profile and send the lens profile to the rendering engine.

12. The system of claim 11, wherein the lens data is fed back into the rendering engine using a plug-in.

13. A system according to any one of claims 8 to 12, wherein: The lens data is synchronized with the corresponding frame of the shot.

14. A system according to any one of claims 8 to 12, wherein: The lens data is synchronized with the time code.

15. The system of claim 8, further comprising the at least one camera, the at least one camera configured to capture an image of a background scene, the at least one camera configured to output the captured image as camera data.

16. A computer program for rendering a video, the computer program comprising executable instructions that cause a computer to: obtaining spatial coordinates of at least one camera tracked during a video sequence forming a shot having a plurality of frames; receiving a lens profile storing lens data corresponding to a lens of the at least one camera during capture, the lens data comprising one or more lens characteristics, wherein: The one or more lens characteristics include: a measure of active sensor dimensions; a distance to the image plane of a node, which is projected onto the image plane and is the point where all beams intersect and cross within the lens; a focal length of each camera; a lens distortion profile; an image center shift; a lens aperture; a focal distance; and vignetting or lens shading; tracing back the movement of the at least one camera during filming; recreating the lens to virtually replicate the lens; and The shot is replicated in a virtual environment using the retroactive camera movement and the recreated lens properties by mimicking the lens and the one or more lens properties of the lens for frames of the shot.

Images