A real-time variable green screen generation method and image matting method based on depth information

Abstract

The present invention relates to a method for virtual production of movies based on an LED background wall, and particularly to a method for real-time variable green screen generation and keying method based on depth information, belonging to the field of virtual production of movies. Aiming at the problems of color spill and green screen exposure in the existing production technology, the present invention proposes a method for real-time variable green screen generation and keying method based on depth information. The depth camera is used to obtain the contour of the subject to be photographed, and the processed contour area of the subject to be photographed is projected onto the LED background wall as a green screen or blue screen area, replacing the original large-area quadrilateral green screen or blue screen, reducing the color spill effect on the subject to be photographed and the probability of green screen exposure, improving the quality of the captured image, and using the green / blue screen contour recorded during the pre-shooting to quickly perform keying production in the post-production. The present invention reduces the area of the blue / green screen in the LED background wall, thereby reducing the influence of the green screen during the production process and improving the quality of the film and television works.

Description

Technical Field

[0001] The present invention relates to a method for virtual production of movies based on an LED background wall, and particularly to a method for real-time variable green screen generation and a keying method based on depth information, belonging to the field of virtual production of movies. Background Art

[0002] In the technology of virtual production of movies based on an LED background wall, in most cases, the three-dimensional image of the background is displayed in real time in the inner viewing cone of the LED background wall, and in-camera visual effects shooting is carried out in real time to complete the film production. The main advantages of this shooting method are as follows: (1) Using an LED screen as the "background wall" behind the subject, the subject directly performs under the surround of the LED background wall, and in-camera visual effects shooting is carried out in real time, eliminating the need to use a green / blue screen for shooting and then synthesizing in a computer; (2) This method provides a real-time and delicate lighting environment, eliminating the trouble of manually moving and adjusting lighting equipment; (3) This method can provide the reflected image for the subject with specular reflection, such as the environment under the reflection of the shiny paint of a car; (4) This method can provide the transmitted image for the subject with transmission, such as the background after transmission through a glass cup. - Due to the limited quality of the three-dimensional images rendered in real time at present, many films still need to be shot with a green screen and then post-produced, but at the same time, an LED background wall is required to provide real-time lighting and the reflected image of the subject. To meet such production requirements, the current practice is to display green or blue color blocks in the quadrilateral inner viewing cone area, and the outer viewing cone area still maintains the three-dimensional real-time rendered image to provide real-time lighting and the reflected and transmitted images of the subject. However, in this case, the green or blue color blocks in the inner viewing cone occupy a relatively large area of the image, and still cause a certain amount of color bleeding on the subject, which brings inconvenience to post-keying, and at certain angles, due to the reflection or transmission on the subject, a green screen exposure phenomenon will occur. In addition, the technology of depth information acquisition devices has been relatively mature, but the depth maps obtained by them still have relatively low accuracy for film keying and cannot be directly used for film keying. Summary of the Invention

[0003] Aiming at the color bleeding and green screen exposure phenomena existing in the existing production technology, the present invention proposes a method for real-time variable green screen generation and a keying method based on depth information, aiming to reduce the area of the blue / green screen in the LED background wall, thereby reducing the influence of the green screen in the production process and improving the production quality of film and television works.

[0004] The object of the present invention is achieved by the following technical solutions:

[0005] The present invention uses a depth camera to obtain the contour of the subject to be photographed, and projects the processed contour area of the subject to be photographed onto the LED background wall as a green screen or a blue screen area, replacing the original large-area quadrilateral green screen or blue screen, reducing the color spill effect on the subject to be photographed and the probability of green screen revealing, and improving the quality of the captured image. And by using the green / blue screen contour recorded during the pre-shooting, keying can be quickly performed during post-production.

[0006] The specific steps of a real-time variable green screen generation method and a keying method based on depth information are as follows:

[0007] Step 1: Connect the camera, the depth camera and the motion tracking device, make the orientations of the three consistent and control the distances between them, and perform compensation for position and attitude calculation according to the measured relative position between the motion tracking device and the camera.

[0008] Step 2: Obtain a depth map through the depth camera.

[0009] Step 3: Perform binarization processing on the obtained depth map.

[0010] Use the motion tracking device to obtain the relative position of the camera, calculate the closest distance between the camera and the LED background wall in real time, calculate the threshold based on this distance, and perform binarization processing on the depth map, that is, the picture area greater than the threshold is black, and vice versa is white. Record and store the processed green / blue screen contour for use during post-production keying.

[0011] Step 4: Perform dilation processing on the binarized contour of the subject to be photographed to generate a dilated binarized depth map, which is the available green / blue screen contour;

[0012] Step 5: Use the green / blue screen contour generated in Step 4 to color the subject area to an appropriate brightness of green or blue, and the other areas are transparent. Adjust the size of the green / blue screen contour to match according to the field of view angle of the depth camera, and merge it with the rendered three-dimensional scene picture to obtain an inner cone picture with a variable green / blue screen;

[0013] Step 6: Use the inner cone picture with a variable green / blue screen obtained in Step 5 to perform off-axis projection according to the relative position between the camera and the LED background wall, and display the picture on the LED background wall;

[0014] Step 7: Use the camera to photograph the picture of the LED background wall and the subject to be photographed to obtain a camera picture for post-processing in the later stage.

[0015] Step 8: Post-production

[0016] Step 8.1: Perform dilation processing on the green / blue screen contour image obtained from the previous shooting. The dilation width should be slightly smaller than that in Step 4 during the previous shooting to obtain a mask after contour dilation.

[0017] Step 8.2: Use the chroma key method to perform keying processing on the camera image to obtain a mask after keying the camera image.

[0018] Step 8.3: Add the dilated mask in Step 1 and the mask after keying the camera image in Step 2 to obtain a combined mask.

[0019] Step 8.4: Use the combined mask obtained in Step 3 to perform keying processing on the camera image to obtain a camera image with an alpha channel.

[0020] Step 8.5: Composite the rendered late-stage 3D virtual image and the camera image with an alpha channel in Step 4 to obtain the final image and complete the late-stage production.

[0021] Beneficial Effects

[0022] 1. The present invention reduces the area of the green screen or blue screen displayed on the LED background wall by using a depth camera to obtain the contour of the subject to be photographed and combining it with the 3D scene image, thereby reducing the color spill caused by the green screen or blue screen on the subject to be photographed and better restoring the environmental image in the reflection and projection areas on the object to be photographed, effectively improving the image quality of blue / green screen production in film virtualization production based on an LED background wall and enhancing the realism of the image. Description of the Drawings

[0023] Figure 1 Schematic diagram of on-site shooting for the real-time variable green screen generation method based on depth information;

[0024] Figure 2 Flowchart of the real-time variable green screen generation method based on depth information;

[0025] Figure 3 Schematic diagram of the late-stage keying production method for the real-time variable green screen based on depth information;

[0026] Figure 4 Flowchart of the late-stage keying production method for the real-time variable green screen based on depth information; Detailed Embodiments

[0027] The present invention will be described in detail below in conjunction with the drawings and embodiments. At the same time, the technical problems solved by the technical solution of the present invention and the beneficial effects are also described. It should be noted that the described embodiments are only for facilitating the understanding of the present invention and do not limit it in any way.

[0028] It should be noted that: for the current depth acquisition device, the depth map obtained is still of low accuracy for film keying and cannot be directly used for film keying. However, it can provide a reliable and real-time changing depth image for the "variable green / blue screen" described in the present invention, thereby meeting the functions of the present invention.

[0029] Preferably, the parameters of the depth camera 3 should meet the following: the maximum measurement distance is greater than 8 meters; the minimum measurement distance is less than 2 meters;

[0030] Preferably, the detection principle of the depth camera 3 should select the time-of-flight ranging method (TOF) to avoid the delay caused by the operation of other ranging methods.

[0031] Step 1:

[0032] As Figure 1 and Figure 2 shown: The camera 2, the depth camera 3 and the motion tracking device 4 are physically connected by hardware, and their orientations are the same. The shooting direction of the camera 2, the shooting direction of the depth camera 3, and the motion tracking axis of the motion tracking device 4 are all guaranteed to be the same. To reduce the difference in the content of the pictures taken by the camera 2 and the depth camera 3, the installation distance between the optical center of the lens of the camera 2 and the optical center of the depth camera 3 should be minimized as much as possible. In the embodiment, the vertical distance between the optical center of the lens of the camera 2 and the optical center of the depth camera 3 is 110 mm, and the offset values in the front-back direction and the left-right direction are 0.

[0033] In addition, the relative position between the motion tracking device 4 and the camera 2 should be accurately measured. In the embodiment, the motion tracking device 4 is 72 mm directly above the optical center of the lens of the camera 2, and the offset values in the front-back direction and the left-right direction are 0. Therefore, a value of 72 mm in the vertical direction is input into the motion tracking software for compensation of position and attitude calculation.

[0034] Before starting to use, the position of the motion tracking device 4 should be calibrated to ensure its accurate position in space. The position of the LED background wall in space is input into the computer 5. During the operation of the device, the computer 5 calculates the distance between the motion tracking device 4 and the LED background wall 1 to obtain the minimum distance between the motion tracking device 4 and the LED background wall 1 in real time.

[0035] Step 2:

[0036] Use the depth camera to photograph the subject and the LED background wall, and obtain the planar depth map of the subject in the direction of the camera, providing a depth map for subsequent extraction of the green / blue screen contour in this direction;

[0037] Step 3:

[0038] The minimum distance obtained above, according to the actual situation of this embodiment, the depth threshold should be obtained by subtracting 0.8 meters. Use this depth threshold to binarize the depth map, that is, the picture area greater than this nearest distance is black, and vice versa is white. Record and store the processed green / blue screen contour for later use in keying production.

[0039] Step 4:

[0040] To facilitate later chroma keying, in the camera picture, the contour of the green / blue screen should completely surround the subject to be photographed, and there should be a certain width of expansion outside the subject to be photographed. In addition, when shooting a picture of the subject to be photographed moving quickly, due to the delay of the entire system, this expansion width should be larger. On the premise of ensuring that the green screen surrounds the subject to be photographed, this expansion width should be minimized as much as possible. Therefore, the binarized contour of the subject to be photographed that has been obtained should be expanded. Expansion means expanding a certain number of widths outward from the existing contour. This expansion width should be 7% of the width of the area covered by the camera's field of view. When shooting a moving shot, this expansion width should be adjusted according to the actual situation to 20% of the width of the area covered by the camera's field of view to generate an expanded binarized depth map, that is, the available green / blue screen contour.

[0041] Step 5:

[0042] Use the generated green / blue screen contour to color the area of the subject to be photographed into green or blue with appropriate brightness according to the actual situation, and other areas are transparent. Adjust the green / blue screen contour to a matching size according to the field of view angle of the depth camera 3 and merge it with the rendered three-dimensional scene picture to obtain an inner visual cone picture with a variable green / blue screen;

[0043] Step 6:

[0044] Use the obtained inner visual cone picture with a variable green / blue screen to perform off-axis projection according to the relative position of the camera 2 and the LED background wall 1, and display this picture on the LED background wall;

[0045] Step 7:

[0046] Use the camera 2 to shoot the picture of the LED background wall 1 and the subject to be photographed 6 to obtain a camera picture for later post-processing.

[0047] Step 8:

[0048] After the above shooting, the corresponding post-production keying method includes the following steps, as Figure 3 And Figure 4 Shown:

[0049] Two sets of pictures obtained during the pre-shooting are: green / blue screen contour picture, camera picture. Use the two sets of pictures to perform the following steps to complete the post-production keying.

[0050] Perform dilation processing on the green / blue screen contour image obtained from the previous shooting. The dilation width should be slightly smaller than that in step 4 during the previous shooting. In the embodiment, it is 4% of the width of the area covered by the camera's field of view during the previous shooting. When shooting a moving shot, this value should be 15%. Finally, obtain a mask after the contour is dilated.

[0051] Use the chroma key method to perform keying processing on the camera image to obtain a mask after the camera image is keyed.

[0052] Add the dilated mask and the mask after the camera image is keyed in step 8.2 to obtain a combined mask.

[0053] Use the obtained combined mask to perform keying processing on the camera image to obtain a camera image with an alpha channel.

[0054] Composite the rendered late-stage 3D virtual image and the camera image with an alpha channel to obtain the final image and complete the late-stage production.

[0055] In the embodiment, since the area of the green screen displayed on the LED background wall in the background is greatly reduced, the reflected and transmitted objects in the shooting scene, such as glasses, helmets, and the paint of vehicles, reflect or transmit less green screen, making the image more natural. At the same time, the steps of dealing with the green screen spill of reflected or transmitted objects in the late-stage production are reduced, greatly improving the production efficiency and quality.

[0056] The above specific description further details the purpose, technical solution, and beneficial effects of the invention. It should be understood that the above is only a specific embodiment of the present invention and is not used to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A real-time variable green screen generation method and keying method based on depth information, characterized in that, It includes the following steps: Step 1: Connect the camera, depth camera and motion tracking device, make the orientations of the three consistent and control the distances between them, and perform compensation for position and attitude calculation according to the measured relative position between the motion tracking device and the camera; Step 2: Obtain a depth map through the depth camera; Step 3: Perform binarization processing on the obtained depth map; The implementation method of Step 3 is: Use the motion tracking device to obtain the relative position of the camera, calculate the closest distance between the camera and the LED background wall in real time, calculate the threshold based on this distance and perform binarization processing on the depth map, that is, the picture area greater than the threshold is black, otherwise it is white; record and store the processed green / blue screen contour for later use in keying production; Step 4: Perform dilation processing on the binarized subject contour to generate a dilated binarized depth map, which is the available green / blue screen contour; Step 5: Use the green / blue screen contour generated in Step 4 to color the subject area with green or blue of appropriate brightness, and the other areas are transparent. Adjust the green / blue screen contour to a matching size according to the field of view angle of the depth camera and merge it with the rendered three-dimensional scene picture to obtain an inner view cone picture with a variable green / blue screen; Step 6: Use the inner view cone picture with a variable green / blue screen obtained in Step 5 to perform off-axis projection according to the relative position between the camera and the LED background wall, and display the picture on the LED background wall; Step 7: Use the camera to shoot the picture of the LED background wall and the subject to obtain a camera picture for later post-processing; Step 8: Post-production; The implementation method of Step 8 is: Step 8.1: Perform dilation processing on the green / blue screen contour picture obtained in the previous shooting. The dilation width should be slightly smaller than the dilation width in Step 4 during the previous shooting to obtain a mask after contour dilation; Step 8.2: Use the chroma key method to perform keying processing on the camera picture to obtain a mask after keying the camera picture; Step 8.3: Add the dilated mask in Step 8.1 and the mask after keying the camera picture in Step 8.2 to obtain a merged mask; Step 8.4: Use the merged mask obtained in Step 8.3 to perform keying processing on the camera picture to obtain a camera picture with an alpha channel; Step 8.5: Synthesize the rendered later three-dimensional virtual picture and the camera picture with an alpha channel in Step 8.4 to obtain the final picture and complete the post-production.

Images