Multi-format keying

Abstract

A method for post-processing a video, comprising receiving an input video stream, detecting both a luma key area and a chroma key area in a portion of the video stream, and replacing the detected areas with areas from a replacement image to construct an output video stream.

Description

Technical Field

[0001] The present invention relates to the capture of video and still images.

Background Art

[0002] There may be a need to replace a portion of the captured images so that they look different. For example, when a video of an actor is captured in a studio, it may be desirable to replace the studio depicted in the video with an outdoor landscape or a fantasy scene.

[0003] One way to do this is to use chroma keying. In chroma keying, the subject, e.g., the actor, is recorded against a background of a specific color. When the video is captured, the video is post-processed so that the regions of the specific color are replaced with the desired replacement image. Typically, the color is green, but other colors can also be used. One drawback of chroma keying is that the subject must not wear the regions of that specific color; if they do, the regions on the subject will also be replaced in post-processing.

[0004] As an alternative to chroma keying, there is luma keying. In luma keying, the background is distinguished from the subject by brightness rather than color. The subject can be photographed against a dark background. If the subject is sufficiently illuminated, it can be expected that the subject will appear brighter than the background. And in post-processing, the dark regions of the video can be replaced with the desired replacement image. One drawback of luma keying is that the non-reflective material that is expected to function best as the background for luma keying has a fine structure that is vulnerable to mechanical damage.

[0005] As another option, a display wall can be used behind the subject. While the video is being captured, the desired background is displayed on the display wall. In this case, there is no need to perform replacement in post-processing. However, the display wall is relatively expensive.

[0006] There is a need for an improved approach for capturing videos in a desired background. SUMMARY OF THE INVENTION

[0007] According to one aspect, a method for post-processing a video is provided, the method including receiving an input video stream, detecting both a luma key area and a chroma key area in a portion of the video stream, and replacing the detected areas with areas from a replacement image to form an output video stream. The step of detecting the luma key area can include detecting a spatial area having a brightness lower than a predetermined brightness. The step of detecting the chroma key area can include detecting an area of a predetermined hue.

[0008] The method includes replacing both the luma key area and the chroma key area in one step. That is, the method can include forming a replacement area including both the detected luma key area and the detected chroma key area, and replacing the input video with the replacement image in the replacement area.

[0009] The replacement image can be a still image or a video.

[0010] The input video stream can include a plurality of frames. The method can include continuously for each frame, detecting both the luma key area and the chroma key area in the frame, and replacing each detected area with an area from the replacement image to form a frame of the output video stream.

[0011] According to another aspect, a computer configured to execute the above-described method is provided.

[0012] According to another aspect, a data store (data storage device) storing code executable by a processor to execute the above-described method is provided.

[0013] According to another aspect, a studio set is provided that includes a floor and a wall behind and above the floor. The wall exposes an extended surface having a reflectivity in the visible spectrum lower than 10%, and the floor exposes an extended surface of a uniform predetermined hue.

[0014] The floor may extend as a horizontal component. The surface of the predetermined hue covers all or part of the floor. Other parts of the floor may optionally be covered by a surface having a relatively low reflectivity.

[0015] The wall may extend as a vertical component. The surface having a relatively low reflectivity can cover all or part of the wall.

[0016] Each extended surface may have a continuous area exceeding 5m 2 .

[0017] The predetermined hue may be blue or green. The surface having a relatively low reflectivity may exhibit black.

[0018] According to another aspect, a method for capturing a video or a still image is provided. This method includes, as described above, directing a camera at the above-described set with one or more subjects on the floor, and the subjects are gathered in front of the portions of the extended surfaces of the wall and the floor as viewed from the perspective of the camera.

[0019] The method includes, after the directing step, post-processing the video or image captured by the camera, and in this post-processing, (a) identifying relatively low-brightness regions in the video or image and replacing the regions with regions of a replacement image, and (b) identifying regions of a predetermined hue in the video or image and replacing the regions with regions of a replacement image.

[0020] According to another aspect, a method for post-processing a video is provided. This method includes performing both luma keying replacement and chroma keying replacement on frames of the video. Both luma keying replacement and chroma keying replacement can be performed on a plurality of frames of the video. The method may include receiving, from a user, an indication of a first spatial region of the video and a second spatial region of the video, preferentially performing luma keying replacement in the first spatial region, and preferentially performing chroma keying replacement in the second spatial region.

[0021] The post-processing is performed by a computer.

[0022] The post-processing can be performed in real time. That is, immediately when the input video is captured, for example, within 1 second of the capture of the input video.

Brief Description of the Drawings

[0023]

Figure 1

Figure 2

Figure 3

Embodiments for Carrying Out the Invention

[0024] Hereinafter, the present invention will be described by way of examples with reference to the drawings.

[0025] FIG. 1 shows a set for capturing a video or a still image. This set is a movie set, a TV studio, a photo studio, or other suitable environment. The examples described below are described with respect to the capture of a video, but the principle is equally applicable to still photos and individual frames of a captured video.

[0026] The subjects 1, 2 are arranged in a set. In this example, the subject 1 is a table and the subject 2 is an actor. The subject can also be other suitable living or non-living things. Usually, the subject is intended to appear in the foreground of the final output video. The subject is placed on the floor 3. The floor may be flat or not flat. The floor extends as a horizontal component, but does not necessarily have to be strictly horizontal. The wall 4 extends upward behind the subject. The wall may be flat or not flat. In the example of FIG. 1, the wall is curved around the vertical axis, and the concave side of the wall faces the subject. The wall extends as a vertical component, but does not necessarily have to be strictly in the vertical direction. The wall abuts against the floor as shown at 5, but there may be a gap between the two.

[0027] The camera 6 is provided to capture the video of the subject against the background provided by the floor 3 and the wall 4. The video captured by the camera is stored in the data store 7.

[0028] FIG. 2 shows a set of post-processing devices for performing post-processing on the video. The set of post-processing devices includes a data store 10. In practice, this data store 10 can be the same as the data store 7. The set of devices also includes a post-processing computer (PPC) 11 and a user terminal 12. The computer includes a processor 13 and a memory 14. The memory stores code in a non-transitory form that is executable by the processor 13. When the processor executes the code, the PPC 11 provides the functions described herein.

[0029] The PPC can perform chroma keying and luma keying. To perform chroma keying, the PPC processes the input video stream stored in the store 10 and replaces a specific portion of the video stream. The output video stream having the resulting replacement portion is stored in the store 10 or output via the data link 15 as the output video stream. The portion to be replaced is determined as follows. The user of the terminal 12 can specify a chroma key color. Alternatively, the PPC may select a default chroma key color, such as green. The user can specify a segment, which is a subset of the input video stream delimited by time, for example, the segment from 1 minute to 2 minutes from the beginning of the stream. Alternatively, the PPC may select a default segment, such as the entire input video stream. The user may specify a position area within the video where chroma key post-processing should be limited, or chroma key post-processing may be performed over the entire area of the video. Further, the user can specify the still image content or video content to be used to replace a part of the video stream. The replacement content can be a previously captured video or a computer-generated image (CGI). The user can specify the conversion between the replacement content and the input video stream. This conversion can adjust any one or more of the registration offset, relative orientation, color shift, luminance shift, and spatial distortion between the replacement content and the input video. The purpose of this conversion is generally to adapt the replacement content to visually match the input video stream. The PPC processes the specified one or more segments and areas of the input stream to detect where the chroma key color appears. The PPC replaces that location with a selected portion of the replacement content to compose the output video stream.

[0030] To perform luma keying, the PPC processes the input video stream stored in store 10 to replace a specific portion of the video stream. The output video stream with the resulting replacement portion is either stored in store 10 or output via data link 15 as the output video stream. The portion to be replaced is determined as follows. The user of terminal 12 can specify a luma key threshold brightness. Alternatively, the PPC may select a default luma key threshold brightness. The user can specify a segment, which is a subset of the input video stream delimited by time, for example, the segment from 1 minute to 2 minutes from the start of the stream. Alternatively, the PPC may select a default segment, such as the entire input video stream. The user may specify a position area within the video where luma key post-processing should be limited, or luma key post-processing may be performed on the entire area of the video. Further, the user can specify the still image content or video content to be used to replace a portion of the video stream. The replacement content can be a previously captured video or a computer-generated image (CGI). The user can specify the conversion between the replacement content and the input video stream. This conversion can adjust any one or more of the registration offset, relative orientation, color shift, luminance shift, and spatial distortion between the replacement content and the input video. The purpose of this conversion is generally to adapt the replacement content to visually match the input video stream. The PPC processes the specified one or more segments and areas of the input stream to detect locations where the brightness is lower than the threshold brightness. The PPC replaces those locations with a selected portion of the replacement content to construct the output video stream.

[0031] The PPC can apply a threshold around the specified chroma key hue to increase the likelihood of successful detection of all chroma key areas.

[0032] PPC can perform chroma keying and luma keying on one input video. Usually, this process will involve performing both types of keying on one time segment of the input video. The spatial region of the video on which the post-processing is performed can be the same, or different spatial parts of the input video can be specified for chroma key post-processing and luma key post-processing.

[0033] In this system, each background for different types of keying is used for the floor 3 and the wall 4. In one arrangement, the floor 3 bears the chroma key background and the wall 4 bears the luma key background.

[0034] The chroma key background is a background of the same color. Most commonly, the color is green or blue, but it can be another color. Preferably, it is a color having a hue such as not black, white, or gray. Such a color makes it easier to avoid the chroma key color existing in the subject. The chroma key background can be achieved by using a carpet or mat of the chroma key color or by coloring the underlying surface with the chroma key color. Commercially available chroma key blue and green paints are available.

[0035] A lumakey background is a background having a relatively low reflectivity. For example, it has a reflectivity of 10% or less, more preferably 8% or less, more preferably 5% or less, more preferably 2% or less, more preferably 1% or less, more preferably 0.5% or less, more preferably 0.1% or less, more preferably 0.01% or less, etc. Such a background should normally appear black to an observer. Such blackness can be provided by a fabric with a sufficiently low reflectivity or by coloring a surface extending thereunder with an appropriate paint. This is either an ordinary black paint or a paint specially adapted to have an especially low reflectivity, for example, by incorporating surface-modifying particles. The background may be provided by a pre-manufactured sheet with a low reflectivity. Examples of materials with an especially low reflectivity are disclosed in "Breakdown of Native Oxide Enables Multifunctional, Free-Form Carbon Nanotube-Metal Hierarchical Architectures" (Cui and Wardle), ACS Appl. Mater. Interfaces 2019, 11, 38, 35212-35220 and EP 3 676 022.

[0036] In the case of a simple arrangement, at least within the field of view of camera 6, the chromakey background is used for the entire floor and the lumakey background is used for the entire wall behind the subject. In the case of another arrangement, the chromakey background is used only for the part of the floor on which the subject is placed, and the lumakey background is used for other parts. An example is shown in FIG. 3. In FIG. 3a, the lumakey background 20 (cross-hatching) is applied to the entire set wall 4 but not to anywhere on the set floor 3, while the chromakey background 21 (dot pattern) is applied to the entire set floor 3 but not to anywhere on the set wall 4. In FIG. 3b, the lumakey background 20 is applied to the entire set wall 4 and also to a part of the set floor 3, and the chromakey background 21 is applied only to a part of the set floor 3. In the case of the arrangement in FIG. 3b, the area of the floor to which the chromakey background is applied is preferably at the horizontal center of the set. Preferably, it may include the part of the floor adjacent to the wall.

[0037] The bed and the wall should preferably be in contact with each other at a curved boundary portion rather than at a sharp angle. The boundary between the bed and the wall can be considered to be at a location where the angle of the camera-facing surface with respect to the vertical is around 45 degrees.

[0038] When a video is captured in the background as described above, the PPC can perform chroma keying on certain parts of the captured video and luma keying on other parts. The PPC can automatically detect the luma key background from its low brightness and the chroma key background from its hue. To assist in the accuracy of the detection, the user of the terminal 12 can specify the spatial region of the video that incorporates the luma key background and the chroma key background.

[0039] With the above configuration, several advantages are provided.

[0040] First, the ultra-black material has a sensitive surface microstructure that is easily damaged by mechanical contact and abrasion. Ordinary black paint is also prone to abrasion and can exhibit gloss. By avoiding the use of the luma key material on the bed or at least on the portion where the subject of the bed is placed, the possibility of damaging the luma key background can be reduced. By maintaining chroma keying in the remaining regions, one type or the other type of keying can still be performed throughout the video frame.

[0041] Furthermore, the above-described configuration can utilize visual considerations that typically act on the upper and lower portions of different videos. Typically, in the upper region of a video above waist height or, for example, above 1 m from the ground, the presence of shadows is generally not as important for the sense of realism, but in this upper region, there are often the faces and costumes of actors. When a chroma key background is used in this region, for example, reflections from the chroma key background resulting from commonly used strong studio lights can project unwanted color tones onto the faces and clothing of actors. This is generally less of a problem in the lower region of the video frame because it is not normal for there to be faces or no light costumes but light costumes. By using a relatively non-reflective background and / or an achromatic (black / grey) background for the upper region background, the above-described reflection effects can be reduced. Furthermore, some objects commonly seen in the upper region of a video are better identified for keying against a lumakey background than against a chroma key background. An example of this is thin lines of blonde hair. On the other hand, in the lower part of the video, it is usually desirable to have shadows in order to enhance the sense of depth and give an illumination feeling from above. When a chroma key background is used on the floor, the subject can cast a shadow on the floor, and the brightness of the chroma key color can be changed so as to be detected by the PPC. When replacing with a replacement image, the PPC can automatically apply a shadow effect to regions of low brightness. In addition, since the chroma key color is typically a bright color, the possibility of such a color existing in objects in the lower part of the scene rather than in the higher parts of the scene is generally low. Objects located below are typically items such as trousers, shoes, or chairs, which are often of subdued colors. Therefore, the possibility of the chroma key color appearing in the foreground objects is lower in the lower part of the scene.

[0042] The studio set can be indoors. The set can be equipped with filming equipment such as one or more cameras, one or more lights, one or more video prompters, and any of one or more diffusers.

[0043] Regions of relatively low reflectivity can have a low reflectivity relative to the chroma key region and / or relative to an absolute reference value such as 10%. The reflectivity can be measured as an average over the entire visible spectrum. The visible spectrum can be considered to consist of wavelengths from 380 to 700 nm.

[0044] The region of relatively low reflectivity may be a region that substantially includes the entire wall area. That is, it can be an extended surface. 5 m 2 or 10 m 2 and may have a continuous area of the above.

[0045] The chroma key region may be a region that substantially includes the entire floor area. That is, it can be an extended surface. 5 m 2 or 10 m 2 and may have a continuous area of the above. The chroma key region can be blue corresponding to wavelengths in the range of, for example, 450 to 495 nm. The chroma key region can be green corresponding to wavelengths in the range of, for example, 495 to 570 nm.

[0046] When a video is captured in a different background as described above, it is passed to the PPC for post - processing. The video can be composed of frames. Each frame represents a portion of the video stream per unit time. The PPC performs replacement based on both chroma keying and luma keying when passing through the entire video once. Its output is an output video in which both the luma key area (with relatively low brightness) and the chroma key area (with a predetermined hue) are replaced. The PPC can be configured to operate on a frame - by - frame basis for the input video. In each frame, the PPC determines areas that match a predetermined criterion for the luma key area or satisfy a predetermined criterion for the chroma key area, and replaces those areas with the corresponding parts of the replacement video. Then, the PPC operates to process the next frame. In this way, the PPC can suitably process the input video in real - time. By identifying the chroma key area and luma key area of the video, it can be executed in a single replacement step. This results in faster processing and better matching between the replacement areas compared to when two separate replacement steps were executed.

[0047] The post - processing as described above can be performed immediately after the input video is captured. The post - processing can be performed on a portion of the captured / input video immediately after capture and during the capture of further video of the same stream.

[0048] The applicant of the present application discloses each of the individual features described in this specification and any combination of two or more such features, without limitation to the scope of the claims, to the extent that they can be implemented based on this specification in light of the common general knowledge of those skilled in the art, regardless of whether the feature or combination of features solves any of the problems disclosed in this specification. The applicant of the present application shows that aspects of the present invention can consist of any such individual features or combinations of features. In view of the above description, it should be apparent to those skilled in the art that various modifications can be made within the scope of the present invention.

[0049] The expressions “configured to” or “arranged to” that follow terms defining conditions or functions are used herein to indicate that the subject of the expression is in a state where it has the condition or can perform the function without being changed or further configured.

Claims

1. A method for post-processing a video, comprising: receiving an input video stream; detecting both a luma key area and a chroma key area in a portion of the video stream; replacing the detected areas with areas from a replacement image to form an output video stream.

2. The method according to claim 1, comprising replacing both the luma key area and the chroma key area in one step.

3. The input video stream includes a plurality of frames, and the method successively for each of the frames detects both the luma key area and the chroma key area in the frame; in the frame, replaces the detected areas with areas from the replacement image to form a frame of the output video stream.

4. A studio set including a floor and a wall behind and above the floor, wherein the wall exposes an extended surface having a reflectance in the visible spectrum lower than 10%, and the floor exposes an extended surface having a uniform predetermined hue.

5. The studio set according to claim 4, wherein the floor extends as a horizontal component.

6. The studio set according to claim 4 or 5, wherein the wall extends as a vertical component.

7. Each of the extended surfaces has a continuous area wider than 5 m ２ The studio set according to any one of claims 4 to 6, having a continuous area wider than ２ .

8. The studio set according to any one of claims 4 to 7, wherein the predetermined hue is blue or green.

9. A method for capturing a video or a still image, comprising: pointing a camera at a set according to any one of claims 4 to 8 and having one or more subjects on the floor, wherein the subjects are gathered in front of portions of the extended surfaces of the wall and the floor as viewed from the perspective of the camera.

10. After the pointing step, post-processing the video or still image captured by the camera, (a) identifying relatively low-luminance areas in the video or still image and replacing the areas with areas from a replacement image, and (b) identifying areas of the predetermined hue in the video or still image and replacing the areas with areas from the replacement image.

Images