Information processing device, information processing method, and computer-readable non-transitory storage medium

Abstract

This information processing device includes a BRDF acquiring unit, a virtual environment map acquiring unit, and a relighting processing unit. The BRDF acquiring unit estimates the BRDF of a subject on the basis of a captured video of the subject. The virtual environment map acquiring unit estimates an environment map that will be obtained when the background scene of the subject is changed to a replacement scene, and acquires the estimated environment map as a virtual environment map. The relighting processing unit generates, on the basis of the BRDF of the subject, a relit video of the subject lit in an illumination environment represented by the virtual environment map.

Description

Information processing device, information processing method, and computer-readable non-transitory storage medium

[0001] The present invention relates to an information processing device, an information processing method, and a computer-readable non-transitory storage medium.

[0002] Virtual production (VP) systems using LED walls are attracting attention. VP is a video production technique that combines live-action footage with CG images in real time. LED walls project background images generated using CG onto a large screen made up of LED panels. The background images are filmed together with real subjects. The background images change in accordance with the movement of the cinema camera, and a fully composited package can be obtained simply by filming. Compared to methods that use green screens, etc., because the background images can be viewed during filming, it is easier for actors and directors to visualize the finished product.

[0003] JP 2003-296759 A JP 2008-102961 A

[0004] The subject is lit by an LED wall. In video production, the background image is often changed after shooting, so assetizing the footage (reusing it in another work with a different background image) is also important for reducing filming costs. In VP, the image of the subject in the foreground is captured live. Therefore, the lighting used during filming is burned into the foreground, and changing the background image can result in a mismatch between the foreground and background lighting.

[0005] Therefore, the present disclosure proposes an information processing device, an information processing method, and a computer-readable non-transitory storage medium that prevent mismatches in lighting between the foreground and background even when the background video is changed.

[0006] According to the present disclosure, there is provided an information processing device including: a BRDF acquisition unit that estimates a BRDF of a subject based on a captured video of the subject; a virtual environment map acquisition unit that estimates an environment map obtained when a background scene of the subject is changed to a replacement scene and acquires the estimated environment map as a virtual environment map; and a relighting processing unit that generates a relighted video of the subject lit in a lighting environment represented by the virtual environment map based on the BRDF of the subject. The present disclosure also provides an information processing method in which information processing of the information processing device is executed by a computer, and a computer-readable non-transitory storage medium that stores a program for causing a computer to realize the information processing of the information processing device.

[0007] FIG. 1 is a diagram illustrating an overview of a VP system. FIG. 2 is a diagram illustrating an overview of a VP system. FIG. 3 is a diagram illustrating a problem related to lighting of a subject. FIG. 4 is a diagram illustrating an overview of a method of the present disclosure. FIG. 5 is a diagram illustrating an example of an environment map. FIG. 6 is a diagram illustrating a method of estimating a BRDF. FIG. 7 is a diagram illustrating a method of estimating a BRDF. FIG. 8 is a diagram illustrating a method of estimating a BRDF. FIG. 9 is a diagram illustrating a relighting process. FIG. 10 is a diagram illustrating an example of a configuration of a VP system of the present disclosure. FIG. 11 is a diagram illustrating an example of a method of acquiring a geometric transformation function. FIG. 12 is a diagram illustrating an example of a method of generating a composite environment map. FIG. 13 is a diagram illustrating an example of brightness calibration. FIG. 14 is a diagram illustrating an example of a method of acquiring a virtual environment map. FIG. 15 is a diagram illustrating an example of a processing flow for advance preparation before VP shooting. FIG. 16 is a diagram illustrating an example of a processing flow during VP shooting. FIG. 17 is a diagram illustrating an example of a hardware configuration of an information processing device.

[0008] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and redundant description will be omitted.

[0009] The description will be given in the following order: [1. Background] [1-1. VP System] [1-2. Problems Related to Lighting of a Subject] [1-3. Overview of the Technique of the Present Disclosure] [2. Configuration of the VP System of the Present Disclosure] [3. Information Processing Method] [4. Example of Hardware Configuration] [5. Effects]

[0010] [1. Background] [1-1. VP System] FIGS. 1 and 2 are diagrams for explaining an overview of the VP system CS.

[0011] VP System CS is LED wall WD and cinema camera CM Ｒ This is a system that synthesizes CG images and live-action images in real time using the LED Wall WD. The LED Wall WD is a large display that combines multiple LED panels. The background CG image (background image I) Ｂ ) is displayed on the screen SCR of the LED wall WD.

[0012] Cinema camera commercial Ｒ The movement of the camera is detected by a tracker. The tracker is an infrared sensor installed on the ceiling of the studio or a cinema camera. Ｒ Cinema camera commercial using IMU etc. Ｒ The tracker tracks the position of the camera in the studio in real time. Ｒ In addition to the position of Cinema Camera CM Ｒ Camera parameters such as orientation, focus, and lens distortion are also recorded as tracking information.

[0013] The tracking information is transmitted to a rendering unit (for example, a rendering PC). The rendering unit performs the tracking of the CG camera CM in the 3DCG space. Ｖ A real-world cinema camera commercial Ｒ The 3DCG space refers to the three-dimensional virtual space that serves as the background for the subject SB. Ｖ is a virtual camera installed in the 3D CG space. Ｖ The shooting process is done by a CG camera. Ｖ This is performed as a rendering process of 3DCG data using the camera parameters of CG camera CM. Ｖ The 3DCG space footage taken in the background video I Ｂ is obtained as

[0014] CG camera commercial Ｖ The position and camera parameters of the cinema camera CM ＲIt is controlled based on the tracking information. Ｖ is a real-world cinema camera commercial. Ｒ When the position and camera parameters of the background image I change, the background image I moves in the same relative direction. Ｂ is a cinema camera commercial Ｒ The background image changes in real time according to the movement of the Ｂ The distortion correction is performed on the background image I after the distortion correction according to the shape of the screen SCR. Ｂ is the displayed image L Ｂ is displayed on the screen SCR.

[0015] Display image L Ｂ is shot with the actual subject SB in the foreground. Cinema camera commercial Ｒ Display image L visible from Ｂ is a CG camera commercial Ｖ 3D CG background image I Ｂ Therefore, the cinema camera commercial Ｒ Just by shooting, you can create a realistic background image of the subject and a CG background image. Ｂ A complete package is obtained in which the above is properly combined.

[0016] [1-2. Problems Related to Lighting of Subject] FIG. 3 is a diagram for explaining problems related to lighting of subject SB.

[0017] In the example of Fig. 3, a real car is placed as the subject SB in the foreground. A daytime landscape is displayed on the screen SCR of the LED wall WD as the background of the subject SB. The subject SB is displayed in the display image L of the screen SCR. Ｂ The scenery displayed on the screen SCR is reflected on the surface of the subject SB facing the screen.

[0018] In video production, the background may be changed after shooting. In this case, the background of the shot image can be changed by post-processing. Ｃ In the example of FIG. 3, the background image I of the replacement scene is changed to another background image I of the replacement scene. Ｂ The sunset scene is used as the background image.Ｃ From the above, the image area of ​​the subject SB is the subject area I ＳＢ The object region I is extracted as ＳＢ Background footage of the replaced scene I Ｂ ' is synthesized into the output composite image I Ｏ is generated.

[0019] Subject area I ＳＢ The image is lit by a bright image showing a daytime scene. The background after the replacement is a dimly lit scene at dusk, so the lighting is very different from when the image was taken. ＳＢ Since the lighting effects are those at the time of shooting, if the background is significantly different from that at the time of shooting, there will be a mismatch in the colors and lighting movement between the foreground and background.

[0020] The following document describes a technique for changing background images as post-processing. However, this technique replaces a low-resolution background image with a high-resolution one, and the lighting conditions remain unchanged before and after the replacement. Therefore, the above-mentioned problem cannot be solved even if the technique described in the following document is used.

[0021] [Literature] <URL: http: / / stage. mo-sys. com / wp-content / uploads / 2021 / 08 / NearTime-Product-Brochure. pdf> (searched on April 28, 2023)

[0022] Furthermore, due to the installation constraints of the LED Wall WD, it is difficult to arrange the LED Wall WD so that it completely surrounds the subject SB. Lighting is only provided from the LED Wall WD side, and lighting cannot be provided from the photographer's side (the opposite side of the subject SB from the LED Wall WD). The studio set on the photographer's side may be reflected in the subject SB, resulting in an unnatural image. The methods described in the above documents cannot address these problems.

[0023] [1-3. Overview of the Technique of the Present Disclosure] The present disclosure has been made in consideration of the above-mentioned problems. ＣThe reflection characteristics (BRDF: Bidirectional Reflectance Distribution Function) of the subject SB are estimated from the image, and the subject SB is relighted based on the replacement scene. The relighted image of the subject SB is then transferred to the background image I. Ｂ ' and output the composite image I Ｏ FIG. 4 is a diagram illustrating an overview of the technique of the present disclosure.

[0024] In order to relight the subject SB, it is necessary to remove the lighting from the surrounding environment that was burned in during shooting. Therefore, first, the BRDF, which corresponds to the reflectance of the subject SB, is estimated. BRDF means the angular distribution characteristics of reflected light when light is incident from a certain angle. Using the BRDF, it is possible to determine the color appearance (reflected light) under any lighting environment. This process allows the replacement background image I to be obtained. Ｂ ' is possible. To estimate the BRDF, the captured image I Ｃ In addition, Environment Map I Ｅ , Normal Map I Ｎ , and data such as coordinate information are required.

[0025] First, VP shooting (Cinema camera commercial) Ｒ As a preliminary step before the actual shooting using the environment map I Ｅ The acquisition work of the environment map I is carried out. Ｅ means an image showing the lighting environment at the position to be measured. Ｅ is acquired as an image of the entire surroundings (panoramic image) seen from the position to be measured. Ｅ As the format of the omnidirectional camera CM, a cube map, a spherical map, etc. are used. Ｓ The spherical map taken with Ｅ It is used as.

[0026] Spherical camera commercial Ｓ is installed at the position (subject position) where the subject SB should be installed in VP photography. ＳIt captures the entire surroundings as seen from the subject's position. Ｓ The footage is used in Environment Map I. Ｅ In the process of acquiring the environmental map, the spherical camera CM Ｓ The coordinate information of the installation position is also acquired as advance information.

[0027] FIG. 5 shows the environment map I. Ｅ In the example of FIG. 5, an omnidirectional image centered on a certain point in the room is used as an environment map I. Ｅ The width of the environment map indicates a range of 360 degrees, and the height indicates a range of 180 degrees. Ｅ uses a special mapping to map points in three-dimensional space to points on a two-dimensional plane in order to fit the entire surrounding scene into a two-dimensional image.

[0028] Cinema camera commercials for VP shooting Ｒ The subject SB is displayed on the LED wall WD. Ｂ The subject SB is photographed together with the display image L Ｂ The LED wall is lit from the WD side. Ｃ The display image L Ｂ The subject SB is lit by the VP shooting. Ｒ The coordinate information is also acquired as photographing information.

[0029] Footage I Ｃ Subject area I ＳＢ By signal processing, the normal map I Ｎ is obtained. Normal map I Ｎ is the object area I ＳＢ The BRDF of the subject SB is the distribution of the normal vectors in the captured image I. Ｃ ,Environment Map I Ｅ , Normal Map I Ｎ , spherical camera commercial Ｓ Coordinate information and cinema camera commercial Ｒ It is estimated using the coordinate information of

[0030] FIG. 6 is a diagram illustrating a method for estimating a BRDF.

[0031] Generally, the rendering equation shown in Equation (1) below is used for CG rendering of points on the surface of an object.

[0032]

[0033] In equation (1), "x" indicates a point on the surface of the object. ０ " indicates the direction of the camera viewpoint (viewpoint direction) as seen from point x. ｉ " indicates the direction of the light source as seen from point x (light source direction). "n(x)" indicates the normal to the object at point x. "θ" indicates the relationship between the normal n(x) and the light source direction ω ｉ "Ω" indicates the angle between the tangent plane of point x and the ω ｉ The integral range is shown.

[0034] "L(x, ω ０ ) is the angle from point x to the viewpoint direction ω ０ It indicates the brightness of the reflected light toward the camera (the brightness of the reflected light incident on the camera). Ｅ (x, ω ｉ ) indicates the luminance of the illumination light directed from the light source to point x (luminance of the light incident on point x). The luminance is expressed as a pixel value. For example, "L(x, ω)" ０ ) is the pixel value I ω０ = (R ω０ , G ω０ , B ω０ ) (see FIG. 8). Specifically, "L(x, ω ０ ) is a filmed video I Ｃ The pixel value is obtained as "L Ｅ (x, ω ｉ ) is the environment map I Ｅ The pixel value is obtained as:

[0035] "f ＢＲＤＦ " indicates the BRDF of the subject SB. The color of point x is determined by the sum of the values ​​obtained by multiplying the various light rays entering from all directions by the reflectance (BRDF) according to the angle of the incident light.

[0036] Cinema camera commercial Ｒ The image acquired by L(x, ω ０ ) corresponds to the environment map I Ｅ Is L Ｅ (x, ωｉ ) BRDF can be estimated as a general inverse rendering problem. Inverse rendering is the process of Ｃ ) refers to a technique for estimating any or all of the geometry, lighting, and surface reflectance properties from a normal map I. Ｎ and Environment Map I Ｅ From this information, the BRDF, which corresponds to the surface reflectance characteristics, can be calculated.

[0037] Several approximation models for BRDF are known. BRDF can be expressed by a combination of multiple parameterized basis functions. For example, BRDF can be expressed by a combination of a specular reflection model and a diffuse reflection model. The following formulas (2) to (4) show examples of BRDF approximation models.

[0038]

[0039]

[0040]

[0041] "f" in Equation (2) and Equation (3) ｒ (ω ０ , ω ｉ ")" in Equation (2) and Equation (4) indicates the BRDF expressed by the specular reflection model (specular reflection BRDF). ｄ (ω ０ , ω ｉ )) represents the BRDF (diffuse reflection BRDF) expressed by the diffuse reflection model. In equation (3), "D" represents the normal distribution function. "G" represents the geometric attenuation term. "F" represents the Fresnel term. "h" represents the half vector. "a" represents the diffuse reflectance. "f ０ " indicates the reflectance of incident light from the normal direction.

[0042] Footage I Ｃ ,Environment Map I Ｅ , Normal Map I Ｎ , spherical camera commercial Ｓ Coordinate information and cinema camera commercial ＲBy applying the coordinate information of the above to equations (1) to (4), the optimal parameters D, G, and F can be found. This can be solved as a general optimization problem using a method such as Bayesian optimization. By finding the parameters D, G, and F, the BRDF is obtained.

[0043] The BRDF estimation process can also be performed using equation (5) instead of equation (1). Figures 7 and 8 are diagrams for explaining a method of estimating the BRDF using equation (5).

[0044]

[0045] Equation (5) is obtained by replacing "cos θ" in equation (1) with "max(cos θ, 0)." max(cos θ, 0) is a function that outputs the larger of cos θ and 0. The side opposite to the side where light enters as viewed from the tangent plane of a certain point is occluded, and light does not enter. By max(cos θ, 0), the calculated value of the occluded area is clamped to 0. This reduces the amount of calculation. In FIGS. 7 and 8, the area clamped to 0 by max(cos θ, 0) is referred to as the "clamped area."

[0046] Once the BRDF of the subject SB is acquired, the background image I of the replacement scene is Ｂ The relighting of the subject BRDF is performed based on the above-mentioned ". The relighting is performed as a process for obtaining the color of the subject SB under an arbitrary light source.

[0047] FIG. 9 is a diagram illustrating the relighting process.

[0048] In the relighting process, the rendering equation shown in the following equation (6) is used.

[0049]

[0050] Formula (6) is the same as "L" and "L" in formula (5). Ｅ " to "L ＲＥ " and "L ＣＧ " has been replaced with "L ＣＧ " is a virtual 360-degree view (Virtual Environment Map I) calculated from the 3DCG data of the replaced scene. ＥＳ(See Figure 14). ＲＥ " is Virtual Environment Map I ＥＳ Relighting image I of subject SB lit in the lighting environment (virtual lighting environment) shown in ＲＥ Indicates "f ＢＲＤＦ The hat next to " indicates that the value is an estimate of the BRDF.

[0051] In FIG. Ｅ " is a commercial for a spherical camera Ｓ The actual 360° view (real environment map) captured by the VP is shown. In the preparation before the VP shooting, the real environment map and the actual captured image I are Ｃ In post-processing after VP shooting, the estimated BRDF of the subject SB and the virtual environment map I are used. ＥＳ Based on this, a relighting image I of the subject SB in the virtual lighting environment is obtained. ＲＥ is estimated.

[0052] 2. Configuration of the VP System of the Present Disclosure FIG. 10 is a diagram showing an example of the configuration of the VP system CS of the present disclosure.

[0053] The VP system CS includes an input unit 10, a 3DCG data holding unit 100, and an information processing device PR. The 3DCG data holding unit 100 holds various 3DCG data D ＣＧ The 3DCG data storage unit 100 stores the 3DCG data D in response to reading by the rendering unit 40. ＣＧ is transferred to the rendering unit 40.

[0054] The input unit 10 performs general VP shooting. The shooting studio is equipped with an LED wall WD on which CG rendering images are displayed and a cinema camera CM. Ｒ Cinema camera commercial Ｒ Operation signal S Ｃ is sent to the captured image acquisition unit 20.

[0055] The image processing is performed by an information processing device PR. For example, the information processing device PR includes a captured image acquisition unit 20, a signal processing unit 30, a rendering unit 40, a background CG display unit 50, an environment map acquisition unit 60, a composite environment map acquisition unit 70, a virtual environment map acquisition unit 80, and an output unit 90.

[0056] The captured image acquisition unit 20 receives an operation signal S Ｃ Cinema camera commercial based on Ｒ The stage to be photographed is photographed by operating the . The subject to be photographed is the subject SB in the foreground and the LED wall WD (display image L) in the background. Ｂ The captured image acquisition unit 20 includes the cinema camera CM Ｒ Footage I Ｃ is sent to the signal processing unit 30.

[0057] The captured image acquisition unit 20 is a cinema camera commercial Ｒ Camera parameters P Ｃ The camera parameters P Ｃ is a frame-by-frame cinema camera commercial Ｒ The captured image acquisition unit 20 acquires the camera parameters P Ｃ to the virtual environment map acquisition unit 80. The captured image acquisition unit 20 Ｒ The movement and orientation of the synchronization signal S Ｓ Cinema camera commercial Ｒ is synchronized with the rendering unit 40.

[0058] The rendering unit 40 receives the 3DCG data D from the 3DCG data storage unit 100. ＣＧ The rendering unit 40 reads the CG camera CM Ｖ 3DCG data D based on the viewpoint ＣＧ The rendering unit 40 renders the CG image (rendered image) and sends the generated CG image to the background CG display unit 50.

[0059] The background CG display unit 50 performs distortion correction on the CG image to produce the display image L Ｂ Distortion correction is the process of correcting the distortion of a CG image by using a cinema camera. ＲThe background CG display unit 50 corrects the display image L obtained by the distortion correction to fit the shape of the screen SCR of the LED wall WD. Ｂ is displayed on the screen SCR.

[0060] The rendering unit 40 generates a background image I as a CG image. Ｂ and Test Video I Ｐ (See FIG. 11) Background image I Ｂ means the image that is the background of the subject SB. Ｐ CG images and environment maps Ｅ The background image I is an image used to determine the geometric correspondence (geometric transformation function g: see FIG. 11) between the background image I and the image I. Ｂ This will be used during the VP shoot. Ｐ is used in advance preparation before VP shooting.

[0061] The geometric transformation function g is Ｂ It is used to synthesize an environment map from the background image I. Ｂ The lighting environment of the subject SB changes from moment to moment in accordance with the movement of the subject SB. The BRDF may change depending on the lighting environment. Ｅ Now, background video I Ｂ It is difficult to accurately estimate the BRDF for each object. Ideally, the environment map I is generated simultaneously with the VP image capture. Ｅ However, there are cases where the stage is being filmed with a spherical camera. Ｓ It is difficult to set up a camera because it interferes with filming.

[0062] Therefore, the synthetic environment map acquisition unit 70 uses the geometric transformation function g to generate the environment map I Ｅ Background video I Ｂ As a result, the synthetic environment map acquisition unit 70 can Ｂ Environment map I according to Ｅ Composite Environment Map I Ｅ The geometric transformation function g can be calculated using a general distortion correction model.

[0063] FIG. 11 is a diagram showing an example of a method for obtaining the geometric transformation function g.

[0064] The rendering unit 40 renders the test video I Ｐ The background CG display unit 50 generates a pattern image having a repeating pattern as follows. For example, the pattern image has a grid pattern in which a plurality of vertical lines and a plurality of horizontal lines are arranged at regular intervals. Ｐ Distortion correction is performed on the displayed image L Ｐ The background CG display unit 50 generates the generated display image L Ｐ is displayed on the screen SCR.

[0065] The environmental map acquisition unit 60 is a celestial sphere camera CM Ｓ , the display image L on the screen SCR. Ｐ The environment map acquisition unit 60 captures the screen SCR using the spherical camera CM. Ｓ The footage was then used to create the Environment Map I Ｅ The environment map acquisition unit 60 acquires the map as follows. Ｓ Camera parameters P Ｅ are sent to the signal processing unit 30 and the virtual environment map acquisition unit 80. The camera parameters P Ｅ is a spherical camera commercial Ｓ This includes the position, orientation and focal length of the

[0066] The synthetic environment map acquisition unit 70 acquires the test image I displayed on the screen SCR. Ｐ and Environment Map I Ｅ Test footage I Ｐ For example, the synthetic environment map acquisition unit 70 calculates a geometric transformation function g that indicates a mapping between the test image I and Ｐ The coordinates (x, y) of each grid point included in the environment map I Ｅ The synthesized environment map acquisition unit 70 calculates the geometric transformation function g using the following equation (7).

[0067]

[0068] FIG. 12 shows the composite environment map I. ＥFIG. 10 is a diagram showing an example of a method for generating a '.

[0069] The composite environment map acquisition unit 70 acquires the environment map I Ｅ The screen position is determined by the environment map I Ｅ The synthesis environment map acquisition unit 70 acquires the image (background image I) on the screen SCR that forms the background of the subject SB. Ｂ ) into the environment map I Ｅ Composite environment map I is synthesized into the LED wall area within Ｅ ' is generated.

[0070] Background image I Ｂ The mapping from the points on the LED wall area to the points on the LED wall area is obtained as a geometric transformation function g. The geometric transformation function g is a function of the background image I Ｂ Transform and create an environment map I Ｅ The synthetic environment map acquisition unit 70 converts the background image I transformed by the geometric transformation function g into a distorted image that can be incorporated into the Ｂ Environment Map I Ｅ Composite onto the LED wall area within Environment Map I Ｅ Background video I Ｂ By combining these, background image I Ｂ the environment map according to Ｅ ' is obtained.

[0071] Here, the test image I output from the rendering unit 40 Ｐ The pixel value of the environment map I Ｅ Test footage I Ｐ The pixel value of the LED wall WD is set as the second pixel value. The first pixel value and the second pixel value are determined based on the brightness level of the LED wall WD and the omnidirectional camera CM. Ｓ This is affected by the exposure settings. Therefore, there may be a discrepancy between the first pixel value and the second pixel value. Although it depends on the characteristics and settings of the equipment, it is generally thought that the second pixel value will be lower than the first pixel value.

[0072] Composite Environment Map I Ｅ ' is the environment map I ＥThis is created by superimposing a rendering image onto the image. Therefore, there is a possibility that pixel values ​​with high or low brightness different from the expected value may be used. In this case, problems such as a decrease in the estimation accuracy of the BRDF and saturation of the brightness level of the lighting added during relighting may occur. For example, when shooting a highly glossy metal surface, a cinema camera commercial Ｒ The brightness of the image may be saturated, making it difficult to accurately estimate the BRDF.

[0073] Therefore, in advance preparation before VP shooting, a known rendering image may be displayed and brightness (color) calibration may be performed. Fig. 13 is a diagram showing an example of brightness calibration.

[0074] The rendering unit 40 generates a rendering image for brightness calibration (brightness calibration image I Ｗ For example, the brightness calibration image I Ｗ is a white image. Ｗ All pixels in the image are set to the same pixel value indicating white. Ｗ is displayed.

[0075] The environmental map acquisition unit 60 is a celestial sphere camera CM Ｓ The display image L on the screen SCR is Ｗ The environment map acquisition unit 60 captures the display image L Ｗ Environment Map I Ｅ The composite environment map acquisition unit 70 acquires the brightness calibration image I displayed on the screen SCR. Ｗ The pixel value (first pixel value) of the environment map I Ｅ Brightness calibration image I Ｗ The synthetic environment map acquisition unit 70 calculates the ratio between the pixel value (second pixel value) of the background image I of the subject SB and the pixel value (second pixel value) of the background image I of the subject SB. Ｂ Environment Map I Ｅ The brightness correction coefficient is obtained when combining the two images.

[0076] The following formula (8) is used to calculate the composite environment map I Ｅ ' is the formula for calculating the pixel value of

[0077] In formula (8), "I Ｅ " is Environment Map I Ｅ The pixel value of the pixel above is shown as "g(I Ｗ ) is transformed into the environment map I by the geometric transformation function g. Ｅ Brightness calibration image I associated with the pixel above Ｗ The pixel value of the pixel above. Ｅ ' is the composite environment map I Ｅ ' indicates the pixel value of the pixel on "g(I Ｂ ) is Environment Map I Ｅ Background image I superimposed on the upper pixel Ｂ The brightness correction coefficient is "I Ｅ / g(I Ｗ )"

[0078] If the subject SB with saturated brightness is captured by the camera, a warning may be displayed.

[0079] The signal processing unit 30 receives the captured image I Ｃ For example, the signal processing unit 30 includes a foreground normal acquisition unit 31, a BRDF acquisition unit 32, a relighting processing unit 33, and a foreground / background composition unit .

[0080] The foreground normal acquisition unit 31 acquires the foreground normal from the captured image I Ｃ is analyzed, and the object region I ＳＢ Normal Map I Ｎ Normal Map I Ｎ means a distribution map of normal vectors. For example, in the object area I ＳＢ The data of the normal vectors obtained for each pixel of the normal map I Ｎ The foreground normal acquisition unit 31 acquires the normal map I Ｎ to the BRDF acquisition unit 32.

[0081] Normal Map I ＮThe acquisition of the normal vector can be performed using a general normal estimation process. For example, the classical SfM (Structure from Motion) can be used, or an inference method from RDB (+depth) images using deep learning can be used. If necessary, a depth sensor can be attached to a cinema camera CM. Ｒ may be attached to improve the accuracy of normal estimation.

[0082] The BRDF acquisition unit 32 acquires the captured image I of the subject SB. Ｃ In the BRDF estimation process, the BRDF of the subject SB is estimated based on the synthesized environment map I Ｅ ', Normal Map I Ｎ , Cinema Camera Commercial Ｒ Camera parameters P Ｃ , and spherical camera commercial Ｓ Camera parameters P Ｅ The BRDF acquisition unit 32 acquires the captured image I Ｃ The object region I extracted from ＳＢ Apply inverse rendering to the composite environment map I Ｅ The BRDF acquisition unit 32 estimates a BRDF that is consistent with the estimated value of the BRDF and various parameters (normal map I Ｎ , camera parameters P Ｃ , camera parameters P Ｅ ) is sent to the relighting processing unit 33.

[0083] The virtual environment map acquisition unit 80 acquires the environment map I obtained when the background of the subject SB is changed to a replacement scene. Ｅ The virtual environment map acquisition unit 80 estimates the estimated environment map I Ｅ Virtual environment map I ＥＳ Obtain as.

[0084] FIG. 14 shows the virtual environment map I. ＥＳ FIG. 10 is a diagram illustrating an example of a method for acquiring the above information.

[0085] The virtual environment map acquisition unit 80 acquires the virtual omnidirectional camera CM ＶＳ It will be installed in the 3D CG space of the replacement scene. Virtual spherical camera commercial ＶＳmeans a virtual spherical camera for capturing omnidirectional images in a 3D CG space. ＶＳ The shooting process is done by a virtual spherical camera ＶＳ This is performed as a rendering process of 3DCG data using the camera parameters.

[0086] The rendering process is performed by the rendering unit 40. The rendering process is performed when the virtual environment map acquisition unit 80 sends the operation command S ＥＳ The operation command S is output. ＥＳ Information on the replacement scene and the virtual spherical camera commercial ＶＳ The rendering unit 40 includes information on the camera parameters of the operation command S ＥＳ Generates a panoramic image according to the virtual environment map I ＥＳ Output as

[0087] Virtual spherical camera commercial ＶＳ The camera parameters are the spherical camera CM Ｓ The same camera parameters as those of the virtual spherical camera CM are used. ＶＳ The installation location is the virtual spherical camera commercial. ＶＳ and CG camera commercial Ｖ The positional relationship with the spherical camera commercial Ｓ and Cinema Camera CM Ｒ The installation position is calculated based on the cinema camera CM. Ｒ Camera parameters P Ｃ and spherical camera commercial Ｓ Camera parameters P Ｅ is used.

[0088] The virtual environment map acquisition unit 80 acquires the virtual omnidirectional camera CM ＶＳ The virtual environment map acquisition unit 80 captures an image of the entire 3D CG space using the virtual omnidirectional camera CM. ＶＳ The footage is then transferred to the virtual environment map I. ＥＳ Obtain as.

[0089] When the subject SB is illuminated by the LED wall WD, the illumination range is limited to the installation range of the screen SCR. However, when a virtual environment map is generated using a rendering image, there is no limitation to the illumination range. The virtual environment map acquisition unit 80 uses the 3DCG data of the replacement scene to generate a virtual environment map I that shows an illumination environment in a range wider than the installation range of the screen SCR. ＥＳ can be obtained.

[0090] The relighting processing unit 33 calculates the BRDF and various parameters (normal map I) of the subject SB. Ｎ , camera parameters P Ｃ , camera parameters P Ｅ ) into the rendering equation of Equation (6). The relighting processing unit 33 generates the virtual environment map I based on the BRDF of the subject SB. ＥＳ An image of the subject SB lit in a lighting environment represented by ＲＥ ) to generate the

[0091] The foreground / background composition unit 34 generates a relighted image I of the subject SB. ＲＥ The background image I of the subject SB generated using the 3DCG data of the replacement scene Ｂ This results in a relighted image I of the subject SB. ＲＥ and Background Video I Ｂ ' and composite image I Ｏ The output unit 90 outputs the generated composite image I Ｏ Final Footage I Ｆ The output unit 90 acquires the composite image I as necessary. Ｏ General post-processing (color conversion, format conversion) may be performed on the image.

[0092] 3. Information Processing Method The flow of video processing according to the present disclosure will be described with reference to Fig. 15 and Fig. 16. Fig. 15 is a diagram showing an example of a processing flow for advance preparation before VP shooting.

[0093] LED wall WD and cinema camera commercial in the studio where the VP is filmed Ｒ General studio sets and spherical camera commercials Ｓis installed (step S1). The rendering unit 40 generates the test video I Ｐ The background CG display unit 50 displays the generated pattern on the screen SCR of the LED wall WD (step S2).

[0094] The environmental map acquisition unit 60 is a celestial sphere camera CM Ｓ camera parameters P including the position, orientation and focal length of Ｅ (Step S3). The environment map acquisition unit 60 acquires the omnidirectional camera CM Ｓ The environmental map acquisition unit 60 captures the celestial sphere camera CM Ｓ The surrounding image captured by the camera is used to create an environment map. Ｅ (step S4).

[0095] The composite environment map acquisition unit 70 receives the rendered image of the background scene from the rendering unit 40 as a background image I Ｂ The composite environment map acquisition unit 70 acquires the environment map I Ｅ The synthesis environment map acquisition unit 70 extracts the LED wall area from the background image I Ｂ A geometric transformation function g is calculated from each pixel in the region to each pixel in the LED wall region (step S5).

[0096] The rendering unit 40 generates a white image for brightness calibration. The background CG display unit 50 displays the generated white image on the screen SCR of the LED wall WD (step S6).

[0097] The environmental map acquisition unit 60 is a celestial sphere camera CM Ｓ The white image was captured on the environmental map I. Ｅ The composite environment map acquisition unit 70 acquires the pixel values ​​of the white image acquired as the rendering image and the environment map I Ｅ Based on the comparison result, the composite environment map acquisition unit 70 converts the rendered image into the environment map I. Ｅ The brightness correction coefficient for combining the two images is obtained (step S7).

[0098] FIG. 16 is a diagram showing an example of a processing flow during VP shooting.

[0099] The captured image acquisition unit 20 is a cinema camera commercial Ｒ The camera CM operates the camera CM1 to capture an image including the subject SB (foreground) and the LED wall WD (background) (step S11). Ｒ Footage I Ｃ and obtain the footage I Ｃ For each frame, a cinema camera commercial linked to the frame number is displayed. Ｒ Camera parameters P Ｃ and background video I Ｂ is acquired (step S12).

[0100] The foreground normal acquisition unit 31 acquires the foreground normal from the captured image I Ｃ From subject area I ＳＢ (Step S13). ＳＢ The extraction process can be performed using a common foreground-background separation technique, such as classical Graphcut or deep learning-based segmentation.

[0101] The foreground normal acquisition unit 31 acquires the object area I ＳＢ Normal Map I Ｎ (Step S14). The synthetic environment map acquisition unit 70 acquires the background image I Ｂ The synthetic environment map acquisition unit 70 applies the geometric transformation function g and the brightness correction coefficient to the background image I Ｂ Environment Map I Ｅ The LED wall area is overlaid with the synthetic environment map I. Ｅ ' is obtained (step S15).

[0102] The BRDF acquisition unit 32 acquires the composite environment map I Ｅ ', Normal Map I Ｎ , the camera parameters P of the spherical camera Ｅ and Cinema Camera Commercial Ｒ Camera parameters P Ｃ Using the above, the object region I ＳＢ The BRDF of each pixel position is obtained (step S16).

[0103] The virtual environment map acquisition unit 80 acquires the virtual omnidirectional camera CM ＶＳ The virtual omnidirectional camera commercial is placed in the 3DCG space of the replacement scene (step S17). ＶＳ The installation location is the virtual spherical camera commercial. ＶＳ and CG camera commercial Ｖ The positional relationship with the spherical camera commercial Ｓ and Cinema Camera CM Ｒ The positional relationship is set to be equal to that of the

[0104] The virtual environment map acquisition unit 80 acquires the virtual omnidirectional camera CM ＶＳ The replacement scene is rendered using the virtual environment map I. ＥＳ (Step S18). The relighting processing unit 33 obtains the estimated value of the BRDF, the virtual environment map I ＥＳ , Normal Map I Ｎ , camera parameters P Ｅ and the camera parameters P Ｃ is used to relight the subject SB in the foreground (step S19).

[0105] The rendering unit 40 generates a background image I from the 3DCG data of the replacement scene. Ｂ The rendering unit 40 generates the background image I Ｂ CG camera commercial to capture Ｖ and Virtual Environment Map I ＥＳ Virtual spherical camera commercial for capturing ＶＳ The foreground / background compositing unit 34 matches the captured frames (rendered frames) of the generated background image I Ｂ ' Relighting Video I ＲＥ Combined with this, composite image I Ｏ is acquired (step S20).

[0106] 4. Example of Hardware Configuration FIG. 17 is a diagram showing an example of the hardware configuration of the information processing device PR.

[0107] The information processing of the information processing device PR is realized by, for example, a computer 1000. The computer 1000 has a CPU (Central Processing Unit) 1100, a RAM (Random Access Memory) 1200, a ROM (Read Only Memory) 1300, a HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. The components of the computer 1000 are connected by a bus 1050.

[0108] The CPU 1100 operates and controls each component based on a program (program data 1450) stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 loads the program stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processing corresponding to the various programs.

[0109] The ROM 1300 stores boot programs such as a Basic Input Output System (BIOS) that is executed by the CPU 1100 when the computer 1000 starts up, as well as programs that depend on the hardware of the computer 1000 .

[0110] The HDD 1400 is a non-transitory computer-readable recording medium that non-temporarily records programs executed by the CPU 1100 and data used by such programs. Specifically, the HDD 1400 is a recording medium that records an information processing program according to an embodiment as an example of program data 1450.

[0111] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0112] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display device, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs recorded on a predetermined recording medium. Examples of media include optical recording media such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), magneto-optical recording media such as an MO (Magneto-Optical Disk), tape media, magnetic recording media, and semiconductor memories.

[0113] For example, when the computer 1000 functions as the information processing device PR according to the embodiment, the CPU 1100 of the computer 1000 executes an information processing program loaded onto the RAM 1200 to realize the functions of the aforementioned components. The information processing program, various models, and various data according to the present disclosure are stored in the HDD 1400. The CPU 1100 reads and executes program data 1450 from the HDD 1400. Alternatively, the CPU 1100 may obtain these programs from another device via an external network 1550.

[0114] [5. Effects] The information processing device PR has a BRDF acquisition unit 32, a virtual environment map acquisition unit 80, and a relighting processing unit 33. The BRDF acquisition unit 32 acquires a captured image I of the subject SB. Ｃ The virtual environment map acquisition unit 80 estimates the BRDF of the subject SB based on the environment map I obtained when the background scene of the subject SB is changed to a replacement scene. Ｅ and estimate the estimated environment map I Ｅ Virtual environment map I ＥＳ The relighting processing unit 33 obtains the virtual environment map I based on the BRDF of the subject SB.ＥＳ A relighting image I of the subject SB lit in a lighting environment represented by ＲＥ In the information processing method of the present disclosure, the processing of the information processing device PR is executed by the computer 1000. The computer-readable non-transitory storage medium of the present disclosure stores a program for causing the computer 1000 to realize the processing of the information processing device PR.

[0115] According to this configuration, the background image I Ｂ Even if you change the background image after the change, Ｂ Therefore, there is no mismatch between the lighting of the foreground and the background.

[0116] The information processing device PR has a composite environment map acquisition unit 70. The composite environment map acquisition unit 70 converts the image on the screen SCR, which is the background of the subject SB, into an environment map I Ｅ Composite environment map I is created by combining it with the position of the screen SCR in the Ｅ The BRDF acquisition unit 32 generates a BRDF from the captured video I Ｃ Inverse rendering is applied to the image area of ​​the subject SB extracted from Ｅ '.

[0117] According to this configuration, the background image I Ｂ An appropriate BRDF is obtained according to the background image I. Ｂ is a cinema camera commercial Ｒ A single pre-photographed environment map I moves with the user's movements. Ｅ It is difficult to estimate the BRDF using only the VP. Ideally, the VP should be captured simultaneously with the environment map I. Ｅ It is desirable to capture this in video, but the stage being filmed is Ｓ In this configuration, it is difficult to install the environment map I. Ｅ Background video I Ｂ Therefore, the simulation is based on the spherical camera CM Ｓ Background video I Ｂ A highly accurate BRDF is obtained according to the above.

[0118] The information processing device PR has an environment map acquisition unit 60. The environment map acquisition unit 60 acquires an environment map I in which the screen SCR is displayed. Ｅ The synthetic environment map acquisition unit 70 acquires the test image I displayed on the screen SCR. Ｐ and Environment Map I Ｅ Test footage I Ｐ The synthetic environment map acquisition unit 70 calculates a geometric transformation function g that indicates a mapping between the image on the screen SCR transformed by the geometric transformation function g and the environment map I. Ｅ The image is composited at the position of the screen SCR within the image.

[0119] According to this configuration, an arbitrary background image I Ｂ Highly accurate synthetic environment map I Ｅ ' is obtained.

[0120] The information processing device PR includes a rendering unit 40. The rendering unit 40 renders a test video I Ｐ A pattern image having a repeating pattern is generated as follows.

[0121] According to this configuration, a highly accurate geometric transformation function g can be obtained.

[0122] The composite environment map acquisition unit 70 acquires the brightness calibration image I displayed on the screen SCR. Ｗ and the pixel values ​​of the environment map I Ｅ Brightness calibration image I Ｗ The composite environment map acquisition unit 70 calculates the ratio between the pixel value of the background image I of the subject SB and the pixel value of the background image I of the subject SB. Ｂ Environment Map I Ｅ The brightness correction coefficient is obtained when combining the two images.

[0123] According to this configuration, a highly accurate synthetic environment map I Ｅ ' is obtained.

[0124] Brightness calibration video I Ｗ is a white image.

[0125] According to this configuration, a highly accurate luminance correction coefficient can be obtained.

[0126] The virtual environment map acquisition unit 80 uses the 3DCG data of the replacement scene to generate a virtual environment map I that shows a lighting environment in a range wider than the installation range of the screen SCR. ＥＳ Get.

[0127] According to this configuration, the background image I projected on the screen SCR Ｂ In normal VP photography, the subject SB is illuminated by the background image I projected on the screen SCR. Ｂ Lighting is done only from the camera side or studio side, where a screen SCR is not installed. Lighting from the camera side or studio side is difficult. If you try to force lighting, studio sets and other things will be reflected, resulting in unnatural images. With this configuration, any lighting space can be created based on the 3DCG data of the replacement scene. Since lighting can be expressed from all 360-degree directions, it is possible to express lighting from the camera side or studio side, which is difficult with normal VP shooting.

[0128] The information processing device PR has a foreground / background synthesis unit 34. The foreground / background synthesis unit 34 synthesizes a relighted image I of the subject SB. ＲＥ The background image I of the subject SB generated using the 3DCG data of the replacement scene Ｂ ' is synthesized.

[0129] This configuration allows for the acquisition of natural-looking images with an appropriate reflection of the background.

[0130] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0131] [Additional Notes] The present technology may also be configured as follows: (1) An information processing device comprising: a BRDF acquisition unit that estimates a BRDF of a subject based on a captured video of the subject; a virtual environment map acquisition unit that estimates an environment map obtained when a background scene of the subject is changed to a replacement scene and acquires the estimated environment map as a virtual environment map; and a relighting processing unit that generates a relit video of the subject lit in a lighting environment represented by the virtual environment map, based on the BRDF of the subject. (2) The information processing device according to (1), further comprising: a composite environment map acquisition unit that generates a composite environment map by composite an image on a screen that forms a background of the subject at a position of the screen in the environment map, wherein the BRDF acquisition unit applies inverse rendering to an image region of the subject extracted from the captured video and estimates the BRDF that is consistent with the composite environment map. (3) The information processing device according to (2), further comprising: an environment map acquisition unit that acquires the environment map onto which the screen is projected, wherein the composite environment map acquisition unit calculates a geometric transformation function that indicates a mapping between a test image displayed on the screen and the test image projected on the environment map, and composites the image on the screen transformed by the geometric transformation function at a position of the screen in the environment map. (4) The information processing device according to (3), further comprising: a rendering unit that generates a pattern image having a repeating pattern as the test image. (5) The information processing device according to (3) or (4), further comprising: the composite environment map acquisition unit calculates a ratio between pixel values ​​of a brightness calibration image displayed on the screen and pixel values ​​of the brightness calibration image projected on the environment map, and acquires the calculated ratio as a brightness correction coefficient for composite a background image of the subject onto the environment map. (6) The information processing device according to (5), further comprising: a white image.(7) The information processing device according to any one of (2) to (6), wherein the virtual environment map acquisition unit acquires the virtual environment map indicating a lighting environment of a range wider than the installation range of the screen using 3DCG data of the replaced scene. (8) The information processing device according to (7), further comprising a foreground / background synthesis unit that synthesizes a relighted video of the subject with a background video of the subject generated using the 3DCG data of the replaced scene. (9) An information processing method executed by a computer, comprising: estimating a BRDF of the subject based on a captured video of the subject; estimating an environment map that will be obtained when a background scene of the subject is changed to a replaced scene, and acquiring the estimated environment map as a virtual environment map; and generating a relighted video of the subject lit in the lighting environment represented by the virtual environment map, based on the BRDF of the subject. (10) A computer-readable non-transitory storage medium storing a program that causes a computer to perform the following operations: estimate a BRDF of a subject based on a captured image of the subject; estimate an environment map obtained when a background scene of the subject is changed to a replacement scene; acquire the estimated environment map as a virtual environment map; and generate a relighted image of the subject lit in a lighting environment represented by the virtual environment map based on the BRDF of the subject.

[0132] 32 BRDF acquisition unit 33 Relighting processing unit 34 Foreground / background synthesis unit 40 Rendering unit 60 Environment map acquisition unit 70 Synthesized environment map acquisition unit 80 Virtual environment map acquisition unit g Geometric transformation function I Ｂ , I Ｂ ′ Background image I Ｃ Footage I Ｅ Environment Map I Ｅ ' Composite Environment Map I ＥＳ Virtual Environment Map I Ｐ Test Footage I ＲＥ Relighting Video I Ｗ Brightness calibration video PR Information processing device SB Subject SCR Screen

Claims

1. An information processing device having: a BRDF acquisition unit that estimates a BRDF of a subject based on a captured image of the subject; a virtual environment map acquisition unit that estimates an environment map obtained when a background scene of the subject is changed to a replacement scene and acquires the estimated environment map as a virtual environment map; and a relighting processing unit that generates a relighted image of the subject lit in a lighting environment represented by the virtual environment map based on the BRDF of the subject.

2. An information processing device as described in claim 1, further comprising a synthetic environment map acquisition unit that generates a synthetic environment map by synthesizing an image on a screen that is the background of the subject with the position of the screen in an environment map, and the BRDF acquisition unit applies inverse rendering to the image area of ​​the subject extracted from the captured image and estimates the BRDF that is consistent with the synthetic environment map.

3. An information processing device as described in claim 2, further comprising an environment map acquisition unit that acquires the environment map in which the screen is displayed, wherein the composite environment map acquisition unit calculates a geometric transformation function that indicates a mapping between the test image displayed on the screen and the test image displayed in the environment map, and composites the image on the screen transformed by the geometric transformation function at the position of the screen in the environment map.

4. The information processing device according to claim 3, further comprising a rendering unit that generates a pattern image having a repeating pattern as the test image.

5. The information processing device of claim 3, wherein the composite environment map acquisition unit calculates a ratio between a pixel value of the brightness calibration image displayed on the screen and a pixel value of the brightness calibration image shown in the environment map, and acquires the calculated ratio as a brightness correction coefficient when synthesizing a background image of the subject with the environment map.

6. The information processing device according to claim 5, wherein the brightness calibration image is a white image.

7. The information processing device according to claim 2, wherein the virtual environment map acquisition unit acquires the virtual environment map showing a lighting environment in a range wider than the installation range of the screen, using 3DCG data of the replaced scene.

8. An information processing device according to claim 7, further comprising a foreground / background synthesis unit that synthesizes the relighted image of the subject with a background image of the subject generated using 3DCG data of the replaced scene.

9. An information processing method executed by a computer, comprising: estimating a BRDF of a subject based on a captured image of the subject; estimating an environment map obtained when a background scene of the subject is changed to a replacement scene, acquiring the estimated environment map as a virtual environment map; and generating a relighted image of the subject lit in a lighting environment represented by the virtual environment map based on the BRDF of the subject.

10. A computer-readable non-transitory storage medium storing a program that causes a computer to perform the following operations: estimating a BRDF of a subject based on a captured image of the subject; estimating an environment map obtained when the background scene of the subject is changed to a replacement scene, acquiring the estimated environment map as a virtual environment map; and generating a relighted image of the subject lit in a lighting environment represented by the virtual environment map based on the BRDF of the subject.

Images