Real-time generation method of shadow and mirror reflection effect in XR studio system

Abstract

The invention discloses a real-time generation method for shadow and mirror reflection effects in an XR studio system, which belongs to the field of video processing, and comprises the following steps: using a digital person as a figure digital branch which has physical attributes in a virtual scene, and generating a shadow and mirror reflection effect in the virtual scene; the virtual manufacturing software can carry out rendering calculation processing such as environment light, environment shadow, shadow shielding and light tracing by setting a light source, so that a shadow effect of a figure meeting physical rules under the light source and a mirror surface effect generated by interaction with a reflective material object are simulated, and the shadow effect is migrated to a real figure image, so that the real figure image is displayed. And the immersion and reality sense of people in the virtual scene are brought to audiences, so that the problem that the shadow and mirror reflection effect is difficult to generate in real time to improve the immersion and reality sense of the program when the XR studio system carries out the live broadcast service is solved.

Description

Technical Field

[0001] The present invention relates to the field of video processing, and more specifically, to a method for real-time generation of shadow and mirror reflection effects in an XR studio system. Background Art

[0002] The XR presentation screen is formed by superimposing the screen 1 of the virtual world taken by the virtual camera and the screen 2 of the real world taken by the camera. When the camera moves, the virtual camera will keep in sync with the movement of the camera. Since XR is a combination of real people and virtual scenes, the shadow generation method in the existing XR system usually uses target detection, natural scene matting and shadow calculation to generate shadow effects for the output screen. The generated shadow is only projected on the floor, has no interaction with objects in the virtual scene, and usually does not provide mirror reflection effects, resulting in the presentation screen being difficult to reflect the real effect of the character's shadow being blocked or deformed, as well as the effect of the virtual and real interaction between the character and the water surface, glass, etc., which seriously affects the immersion and realism of the XR broadcast. Summary of the invention

[0003] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a method for real-time generation of shadows and mirror reflection effects in an XR studio system, which solves the problem that it is difficult for the XR studio system to generate shadows and mirror reflection effects in real time to enhance the immersion and realism of the program when conducting live broadcast business.

[0004] The object of the present invention is achieved through the following solutions:

[0005] A method for real-time generation of shadow and specular reflection effects in an XR studio system comprises the following steps:

[0006] S1: 3D modeling of the characters in the XR broadcast program to form digital twins;

[0007] S2: Importing digital humans into virtual production software;

[0008] S3: According to the geometric relationship between the position and size of the person in the camera and the image displayed on the screen, the position of the digital human in the virtual scene in the virtual production software is calculated, and the size of the digital human is adjusted to ensure that the size of the rendered digital human image is consistent with the image of the person in the camera;

[0009] S4: Perform real-time matting of the person in the camera to obtain a sequence of person images [R1, R2, R3, R4...R n ];

[0010] S5: Extract the skeleton of the person in the camera and obtain the skeleton information [G1, G2, G3, G4...G n ];

[0011] S6: In the virtual production software, the skeleton of the character is bound to the digital human, and the skeleton information [G1, G2, G3, G4...G n ] Realize the motion driving and synchronization of human figures and digital humans under a single camera;

[0012] S7: Perform real-time rendering in the virtual production software to generate shadows and mirror reflection effects corresponding to the digital human, and obtain a virtual background image sequence [B1, B2, B3, R4...B n ] and the image sequence of the virtual character [V1, V2, V3, V4...V n ], where the avatar is a separate layer;

[0013] S8: [R1, R2, R3, R4...R n ]、[B1、B2、B3、R4...B n ] and {P} are synthesized to output the final presentation image of the XR studio system.

[0014] Furthermore, in step S1, the clothes, hair, and body shape of the twin digital human are consistent with those of the character being played.

[0015] Further, in step S2, the virtual production software includes UE production software.

[0016] Furthermore, in step S4, the real-time matting of the person in the camera is performed using a matting model PP-HumanMatting.

[0017] Further, in step S5, the skeleton of the person in the camera is extracted to obtain skeleton information [G1, G2, G3, G4...G n ], specifically including the sub-steps of: using the ViTPose model to extract the skeleton of the person in the camera.

[0018] Furthermore, the method of extracting the skeleton of the person in the camera by using the ViTPose model specifically includes the following sub-steps:

[0019] S51: Use the top-down paradigm for model training and testing;

[0020] S52: Use the ViT-B network with a medium number of parameters as the basic network for extracting image features from ViTPose;

[0021] S53: The decoder uses bilinear interpolation to perform multiple upsampling on the feature map extracted by Vit-B, and then inputs it into the ReLU activation function and the convolution layer to predict the bone information; the multiple includes 4 times; the activation function includes the ReLU activation function.

[0022] Furthermore, in step S7, real-time rendering is performed in the virtual production software to generate shadows and mirror reflection effects corresponding to the digital human, and a virtual background image sequence [B1, B2, B3, R4...B n ] and the image sequence of the virtual character [V1, V2, V3, V4...V n ], where the avatar is a separate layer, specifically including the following sub-steps:

[0023] S71: If the image of the virtual character V x If an environmental object casts a shadow on the virtual character, V x Use matting to get the shadow area image P x ';

[0024] S72: P x 'The shadow area image is replaced with black and the transparency is set to get the image P x ;

[0025] S73: Repeat steps S71 and S72 to obtain a shadow mask image sequence {P} on the virtual character.

[0026] Furthermore, the setting of transparency specifically includes setting it to 20%.

[0027] Further, in step S8, [R1, R2, R3, R4...R n ]、[B1、B2、B3、R4...B n ] and {P} are synthesized to output the final presentation screen of the XR studio system, which specifically includes sub-steps: using a pipeline method to perform parallel calculations on character matting, skeleton extraction, posture driving, rendering, shadow matting, and synthesis output, and setting the delay of all parallel calculation processes to t.

[0028] Furthermore, the output of the final presentation screen of the XR studio system also includes a sub-step: during live broadcast, at least increasing the screen delay t to ensure smooth live broadcast.

[0029] The beneficial effects of the present invention include:

[0030] The technical solution of the present invention utilizes a digital human as a digital avatar of a character, which has physical properties in a virtual scene. The virtual production software can perform rendering calculations such as ambient light, ambient shadows, shadow shielding, and ray tracing by setting the light source, thereby simulating the shadow effect of the character that conforms to physical rules under the light source, as well as the mirror effect generated by the interaction with reflective material objects, and migrates the shadow effect to the real character image, giving the audience a sense of immersion and reality as if the character is in a virtual scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0032] Figure 1 For the original shooting process and presentation effect;

[0033] Figure 2 It is a schematic diagram of the process of the present invention;

[0034] Figure 3 Schematic diagram of pipeline calculation in the method of the present invention. DETAILED DESCRIPTION

[0035] All features disclosed in all embodiments in this specification, or steps in all methods or processes implicitly disclosed, except for mutually exclusive features and / or steps, can be combined and / or expanded or replaced in any manner.

[0036] Explanation of terms

[0037] The top-down paradigm is an autoregressive training model that first detects people and then estimates their poses.

[0038] The present invention solves the following technical problems: Figure 1 As shown, the shadow generation method in the existing XR system usually uses target detection, natural scene matting and shadow calculation to generate shadow effects on the output screen. The generated shadow is only projected on the floor, has no interaction with the objects in the virtual scene, and usually does not provide a mirror reflection effect. Therefore, in order to solve the problem that it is difficult for the XR studio system to generate shadows and mirror reflection effects in real time to enhance the immersion and realism of the program when conducting live broadcast services, the present invention proposes a real-time generation method of shadows and mirror reflection effects in an XR studio system. In the specific implementation process, such as Figure 2 and Figure 3 As shown, the following steps are included:

[0039] S1: Create three-dimensional models of the characters in XR broadcast programs to form twin digital humans, whose clothes, hair, and body shape are consistent with the characters.

[0040] S2: Import digital humans into virtual production software such as UE (Unreal Engine).

[0041] S3: According to the geometric relationship between the position and size of the person in the camera and the image presented by the three screens (ground screen + two vertical screens), the position of the digital human in the virtual scene in the virtual production software is calculated, and the size of the digital human is adjusted to ensure that the size of the rendered digital human image is consistent with the image of the person in the camera.

[0042] S4: Perform real-time matting of the person in the camera to obtain a sequence of person images [R1, R2, R3, R4...R n ].

[0043] S41: To ensure a faster matting speed under ultra-high-definition cameras, the matting model uses PP-HumanMatting.

[0044] S5: Using monocular human posture estimation technology, the ViTPose model can be used to extract the skeleton of the person in the camera and obtain the skeleton information [G1, G2, G3, G4...G n ].

[0045] S51: The top-down paradigm is adopted, that is, the autoregressive training mode of first performing person detection and then posture estimation from top to bottom, for model training and testing.

[0046] S52: In order to reduce the amount of calculation and improve the training and reasoning speeds, the ViT-B network with a medium number of parameters is used as the basic network for ViTPose to extract image features, so as to ensure good estimation accuracy while achieving real-time posture estimation.

[0047] S53: To reduce the amount of calculation, the decoder uses bilinear interpolation to upsample the feature map extracted by Vit-B by 4 times, and then inputs it into the ReLU activation function and convolution layer to predict the skeleton information.

[0048] S6: In the virtual production software, the skeleton of the character is bound to the digital human, and the skeleton information [G1, G2, G3, G4...G n ] Realize the motion driving and synchronization of human figures and digital humans under a single camera.

[0049] S7: Perform real-time rendering in the virtual production software to generate shadows and mirror reflection effects corresponding to the digital human, and obtain a virtual background image sequence [B1, B2, B3, R4...B n ] and the image sequence of the virtual character [V1, V2, V3, V4...V n ], where the avatar is a separate layer.

[0050] S71: If V x Environmental objects cast shadows on virtual characters, such as when a person stands under a tree and has shadows cast by leaves and branches on his body.x Use matting to get the shadow area image P x '.

[0051] S72: Due to P x 'With a certain degree of transparency, the image contains the digital human's clothes, skin and other contents, which will appear unnatural if directly transferred to the human. The shadow area image is replaced with black and the transparency is set to 20%. x .

[0052] S73: Repeat the above operation to obtain a shadow mask image sequence {P} on the virtual character.

[0053] S8: [R1, R2, R3, R4...R n ]、[B1、B2、B3、R4...B n ] and {P} to synthesize and output the final presentation screen of the XR studio system. Specifically, a pipeline method is used to parallelly calculate character matting, skeleton extraction, posture driving, rendering, shadow matting, and synthesis output to improve processing efficiency. Among them, including shadow matting, the delay of executing all calculation processes is t. During live broadcast, the screen delay should be increased by at least t, and 2*t is recommended to ensure smooth live broadcast.

[0054] It should be noted that within the scope of protection defined in the claims of the present invention, the following embodiments can be combined and / or expanded or replaced in any logical way from the above specific implementation methods, such as disclosed technical principles, disclosed technical features or implicitly disclosed technical features.

[0055] Example 1

[0056] A method for real-time generation of shadow and specular reflection effects in an XR studio system comprises the following steps:

[0057] S1: 3D modeling of the characters in the XR broadcast program to form digital twins;

[0058] S2: Importing digital humans into virtual production software;

[0059] S3: According to the geometric relationship between the position and size of the person in the camera and the image displayed on the screen, the position of the digital human in the virtual scene in the virtual production software is calculated, and the size of the digital human is adjusted to ensure that the size of the rendered digital human image is consistent with the image of the person in the camera;

[0060] S4: Perform real-time matting of the person in the camera to obtain a sequence of person images [R1, R2, R3, R4...R n ];

[0061] S5: Extract the skeleton of the person in the camera and obtain the skeleton information [G1, G2, G3, G4...G n ];

[0062] S6: In the virtual production software, the skeleton of the character is bound to the digital human, and the skeleton information [G1, G2, G3, G4...G n ] Realize the motion driving and synchronization of human figures and digital humans under a single camera;

[0063] S7: Perform real-time rendering in the virtual production software to generate shadows and mirror reflection effects corresponding to the digital human, and obtain a virtual background image sequence [B1, B2, B3, R4...B n ] and the image sequence of the virtual character [V1, V2, V3, V4...V n ], where the avatar is a separate layer;

[0064] S8: [R1, R2, R3, R4...R n ]、[B1、B2、B3、R4...B n ] and {P} are synthesized to output the final presentation image of the XR studio system.

[0065] Example 2

[0066] Based on Example 1, in step S1, the clothes, hair, and body shape of the twin digital human are consistent with those of the character.

[0067] Example 3

[0068] Based on Embodiment 1, in step S2, the virtual production software includes UE production software.

[0069] Example 4

[0070] On the basis of Example 1, in step S4, the real-time matting of the person in the camera is performed by using a matting model PP-HumanMatting.

[0071] Example 5

[0072] On the basis of Example 1, in step S5, the skeleton of the person in the camera is extracted to obtain skeleton information [G1, G2, G3, G4...G n ], specifically including the sub-steps of: using the ViTPose model to extract the skeleton of the person in the camera.

[0073] Example 6

[0074] Based on Example 5, the method of using the ViTPose model to extract the skeleton of the person in the camera specifically includes the following sub-steps:

[0075] S51: Use the top-down paradigm for model training and testing;

[0076] S52: Use the ViT-B network with a medium number of parameters as the basic network for extracting image features from ViTPose;

[0077] S53: The decoder uses bilinear interpolation to perform multiple upsampling on the feature map extracted by Vit-B, and then inputs it into the ReLU activation function and the convolution layer to predict the bone information; the multiple includes 4 times; the activation function includes the ReLU activation function.

[0078] Example 7

[0079] On the basis of Example 1, in step S7, real-time rendering is performed in the virtual production software to generate shadows and mirror reflection effects corresponding to the digital human, and a virtual background image sequence [B1, B2, B3, R4...B n ] and the image sequence of the virtual character [V1, V2, V3, V4...V n ], where the avatar is a separate layer, specifically including the following sub-steps:

[0080] S71: If the image of the virtual character V x If an environmental object casts a shadow on the virtual character, V x Use matting to get the shadow area image P x ';

[0081] S72: P x 'The shadow area image is replaced with black and the transparency is set to get the image P x ;

[0082] S73: Repeat steps S71 and S72 to obtain a shadow mask image sequence {P} on the virtual character.

[0083] Example 8

[0084] Based on Example 7, the setting of transparency specifically includes setting it to 20%.

[0085] Example 9

[0086] Based on Example 7, in step S8, [R1, R2, R3, R4...R n ]、[B1、B2、B3、R4...B n] and {P} are synthesized to output the final presentation screen of the XR studio system, which specifically includes sub-steps: using a pipeline method to perform parallel calculations on character matting, skeleton extraction, posture driving, rendering, shadow matting, and synthesis output, and setting the delay of all parallel calculation processes to t.

[0087] Example 10

[0088] On the basis of Example 9, the output of the final presentation screen of the XR studio system also includes a sub-step: during live broadcast, at least increase the screen delay t to ensure smooth live broadcast.

[0089] The units involved in the embodiments of the present invention may be implemented by software or hardware, and the units described may also be arranged in a processor. The names of these units do not, in some cases, limit the units themselves.

[0090] According to one aspect of an embodiment of the present invention, a computer program product or a computer program is provided, the computer program product or the computer program includes a computer instruction, and the computer instruction is stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in the above various optional implementations.

[0091] As another aspect, an embodiment of the present invention further provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the method described in the above embodiment.

Claims

1. A method for real-time generation of shadow and specular reflection effects in an XR studio system, characterized in that: The following steps are involved: S1: 3D modeling of the characters in the XR broadcast program to form digital twins; S2: Importing digital humans into virtual production software; S3: According to the geometric relationship between the position and size of the person in the camera and the image displayed on the screen, the position of the digital human in the virtual scene in the virtual production software is calculated, and the size of the digital human is adjusted to ensure that the size of the rendered digital human image is consistent with the image of the person in the camera; S4: Perform real-time matting of the person in the camera to obtain a sequence of person images [R1, R2, R3, R4...R n ]; S5: Extract the skeleton of the person in the camera and obtain the skeleton information [G1, G2, G3, G4...G n ]; S6: In the virtual production software, the skeleton of the character is bound to the digital human, and the skeleton information [G1, G2, G3, G4...G n ] Realize the motion driving and synchronization of human figures and digital humans under a single camera; S7: Perform real-time rendering in the virtual production software to generate shadows and mirror reflection effects corresponding to the digital human, and obtain a virtual background image sequence [B1, B2, B3, R4...B n ] and the image sequence of the virtual character [V1, V2, V3, V4...V n ], where the avatar is a separate layer; S8: [R1, R2, R3, R4...R n ]、[B1、B2、B3、R4...B n ] and {P} are synthesized to output the final presentation image of the XR studio system.

2. The method for real-time generation of shadow and specular reflection effects in an XR studio system according to claim 1, characterized in that: In step S1, the clothes, hair, and body shape of the twin digital human are consistent with those of the character.

3. The method for real-time generation of shadow and specular reflection effects in an XR studio system according to claim 1, characterized in that: In step S2, the virtual production software includes UE production software.

4. The method for real-time generation of shadow and specular reflection effects in an XR studio system according to claim 1, characterized in that: In step S4, the real-time matting of the person in the camera is performed using a matting model PP-HumanMatting.

5. The method for real-time generation of shadow and specular reflection effects in an XR studio system according to claim 1, characterized in that: In step S5, the skeleton of the person in the camera is extracted to obtain skeleton information [G1, G2, G3, G4...G n ], specifically including the sub-steps of: using the ViTPose model to extract the skeleton of the person in the camera.

6. The method for real-time generation of shadow and specular reflection effects in an XR studio system according to claim 5, characterized in that: The method of extracting the skeleton of the person in the camera using the ViTPose model specifically includes the following sub-steps: S51: Use the top-down paradigm for model training and testing; S52: Use the ViT-B network with a medium number of parameters as the basic network for extracting image features from ViTPose; S53: The decoder uses bilinear interpolation to perform multiple upsampling on the feature map extracted by Vit-B, and then inputs it into the ReLU activation function and the convolution layer to predict the bone information; the multiple includes 4 times; the activation function includes the ReLU activation function.

7. The method for real-time generation of shadow and specular reflection effects in an XR studio system according to claim 1, characterized in that: In step S7, the virtual production software performs real-time rendering to generate shadows and mirror reflection effects corresponding to the digital human, and obtains a virtual background image sequence [B1, B2, B3, R4...B n ] and the image sequence of the virtual character [V1, V2, V3, V4...V n ], where the avatar is a separate layer, specifically including the following sub-steps: S71: If the image of the virtual character V x If an environmental object casts a shadow on the virtual character, V x Use matting to get the shadow area image P x '; S72: P x 'The shadow area image is replaced with black and the transparency is set to get the image P x ; S73: Repeat steps S71 and S72 to obtain a shadow mask image sequence {P} on the virtual character.

8. The method for real-time generation of shadow and specular reflection effects in an XR studio system according to claim 7, characterized in that: The setting of transparency specifically includes setting it to 20%.

9. The method for real-time generation of shadow and specular reflection effects in an XR studio system according to claim 7, characterized in that: In step S8, the [R1, R2, R3, R4 ... R n ]、[B1、B2、B3、R4...B n ] and {P} are synthesized to output the final presentation screen of the XR studio system, which specifically includes sub-steps: using a pipeline method to perform parallel calculations on character matting, skeleton extraction, posture driving, rendering, shadow matting, and synthesis output, and setting the delay of all parallel calculation processes to t.

10. The method for real-time generation of shadow and specular reflection effects in an XR studio system according to claim 9, characterized in that: The output of the final presentation screen of the XR studio system also includes a sub-step: during live broadcast, at least increasing the screen delay t to ensure smooth live broadcast.

Images