An image processing method for virtual shooting and a related device

Abstract

The application provides an image processing method for virtual shooting and related equipment, which is suitable for the technical field of virtual shooting, generates a human body model parameter according to a monocular image, and generates a three-dimensional human body grid in the human body model parameter; a digital human model is generated by using the three-dimensional human body grid of the monocular image; a human body model parameter sequence of each frame of monocular image is calculated; the digital human model is driven by using the human body model parameter sequence, and a target three-dimensional human body grid is obtained; the target three-dimensional human body grid of the monocular image is rendered by using a virtual camera, and a depth map and a normal map of the target three-dimensional human body grid are obtained; synthesis is performed by using a human body patch shader, the depth map and the normal map of the target three-dimensional human body grid, and an image meeting the high-quality shooting requirement is obtained, so that the problems that a depth-of-field effect cannot be generated and the normal distribution of unevenness of each part of a human body cannot be simulated in the prior art are solved, and the post-production depth-of-field effect, the color adjustment period, the cost are short, low, and do not need to rely on manual processing.

Description

Technical Field

[0001] This invention relates to the field of virtual photography technology, and more specifically, to an image processing method and related equipment for virtual photography. Background Technology

[0002] The basic principle of green screen-based virtual shooting is to extract green from real-time images of the human body captured by a real camera, use this green as a texture, and then apply it to a plane in the scene. A virtual camera then renders the scene, outputting a composite image of the human body and the 3D scene. Figure 1 As shown, it is widely used in the production of various types of content such as live streaming, short dramas, films and television dramas, and variety shows.

[0003] However, because the human body is first photographed as a 2D image by a real camera, and then processed to obtain a human body image before being pasted onto a plane, it lacks realistic depth and normal information. Furthermore, when a virtual camera captures an image, the entire human body is at the same depth (for example, the depth of the nose and ears is the same relative to the camera), making it impossible to produce depth-of-field effects such as localized blurring or focus shift. It also cannot simulate the uneven distribution of normals across the human body to correctly receive virtual lighting, thus failing to meet the demands of high-quality shooting. Relying on post-production to create depth-of-field effects and color grading is time-consuming, costly, and entirely reliant on manually defining the areas to be blurred, resulting in low-quality images. Summary of the Invention

[0004] In view of this, the present invention provides an image processing method and related equipment for virtual shooting, so as to obtain images that meet the requirements of high-quality shooting. It solves the problems in the prior art that it cannot produce depth-of-field effects such as local defocus and focus movement, and cannot simulate the uneven normal distribution of various parts of the human body to correctly receive virtual lighting. It also aims to achieve short post-production depth-of-field effects and color grading cycle, low cost, and no need to rely on manual processing.

[0005] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0006] The first aspect of this invention discloses an image processing method for virtual photography, the method comprising:

[0007] Acquire the video to be processed, wherein the video to be processed includes: a series of multiple monocular images;

[0008] Human body model parameters are generated based on the monocular image, and a three-dimensional human body mesh is generated based on the human body model parameters;

[0009] Using the three-dimensional human body mesh of the monocular image, a digital human model of the monocular image is generated;

[0010] Calculate the sequence of human body model parameters for each frame of the monocular image;

[0011] The human body model parameter sequence of the monocular image is used to drive the digital human model of the monocular image to obtain the target three-dimensional human body mesh of the monocular image;

[0012] The target 3D human body mesh of the monocular image is rendered using a virtual camera to obtain the depth map and normal map of the target 3D human body mesh;

[0013] The final image of the monocular image is obtained by synthesizing the human face shader, the depth map and the normal map of the target 3D human body mesh, wherein the final image includes human face patches and virtual scene.

[0014] Preferably, generating human body model parameters based on the monocular image, and generating a three-dimensional human body mesh based on the human body model parameters, includes:

[0015] The encoder is used to extract features from the monocular image to obtain image features, and the decoder is used to regress the parameterization of the human body model parameters from the image features.

[0016] A three-dimensional mesh is generated by using the parameters of the human body model through a preset parametric model, and then the three-dimensional mesh vertices are globally transformed using global transformation parameters to generate a three-dimensional human body mesh.

[0017] Preferably, calculating the human model parameter sequence for each frame of the monocular image includes:

[0018] For each frame of the monocular image, the monocular image is downsampled to obtain the target monocular image;

[0019] The target monocular image is extracted using an encoder to obtain target image features, and the target human body model parameters are regressed and parameterized from the target image features using a decoder.

[0020] The target human body model parameters are filtered, and a specified neural network is used to smooth the target human body model parameters in a sliding window to obtain the human body model parameter sequence of the target monocular image.

[0021] Preferably, the target 3D human body mesh of the monocular image is rendered using a virtual camera to obtain the depth map and normal map of the target 3D human body mesh, including:

[0022] Create a corresponding virtual camera based on the camera parameters of the camera that captured the video to be processed;

[0023] The depth value of each pixel in the target 3D human body mesh is rendered into a high-precision texture using the virtual camera to obtain the depth map of the target 3D human body mesh.

[0024] The world space normal vector of each pixel in the target 3D human body mesh is calculated using the virtual camera, and the world space normal vector of each pixel is written into another high-precision texture to obtain the normal map of the target 3D human body mesh.

[0025] Preferably, the final image of the monocular image is obtained by synthesizing the human face patch shader, the depth map and the normal map of the target 3D human body mesh, including:

[0026] Obtain the current pixel of the virtual camera and the pixel coordinates of the bottom corner of the current viewport of the virtual camera;

[0027] Determine the corresponding texture coordinates based on the current pixel and the pixel coordinates;

[0028] The depth blur is calculated using the texture coordinates and the depth values ​​in the depth map to obtain the color of the human face patch;

[0029] The world normal is decoded using the normal map, and the human face patch lighting information is determined based on the texture coordinates and the world normal.

[0030] The final image of the monocular image is obtained by fusing the texture coordinates, the human face shader, the human face color, and the human face lighting information.

[0031] A second aspect of this invention discloses an image processing apparatus for virtual photography, the apparatus comprising:

[0032] An acquisition unit is used to acquire a video to be processed, wherein the video to be processed includes a series of consecutive monocular images;

[0033] The first generation unit is used to generate human body model parameters based on the monocular image, and to generate a three-dimensional human body mesh based on the human body model parameters.

[0034] The second generation unit is used to generate a digital human model of the monocular image using the three-dimensional human body mesh of the monocular image;

[0035] A computing unit is used to calculate the sequence of human model parameters for each frame of the monocular image;

[0036] A driving unit is used to drive the digital human model of the monocular image using the human body model parameter sequence of the monocular image to obtain the target three-dimensional human body mesh of the monocular image.

[0037] The rendering unit is used to render the target three-dimensional human body mesh of the monocular image using a virtual camera, and obtain the depth map and normal map of the target three-dimensional human body mesh.

[0038] The processing unit is used to synthesize the final image of the monocular image by using a human face shader, the depth map and normal map of the target three-dimensional human body mesh, wherein the final image includes human face patches and a virtual scene.

[0039] Preferably, the first generating unit is specifically used for:

[0040] The encoder is used to extract features from the monocular image to obtain image features, and the decoder is used to regress the parameterization of the human body model parameters from the image features.

[0041] A three-dimensional mesh is generated by using the parameters of the human body model through a preset parametric model, and then the three-dimensional mesh vertices are globally transformed using global transformation parameters to generate a three-dimensional human body mesh.

[0042] Preferably, the computing unit is specifically used for:

[0043] For each frame of the monocular image, the monocular image is downsampled to obtain the target monocular image;

[0044] The target monocular image is extracted using an encoder to obtain target image features, and the target human body model parameters are regressed and parameterized from the target image features using a decoder.

[0045] The target human body model parameters are filtered, and a specified neural network is used to smooth the target human body model parameters in a sliding window to obtain the human body model parameter sequence of the target monocular image.

[0046] A third aspect of the present invention discloses an electronic device, comprising: a processor and a memory, the processor and the memory being connected via a bus; wherein the processor is configured to call and execute a program stored in the memory; the memory is configured to store the program, the program being configured to implement the image processing method for virtual shooting disclosed in the first aspect of the present invention.

[0047] A fourth aspect of the present invention discloses a computer-readable storage medium storing computer-executable instructions for performing the image processing method for virtual shooting disclosed in the first aspect of the present invention.

[0048] This application provides an image processing method and related equipment for virtual photography. The method involves acquiring a video to be processed, wherein the video includes multiple consecutive monocular images; generating human model parameters based on the monocular images, and generating a three-dimensional human mesh based on the human model parameters; generating a digital human model of the monocular image using the three-dimensional human mesh; calculating a sequence of human model parameters for each frame of the monocular image; driving the digital human model of the monocular image using the sequence of human model parameters to obtain a target three-dimensional human mesh of the monocular image; rendering the target three-dimensional human mesh of the monocular image using a virtual camera to obtain a depth map and a normal map of the target three-dimensional human mesh; and synthesizing the target three-dimensional human mesh using a human face shader, the depth map, and the normal map of the target three-dimensional human mesh to obtain a final image of the monocular image, wherein the final image includes human face patches and a virtual scene. Therefore, the technical solution provided in this application first uses the human body model parameters of the monocular image to generate a three-dimensional human body mesh of the monocular image, and then uses the three-dimensional human body mesh to generate a digital human model to achieve efficient human body reconstruction; calculates the human body model parameter sequence of the monocular image to achieve vision-based wearless motion capture; uses the human body model parameter sequence of the monocular image to drive the digital human model of the monocular image to quickly generate a digital human model (target three-dimensional human body mesh) with the same height as the photographed human body; uses a virtual camera to render the target three-dimensional human body mesh of the monocular image, and after obtaining the depth map and normal map of the target three-dimensional human body mesh, uses the depth map and normal map in the human body face shader to obtain a final image that integrates accurate depth of field, lighting and cinematic realism, that is, an image that meets the requirements of high-quality shooting, thereby solving the problems in the prior art that cannot produce depth of field effects such as local defocus and focus movement, and cannot simulate the uneven normal distribution of various parts of the human body to correctly receive virtual lighting, and no post-processing is required. Not only is the post-production of depth of field effects and color grading cycle short and low cost, but it also does not rely on manual processing. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0050] Figure 1 An example image provided by existing technology for fusing human body face patches and 3D scene;

[0051] Figure 2A flowchart illustrating an image processing method for virtual photography provided in an embodiment of this application;

[0052] Figure 3 An example diagram of generating a three-dimensional human body mesh provided in this application embodiment;

[0053] Figure 4 An example diagram illustrating the generation of a human body model parameter sequence provided in this application embodiment;

[0054] Figure 5 An example diagram illustrating the generation of the final image provided in this application embodiment;

[0055] Figure 6 This is a schematic diagram of the structure of an image processing system for virtual shooting provided in an embodiment of the present invention;

[0056] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] In this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0059] See Figure 2 The diagram illustrates a flowchart of an image processing method for virtual photography provided in an embodiment of this application, specifically including the following steps:

[0060] S201: Obtain the video to be processed, wherein the video to be processed includes a series of consecutive monocular images.

[0061] In this embodiment of the application, the obtained video to be processed can be a monocular RGB video, wherein the monocular RGB video includes a monocular RGB image sequence (image sequence), and the monocular RGB image sequence includes a series of consecutive monocular RGB images (monocular images).

[0062] S202: Generate human body model parameters from monocular images, and generate a 3D human body mesh from the human body model parameters.

[0063] In the specific execution step S202, for each frame of monocular image in the video to be processed, feature extraction and parameter regression processing can be performed on the monocular image to obtain human body model parameters, and then a three-dimensional human body mesh can be generated using these human body model parameters. The human body model parameters include target pose parameters and target shape parameters.

[0064] In some embodiments, an end-to-end deep learning network can be used to perform feature extraction and parameter regression processing on monocular images to obtain human model parameters; wherein, the end-to-end deep learning network consists of an encoder and a decoder architecture, including an encoder and a decoder, the encoder being a feature extraction encoder (HRNet / ResNet, etc.), and the decoder being a parameter regression decoder (fully connected network (Multi-Person Linear, MLP)).

[0065] Optionally, the process of generating human body model parameters from monocular images and generating a three-dimensional human body mesh from the human body model parameters can be as follows: using an encoder to extract features from the monocular image to obtain image features, and using a decoder to regress and parameterize the human body model parameters from the image features; using the human body model parameters to generate three-dimensional mesh vertices through a preset parameterized model, and using global transformation parameters to perform global transformation on the three-dimensional mesh vertices to generate a three-dimensional human body mesh.

[0066] In some examples, the pre-defined parametric model can be a parametric 3D human body model.

[0067] In this embodiment, a parametric 3D human body model (Skinned Multi-Person Linear, SMPL) is defined, which can automatically deform into a suitable shape based on a set of human body parameters. The parametric 3D human body model includes 72 pose parameters θ∈R. 72 (Rotation of 23 joints + rotation of 1 root joint), 10 shape parameters β∈R 10 and 3 global rotation parameters L∈R 3 and 3 global translation parameters T∈R 3 .

[0068] In practical applications, see Figure 3For each frame of monocular RGB image, the monocular image is input into an end-to-end deep learning network to extract high-dimensional image features from the monocular RGB image using the feature extraction encoder in the end-to-end deep learning network, and the parameter regression decoder (fully connected network MLP) is used to directly regress the parameterization of the human body model parameters from the image features. The target pose parameter θ and target shape parameter β in the regressed human body model parameters are input into the SMPL model, so that the SMPL model uses function M to generate the corresponding three-dimensional mesh vertex V in the standard pose according to the input parameters (θ,β). Then, the three-dimensional mesh vertex V is globally transformed to obtain the three-dimensional human body mesh (3D human body mesh with pose) in the world coordinate system, as shown in formula (1).

[0069] V world =L·M(θ,β)+T(1;

[0070] Among them, V world V refers to the world coordinates of each 3D mesh vertex V, which is the shape of the 3D human body mesh. L refers to the global rotation parameter, T is the global translation parameter, and M is the function of the SMPL model.

[0071] It should be noted that this application generates human body model parameters from monocular images using a regression neural network. This regression neural network consists of an end-to-end deep learning network and a parametric regression decoder. It aligns a pre-defined parametric model (i.e., the SMPL model) with 2D observations (keypoints, contours) in perspective projection space by explicitly modeling perspective camera parameters (focal length, camera distance), rather than the traditional weak perspective space. This allows for the creation of a 3D human body mesh with higher dimensional accuracy, ensuring better model fit from the camera's perspective. Here, the camera parameters refer to the camera parameters of the camera that captured the monocular images; the 2D observations are image features.

[0072] S203: Generate a digital human model from a monocular image using a 3D human body mesh.

[0073] In the specific execution step S203, after obtaining the three-dimensional human body mesh of the monocular image, the three-dimensional human body mesh of the monocular image can be further used to generate human skeleton information and skin information under the monocular image; specifically, since the SMPL model itself has a built-in standard human skeleton hierarchy structure and pre-calculated skin weight matrix W, once the corresponding target pose parameter θ and target shape parameter β are regressed from the monocular image, the corresponding human skeleton information and skin information are readily available and do not require additional calculation. Therefore, the reconstructed three-dimensional tenting mesh naturally supports the digital human model of the monocular image driven by linear hybrid skin (LBS), wherein the digital human model contains human skeleton information and skin information; the method of using linear hybrid skin (LBS) to drive the digital human model containing human skeleton information and skin information is shown in formula (2).

[0074] DigitalHuman=(M(θ,β),Skeleton,W)(2);

[0075] In this model, DigitalHuman represents the digital human model, W is the skinning weight matrix, θ is the target pose parameter, β is the target shape parameter, and M is the shape. In other words, the digital human model contains a shape M, a skeleton (Skeleton), and a set of skinning weights.

[0076] S204: Calculate the sequence of human body model parameters for each frame of monocular image.

[0077] In the specific execution step S204, for each frame of monocular image in the video to be processed, the monocular image can be downsampled to obtain the target monocular image; the target human body model parameters of the target monocular image are calculated, and the target human body model parameters are subjected to temporal smoothing optimization processing to obtain the human body model parameter sequence of the monocular image.

[0078] It should be noted that by performing temporal smoothing optimization on the target human body model parameters of monocular images, it is possible not only to effectively overcome the inter-frame jitter and depth ambiguity of monocular reconstruction, but also to stably estimate the human body posture of each frame of the bullet screen image from the video sequence (image sequence), thereby achieving the corresponding motion capture.

[0079] Optionally, the process of calculating the human model parameter sequence for each frame of monocular image can be as follows: for each frame of monocular image, downsample the monocular image to obtain the target monocular image; use an encoder to extract features from the target monocular image to obtain target image features, and use a decoder to regress the target human model parameters from the target image features; filter the target human model parameters, and use a specified neural network (e.g., a parameter smoothing neural network) in a sliding window to smooth the target human model parameters to obtain the human model parameter sequence of the target monocular image.

[0080] It should be noted that the process of generating the target human body model parameters is the same as the process of generating the human body model parameters in step S202. Please refer to the corresponding part in step S202. This embodiment of the application does not limit it here.

[0081] In practical applications, see Figure 4 The process involves obtaining an image sequence containing multiple consecutive frames of monocular RGB images. First, for each frame of monocular RGB image, a lightweight single-frame parametric regression network is used to downsample the monocular RGB image to obtain the target monocular image. Then, feature extraction is performed on the target monocular image to obtain the target image features. The target human model parameters (θ_raw, β, R, T_raw) are regressed from the target image features using a parametric regression decoder. The Kalman filter (to suppress high-frequency jitter) is used to filter θ_raw in the target human model parameters. At the same time, in a sliding window, an RNN or Transformer neural network is used to regress T_raw in the target human model parameters using the features (contextual information) of the past N frames to smooth the result, resulting in a more stable sequence of human model parameters [θt, β, R, Tt].

[0082] It should be noted that by using a neural network to smooth the parameters of the target human model through contextual information within a sliding window, the accuracy and smoothness of the obtained human model parameter sequence can be effectively improved, resulting in a more time-series-smooth and stable human model parameter sequence.

[0083] S205: Use the human body model parameter sequence of monocular image to drive the digital human model of monocular image to obtain the target three-dimensional human body mesh of monocular image.

[0084] In the specific execution step S205, for each frame of monocular image, the human body model parameter sequence of the monocular image can be applied to the digital human model to drive the movement of the digital human model and obtain the target three-dimensional human body mesh of the monocular image.

[0085] Specifically, for each frame of monocular image, the optimized human model parameter sequence [θt,β,R,Tt] can be directly input into the SMPL model; the function of the SMPL model drives the corresponding digital human model according to the human model parameter sequence through linear hybrid skinning (LBS) technology to generate the accurate three-dimensional human mesh Mt corresponding to the monocular image.

[0086] It should be noted that the SMPL model's function uses Linear Blend Skinning (LBS) technology to drive the digital human model to generate 3D mesh vertices v based on the sequence of human model parameters. i The system then uses the 3D mesh vertices vi to generate a precise 3D human body mesh Mt corresponding to the monocular image, where the 3D mesh vertices v i final position v i 'Determined by skin weight and bone transformation, as shown in formula (3).

[0087] v i '=∑w ij ·G j ·(θ,R)·v i ,j=1,2,...,k(3;

[0088] Among them, G j (θ,R) is the global transformation matrix of the j-th bone, k is the total number of bones, and v i ' represents the final position of the i-th 3D mesh vertex v.

[0089] S206: Render the target 3D human body mesh from a monocular image using a virtual camera to obtain the depth map and normal map of the target 3D human body mesh.

[0090] In the specific execution step S206, a corresponding virtual camera can be created according to the camera parameters of the camera that captures the video to be processed, so as to use the virtual camera to perform depth channel rendering and normal channel rendering on the target 3D human body mesh of the monocular image, and obtain the depth map and normal map of the target 3D human body mesh of the monocular image.

[0091] Optionally, the process of rendering the target 3D human body mesh from a monocular image using a virtual camera to obtain the depth map and normal map of the target 3D human body mesh is as follows: create a corresponding virtual camera based on the camera parameters of the camera that captured the video to be processed; use the virtual camera to render the depth value of each pixel in the target 3D human body mesh onto a high-precision texture to obtain the depth map of the target 3D human body mesh; use the virtual camera to calculate the world space normal vector of each pixel in the target 3D human body mesh, and write the world space normal vector of each pixel into another high-precision texture to obtain the normal map of the target 3D human body mesh.

[0092] It should be noted that in order to apply the target 3D human body mesh to the human body surface, the target 3D human body mesh needs to be rendered into a texture first to obtain the corresponding depth map and normal map; specifically, the depth and discovery information of the dynamic digital human mesh (target 3D human body mesh) are rendered into two high-precision textures respectively to obtain the corresponding depth map and normal map.

[0093] In practical applications, firstly, a virtual camera V_virtual is created on the GPU. The intrinsic parameters (FOV, resolution) and pose (extrinsic parameters) of this virtual camera are completely identical to the real main camera V_main (used to capture the video to be processed). Secondly, the driven dynamic mesh Mt (the target 3D human mesh) is fed into the rendering pipeline, and V_virtual is used as the viewing camera for rendering, obtaining the corresponding depth map and normal map. The rendering includes depth rendering (RT) and normal rendering (RT).

[0094] It should be noted that the depth RT rendering process is as follows: using a shader that only writes depth values, the depth value (Z value) of each pixel in Mt is rendered to a high-precision texture (such as R32_FLOAT format) to obtain the depth map Dt. The normal RT rendering process is as follows: using a shader to calculate the world space normal vector (Nx, Ny, Nz) of each fragment (pixel) in Mt, and encoding it (e.g., mapping from [-1,1] to [0,1]) and writing it to another high-precision texture (such as RGB8_UNORM) to obtain the normal map Nt.

[0095] S207: The final image of the monocular image is obtained by using the human face shader, the depth map and normal map of the target 3D human body mesh. The final image includes human face patches and virtual scene.

[0096] In the specific execution step S207, the current pixel of the virtual camera and the pixel coordinates of the bottom corner of the current viewport of the virtual camera are obtained; the corresponding texture coordinates are determined based on the current pixel and pixel coordinates; depth blur calculation is performed using the texture coordinates and the depth value in the depth map to obtain the human face color; the world normal is decoded using the texture coordinates and normal map, and the human face lighting information is determined based on the world normal; the texture coordinates, human face shader, human face color and human face lighting information are fused to obtain the final image of the monocular image.

[0097] In practical applications, such as Figure 5As shown, a virtual camera V_virtual can be created based on the parameters (intrinsic and extrinsic) of the main camera capturing the video to be processed. After obtaining the driven digital human mesh Mt (target 3D human mesh), the virtual camera is used to perform depth channel rendering and normal channel rendering on the target 3D human mesh of the monocular image, respectively, to obtain the depth map Dt and normal map Nt of the target 3D human mesh of the monocular image. The depth map and normal map are sampled in combination with texture coordinates to obtain texture coordinates. The depth map is processed with texture coordinates to obtain the human face color, and the normal map is used to calculate the lighting information of the human face. The human face shader, human face color and human face lighting information are fused to obtain the final image of the monocular image.

[0098] In some embodiments, the process of sampling the depth map and normal map to obtain texture coordinates can be as follows: In Unreal Engine, for RT rendering using SceneCapture2DActor, the corresponding texture coordinates uv are obtained. The generation process of texture coordinate uv is shown in the following code:

[0099] float2uv=(pixel_position+View.ViewRectMin.xy)*View.BufferSizeAndInvSize.zw;

[0100] Where pixel_position is the pixel coordinate of the current pixel of the virtual camera; View.ViewRectMin.xy is the pixel coordinate of the bottom corner of the current viewport of the virtual camera, which is usually (0,0); View.BufferSizeAndInvSize.zw is the coefficient for converting the pixel coordinate of the bottom corner of the current viewport to normalized coordinates.

[0101] In other embodiments, the process of sampling the depth map and normal map to obtain texture coordinates can be as follows: Custom RT rendering using C++'s Draw family of functions is used to obtain texture coordinates (uv). The generation process of texture coordinates (uv) is shown in the following code:

[0102] float2 uv=pixel_position*View.ViewSizeAndInvSize.zw;

[0103] Where pixel_position is the pixel coordinate of the current pixel of the virtual camera, and View.ViewSizeAndInvSize.zw is the coefficient for converting pixel coordinates to normalized coordinates for the rendering target texture used.

[0104] In this embodiment, the depth value in Dt can be used as the true depth of the current pixel based on the focus and aperture parameters of the main camera, participating in depth-of-field blur calculation (such as Circle of Confusion calculation) to obtain the final pixel color (human face color), i.e., human face color = human face color after applying physically accurate blur based on Dt; the world normal decoded in Nt is used to replace the default plane normal of the face, and interacts with the lights in the CG scene (virtual scene) to generate accurate specular highlights, diffuse reflections, and contact shadows (human face lighting information), so that the human body can realistically "blend" into the lighting of the virtual environment. Finally, the human face shader, human face color, and human face lighting information are fused to obtain a final image that integrates accurate depth of field, lighting, and cinematic realism.

[0105] This application provides an image processing method for virtual photography. The method involves acquiring a video to be processed, comprising multiple consecutive monocular images; generating human model parameters based on the monocular images, and generating a 3D human mesh based on these parameters; generating a digital human model of the monocular image using the 3D human mesh; calculating the sequence of human model parameters for each frame of the monocular image; driving the digital human model of the monocular image using the sequence of human model parameters to obtain a target 3D human mesh; rendering the target 3D human mesh of the monocular image using a virtual camera to obtain a depth map and a normal map of the target 3D human mesh; and synthesizing the target 3D human mesh using a human face shader, the depth map, and the normal map of the target 3D human mesh to obtain a final image of the monocular image, wherein the final image includes human face patches and a virtual scene. Therefore, the technical solution provided in this application first uses the human body model parameters of the monocular image to generate a three-dimensional human body mesh of the monocular image, and then uses the three-dimensional human body mesh to generate a digital human model to achieve efficient human body reconstruction; calculates the human body model parameter sequence of the monocular image to achieve vision-based wearless motion capture; uses the human body model parameter sequence of the monocular image to drive the digital human model of the monocular image to quickly generate a digital human model (target three-dimensional human body mesh) with the same height as the photographed human body; uses a virtual camera to render the target three-dimensional human body mesh of the monocular image, and after obtaining the depth map and normal map of the target three-dimensional human body mesh, uses the depth map and normal map in the human body face shader to obtain a final image that integrates accurate depth of field, lighting and cinematic realism, that is, an image that meets the requirements of high-quality shooting, thereby solving the problems in the prior art that cannot produce depth of field effects such as local defocus and focus movement, and cannot simulate the uneven normal distribution of various parts of the human body to correctly receive virtual lighting, and no post-processing is required. Not only is the post-production of depth of field effects and color grading cycle short and low cost, but it also does not rely on manual processing.

[0106] Based on the above embodiments of the present invention, an image processing method for virtual photography is provided. Correspondingly, the embodiments of the present invention also provide an image processing device for virtual photography, such as... Figure 6 As shown, the device includes:

[0107] The acquisition unit 601 is used to acquire the video to be processed, wherein the video to be processed includes a series of multiple monocular images;

[0108] The first generation unit 602 is used to generate human body model parameters based on the monocular image and generate a three-dimensional human body mesh based on the human body model parameters.

[0109] The second generation unit 603 is used to generate a digital human model of a monocular image using a three-dimensional human body mesh of a monocular image.

[0110] The computing unit 604 is used to calculate the sequence of human body model parameters for each frame of monocular image;

[0111] The driving unit 605 is used to drive the digital human model of the monocular image using the human body model parameter sequence of the monocular image to obtain the target three-dimensional human body mesh of the monocular image.

[0112] The rendering unit 606 is used to render the target 3D human body mesh from a monocular image using a virtual camera, and obtain the depth map and normal map of the target 3D human body mesh.

[0113] The processing unit 607 is used to synthesize a final monocular image by using a human face shader, a depth map and a normal map of the target 3D human body mesh, wherein the final image includes human face patches and a virtual scene.

[0114] Optionally, based on the image processing apparatus for virtual shooting shown in the above embodiments of the present invention, the first generation unit 602 is specifically used for:

[0115] The encoder is used to extract features from the monocular image to obtain image features, and the decoder is used to regress the parameters of the human body model from the image features.

[0116] The three-dimensional mesh vertices are generated by using the human body model parameters through a preset parametric model, and the three-dimensional mesh vertices are then globally transformed using global transformation parameters to generate a three-dimensional human body mesh.

[0117] Optionally, based on the image processing apparatus for virtual shooting shown in the above embodiments of the present invention, the computing unit 604 is specifically used for:

[0118] For each frame of monocular image, the monocular image is downsampled to obtain the target monocular image;

[0119] The encoder is used to extract features from the target monocular image to obtain the target image features, and the decoder is used to regress the parameterization of the target human body model parameters from the target image features.

[0120] The target human body model parameters are filtered, and a specified neural network is used to smooth the target human body model parameters in a sliding window to obtain the human body model parameter sequence of the target monocular image.

[0121] Optionally, based on the image processing apparatus for virtual shooting shown in the above embodiments of the present invention, the rendering unit 606 is specifically used for:

[0122] Create a corresponding virtual camera based on the camera parameters of the camera that captured the video to be processed;

[0123] Using a virtual camera, the depth value of each pixel in the target 3D human body mesh is rendered into a high-precision texture to obtain the depth map of the target 3D human body mesh.

[0124] The world space normal vector of each pixel in the target 3D human body mesh is calculated using a virtual camera, and then written into another high-precision texture to obtain the normal map of the target 3D human body mesh.

[0125] Optionally, based on the image processing apparatus for virtual shooting shown in the above embodiments of the present invention, the processing unit 607 is specifically used for:

[0126] Get the current pixel of the virtual camera and the pixel coordinates of the bottom corner of the current viewport of the virtual camera;

[0127] Determine the corresponding texture coordinates based on the current pixel and pixel coordinates;

[0128] Depth blur is calculated using texture coordinates and depth values ​​in the depth map to obtain the color of the human face patch;

[0129] World normals are decoded using texture coordinates and normal maps, and lighting information for human face patches is determined based on world normals;

[0130] The final monocular image is obtained by fusing texture coordinates, human face shaders, human face colors, and human face lighting information.

[0131] This application also provides a storage medium storing program instructions, which, when loaded and executed by a processor, implement any of the above-described embodiments of the image processing method for virtual shooting.

[0132] This application also provides an electronic device, such as Figure 7As shown, the device includes a processor 701 and a memory 702, which are connected via a bus; the memory stores program instructions; the processor calls the program instructions in the memory to execute any of the above-described embodiments of the image processing method for virtual shooting.

[0133] The processor mentioned in this article can be the terminal's CPU, an integrated MCU within the terminal, or a combination of a CPU and an MCU. Furthermore, the processor contains a kernel that retrieves the corresponding program from memory; one or more kernels can be configured.

[0134] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0135] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0136] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0137] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0138] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An image processing method for virtual photography, characterized in that, The method includes: Acquire the video to be processed, wherein the video to be processed includes: a series of multiple monocular images; Human body model parameters are generated based on the monocular image, and a three-dimensional human body mesh is generated based on the human body model parameters; Using the three-dimensional human body mesh of the monocular image, a digital human model of the monocular image is generated; Calculate the sequence of human body model parameters for each frame of the monocular image; The human body model parameter sequence of the monocular image is used to drive the digital human model of the monocular image to obtain the target three-dimensional human body mesh of the monocular image; Rendering a target 3D human body mesh from a monocular image using a virtual camera to obtain a depth map and a normal map of the target 3D human body mesh; rendering a target 3D human body mesh from a monocular image using a virtual camera to obtain a normal map of the target 3D human body mesh includes: calculating the world space normal vector of each pixel in the target 3D human body mesh using the virtual camera, and writing the world space normal vector of each pixel into another high-precision texture to obtain a normal map of the target 3D human body mesh; Obtain the current pixel of the virtual camera and the pixel coordinates of the bottom corner of the current viewport of the virtual camera; Determine the corresponding texture coordinates based on the current pixel and the pixel coordinates; The depth-of-field blur is calculated using the texture coordinates, the depth values ​​in the depth map, and the focus and aperture parameters of the main camera corresponding to the virtual camera to obtain the color of the human face patch. The world normal is decoded using the normal map, and the human face patch lighting information is determined based on the texture coordinates and the world normal. The texture coordinates, human face shader, human face color, and human face lighting information are fused to obtain the final image of the monocular image, wherein the final image includes the human face and the virtual scene.

2. The method according to claim 1, characterized in that, Generating human body model parameters from the monocular image, and generating a 3D human body mesh from the human body model parameters, including: The encoder is used to extract features from the monocular image to obtain image features, and the decoder is used to regress the parameterization of the human body model parameters from the image features. A three-dimensional mesh is generated by using the parameters of the human body model through a preset parametric model, and then the three-dimensional mesh vertices are globally transformed using global transformation parameters to generate a three-dimensional human body mesh.

3. The method according to claim 1, characterized in that, Calculate the sequence of human model parameters for each frame of the monocular image, including: For each frame of the monocular image, the monocular image is downsampled to obtain the target monocular image; The target monocular image is extracted using an encoder to obtain target image features, and the target human body model parameters are regressed and parameterized from the target image features using a decoder. The target human body model parameters are filtered, and a specified neural network is used to smooth the target human body model parameters in a sliding window to obtain the human body model parameter sequence of the target monocular image.

4. The method according to claim 1, characterized in that, The target 3D human body mesh of the monocular image is rendered using a virtual camera to obtain the depth map and normal map of the target 3D human body mesh, including: Create a corresponding virtual camera based on the camera parameters of the camera that captured the video to be processed; The depth value of each pixel in the target 3D human body mesh is rendered into a high-precision texture using the virtual camera to obtain the depth map of the target 3D human body mesh.

5. An image processing apparatus for virtual shooting, characterized in that, The device includes: An acquisition unit is used to acquire a video to be processed, wherein the video to be processed includes a series of consecutive monocular images; The first generation unit is used to generate human body model parameters based on the monocular image, and to generate a three-dimensional human body mesh based on the human body model parameters. The second generation unit is used to generate a digital human model of the monocular image using the three-dimensional human body mesh of the monocular image; A computing unit is used to calculate the sequence of human model parameters for each frame of the monocular image; A driving unit is used to drive the digital human model of the monocular image using the human body model parameter sequence of the monocular image to obtain the target three-dimensional human body mesh of the monocular image. The rendering unit is used to render the target 3D human body mesh of the monocular image using a virtual camera to obtain the depth map and normal map of the target 3D human body mesh; wherein, rendering the target 3D human body mesh of the monocular image using a virtual camera to obtain the normal map of the target 3D human body mesh includes: calculating the world space normal vector of each pixel in the target 3D human body mesh using the virtual camera, and writing the world space normal vector of each pixel into another high-precision texture to obtain the normal map of the target 3D human body mesh; The processing unit is configured to acquire the current pixel of the virtual camera and the pixel coordinates of the bottom corner of the current viewport of the virtual camera; determine the corresponding texture coordinates based on the current pixel and the pixel coordinates; perform depth-of-field blur calculation using the texture coordinates, the depth value in the depth map, and the focus and aperture parameters of the main camera corresponding to the virtual camera to obtain the human face color; decode the world normal using the normal map, and determine the human face lighting information based on the texture coordinates and the world normal; and fuse the texture coordinates, the human face shader, the human face color, and the human face lighting information to obtain the final image of the monocular image, wherein the final image includes the human face and the virtual scene.

6. The apparatus according to claim 5, characterized in that, The first generation unit is specifically used for: The encoder is used to extract features from the monocular image to obtain image features, and the decoder is used to regress the parameterization of the human body model parameters from the image features. A three-dimensional mesh is generated by using the parameters of the human body model through a preset parametric model, and then the three-dimensional mesh vertices are globally transformed using global transformation parameters to generate a three-dimensional human body mesh.

7. The apparatus according to claim 5, characterized in that, The computing unit is specifically used for: For each frame of the monocular image, the monocular image is downsampled to obtain the target monocular image; The target monocular image is extracted using an encoder to obtain target image features, and the target human body model parameters are regressed and parameterized from the target image features using a decoder. The target human body model parameters are filtered, and a specified neural network is used to smooth the target human body model parameters in a sliding window to obtain the human body model parameter sequence of the target monocular image.

8. An electronic device, characterized in that, include: A processor and a memory are connected via a bus; wherein the processor is used to call and execute a program stored in the memory; The memory is used to store a program for implementing the image processing method for virtual shooting as described in any one of claims 1-4.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for performing the image processing method for virtual shooting as described in any one of claims 1-4.

Images