Three-dimensional virtual human video synthesis method, system and device and storage medium

Abstract

The embodiment of the invention provides a three-dimensional virtual human video synthesis method, system and device and a storage medium, and the method comprises the steps: employing a unified camera position to separately shoot a site video and a field video, carrying out the detection of a person in each frame of image of the field video, and generating a pixel-level mask of each human body; on the basis of a parameterized SMPL human body model, registering human body contours formed by masks in each frame of mask grey map to obtain SMPL parameters of each registered human body; and taking the registered SMPL parameters of each human body as a new view angle of the implicit neural radiation field, synthesizing a video picture under the new view angle under volume rendering, and fusing the video picture with a three-dimensional virtual human to generate a video. The problem that in an existing method, registration optimization of a parameterized three-dimensional human body and the contour of human body segmentation is difficult to achieve is avoided, and the visual defect that in an existing image synthesis method, the posture of the virtual human in a video picture is adjusted only based on the human body posture, and consequently the virtual human foot suspension or foot sliding animation is likely to be generated can be avoided.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of multimodal fusion AI digital human technology, and specifically to a three-dimensional virtual human video synthesis method, system, device and storage medium. Background Art

[0002] The main technologies for automatically generating digital humans by replacing the characters in real-life videos with 3D virtual images are: based on adversarial generative networks, such as pix2pix, vid2vid, and Vid2Game frameworks. This type of method mainly transfers the human body posture of a character in a video to the character in the video, and this type of method has certain feasibility.

[0003] Although it is possible to imitate the movement of people in the video and synthesize the video, when replacing the people in the video with 3D virtual images, the outline of the people and the virtual image cannot be aligned, and the feet of the virtual image will be suspended from the ground. The main reason is that this type of method cannot achieve the technical problem of alignment and registration between the outline of the person and the virtual image. Summary of the invention

[0004] To this end, the embodiments of the present invention provide a three-dimensional virtual human video synthesis method, system, device and storage medium to solve the current technical problem of using 3D virtual images to replace the characters in real-life videos to synthesize videos, and being unable to achieve alignment and registration between the character outline and the virtual image.

[0005] In order to achieve the above purpose, the embodiment of the present invention provides the following technical solutions: According to a first aspect of an embodiment of the present invention, an embodiment of the present application provides a method for synthesizing a three-dimensional virtual human video, the method comprising: Receive venue and live video shot from the same camera position; Converting the scene described by the venue video into an implicit neural radiation field; Detect and track the people in each frame of the live video image, obtain the boundary annotation box of each individual human body and record the corresponding identification ID; Perform human body segmentation on all areas within the boundary annotation boxes in each frame of the live video image, generate pixel-level masks of each human body, and obtain the corresponding mask gray map; Based on a parameterized SMPL human body model, the human body contour formed by the mask in the mask gray image of each frame is registered and the SMPL parameters are updated to obtain the SMPL parameters of each human body after registration; The SMPL parameters of each human body after registration are used as the new perspective of the implicit neural radiation field, and the video images under the new perspective are synthesized under volume rendering; The three-dimensional virtual human provided by the user is integrated with the synthesized video image to generate a video; The SMPL parameters of each human body after registration are used as the new perspective of the implicit neural radiation field, and the video images under the new perspective are synthesized under volume rendering, including: The SMPL parameters of each human body after registration are used as new perspective data, and added to the camera perspective track corresponding to the site video in the form of a new camera perspective; Based on the guidance of the mask gray map, rays are emitted in the neural radiation field, volume rendering is performed, the designated person in the live video is removed, and the video picture under a new perspective is synthesized.

[0006] Furthermore, the camera viewing angle trajectory of the on-site video and the camera viewing angle trajectory of the venue video overlap in space.

[0007] Furthermore, the scene described by the venue video is converted into an implicit neural radiation field, including: Acquire each frame of the venue video image for the venue video; Based on the video images of each frame of the venue, the relative position of the camera during imaging is estimated, so as to construct a sparse three-dimensional structure of the venue scene space; Obtaining the three-dimensional spatial position of the captured site video scene and the camera posture of each frame of the site video image; wherein the camera posture is represented by the camera intrinsic parameters and extrinsic parameters in the pinhole camera model of the corresponding frame of the site video image; Encoding the three-dimensional spatial position through a multi-layer perception network; The camera posture is used as the camera view angle, and rays are emitted from each three-dimensional space position along each camera view angle, and the density and color of the corresponding pixel point are obtained by integrating the rays.

[0008] Furthermore, the person in each frame of the live video image is detected and tracked, and the boundary annotation box of each individual person is obtained and the corresponding identification ID is recorded, including: Acquire a first preset number of pre-labeled images having human bodies; Draw a line from the top to the middle of the bottom of each human head as the dimension of the annotated bounding box; generating a width of the bounding box with a preset aspect ratio based on the dimensions of the bounding box; Determine the bounding box of the human body to be annotated by using the length and width of the bounding box, and annotate a complete bounding box for each individual human body in each pre-annotated image; Based on the annotated complete bounding box, crop each human body instance with a bounding box from the pre-annotated image, save the human body instance as training data, and form a training data set; Using the training data set to train a human body detection algorithm model; Input each frame of the live video image into the trained human body detection algorithm model, annotate the boundary annotation box for each individual human body and record the corresponding identification ID.

[0009] Furthermore, human body segmentation is performed on all regions within the boundary annotation boxes in each frame of the live video image to generate pixel-level masks of each human body and obtain corresponding mask gray images, including: Extract feature points from the areas within each boundary annotation box in each frame of the live video image to obtain a corresponding feature map; Setting a second preset number of first target regions for each point in each feature map as candidate target regions; Send the candidate target regions to the region generation network for binary classification and bounding box regression, and filter out some candidate target regions; Match the pixels of each frame of the live video image with the corresponding feature map, perform an alignment operation on the remaining candidate target areas, and obtain the corresponding second target area; Performing mask generation on the second target area to obtain a corresponding mask gray image; The masked area in the masked gray image is black, and the corresponding value of the masked area is 0; the non-masked area in the masked gray image is white, and the corresponding value of the non-masked area is 1.

[0010] Furthermore, based on a parameterized SMPL human body model, the human body contour formed by the mask in the mask gray image of each frame is registered and the SMPL parameters are updated to obtain the SMPL parameters of each human body after registration, including: Extracting the human body contour formed by the mask from the mask gray image of each frame to obtain the corresponding first human body two-dimensional contour image; Using the camera view and camera posture rotation matrix R and translation matrix T, the SMPL parameters of the SMPL three-dimensional human body model are initialized to obtain the first global human body posture data and the first skeletal joint posture data; Driven by the first skeletal joint posture data, the SMPL three-dimensional human body model is deformed by skinning to obtain a new SMPL three-dimensional human body model; Using the first global human body posture data as a projection matrix, rendering through a pinhole camera perspective projection, to obtain a second two-dimensional human body contour image; Obtaining a similarity error between the second two-dimensional human body contour image and the first two-dimensional human body contour image by a least square method; Determine whether the similarity error is less than a preset threshold; If the similarity error is greater than or equal to a preset threshold, registering the second two-dimensional human body contour map with the first two-dimensional human body contour map, updating the SMPL parameters, and obtaining second global human body posture data and second skeletal joint posture data; The first global human body posture data and the first skeletal joint posture data are respectively updated to the second global human body posture data and the second skeletal joint posture data, and the process is repeated until the SMPL three-dimensional human body model is deformed by skinning under the drive of the first skeletal joint posture data to obtain a new SMPL three-dimensional human body model; If the similarity error is less than a preset threshold, the currently updated SMPL parameters are used as the registered SMPL parameters.

[0011] Further, rays are emitted at pixel positions where the mask value is 1 in the mask gray map, and rays are not emitted at pixel positions where the mask value is 0 in the mask gray map.

[0012] Furthermore, the method further comprises: The rotation of each joint of the SMPL human body model is applied to the corresponding joint of the three-dimensional virtual human, so as to realize the use of real body movements to drive the body animation of the three-dimensional virtual human.

[0013] According to a second aspect of an embodiment of the present invention, an embodiment of the present application provides a three-dimensional virtual human video synthesis system, the system comprising: A receiving module, used for receiving the site video and the on-site video shot by the same camera position; A scene conversion module, used to convert the scene described by the venue video into an implicit neural radiation field; The human body detection and tracking module is used to detect and track the people in each frame of the live video image, obtain the boundary annotation box of each individual human body and record the corresponding identification ID; The human body segmentation module is used to perform human body segmentation on the areas within all boundary annotation boxes in each frame of the live video image, generate pixel-level masks of each human body, and obtain the corresponding mask gray map; A registration module is used to register the human body contour formed by the mask in the mask gray image of each frame based on a parameterized SMPL human body model and update the SMPL parameters to obtain the SMPL parameters of each human body after registration; A new perspective synthesis module is used to use the SMPL parameters of each human body after registration as the new perspective of the implicit neural radiation field, and synthesize the video images under the new perspective under volume rendering; A synthesis module is used to fuse the 3D virtual human provided by the user with the synthesized video image to generate a video; The SMPL parameters of each human body after registration are used as the new perspective of the implicit neural radiation field, and the video images under the new perspective are synthesized under volume rendering, including: The SMPL parameters of each human body after registration are used as new perspective data, and added to the camera perspective track corresponding to the site video in the form of a new camera perspective; Based on the guidance of the mask gray map, rays are emitted in the neural radiation field, volume rendering is performed, the designated person in the live video is removed, and the video picture under a new perspective is synthesized.

[0014] Furthermore, the system also includes: a control module, which is used to apply the rotation of each joint of the SMPL human model to the corresponding joint of the three-dimensional virtual human, so as to achieve the use of real body movements to drive the body animation of the three-dimensional virtual human.

[0015] According to a third aspect of an embodiment of the present invention, there is provided a 3D virtual human video synthesis device, the device comprising: a processor and a memory; The memory is used to store one or more program instructions; The processor is used to run one or more program instructions to execute the steps of a three-dimensional virtual human video synthesis method as described in any one of the above items.

[0016] According to a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of a three-dimensional virtual human video synthesis method as described in any one of the above items are implemented.

[0017] Compared with the prior art, the embodiment of the present application provides a method, system, device and storage medium for synthesizing a three-dimensional virtual human video. The method adopts a unified camera position to shoot the site video and the live video separately, detects the characters in each frame of the live video, and generates a pixel-level mask for each human body; based on the parameterized SMPL human body model, the human body contours formed by the mask in the gray image of each frame mask are aligned to obtain the SMPL parameters of each human body after alignment; the SMPL parameters of each human body after alignment are used as a new perspective of the implicit neural radiation field, and the video screen under the new perspective is synthesized under volume rendering, and the video is fused with the three-dimensional virtual human to generate a video. Avoiding the alignment optimization of the contours of the parameterized three-dimensional human body and the human body segmentation that is difficult to achieve in the existing methods can avoid the visual defects of the virtual human's "feet hanging in the air" or "foot sliding" that are easy to produce in the existing image synthesis method, which is based only on the human body posture to adjust the posture of the virtual human in the video screen. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0019] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0020] Figure 1 A schematic diagram of the logical structure of a three-dimensional virtual human video synthesis system provided by an embodiment of the present invention; Figure 2 A schematic diagram of a flow chart of a three-dimensional virtual human video synthesis method provided by an embodiment of the present invention; Figure 3 A schematic diagram of a process for converting a scene described by the venue video into an implicit neural radiation field provided by an embodiment of the present invention; Figure 4 A schematic diagram of a process for detecting and tracking a person in each frame of a live video image provided by an embodiment of the present invention; Figure 5 A schematic diagram of a process for performing human body segmentation on all areas within the boundary annotation boxes in each frame of a live video image provided by an embodiment of the present invention; Figure 6 A schematic diagram of a process for registering a human body contour formed by a mask in a mask gray image of each frame based on a parameterized SMPL human body model and updating SMPL parameters is provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] The purpose of this application is to solve the technical problem that the current use of 3D virtual images to replace the synthetic video of the characters in the real-life video cannot achieve the alignment and registration between the character outline and the virtual image.

[0023] In order to solve the above technical problems, Figure 1 As shown, an embodiment of the present application provides a three-dimensional virtual human video synthesis system, which is applied in the cultural and tourism metaverse to replace the characters in the real-life video with 3D virtual images and automatically generate digital humans.

[0024] Specifically, a three-dimensional virtual human video synthesis system provided in an embodiment of the present application includes: a receiving module 1, a scene conversion module 2, a human body detection and tracking module 3, a human body segmentation module 4, a registration module 5, a new perspective synthesis module 6 and a synthesis module 7.

[0025] Furthermore, the receiving module 1 is used to receive the venue video and the live video shot by the same camera position; the scene conversion module 2 is used to convert the scene described by the venue video into an implicit neural radiation field; the human body detection and tracking module 3 is used to detect and track the characters in each frame of the live video image, obtain the boundary annotation box of each individual human body and record the corresponding identification ID; the human body segmentation module 4 is used to perform human body segmentation on the areas within all boundary annotation boxes in each frame of the live video image, generate a pixel-level mask of each human body, and obtain a corresponding mask gray map; the registration module 5 is used to register the human body contour formed by the mask in the mask gray map of each frame based on a parameterized SMPL human body model and update the SMPL parameters to obtain the SMPL parameters of each human body after registration; the new perspective synthesis module 6 is used to use the SMPL parameters of each human body after registration as the new perspective of the implicit neural radiation field, and synthesize the video picture under the new perspective under volume rendering; the synthesis module 7 is used to fuse the three-dimensional virtual person provided by the user with the synthesized video picture to generate a video.

[0026] Compared with the prior art, the three-dimensional virtual human video synthesis system provided by the embodiment of the present application adopts a unified camera position to shoot the site video and the on-site video separately, detects the characters in each frame of the on-site video, and generates a pixel-level mask for each human body; based on the parameterized SMPL human body model, the human body contours formed by the mask in the gray image of each frame mask are registered to obtain the SMPL parameters of each human body after registration; the SMPL parameters of each human body after registration are used as the new perspective of the implicit neural radiation field, and the video screen under the new perspective is synthesized under volume rendering, and the video is fused with the three-dimensional virtual human to generate the video. Avoiding the registration optimization of the contours of the parameterized three-dimensional human body and the human body segmentation that is difficult to achieve in the existing methods can avoid the visual defects of the virtual human's "feet hanging in the air" or "foot sliding" that are easy to produce in the existing image synthesis method, which is based only on the human body posture to adjust the posture of the virtual human in the video screen.

[0027] "Foot sliding" is a common visual defect in 3D animation or virtual character movements. It refers to the unnatural animation effect of the character's feet contacting the ground when walking, running or other movements. It looks like sliding on the ground (as if "the soles of the feet are slipping") instead of actually interacting with the ground mechanically. This problem will destroy the realism and immersion of the action.

[0028] Furthermore, the 3D virtual human video synthesis system provided in the embodiment of the present application further includes: a control module 8, which is used to apply the rotation of each joint of the SMPL human model to the corresponding joint of the 3D virtual human, so as to realize the use of real human body movements to drive the body animation of the 3D virtual human. The synthesized virtual human body movements can be automatically controlled or secondary controlled.

[0029] Corresponding to the above disclosed 3D virtual human video synthesis system, the embodiment of the present invention further discloses a 3D virtual human video synthesis method. The following describes in detail the 3D virtual human video synthesis method disclosed in the embodiment of the present invention in combination with the above described 3D virtual human video synthesis system.

[0030] like Figure 2 As shown, the specific steps of a three-dimensional virtual human video synthesis method provided in an embodiment of the present application are described in detail below.

[0031] A three-dimensional virtual human video synthesis method provided in an embodiment of the present application is applied in the cultural and tourism metaverse to replace the characters in the real-life video with 3D virtual images and automatically generate digital humans.

[0032] The receiving module 1 receives the venue video and the on-site video shot by the same camera position.

[0033] In the embodiment of the present invention, a camera is used to shoot a video of the venue, which requires that the camera angles between video frames overlap and form a smooth angle track. In practice, a photography mechanical arm is generally used to complete the panoramic shooting of the interior of the venue.

[0034] In addition, the live video is shot with the same camera position as the site video. The same camera position ensures a unified camera internal parameter, which is critical for human posture estimation, and also ensures that the view trajectory of the live video camera overlaps with the view trajectory of the site video camera in space, which is critical for the process of synthesizing new view angles from the neural radiation field.

[0035] The scene described by the venue video is converted into an implicit neural radiation field by the scene conversion module 2.

[0036] In an embodiment of the present invention, the scene described by the venue video is converted into an implicit neural radiance field representation. A neural rendering algorithm is used, for example, the Instant-NGP (Instant Neural Graphics Primitives) method is used, which has a fast calculation speed, to convert the scene described by the venue video into a neural radiance field (NERF). As an implicit representation, NERF encodes the three-dimensional spatial position of the captured scene through a multi-layer perceptual network, and emits rays from each spatial position along the camera's viewing angle. The density and color of the pixel points can be obtained by integrating the rays. In an embodiment of the present invention, a structure from motion algorithm can be used, which estimates the relative pose / camera posture of the camera during imaging by taking a series of ordinary RGB images, thereby constructing a sparse three-dimensional structure of the scene space.

[0037] refer to Figure 3 The above steps specifically include: obtaining each frame of the venue video image for the venue video; estimating the relative position of the camera during imaging based on each frame of the venue video image, so as to construct a sparse three-dimensional structure of the venue scene space; obtaining the three-dimensional space position of the captured venue video scene and the camera posture of each frame of the venue video image; encoding the three-dimensional space position through a multi-layer perception network; using the camera posture as the camera angle of view, emitting rays from each three-dimensional space position along each camera angle of view, and integrating on the rays to obtain the density and color of the corresponding pixel points.

[0038] In an embodiment of the present application, the three-dimensional spatial position of the captured site video scene is obtained by a structure from motion (SFM) algorithm. Structure from motion (SFM) is to estimate the position of 3D points by giving a sparse corresponding set of multiple images and their image features. This solution process usually involves simultaneous estimation of 3D geometry (structure) and camera posture (motion). Structure from motion (SFM) has been one of the hot research directions in the field of computer vision for decades, and has achieved many practical applications, especially in close-range three-dimensional reconstruction. The algorithm is triggered by a series of images of the target object, and finally obtains a sparse three-dimensional point cloud of the target object with higher precision. Among them, the camera posture is represented by the camera intrinsic parameters and extrinsic parameters in the pinhole camera model of the corresponding frame site video image. The camera intrinsic parameters, that is, the camera intrinsic parameter matrix, are the change relationship between the 3D coordinates in the camera coordinate system and the pixel coordinates. The camera extrinsic parameters, that is, the camera extrinsic matrix, describe the process of converting the coordinates in the world coordinate system to the coordinates in the camera coordinate system. Both coordinate systems are three-dimensional, and the conversion process is a rigid body transformation of rotation and translation operations.

[0039] The human body detection and tracking module 3 detects and tracks the human body in each frame of the live video image, obtains the boundary annotation frame of each individual human body and records the corresponding identification ID.

[0040] As a method for implementing the above steps in the embodiment of the present invention, a human body detection and tracking algorithm based on a deep model is applied to track people in the video in real time. For example, a YOLOv5 model trained from video data with human body bounding boxes annotated is used as a human body detection algorithm, and a classic Kalman filter is used as a human body tracking algorithm.

[0041] refer to Figure 4 In an embodiment of the present invention, the above steps specifically include: obtaining a first preset number of pre-annotated images with human bodies; drawing a line from the top of each human head to the middle of the bottom as the degree of the annotated bounding box; generating the width of the bounding box based on the degree of the bounding box with a preset aspect ratio; using the degree and width of the bounding box, determining the bounding box of the human body to be annotated, and annotating a complete bounding box for each individual human body in each pre-annotated image; based on the annotated complete bounding box, cropping each human body instance with a bounding box from the pre-annotated image, saving it as training data, and forming a training data set; using the training data set to train a human body detection algorithm model; inputting each frame of the live video image into the trained human body detection algorithm model, annotating a bounding annotation box for each individual human body and recording the corresponding identification ID.

[0042] In an embodiment of the present invention, the first preset number is greater than or equal to 1000. In order to obtain a training data set for a human body detection algorithm model, no less than 1000 pre-labeled images with human bodies are required.

[0043] In the embodiment of the present invention, the specific method of annotating the training data set is as follows: annotating the complete bounding box of each individual human body in the image in detail. If there is partial occlusion, the annotator is required to complete the invisible part and draw a complete bounding box.

[0044] In addition, the output boundary annotation box of the marked human body and the corresponding identification ID are saved as required for the subsequent steps of the embodiment of the present invention, wherein the identification ID will be used to specify the person who needs to be replaced by the virtual person, and the boundary box reduces the requirements for the accuracy of the human body detection algorithm, and only a coarse-grained quadrilateral boundary box is required, rather than a pixel-level human body boundary contour.

[0045] The human body segmentation module 4 performs human body segmentation on all areas within the boundary annotation boxes in each frame of the live video image, generates a pixel-level mask of each human body, and obtains a corresponding mask gray map.

[0046] In the embodiment of the present invention, the human body posture includes global human body posture and skeletal joint posture. Since the human body boundary annotation box is pre-provided in the aforementioned human body detection and tracking steps, it is only necessary to perform human body posture estimation on the area within the boundary annotation box.

[0047] Specifically, a human body segmentation algorithm is implemented on all areas within the boundary annotation boxes in each frame of the live video image. For example, the classic Mask R-CNN deep model is a human body segmentation algorithm suitable for the embodiment of the present application, thereby segmenting the human body and generating a pixel-level mask corresponding to each human body, which is represented by a semi-transparent color area superimposed on the human body.

[0048] refer to Figure 5 The above steps specifically include: extracting feature points from the areas within each boundary annotation box in each frame of the live video image to obtain a corresponding feature map; setting a second preset number of first target areas for each point in each feature map as candidate target areas; sending the candidate target areas to the region generation network for binary classification and bounding box regression, filtering out a part of the candidate target areas; matching the pixels of each frame of the live video image with the corresponding feature map, aligning the remaining candidate target areas, and obtaining the corresponding second target area; performing mask generation on the second target area to obtain a corresponding mask gray map; wherein the mask area in the mask gray map is black, and the corresponding value of the mask area is 0, and the non-mask area in the mask gray map is white, and the corresponding value of the non-mask area is 1.

[0049] In the embodiment of the present application, human body detection and segmentation are achieved through a target detection model. Target detection refers to the calibration of the bounding boxes and corresponding categories of all objects in a given image when the categories of all target objects have been determined. The input of the model is a picture, and the output consists of the following three parts. (1) The bounding box of each object. The bounding box refers to a rectangular box that limits the position range of the object. The model determines the bounding box corresponding to each object in the form of the two-dimensional coordinates of the output rectangular box in the picture. If the model outputs a bounding box, it means that it believes that there is an object in the bounding box. (2) The category probability of each object. The category probability refers to the probability that the object in the bounding box belongs to a certain category. (3) The confidence probability of each object. The confidence probability refers to the probability that the object in the bounding box exists.

[0050] A pre-trained neural convolution network is used to extract feature points from three dimensions of skin color, contour, and texture of the human body to obtain a feature map. In an embodiment of the present application, the first target area or the candidate target area is also obtained through a prediction model, and the prediction model is obtained by training a positive sample set and a negative sample set. For example, samples corresponding to a confidence probability value greater than or equal to 0.7 and less than or equal to 0.8 are taken to form a positive sample set, and samples corresponding to a confidence probability value less than 0.7 are taken to form a negative sample set. The second preset number is set based on the number of human bodies in the live video image. For example, the second preset number is greater than or equal to the number of human bodies in the live video image.

[0051] In addition, the region generation network is used to perform binary classification on the candidate target region in order to determine whether the candidate target region is the foreground or the background. For example, the average value of the confidence probability value of each candidate target region is used for binary classification. The bounding box regression is to identify the boundary feature points, and the identified boundary feature points can be used to obtain the boundary of the segmented human body.

[0052] The registration module 5 registers the human body contour formed by the mask in the mask gray image of each frame based on a parameterized SMPL (Skinned Multi-Person Linear) human body model and updates the SMPL parameters to obtain the registered SMPL parameters of each human body.

[0053] The SMPL human model is a statistical model that includes a set of posture parameters with 24×3 dimensional numbers to describe the human body's motion posture at a certain moment, where 24 represents 24 defined human joints. The SMPL parameters are a set of data representing the global human body posture and skeletal joint posture, so that the human body posture data can be directly obtained. As a three-dimensional human body model, SMPL can be perspective projected through the pinhole camera model. Here, the global human body posture data is used as the perspective projection matrix to obtain a two-dimensional contour map of the SMPL human body model on a two-dimensional plane. The SMPL parameters are continuously updated to obtain a new human body three-dimensional model under the new skeletal joint posture. In this way, the new contour map is iteratively obtained under the updated perspective projection and is aligned with the mask contour map until the two contour maps are highly overlapped. The current SMPL parameters are the current human body posture data. The perspective projection matrix is ​​the global human body posture data, which is composed of the internal and external parameters of the pinhole camera model, where the external parameters are the R and T of the posture. The skeletal joint posture data is represented according to SMPL. Registration is an iterative optimization process. The optimization objective function takes the global human body posture data and the skeletal joint posture data as parameters, and minimizes the registration error as the goal.

[0054] refer to Figure 6The above steps specifically include: extracting the human body contour formed by the mask from the mask gray image of each frame to obtain the corresponding first human body two-dimensional contour image; using the camera view and camera posture rotation matrix R and translation matrix T to initialize the SMPL parameters of the SMPL three-dimensional human body model to obtain the first global human body posture data and the first bone joint posture data; under the drive of the first bone joint posture data, the SMPL three-dimensional human body model is deformed by skinning to obtain a new SMPL three-dimensional human body model; using the first global human body posture data as the projection matrix, rendering through pinhole camera perspective projection, to obtain a second human body two-dimensional contour image; using the least squares method to obtain the relationship between the second human body two-dimensional contour image and the first human body two-dimensional contour image similarity error; judging whether the similarity error is less than a preset threshold; if the similarity error is greater than or equal to the preset threshold, registering the second two-dimensional human body contour map with the first two-dimensional human body contour map, updating the SMPL parameters, and obtaining the second global human body posture data and the second skeletal joint posture data; updating the first global human body posture data and the first skeletal joint posture data to the second global human body posture data and the second skeletal joint posture data respectively, and looping to the SMPL three-dimensional human body model driven by the first skeletal joint posture data to complete the deformation through skinning, and obtain a new SMPL three-dimensional human body model; if the similarity error is less than the preset threshold, using the currently updated SMPL parameters as the registered SMPL parameters.

[0055] Specifically, contour extraction is first performed, that is, edge information is extracted using the canny operator, etc., and then the edges are sampled and converted into sampling points, and the context information of each point and the cost matrix are calculated; finally, the Hungarian matching algorithm is used for point matching. The degree of matching is determined by shape similarity evaluation, for example, by calculating the differences in features such as shape, area, and perimeter of the contour and comparing them. Repeat the above steps until the similarity error is less than the preset threshold, which can be specifically set to 0.001. If it is not met, the global human posture data and bone joint posture data are optimized and updated.

[0056] The new perspective synthesis module 6 uses the SMPL parameters of each human body after registration as the new perspective of the implicit neural radiation field, and synthesizes the video picture under the new perspective under volume rendering.

[0057] Specifically, the above steps specifically include: using the SMPL parameters of each human body after registration as new perspective data, and adding it to the camera perspective trajectory corresponding to the site video in the form of a new camera perspective; based on the guidance of the mask gray map, emitting rays in the neural radiation field, performing volume rendering, removing the specified person in the live video, and synthesizing the video screen under the new perspective; wherein, in the mask gray map, rays are emitted at the pixel position where the mask map value is 1, and in the mask gray map, rays are not emitted at the pixel position where the mask map value is 0.

[0058] The following is a description of the specific process of adding global human posture data to the camera view trajectory corresponding to the site video. The global human posture data is represented by the internal and external parameters in the pinhole camera model. The internal parameter is represented by the internal parameter of the camera posture corresponding to the image where the human body is located, and the external parameter is the matrix composed of the rotation R and translation T of the global human posture. Representation, so that the global human pose data and the existing camera pose use a common representation. The global human pose data and camera pose data are normalized, and a unit sphere is used as a three-dimensional geometric body to represent the three-dimensional space of the scene. This is also the representation method used in panoramic videos. The specific method of normalization is to first calculate the center position of all cameras c The average . Then all camera centers are moved to the unit sphere with the average value as the center, and the calculation is Get one , and find the norm of the matrix according to the column vector and get the maximum value of the norm vector Finally, the vector that needs to be offset from the camera center is , scaling factor: . Thus the camera center is given by the formula Normalized to the unit sphere. The camera is visualized using a frustum, where the four vertices of the far clipping plane are determined by the horizontal and vertical fields of view and the frustum origin. The distance from the origin to the far clipping plane is l , the value range is The half width (halfWidth) and half height (halfHeight) of the far clipping plane are determined by the field of view angle and the distance from the origin to the far clipping plane. l Decide:

[0059] in: is the horizontal viewing angle, is the vertical viewing angle, and It must be in radians. The four vertices are: upper left vertex (-halfWidth, halfHeight, l), upper right vertex (halfWidth, halfHeight, l), lower right vertex (halfWidth, -halfHeight, l), and lower left vertex (-halfWidth, -halfHeight, l). Finally, the view cone is rotated and translated according to the normalized six degrees of freedom (6-DOF) of the camera, and the camera perspective is visualized. Adding global human pose data to the existing camera perspective trajectory can be directly reflected by the position of the global human pose in the sphere.

[0060] According to the human posture data provided in the previous steps, a neural rendering algorithm is used to synthesize the video screen from a new perspective. In order to remove the specified person in the live video, the emission process in the neural radiation field is guided by the mask gray map, specifically, the pixel position where the mask map value is 1 emits rays, and the mask map value is 0. No rays are emitted. Therefore, in the process of implementing the existing volume rendering method, the pixels that do not emit rays will not calculate the density and color value of the pixel points through integration, so that the new perspective synthesized picture will not render the characters. Moreover, since the captured site video and the live video limit the same camera position, there will not be a large difference in the image resolution of the synthesized new picture and the corresponding picture in the live video. In this way, the new synthesized video erases the person and replaces him with a virtual person.

[0061] The synthesis module 7 fuses the three-dimensional virtual human provided by the user with the synthesized video image to generate a video.

[0062] Furthermore, the density of the neural rendering output is a value that represents the degree of transparency of the pixel color. In the synthesis, it can be used as depth information to deal with occlusion problems between the virtual human and the video screen. First, the global posture of the virtual human is updated according to the calculated global human posture data, and the virtual human is projected to the new perspective screen according to the pinhole camera model with the global posture as the projection matrix. In this way, the virtual human is superimposed on the synthetic screen, and finally, the density value of the pixel is used as the alpha channel, and weighted fusion is performed to make the virtual human as the foreground and the new screen as the background. Using the camera posture and human posture information, the virtual human is placed in the new synthetic video.

[0063] Compared with the prior art, the embodiment of the present application provides a method for synthesizing a three-dimensional virtual human video, which uses a unified camera position to shoot the site video and the on-site video separately, detects the characters in each frame of the on-site video, and generates a pixel-level mask for each human body; based on the parameterized SMPL human body model, the human body contours formed by the mask in the gray image of each frame mask are aligned to obtain the SMPL parameters of each human body after alignment; the SMPL parameters of each human body after alignment are used as a new perspective of the implicit neural radiation field, and the video screen under the new perspective is synthesized under volume rendering, and merged with the three-dimensional virtual human to generate a video. Avoiding the alignment optimization of the contours of the parameterized three-dimensional human body and the human body segmentation that is difficult to achieve in the existing methods can avoid the visual defects of the virtual human's "feet hanging in the air" or "foot sliding" that are easy to produce in the existing image synthesis method, which is based only on the human body posture to adjust the posture of the virtual human in the video screen.

[0064] Further, refer to Figure 1The embodiment of the present application provides a method for synthesizing a 3D virtual human video, further comprising: applying the rotation of each joint of the SMPL human model to the corresponding joint of the 3D virtual human by the control module 8, thereby realizing the use of real human body movements to drive the limb animation of the 3D virtual human. The synthesized limb movements of the virtual human can be automatically controlled or secondary controlled.

[0065] In practical applications, if there is a difference between the skeleton structure of the virtual human and the SMPL skeleton structure, it is necessary to manually match the SMPL joints with the virtual human joints and redirect the movements through skeleton redirection in advance. Both can be completed using existing methods to help achieve the visual effect produced by the proposed method. Specifically, a1 is the reference pose of the SMPL skeleton, b1 is the reference pose of the real skeleton, the pose of a certain frame in the SMPL skeleton animation is a2, and the desired result is the real skeleton animation b2 of the current frame. The difference between a2 and the reference pose a1 should be the same as the difference between b2 and its corresponding reference pose b1. That is: a2 -a1 = b2 - b1, and the calculation process can be obtained as: b2 = a2 - a1 + b1 = a2 + (b1 - a1). Not limited to the character movements of the live video, users can also use other data sources such as their own body movements to drive the virtual human body animation. For example, using the existing motion capture data to match it to the SMPL human character model, the pose data of each bone joint in the motion data is assigned to the bone joint pose of the SMPL model after the above redirection. Driven by the new posture data, the SMPL model is skinned, a method of binding the vertices of the three-dimensional mesh to the bones. The essence of skeletal animation is to use bones to control the movement of the three-dimensional model mesh. The essence of the mesh movement is the change of vertex positions, thereby generating a new human character animation.

[0066] In addition, an embodiment of the present invention also provides a three-dimensional virtual human video synthesis device, which includes: a processor and a memory; the memory is used to store one or more program instructions; the processor is used to run one or more program instructions to execute the steps of a three-dimensional virtual human video synthesis method as described in any of the above items.

[0067] In addition, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of a three-dimensional virtual human video synthesis method as described in any one of the above items are implemented.

[0068] In the embodiment of the present invention, the processor may be an integrated circuit chip having the signal processing capability. The processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0069] The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present invention can be directly embodied as a hardware decoding processor for execution, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The processor reads the information in the storage medium and completes the steps of the above method in combination with its hardware.

[0070] The storage medium may be a memory, which may be, for example, a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memory.

[0071] Among them, the non-volatile memory can be a read-only memory (ROM), a programmable ROM (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory.

[0072] The volatile memory may be a random access memory (RAM) which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM).

[0073] The storage media described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0074] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the present invention can be implemented using a combination of hardware and software. When software is used, the corresponding functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any media that facilitates the transmission of computer programs from one place to another. Storage media can be any available media that can be accessed by general or special-purpose computers.

[0075] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. A three-dimensional virtual human video synthesis method, characterized in that: The method comprises: Receive venue and live video shot from the same camera position; Converting the scene described by the venue video into an implicit neural radiation field; Detect and track the people in each frame of the live video image, obtain the boundary annotation box of each individual human body and record the corresponding identification ID; Perform human body segmentation on all areas within the boundary annotation boxes in each frame of the live video image, generate pixel-level masks of each human body, and obtain the corresponding mask gray map; Based on a parameterized SMPL human body model, the human body contour formed by the mask in the mask gray image of each frame is registered and the SMPL parameters are updated to obtain the SMPL parameters of each human body after registration; The SMPL parameters of each human body after registration are used as the new perspective of the implicit neural radiation field, and the video images under the new perspective are synthesized under volume rendering; The three-dimensional virtual human provided by the user is integrated with the synthesized video image to generate a video; The SMPL parameters of each human body after registration are used as the new perspective of the implicit neural radiation field, and the video images under the new perspective are synthesized under volume rendering, including: The SMPL parameters of each human body after registration are used as new perspective data, and added to the camera perspective track corresponding to the site video in the form of a new camera perspective; Based on the guidance of the mask gray map, rays are emitted in the neural radiation field, volume rendering is performed, the designated person in the live video is removed, and the video picture under a new perspective is synthesized.

2. A three-dimensional virtual human video synthesis method as claimed in claim 1, characterized in that: Converting the scene described by the venue video into an implicit neural radiation field, including: Acquire each frame of the venue video image for the venue video; Based on the video images of each frame of the venue, the relative position of the camera during imaging is estimated, so as to construct a sparse three-dimensional structure of the venue scene space; Obtaining the three-dimensional spatial position of the captured site video scene and the camera posture of each frame of the site video image; wherein the camera posture is represented by the camera intrinsic parameters and extrinsic parameters in the pinhole camera model of the corresponding frame of the site video image; Encoding the three-dimensional spatial position through a multi-layer perception network; The camera posture is used as the camera view angle, and rays are emitted from each three-dimensional space position along each camera view angle, and the density and color of the corresponding pixel point are obtained by integrating the rays.

3. A three-dimensional virtual human video synthesis method as claimed in claim 2, characterized in that: Detect and track the people in each frame of the live video image, obtain the boundary annotation box of each individual human body and record the corresponding identification ID, including: Acquire a first preset number of pre-labeled images having human bodies; Draw a line from the top to the middle of the bottom of each human head as the dimension of the annotated bounding box; generating a width of the bounding box with a preset aspect ratio based on the dimensions of the bounding box; Determine the bounding box of the human body to be annotated by using the length and width of the bounding box, and annotate a complete bounding box for each individual human body in each pre-annotated image; Based on the annotated complete bounding box, crop each human body instance with a bounding box from the pre-annotated image, save the human body instance as training data, and form a training data set; Using the training data set to train a human body detection algorithm model; Input each frame of the live video image into the trained human body detection algorithm model, annotate the boundary annotation box for each individual human body and record the corresponding identification ID.

4. A three-dimensional virtual human video synthesis method as claimed in claim 3, characterized in that: Perform human body segmentation on all the areas within the boundary annotation boxes in each frame of the live video image, generate pixel-level masks of each human body, and obtain the corresponding mask gray map, including: Extract feature points from the areas within each boundary annotation box in each frame of the live video image to obtain a corresponding feature map; Setting a second preset number of first target regions for each point in each feature map as candidate target regions; Send the candidate target regions to the region generation network for binary classification and bounding box regression, and filter out some candidate target regions; Match the pixels of each frame of the live video image with the corresponding feature map, perform an alignment operation on the remaining candidate target areas, and obtain the corresponding second target area; Performing mask generation on the second target area to obtain a corresponding mask gray image; The masked area in the masked gray image is black, and the corresponding value of the masked area is 0; the non-masked area in the masked gray image is white, and the corresponding value of the non-masked area is 1.

5. A three-dimensional virtual human video synthesis method as claimed in claim 4, characterized in that: Based on a parameterized SMPL human body model, the human body contour formed by the mask in the gray image of each frame is registered and the SMPL parameters are updated to obtain the SMPL parameters of each human body after registration, including: Extracting the human body contour formed by the mask from the mask gray image of each frame to obtain the corresponding first human body two-dimensional contour image; Using the camera view and camera posture rotation matrix R and translation matrix T, the SMPL parameters of the SMPL three-dimensional human body model are initialized to obtain the first global human body posture data and the first skeletal joint posture data; Driven by the first skeletal joint posture data, the SMPL three-dimensional human body model is deformed by skinning to obtain a new SMPL three-dimensional human body model; Using the first global human posture data as a projection matrix, rendering through a pinhole camera perspective projection, to obtain a second two-dimensional human body contour image; Obtaining a similarity error between the second two-dimensional human body contour image and the first two-dimensional human body contour image by a least square method; Determine whether the similarity error is less than a preset threshold; If the similarity error is greater than or equal to a preset threshold, registering the second two-dimensional human body contour map with the first two-dimensional human body contour map, updating the SMPL parameters, and obtaining second global human body posture data and second skeletal joint posture data; The first global human body posture data and the first skeletal joint posture data are respectively updated to the second global human body posture data and the second skeletal joint posture data, and the process is repeated until the SMPL three-dimensional human body model is deformed by skinning under the drive of the first skeletal joint posture data to obtain a new SMPL three-dimensional human body model; If the similarity error is less than a preset threshold, the currently updated SMPL parameters are used as the registered SMPL parameters.

6. A three-dimensional virtual human video synthesis method as claimed in claim 5, characterized in that: In the mask gray map, rays are emitted from pixel positions where the mask map value is 1, and rays are not emitted from pixel positions where the mask map value is 0 in the mask gray map.

7. A three-dimensional virtual human video synthesis method according to any one of claims 1 to 6, characterized in that: The method further comprises: The rotation of each joint of the SMPL human body model is applied to the corresponding joint of the three-dimensional virtual human, so as to realize the use of real body movements to drive the body animation of the three-dimensional virtual human.

8. A three-dimensional virtual human video synthesis system, characterized in that: The system comprises: A receiving module, used for receiving the site video and the on-site video shot by the same camera position; A scene conversion module, used to convert the scene described by the venue video into an implicit neural radiation field; The human body detection and tracking module is used to detect and track the people in each frame of the live video image, obtain the boundary annotation box of each individual human body and record the corresponding identification ID; The human body segmentation module is used to perform human body segmentation on the areas within all boundary annotation boxes in each frame of the live video image, generate pixel-level masks of each human body, and obtain the corresponding mask gray map; A registration module is used to register the human body contour formed by the mask in the mask gray image of each frame based on a parameterized SMPL human body model and update the SMPL parameters to obtain the SMPL parameters of each human body after registration; A new perspective synthesis module is used to use the SMPL parameters of each human body after registration as the new perspective of the implicit neural radiation field, and synthesize the video images under the new perspective under volume rendering; A synthesis module is used to fuse the 3D virtual human provided by the user with the synthesized video image to generate a video; The SMPL parameters of each human body after registration are used as the new perspective of the implicit neural radiation field, and the video images under the new perspective are synthesized under volume rendering, including: The SMPL parameters of each human body after registration are used as new perspective data, and added to the camera perspective track corresponding to the site video in the form of a new camera perspective; Based on the guidance of the mask gray map, rays are emitted in the neural radiation field, volume rendering is performed, the designated person in the live video is removed, and the video picture under a new perspective is synthesized.

9. A three-dimensional virtual human video synthesis device, characterized in that: The device comprises: a processor and a memory; The memory is used to store one or more program instructions; The processor is used to run one or more program instructions to execute the steps of a three-dimensional virtual human video synthesis method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the three-dimensional virtual human video synthesis method as claimed in any one of claims 1 to 7 are implemented.

Images