Method and device for replacing character in video with three-dimensional virtual character

Abstract

The invention provides a method and device for replacing a character in a video with a three-dimensional virtual character, and the method comprises the steps: obtaining a to-be-processed video, and cutting the to-be-processed video into a video frame sequence; performing image segmentation processing on the video frame sequence to obtain a background image frame sequence; performing action estimation processing on the character in the video frame sequence to obtain action data of the character in the video frame sequence; and performing action redirection processing on the basis of the action data and a preset three-dimensional virtual character to obtain a three-dimensional virtual character motion image sequence, overlapping the three-dimensional virtual character motion image sequence with the background image frame sequence, and generating a target video after character replacement according to a target video frame sequence obtained by overlapping. The consistent background image is obtained by quickly eliminating the foreground, and action redirection is performed based on the action data of the character in the video and the preset three-dimensional virtual character, so that the appearance of the character in the generated target video is always kept consistent, and the precision of the generated target video is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of video processing, and in particular to a method and device for replacing a person in a video with a three-dimensional virtual character. Background Art

[0002] In the prior art, video character replacement mainly relies on taking the target 2D character image and action sequence as the input of the neural network model, and directly generating a video of the target character performing a specified action by training the neural network model. The model usually consists of two parts, one for learning the appearance features of the target character and the other for learning the action features. The two parts of the model work together to generate the replaced video. Some generative models also design additional spatial attention modules and temporal attention modules to improve temporal consistency and spatial consistency.

[0003] For the method of using neural network to generate replacement videos, each replacement requires a certain number of iterations for training, which takes a long time. In addition, this method requires appropriate parameter debugging of the network to generate a good replacement video, otherwise the generated video will have problems such as changes in character appearance and inconsistent background. The long time required for network training and parameter debugging means that this type of method is difficult to be widely used. Summary of the invention

[0004] The present invention provides a method for replacing a character in a video with a three-dimensional virtual character, so as to solve the defects of the prior art method of using a neural network to generate a replacement video, which is time-consuming and has low precision, and achieves the effect of quickly generating a character replacement video with higher accuracy of the generated result.

[0005] The present invention provides a method for replacing a person in a video with a three-dimensional virtual character, comprising the following steps.

[0006] Acquire a video to be processed, and cut the video to be processed into a video frame sequence, wherein each video frame in the video frame sequence includes at least one person; Performing image segmentation processing on the video frame sequence respectively to obtain a background image frame sequence, wherein the background image frame sequence is a video frame sequence without the person; Performing motion estimation processing on the characters in the video frame sequence to obtain motion data of the characters in the video frame sequence; Based on the action data and the preset three-dimensional virtual character, action redirection processing is performed to obtain a three-dimensional virtual character motion image sequence, and the three-dimensional virtual character motion image sequence is superimposed with the background image frame sequence, and a target video after replacing the character is generated based on the target video frame sequence obtained by superposition.

[0007] According to a method for replacing a person in a video with a three-dimensional virtual character provided by the present invention, the video frame sequence is subjected to image segmentation processing to obtain a background image frame sequence, including: The following processing is performed for each video frame in the video frame sequence: obtaining key points of the characters in the video frame; Removing the person in the video frame based on the key points to obtain a background image frame; The background image frames corresponding to each video frame are combined in sequence to form the background image frame sequence.

[0008] According to a method for replacing a person in a video with a three-dimensional virtual character provided by the present invention, the person in the video frame is removed based on the key point to obtain a background image frame, including: Based on the key points, calling a semantic segmentation algorithm to perform semantic segmentation processing on the video frame to obtain a mask image for the person; Performing morphological opening and closing operation processing on the mask image to obtain a processed mask image; The person in the video frame is removed based on the processed mask image to obtain the background image frame.

[0009] According to a method for replacing a person in a video with a three-dimensional virtual character provided by the present invention, the method of performing action redirection processing based on the action data and a preset three-dimensional virtual character to obtain a three-dimensional virtual character motion image sequence includes: Writing the motion data into the SMPL-X model to obtain a three-dimensional motion model; The three-dimensional action model is redirected to the preset three-dimensional virtual character to obtain the three-dimensional virtual character motion image sequence.

[0010] According to a method for replacing a person in a video with a three-dimensional virtual character provided by the present invention, redirecting the three-dimensional action model to the preset three-dimensional virtual character to obtain a motion image sequence of the three-dimensional virtual character includes: Creating a controller for the preset three-dimensional virtual character to control a model corresponding to the preset three-dimensional virtual character; Calling the controller to adjust the size of the model corresponding to the preset three-dimensional virtual character so that the size of the model corresponding to the preset three-dimensional virtual character is the same as the size of the three-dimensional action model; The skeleton of the model corresponding to the preset three-dimensional virtual character is matched one by one with the skeleton of the three-dimensional action model, and the model corresponding to the preset three-dimensional virtual character is driven by the action data, and the three-dimensional virtual character motion image sequence is obtained by Blender rendering.

[0011] According to a method for replacing a person in a video with a three-dimensional virtual character provided by the present invention, the method further comprises: When the resolution of the video frame sequence is greater than or equal to a resolution threshold, the video frame sequence is scaled to obtain a processed video frame sequence, and the processed video frame sequence is used as the video frame sequence.

[0012] The present invention also provides a device for replacing a person in a video with a three-dimensional virtual character, comprising the following modules: An acquisition module, used for acquiring a video to be processed, and cutting the video to be processed into a video frame sequence, wherein each video frame in the video frame sequence includes at least one person; An image segmentation module, used to perform image segmentation processing on the video frame sequence to obtain a background image frame sequence, wherein the background image frame sequence is a video frame sequence without the character; An action estimation module, used for performing action estimation processing on the characters in the video frame sequence to obtain action data of the characters in the video frame sequence; The action redirection module is used to perform action redirection processing based on the action data and the preset three-dimensional virtual character to obtain a three-dimensional virtual character motion image sequence, and to superimpose the three-dimensional virtual character motion image sequence with the background image frame sequence, and to generate a target video frame sequence after replacing the character according to the superposition.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method of replacing a person in a video with a three-dimensional virtual character as described in any one of the above methods is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method of replacing a person in a video with a three-dimensional virtual character as described in any one of the above methods is implemented.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method for replacing a person in a video with a three-dimensional virtual character as described in any one of the above methods is implemented.

[0016] The method and device for replacing a character in a video with a three-dimensional virtual character provided by the present invention perform image segmentation processing on a video frame sequence to quickly obtain a background image sequence of the video frame sequence, and perform motion estimation processing on the character in the video frame sequence based on the key points of the character in the video frame sequence to obtain the motion data of the character; then, based on the motion data of the character in the video, motion redirection is performed with a preset three-dimensional virtual character, thereby ensuring that the appearance of the character in the generated target video is always consistent, thereby improving the accuracy of the generated target video. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 This is one of the flow charts of the method for replacing a person in a video with a three-dimensional virtual character provided by the present invention.

[0019] Figure 2 This is the second flow chart of the method for replacing a person in a video with a three-dimensional virtual character provided by the present invention.

[0020] Figure 3 It is a structural schematic diagram of a device for replacing a person in a video with a three-dimensional virtual character provided by the present invention.

[0021] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] Figure 1 FIG. 1 is one of the flow charts of the method for replacing a person in a video with a three-dimensional virtual character provided by the present invention, such as Figure 1 As shown, the method includes the following steps 101 to 104.

[0024] In some embodiments, see Figure 2 , Figure 2This is a second flow chart of the method for replacing a person in a video with a three-dimensional virtual character provided by the present invention. Figure 1 and Figure 2 Provide explanation.

[0025] Step 101: Obtain a video to be processed, and cut the video to be processed into a video frame sequence.

[0026] Each video frame in the video frame sequence includes at least one person.

[0027] In some embodiments, the video to be processed is a video that needs character replacement, and the entire video to be processed includes at least one character, that is, the character that needs to be replaced with a three-dimensional virtual character, and the age and gender of the character are not limited.

[0028] In some embodiments, only some segments of the video to be processed include at least one person. When the video to be processed is cut, only some video frames containing the person are combined to obtain a video frame sequence.

[0029] In some embodiments, the video to be processed can be cut into video frame sequences through the multimedia processing tool Ffmpeg, extracted from the computer vision library OpenCV through the Python interface, or completed through video editing software, or it can also be completed through an online video frame capture tool. The order of the cut video frame sequence needs to be consistent with the time sequence of the video.

[0030] In some embodiments, step 101 further includes the following steps: when the resolution of the video frame sequence is greater than or equal to a resolution threshold, scaling the video frame sequence to obtain a processed video frame sequence, and using the processed video frame sequence as the video frame sequence.

[0031] Since the resolution of the video indicates the resolution of each frame of the video frame sequence, when it is detected that the resolution of the video is greater than or equal to the resolution threshold, it is necessary to perform frame-by-frame scaling processing on the video frame sequence to obtain a scaled video frame sequence, and perform subsequent steps 102 to 104 on the processed video frame sequence, thereby reducing the resolution of the video frame, thereby facilitating improving the efficiency of subsequent processing of the video frame.

[0032] Step 102: perform image segmentation processing on the video frame sequence to obtain a background image frame sequence, wherein the background image frame sequence is a video frame sequence without the character.

[0033] In some embodiments, step 102 can be implemented in the following manner: performing the following processing for each video frame in the video frame sequence: obtaining the key points of the characters in the video frame; removing the characters in the video frame based on the key points to obtain the background image frame; and combining the background image frames corresponding to each video frame in sequence into the background image frame sequence.

[0034] Specifically, the key points of the characters in the video frame are two-dimensional (2D) key points. 2D key point detection is an important task in the field of computer vision, which aims to locate specific parts of the human body or other objects in the image. These key points can be used to describe information such as the posture, shape or movement of the object.

[0035] 2D key point detection methods can be divided into regression and heatmap. The regression method usually uses CNN to extract features, and then uses the fully connected layer to directly regress the coordinates of the key points. The heatmap method predicts the position of the key points by using heat maps, where each channel represents a key point of a category. The result of network prediction is also a heat map. The most direct way to extract the coordinates is to extract the point in a channel where the pixel response is greater than a certain threshold and the response is the largest. The coordinates of this point are the coordinates of the key point of this category.

[0036] For example, MMPose can be used to detect 2D key points of people in video frames.

[0037] In some embodiments, removing the characters in the video frame based on key points to obtain a background image frame can be achieved through the following steps: based on the key points, calling a semantic segmentation algorithm to perform semantic segmentation processing on the video frame to obtain a mask image for the characters; performing morphological opening and closing operations on the mask image to obtain a processed mask image; removing the characters in the video frame based on the processed mask image to obtain the background image frame.

[0038] Specifically, semantic segmentation is another important task in computer vision, which aims to assign each pixel in an image frame to a specific category. When the 2D key points of the characters in the video frame are used as input to the semantic segmentation algorithm, these key points can be used as context clues or prior knowledge to help the semantic segmentation algorithm perform segmentation more accurately. For example, the semantic segmentation algorithm can choose Fully Convolutional Networks (FCN), Deeplab, U-Net or Mask R-CNN.

[0039] Taking the SAM model (Segment Anything Model) as an example, the key points corresponding to each video frame in the input video frame sequence are generated by the SAM model according to the input prompts, that is, the mask image (MASK) of the character.

[0040] Then, the mask image needs to be processed by morphological opening and closing operations to reduce the noise in the mask image and fill the holes in the mask image. Here, the mask image is processed by morphological opening and closing operations, and either the morphological opening operation or the morphological closing operation can be performed, or both operations can be performed, and the execution order is not limited.

[0041] Morphological opening is an image processing technique based on mathematical morphology that can remove small noise points in a mask image. The opening operation is achieved by first performing an erosion operation on the image and then performing a dilation operation. The erosion operation can remove small foreground objects, while the dilation operation can restore the shape of the main object, but will not restore the removed small objects.

[0042] Morphological closing is mainly used to fill small holes in the mask image. Closing is achieved by first dilating the image and then eroding it. Dilation can fill small background holes, while erosion can restore the shape of the main object but not the filled holes.

[0043] In some embodiments, after the MASK is opened and closed, a dilation operation may be performed on the MASK image of the largest connected area to expand the character area in the MASK image, so as to obtain a background image frame with higher accuracy.

[0044] The dilation operation can be implemented by using the dilate function in OpenCV, which accepts three parameters: the input image, the structural element (ie, the kernel), and the number of iterations. The structural element determines the shape and size of the dilation. After the dilation operation, the edges of the character area will become smooth, and the size of the MASK area will increase. This is because the dilation operation checks every pixel within the structural element and applies all or part of their assigned values ​​to the corresponding pixels in the MASK image.

[0045] For the MASK sequence composed of processed mask images corresponding to each video frame, the video editing algorithm ProPainter is used to remove the characters in the video frame sequence to obtain the background image frame.

[0046] In the embodiment of the present invention, by eliminating the foreground of the person in the video frame sequence through the above-mentioned image editing algorithm, a consistent background image frame can be quickly obtained, thereby improving the replacement efficiency.

[0047] Step 103: Perform motion estimation processing on the characters in the video frame sequence to obtain motion data of the characters in the video frame sequence.

[0048] In some embodiments, a posture estimation algorithm WHAM (Reconstructing World-grounded Humans with Accurate 3D Motion) is used to perform motion estimation processing on a video frame sequence, that is, the video is input into WHAM for processing, and SMPL motion sequence parameters, that is, the result of motion estimation, are generated according to the video. The motion estimation result is expanded into an SMPL-X model sequence, and is written into an NPZ file in the Amass dataset format.

[0049] It should be noted that the data finally written into the NPZ file include gender parameters, frame rate, 16-dimensional body parameters, and motion data, among which the gender parameter and body parameter can be set to a constant; the frame rate is equivalent to the frame rate of the video to be processed; the motion data consists of the rotation posture of the body joints and the motion trajectory of the root node. The rotation of the joints and the movement of the root node can be performed along the xyz direction of the coordinate system, so the rotation and trajectory are both represented by three-dimensional vectors.

[0050] The posture estimation algorithm WHAM uses the SMPL skeleton without fingers as the action skeleton, sets the rotation of the finger sequence to 0, expands the action estimation result of WHAM to the SMPL-X model sequence, and writes it into the NPZ file according to the Amass dataset format to obtain the action data.

[0051] It should be noted that the SMPL model (Skinned Multi-Person Linear Model) is a differentiable model that is widely used in the field of 3D modeling. The model supports soft tissue dynamics and has multiple derivative versions, such as SMPL+H (a model including body and hands), SMPL-X (a model including body, hands and face), etc.

[0052] The Amass dataset is a dataset specifically used for human pose estimation and shape estimation. Its format usually includes image data, corresponding human pose and shape labels, and other information. To write the SMPL-X model sequence into an NPZ file, it is necessary to ensure that the model can output a structure that conforms to the Amass dataset format.

[0053] To write the SMPL-X model sequence into an NPZ file, you can use Python's numpy.savez function, which can save multiple arrays into a file as a group.

[0054] In the embodiment of the present invention, the motion data of the character in the video frame sequence can be quickly acquired through the posture estimation algorithm, which facilitates the subsequent task replacement according to the motion data, thereby improving the replacement efficiency.

[0055] Step 104: perform action redirection processing based on the action data and the preset three-dimensional virtual character to obtain a three-dimensional virtual character motion image sequence, and superimpose the three-dimensional virtual character motion image sequence with the background image frame sequence, and generate a target video after replacing the character based on the target video frame sequence obtained by superposition.

[0056] In some embodiments, the action redirection processing based on the action data and the preset three-dimensional virtual character in step 104 to obtain a three-dimensional virtual character motion image sequence can be specifically implemented in the following ways: writing the action data into the SMPL-X model to obtain a three-dimensional action model; redirecting the three-dimensional action model to the preset three-dimensional virtual character to obtain the three-dimensional virtual character motion image sequence.

[0057] In some embodiments, the Blender plug-in of SMPL-X may be used to write the NPZ motion data obtained in step 103 into the SMPL-X model to obtain a three-dimensional motion model.

[0058] In some embodiments, the three-dimensional action model is redirected to the preset three-dimensional virtual character to obtain the three-dimensional virtual character motion image sequence, which can be specifically achieved in the following ways: creating a controller for the preset three-dimensional virtual character to control the model corresponding to the preset three-dimensional virtual character; calling the controller to adjust the size of the model corresponding to the preset three-dimensional virtual character so that the size of the model corresponding to the preset three-dimensional virtual character is the same as the size of the three-dimensional action model; matching the skeleton of the model corresponding to the preset three-dimensional virtual character with the skeleton of the three-dimensional action model one by one, and using the action data to drive the model corresponding to the preset three-dimensional virtual character, and obtaining the three-dimensional virtual character motion image sequence through Blender rendering.

[0059] Specifically, a virtual character downloaded from the Mixamo website (i.e., a preset three-dimensional virtual character) is imported into Blender, and then a controller system is created for the virtual character using Mixamo's Blender plug-in, and control bones and surface meshes of corresponding parts of the virtual character are bound to control the model corresponding to the preset three-dimensional virtual character.

[0060] Then use the Auto-Rig Pro plug-in to redirect the SMPL-X model (i.e., 3D action model) action to the virtual character. Specifically, first make the two models the same size through scaling operations, then match the SMPL-X skeleton and Mixamo's controller skeleton one by one, and then select the root node of the model and the number of redirected animation frames. That is, use the action data to drive the model corresponding to the preset 3D virtual character to complete the redirection operation.

[0061] In some embodiments, after the redirection operation is completed, a camera and a light source are added in Blender and their positions and other properties are adjusted, and the virtual character animation (i.e., the animation corresponding to the three-dimensional virtual character motion image sequence) is rendered into a PNG image format with transparency. Finally, with the help of the transparent channel of the PNG image, the character animation is superimposed on the background image frame sequence to obtain a target video frame sequence, and the target video frame sequence is written into the result video to obtain the final output, i.e., the target video.

[0062] It should be noted that for portable use, all the above Blender operations can be encapsulated as Blender Python scripts.

[0063] In an embodiment of the present invention, image segmentation and video editing technology are first used to remove the portrait to obtain the background. Compared with directly generating a new video, this technology ensures that no new noise will be generated in the background. Secondly, Blender software is used to render the 3D virtual character offline. This step also does not cause changes in the appearance of the characters in the video. In terms of generalization and time consumption, this method also has certain advantages. For an original video, its action estimation only needs to be performed once and can be applied to multiple virtual characters at the same time without the need for additional network training. The use of a generative model requires retraining for each new character and new action. Therefore, the method provided by the present invention to replace the characters in the video with three-dimensional virtual characters can effectively improve the replacement efficiency and accuracy.

[0064] The following is a description of the device for replacing characters in a video with three-dimensional virtual characters provided by the present invention. The device for replacing characters in a video with three-dimensional virtual characters described below and the method for replacing characters in a video with three-dimensional virtual characters described above can be referenced to each other.

[0065] See also Figure 3 , Figure 3 is a schematic diagram of the structure of the device for replacing a person in a video with a three-dimensional virtual character provided by the present invention, such as Figure 3As shown, the device 300 for replacing a character in a video with a three-dimensional virtual character provided by an embodiment of the present invention includes an acquisition module 301, an image segmentation module 302, an action estimation module 303 and an action redirection module 304; wherein the acquisition module 301 is used to acquire a video to be processed and cut the video to be processed into a video frame sequence, wherein each video frame in the video frame sequence includes at least one character; the image segmentation module 302 is used to perform image segmentation processing on the video frame sequence respectively to obtain a background image frame sequence, wherein the background image frame sequence is a video frame sequence without the character; the action estimation module 303 is used to perform action estimation processing on the character in the video frame sequence to obtain action data of the character in the video frame sequence; the action redirection module 304 is used to perform action redirection processing based on the action data and a preset three-dimensional virtual character to obtain a three-dimensional virtual character motion image sequence, and superimpose the three-dimensional virtual character motion image sequence with the background image frame sequence, and generate a target video after replacing the character according to the superimposed target video frame sequence.

[0066] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute a method for replacing a character in a video with a three-dimensional virtual character, the method comprising: obtaining a video to be processed, and cutting the video to be processed into a video frame sequence, wherein each video frame in the video frame sequence includes at least one character; performing image segmentation processing on the video frame sequence to obtain a background image frame sequence, wherein the background image frame sequence is a video frame sequence without the character; performing motion estimation processing on the character in the video frame sequence to obtain motion data of the character in the video frame sequence; performing motion redirection processing based on the motion data and a preset three-dimensional virtual character to obtain a three-dimensional virtual character motion image sequence, and superimposing the three-dimensional virtual character motion image sequence with the background image frame sequence, and generating a target video after replacing the character according to the superimposed target video frame sequence.

[0067] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0068] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the method provided by the above-mentioned methods to replace the characters in the video with three-dimensional virtual characters. The method includes: obtaining a video to be processed, and cutting the video to be processed into a video frame sequence, wherein each video frame in the video frame sequence includes at least one character; performing image segmentation processing on the video frame sequence to obtain a background image frame sequence, wherein the background image frame sequence is a video frame sequence without the characters; performing motion estimation processing on the characters in the video frame sequence to obtain motion data of the characters in the video frame sequence; performing motion redirection processing based on the motion data and a preset three-dimensional virtual character to obtain a three-dimensional virtual character motion image sequence, and superimposing the three-dimensional virtual character motion image sequence and the background image frame sequence, and generating a target video after replacing the characters according to the superimposed target video frame sequence.

[0069] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the method provided by the above-mentioned methods for replacing characters in a video with three-dimensional virtual characters, the method comprising: obtaining a video to be processed, and cutting the video to be processed into a video frame sequence, wherein each video frame in the video frame sequence includes at least one character; performing image segmentation processing on the video frame sequence to obtain a background image frame sequence, wherein the background image frame sequence is a video frame sequence without the characters; performing motion estimation processing on the characters in the video frame sequence to obtain motion data of the characters in the video frame sequence; performing motion redirection processing based on the motion data and a preset three-dimensional virtual character to obtain a three-dimensional virtual character motion image sequence, and superimposing the three-dimensional virtual character motion image sequence and the background image frame sequence, and generating a target video after replacing the characters based on the superimposed target video frame sequence.

[0070] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0071] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0072] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for replacing a character in a video with a three-dimensional virtual character, characterized in that: include: Acquire a video to be processed, and cut the video to be processed into a video frame sequence, wherein each video frame in the video frame sequence includes at least one person; Performing image segmentation processing on the video frame sequence respectively to obtain a background image frame sequence, wherein the background image frame sequence is a video frame sequence without the person; Performing motion estimation processing on the characters in the video frame sequence to obtain motion data of the characters in the video frame sequence; Based on the action data and the preset three-dimensional virtual character, action redirection processing is performed to obtain a three-dimensional virtual character motion image sequence, and the three-dimensional virtual character motion image sequence is superimposed with the background image frame sequence, and a target video after replacing the character is generated based on the target video frame sequence obtained by superposition.

2. The method for replacing a person in a video with a three-dimensional virtual character according to claim 1, characterized in that: The step of performing image segmentation processing on the video frame sequence to obtain a background image frame sequence comprises: The following processing is performed for each video frame in the video frame sequence: obtaining key points of the characters in the video frame; Removing the person in the video frame based on the key points to obtain a background image frame; The background image frames corresponding to each video frame are combined in sequence to form the background image frame sequence.

3. The method for replacing a person in a video with a three-dimensional virtual character according to claim 2, characterized in that: The removing of the characters in the video frame based on the key points to obtain a background image frame includes: Based on the key points, calling a semantic segmentation algorithm to perform semantic segmentation processing on the video frame to obtain a mask image for the person; Performing morphological opening and closing operation processing on the mask image to obtain a processed mask image; The person in the video frame is removed based on the processed mask image to obtain the background image frame.

4. The method for replacing a person in a video with a three-dimensional virtual character according to claim 1, characterized in that: The step of performing action redirection processing based on the action data and a preset three-dimensional virtual character to obtain a three-dimensional virtual character motion image sequence includes: Writing the motion data into the SMPL-X model to obtain a three-dimensional motion model; The three-dimensional action model is redirected to the preset three-dimensional virtual character to obtain a motion image sequence of the three-dimensional virtual character.

5. The method for replacing a person in a video with a three-dimensional virtual character according to claim 4, characterized in that: The step of redirecting the three-dimensional action model to the preset three-dimensional virtual character to obtain a motion image sequence of the three-dimensional virtual character includes: Creating a controller for the preset three-dimensional virtual character to control a model corresponding to the preset three-dimensional virtual character; Calling the controller to adjust the size of the model corresponding to the preset three-dimensional virtual character so that the size of the model corresponding to the preset three-dimensional virtual character is the same as the size of the three-dimensional action model; The skeleton of the model corresponding to the preset three-dimensional virtual character is matched one by one with the skeleton of the three-dimensional action model, and the model corresponding to the preset three-dimensional virtual character is driven by the action data, and the three-dimensional virtual character motion image sequence is obtained by Blender rendering.

6. The method for replacing a person in a video with a three-dimensional virtual character according to claim 1, characterized in that: The method further comprises: When the resolution of the video frame sequence is greater than or equal to a resolution threshold, the video frame sequence is scaled to obtain a processed video frame sequence, and the processed video frame sequence is used as the video frame sequence.

7. A device for replacing a person in a video with a three-dimensional virtual character, characterized in that: include: An acquisition module, used for acquiring a video to be processed, and cutting the video to be processed into a video frame sequence, wherein each video frame in the video frame sequence includes at least one person; An image segmentation module, used to perform image segmentation processing on the video frame sequence to obtain a background image frame sequence, wherein the background image frame sequence is a video frame sequence without the character; An action estimation module, used for performing action estimation processing on the characters in the video frame sequence to obtain action data of the characters in the video frame sequence; The action redirection module is used to perform action redirection processing based on the action data and a preset three-dimensional virtual character to obtain a three-dimensional virtual character motion image sequence, and to superimpose the three-dimensional virtual character motion image sequence with the background image frame sequence, and to generate a target video after replacing the character based on the target video frame sequence obtained by superposition.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method for replacing a person in a video with a three-dimensional virtual character as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for replacing a person in a video with a three-dimensional virtual character as claimed in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for replacing a person in a video with a three-dimensional virtual character as claimed in any one of claims 1 to 6 is implemented.

Images