Digital human mouth shape inference material processing method and system

Abstract

The application provides a digital human mouth shape inference material processing method and system, and the method comprises the following steps: acquiring an original material video; inputting the original material video into MuseTalk for preprocessing to obtain a mask graph; acquiring a trained UNet and a VAE decoder; performing lip shape inference on the mask graph based on the trained UNet and the VAE decoder to obtain an inference image; performing adjustment and superposition processing on the inference image based on the mask graph to obtain a processing image; performing virtual image elimination on the processing image to obtain an elimination image; and storing the elimination image and the mask graph in a processing material set to complete the processing of the digital human mouth shape inference material. Through a series of processing on the image, the virtual image in the image can be effectively removed, the original bulging problem can be solved, and the problem of existing stripes on the edge can be solved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to a digital human mouth shape reasoning material processing method and system. BACKGROUND

[0002] With the rise and development of natural language large models and mouth shape reasoning technology, the fusion of AI digital humans and humans will become a major trend in the future. At present, e-commerce, finance, entertainment and the like are relatively mature, and begin to have numerous service providers and users, and communication, tourism, automobiles, government affairs and even public welfare and charity fields also begin to usher in large-scale exploration. In the process of real-time interaction with digital humans, the dialogue mouth shape of the digital human is particularly obvious and important. Musetalk is a real-time audio-driven lip synchronization model. The model can automatically adjust the facial image of the digital human according to the input audio signal, so that the lip shape is highly synchronized with the audio content.

[0003] However, the original application of the Musetalk model is to generate highly realistic virtual human speech or performance videos, and the model has the function of automatically beautifying the input material during reasoning. However, for real-time interactive digital humans, the pre-recorded action material will be cut into multiple pictures before mouth shape reasoning. When audio-driven mouth shape reasoning is performed, the cut-out pictures are used as the basis, the mouth shape region after reasoning is replaced in the original picture, and the effect of real-time mouth shape reasoning is realized.

[0004] In actual application, the following disadvantages exist in real-time mouth shape reasoning by Musetalk alone:

[0005] 1. There are stripes on the edge of the replacement region: before training the mouth shape reasoning model, Musetalk performs facial feature point detection on each cut-out picture and calculates the rectangular region of the framed face. The built-in face recognition model and framing region algorithm are prone to cause the region to be too close to the edge of the face, resulting in reduced tolerance when the pictures are superimposed, and thus stripes appear on the edge.

[0006] 2. Facial bulge appears at the moment of speaking and silence: during real-time mouth shape reasoning, the pre-processed picture frame is directly played when in a silent state. Therefore, when the face of the material person is relatively full, the built-in implementation effect of Musetalk will cause the facial contour after mouth shape reasoning to be smaller than the original contour by one circle, resulting in a facial bulge at the moment of speaking and silence.

[0007] 3. There is a circle of virtual shadow outside the facial contour during speaking: during real-time mouth shape reasoning, the mouth shape region after reasoning is superimposed on the original picture after edge Gaussian blur. Because the facial region after reasoning is reduced, it cannot completely cover the facial edge of the original picture when superimposed on the original picture, thereby causing a circle of virtual shadow outside the facial contour during speaking. Summary of the Invention

[0008] In order to solve the above technical problems, the present invention provides a method and system for processing digital human mouth shape reasoning materials, which are used to solve the technical problems raised in the prior art.

[0009] In a first aspect, an embodiment of the present invention provides the following technical solution: a method for processing digital human lip shape inference material, comprising:

[0010] Obtaining an original material video, and inputting the original material video into MuseTalk for preprocessing to obtain a mask image;

[0011] Obtaining a trained UNet and VAE decoder, and performing lip shape inference on the mask image based on the trained UNet and VAE decoder to obtain an inference image;

[0012] Adjusting and overlaying the inference image based on the mask image to obtain a processed image;

[0013] Performing ghost removal on the processed image to obtain a ghost-removed image;

[0014] The eliminated image and the mask image are stored in a processing material set to complete the processing of the digital human mouth shape reasoning material.

[0015] Compared with the prior art, the beneficial effects of the present application are as follows: the present invention first obtains the original material video, inputs the original material video into MuseTalk for preprocessing to obtain a mask image; then obtains the trained UNet and VAE decoder, and performs lip shape inference on the mask image based on the trained UNet and VAE decoder to obtain an inference image; then adjusts and overlays the inference image based on the mask image to obtain a processed image; then performs ghost removal on the processed image to obtain an eliminated image; finally, the eliminated image and the mask image are stored in the processing material set to complete the processing of the digital human lip shape inference material. The present invention can effectively remove ghosts in the image, solve the original bulging problem and the problem of stripes on the edge by performing a series of processing on the image. In the actual inference process, the unbeautified material is still used for inference, but the inferred face will be superimposed on the picture after face slimming, so that no ghost will be generated during real-time inference. When jumping from speaking to stillness, since the face shape has been processed to the same intensity of beautification, the original bulging problem is also solved.

[0016] Preferably, the step of inputting the original material video into MuseTalk for preprocessing to obtain the mask image includes:

[0017] Replace the first face feature detection model in MuseTalk with a second face feature detection model to obtain a replacement MuseTalk;

[0018] Input the original material video into the replacement MuseTalk and extract the vertex coordinates of the framed face region in the original material video through the replacement MuseTalk;

[0019] Add pixel point offsets to the vertex coordinates in proportion to the resolution size of the original material video to obtain offset vertex coordinates;

[0020] Based on the offset vertex coordinates and according to the pre-processing script in the replacement MuseTalk, perform image splitting and face mask region generation on the original material video to obtain generated images;

[0021] Gaussian blur the edges of the generated images and perform binary processing on the generated images to obtain a mask image.

[0022] Preferably, the first face feature detection model is specifically a dw-ll_ucoco_384.pth model, and the second face feature detection model is specifically a shape_predictor_68_face_landmarks.dat model.

[0023] Preferably, the step of performing lip reasoning on the mask image based on the trained UNet and VAE decoder to obtain a reasoning image comprises:

[0024] Set a numpy array of a silent audio frame, input the numpy array into a preset encoding model in MuseTalk for encoding to obtain text and feature representation;

[0025] Input the text and feature representation into the trained UNet and VAE decoder, and input the mask image and the corresponding mask region into a real-time live streaming framework of a real person digital human according to the trained UNet and VAE decoder for lip reasoning to obtain a reasoning image.

[0026] Preferably, the preset encoding model is a frozen whisper-tiny model.

[0027] Preferably, the step of adjusting and superimposing the reasoning image based on the mask image to obtain a processed image comprises:

[0028] Segment and frame the face lower region of the mask image and the reasoning image to obtain a first segmented image and a second segmented image;

[0029] The first segmented image is polygonally cut using a polygon cutting method in Opencv to obtain a cut image.

[0030] The edges of the cut image are blurred to obtain a blurred image, and the blurred image is superimposed on the second segmented image using an addWeighted method in Opencv to obtain a processed image.

[0031] Preferably, the step of eliminating virtual shadows from the processed image to obtain an eliminated image comprises:

[0032] Face feature points of a lower half of the face of the processed image are identified, and the face feature points are sequentially connected to obtain a lower half face line.

[0033] Converging points of the face feature points on the lower half face line are calculated, and all face feature points are diffused in the direction of the corresponding converging points to obtain diffused feature points.

[0034] The area surrounded between the diffused feature points and the face feature points is blurred to obtain a diffused image.

[0035] The diffused image is overlaid on the corresponding position of the processed image to obtain an eliminated image.

[0036] In a second aspect, the embodiments of the present application provide the following technical solutions, a digital person mouth shape reasoning material processing system, the system comprises:

[0037] A processing module is configured to obtain an original material video, input the original material video into MuseTalk for preprocessing to obtain a mask image.

[0038] A reasoning module is configured to obtain a trained UNet and a VAE decoder, perform lip shape reasoning on the mask image based on the trained UNet and the VAE decoder to obtain a reasoning image.

[0039] An adjustment module is configured to perform adjustment superposition processing on the reasoning image based on the mask image to obtain a processed image.

[0040] An elimination module is configured to eliminate virtual shadows from the processed image to obtain an eliminated image.

[0041] An output module is configured to store the eliminated image and the mask image in a processing material set to complete processing of the digital person mouth shape reasoning material.

[0042] In a third aspect, an embodiment of the present application provides the technical scheme below, a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the digital human mouth shape reasoning material processing method described above when executing the computer program.

[0043] In a fourth aspect, an embodiment of the present application provides the technical scheme below, a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the digital human mouth shape reasoning material processing method described above. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or prior art description will be briefly introduced below, and obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0045] Figure 1 The flowchart of the digital human mouth shape reasoning material processing method provided in the embodiment of the present application is shown in the figure.

[0046] Figure 2 The structural block diagram of the digital human mouth shape reasoning material processing system provided in the embodiment of the present application is shown in the figure.

[0047] Figure 3 The hardware structure schematic diagram of the computer device provided in the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0048] The embodiments of the present application will be described in detail below, and the examples of the embodiments are shown in the drawings, wherein the same or similar reference signs represent the same or similar elements or elements with the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and are intended to explain the embodiments of the present application, and cannot be understood as a limitation of the present application.

[0049] Embodiment one

[0050] In the embodiment one of the present application, as shown in the figure, Figure 1 a digital human mouth shape reasoning material processing method comprises:

[0051] S1, obtaining an original material video, inputting the original material video into MuseTalk for preprocessing to obtain a mask map;

[0052] The step S1 comprises:

[0053] S1, replace the first face feature detection model in MuseTalk with a second face feature detection model to obtain a replacement MuseTalk;

[0054] Specifically, the first face feature detection model is specifically a dw-ll_ucoco_384.pth model, and the second face feature detection model is specifically a shape_predictor_68_face_landmarks.dat model.

[0055] The dw-ll_ucoco_384.pth model is a feature detection model in MuseTalk, which may have a problem of error in frame selection of facial contour feature points in some skewed face images. After the contour of the face is framed, the dw-ll_ucoco_384.pth model calculates the maximum and minimum values of the horizontal and vertical coordinates of all feature points in the coordinate axis to find the rectangular four vertex coordinates of the framed face region. At this time, the error caused by feature point recognition cannot be processed, so the shape_predictor_68_face_landmarks.dat model is used.

[0056] S2, input the original material video into the replacement MuseTalk and extract the vertex coordinates of the framed face region in the original material video through the replacement MuseTalk.

[0057] S3, add pixel point offset to the vertex coordinates according to the resolution size of the original material video to obtain offset vertex coordinates;

[0058] Specifically, by adding a certain pixel point offset, it can be ensured that the chin, forehead and both cheeks of the face can be completely contained in the rectangular region.

[0059] S4, based on the offset vertex coordinates and according to the pre-processing script in the replacement MuseTalk, image splitting and face mask region generation are performed on the original material video to obtain a generated image;

[0060] S5, Gaussian blur is performed on the edges of the generated image, and the generated image is binarized to obtain a mask image;

[0061] Specifically, the splitting of the image of the image is calculated according to the frame rate of the imported video to calculate how many frame images should be sampled per second of video, and then the video path, frame rate, resolution and other parameters are given to the ffmpeg tool to realize picture splitting, and the face mask is obtained by using the previous face region framing and face feature point detection to obtain the parameters of the face of the current frame picture, and the face-parsing.PyTorch integrated by MuseTalk divides the face region (irregular region) and converts it into a binary image after Gaussian blur, that is, the mask image is obtained.

[0062] S2, obtaining a trained UNet and VAE decoder, and performing lip reasoning on the mask image based on the trained UNet and VAE decoder to obtain a reasoning image;

[0063] Specifically, the UNet and VAE decoder here can be trained using some template reasoning images.

[0064] The step S2 includes:

[0065] S21, setting a numpy array of a silent audio frame, inputting the numpy array into a preset encoding model in MuseTalk for encoding to obtain text and feature representation;

[0066] Specifically, the preset encoding model is a frozen whisper-tiny model, and a numpy array of a silent audio frame is artificially set. MuseTalk uses the frozen whisper-tiny model to encode the audio, which converts the audio into corresponding text and feature representation. Then, combining the text and feature representation, MuseTalk internally uses the UNet and VAE decoder trained above to perform lip reasoning on the picture frame.

[0067] S22, inputting the text and feature representation into the trained UNet and VAE decoder, and inputting the mask image and the corresponding mask region into a real-time live streaming framework of a real person digital human according to the trained UNet and VAE decoder for lip reasoning to obtain a reasoning image;

[0068] Specifically, even in a mute state, the lip shape will change during the reasoning process of MuseTalk, but because MuseTalk automatically thins the face during reasoning, after a round of reasoning, the corresponding picture frame after face thinning (beautification) but with open and closed mouth can also be obtained.

[0069] S3, adjusting and superimposing the reasoning image based on the mask image to obtain a processing image;

[0070] The step S3 comprises:

[0071] S31, segment and frame the face lower half region of the mask image and the inference image to obtain a first segmented image and a second segmented image.

[0072] S32, use a polygon clipping method in Opencv to clip the first segmented image to obtain a clipped image.

[0073] Specifically, since the face lower half region is an irregular region, it can be understood that the feature points of the two cheeks are sequentially connected with other feature points in the chin direction, and finally a closed polygon region is formed.

[0074] S33, edge blur the edges of the clipped image to obtain a blurred image, and use the addWeighted method in Opencv to superimpose the blurred image on the second segmented image to obtain a processed image.

[0075] Each frame of the obtained processed image has the same expression as the image in the original material video.

[0076] S4, eliminate the virtual image of the processed image to obtain an eliminated image.

[0077] The step S4 comprises:

[0078] S41, identify the face feature points of the face lower half region of the processed image, and sequentially connect the face feature points to obtain a lower half face line.

[0079] Specifically, in the above step, the image after beautifying, i.e., the processed image, has the thin face region superimposed on the processed mask image, so the contour virtual image of the face already exists, and therefore the contour virtual image can be removed in this step.

[0080] S42, calculate the convergence point of the face feature points on the lower half face line, and diffuse all face feature points towards the corresponding convergence point to obtain diffusion feature points.

[0081] Specifically, each face feature point has the same convergence point, and then the face feature points are expanded outward along the direction from the face feature points to the convergence point, and in this application, the diffusion distance is 15 pixel distances for a resolution of 1K.

[0082] S43, blur the region surrounded by the diffusion feature points and the face feature points to obtain a diffusion image.

[0083] S44, overlaying the diffusion image at the corresponding position on the processing image to obtain an elimination image;

[0084] Specifically, the above process is similar to the process of copying a seal, and the processed diffusion image is overlaid at the corresponding area on the original image, so that the ghost effect is realized.

[0085] S5, storing the elimination image and the mask map in a processing material set to complete the processing of the digital person mouth reasoning material.

[0086] The digital person mouth reasoning material processing method provided by the embodiment one of the application first acquires an original material video, inputs the original material video into MuseTalk for preprocessing to obtain a mask map; then acquires a trained UNet and a VAE decoder, performs lip reasoning on the mask map based on the trained UNet and the VAE decoder to obtain a reasoning image; then performs adjustment and superposition processing on the reasoning image based on the mask map to obtain a processing image; then eliminates the ghost from the processing image to obtain an elimination image; finally, stores the elimination image and the mask map in a processing material set to complete the processing of the digital person mouth reasoning material. Through a series of processing on the image, the ghost in the image can be effectively removed, the original bulging problem and the problem of existing stripes on the edge can be solved. In the actual reasoning process, the material without beautifying is still used for reasoning, but the face after reasoning will be superimposed on the picture after face thinning, so that the ghost will not be generated during real-time reasoning. When speaking to static transition, since the face shape has been processed to the same intensity of beautifying state, the original bulging problem is also solved.

[0087] Embodiment two

[0088] As shown in the embodiment two of the application, a digital person mouth reasoning material processing system is provided, and the system comprises: Figure 2 The processing module 1 is configured to acquire an original material video, input the original material video into MuseTalk for preprocessing to obtain a mask map;

[0089] The reasoning module 2 is configured to acquire a trained UNet and a VAE decoder, perform lip reasoning on the mask map based on the trained UNet and the VAE decoder to obtain a reasoning image;

[0090] The adjustment module 3 is configured to perform adjustment and superposition processing on the reasoning image based on the mask map to obtain a processing image;

[0091] The elimination module 4 is configured to eliminate the ghost from the processing image to obtain an elimination image;

[0092]

[0093] ​An output module 5 is configured to store the elimination image and the mask image in a processing material set to complete processing of the digital human mouth shape inference material;

[0094] The processing module 1 comprises:

[0095] A replacement submodule is configured to replace a first face feature detection model in MuseTalk with a second face feature detection model to obtain a replacement MuseTalk.

[0096] A coordinate submodule is configured to input the original material video into the replacement MuseTalk and extract vertex coordinates of a framed face region in the original material video through the replacement MuseTalk.

[0097] An offset submodule is configured to add pixel point offsets to the vertex coordinates in proportion to the resolution size of the original material video to obtain offset vertex coordinates.

[0098] A generation submodule is configured to perform image splitting and face mask region generation on the original material video based on the offset vertex coordinates and according to a preprocessing script in the replacement MuseTalk to obtain a generated image.

[0099] A binarization submodule is configured to perform Gaussian blur on an edge of the generated image and perform binarization processing on the generated image to obtain a mask image.

[0100] The inference module 2 comprises:

[0101] A setting submodule is configured to set a numpy array of a mute audio frame, input the numpy array into a preset encoding model in MuseTalk for encoding to obtain text and feature representation.

[0102] An inference submodule is configured to input the text and feature representation into a trained UNet and VAE decoder, and input the mask image and a corresponding mask region into a real-time live streaming framework of a real human digital human according to the trained UNet and VAE decoder for lip shape inference to obtain an inference image.

[0103] The adjustment module 3 comprises:

[0104] A segmentation submodule is configured to perform segmentation and framing of a face lower part region on the mask image and the inference image to obtain a first segmented image and a second segmented image.

[0105] A cutting submodule is configured to use a polygon cutting method in Opencv to perform polygon cutting on the first segmented image to obtain a cut image.

[0106] The blurring sub-module is configured to perform edge blurring on edges of the cropped image to obtain a blurred image, and the blurred image is superimposed on the second divided image by using an addWeighted method in OpenCV to obtain a processed image.

[0107] The eliminating module comprises:

[0108] The connecting sub-module is configured to identify facial feature points in a lower half of the face region of the processed image, and sequentially connect the facial feature points to obtain a lower half face line;

[0109] The diffusion sub-module is configured to calculate a convergence point with the same distance of the facial feature points on the lower half face line, and diffuse all the facial feature points towards the corresponding convergence point to obtain diffusion feature points;

[0110] The blurring sub-module is configured to blur a region surrounded by the diffusion feature points and the facial feature points to obtain a diffusion image;

[0111] The covering sub-module is configured to cover the diffusion image on a corresponding position on the processed image to obtain an eliminating image.

[0112] In some embodiments of the present application, a computer is provided, which comprises a memory 102, a processor 101, and a computer program stored in the memory 102 and executable on the processor 101, and the processor 101 implements the digital human mouth shape reasoning material processing method as described above when executing the computer program.

[0113] Specifically, the processor 101 can comprise a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present application.

[0114] The memory 102 can include mass storage for data or instructions. By way of example, and not limitation, the memory 102 can include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), flash memory, a USB drive, a magneto-optical disk, optical disks, such as CDs and DVDs, magnetic tapes, or other types of storage media, which can be removable or non-removable (or fixed). In some embodiments, the memory 102 is non-volatile memory. In some embodiments, the memory 102 includes read-only memory (ROM) and random access memory (RAM). In some embodiments, the ROM can be mask- programmed ROM, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), electrically alterable ROM (EAROM), or FLASH, or combinations of two or more of these. In some embodiments, the RAM can be static RAM (SRAM) or dynamic RAM (DRAM), which can be Fast Page Mode DRAM (FPM DRAM), Extended Data Output DRAM (EDO DRAM), synchronous DRAM (SDRAM), etc.

[0115] The memory 102 can be used to store or buffer various data files needed for processing and / or communication, and possible computer program instructions executed by the processor 101.

[0116] The processor 101 realizes the digital human mouth shape inference material processing method described above by reading and executing the computer program instructions stored in the memory 102.

[0117] In some embodiments, the computer can also include a communication interface 103 and a bus 100. As shown, the processor 101, the memory 102, and the communication interface 103 are connected through the bus 100 and complete communication with each other. Figure 3

[0118] The communication interface 103 is used to realize the communication between the modules, devices, units and / or equipment in the embodiments of the present application. The communication interface 103 can also realize data communication with other components, such as external devices, image / data acquisition devices, databases, external storage, image / data processing workstations, etc.

[0119] ​Bus 100 includes hardware, software, or both, to couple components of computer device to each other and to couple components of computer device to other devices. Bus 100 includes, but is not limited to, at least one of the following: a data bus, an address bus, a control bus, an expansion bus, a local bus, and the like. By way of example and not limitation, bus 100 can include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or another suitable bus or combination of two or more of these. Where appropriate, bus 100 can include one or more buses. Although the present embodiments describe and show a particular bus, the present embodiments contemplate any suitable bus or interconnect.

[0120] The computer can acquire the digital human mouth shape reasoning material processing system, execute the digital human mouth shape reasoning material processing method of the present application, and thus realize digital human mouth shape reasoning material processing.

[0121] In some embodiments of the present application, in combination with the above-mentioned digital human mouth shape reasoning material processing method, the present embodiments provide the following technical solutions: a storage medium, which has a computer program stored thereon, and the computer program is executed by a processor to realize the above-mentioned digital human mouth shape reasoning material processing method.

[0122] Those skilled in the art will appreciate that the logic and / or steps represented in the flow diagrams and / or otherwise described herein, for example, can be thought of as a list of executable instructions for implementing logic functions, and can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor- containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions, or a combination thereof. For the purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.

[0123] More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection (electronic) having one or more wires, a portable computer diskette (magnetic), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can also be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example, via an optical scanner, then compiled, interpreted, or otherwise processed, and stored in a computer memory in a form that is then employable by a computer. In this context, the above-mentioned media are considered machine-readable media.

[0124] It should be understood that aspects of the application can be implemented in hardware, software, firmware or combinations thereof. In the above embodiments, various steps or methods can be implemented in software or firmware that is stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any of the following technologies, known in the art, or combinations thereof, can be used: a discrete logic circuit having logic gates for implementing logic functions upon data signals, an application specific integrated circuit having appropriate combinational logic gates, a programmable gate array (PGA), a field programmable gate array (FPGA), and / or the like.

[0125] The technical features of the above-described embodiments can be combined in any manner, and for the sake of brevity, not all possible combinations are described, however, it is to be understood that any combination of the technical features is within the scope of the present specification.

[0126] The above embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but should not be understood as a limitation on the patent scope of the present application. It should be noted that, for ordinary skilled persons in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A method for processing digital human lip shape reasoning materials, characterized in that: include: Obtaining an original material video, replacing a first facial feature detection model provided in MuseTalk with a second facial feature detection model to obtain a replacement MuseTalk, inputting the original material video into the replacement MuseTalk, and extracting vertex coordinates of a framed facial area in the original material video through the replacement MuseTalk, adding pixel offsets to the vertex coordinates in proportion to a resolution of the original material video to obtain offset vertex coordinates, performing image segmentation and facial mask region generation on the original material video based on the offset vertex coordinates and according to a preprocessing script in the replacement MuseTalk to obtain a generated image, performing Gaussian blur on edges of the generated image, and binarizing the generated image to obtain a mask map; Obtaining a trained UNet and VAE decoder, and performing lip shape inference on the mask image based on the trained UNet and VAE decoder to obtain an inference image; Adjusting and overlaying the inference image based on the mask image to obtain a processed image; Performing ghost removal on the processed image to obtain a ghost-removed image; The eliminated image and the mask image are stored in a processing material set to complete the processing of the digital human mouth shape reasoning material.

2. The method for processing digital human lip shape reasoning materials according to claim 1, characterized in that: The first facial feature detection model is specifically a dw-ll_ucoco_384.pth model, and the second facial feature detection model is specifically a shape_predictor_68_face_landmarks.dat model.

3. The method for processing digital human lip shape reasoning materials according to claim 1, characterized in that: The step of performing lip shape inference on the mask image based on the trained UNet and VAE decoder to obtain an inference image includes: Set a numpy array of silent audio frames, and input the numpy array into the preset encoding model in MuseTalk for encoding to obtain text and feature representation; The text and feature representation are input into the trained UNet and VAE decoder, and the mask image and the corresponding mask area are input into the real-time digital human streaming framework for lip shape inference according to the trained UNet and VAE decoder to obtain an inference image.

4. The method for processing digital human lip shape reasoning materials according to claim 3, characterized in that: The preset encoding model is a frozen whisper-tiny model.

5. The method for processing digital human lip shape reasoning materials according to claim 1, characterized in that: The step of adjusting and superimposing the inference image based on the mask image to obtain a processed image includes: Segmenting and selecting the lower half of the face on the mask image and the inference image to obtain a first segmented image and a second segmented image; Performing polygon cropping on the first segmented image using a polygon cropping method in OpenCV to obtain a cropped image; The edges of the cropped image are blurred to obtain a blurred image, and the addWeighted method in OpenCV is used to superimpose the blurred image on the second separated image to obtain a processed image.

6. The method for processing digital human lip shape reasoning materials according to claim 1, characterized in that: The step of performing ghost removal on the processed image to obtain an eliminated image comprises: Identifying facial feature points in the lower half of the face of the processed image, and sequentially connecting the facial feature points to obtain lower half facial lines; Calculating convergence points with the same distance from facial feature points on the lower half of the face line, and diffusing all facial feature points toward the corresponding convergence points to obtain diffuse feature points; performing blurring processing on the area enclosed by the diffusion feature points and the facial feature points to obtain a diffusion image; The diffusion image is overlaid on the corresponding position of the processed image to obtain a cancellation image.

7. A digital human lip shape reasoning material processing system, characterized in that: include: a processing module for obtaining an original material video, replacing a first facial feature detection model provided in MuseTalk with a second facial feature detection model to obtain a replacement MuseTalk, inputting the original material video into the replacement MuseTalk, and extracting vertex coordinates of a framed facial area in the original material video through the replacement MuseTalk, adding pixel offsets to the vertex coordinates in proportion to a resolution of the original material video to obtain offset vertex coordinates, performing image segmentation and facial mask region generation on the original material video based on the offset vertex coordinates and according to a preprocessing script in the replacement MuseTalk to obtain a generated image, performing Gaussian blur on edges of the generated image, and performing binarization processing on the generated image to obtain a mask map; An inference module is used to obtain a trained UNet and VAE decoder, and perform lip shape inference on the mask image based on the trained UNet and VAE decoder to obtain an inference image; an adjustment module, configured to adjust and overlay the inference image based on the mask image to obtain a processed image; an elimination module, configured to eliminate ghost images from the processed image to obtain an eliminated image; The output module is used to store the eliminated image and the mask image into a processing material set to complete the processing of the digital human mouth shape reasoning material.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method for processing digital human mouth shape inference materials according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for processing digital human lip shape inference materials according to any one of claims 1 to 6 is implemented.

Images