Digital human mouth shape reasoning material processing method and system

Abstract

The invention provides a digital human mouth shape reasoning material processing method and system, and the method comprises the steps: obtaining an original material video, inputting the original material video into Muse Talk, and carrying out the preprocessing, so as to obtain a masking map; obtaining a trained UNet and a trained VAE decoder, and performing lip shape reasoning on the mask pattern based on the trained UNet and the trained VAE decoder to obtain a reasoning image; performing adjustment and superposition processing on the inference image based on the masking image to obtain a processed image; performing ghosting elimination on the processed image to obtain an eliminated image; and the elimination image and the masking image are stored in a processing material set so as to complete processing of the digital human mouth shape reasoning material, a series of processing is carried out on the image, ghosting in the image can be effectively removed, and the original problem of swelling and the problem that stripes exist on the edge are solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a method and system for processing digital human mouth shape reasoning materials. Background Art

[0002] With the rise and development of large natural language models and lip shape reasoning technology, the integration of AI digital people and humans will become a major trend in the future. At present, e-commerce, finance, entertainment, etc. have become relatively mature and have begun to have many service providers and users, while communications, cultural tourism, automobiles, government affairs, and even public welfare and charity have also begun to usher in large-scale exploration. In the process of real-time interaction between digital people, the conversational lip shape of digital people is particularly obvious and important. Musetalk is a real-time audio-driven lip synchronization model. The model can automatically adjust the facial image of the digital character according to the input audio signal so that its lip shape is highly synchronized with the audio content.

[0003] However, the original intention of the Musetalk model is to generate highly realistic virtual human speeches or performance videos, and it will automatically beautify the input materials during the inference process. However, for real-time interactive digital humans, the pre-recorded action materials will be divided into multiple pictures before lip shape inference. When audio-driven lip shape inference is performed, the segmented pictures will be used as the basis, and the mouth shape area after inference will be replaced with the original picture and output to achieve the effect of real-time lip shape inference;

[0004] In actual application, real-time lip shape inference based on Musetalk alone has the following disadvantages:

[0005] 1. There are stripes on the edge of the replacement area: Before training the mouth shape inference model, MuseTalk will detect facial feature points for each segmented image and calculate the rectangular area of ​​the framed face. The built-in facial recognition model and framed area algorithm can easily cause the area to fit the edge of the face too closely, resulting in reduced error tolerance when the images are superimposed later, resulting in stripes on the edge.

[0006] 2. Facial bulges appear at the moment of speaking and being silent: During the real-time lip shape inference process, when in silent state, the pre-processed image frames will be used for direct playback. Therefore, when the face of the material character is fuller, the implementation effect of MuseTalk will cause the facial contour after the inferred lip shape to be smaller than the original contour, resulting in facial bulges at the moment of speaking and being silent.

[0007] 3. There is a circle of virtual shadow outside the facial contour during speech: During the real-time lip shape inference process, the inferred mouth shape area will be superimposed on the original image after edge Gaussian blur. Because the facial area is reduced after inference, it cannot completely cover the facial edge of the original image when superimposed on the original image, resulting in a circle of virtual shadow outside the facial contour during speech. Summary of the invention

[0008] In order to solve the above technical problems, the present invention provides a method and system for processing digital human mouth shape reasoning materials, which are used to solve the technical problems raised in the prior art.

[0009] In a first aspect, an embodiment of the present invention provides the following technical solution, a method for processing digital human lip shape reasoning materials, comprising:

[0010] Obtaining an original material video, and inputting the original material video into MuseTalk for preprocessing to obtain a mask image;

[0011] Obtaining trained UNet and VAE decoders, and performing lip shape inference on the mask image based on the trained UNet and VAE decoders to obtain an inference image;

[0012] Adjusting and superimposing the inference image based on the mask image to obtain a processed image;

[0013] Performing ghost removal on the processed image to obtain a removed image;

[0014] The eliminated image and the mask image are stored in a processing material set to complete the processing of the digital human mouth shape reasoning material.

[0015] Compared with the prior art, the beneficial effects of the present application are as follows: the present invention first obtains the original material video, inputs the original material video into MuseTalk for preprocessing to obtain a mask image; then obtains the trained UNet and VAE decoder, and performs lip shape inference on the mask image based on the trained UNet and VAE decoder to obtain an inference image; then adjusts and overlays the inference image based on the mask image to obtain a processed image; then performs ghost removal on the processed image to obtain an eliminated image; finally, the eliminated image and the mask image are stored in a processing material set to complete the processing of the digital human lip shape inference material. The present invention can effectively remove ghost images in the image, solve the original bulging problem and the problem of stripes on the edge by performing a series of processing on the image. In the actual inference process, the unbeautified material is still used for inference, but the inferred face will be superimposed on the picture after face slimming, so that no ghost images will be generated during real-time inference. When jumping from speaking to stillness, since the face shape has been processed to the same intensity of beauty state, the original bulging problem is also solved.

[0016] Preferably, the step of inputting the original material video into MuseTalk for preprocessing to obtain a mask image comprises:

[0017] Replace the first facial feature detection model provided in MuseTalk with the second facial feature detection model to obtain a replacement MuseTalk;

[0018] Input the original material video into the replacement MuseTalk and extract the vertex coordinates of the framed facial area in the original material video through the replacement MuseTalk;

[0019] Adding pixel offsets to the vertex coordinates in proportion to the resolution of the original material video to obtain offset vertex coordinates;

[0020] Based on the offset vertex coordinates and according to the preprocessing script in the replacement MuseTalk, the original material video is subjected to image segmentation and facial mask area generation to obtain a generated image;

[0021] Gaussian blur is performed on the edge of the generated image and binarization is performed on the generated image to obtain a mask image.

[0022] Preferably, the first facial feature detection model is specifically a dw-ll_ucoco_384.pth model, and the second facial feature detection model is specifically a shape_predictor_68_face_landmarks.dat model.

[0023] Preferably, the step of performing lip shape inference on the mask image based on the trained UNet and VAE decoder to obtain an inference image includes:

[0024] Set a numpy array of silent audio frames, and input the numpy array into a preset encoding model in MuseTalk for encoding to obtain text and feature representations;

[0025] The text and feature representation are input into the trained UNet and VAE decoder, and the mask map and the corresponding mask area are input into the real-time digital human streaming framework for lip shape inference according to the trained UNet and VAE decoder to obtain an inference image.

[0026] Preferably, the preset encoding model is a frozen whisper-tiny model.

[0027] Preferably, the step of adjusting and superimposing the inference image based on the mask image to obtain a processed image includes:

[0028] Segmenting and selecting the lower half of the face region on the mask image and the inference image to obtain a first segmented image and a second segmented image;

[0029] Using the polygon clipping method in Opencv to perform polygon clipping on the first segmented image to obtain a clipped image;

[0030] The edge of the cropped image is blurred to obtain a blurred image, and the addWeighted method in Opencv is used to superimpose the blurred image on the second separated image to obtain a processed image.

[0031] Preferably, the step of performing ghost removal on the processed image to obtain the removed image comprises:

[0032] Identifying facial feature points in the lower half of the face area of ​​the processed image, and sequentially connecting the facial feature points to obtain lower half facial lines;

[0033] Calculating convergence points with the same distance from the facial feature points on the lower half of the face line, and diffusing all the facial feature points toward the corresponding convergence points to obtain diffuse feature points;

[0034] Performing blur processing on the area enclosed by the diffusion feature points and the facial feature points to obtain a diffusion image;

[0035] The diffused image is overlaid on the corresponding position on the processed image to obtain a removed image.

[0036] In a second aspect, an embodiment of the present invention provides the following technical solution, a digital human lip shape reasoning material processing system, the system comprising:

[0037] A processing module, used for obtaining an original material video, and inputting the original material video into MuseTalk for preprocessing to obtain a mask image;

[0038] An inference module, used for obtaining a trained UNet and VAE decoder, and performing lip shape inference on the mask image based on the trained UNet and VAE decoder to obtain an inference image;

[0039] An adjustment module, used for adjusting and superimposing the inference image based on the mask image to obtain a processed image;

[0040] An elimination module, used for performing ghost elimination on the processed image to obtain an eliminated image;

[0041] The output module is used to store the eliminated image and the mask image into a processing material set to complete the processing of the digital human mouth shape reasoning material.

[0042] In a third aspect, an embodiment of the present invention provides the following technical solution: a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned method for processing digital human mouth shape inference materials when executing the computer program.

[0043] In a fourth aspect, an embodiment of the present invention provides the following technical solution: a computer-readable storage medium having a computer program stored thereon, and the computer program, when executed by a processor, implements the above-mentioned method for processing digital human lip shape inference materials. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0045] Figure 1 A flowchart of a method for processing digital human mouth shape inference materials provided by an embodiment of the present invention;

[0046] Figure 2 A structural block diagram of a digital human lip shape inference material processing system provided by an embodiment of the present invention;

[0047] Figure 3 A schematic diagram of the hardware structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the embodiments of the present invention, and should not be construed as limiting the present invention.

[0049] Embodiment 1

[0050] In the first embodiment of the present invention, Figure 1 As shown, a method for processing digital human mouth shape reasoning materials includes:

[0051] S1. Obtain an original material video, and input the original material video into MuseTalk for preprocessing to obtain a mask image;

[0052] Wherein, the step S1 comprises:

[0053] S1. Replace the first facial feature detection model in MuseTalk with the second facial feature detection model to obtain a replacement MuseTalk;

[0054] Specifically, the first facial feature detection model is a dw-ll_ucoco_384.pth model, and the second facial feature detection model is a shape_predictor_68_face_landmarks.dat model.

[0055] Among them, the dw-ll_ucoco_384.pth model is a feature detection model that comes with MuseTalk. In some images of people with crooked faces, there will be problems with incorrect selection of facial contour feature points. After selecting the facial contour, the dw-ll_ucoco_384.pth model will select and calculate the maximum and minimum values ​​of the horizontal and vertical coordinates of all feature points in the coordinate axis to find the coordinates of the four vertices of the rectangle that selects the facial area. At this time, the error caused by feature point recognition cannot be processed. Therefore, the present invention adopts the shape_predictor_68_face_landmarks.dat model.

[0056] S2. Input the original material video into the replacement MuseTalk and extract the vertex coordinates of the framed facial area in the original material video through the replacement MuseTalk.

[0057] S3, adding pixel offsets to the vertex coordinates in proportion to the resolution of the original material video to obtain offset vertex coordinates;

[0058] Specifically, by adding a certain pixel offset, it can be ensured that the chin, forehead, and cheeks of the face are completely contained in the rectangular area.

[0059] S4, performing image segmentation and facial mask region generation on the original material video based on the offset vertex coordinates and according to the preprocessing script in the replacement MuseTalk to obtain a generated image;

[0060] S5, performing Gaussian blur on the edge of the generated image and performing binarization processing on the generated image to obtain a mask image;

[0061] Specifically, the image segmentation will calculate how many frames of images should be sampled per second according to the frame rate of the imported video, and then give the video path, frame rate, resolution and other parameters to the ffmpeg tool to achieve image segmentation. The facial mask uses the previous facial area selection and facial feature point detection to obtain the parameters of the face in the current frame image. The face segmentation model face-parsing.PyTorch integrated by MuseTalk segments the facial area (irregular area) and converts the edge into a binary image after Gaussian blurring, that is, the mask image is obtained.

[0062] S2, obtaining the trained UNet and VAE decoders, and performing lip shape inference on the mask image based on the trained UNet and VAE decoders to obtain an inference image;

[0063] Specifically, the UNet and VAE decoders here can be trained using some template inference images.

[0064] Wherein, the step S2 comprises:

[0065] S21, setting a numpy array of silent audio frames, and inputting the numpy array into a preset encoding model in MuseTalk for encoding to obtain text and feature representation;

[0066] Specifically, the preset encoding model is a frozen whisper-tiny model. A numpy array of silent audio frames is manually set. MuseTalk will use the frozen whisper-tiny model to encode the audio. This process will convert the audio into corresponding text and feature representations. After combining the text and feature representations, MuseTalk will use the UNet and VAE decoders produced by the above training to perform lip shape inference on the picture frames.

[0067] S22, inputting the text and feature representation into the trained UNet and VAE decoders, and inputting the mask image and the corresponding mask area into the real-time digital human streaming framework for lip shape reasoning according to the trained UNet and VAE decoders to obtain a reasoning image;

[0068] Specifically, even in silent mode, MuseTalk's lip shape will change during the inference process. However, since MuseTalk automatically slims the face during inference, after a round of inference, we can get the picture frame with slimmed face (beautified) but with the mouth opened and closed for each frame.

[0069] S3, adjusting and superimposing the inference image based on the mask image to obtain a processed image;

[0070] Wherein, the step S3 comprises:

[0071] S31, segmenting and selecting the lower half of the face area on the mask image and the inference image to obtain a first segmented image and a second segmented image.

[0072] S32, using the polygon clipping method in Opencv to perform polygon clipping on the first segmented image to obtain a clipped image;

[0073] Specifically, since the lower half of the face is an irregular area, it can be understood that the feature points of the two cheeks are sequentially connected with other feature points in the chin direction, and finally a closed polygonal area can be formed.

[0074] S33, blurring the edge of the cropped image to obtain a blurred image, and superimposing the blurred image on the second separated image using the addWeighted method in Opencv to obtain a processed image;

[0075] Each frame of the processed image has the same expression as the image in the original material video.

[0076] S4, performing ghost removal on the processed image to obtain a ghost-removed image;

[0077] Wherein, the step S4 comprises:

[0078] S41, identifying facial feature points in the lower half of the face area of ​​the processed image, and sequentially connecting the facial feature points to obtain lower half facial lines;

[0079] Specifically, in the above steps, the image after beautification, that is, the processed image, has the face-slimming area superimposed on the processed mask image, so the contour ghost of the face already exists, so in this step, the contour ghost can be removed.

[0080] S42, calculating the convergence points with the same distances from the facial feature points on the lower half of the face line, and diffusing all the facial feature points toward the corresponding convergence points to obtain diffuse feature points;

[0081] Specifically, each facial feature point has the same convergence point, and then expands outward along the direction from the facial feature point to the convergence point. In the present application, for a resolution of 1K, the diffusion distance is 15 pixels.

[0082] S43: performing blur processing on the area enclosed by the diffusion feature points and the facial feature points to obtain a diffusion image.

[0083] S44, overlaying the diffusion image at a corresponding position on the processed image to obtain an eliminated image;

[0084] Specifically, the above process is similar to the process of cloning a stamp, and the processed diffusion image is covered on the corresponding area of ​​the original image to achieve the effect of removing the ghost image.

[0085] S5. Storing the eliminated image and the mask image in a processing material set to complete the processing of the digital human mouth shape reasoning material.

[0086] The method for processing digital human lip shape reasoning materials provided in the first embodiment of the present invention first obtains the original material video, inputs the original material video into MuseTalk for preprocessing to obtain a mask map; then obtains the trained UNet and VAE decoders, and performs lip shape reasoning on the mask map based on the trained UNet and VAE decoders to obtain a reasoning image; then adjusts and superimposes the reasoning image based on the mask map to obtain a processed image; then performs ghost removal on the processed image to obtain a removed image; finally, the removed image and the mask map are stored in a processing material set to complete the processing of the digital human lip shape reasoning material. The present invention can effectively remove ghosts in the image, solve the original bulging problem and the problem of stripes on the edge by performing a series of processing on the image. In the actual reasoning process, the material without beautification is still used for reasoning, but the face after reasoning will be superimposed on the picture after face thinning, so that no ghost will be generated during real-time reasoning. When jumping from speaking to stillness, since the face shape has been processed to the same intensity of beauty state, the original bulging problem is also solved.

[0087] Embodiment 2

[0088] like Figure 2 As shown, in the second embodiment of the present invention, a digital human mouth shape reasoning material processing system is provided, and the system includes:

[0089] Processing module 1 is used to obtain the original material video, and input the original material video into MuseTalk for preprocessing to obtain a mask image;

[0090] Inference module 2, used for obtaining the trained UNet and VAE decoders, and performing lip shape inference on the mask image based on the trained UNet and VAE decoders to obtain an inference image;

[0091] An adjustment module 3, configured to adjust and overlay the inference image based on the mask image to obtain a processed image;

[0092] The elimination module 4 is used to eliminate ghost images from the processed image to obtain an eliminated image;

[0093] Output module 5, used for storing the eliminated image and the mask image into a processing material set to complete the processing of the digital human mouth shape reasoning material;

[0094] The processing module 1 comprises:

[0095] A replacement submodule, used to replace the first facial feature detection model provided in MuseTalk with the second facial feature detection model to obtain a replacement MuseTalk;

[0096] A coordinate submodule, for inputting the original material video into the replacement MuseTalk and extracting the vertex coordinates of the framed facial area in the original material video through the replacement MuseTalk;

[0097] An offset submodule, used to increase pixel offsets for the vertex coordinates in proportion to the resolution of the original material video to obtain offset vertex coordinates;

[0098] A generation submodule, for performing image segmentation and facial mask region generation on the original material video based on the offset vertex coordinates and according to the preprocessing script in the replacement MuseTalk, so as to obtain a generated image;

[0099] The binarization submodule is used to perform Gaussian blur on the edge of the generated image and perform binarization processing on the generated image to obtain a mask image.

[0100] The reasoning module 2 comprises:

[0101] A setting submodule is used to set a numpy array of a silent audio frame, and input the numpy array into a preset encoding model in MuseTalk for encoding to obtain text and feature representation;

[0102] The reasoning submodule is used to input the text and feature representation into the trained UNet and VAE decoders, and input the mask map and the corresponding mask area into the real-time digital human streaming framework for lip shape reasoning according to the trained UNet and VAE decoders to obtain the reasoning image.

[0103] The adjustment module 3 comprises:

[0104] A segmentation submodule, used for segmenting and selecting the lower half of the face area of ​​the mask image and the inference image to obtain a first segmented image and a second segmented image;

[0105] A cutting submodule, used for performing polygon cutting on the first segmented image using a polygon cutting method in Opencv to obtain a cut image;

[0106] The blur submodule is used to blur the edges of the cropped image to obtain a blurred image, and the addWeighted method in Opencv is used to superimpose the blurred image on the second separated image to obtain a processed image.

[0107] The elimination module comprises:

[0108] A line connection submodule, used for identifying facial feature points in the lower half of the face area of ​​the processed image, and sequentially connecting the facial feature points to obtain the lower half of the face lines;

[0109] A diffusion submodule, used for calculating the convergence points with the same distance from the facial feature points on the lower half of the face line, and diffusing all the facial feature points toward the corresponding convergence points to obtain diffuse feature points;

[0110] A fuzzy submodule, used for fuzzifying the area enclosed by the diffusion feature points and the facial feature points to obtain a diffusion image;

[0111] The covering submodule is used to cover the diffusion image at a corresponding position on the processed image to obtain a eliminated image.

[0112] In some other embodiments of the present invention, the embodiments of the present invention provide the following technical solutions: a computer, comprising a memory 102, a processor 101, and a computer program stored in the memory 102 and executable on the processor 101; the processor 101 implements the above-mentioned method for processing digital human mouth shape inference materials when executing the computer program.

[0113] Specifically, the processor 101 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiment of the present invention.

[0114] Among them, the memory 102 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 102 may include a hard disk drive (HDD), a floppy disk drive, a solid state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 102 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 102 may be inside or outside a data processing device. In a specific embodiment, the memory 102 is a non-volatile memory. In a specific embodiment, the memory 102 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (Programmable Read-Only Memory, PROM for short), an erasable PROM (ErasableProgrammable Read-Only Memory, EPROM for short), an electrically erasable PROM (Electrically ErasableProgrammable Read-Only Memory, EEPROM for short), an electrically alterable ROM (ElectricallyAlterable Read-Only Memory, EAROM for short) or a flash memory (FLASH) or a combination of two or more of these. Under appropriate circumstances, the RAM can be a static random access memory (SRAM) or a dynamic random access memory (DRAM), wherein the DRAM can be a fast page mode dynamic random access memory (FPMDRAM), an extended data output dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.

[0115] The memory 102 may be used to store or cache various data files that need to be processed and / or used for communication, as well as possible computer program instructions executed by the processor 101 .

[0116] The processor 101 implements the above-mentioned digital human mouth shape inference material processing method by reading and executing the computer program instructions stored in the memory 102.

[0117] In some embodiments, the computer may further include a communication interface 103 and a bus 100. Figure 3 As shown, the processor 101, the memory 102, and the communication interface 103 are connected via a bus 100 and communicate with each other.

[0118] The communication interface 103 is used to implement communication between the modules, devices, units and / or equipment in the embodiment of the present invention. The communication interface 103 can also implement data communication with other components such as: external devices, image / data acquisition equipment, databases, external storage, and image / data processing workstations.

[0119] The bus 100 includes hardware, software or both, and couples the components of the computer device to each other. The bus 100 includes but is not limited to at least one of the following: a data bus, an address bus, a control bus, an expansion bus, and a local bus. By way of example and not limitation, bus 100 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of the above. Where appropriate, bus 100 may include one or more buses. Although embodiments of the present invention describe and illustrate a particular bus, the present invention contemplates any suitable bus or interconnect.

[0120] The computer can execute the digital human mouth shape reasoning material processing method of the present invention based on the digital human mouth shape reasoning material processing system obtained, thereby realizing the digital human mouth shape reasoning material processing.

[0121] In some further embodiments of the present invention, in combination with the above-mentioned method for processing digital human lip shape inference materials, the embodiments of the present invention provide the following technical solutions: a storage medium having a computer program stored thereon, and the computer program implements the above-mentioned method for processing digital human lip shape inference materials when executed by a processor.

[0122] Those skilled in the art will appreciate that the logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable instructions for implementing logical functions, and may be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For purposes of this specification, a "computer-readable medium" may be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0123] More specific examples of readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0124] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or a combination thereof: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0125] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0126] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A method for processing digital human lip shape reasoning materials, characterized in that: include: Obtaining an original material video, and inputting the original material video into MuseTalk for preprocessing to obtain a mask image; Obtaining trained UNet and VAE decoders, and performing lip shape inference on the mask image based on the trained UNet and VAE decoders to obtain an inference image; Adjusting and superimposing the inference image based on the mask image to obtain a processed image; Performing ghost removal on the processed image to obtain a removed image; The eliminated image and the mask image are stored in a processing material set to complete the processing of the digital human mouth shape reasoning material.

2. The method for processing digital human lip shape reasoning materials according to claim 1, characterized in that: The step of inputting the original material video into MuseTalk for preprocessing to obtain a mask image comprises: Replace the first facial feature detection model provided in MuseTalk with the second facial feature detection model to obtain a replacement MuseTalk; Input the original material video into the replacement MuseTalk and extract the vertex coordinates of the framed facial area in the original material video through the replacement MuseTalk; Adding pixel offsets to the vertex coordinates in proportion to the resolution of the original source video to obtain offset vertex coordinates; Based on the offset vertex coordinates and according to the preprocessing script in the replacement MuseTalk, the original material video is subjected to image segmentation and facial mask area generation to obtain a generated image; Gaussian blur is performed on the edge of the generated image and binarization is performed on the generated image to obtain a mask image.

3. The method for processing digital human lip shape reasoning materials according to claim 2, characterized in that: The first facial feature detection model is specifically a dw-ll_ucoco_384.pth model, and the second facial feature detection model is specifically a shape_predictor_68_face_landmarks.dat model.

4. The method for processing digital human lip shape reasoning materials according to claim 1, characterized in that: The step of performing lip shape reasoning on the mask image based on the trained UNet and VAE decoder to obtain the reasoning image includes: Set a numpy array of silent audio frames, and input the numpy array into a preset encoding model in MuseTalk for encoding to obtain text and feature representations; The text and feature representation are input into the trained UNet and VAE decoder, and the mask map and the corresponding mask area are input into the real-time digital human streaming framework for lip shape inference according to the trained UNet and VAE decoder to obtain an inference image.

5. The method for processing digital human lip shape reasoning materials according to claim 4, characterized in that: The preset encoding model is a frozen whisper-tiny model.

6. The method for processing digital human lip shape reasoning materials according to claim 1, characterized in that: The step of adjusting and superimposing the inference image based on the mask image to obtain a processed image comprises: Segmenting and selecting the lower half of the face region on the mask image and the inference image to obtain a first segmented image and a second segmented image; Using the polygon clipping method in Opencv to perform polygon clipping on the first segmented image to obtain a clipped image; The edge of the cropped image is blurred to obtain a blurred image, and the addWeighted method in Opencv is used to superimpose the blurred image on the second separated image to obtain a processed image.

7. The method for processing digital human lip shape reasoning materials according to claim 1, characterized in that: The step of performing ghost removal on the processed image to obtain a ghost-removed image comprises: Identifying facial feature points in the lower half of the face area of ​​the processed image, and sequentially connecting the facial feature points to obtain lower half facial lines; Calculating convergence points with the same distance from the facial feature points on the lower half of the face line, and diffusing all the facial feature points toward the corresponding convergence points to obtain diffuse feature points; Performing blur processing on the area enclosed by the diffusion feature points and the facial feature points to obtain a diffusion image; The diffused image is overlaid on the corresponding position on the processed image to obtain a removed image.

8. A digital human lip shape reasoning material processing system, characterized in that: include: A processing module, used for obtaining an original material video, and inputting the original material video into MuseTalk for preprocessing to obtain a mask image; An inference module, used for obtaining a trained UNet and VAE decoder, and performing lip shape inference on the mask image based on the trained UNet and VAE decoder to obtain an inference image; An adjustment module, used for adjusting and superimposing the inference image based on the mask image to obtain a processed image; An elimination module, used for performing ghost elimination on the processed image to obtain an eliminated image; The output module is used to store the eliminated image and the mask image into a processing material set to complete the processing of the digital human mouth shape reasoning material.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for processing digital human mouth shape inference materials as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for processing digital human lip shape inference materials according to any one of claims 1 to 7 is implemented.

Images