A multi-task real-time cutout method and system for digital population synthesis
Through the deep learning model and multi-task face image generation model, combining audio and lip-like image features, multi-task real-time cutout of digital lip-like synthesis is realized, solving the problems of high cost and low efficiency in the existing technology, and improving real-time Sex and matching, suitable for a variety of scenarios.
Patent Information
- Application Number
- CN202411215967.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-02
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-09-02
AI Technical Summary
The existing digital verbal synthesis technology is limited by high cost and low efficiency in the process of cutting pictures, especially the dependence on expert knowledge in the audio field and real-time requirements. Traditional methods cannot meet the needs of multi-task real-time cutting pictures.
Deep learning model is used to extract audio feature and extract lip image features, combine multi-task face image generation model, and build pre-trained models for real-time cutout by obtaining and fusion features at one time, reducing dependence on third-party APIs, and Design the multi-task network output alpha channel image for cutting pictures in the model.
It reduces project delivery time and cost, realizes flexibility for green screen and real scene background, and improves delay performance in live broadcast scenarios, with lip matching reaching 98% and alpha channel matching reaching 99.2%.
Smart Images

Figure CN118735935B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical fields of artificial intelligence, digital humans and image synthesis, and in particular relates to a multi-task real-time cutout method and system in digital population synthesis. Background Art
[0002] Due to the current national emphasis on digital systems and the vigorous development of the current e-commerce environment, digital population synthesis technology has emerged. The traditional method uses expert knowledge in the audio field to manually extract audio features, and uses CNN (convolutional neural network) to extract lip picture features. The two features are fused and input into the generative model (generative adversarial network GAN, variational encoder VAE, etc.) for model training. This solution is greatly limited by expert knowledge in the audio field and has a high implementation cost. Manually extracted audio features have a low matching degree of the lip shape synthesized by the features, which seriously affects the perception of the synthesized video. With the development of deep learning, audio features are extracted using the huber (speech recognition pre-training) model, and lip picture features are extracted using the transformer architecture model. The extracted features are fused and input into the generative model for model training, which can alleviate the disadvantages of traditional methods to a certain extent.
[0003] In order to adapt the digital human to be used in various occasions, it is necessary to replace the background of the model output lip picture. Regardless of the traditional method or the deep learning method, the lip picture output by the model needs to be cut out. Generally, the following two solutions are used for cutout. Solution 1: Use an open source cutout model, such as Baidu's open source PP-Matting and the RMBG-v1.4 model developed by BRIAAI. Solution 2: Directly purchase a paid cutout solution, such as the cutout API (Application Programming Interface) provided by a third party. The above two cutout solutions have their own defects. Solution 1 is heavily dependent on open source solutions, cannot be changed twice, and commercial use is limited. Solution 2 will incur high costs. Finally, both require the lip picture to be generated before the cutout operation, and the real-time performance is severely limited. Summary of the invention
[0004] In order to solve the problems in the prior art, the present invention provides a multi-task real-time cutout method and system in digital population synthesis.
[0005] The technical solution adopted by the present invention is as follows:
[0006] In a first aspect, the present invention discloses a multi-task real-time cutout method in digital population synthesis, comprising the following steps:
[0007] 1) Get the complete oral video of the specified text as the historical video, and obtain the audio data, RGB lip shape image and alpha channel image of the historical video;
[0008] 2) obtaining audio features of the audio data and facial region images corresponding to the RGB lip shape image and the alpha channel image, and aligning the audio features with the two facial region images; performing feature extraction on the aligned audio features and the facial region images corresponding to the RGB lip shape image, respectively, and performing feature fusion on the two feature extraction results;
[0009] 3) constructing a multi-task face image generation model, inputting the feature fusion result into the model, obtaining a first RGB lip shape picture and a first alpha channel picture, updating the parameters of the model according to the first RGB lip shape picture and the first alpha channel picture, and obtaining a pre-trained model;
[0010] 4) Use the pre-trained model to process the audio of the face image to be generated, obtain the corresponding RGB lip shape picture and alpha channel picture, and cut out the RGB lip shape picture based on the alpha channel picture to complete the cutout.
[0011] In a second aspect, the present invention discloses a multi-task real-time cutout system in digital population synthesis, comprising:
[0012] The data and image acquisition module is used to obtain the complete oral video of the specified text as the historical video, and obtain the audio data, RGB lip shape image and alpha channel image of the historical video;
[0013] A feature fusion acquisition module is used to obtain the audio features of the audio data and the face area pictures corresponding to the RGB lip shape picture and the alpha channel picture, align the audio features with the two face area pictures; perform feature extraction on the aligned audio features and the face area pictures corresponding to the RGB lip shape pictures, respectively, and perform feature fusion on the two feature extraction results;
[0014] A model building and training module, which is used to build a multi-task face image generation model, input the feature fusion result into the model, obtain a first RGB lip shape picture and a first alpha channel picture, update the parameters of the model according to the first RGB lip shape picture and the first alpha channel picture, and obtain a pre-trained model;
[0015] The image generation and cutout module is used to process the audio of the face image to be generated using a pre-trained model to obtain the corresponding RGB lip shape image and alpha channel image, and cut out the RGB lip shape image based on the alpha channel image to complete the cutout.
[0016] In a third aspect, the present invention discloses an electronic device, including a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the above-mentioned multi-task real-time cutout method in digital population synthesis.
[0017] In a fourth aspect, the present invention discloses a machine-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, they are used to implement the above-mentioned multi-task real-time cutout method in digital population synthesis.
[0018] Compared with the prior art, the invention has the following innovative features and beneficial effects:
[0019] (1) Since deep learning models are used throughout the process to extract audio features, the reliance on expert knowledge in the audio field is reduced. During the specific implementation of the project, the project delivery time can be quickly improved, and the fees paid to relevant experts can be reduced, thereby reducing costs.
[0020] (2) Since the present invention only needs to call the cutout API once during the data processing process, it is no longer necessary to call the cutout API during the subsequent model inference process, thereby reducing the cost expenditure of this part. At the same time, it also reduces the dependence on the cutout requirements and is not limited by the cutout capabilities of third parties.
[0021] (3) The present invention is not limited to any background image input requirements, that is, it can be a green screen background or a real scene background.
[0022] (4) The present invention outputs the corresponding alpha channel image at the same time as the lip-sync image, without the need for secondary cutouts, which makes it possible for live broadcasts, which have very high latency requirements. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1 is a flow chart of the present invention;
[0024] Figure 2 is a flow chart of the present invention for processing training data;
[0025] Figure 3 is a structural diagram of an encoder of the present invention;
[0026] Figure 4 is a structural diagram of the multi-task face image generation model of the present invention;
[0027] Figure 5 It is a schematic diagram of an electronic device of the present invention. DETAILED DESCRIPTION
[0028] The present invention is further described and illustrated below in conjunction with specific embodiments. The embodiments are merely exemplary of the present disclosure and do not define the scope of limitation. The technical features of each embodiment of the present invention may be combined accordingly without conflicting with each other.
[0029] The present invention is proposed to solve the problem of real-time cutout in digital population synthesis system. The present invention adopts a self-developed generative adversarial network architecture and a multi-task learning algorithm that integrates the cutout solution into the real-time generation of the model, which solves the above two difficulties very well.
[0030] like Figure 1 As shown, the present invention is achieved through the following technical solutions:
[0031] Step 1: First, call the third-party excellent cutout API (Application Programming Interface) with the training data (usually a complete oral video of a specified text) to get the corresponding webm format video, and then use the ffmpeg tool to extract the corresponding audio data, RGB lip shape image and alpha channel image from the webm format video; Figure 2 shown.
[0032] Step 2: extract features from the audio data obtained above through the wenet model, call the open source face recognition model to identify the corresponding face area of the RGB lip shape picture and the alpha channel picture, and the open source face recognition model can be the SCRFD (Sample and Computation Re-allocation for Face Detection) model; align the audio features, the face area picture of the RGB lip shape picture, and the face area picture of the alpha channel picture to prepare for the input data of the model. The specific processing process is explained in detail below.
[0033] Step 3: Construct a multi-task face image generation model, which is a GAN network based on Transformer and CNN. The aligned audio features and the face area image corresponding to the RGB lip shape image are processed separately, and the two processing results are feature fused; wherein, the aligned audio features are encoded by an encoder, which is composed of multiple two-dimensional convolutional layers connected by residuals, and the specific structure is as follows: Figure 3As shown in the figure, a downsampling convolutional neural network structure is used to extract features from the face area image corresponding to the RGB lip shape image. The structure is composed of 4 layers of convolutional neural networks in series, and the image dimension is halved after each layer of convolution operation. The two processing results are feature fused (concat), and finally the feature fusion result is input into the multi-task face image generation model. The specific structure of the multi-task face image generation model is shown in the figure. Figure 4 As shown, the multi-task face image generation model includes a generator module and a discriminator module. The generator module includes a layer of Transformer network, a layer of Resnet network composed of CNN and a multi-task network. The feature fusion result is input into the Transformer network for primary image information extraction to obtain a first result and input into the Resnet network. The Resnet network further learns the image information. The Resnet network outputs a second result to the multi-task network. The purpose of the multi-task network is to enable the model to directly output an alpha channel image while outputting a lip shape image. The multi-task network includes two CNN convolutional neural networks. The second result is respectively input into the two CNN convolutional neural networks. The multi-task network outputs a first RGB lip shape image and a first alpha channel image to the discriminator module, and simultaneously inputs a face area image corresponding to the aligned alpha channel image to the discriminator module. The discriminator module calculates a loss value based on the first RGB lip shape image, the first alpha channel image, the face area image corresponding to the aligned RGB lip shape image and the face area image corresponding to the alpha channel image using a loss function, updates the parameters of the model, and obtains a pre-trained multi-task face image generation model.
[0034] For the multi-task network, the present invention designs a multi-task algorithm solution, that is, two classification heads are designed, the first classification head outputs RGB lip shape pictures, and the second classification head outputs alpha channel pictures, and the two classification heads are both composed of CNN convolutional neural networks.
[0035] The discriminator module adopts the discriminator network structure in the open source PatchGan. In terms of loss calculation, the present invention has undergone some special designs, which will be explained in detail below.
[0036] Step 4: Use the pre-trained model to process the audio of the face image to be generated, obtain the corresponding RGB lip shape picture and alpha channel picture, and cut out the RGB lip shape picture based on the alpha channel picture to complete the cutout.
[0037] Videos in webm format can be understood as videos with transparent channels, that is, extracting a picture from a webm video can be separated into four-channel PNG pictures: R, G, B, and alpha. The RGB picture is the original picture with lip shapes, and the alpha channel can be understood as a mask corresponding to the RGB lip shape picture. The alpha channel can be used to cut out the original picture, that is, the innovation of this solution is to design a multi-task solution, which outputs the corresponding alpha channel at the same time as the corresponding lip shape picture.
[0038] Furthermore, the step 2 is specifically as follows: the audio data, RGB lip shape picture and alpha channel picture obtained in step 1 need to be aligned before being input into the model. The audio data is uniformly resampled to a sampling rate of 16000 (16000 sampling points per second), and then the wenet model is used to extract audio features, with a hop_size of 320 (the window size during the audio feature extraction process, in order to align with the RGB lip shape picture later); for the original video data, before calling the ffmpeg tool to obtain the PNG picture, it is necessary to first adjust the FPS (frame rate, that is, 25 png pictures can be disassembled from one second of video) to 25, the purpose is to align with the audio features of the previous step, and the alignment calculation process is as follows: 16000 / 320 = 50, 50 audio features can be extracted in one second, and there are 25 lip shape pictures in one second of video, that is, one picture corresponds to 2 audio features.
[0039] The RGB lip shape images and alpha channel images are both used to perform facial area recognition by calling the open source avatar recognition model (SCRFD model). The facial area images corresponding to the RGB lip shape images and alpha channel images are obtained and cropped to 512*512 before being input into the model.
[0040] Further detailed explanation at the model level is as follows:
[0041] This model architecture design is a model architecture that combines multi-task, convolutional neural network and transformer.
[0042] The multi-task is reflected in that the general lip shape synthesis model finally outputs only one lip shape picture, while the model of the present invention outputs a corresponding alpha channel picture at the same time as the lip shape picture for subsequent cutout use. From the perspective of model design, two branches (RGB lip shape picture and alpha channel picture) are output in the last layer of the model. Figure 4 shown.
[0043] A detailed explanation of the loss in step 3 above: In order to increase the output of the alpha channel so that it can converge quickly, this method proposes a new data balance loss in loss design. Since the obtained alpha channel values are normalized to between 0 and 1, most of the data are around 1, which will lead to category imbalance. In order to solve this problem, experimental data shows that adjusting the alpha channel data and combining it with the loss designed by this method will have the best effect.
[0044] The specific manifestation of category balance is that the alpha value greater than 0.98 in the picture is forced to be set to 1, and the data less than 0.98 is forced to be designed to 50, that is, , ,in, It is the alpha channel data of the face area image corresponding to the aligned alpha channel image; It is the alpha channel data after setting; The alpha channel data of the first alpha channel image. The final loss The calculation formula is as follows:
[0045] ;
[0046] in, is a hyperparameter with values of 0.5 and 10 respectively.
[0047] For step four, in actual application, the audio of the face image to be generated is obtained, and the audio is resampled in the same way as during training. In order to be able to directly input it into the pre-trained model later, this step uses the wenet model during training to extract features of the resampled audio to obtain audio features; the audio features are encoded by the encoder during training, and the encoding results are fused with the processed face area image input during model training. The fusion result is input into the pre-trained model to obtain the RGB lip shape image and alpha channel image corresponding to the audio. Finally, the RGB lip shape image is cut out based on the alpha channel image to complete the cutout. Finally, in order to measure the performance of the method of the present invention, two indicators, namely, lip shape matching and alpha channel matching, are used. The lip shape matching refers to that, as can be seen from the above step 1, the training data includes audio data, RGB lip shape pictures and alpha channel pictures. By arbitrarily specifying a section of oral video data, the input data is obtained and input into the model. The model generates a first RGB lip shape picture and a first alpha channel picture. The generated first RGB lip shape picture and the first alpha channel picture respectively capture the mouth shape part, and then calculate the similarity between the two. The final lip shape matching of the method of the present invention can reach 98%.
[0048] When calculating the matching degree of the alpha channel feature, no interception operation is performed. The alpha channel image output by the entire model and all the pixels of the alpha channel input are directly calculated. The final alpha channel feature reaches 99.2%.
[0049] In this embodiment, a multi-task real-time cutout system for digital population synthesis is also provided, including:
[0050] The data and image acquisition module is used to obtain the complete oral video of the specified text as the historical video, and obtain the audio data, RGB lip shape image and alpha channel image of the historical video;
[0051] A feature fusion acquisition module is used to obtain the audio features of the audio data and the face area images corresponding to the RGB lip shape image and the alpha channel image, and align the audio features with the two face area images; process the aligned audio features and the face area images corresponding to the RGB lip shape image respectively, and perform feature fusion on the two processing results;
[0052] A model building and training module, which is used to build a multi-task face image generation model, input the feature fusion result into the model, obtain a first RGB lip shape picture and a first alpha channel picture, update the parameters of the model according to the first RGB lip shape picture and the first alpha channel picture, and obtain a pre-trained model;
[0053] The image generation and cutout module is used to process the audio of the face image to be generated using a pre-trained model to obtain the corresponding RGB lip shape image and alpha channel image, and cut out the RGB lip shape image based on the alpha channel image to complete the cutout.
[0054] For the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment, and the implementation methods of the remaining modules will not be repeated here. The system embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.
[0055] The embodiments of the system of the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The system embodiments can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, the corresponding computer program instructions in the non-volatile memory are read into the memory by the processor of any device with data processing capabilities and run.
[0056] like Figure 5 As shown, an embodiment of the present invention further provides an electronic device, including a memory and a processor;
[0057] The memory is used to store computer programs;
[0058] The processor is used to implement the above-mentioned multi-task real-time cutout method in digital population synthesis when executing the computer program.
[0059] The embodiment of the present invention further provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the multi-task real-time cutout method in digital population synthesis is implemented.
[0060] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.
[0061] Obviously, the above-described embodiments and drawings are only some examples of the present application. For those of ordinary skill in the art, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that, although the work done in this development process may be complicated and lengthy, for those of ordinary skill in the art, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application. Without departing from the concept of the present application, several variations and improvements can also be made, which all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.
Claims
1. A multi-task real-time cutout method for digital population synthesis, characterized in that: The following steps are involved: 1) Get the complete oral video of the specified text as the historical video, and obtain the audio data, RGB lip shape image and alpha channel image of the historical video; 2) obtaining the audio features of the audio data and the facial region images corresponding to the RGB lip shape image and the alpha channel image, and aligning the audio features with the two facial region images; Feature extraction is performed on the aligned audio features and the facial region image corresponding to the RGB lip shape image, and the two results are fused; 3) constructing a multi-task face image generation model, inputting the feature fusion result into the model, obtaining a first RGB lip shape picture and a first alpha channel picture, updating the parameters of the model according to the first RGB lip shape picture and the first alpha channel picture, and obtaining a pre-trained model; 4) Use the pre-trained model to process the audio of the face image to be generated, obtain the corresponding RGB lip shape picture and alpha channel picture, and perform cutout on the RGB lip shape picture based on the alpha channel picture to complete the cutout; In step 3), the multi-task face image generation model is a GAN network based on Transformer and CNN, and the multi-task face image generation model includes a generator module and a discriminator module. The generator module includes a layer of Transformer network, a layer of Resnet network composed of CNN, and a multi-task network. The feature fusion result is input into the Transformer network to obtain a first result and input into the Resnet network. The Resnet network outputs a second result to the multi-task network. The multi-task network includes two CNN convolutional neural networks. The second result is respectively input into the two CNN convolutional neural networks. The multi-task network outputs a first RGB lip shape picture and a first alpha channel picture to the discriminator module, and at the same time, a face area picture corresponding to the aligned alpha channel picture is input into the discriminator module. The discriminator module calculates the loss based on the first RGB lip shape picture, the first alpha channel picture, the face area picture corresponding to the aligned RGB lip shape picture, and the face area picture corresponding to the alpha channel picture using a loss function, updates the parameters of the model, and obtains a pre-trained multi-task face image generation model; The discriminator module is the discriminator network structure in PatchGan; The loss for: ; ; in, It is the alpha channel data of the face area image corresponding to the aligned alpha channel image; It is the alpha channel data after setting; The alpha channel data of the first alpha channel image; is a hyperparameter.
2. The multi-task real-time cutout method in digital population synthesis according to claim 1, characterized in that: In step 1), the step of obtaining audio data, RGB lip shape images and alpha channel images of historical videos includes: Obtain a historical video in a webm format, and then obtain corresponding audio data, RGB lip shape images, and alpha channel images from the historical video in the webm format using an ffmpeg tool.
3. The multi-task real-time cutout method in digital population synthesis according to claim 1, characterized in that: In step 2), the step of obtaining the audio features of the audio data and the facial region images corresponding to the RGB lip shape image and the alpha channel image respectively includes: Using a wenet model to extract features from the audio data to obtain audio features of the audio data; The SCRFD model is used to perform facial region recognition on the RGB lip shape image and the alpha channel image to obtain the facial region images corresponding to the RGB lip shape image and the alpha channel image respectively.
4. The multi-task real-time cutout method in digital population synthesis according to claim 1, characterized in that: In step 2), the aligned audio features and the facial region images corresponding to the RGB lip shape images are processed separately, including: The aligned audio features are extracted through an encoder, wherein the encoder is composed of multiple two-dimensional convolutional layers connected by residual connections; the face area image corresponding to the RGB lip shape image is extracted through a downsampling convolutional neural network structure, wherein the downsampling convolutional neural network structure is composed of 4 layers of convolutional neural networks connected in series.
5. A multi-task real-time cutout system for digital population synthesis, characterized in that: include: The data and image acquisition module is used to obtain the complete oral video of the specified text as the historical video, and obtain the audio data, RGB lip shape image and alpha channel image of the historical video; A feature fusion acquisition module, which is used to obtain the audio features of the audio data and the face area images corresponding to the RGB lip shape image and the alpha channel image, and align the audio features with the two face area images; Feature extraction is performed on the aligned audio features and the facial region image corresponding to the RGB lip shape image, and the two results are fused; A model building and training module, which is used to build a multi-task face image generation model, input the feature fusion result into the model, obtain a first RGB lip shape picture and a first alpha channel picture, update the parameters of the model according to the first RGB lip shape picture and the first alpha channel picture, and obtain a pre-trained model; The multi-task face image generation model is a GAN network based on Transformer and CNN. The multi-task face image generation model includes a generator module and a discriminator module. The generator module includes a layer of Transformer network, a layer of Resnet network composed of CNN and a multi-task network. The feature fusion result is input into the Transformer network to obtain a first result and input into the Resnet network. The Resnet network outputs a second result to the multi-task network. The multi-task network includes two CNN convolutional neural networks. The second result is respectively input into the two CNN convolutional neural networks. The multi-task network outputs a first RGB lip shape picture and a first alpha channel picture to the discriminator module, and at the same time inputs a face area picture corresponding to the aligned alpha channel picture to the discriminator module. The discriminator module calculates the loss based on the first RGB lip shape picture, the first alpha channel picture, the face area picture corresponding to the aligned RGB lip shape picture and the face area picture corresponding to the alpha channel picture using a loss function, updates the parameters of the model, and obtains a pre-trained multi-task face image generation model. The discriminator module is the discriminator network structure in PatchGan; The loss for: ; ; in, It is the alpha channel data of the face area image corresponding to the aligned alpha channel image; It is the alpha channel data after setting; The alpha channel data of the first alpha channel image; is a hyperparameter; The image generation and cutout module is used to process the audio of the face image to be generated using a pre-trained model to obtain the corresponding RGB lip shape image and alpha channel image, and cut out the RGB lip shape image based on the alpha channel image to complete the cutout.
6. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the multi-task real-time cutout method in digital population synthesis as described in any one of claims 1 to 4.
7. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores machine-executable instructions, which, when called and executed by a processor, are used to implement the multi-task real-time cutout method in digital population synthesis as described in any one of claims 1 to 4.
Citation Information
Patent Citations
Multimedia content display method and device, terminal and storage medium
CN111813970A
Video portrait matting method and system and medium
CN113253890A
Image matting method and device, electronic equipment and storage medium
CN114187317A
Digital human generation model, model training method and digital human generation method
CN114419702A
Virtual digital human lip shape synchronization method and system
CN116668611A
