Edge side equipment digital human generation system

Abstract

The invention provides an edge side device digital human generation system, and relates to the technical field of artificial intelligence, the system comprises a data acquisition module, the data acquisition module is configured to acquire a target audio feature and a target face image feature map of a source video; the digital human generation model completes training processing in the cloud server in advance and is synchronized to the edge side equipment through communication connection; the digital human generation model is configured to perform target generation processing according to the target audio features and the target face image feature map to obtain a target video; the target generation processing comprises a plurality of times of depth separable convolution, a plurality of times of deconvolution, convolution channel dimension splicing and twice down-sampling processing. According to the system, the limitation of the existing digital human generation technology on edge side equipment is solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a digital human generation system for edge devices. Background Art

[0002] 2D digital human generation technology, especially voice-driven digital human technology, has been a research hotspot in the fields of computer graphics and artificial intelligence in recent years. This technology can automatically synthesize lip animations that match the input audio by inputting audio signals, allowing digital humans to "speak" in real time and naturally in the video. Specifically, the core of this technology is to analyze the voice features in the audio, such as pitch, speech speed, volume, etc., through algorithms, and then adjust the digital human mouth shape in the template video according to these features to achieve synchronization of voice and lip shape.

[0003] At present, the mainstream method to achieve this technology is to use deep learning technology. Deep learning learns the complex correspondence between speech features and lip animation by training large-scale neural network models. The advantage of this method is that it can handle complex speech and lip changes and generate more natural and realistic animation effects. However, deep learning models usually have high requirements for computing power and hardware resources, including high-performance GPUs, sufficient memory and storage space, etc. This has limited the application of 2D digital human generation technology in a wider range of scenarios to a certain extent.

[0004] Especially on edge devices (such as smartphones, tablets, etc.), the application of 2D digital human generation technology faces many challenges. First, due to the large number of parameters and computational complexity of deep learning models, it is very difficult to deploy these models on edge devices. Even if they can be deployed, they may be inefficient due to device performance limitations, affecting user experience. Secondly, when edge devices generate 2D digital humans offline, manual intervention is often required for parameter adjustment or model optimization, which not only increases operating costs, but also reduces the practicality and convenience of the technology. In actual applications, users may prefer to use automated solutions that do not require manual intervention. And because the computing power of edge devices is limited, even if deep learning models can be run, the generated 2D digital humans may have certain defects due to insufficient computing power, such as lip animation that is not smooth enough and expression that is not natural enough. These problems will affect the realism of digital humans and the user's sense of immersion. Summary of the invention

[0005] The present application provides a digital human generation system for edge devices to address the limitations of existing digital human generation technologies on edge devices.

[0006] The system is arranged in an edge side device, and includes:

[0007] A data acquisition module, wherein the data acquisition module is configured to acquire target audio features and target facial image feature maps of a source video;

[0008] A digital human generation model, wherein the digital human model is pre-trained on a cloud server and synchronized to an edge device via a communication connection; the digital human generation model is configured to perform target generation processing according to the target audio features and the target facial image feature map to obtain a target video; the target generation processing includes a plurality of sub-depth separable convolutions, a plurality of deconvolutions, convolution channel dimension splicing, and 2x downsampling processing.

[0009] Preferably, the data acquisition module includes:

[0010] A data acquisition unit, wherein the data acquisition unit is configured to acquire the source video;

[0011] A preprocessing unit, wherein the preprocessing unit is configured to preprocess the source video to obtain a target source video;

[0012] A feature extraction unit is configured to perform feature extraction based on the target source video to obtain the target audio feature and the target face image feature map.

[0013] Preferably, the feature extraction unit is further configured to:

[0014] Extracting audio features according to the target source video to obtain audio features;

[0015] Wenet feature extraction is performed based on the audio feature to obtain the target audio feature.

[0016] Preferably, the feature extraction unit is further configured to:

[0017] Perform image extraction according to the target source video to obtain several frames of initial images;

[0018] Performing face cropping processing on all the initial images to obtain a plurality of frames of face images;

[0019] All the face images are scaled to obtain several frames of target face images, and the key point coordinates of the target face images are recorded; the target face image feature map includes at least two frames of the target face images and the key point coordinates corresponding to the target face images.

[0020] Preferably, the digital human generation model includes:

[0021] An input unit, the input unit being configured to perform input processing on the target face image feature map to obtain an input image feature map;

[0022] An encoding unit, wherein the encoding unit is configured to perform encoding processing according to the input image feature map and the target audio feature to obtain a plurality of encoding features; the encoding processing includes a plurality of depthwise separable convolutions and a 2x downsampling process;

[0023] A decoding unit, the decoding unit being configured to perform decoding processing according to all the encoding features to obtain target decoding features; the decoding processing includes several deconvolutions, several depth-separable convolutions and convolution channel dimension splicing;

[0024] An output unit is configured to generate a video according to the target decoding feature to obtain an output video; the output video includes an output face image and output key point coordinates corresponding to the output face image.

[0025] Preferably, the digital human generation model further comprises an attention unit, wherein the attention unit is configured to calculate attention weights according to the coding features, and input the coding features after the attention weights are assigned into the decoding unit for decoding processing;

[0026] Wherein, the attention unit is further configured as:

[0027] Mapping each token in the encoding feature to a query vector, a key vector, and a value vector to obtain a query vector space, a key vector space, and a value vector space;

[0028] The query vector is segmented to obtain a plurality of query vector subspaces; the key vector and the value vector are dimensionally reduced to obtain a key vector and a value vector after dimension reduction;

[0029] Attention weights are calculated for each of the query vector subspaces according to the reduced-dimensional key vector and value vector, and attention weight calculation results of multiple query vector subspaces are concatenated to obtain the attention weights corresponding to the encoding features.

[0030] Preferably, the target facial image feature map includes a first facial image and a second facial image, and the input unit is further configured as follows:

[0031] Masking a preset area of ​​the first facial image to obtain a masked image;

[0032] Concatenate the mask image and the second face image in the channel dimension to obtain the input image feature map;

[0033] The target audio features corresponding to the first facial image and the image feature map are sent to the encoding unit.

[0034] Preferably, the encoding unit comprises:

[0035] A first encoder, wherein the first encoder is configured to perform length padding and conventional convolution processing on the input image feature map to obtain a first encoded feature;

[0036] A second encoder, wherein the second encoder is configured to perform a 2-fold downsampling process and a plurality of depth-wise separable convolutions on the first encoding feature to obtain a second encoding feature;

[0037] A third encoder, wherein the third encoder is configured to perform a 2-fold downsampling process and a plurality of depth-wise separable convolutions on the second encoding feature to obtain a third encoding feature;

[0038] A fourth encoder, wherein the fourth encoder is configured to perform a 2-fold downsampling process and a plurality of depth-wise separable convolutions on the third encoding feature to obtain a fourth encoding feature;

[0039] A fifth encoder, wherein the fifth encoder is configured to perform a 2-fold downsampling process and a plurality of depth-wise separable convolutions on the fourth encoding feature to obtain a fifth encoding feature;

[0040] A sixth encoder, wherein the sixth encoder is configured to perform a 2-fold downsampling process and a plurality of depth-wise separable convolutions on the fifth encoded feature to obtain a sixth encoded feature;

[0041] An audio encoder is configured to perform several conventional convolutions and several depth-separable convolutions on the target audio features to obtain audio coding features.

[0042] Preferably, the depth-wise separable convolution includes a first depth-wise separable convolution and a second depth-wise separable convolution; the first depth-wise separable convolution is used to set a convolution kernel of 1×1 for each convolution channel of the feature map to obtain channel features corresponding to each convolution channel; the second depth-wise separable convolution is used to convolve the feature map or target audio features according to the channel features of different convolution channels.

[0043] Preferably, the decoding unit comprises:

[0044] A first decoder, wherein the first decoder is configured to sequentially perform deconvolution, convolution channel dimension splicing, and depth-separable convolution according to the audio coding feature and the sixth coding feature to obtain a first decoding feature;

[0045] A second decoder, wherein the second decoder is configured to sequentially perform deconvolution, convolution channel dimension splicing, and depthwise separable convolution according to the first decoding feature and the fifth encoding feature to obtain a second decoding feature;

[0046] A third decoder, wherein the third decoder is configured to sequentially perform deconvolution, convolution channel dimension splicing, and depth-separable convolution according to the second decoding feature and the fourth encoding feature to obtain a third decoding feature;

[0047] a fourth decoder, the fourth decoder being configured to sequentially perform deconvolution, convolution channel dimension splicing, and depthwise separable convolution according to the third decoding feature and the third encoding feature to obtain a fourth decoding feature;

[0048] a fifth decoder, the fifth decoder being configured to sequentially perform deconvolution, convolution channel dimension splicing, and depth-wise separable convolution according to the fourth decoding feature and the second encoding feature to obtain a fifth decoding feature;

[0049] a sixth decoder, the sixth decoder being configured to perform deconvolution in sequence according to the fourth decoding feature and the second encoding feature to obtain the target decoding feature;

[0050] The size of the target decoding feature is the same as the size of the first encoding feature.

[0051] Preferably, the digital human generation model and the discriminator are trained; the learning rate ratio of the digital human generation model and the discriminator is 1000 / 1; the discrimination ability of the discriminator is proportional to the training progress of the digital human generation model; and the digital human generation model uses distributed data for video generation.

[0052] As can be seen from the above content, the present application provides a digital human generation system for edge devices, the system includes a data acquisition module, the data acquisition module is configured to obtain the target audio features and target facial image feature map of the source video; a digital human generation model, the digital human model is pre-trained on a cloud server and synchronized to the edge device through a communication connection; the digital human generation model is configured to perform target generation processing according to the target audio features and the target facial image feature map to obtain a target video; the target generation processing includes several sub-depth separable convolutions, several deconvolutions, convolution channel dimension splicing, and 2x downsampling processing. The present application solves the limitations of existing digital human generation technology on edge devices through the above system. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0054] Figure 1A schematic diagram of a digital human generation system for edge devices in this application;

[0055] Figure 2 This is a specific schematic diagram of a data acquisition module in a digital human generation system of an edge device in this application;

[0056] Figure 3 This is a specific schematic diagram of a digital human generation model in a digital human generation system for an edge device of the present application;

[0057] Figure 4 This is a specific schematic diagram of an encoding unit and a decoding unit in a digital human generation system of an edge device of the present application;

[0058] Figure 5 This is a comparison diagram of the input processing effects in a digital human generation system of an edge device in this application. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0061] It should be noted that, in this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0062] 2D digital human, as an important concept in the field of modern digital technology and media, is gradually becoming known to the public. It is a virtual character image based on two-dimensional graphics technology, widely used in graphic design, game development, animation production, virtual reality and other fields.

[0063] Specifically, 2D digital humans are often referred to as "paper humans" and are the primary form of digital humans. The production process is relatively simple and quick. Usually, a real-life video or just a photo is collected, and the facial expressions and body movements are quickly identified and bound through machine pre-learning training. Then, a humanized video can be generated within a few minutes through text or voice drive. This method can greatly reduce the labor and time costs of video creation and solve the various troubles of real-life appearances.

[0064] From a technical perspective, the production of 2D digital humans mainly relies on the "cutout" of real-life videos and speech synthesis technology. By pre-recording real-life videos and using AI technology for lip-matching and background replacement, a seemingly virtual digital human image is generated. Although its movements and expressions are relatively limited, they can usually meet some basic interaction needs. For example, based on a given knowledge graph, 2D digital humans can perform simple question-and-answer, voice broadcast and other operations.

[0065] With the development of smart terminals, the demand for deploying models on edge devices such as mobile terminals has become an increasing demand for relevant application scenarios. However, this demand is in conflict with the mainstream method of generating 2D digital humans. This is mainly reflected in the following aspects:

[0066] 1. In order to achieve a relatively ideal effect, 2D digital human generation often relies on the design of a relatively complex network, which has a large number of parameters and a large amount of calculation. Therefore, the 2D digital human generation models in related technologies are mostly deployed on PCs or in the cloud, and use GPU computing power for model calculations. These limitations make it very difficult to deploy existing models or their improvements directly on edge devices such as mobile terminals.

[0067] 2. There are some models in the related technology that can run offline on the above-mentioned edge devices, but they have poor stability and are prone to defects in effect, such as background generation problems, color difference problems, unclear teeth / tooth flickering problems, etc. Therefore, they require manual intervention for targeted processing during the generation process, which in turn consumes manpower and time costs.

[0068] 3. The computing power of edge devices is limited, so the model deployed offline on the device is limited in both computing power and parameter quantity. This also leads to limited generation capabilities of the model. For some videos with complex backgrounds, especially dynamic background videos, the background generation effect around the face has certain flaws.

[0069] 4. Most 2D digital human generation models use generative adversarial networks, which have the characteristics of unstable training. The models deployed on edge devices are bound to be simplified compared to conventional models. The simplified models will further amplify the unstable characteristics of GAN and increase the difficulty of training.

[0070] The above problems make the deployment effect of the 2D digital human generation model in the related technology on the edge side device not ideal. For the above technical problems, no effective solutions have been proposed in the related technology.

[0071] Based on the above problems, the present application provides the following implementation methods.

[0072] Figure 1 This is a schematic diagram of a digital human generation system for an edge device in the present application.

[0073] See also Figure 1 It can be seen that this embodiment provides a digital human generation system for edge devices, and the system includes:

[0074] The data acquisition module 100 is configured to acquire the target audio features and target facial image feature map of the source video. Specifically, in this embodiment, the audio and image in the source video are acquired by the data acquisition module 100, and the features of the audio and image are extracted to obtain the target audio features and target facial image feature map. The source video is input into the data acquisition module 100, and the corresponding features are extracted by the data acquisition module 100.

[0075] The system further comprises:

[0076] A digital human generation model 200, wherein the digital human model 200 is pre-trained in a cloud server and synchronized to an edge device via a communication connection; the digital human generation model 200 is configured to perform target generation processing according to the target audio features and the target face image feature map to obtain a target video; the target generation processing includes a plurality of sub-depth separable convolutions, a plurality of deconvolutions, convolution channel dimension splicing, and 2x downsampling processing. Specifically, in this embodiment, the generation of a digital human is completed by the trained digital human generation model 200, wherein the training in the cloud server is completed. The digital human generation model 200 is sent to the edge device through the relevant parameters of the model from the cloud server. Therefore, it can be understood that the digital human generation model 200 has been trained in the cloud server, and the untrained digital human generation model 200 is placed on the edge device. Through the communication between the cloud device and the edge device, the relevant parameters of the trained digital human generation model 200 are transmitted to the edge device, so that the edge device can use the technical effect of the model without training, thereby avoiding the problem of poor digital human generation effect due to insufficient computing power of the edge device itself.

[0077] It should be noted that in order to achieve better training results and make the trained model generate images with higher clarity, the input video selected during the model training phase must contain only one person in the scene and only the person's voice without other noise. Videos that do not meet the requirements can be manually processed in advance.

[0078] Figure 2 This is a specific schematic diagram of a data acquisition module in a digital human generation system of an edge device in the present application.

[0079] See also Figure 2 It can be seen that, further, in some embodiments, the data acquisition module 100 includes:

[0080] The data acquisition unit 110 is configured to acquire the source video. Specifically, in this embodiment, the data acquisition unit 110 is used to directly acquire the source video. It can be understood that the data acquisition unit 110 is a port for the user to input the source video.

[0081] The data acquisition module 100 also includes:

[0082] The preprocessing unit 120 is configured to preprocess the source video to obtain a target source video. Specifically, in this embodiment, due to the video quality and audio quality of the source video, if feature extraction is performed using a video of poor quality, it will have a certain impact on the subsequent generation of digital humans. Therefore, before feature extraction, the source video needs to be preprocessed by the preprocessing unit 120 to obtain a video of higher quality and a single sound source.

[0083] It should be noted that it is necessary to distinguish it from the preprocessing of the video during the model training process. The function of the preprocessing unit 120 is to improve the quality of the digital human generated by the model after the model training is completed, while the preprocessing of the video during the model training process is to improve the quality of the model. It can be understood that the video needs to undergo two preprocessings during the model training process, while in the formal video generation process, it only needs to be preprocessed once through the preprocessing unit 120.

[0084] The data acquisition module 100 also includes:

[0085] The feature extraction unit 130 is configured to perform feature extraction based on the target source video to obtain the target audio features and the target facial image feature map. Specifically, in this embodiment, the feature extraction unit 130 performs feature extraction on the target source video that has completed preprocessing, so as to respectively extract the audio features and image features corresponding to the target source video, and obtain the target audio features and the target facial image feature map.

[0086] The step of extracting audio features by the feature extraction unit 130 includes:

[0087] Audio features are extracted according to the target source video to obtain audio features; wenet features are extracted according to the audio features to obtain the target audio features, wherein the type of the target audio features is wenet features.

[0088] Exemplarily, the image part and the audio part of the target source video are extracted respectively, and the corresponding wenet features are further extracted for the extracted audio, and the audio features are extracted with the help of the open source wenet speech recognition model, mainly using the wenet encoder coding network. The reason why the audio is characterized by wenet features in this embodiment is that different audio features have a great influence on the final lip shape effect. After actual testing, wenet features have better lip shape effects.

[0089] In terms of speech recognition, Wenet uses advanced acoustic models and language models, which can accurately identify speech content in multiple language environments, including dialects, speech in noisy environments, etc. In terms of speech synthesis, Wenet optimizes the waveform generation algorithm to make the synthesized speech closer to real people in terms of sound quality, intonation, and speaking speed, bringing users a more natural and comfortable listening experience.

[0090] Wenet technology can convert voice signals into text content in real time, and further analyze the emotions, intonation and other information in the voice. Based on this information, the system can automatically generate digital human expressions and movements that match the voice content, making the digital human more vivid and natural when expressing emotions and telling stories.

[0091] Through Wenet technology, digital humans can achieve real-time voice interaction with users. Users can control the behavior of digital humans through voice commands, and digital humans can immediately understand and give corresponding feedback. This interactive method not only enhances the immersiveness of user experience, but also brings new application scenarios to digital entertainment, online education and other fields.

[0092] Wenet technology also supports speech recognition and synthesis in multiple languages, which enables digital humans to overcome language barriers and communicate seamlessly with users around the world. In cross-cultural communication, digital humans can automatically switch to the corresponding language mode based on the user's language preference, providing users with more intimate and convenient services.

[0093] The step of extracting image features by the feature extraction unit 130 includes:

[0094] Image extraction is performed according to the target source video to obtain several frames of initial images; face cropping is performed on all the initial images to obtain several frames of face images; scaling is performed on all the face images to obtain several frames of target face images, and key point coordinates of the target face images are recorded; the target face image feature map includes at least two frames of the target face images and key point coordinates corresponding to the target face images.

[0095] Exemplarily, face cropping processing is performed on the image portion of the target source video, and the face of each frame in the video is cropped out using a face detection model and scaled to a specified size (160×160) to obtain a corresponding face image; the coordinates of the cropped face key points are recorded to obtain the corresponding face key points.

[0096] After the above preprocessing, the target face image feature map, target audio features (audio wenet features) and key point coordinates can be obtained, based on which the correspondence between each frame of the image and the audio features can be obtained.

[0097] Figure 3 This is a specific schematic diagram of a digital human generation model in a digital human generation system for an edge device in the present application.

[0098] See also Figure 3 It can be seen that, further, in some embodiments, the digital human generation model 200 includes:

[0099] The input unit 210 is configured to perform input processing on the target facial image feature map to obtain an input image feature map. Specifically, in this embodiment, the input unit 210 is used to perform necessary input processing on the target facial image feature map.

[0100] The target face image feature map includes a first face image and a second face image, and the input unit 210 performs input processing including:

[0101] Mask a preset area of ​​the first facial image to obtain a masked image; splice the masked image and the second facial image in the channel dimension to obtain the input image feature map; and send the target audio features corresponding to the first facial image and the image feature map to the encoding unit 220.

[0102] Figure 5 This is a comparison diagram of the input processing effects in a digital human generation system of an edge device in this application.

[0103] The effect diagram of covering the preset area of ​​the first face image is shown in FIG. Figure 5 See also Figure 5 , covering the first face image can be understood as painting the preset area black.

[0104] Exemplarily, the implementation of the encoding unit 220 is as follows:

[0105] The input face image, that is, the input image is composed of the following:

[0106] (1) Face image 1: A face image (3 channels) is randomly selected and its mouth area is masked, that is, the mouth is painted black. The final dimension of face image 1 is 160×160×3.

[0107] (2) Face image 2: In the same video as face1, another image different from face image 1 is selected as the reference image (3 channels). The final dimension of face image 2 is 160×160×3;

[0108] (3) concatenate face image 1 and face image 2 together in the channel dimension, that is, the final dimension of the face image is 192×192×6;

[0109] The input audio, that is, the input audio feature is obtained by the following method: the audio feature corresponding to the face image 1 is obtained, and the size is 1×20×256.

[0110] The digital human generation model 200 also includes:

[0111] The encoding unit 220 is configured to perform encoding processing according to the input image feature map and the target audio feature to obtain a plurality of encoding features; the encoding processing includes a plurality of depth-separable convolutions and a 2x downsampling process. Specifically, in the present embodiment, the encoding unit 220 is used to perform encoding processing according to the input image feature map and the target audio feature.

[0112] Figure 4 This is a specific schematic diagram of an encoding unit and a decoding unit in a digital human generation system of an edge device in the present application.

[0113] See also Figure 4 It can be seen that the encoding unit 220 includes:

[0114] A first encoder 221, wherein the first encoder 221 is configured to perform length padding and conventional convolution processing on the input image feature map to obtain a first encoding feature;

[0115] A second encoder 222, wherein the second encoder 222 is configured to perform a 2-fold downsampling process and a plurality of depth-wise separable convolutions on the first encoding feature to obtain a second encoding feature;

[0116] A third encoder 223, wherein the third encoder 223 is configured to perform a 2-fold downsampling process and a plurality of depth-wise separable convolutions on the second encoding feature to obtain a third encoding feature;

[0117] A fourth encoder 224, wherein the fourth encoder 224 is configured to perform a 2-fold downsampling process and a plurality of depth-wise separable convolutions on the third encoding feature to obtain a fourth encoding feature;

[0118] A fifth encoder 225, wherein the fifth encoder 225 is configured to perform a 2-fold downsampling process and a plurality of depth-wise separable convolutions on the fourth encoding feature to obtain a fifth encoding feature;

[0119] A sixth encoder 226, wherein the sixth encoder 226 is configured to perform a 2-fold downsampling process and a plurality of depth-wise separable convolutions on the fifth encoded feature to obtain a sixth encoded feature;

[0120] The audio encoder 227 is configured to perform several conventional convolutions and several depth-wise separable convolutions on the target audio features to obtain audio coding features.

[0121] Among them, the depth-wise separable convolution includes a first depth-wise separable convolution and a second depth-wise separable convolution; the first depth-wise separable convolution is used to set the convolution kernel to 1×1 for each convolution channel of the feature map separately to obtain the channel characteristics corresponding to each convolution channel; the second depth-wise separable convolution is used to convolve the feature map or target audio features according to the channel characteristics of different convolution channels.

[0122] Exemplarily, the specific structure and workflow of the encoding unit 220 are as follows:

[0123] The first encoder 221: uses the pad+conv+bn+relu combination, the pad parameter is 1, the convolution kernel is 3×3, the stride: 1, and the output size is 160×160.

[0124] Based on the output of the first encoder 221, the second encoder 222, the third encoder 223, the fourth encoder 224, the fifth encoder 225 and the sixth encoder 226 are further set, and the outputs of the second encoder 222, the third encoder 223, the fourth encoder 224, the fifth encoder 225 and the sixth encoder 226 are all downsampled by 2 times, that is, the output size of the second encoder 222 is 80×80, and so on, until the output size of the sixth encoder 226 is 5×5. The second encoder 222, the third encoder 223, the fourth encoder 224, the fifth encoder 225 and the sixth encoder 226 contain different numbers of depth-separable convolutions; the depth-separable convolution here consists of two parts:

[0125] First depth-wise separable convolution: For each convolution channel of the feature map (i.e., the output of the previous encoder), a 1x1 convolution kernel is set separately for convolution processing to obtain the channel features corresponding to each convolution channel;

[0126] Second depth-wise separable convolution: further convolution processing is performed on the above different channel features to obtain the corresponding output.

[0127] Compared with traditional convolution operations, the above-mentioned depthwise separable convolution can significantly reduce the amount of calculation and the number of model parameters. Generally speaking, the number of parameters for convolution calculation for the same image is 1 / 3 of that of the traditional convolution method, which allows the model in this embodiment to be deployed in edge devices such as mobile phones.

[0128] On this basis, the specific structures and outputs of the second encoder 222, the third encoder 223, the fourth encoder 224, the fifth encoder 225 and the sixth encoder 226 are as follows:

[0129] The second encoder 222 includes a depthwise separable convolution with stride=2 and a depthwise separable convolution module with stride=1, and the output size is 80×80;

[0130] The third encoder 223: includes a depthwise separable convolution module with stride=2 and three depthwise separable convolution modules with stride=1, and the output size is 40×40;

[0131] The fourth encoder 224 includes one depthwise separable convolution module with stride=2 and six depthwise separable convolution modules with stride=1, and the output size is 20×20;

[0132] The fifth encoder 225 includes one depthwise separable convolution module with stride=2 and four depthwise separable convolution modules with stride=1, and the output size is 10×10;

[0133] The sixth encoder 226 includes one depthwise separable convolution module with stride=2 and three depthwise separable convolution modules with stride=1, and the output size is 5×5.

[0134] Audio encoder 227: After 2 normal 2D convolutions + batchnorm + relu and a depthwise separable convolution module with stride = 1, and then 2 normal 2D convolutions + batchnorm + relu and a depthwise separable convolution module with stride = 1, the final output is 5×5.

[0135] The digital human generation model 200 also includes:

[0136] The decoding unit 230 is configured to perform decoding processing according to all the coding features to obtain target decoding features; the decoding processing includes several deconvolutions, several depth-separable convolutions and convolution channel dimension splicing. Specifically, in this embodiment, the decoding unit 230 is used to perform decoding processing on the coding features to obtain the target decoding features, thereby realizing the release of features.

[0137] Among them, see Figure 4 It can be seen that the decoding unit 230 includes:

[0138] A first decoder 231, wherein the first decoder 231 is configured to sequentially perform deconvolution, convolution channel dimension splicing, and depth-separable convolution according to the audio coding feature and the sixth coding feature to obtain a first decoding feature;

[0139] A second decoder 232, wherein the second decoder 232 is configured to sequentially perform deconvolution, convolution channel dimension splicing, and depth-separable convolution according to the first decoding feature and the fifth encoding feature to obtain a second decoding feature;

[0140] A third decoder 233, wherein the third decoder 233 is configured to sequentially perform deconvolution, convolution channel dimension splicing, and depth-separable convolution according to the second decoding feature and the fourth encoding feature to obtain a third decoding feature;

[0141] A fourth decoder 234, wherein the fourth decoder 234 is configured to sequentially perform deconvolution, convolution channel dimension splicing, and depth-separable convolution according to the third decoding feature and the third encoding feature to obtain a fourth decoding feature;

[0142] A fifth decoder 235, wherein the fifth decoder 235 is configured to sequentially perform deconvolution, convolution channel dimension splicing, and depthwise separable convolution according to the fourth decoding feature and the second encoding feature to obtain a fifth decoding feature;

[0143] a sixth decoder 236, wherein the sixth decoder 236 is configured to perform deconvolution in sequence according to the fourth decoding feature and the second encoding feature to obtain the target decoding feature;

[0144] The size of the target decoding feature is the same as the size of the first encoding feature.

[0145] Exemplarily, the specific structure and workflow of the decoding unit 230 are as follows:

[0146] The first decoder 231: deconvolves the audio encoder 227 and then concatenates it with the sixth encoder 226 in the channel dimension, passes through a depth-separable convolution module with stride=1, and the output size is 5×5.

[0147] The second decoder 232 is a decoder that performs a deconvolution on the first decoder 231 and then concatenates it with the fifth encoder 225 in the channel dimension, and then passes through a depth-separable convolution module with stride=1, and the output size is 10×10.

[0148] The third decoder 233: deconvolves the second decoder 232 and then concatenates it with the fourth encoder 224 in the channel dimension, passes through a depth-separable convolution module with stride=1, and the output size is 20×20.

[0149] The fourth decoder 234 is a decoder that performs a deconvolution on the third decoder 233 and then concatenates it with the third encoder 223 in the channel dimension, and then passes through a depth-separable convolution module with stride=1, and the output size is 40×40.

[0150] The fifth decoder 235 is a decoder that performs a deconvolution on the fourth decoder 234 and then concatenates it with the second encoder 222 in the channel dimension, and then passes through a depth-separable convolution module with stride=1, and the output size is 80×80.

[0151] The sixth decoder 236: After a deconvolution, the output size is 160×160.

[0152] Furthermore, in some embodiments, the digital human generation model 200 further includes an attention unit 250, and the attention unit 250 is configured to calculate attention weights according to the coding features, and input the coding features after the attention weights are assigned to the decoding unit 230 for decoding processing;

[0153] The attention unit 250 is further configured as follows:

[0154] Mapping each token in the encoding feature to a query vector, a key vector, and a value vector to obtain a query vector space, a key vector space, and a value vector space;

[0155] The query vector is segmented to obtain a plurality of query vector subspaces; the key vector and the value vector are dimensionally reduced to obtain a key vector and a value vector after dimension reduction;

[0156] Attention weights are calculated for each of the query vector subspaces according to the reduced-dimensional key vector and value vector, and attention weight calculation results of multiple query vector subspaces are concatenated to obtain the attention weights corresponding to the encoding features.

[0157] Specifically, in the present embodiment, the attention unit 250 is an improvement over the traditional multi-head attention mechanism. In the traditional multi-head attention mechanism, multiple attention heads are formed, and each attention head corresponds to an independent query vector, key vector, and value vector. This method requires a large number of memory read operations during implementation, thereby forming a large computational load. In this regard, the attention unit 250 only performs segmentation processing on the query vector, and performs dimensionality reduction processing on the key vector and the value vector. In this way, the subspaces of multiple query vectors share the same set of key vectors and value vectors, so that the key vector and the value vector only need to be read from memory once, thereby effectively reducing the computational load and better adapting to the requirements of this application deployed on edge devices. At the same time, the key vector and the value vector are reduced in dimension during the above calculation process, while the query vector is not reduced in dimension. This method can further reduce the computational load while ensuring that the query vector subspace maintains a high-dimensional calculation during the attention weight calculation process.

[0158] The attention unit 250 significantly reduces the load in the calculation process while having limited impact on the model reasoning accuracy, thereby making the deployment effect of this embodiment on the edge side device more ideal.

[0159] The system further comprises:

[0160] Output unit 240, the output unit 240 is configured to generate a video according to the target decoding features to obtain an output video; the output video includes an output face image and output key point coordinates corresponding to the output face image. Specifically, in this embodiment, the features are converted into a video by the output unit 240, so as to realize the generation of a digital human.

[0161] Exemplarily, the specific structure and workflow of the output unit 240 are as follows:

[0162] Face image output: Based on the sixth decoder 236, pad(1)+conv(3×3, pad=0, stride=1)+tanh activation is performed to output a face result with a size of 160×160×3.

[0163] Key point coordinate output: Based on the sixth decoder 236, after a full connection layer + sigmoid activation, the facial key point coordinates corresponding to the synthesized face are output, with a size of 68×2.

[0164] In addition to the beneficial effects brought about by the improvement of the model infrastructure, the improvements of this embodiment are also reflected in the following points:

[0165] 1. In the encoding part, the size of the feature map obtained by the first decoder 231 after convolution remains unchanged, which is still 160×160. The advantage of this is that the model can learn as much information as possible from the input, especially local texture information.

[0166] 2. In the decoding part, the sixth decoder 236 outputs the face image. The size of the feature map output by the sixth decoder 236 is already 160×160. At this time, for the generalization of the model, convolution processing is performed on the basis of the sixth decoder 236. The feature map remains unchanged. In addition to adding some parameters, more semantic information is given to the decoder to improve the generation quality.

[0167] 3. The last batch normalization layer (BN layer) of the depth-wise separable convolution module with stride=1 is changed to a group normalization layer (GroupNorm, GN layer), and the parameters of groupnorm are set to (numgroups=1, num_features=number of output channels). The purpose of this is that the mean and variance of the BN layer are statistical values ​​learned from a large number of training samples. During inference, these statistical values ​​may be different from the actual input, resulting in color difference problems for some images. In contrast, the mean and variance of the GN layer used in the present invention are calculated from the input data, which will make some corrections to the BN results and improve generalization.

[0168] 4. The sixth decoder 236 outputs the key point coordinates, and directly outputs the key point coordinates corresponding to the synthesized face through the model. During inference, the post-processing fusion module will fuse the synthesized area and the original face in a specific area to reduce edge defects. The previous approach used fixed area fusion, which in some cases would cover more background. For some videos with complex backgrounds, the effect would be flawed. After optimization, the union of the predicted key points and the original key points is used as the fusion area, which basically eliminates redundant background information and generates better results.

[0169] Furthermore, in some embodiments, the digital human generation model 200 and the discriminator are trained; the learning rate ratio of the digital human generation model 200 and the discriminator is 1000 / 1; the discrimination ability of the discriminator is proportional to the training progress of the digital human generation model 200; the digital human generation model 200 uses distributed data for video generation. Specifically, in this embodiment, the training of the digital human generation model 200 and the discriminator is completed in the cloud server.

[0170] Among them, the computational complexity and parameter volume of the digital human generation model 200 are 729M and 7.5M respectively, which is a lightweight network. During the training process, it is difficult to rely solely on the digital human generation model 200 to fit the distribution of audio to lip shape and generate a relatively good effect. Therefore, this embodiment introduces an adversarial generation mechanism, and uses the digital human generation model 200 as the generator of the GAN network to train it together with the discriminator. The training process of the generative adversarial network is a process of mutual game between the generator and the discriminator. During the training process, further improvements are made through the following points:

[0171] Using the pre-trained model, due to the change in input size, a model with an input of 160×160 was trained using 1 million data. When the generator was trained, this embodiment directly loaded the weights of the encoder part of the training model, so that the generator has a stronger learning ability from the beginning.

[0172] In actual training, the training is relatively stable at the beginning, but later the discriminator may be too strong, resulting in the generator being unable to obtain positive feedback. Therefore, this embodiment adopts a step-by-step update strategy for the parameters of the discriminator, that is, at the beginning, all the parameters of the discriminator are solidified and not allowed to be updated, and 3% (empirical value) is released every 10,000 iterations until they are completely released.

[0173] In order to make the generator learn faster and more stably, this embodiment tries to limit the data distribution within a certain range. That is, at the beginning, this embodiment only uses video data of a small number of people. This strategy is related to both people and videos (the same person is considered one person only under the same shooting conditions). When the generator fits the small range of data almost well, further data is added to continue training.

[0174] Practical application examples:

[0175] Deploy the model on the mobile phone and transplant it to the mobile phone (all other processes except audio feature extraction are handled on the PC, and the mobile phone is only responsible for audio feature extraction and network inference and pasting the synthesized face back to the original image). Considering only the inference speed, it reaches 10ms, far exceeding real time.

[0176] In summary, the advantages of this embodiment include:

[0177] 1. Use wnet audio features to train and drive lip shape, which is more natural and realistic than other audio features.

[0178] 2. Propose stable training methods and strategies for digital human applications based on the optimization of MobileNETv2 and UNET architecture at the network level.

[0179] 3. A normalization method combining batchnorm and groupnorm is proposed, which combines the characteristics of fast bn and strong gn generation ability, improving the generation quality and effect stability while ensuring the inference speed.

[0180] 4. A new fusion solution is proposed to solve background defects. The face key points are predicted through the network, and the union of the predicted key points and the input key points is taken as the dynamic fusion area.

[0181] 5. A lightweight network is proposed. The inference speed of this network on an existing flagship mobile phone is 10ms, which meets the offline deployment requirements of most edge devices.

[0182] For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above discussion in some embodiments is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. According to the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are intended to better explain the content of this disclosure, so that those skilled in the art can better use the embodiments.

Claims

1. A digital human generation system for edge devices, the system being arranged in an edge device, characterized in that: The system comprises: A data acquisition module (100), wherein the data acquisition module (100) is configured to acquire target audio features and target facial image feature maps of a source video; A digital human generation model (200), wherein the digital human model (200) is pre-trained on a cloud server and synchronized to an edge device via a communication connection; the digital human generation model (200) is configured to perform target generation processing according to the target audio features and the target face image feature map to obtain a target video; the target generation processing includes a plurality of sub-depth separable convolutions, a plurality of deconvolutions, convolution channel dimension splicing, and 2-fold downsampling processing.

2. The edge device digital human generation system according to claim 1, characterized in that: The data acquisition module (100) comprises: A data acquisition unit (110), the data acquisition unit (110) being configured to acquire the source video; A preprocessing unit (120), the preprocessing unit (120) being configured to preprocess the source video to obtain a target source video; A feature extraction unit (130), wherein the feature extraction unit (130) is configured to perform feature extraction based on the target source video to obtain the target audio feature and the target face image feature map.

3. The edge device digital human generation system according to claim 2, characterized in that: The feature extraction unit (130) is further configured to: Extracting audio features according to the target source video to obtain audio features; Wenet feature extraction is performed based on the audio feature to obtain the target audio feature.

4. The edge device digital human generation system according to claim 2, characterized in that: The feature extraction unit (130) is further configured to: Perform image extraction according to the target source video to obtain several frames of initial images; Performing face cropping processing on all the initial images to obtain a plurality of frames of face images; All the face images are scaled to obtain several frames of target face images, and the key point coordinates of the target face images are recorded; the target face image feature map includes at least two frames of the target face images and the key point coordinates corresponding to the target face images.

5. The edge device digital human generation system according to claim 4, characterized in that: The digital human generation model (200) comprises: An input unit (210), the input unit (210) being configured to perform input processing on the target face image feature map to obtain an input image feature map; An encoding unit (220), the encoding unit (220) being configured to perform encoding processing according to the input image feature map and the target audio feature to obtain a plurality of encoding features; the encoding processing comprises a plurality of depthwise separable convolutions and a 2-fold downsampling process; A decoding unit (230), the decoding unit (230) being configured to perform decoding processing according to all the encoding features to obtain target decoding features; the decoding processing comprises a plurality of deconvolutions, a plurality of depth-wise separable convolutions and convolution channel dimension splicing; An output unit (240) is configured to generate a video according to the target decoding feature to obtain an output video; the output video includes an output face image and output key point coordinates corresponding to the output face image.

6. The edge device digital human generation system according to claim 5, characterized in that: The digital human generation model (200) further comprises an attention unit (250), wherein the attention unit (250) is configured to calculate an attention weight according to the coding feature, and input the coding feature after the attention weight is assigned into the decoding unit (230) for decoding processing; Wherein, the attention unit (250) is further configured to: Mapping each token in the encoding feature to a query vector, a key vector, and a value vector to obtain a query vector space, a key vector space, and a value vector space; The query vector is segmented to obtain a plurality of query vector subspaces; the key vector and the value vector are dimensionally reduced to obtain a key vector and a value vector after dimension reduction; Attention weights are calculated for each of the query vector subspaces according to the reduced-dimensional key vector and value vector, and attention weight calculation results of multiple query vector subspaces are concatenated to obtain the attention weights corresponding to the encoding features.

7. The edge device digital human generation system according to claim 5, characterized in that: The target facial image feature map includes a first facial image and a second facial image, and the input unit (210) is further configured to: Masking a preset area of ​​the first facial image to obtain a masked image; Concatenate the mask image and the second face image in the channel dimension to obtain the input image feature map; The target audio features corresponding to the first facial image and the image feature map are sent to the encoding unit (220).

8. The edge device digital human generation system according to claim 5, characterized in that: The encoding unit (220) comprises: A first encoder (221), wherein the first encoder (221) is configured to perform length padding and conventional convolution processing on the input image feature map to obtain a first encoded feature; A second encoder (222), wherein the second encoder (222) is configured to perform a 2-fold downsampling process and a plurality of depth-wise separable convolutions on the first encoding feature to obtain a second encoding feature; A third encoder (223), wherein the third encoder (223) is configured to perform a 2-fold downsampling process and a plurality of depth-wise separable convolutions on the second encoding feature to obtain a third encoding feature; A fourth encoder (224), wherein the fourth encoder (224) is configured to perform a 2-fold downsampling process and a plurality of depth-wise separable convolutions on the third encoding feature to obtain a fourth encoding feature; A fifth encoder (225), wherein the fifth encoder (225) is configured to perform a 2-fold downsampling process and a plurality of depth-wise separable convolutions on the fourth encoded feature to obtain a fifth encoded feature; A sixth encoder (226), wherein the sixth encoder (226) is configured to perform a 2-fold downsampling process and a plurality of depth-wise separable convolutions on the fifth encoded feature to obtain a sixth encoded feature; An audio encoder (227), wherein the audio encoder (227) is configured to perform several conventional convolutions and several depth-wise separable convolutions on the target audio features to obtain audio coding features.

9. The edge device digital human generation system according to claim 8, characterized in that: The depth-wise separable convolution includes a first depth-wise separable convolution and a second depth-wise separable convolution; the first depth-wise separable convolution is used to set a convolution kernel of 1×1 for each convolution channel of the feature map to obtain channel features corresponding to each convolution channel; the second depth-wise separable convolution is used to convolve the feature map or target audio features according to the channel features of different convolution channels.

10. The edge device digital human generation system according to claim 8, characterized in that: The decoding unit (230) comprises: A first decoder (231), wherein the first decoder (231) is configured to sequentially perform deconvolution, convolution channel dimension splicing, and depthwise separable convolution according to the audio coding feature and the sixth coding feature to obtain a first decoding feature; A second decoder (232), wherein the second decoder (232) is configured to sequentially perform deconvolution, convolution channel dimension splicing, and depth-wise separable convolution according to the first decoding feature and the fifth encoding feature to obtain a second decoding feature; A third decoder (233), wherein the third decoder (233) is configured to sequentially perform deconvolution, convolution channel dimension splicing, and depth-wise separable convolution according to the second decoding feature and the fourth encoding feature to obtain a third decoding feature; A fourth decoder (234), the fourth decoder (234) being configured to sequentially perform deconvolution, convolution channel dimension splicing, and depthwise separable convolution according to the third decoding feature and the third encoding feature to obtain a fourth decoding feature; A fifth decoder (235), the fifth decoder (235) being configured to sequentially perform deconvolution, convolution channel dimension splicing, and depthwise separable convolution according to the fourth decoding feature and the second encoding feature to obtain a fifth decoding feature; a sixth decoder (236), the sixth decoder (236) being configured to perform deconvolution in sequence according to the fourth decoding feature and the second encoding feature to obtain the target decoding feature; The size of the target decoding feature is the same as the size of the first encoding feature.

11. The edge device digital human generation system according to claim 1, characterized in that: The digital human generation model (200) and the discriminator are trained; the learning rate ratio of the digital human generation model (200) and the discriminator is 1000 / 1; the discriminant ability of the discriminator is proportional to the training progress of the digital human generation model (200); and the digital human generation model (200) uses distributed data for video generation.

Images