Digital human generation method and device, computer equipment and storage medium

Abstract

The invention belongs to the field of artificial intelligence, is applied to the field of financial virtual customer service, and relates to a digital human generation method which comprises the following steps: receiving an audio signal and initial digital human image data, and converting the audio signal and the initial digital human image data into audio features and initial image implicit features; inputting the audio features and the initial image implicit features into an autoregressive Transform model, generating image implicit features corresponding to each time step of the audio features through the autoregressive Transform model, and taking the image implicit features as target image implicit features; decoding the implicit features of the target image to obtain a target digital human image; a digital human video sequence is generated based on the audio features and the target digital human image. The invention further provides a digital human generation device, computer equipment and a storage medium. In addition, the invention also relates to a block chain technology, and the digital human video sequence can be stored in a block chain. According to the invention, the trueness and fluency of the generated digital human are effectively improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, computer equipment and storage medium for generating a digital human. Background Art

[0002] In today's digital age, digital human generation technology has become an important branch of artificial intelligence and image processing. This technology aims to create highly realistic and natural digital human images through computer algorithms and deep learning technology.

[0003] At present, digital human generation technology is mainly based on artificial 3D modeling and deep learning large models (LargeModels) technology, such as using generative adversarial networks (GANs), variational autoencoders (VAEs) and Transformer models to generate high-quality digital human images. Although the current methods have certain performance, they also have certain limitations: 1) Methods based on artificial 3D modeling and motion capture technology require high labor costs and long modeling time, and subsequent expression customization also requires additional expenses. 2) Digital human generation technologies based on deep learning methods such as TalkLip, video-retalking, etc. can automatically learn the complex mapping relationship between facial features, audio and lip shape, thereby realizing the generation of digital humans with audio and lip shape synchronization. However, there is still room for improvement in the consistency of facial appearance between the upper and lower frames of the generated video.

[0004] In summary, while pursuing efficiency and automation, these methods also face the challenge of maintaining the consistency and naturalness of facial appearance between video frames. In particular, when dealing with complex dynamic scenes and continuous conversations, problems such as distortion of details or lack of smoothness may occur. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a method, apparatus, computer device and storage medium for generating a digital human, so as to improve the authenticity and fluency of the generated digital human.

[0006] In order to solve the above technical problems, the present application embodiment provides a method for generating a digital human, which adopts the following technical solution:

[0007] A method for generating a digital human comprises the following steps:

[0008] Receiving an audio signal and initial digital human image data, and converting the audio signal and the initial digital human image data into audio features and initial image latent features respectively;

[0009] Inputting the audio features and the initial image latent features into a pre-trained autoregressive Transformer model, and generating image latent features corresponding to each time step of the audio features through the pre-trained autoregressive Transformer model as target image latent features;

[0010] Decoding the implicit features of the target image to obtain a target digital human image;

[0011] A digital human video sequence is generated based on the audio features and the target digital human image.

[0012] Furthermore, before the step of inputting the audio features and the initial image latent features into the pre-trained autoregressive Transformer model, the method further includes:

[0013] Receiving training data, wherein the training data includes a training audio signal and continuous digital human video frames, and converting the training audio signal and the digital human video frames into training audio features and training latent features respectively;

[0014] Generate the image latent features of the next time step based on the trained audio features, trained latent features and the preset initial autoregressive Transformer model corresponding to any time step;

[0015] Performing a dimensionality reduction operation on the image implicit features of the next time step to obtain reduced dimensionality features, and calculating image loss, perception loss, and synchronization loss based on the reduced dimensionality features;

[0016] The initial autoregressive Transformer model is adjusted based on the image loss, the perceptual loss, and the synchronization loss to obtain the pre-trained autoregressive Transformer model.

[0017] Furthermore, the step of generating the image implicit features of the next time step based on the training implicit features corresponding to any time step, the training audio features and the preset initial autoregressive Transformer model includes:

[0018] Identifying the facial position of the digital human video frame, performing a half-face occlusion operation on the facial position to obtain a reference image, and converting the reference image into a reference feature;

[0019] Performing a splicing operation on the training implicit feature corresponding to the current time step and the reference feature to obtain a spliced ​​feature;

[0020] The concatenated features and the training audio features are input into the initial autoregressive Transformer model to obtain the image implicit features of the next time step output by the initial autoregressive Transformer model.

[0021] Furthermore, the step of inputting the concatenated features and the training audio features into the initial autoregressive Transformer model to obtain the image implicit features of the next time step output by the initial autoregressive Transformer model includes:

[0022] Inputting the concatenated features and the training audio features into the initial autoregressive Transformer model;

[0023] The splicing features and the training audio features are fused through the cross attention mechanism of the initial autoregressive Transformer model to obtain the image implicit features of the next time step.

[0024] Furthermore, the step of calculating the image loss based on the dimensionality reduction feature includes:

[0025] Identify the facial position of the digital human video frame, perform a half-face occlusion operation on the facial position to obtain a reference image, and convert the reference image into a reference feature;

[0026] An image loss is calculated based on the dimension reduction feature and the reference feature.

[0027] Furthermore, the synchronization loss is calculated by the following steps:

[0028] Performing a decoding operation on the dimension reduction feature to obtain a dimension reduction image;

[0029] Inputting the dimension-reduced image and the training audio signal into a pre-trained SyncNet model to obtain an image vector and an audio vector output by the SyncNet model;

[0030] Calculating the similarity between the image vector and the audio vector to obtain a target similarity;

[0031] The synchronization loss is calculated based on the target similarity.

[0032] Furthermore, the perceptual loss is calculated by the following steps:

[0033] Performing a decoding operation on the dimension reduction feature to obtain a dimension reduction image;

[0034] Identify the facial position of the digital human video frame, perform a half-face occlusion operation on the facial position, and obtain a reference image;

[0035] A perceptual loss is calculated based on the reference image and the reduced-dimensionality image.

[0036] In order to solve the above technical problems, the embodiment of the present application further provides a digital human generation device, which adopts the following technical solution:

[0037] A receiving module, used for receiving an audio signal and initial digital human image data, and converting the audio signal and the initial digital human image data into audio features and initial image latent features respectively;

[0038] An input module, used for inputting the audio features and the initial image latent features into a pre-trained autoregressive Transformer model, and generating image latent features corresponding to each time step of the audio features through the pre-trained autoregressive Transformer model as target image latent features;

[0039] A decoding module, used for decoding the implicit features of the target image to obtain a target digital human image;

[0040] A generating module is used to generate a digital human video sequence based on the audio features and the target digital human image.

[0041] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0042] A computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned digital human generation method when executing the computer-readable instructions.

[0043] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0044] A computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the above-mentioned digital human generation method.

[0045] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0046] This application receives audio signals and initial digital human image data, and converts them into audio features and initial image implicit features respectively, thereby retaining the key information of the data and achieving accurate processing of the data. Using the pre-trained autoregressive Transformer model, the generated image achieves a high level of visual quality, with rich details and clear textures, making the appearance of the digital human more natural and realistic. Based on the audio features, the implicit features of the target image are gradually generated at each time step, and the digital human video sequence is generated based on the implicit features of the target image. This method not only improves the real-time performance of the generation process, but also can flexibly respond to different audio inputs, realize real-time audio-driven digital human generation, and ensure that the generated digital human can accurately express the audio content, so that the digital human is closely integrated with the audio content, effectively improving the authenticity and fluency of the generated digital human. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0048] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0049] Figure 2 A flowchart of an embodiment of a method for generating a digital human according to the present application;

[0050] Figure 3 is a structural schematic diagram of an embodiment of a digital human generation device according to the present application;

[0051] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0053] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0054] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0055] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0056] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0057] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to a laptop computer 1011, a tablet computer 1012 or a mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0058] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0059] It should be noted that the digital human generation method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the digital human generation device is generally set in the server / terminal device.

[0060] It should be understood that Figure 1The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0061] Continue to refer Figure 2 , shows a flow chart of an embodiment of a method for generating a digital human according to the present application. The method for generating a digital human comprises the following steps:

[0062] Step S201: receiving an audio signal and initial digital human image data, and converting the audio signal and the initial digital human image data into audio features and initial image latent features respectively.

[0063] In this embodiment, the electronic device (eg, Figure 1 The server / terminal device shown in the figure) can transmit the audio signal and the initial digital human image data through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (Ultra Wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0064] In this embodiment, a pre-trained Whisper model can be used as an audio encoder to convert the input audio signal into high-dimensional audio features. And a pre-trained VAE model can be used to convert the initial digital human image data into high-dimensional initial image latent features. Audio feature extraction can map the audio signal to different feature dimensions, thereby simplifying the subsequent processing process and improving the overall processing efficiency. The initial latent features can more accurately reflect the essential information in the image, which helps to improve the accuracy and efficiency of image processing.

[0065] Step S202: Input the audio features and the initial image latent features into a pre-trained autoregressive Transformer model, and generate the image latent features corresponding to each time step of the audio features through the pre-trained autoregressive Transformer model as target image latent features.

[0066] In this embodiment, the autoregressive Transformer model can gradually generate corresponding image implicit features based on the time step of the audio feature. This gradual generation method ensures the synchronization of the image and the audio in time, so that the generated image can more accurately reflect the changes in the audio content.

[0067] Step S203: decoding the implicit features of the target image to obtain a target digital human image.

[0068] In this embodiment, since the target image implicit features are generated based on the audio features, the digital human image obtained after decoding is consistent with the audio signal in terms of time and content. This consistency enables the generated image to more accurately reflect the information in the audio signal, enhancing the semantics and readability of the image. The decoding operation is usually performed on pre-trained deep learning models, which have been trained and optimized a lot and can generate high-quality images in a short time. Therefore, by decoding the target image implicit features, the target digital human image can be quickly obtained, which improves the efficiency of image generation.

[0069] Step S204: Generate a digital human video sequence based on the audio features and the target digital human image.

[0070] In this embodiment, according to the association between the audio features and the target digital human image at the time step, the audio features and the target digital human image are combined to generate a digital human video sequence with accurate lip synchronization, realistic facial behavior and natural head movement. Such a video sequence can more accurately reflect the audio content and enhance the realism and vividness of the video.

[0071] This application receives audio signals and initial digital human image data, and converts them into audio features and initial image implicit features respectively, thereby retaining the key information of the data and achieving accurate processing of the data. Using the pre-trained autoregressive Transformer model, the generated image achieves a high level of visual quality, with rich details and clear textures, making the appearance of the digital human more natural and realistic. Based on the audio features, the implicit features of the target image are gradually generated at each time step, and the digital human video sequence is generated based on the implicit features of the target image. This method not only improves the real-time performance of the generation process, but also can flexibly respond to different audio inputs, realize real-time audio-driven digital human generation, and ensure that the generated digital human can accurately express the audio content, so that the digital human is closely integrated with the audio content, effectively improving the authenticity and fluency of the generated digital human.

[0072] In some optional implementations of this embodiment, before step 202, that is, before the step of inputting the audio features and the initial image latent features into the pre-trained autoregressive Transformer model, the electronic device may further perform the following steps:

[0073] Receiving training data, wherein the training data includes a training audio signal and continuous digital human video frames, and converting the training audio signal and the digital human video frames into training audio features and training latent features respectively;

[0074] Generate the image latent features of the next time step based on the trained audio features, trained latent features and the preset initial autoregressive Transformer model corresponding to any time step;

[0075] Performing a dimensionality reduction operation on the image implicit features of the next time step to obtain reduced dimensionality features, and calculating image loss, perception loss, and synchronization loss based on the reduced dimensionality features;

[0076] The initial autoregressive Transformer model is adjusted based on the image loss, the perceptual loss, and the synchronization loss to obtain the pre-trained autoregressive Transformer model.

[0077] In this embodiment, the time step of the present application can be in frames, or other suitable metrics can be selected as time steps. Feature generation is performed through a preset initial autoregressive Transformer model. This model uses the training implicit features of the current image at each time step, and through the multi-layer Transformer decoder of the initial autoregressive Transformer model, each layer in the decoder is fused with the training audio features of the current time step through a cross-attention mechanism to generate the image implicit features of the next time step. In addition, a preset convolutional layer can be selected to reduce the dimension of the image implicit features of the next time step to half, and obtain the reduced dimension features for calculating the image loss (L1). Using continuous digital human video frames and training audio signals for training can enable the model to learn more abundant information, thereby improving its generalization ability. By calculating image loss, perceptual loss and synchronization loss, and using these losses as the target of model optimization, the training process of the model can be made clearer and more targeted. This helps to speed up the training speed of the model, improve training efficiency, and reduce the risk of overfitting. Through the comprehensive optimization of multiple loss functions, it is ensured that the generated digital human image is visually realistic and highly synchronized with the audio content.

[0078] In some optional implementations, the step of generating the image latent features of the next time step based on the trained latent features corresponding to any time step, the trained audio features and the preset initial autoregressive Transformer model includes:

[0079] Identifying the facial position of the digital human video frame, performing a half-face occlusion operation on the facial position to obtain a reference image, and converting the reference image into a reference feature;

[0080] Performing a splicing operation on the training implicit feature corresponding to the current time step and the reference feature to obtain a spliced ​​feature;

[0081] The concatenated features and the training audio features are input into the initial autoregressive Transformer model to obtain the image implicit features of the next time step output by the initial autoregressive Transformer model.

[0082] In this embodiment, in order to ensure the structural integrity of the input data, a time step is randomly selected, and the training latent features corresponding to the current time step are spliced ​​with the reference features in the channel dimension as the input of the initial autoregressive Transformer model. Among them, the reference features are latent features of the reference image (ground truth image) after half-face occlusion. By performing a half-face occlusion operation on the facial position of the digital human video frame, the partial occlusion or missing situation that may occur in practical applications can be simulated. This operation enables the model to learn how to generate high-quality images in the case of incomplete information during the training process. The occluded reference features are spliced ​​with the training latent features of the current time step and input into the model, so that the model can pay more attention to the changes and predictions of the occluded area when generating the image latent features of the next time step. This training method helps the model learn how to infer and complete the missing information under occlusion, thereby improving the model's prediction ability for the occluded area. By inputting the spliced ​​features together with the training audio features into the initial autoregressive Transformer model, the model can promote the fusion of audio and image features. This fusion helps the model better understand the relationship between audio signals and image content, thereby generating more accurate and natural digital human video sequences.

[0083] Specifically, the step of inputting the concatenated features and the training audio features into the initial autoregressive Transformer model to obtain the image implicit features of the next time step output by the initial autoregressive Transformer model includes:

[0084] Inputting the concatenated features and the training audio features into the initial autoregressive Transformer model;

[0085] The splicing features and the training audio features are fused through the cross attention mechanism of the initial autoregressive Transformer model to obtain the image implicit features of the next time step.

[0086] In this embodiment, the cross-attention mechanism can effectively fuse information from different modalities, such as images and audio. In this fusion process, the model can capture the associations and complementarities between different modalities, thereby generating richer and more accurate outputs. By introducing the cross-attention mechanism, the model can more accurately understand the structure and semantics of the input data, thereby showing a higher consistency between the image and audio when generating the implicit features of the image at the next time step. This application not only takes into account the correspondence between audio and image, but also makes full use of the advantages of the Transformer model in processing sequence data to ensure that the subsequently generated target digital human image can accurately reflect the audio content.

[0087] In some optional implementations, the step of calculating the image loss based on the dimensionality reduction feature includes:

[0088] Identify the facial position of the digital human video frame, perform a half-face occlusion operation on the facial position to obtain a reference image, and convert the reference image into a reference feature;

[0089] An image loss is calculated based on the dimension reduction feature and the reference feature.

[0090] Specifically, the image loss can be calculated according to the following formula:

[0091]

[0092] Wherein, L1 is the image loss, L g is the dimension reduction feature, L G is the reference feature.

[0093] In this embodiment, the image loss (L1) is used to ensure the pixel-level accuracy of the generated image. When calculating the image loss, the difference between the generated image and the real image needs to be considered. Since the reference image is obtained by half-face occlusion of the original image, the generated image needs to be reasonably inferred and completed in the occluded area. By calculating the image loss, the model can learn how to infer the missing facial information under occlusion, thereby generating a more natural and realistic image. In addition, by performing a half-face occlusion operation on the facial position of the digital human video frame and calculating the image loss, the model can learn richer facial features and change patterns. This learning helps the model to accurately identify or generate when facing new and unknown facial images. Therefore, this process improves the generalization ability of the model, enabling it to better adapt to various complex application scenarios.

[0094] In some optional implementations, the synchronization loss is calculated by the following steps:

[0095] Performing a decoding operation on the dimension reduction feature to obtain a dimension reduction image;

[0096] Inputting the dimension-reduced image and the training audio signal into a pre-trained SyncNet model to obtain an image vector and an audio vector output by the SyncNet model;

[0097] Calculating the similarity between the image vector and the audio vector to obtain a target similarity;

[0098] The synchronization loss is calculated based on the target similarity.

[0099] Specifically, the target similarity is calculated by the following formula:

[0100]

[0101] Among them, P sync is the target similarity, v is the image vector, s is the audio vector, and ∈ is a preset parameter.

[0102] In this embodiment, a VAE decoder can be used to decode the reduced-dimensional features into an image to obtain a reduced-dimensional image for calculating the synchronization loss (L sync ). By inputting the reduced-dimensional image and the training audio signal into the SyncNet model and calculating the similarity between the image vector and the audio vector, the degree of synchronization between the audio and the image can be quantified. This quantification method helps the model learn the correspondence between audio and image during training, thereby improving the synchronization between audio and image when generating digital human videos. The calculation of synchronization loss provides clear guidance for model optimization. The synchronization loss is used to ensure the temporal consistency of the generated image and audio features, further improving the performance of the model. During training, the model will continuously adjust its parameters to reduce synchronization loss, thereby improving the synchronization between audio and image. This clear guidance helps the model converge to the optimal solution faster and improves training efficiency.

[0103] Among them, the synchronization loss (L sync ) is calculated as:

[0104]

[0105] Among them, L sync is the synchronization loss, P sync is the target similarity.

[0106] In some optional implementations, the perceptual loss is calculated by the following steps:

[0107] Performing a decoding operation on the dimension reduction feature to obtain a dimension reduction image;

[0108] Identify the facial position of the digital human video frame, perform a half-face occlusion operation on the facial position, and obtain a reference image;

[0109] A perceptual loss is calculated based on the reference image and the reduced-dimensionality image.

[0110] Specifically, the perception loss is calculated according to the following formula:

[0111]

[0112] Where VGG is the pre-trained frozen VVG19 model, j is the number of layers of the frozen VVG19 model, and L r is the dimension reduction image, L R is the reference image.

[0113] In this embodiment, the perceptual loss uses high-level features extracted by a pre-trained large VGG network to ensure the perceptual quality of the generated image. The frozen VGG19 model can be used as a powerful feature extractor to accurately extract the features of the image. By decoding the reduced-dimensional features, a reduced-dimensional image can be obtained. This step is actually a reconstruction of the original image. The calculation of the perceptual loss usually determines the difference between the reduced-dimensional image and the reference image. By minimizing the perceptual loss, the model can be prompted to retain more key information during the dimensionality reduction and decoding process, thereby improving the quality of image reconstruction.

[0114] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned digital human video sequence, the above-mentioned digital human video sequence can also be stored in a node of a blockchain.

[0115] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0116] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0117] The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning. This application can be applied to the field of financial virtual customer service, thereby promoting the construction of smart cities.

[0118] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0119] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0120] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a digital human generation device, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0121] like Figure 3 As shown, the digital human generation device 300 described in this embodiment includes: a receiving module 301, an input module 302, a decoding module 303 and a generation module 304. Among them:

[0122] The receiving module 301 is used to receive an audio signal and initial digital human image data, and convert the audio signal and the initial digital human image data into audio features and initial image latent features respectively;

[0123] An input module 302 is used to input the audio feature and the initial image latent feature into a pre-trained autoregressive Transformer model, and generate an image latent feature corresponding to each time step of the audio feature through the pre-trained autoregressive Transformer model as a target image latent feature;

[0124] A decoding module 303 is used to decode the implicit features of the target image to obtain a target digital human image;

[0125] The generating module 304 is configured to generate a digital human video sequence based on the audio features and the target digital human image.

[0126] In this embodiment, the application retains the key information of the data and realizes accurate processing of the data by receiving the audio signal and the initial digital human image data, and converting them into audio features and initial image implicit features respectively. Using the pre-trained autoregressive Transformer model, the generated image reaches a high level of visual quality, with rich details and clear textures, making the appearance of the digital human more natural and realistic. Based on the audio features, the implicit features of the target image are gradually generated at each time step, and the digital human video sequence is generated according to the implicit features of the target image. This method not only improves the real-time performance of the generation process, but also can flexibly respond to different audio inputs, realize real-time audio-driven digital human generation, and ensure that the generated digital human can accurately express the audio content, so that the digital human is closely integrated with the audio content, effectively improving the authenticity and fluency of the generated digital human.

[0127] In some optional implementations of this embodiment, the digital human generation device further includes a training module; the training module includes a receiving submodule, a generating submodule, a calculating submodule and an adjusting submodule. Among them:

[0128] A receiving submodule, configured to receive training data, wherein the training data includes a training audio signal and continuous digital human video frames, and convert the training audio signal and the digital human video frames into training audio features and training implicit features respectively;

[0129] A generation submodule is used to generate the image latent features of the next time step based on the training audio features, training latent features and the preset initial autoregressive Transformer model corresponding to any time step;

[0130] A calculation submodule, used for performing a dimensionality reduction operation on the image implicit features of the next time step to obtain reduced dimensionality features, and calculating image loss, perception loss and synchronization loss based on the reduced dimensionality features;

[0131] An adjustment submodule is used to adjust the initial autoregressive Transformer model based on the image loss, the perceptual loss and the synchronization loss to obtain the pre-trained autoregressive Transformer model.

[0132] In some optional implementations of this embodiment, the generation submodule includes a recognition unit, a splicing unit, and an input unit.

[0133] The recognition unit is used to recognize the facial position of the digital human video frame, perform a half-face occlusion operation on the facial position to obtain a reference image, and convert the reference image into a reference feature; the splicing unit is used to perform a splicing operation on the training implicit feature corresponding to the current time step and the reference feature to obtain a spliced ​​feature; the input unit is used to input the spliced ​​feature and the training audio feature into the initial autoregressive Transformer model to obtain the image implicit feature of the next time step output by the initial autoregressive Transformer model.

[0134] Specifically, the input unit is also used to input the splicing features and the training audio features into the initial autoregressive Transformer model; the splicing features and the training audio features are fused through the cross-attention mechanism of the initial autoregressive Transformer model to obtain the implicit features of the image at the next time step.

[0135] In some optional implementations of this embodiment, the calculation submodule is also used to identify the facial position of the digital human video frame, perform a half-face occlusion operation on the facial position to obtain a reference image, and convert the reference image into a reference feature; and calculate the image loss based on the dimensionality reduction feature and the reference feature.

[0136] In some optional implementations of this embodiment, the calculation submodule includes a first decoding unit, a data input unit, a first calculation unit, and a second calculation unit.

[0137] The first decoding unit is used to perform a decoding operation on the dimensionality reduction feature to obtain a dimensionality reduction image; the data input unit is used to input the dimensionality reduction image and the training audio signal into a pre-trained SyncNet model to obtain an image vector and an audio vector output by the SyncNet model; the first calculation unit is used to calculate the similarity between the image vector and the audio vector to obtain a target similarity; and the second calculation unit is used to calculate the synchronization loss based on the target similarity.

[0138] In some optional implementations of this embodiment, the calculation submodule further includes a second decoding unit, an occlusion unit, and a third calculation unit, wherein:

[0139] The second decoding unit is used to perform a decoding operation on the dimensionality reduction feature to obtain a dimensionality reduction image; the occlusion unit is used to identify the facial position of the digital human video frame, perform a half-face occlusion operation on the facial position, and obtain a reference image; the third calculation unit is used to calculate the perceptual loss based on the reference image and the dimensionality reduction image.

[0140] This application receives audio signals and initial digital human image data, and converts them into audio features and initial image implicit features respectively, thereby retaining the key information of the data and achieving accurate processing of the data. Using the pre-trained autoregressive Transformer model, the generated image achieves a high level of visual quality, with rich details and clear textures, making the appearance of the digital human more natural and realistic. Based on the audio features, the implicit features of the target image are gradually generated at each time step, and the digital human video sequence is generated based on the implicit features of the target image. This method not only improves the real-time performance of the generation process, but also can flexibly respond to different audio inputs, realize real-time audio-driven digital human generation, and ensure that the generated digital human can accurately express the audio content, so that the digital human is closely integrated with the audio content, effectively improving the authenticity and fluency of the generated digital human.

[0141] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0142] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 4 with a memory 41, a processor 42, and a network interface 43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0143] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0144] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 4. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of the digital human generation method, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0145] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run the computer-readable instructions stored in the memory 41 or process data, such as running the computer-readable instructions of the digital human generation method.

[0146] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0147] This application receives audio signals and initial digital human image data, and converts them into audio features and initial image implicit features respectively, thereby retaining the key information of the data and achieving accurate processing of the data. Using the pre-trained autoregressive Transformer model, the generated image achieves a high level of visual quality, with rich details and clear textures, making the appearance of the digital human more natural and realistic. Based on the audio features, the implicit features of the target image are gradually generated at each time step, and the digital human video sequence is generated based on the implicit features of the target image. This method not only improves the real-time performance of the generation process, but also can flexibly respond to different audio inputs, realize real-time audio-driven digital human generation, and ensure that the generated digital human can accurately express the audio content, so that the digital human is closely integrated with the audio content, effectively improving the authenticity and fluency of the generated digital human.

[0148] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the digital human generation method as described above.

[0149] This application receives audio signals and initial digital human image data, and converts them into audio features and initial image implicit features respectively, thereby retaining the key information of the data and achieving accurate processing of the data. Using the pre-trained autoregressive Transformer model, the generated image achieves a high level of visual quality, with rich details and clear textures, making the appearance of the digital human more natural and realistic. Based on the audio features, the implicit features of the target image are gradually generated at each time step, and the digital human video sequence is generated based on the implicit features of the target image. This method not only improves the real-time performance of the generation process, but also can flexibly respond to different audio inputs, realize real-time audio-driven digital human generation, and ensure that the generated digital human can accurately express the audio content, so that the digital human is closely integrated with the audio content, effectively improving the authenticity and fluency of the generated digital human.

[0150] The non-Company software tools or components appearing in the embodiments of this application are merely examples and do not represent actual use.

[0151] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0152] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A method for generating a digital human, characterized in that: The steps include: Receiving an audio signal and initial digital human image data, and converting the audio signal and the initial digital human image data into audio features and initial image latent features respectively; Inputting the audio features and the initial image latent features into a pre-trained autoregressive Transformer model, and generating image latent features corresponding to each time step of the audio features through the pre-trained autoregressive Transformer model as target image latent features; Decoding the implicit features of the target image to obtain a target digital human image; A digital human video sequence is generated based on the audio features and the target digital human image.

2. The method for generating a digital human according to claim 1, characterized in that: Before the step of inputting the audio features and the initial image latent features into the pre-trained autoregressive Transformer model, the method further includes: Receiving training data, wherein the training data includes a training audio signal and continuous digital human video frames, and converting the training audio signal and the digital human video frames into training audio features and training latent features respectively; Generate the image latent features of the next time step based on the trained audio features, trained latent features and the preset initial autoregressive Transformer model corresponding to any time step; Performing a dimensionality reduction operation on the image implicit features of the next time step to obtain reduced dimensionality features, and calculating image loss, perception loss, and synchronization loss based on the reduced dimensionality features; The initial autoregressive Transformer model is adjusted based on the image loss, the perceptual loss, and the synchronization loss to obtain the pre-trained autoregressive Transformer model.

3. The method for generating a digital human according to claim 2, characterized in that: The step of generating the image implicit features of the next time step based on the training implicit features corresponding to any time step, the training audio features and the preset initial autoregressive Transformer model includes: Identify the facial position of the digital human video frame, perform a half-face occlusion operation on the facial position to obtain a reference image, and convert the reference image into a reference feature; Performing a splicing operation on the training implicit feature corresponding to the current time step and the reference feature to obtain a spliced ​​feature; The concatenated features and the training audio features are input into the initial autoregressive Transformer model to obtain the image implicit features of the next time step output by the initial autoregressive Transformer model.

4. The method for generating a digital human according to claim 3, characterized in that: The step of inputting the concatenated features and the training audio features into the initial autoregressive Transformer model to obtain the image implicit features of the next time step output by the initial autoregressive Transformer model comprises: Inputting the concatenated features and the training audio features into the initial autoregressive Transformer model; The splicing features and the training audio features are fused through the cross attention mechanism of the initial autoregressive Transformer model to obtain the image implicit features of the next time step.

5. The method for generating a digital human according to claim 2, characterized in that: The step of calculating the image loss based on the dimension reduction feature comprises: Identify the facial position of the digital human video frame, perform a half-face occlusion operation on the facial position to obtain a reference image, and convert the reference image into a reference feature; An image loss is calculated based on the dimension reduction feature and the reference feature.

6. The method for generating a digital human according to claim 2, characterized in that: The synchronization loss is calculated by the following steps: Performing a decoding operation on the dimension reduction feature to obtain a dimension reduction image; Inputting the dimension-reduced image and the training audio signal into a pre-trained SyncNet model to obtain an image vector and an audio vector output by the SyncNet model; Calculating the similarity between the image vector and the audio vector to obtain a target similarity; The synchronization loss is calculated based on the target similarity.

7. The method for generating a digital human according to claim 2, characterized in that: The perceptual loss is calculated by the following steps: Performing a decoding operation on the dimension reduction feature to obtain a dimension reduction image; Identify the facial position of the digital human video frame, perform a half-face occlusion operation on the facial position, and obtain a reference image; A perceptual loss is calculated based on the reference image and the reduced-dimensionality image.

8. A digital human generation device, characterized in that: include: A receiving module, used for receiving an audio signal and initial digital human image data, and converting the audio signal and the initial digital human image data into audio features and initial image latent features respectively; An input module, used for inputting the audio features and the initial image latent features into a pre-trained autoregressive Transformer model, and generating image latent features corresponding to each time step of the audio features through the pre-trained autoregressive Transformer model as target image latent features; A decoding module, used for decoding the implicit features of the target image to obtain a target digital human image; A generating module is used to generate a digital human video sequence based on the audio features and the target digital human image.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the digital human generation method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the digital human generation method according to any one of claims 1 to 7.

Images