Audio and video generation method and related device and electronic equipment for virtual image

Abstract

The application discloses an audio and video generation method and related device and electronic equipment about virtual image, comprising: processing a visual feature sequence and an audio feature sequence based on a first diffusion generation network to obtain a target hidden layer representation; processing the target hidden layer representation based on a second diffusion generation network to obtain an audio hidden layer representation, and processing the target hidden layer representation based on a third diffusion generation network to obtain a video hidden layer representation; decoding based on the audio hidden layer representation to obtain a target audio sequence, and decoding based on the video hidden layer representation to obtain a target video sequence; and fusing based on the target audio sequence and the target video sequence to obtain a target audio and video about the virtual image. The above scheme can improve the data quality of the virtual image audio and video, and the coordination and consistency between the audio and the picture.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a method for generating audio and video about a virtual image and related devices and electronic equipment. BACKGROUND

[0002] With the development of artificial intelligence technology, multi-modal interaction with digital people and virtual images as carriers has been widely used in intelligent education, intelligent customer service and many other scenarios.

[0003] In the prior art, for the content to be broadcast by a virtual image, a pre-recorded audio or an audio generated by means of speech synthesis technology is usually used, and then the audio is synthesized with a dynamic virtual image modeled to obtain an audio and video about the virtual image. However, the above-mentioned method usually has the problem of asynchrony between the virtual image and the content to be broadcast, which makes the overall effect of the synthesized audio and video appear stiff and unrealistic. Therefore, how to improve the data quality of the virtual image audio and video and the coordination and consistency between the audio and the picture has become a problem to be solved. SUMMARY

[0004] The technical problem solved by the present application is to provide a method for generating audio and video about a virtual image and related devices and electronic equipment, which can improve the data quality of the virtual image audio and video and the coordination and consistency between the audio and the picture.

[0005] To solve the above technical problem, the first aspect of the present application provides a method for generating audio and video about a virtual image, comprising: performing feature extraction based on reference image data of the virtual image to obtain a visual feature sequence, and performing feature extraction based on audio data to be broadcast by the virtual image to obtain an audio feature sequence; processing the visual feature sequence and the audio feature sequence based on a first diffusion generative network to obtain a target hidden layer representation; processing the target hidden layer representation based on a second diffusion generative network to obtain an audio hidden layer representation, and processing the target hidden layer representation based on a third diffusion generative network to obtain a video hidden layer representation; wherein the first diffusion generative network, the second diffusion generative network and the third diffusion generative network share time steps in the diffusion process; decoding based on the audio hidden layer representation to obtain a target audio sequence, and decoding based on the video hidden layer representation to obtain a target video sequence; fusing based on the target audio sequence and the target video sequence to obtain a target audio and video about the virtual image.

[0006] To solve the above technical problems, the second aspect of the present application provides an audio and video generation device about a virtual image, comprising: a feature extraction module, a first processing module, a second processing module, a feature decoding module and a sequence fusion module, the feature extraction module is used for feature extraction based on the reference image data of the virtual image to obtain a visual feature sequence, and feature extraction based on audio data to be broadcast of the virtual image to obtain an audio feature sequence; the first processing module is used for processing the visual feature sequence and the audio feature sequence based on a first diffusion generative network to obtain a target hidden layer representation; the second processing module is used for processing the target hidden layer representation based on a second diffusion generative network to obtain an audio hidden layer representation, and processing the target hidden layer representation based on a third diffusion generative network to obtain a video hidden layer representation; wherein the first diffusion generative network, the second diffusion generative network and the third diffusion generative network share time steps in the diffusion process; the feature decoding module is used for decoding based on the audio hidden layer representation to obtain a target audio sequence, and decoding based on the video hidden layer representation to obtain a target video sequence; the sequence fusion module is used for fusion based on the target audio sequence and the target video sequence to obtain a target audio and video about the virtual image.

[0007] To solve the above technical problems, the third aspect of the present application provides an electronic device, comprising a memory and a processor coupled to each other, the memory at least stores program instructions, and the processor is used to execute the program instructions to realize the audio and video generation method about the virtual image in the first aspect.

[0008] To solve the above technical problems, the fourth aspect of the present application provides a computer readable storage medium, which stores program instructions capable of being run by a processor, and the program instructions are used to realize the audio and video generation method about the virtual image in the first aspect.

[0009] The scheme extracts features based on the reference image data of the virtual image to obtain a visual feature sequence, extracts features based on audio data to be broadcast of the virtual image to obtain an audio feature sequence, processes the visual feature sequence and the audio feature sequence based on a first diffusion generation network to obtain a target hidden layer representation, processes the target hidden layer representation based on a second diffusion generation network to obtain an audio hidden layer representation, and processes the target hidden layer representation based on a third diffusion generation network to obtain a video hidden layer representation. The first diffusion generation network, the second diffusion generation network, and the third diffusion generation network share time steps in the diffusion process. The target audio sequence is obtained based on the audio hidden layer representation, and the target video sequence is obtained based on the video hidden layer representation. The target audio sequence and the target video sequence are fused to obtain a target audio-video about the virtual image. On the one hand, the diffusion network is used to generate the target audio sequence and the target video sequence, which can capture as much visual feature and audio feature about the virtual image as possible. The visual feature sequence and the audio feature sequence are processed by the first diffusion generation network to realize data processing in the overall dimension of audio-video and learn the correlation between cross-modal data, so as to realize the interaction and alignment between audio data and video data, improve the realism and naturalness of the generated audio-video, and further improve the detail quality of the audio and the video by respectively polishing the details of different modal data based on the second diffusion generation network and the third diffusion generation network. On the other hand, the first diffusion generation network, the second diffusion generation network, and the third diffusion generation network share time steps in the diffusion process, so that the three networks can maintain synchronization when processing data, which helps to improve the overall coordination and consistency of the generated audio-video. Therefore, the data quality of the virtual image audio-video and the coordination and consistency between the audio and the picture are improved. BRIEF DESCRIPTION OF DRAWINGS DETAILED DESCRIPTION BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 is a flowchart of an embodiment of the audio-video generation method of the virtual image according to the present application;

[0011] Figure 2 is a schematic diagram of an embodiment of the target synthesis system in the audio-video generation method of the virtual image according to the present application;

[0012] Figure 3 is a framework schematic diagram of an embodiment of the audio-video generation device according to the present application;

[0013] Figure 4 is a framework schematic diagram of an embodiment of the electronic device according to the present application;

[0014] Figure 5 is a framework schematic diagram of an embodiment of the computer readable storage medium according to the present application. DETAILED DESCRIPTION

[0015] The scheme of the embodiments of the present application will be described in detail below with reference to the accompanying drawings of the specification.

[0016] In the following description, for the purpose of explanation and not limitation, specific details are set forth, such as particular system configurations, interfaces, techniques, in order to provide a thorough understanding of the present application.

[0017] The terms "system" and "network" are often used interchangeably herein. The term "and / or" herein is merely an associative relationship between the associated objects, which means that there can be three relationships, for example, A and / or B, which means that there are three cases: A alone, A and B together, and B alone. In addition, the segment " / " herein generally represents an "or" relationship between the associated objects. In addition, "multiple" herein means two or more.

[0018] If the technical scheme of the present application involves personal information, the product applying the technical scheme of the present application has been explicitly informed of the personal information processing rules before processing the personal information, and has obtained the personal independent consent. If the technical scheme of the present application involves sensitive personal information, the product applying the technical scheme of the present application has obtained the personal independent consent before processing the sensitive personal information, and at the same time meets the requirement of "explicit consent". For example, at the personal information collection device such as camera, a clear and prominent sign is set to inform that the personal information collection range has been entered and the personal information will be collected. If the person voluntarily enters the collection range, it is considered to agree to collect the personal information. Or, in the case of using obvious signs / information to inform the personal information processing rules on the device for processing personal information, the personal authorization is obtained by pop-up information or by asking the person to upload his / her personal information, etc. The personal information processing rules can include personal information processor, personal information processing purpose, processing method, and personal information type, etc.

[0019] Please refer to Figure 1 , Figure 1 is a flowchart of an embodiment of the audio and video generation method of the virtual image according to the present application. Specifically, it can include the following steps:

[0020] Step S11: Based on the reference image data of the virtual image, feature extraction is performed to obtain a visual feature sequence, and based on the audio data to be broadcast of the virtual image, feature extraction is performed to obtain an audio feature sequence.

[0021] In the embodiments of the present disclosure, the virtual image can be a two-dimensional image, a three-dimensional model or other forms of digital image representation, and the virtual image includes but is not limited to digital people, cartoon characters, virtual anchors, etc. Specifically, the reference image data of the virtual image can be image data, model parameter data, etc. about the virtual image, and the reference image data can at least reflect the visual characteristics of the virtual image.

[0022] In one implementation scenario, for the reference avatar data, visual processing techniques such as convolutional neural network (CNN), image encoder, etc. can be used to extract visual feature sequences that can represent the visual characteristics of the virtual avatar, which can include facial features, body movement features, etc. of the virtual avatar.

[0023] In one specific implementation scenario, before feature extraction based on the reference avatar data of the virtual avatar to obtain the visual feature sequence, a reference video indicating the audio-video generation is obtained. Specifically, the reference video can be a pre-uploaded video related to the theme of the virtual avatar to be generated, for example, when generating an audio-video of a virtual anchor reporting news, the reference video can be a real news anchor reporting video, or a reference video of a virtual avatar reporting other news audio. Feature extraction based on the reference avatar data and the reference video obtains the visual feature sequence, which can extract relevant visual feature information from the reference video, such as facial expression change rules, body movement patterns, or scene information strongly associated with the virtual avatar, as auxiliary information to optimize the feature extraction process of the virtual avatar reference avatar data, so that the extracted visual feature sequence is more consistent with the actual scene performance.

[0024] In one specific implementation scenario, feature extraction based on the reference avatar data and the reference video can first extract static core features from the reference avatar data, including facial key points, topological structure, texture material, bone binding relationship, etc. to construct a basic feature library, then analyze dynamic features from the reference video, obtain posture changes, motion trajectories, timing dynamic rules, etc. through motion capture, optical flow analysis, expression sequence extraction, and finally through a feature alignment module, map and fuse the dynamic features with the static features of the reference avatar in space and time to obtain the visual feature sequence, ensuring that the dynamic features adapt to the structural properties of the reference avatar.

[0025] In one implementation scenario, the audio data to be broadcast by the virtual avatar is derived from the content to be broadcast, which can include text, voice, etc. in various forms. Specifically, when the content to be broadcast is text, voice synthesis technology can be used to convert it into audio data.

[0026] In one specific implementation scenario, as one possible implementation, a speech synthesis model can be pre-trained for synthesizing audio data to be broadcast based on the content to be broadcast. The speech synthesis model can be constructed based on a deep learning algorithm, such as using a recurrent neural network (RNN), a long short-term memory network (LSTM), or a Transformer structure, to learn the mapping relationship between text and speech through learning a large amount of speech data, so as to generate natural and fluent audio data according to the input text. It should be noted that the audio data generated based on the speech synthesis model can be used as a reference audio related only to the content to be broadcast.

[0027] In one specific implementation scenario, the extraction of the audio feature sequence can be implemented through audio processing techniques, such as using a mel-frequency cepstral coefficient (MFCC) extraction method, or using audio analysis techniques such as short-time Fourier transform (STFT) and linear predictive coding (LPC), which are not limited in the present application.

[0028] Step S12: processing the visual feature sequence and the audio feature sequence based on the first diffusion generation network to obtain a target hidden layer representation.

[0029] In one implementation scenario, the first diffusion generation network has a network architecture that integrates a diffusion mechanism, which can jointly process the visual feature sequence and the audio feature sequence to capture the cross-modal correlation relationship between them. Specifically, in the first diffusion generation network, the diffusion mechanism gradually adds Gaussian noise to the initial visual and audio feature sequence to approximate a random distribution, and then iteratively denoises and optimizes it in reverse. For example, in each step of denoising, the first diffusion generation network combines a cross-modal attention mechanism to dynamically capture the corresponding relationship between lip movements and speech rhythm, expression changes and intonation fluctuations, and to refine the feature correlation through multiple iterations to output a time-synchronized and semantically consistent target hidden layer representation.

[0030] In one specific implementation scenario, the first diffusion generation network is specifically a multimodal diffusion Transformer (MDiT) network or a diffusion convolutional neural network. Specifically, the diffusion Transformer network combines the self-attention mechanism of the Transformer and the step-by-step denoising property of the diffusion model, and can better capture long-distance dependencies and complex feature interactions when processing cross-modal data.

[0031] In a specific implementation scenario, taking the first diffusion generation network as an example, the multi-modal diffusion Transformer network is composed of a plurality of DiT Blocks connected in series, each DiT Block containing a self-attention layer, a cross-attention layer and a diffusion denoising layer. Specifically, the self-attention layer is used to mine the internal relationship between features in the same modality, for example, capturing the correlation between features of different regions of a face in a visual feature sequence, analyzing the time continuity of speech spectrum features in an audio feature sequence, the cross-attention layer focuses on the cross-modal association between different modal features, such as associating lip movements with speech pronunciation, expression changes with emotional intonation, etc., realizing the deep fusion of visual and audio features, and the diffusion denoising layer gradually removes the added noise according to the preset diffusion steps, gradually restores the clear, accurate and rich semantic target hidden layer representation from the approximately random distributed features, and through multiple rounds of iteration processing, the multi-modal diffusion Transformer network can output the target hidden layer representation that is time-synchronized and semantically consistent.

[0032] In one implementation scenario, before processing the visual feature sequence and the audio feature sequence based on the first diffusion generation network to obtain the target hidden layer representation, a reference audio indicating the generation of the audio-video can be obtained, and reference acoustic features of the reference audio can be extracted. Specifically, the reference audio can be an audio related to the theme of the virtual image audio-video to be generated and uploaded in advance, for example, when generating an audio-video of a virtual anchor reporting news, the reference audio can be a real news anchor reporting audio, or a reference audio of a virtual image reporting other news audio. Reference acoustic features are extracted based on the reference audio, which can include pitch, intensity, timbre, speech rate, etc. The visual feature sequence, the audio feature sequence and the reference acoustic features are processed based on the first diffusion generation network to obtain the target hidden layer representation.

[0033] In a specific implementation scenario, the reference acoustic features can be extracted using audio analysis techniques such as fundamental frequency extraction, energy calculation, Mel frequency cepstrum coefficient (MFCC) extraction, etc., which are not limited in the present application.

[0034] In one implementation scenario, before processing the visual feature sequence and the audio feature sequence based on the first diffusion generation network to obtain the target hidden layer representation, scene action descriptions are generated based on the benchmark avatar data and the to-be-broadcast content, the scene action descriptions representing scene descriptions and corresponding action descriptions matched by the virtual avatar when broadcasting the to-be-broadcast content. For example, in the generation of an audio-video scene in which a virtual anchor broadcasts news, the scene action descriptions can include scene descriptions of the background environment setting for news broadcasting, such as the arrangement of a news studio, light effects, and action descriptions of corresponding body movements, gesture use, and eye contact of the virtual anchor in the broadcasting process. Based on the benchmark avatar data, the basic appearance characteristics of the virtual anchor, such as appearance and body shape, can be determined, and in combination with the to-be-broadcast content, the scene style suitable for the news theme and the action performance of the virtual anchor can be analyzed. For example, if the to-be-broadcast content is serious political news, the scene description can be a solemn studio scene with a dark background, a professional news broadcasting station, and the action description can be a correct posture of the virtual anchor, a steady gesture, and a focused gaze. The scene action descriptions are converted into first embedding vectors matching the dimensions of the first diffusion generation network, the visual feature sequence, the audio feature sequence, and the first embedding vectors are fused to obtain first fusion features, and the first diffusion generation network processes the first fusion features to obtain the target hidden layer representation. The above scheme introduces scene action descriptions and converts them into first embedding vectors matching the first diffusion generation network, and fuses them with the visual feature sequence and the audio feature sequence, so that the first diffusion generation network can fully consider the scene and the corresponding action performance of the virtual avatar in the processing process, thereby further improving the generation quality of the generated audio-video.

[0035] In one specific implementation scenario, the fusion manner of the first fusion features can be implemented by input layer splicing, cross-attention mechanism, or time step embedding fusion, for example, input layer splicing can directly splice the visual feature sequence (dimension Dv x T), the audio feature sequence (dimension Da x T), and the first embedding vector (dimension De, which is expanded to De x T through position encoding) in the feature dimension to form a joint feature matrix with a dimension of (Dv + Da + De) x T, or through the cross-attention mechanism, the visual feature is used as the query vector, the audio feature and the first embedding vector are used as the key-value pair vector, the association information between the visual frame and the audio time sequence and the scene action is mined by calculating the attention weight, and the bidirectional attention mapping of the audio to the visual and the scene action to the audio-video is reversely constructed to strengthen the cross-modal collaboration, or the time step embedding fusion first converts the first embedding vector into a time sequence embedding vector matching the time steps of the feature sequence, and then at each time step, the embedding vector is added or multiplied with the corresponding visual feature and audio feature at the corresponding time step, which is not limited in the present application.

[0036] In a specific implementation scenario, based on the benchmark image data and the to-be-broadcast content, a large model instruction is constructed, and the large model instruction is used to instruct a large language model to generate a scene action description text matched with the benchmark image data and the to-be-broadcast content. Specifically, the large language model is a pre-trained model with natural language understanding and generation capabilities, which can understand complex semantic information and generate text content conforming to logic. The output text of the large language model in response to the large model instruction is taken as the scene action description.

[0037] In a specific implementation scenario, in the case where the to-be-broadcast content is incomplete, the large language model can also be used to generate specific broadcast text based on the benchmark image data and the to-be-broadcast content. For example, when the to-be-broadcast content only gives a general theme or part of the key information, the large language model can expand and perfect the content intelligently according to the role characteristics and style tendency set by the benchmark image data of the virtual image, and the theme category and key points of the to-be-broadcast content. For example, if the benchmark image data sets a lively and cute virtual anchor image, and the to-be-broadcast content only mentions “technology new product launch meeting”, the large language model can generate a broadcast text such as “Hey, everyone! Today is a super good day, a super cool technology new product launch meeting is about to start! It is said that this time’s new product will bring an unprecedented surprise experience, let us wait and see!” which conforms to the image style and is complete in content, or the to-be-broadcast content only mentions “reading poem A”, and the large language model can generate the content of poem A as the to-be-broadcast text.

[0038] Step S13: processing the target hidden layer representation based on the second diffusion generation network to obtain an audio hidden layer representation, and processing the target hidden layer representation based on the third diffusion generation network to obtain a video hidden layer representation.

[0039] In the embodiments of the present disclosure, the first diffusion generation network, the second diffusion generation network and the third diffusion generation network share time steps in the diffusion process. It can be understood that sharing time steps in the diffusion process means that the first diffusion generation network, the second diffusion generation network and the third diffusion generation network use the same time interval or time schedule to advance the processing and conversion of features in each step of the diffusion operation. This design of sharing time steps can ensure that the three networks maintain consistency in time sequence when processing cross-modal features, avoiding feature misplacement or time sequence synchronization problems caused by inconsistent time steps. Specifically, in each step of the diffusion process, the three networks simultaneously receive the input features of the current step and perform feature transformation and update according to their respective network architectures. For example, at a certain time step, the first diffusion generation network jointly processes the visual feature sequence and the audio feature sequence, captures the cross-modal correlation, and outputs the target hidden layer representation of the current step; at the same time, the second diffusion generation network and the third diffusion generation network respectively perform further refinement and optimization of the audio hidden layer representation and the video hidden layer representation based on the target hidden layer representation. By sharing time steps, the three networks can work cooperatively to ensure that the visual, audio and video features remain synchronized in time sequence in each step of the diffusion process, thereby ultimately generating virtual avatar audio-video content that is consistent in time sequence and coherent in semantics.

[0040] In one implementation scenario, the second diffusion generation network and the third diffusion generation network adopt the same diffusion network structure, which helps to simplify the complexity of the network architecture and facilitate unified optimization and training. Specifically, the second diffusion generation network and the third diffusion generation network can both adopt a network architecture similar to the fusion diffusion mechanism in the first diffusion generation network, such as the diffusion Transformer (DiT) structure, but the specific parameter settings and processing details will be adjusted accordingly for the different characteristics of audio and video. For example, when processing the audio hidden layer representation, the second diffusion generation network will focus more on the deep mining and optimization of audio features, such as further refining the variation rules of pitch, intensity, timbre and other features of speech, to generate more natural and smooth audio hidden layer representations with rich expressiveness; while the third diffusion generation network will focus on the fine processing of features such as facial expressions, body movements and scene transitions of the virtual avatar in the video, to ensure that the virtual avatar in the video is coherent in action and lively in expression, matching the generated audio.

[0041] In one specific implementation scenario, taking the second diffusion generation network and the third diffusion generation network as both being diffusion Transformer (DiT) structures for example, the second diffusion generation network is composed of multiple DiT Blocks in series, and the third diffusion generation network is composed of several DiT Blocks in series.

[0042] In a specific implementation scenario, taking the first diffusion generation network, the second diffusion generation network, and the third diffusion generation network as diffusion Transformer (DiT) structures for example, in the process of overall diffusion iteration of the first diffusion generation network, the second diffusion generation network, and the third diffusion generation network, the self-attention (Self-Attention) calculation module of the first diffusion generation network splices the QKV (Query, Key, Value) tensors of the second diffusion generation network branch and the QKV tensors of the third diffusion generation network branch in the sequence dimension, and then jointly performs self-attention calculation to realize cross-network feature interaction and information sharing. This design enables different network branches to learn from each other's feature information during diffusion iteration, thereby more comprehensively capturing the complex correlations between cross-modal features.

[0043] In one implementation scenario, the target hidden layer representation and the audio feature sequence are fused to obtain fourth fusion features. The specific steps of feature fusion can be referred to in the foregoing embodiments, and are not described herein again for brevity. The second diffusion generation network processes the fourth fusion features to obtain the audio hidden layer representation. The above scheme fuses the target hidden layer representation and the audio feature sequence to obtain fourth fusion features, and uses the second diffusion generation network to process the fourth fusion features to generate the audio hidden layer representation. The generation of the audio hidden layer representation is constrained by the audio feature sequence, so that the generated audio hidden layer representation can better retain key information in the original audio, such as text content, thereby improving the accuracy and naturalness of audio generation.

[0044] In a specific implementation scenario, after generating the scene action description based on the benchmark image data and the to-be-broadcast content, the scene action description is converted into a second embedding vector matching the dimension of the second diffusion generation network. The target hidden layer representation and the second embedding vector are fused to obtain second fusion features. The second diffusion generation network processes the second fusion features to obtain the audio hidden layer representation. The specific steps of feature fusion can be referred to in the foregoing embodiments, and are not described herein again for brevity. The above scheme constrains the generation of the audio hidden layer representation by the scene action description, so that the generated audio hidden layer representation can better fit the action performance of the virtual image in a specific scene, thereby enhancing the relevance and coordination between the audio and the scene and the action.

[0045] In a specific implementation scenario, after generating the scene action description based on the benchmark avatar data and the to-be-broadcast content, the scene action description is converted into a third embedding vector matching the third diffusion generation network dimension, the target hidden layer representation and the third embedding vector are fused to obtain a third fusion feature, and the third diffusion generation network processes the third fusion feature to obtain the video hidden layer representation. For brevity, the specific steps of feature fusion will not be repeated here. According to the above scheme, the generation of the video hidden layer representation is constrained by the scene action description, so that the generated video hidden layer representation can accurately present the action performance of the virtual avatar in a specific scene, ensuring that the action, expression, and the like of the virtual avatar in the video image are highly matched with the scene setting, and enhancing the realism and immersion of the video content.

[0046] Step S14: decoding based on the audio hidden layer representation to obtain a target audio sequence, and decoding based on the video hidden layer representation to obtain a target video sequence.

[0047] In one implementation scenario, the audio hidden layer representation subjected to decoding is the final audio hidden layer representation output by the second diffusion generation network after the overall diffusion iteration of the first diffusion generation network, the second diffusion generation network, and the third diffusion generation network, and the video hidden layer representation subjected to decoding is the final video hidden layer representation output by the third diffusion generation network after the overall diffusion iteration of the first diffusion generation network, the second diffusion generation network, and the third diffusion generation network. It can be understood that, since the first diffusion generation network, the second diffusion generation network, and the third diffusion generation network share the time step during diffusion, the final audio hidden layer representation and the final video hidden layer representation output by each network are consistent in timing after the overall diffusion iteration is completed. During decoding, for the audio hidden layer representation, it will be gradually restored to a continuous audio signal sequence, i.e., a target audio sequence, according to the encoding rules and language model of the audio, which can accurately present the speech content expressed by the virtual avatar, including the pitch, intensity, timbre, and the like of the speech. For the video hidden layer representation, it will be converted into a series of continuous video frames according to the encoding rules and image generation model of the video, which are combined into a target video sequence, and these video frames can clearly show the action, expression, and dynamic changes of the scene of the virtual avatar in a specific scene.

[0048] In a specific implementation scenario, when decoding based on the audio hidden layer representation, complete and smooth audio waveforms can be gradually constructed according to the pitch, intensity, timbre, and the like of the features information contained in the audio hidden layer representation, ensuring that the generated target audio sequence achieves a natural and realistic effect in terms of sound quality, tone, and the like. For example, for an audio hidden layer representation of a virtual anchor broadcasting news, the target audio sequence obtained after decoding can clearly present the calm and professional tone, accurate pronunciation, and appropriate speech rate of the news anchor.

[0049] In a specific implementation scenario, for decoding the target video sequence based on the video latent representation, the feature information can be converted into specific video frames frame by frame in chronological order, thereby generating a coherent and vivid target video sequence. Taking the video latent representation of the virtual anchor reporting news as an example, the decoded target video sequence can accurately present the dignified posture, natural gesture use and focused eye contact of the virtual anchor in the news studio, and can also clearly show the arrangement, lighting effect and other scene details of the news studio, making the entire video picture lifelike and giving the audience an immersive viewing experience.

[0050] Step S15: Fusion based on the target audio sequence and the target video sequence to obtain a target audio-video about the virtual image.

[0051] In an implementation scenario, when fusing the target audio sequence and the target video sequence, a synchronous alignment method can be used to ensure the matching of audio and video on the time axis. Specifically, each speech segment in the audio can be associated with the corresponding picture frame in the video according to the timestamp markers of the target audio sequence and the target video sequence.

[0052] In a specific implementation scenario, in order to further improve the fusion effect, some image processing and audio processing techniques can also be used. For example, in terms of image processing, color correction, brightness adjustment and other operations can be performed on the video picture to make the picture clearer and brighter, enhance the visual effect, etc.

[0053] Please refer to Figure 2 , Figure 2 is a schematic diagram of an embodiment of a target synthesis system in the method for generating an audio-video of a virtual image. As shown in Figure 2 , the target audio-video is generated by a target synthesis system, and the target synthesis system includes an audio-video generation model, which includes sequentially connected data input module, data processing module, data decoding module and data output module. The data input module includes a visual coding network and an audio coding network. The data processing module includes a first diffusion generation network, and a second diffusion generation network and a third diffusion generation network connected to the output end of the first diffusion generation network, respectively. The data decoding module includes an audio decoding network connected to the output end of the second diffusion generation network and a video decoding network connected to the output end of the third diffusion generation network.

[0054] In a specific implementation scenario, the target synthesis system further includes a large language model for generating scene action descriptions. For brevity, details are not repeated here.

[0055] The above scheme is based on the reference image data of the virtual image to extract features to obtain a visual feature sequence, and based on audio data to be broadcast by the virtual image to extract features to obtain an audio feature sequence, the visual feature sequence and the audio feature sequence are processed based on the first diffusion generation network to obtain a target hidden layer representation, the target hidden layer representation is processed based on the second diffusion generation network to obtain an audio hidden layer representation, and the target hidden layer representation is processed based on the third diffusion generation network to obtain a video hidden layer representation, and the first diffusion generation network, the second diffusion generation network and the third diffusion generation network share time steps in the diffusion process, the audio hidden layer representation is decoded to obtain a target audio sequence, and the video hidden layer representation is decoded to obtain a target video sequence, the target audio sequence and the target video sequence are fused to obtain a target audio and video about the virtual image. On the one hand, the diffusion network is used to generate the target audio sequence and the target video sequence, which can capture as much visual and audio features about the virtual image as possible, and the visual feature sequence and the audio feature sequence are processed by the first diffusion generation network to realize data processing in the overall dimension of audio and video to learn the correlation between cross-modal data, which can realize the interaction and alignment between audio data and video data, improve the realism and naturalness of the generated audio and video as a whole, and then generate the audio hidden layer representation and the video hidden layer representation based on the second diffusion generation network and the third diffusion generation network respectively to finely polish the details of different modal data, further improving the detail quality of audio and video; on the other hand, the first diffusion generation network, the second diffusion generation network and the third diffusion generation network share time steps in the diffusion process, so that the three networks can maintain synchronization when processing data, which helps to improve the overall coordination and consistency of the generated audio and video. Therefore, the data quality of the virtual image audio and video and the coordination and consistency between the audio and the picture can be improved.

[0056] Please refer to Figure 3 , Figure 3 is a schematic diagram of an embodiment of the virtual image audio and video generation device of the present application. As Figure 3As shown, the audio-video generation apparatus 30 about the virtual image comprises a feature extraction module 31, a first processing module 32, a second processing module 33, a feature decoding module 34 and a sequence fusion module 35. The feature extraction module 31 is configured to perform feature extraction based on reference image data of the virtual image to obtain a visual feature sequence, and perform feature extraction based on audio data to be broadcast of the virtual image to obtain an audio feature sequence. The first processing module 32 is configured to process the visual feature sequence and the audio feature sequence based on a first diffusion generative network to obtain a target hidden layer representation. The second processing module 33 is configured to process the target hidden layer representation based on a second diffusion generative network to obtain an audio hidden layer representation, and process the target hidden layer representation based on a third diffusion generative network to obtain a video hidden layer representation. The first diffusion generative network, the second diffusion generative network and the third diffusion generative network share time steps in the diffusion process. The feature decoding module 34 is configured to decode the audio hidden layer representation to obtain a target audio sequence, and decode the video hidden layer representation to obtain a target video sequence. The sequence fusion module 35 is configured to fuse the target audio sequence and the target video sequence to obtain a target audio-video about the virtual image.

[0057] Therefore, the audio-video generation apparatus 30 related to the virtual image performs feature extraction based on the reference image data of the virtual image to obtain a visual feature sequence, and performs feature extraction based on audio data to be broadcast of the virtual image to obtain an audio feature sequence, processes the visual feature sequence and the audio feature sequence based on the first diffusion generation network to obtain a target hidden layer representation, processes the target hidden layer representation based on the second diffusion generation network to obtain an audio hidden layer representation, and processes the target hidden layer representation based on the third diffusion generation network to obtain a video hidden layer representation, and the first diffusion generation network, the second diffusion generation network and the third diffusion generation network share time steps in the diffusion process, decodes based on the audio hidden layer representation to obtain a target audio sequence, and decodes based on the video hidden layer representation to obtain a target video sequence, and fuses based on the target audio sequence and the target video sequence to obtain a target audio-video related to the virtual image. On the one hand, the target audio sequence and the target video sequence are generated based on the diffusion network, which can capture as much visual feature and audio feature of the virtual image to be generated as possible, and the visual feature sequence and the audio feature sequence are processed by the first diffusion generation network, which realizes data processing of the overall dimension of audio-video to learn the correlation between cross-modal data, realizes the interaction and alignment between audio data and video data, improves the realism and naturalness of the generated audio-video as a whole, and then generates the audio hidden layer representation and the video hidden layer representation based on the second diffusion generation network and the third diffusion generation network respectively to finely polish the details of different modal data respectively, further improving the detail quality of the audio and the video. On the other hand, the first diffusion generation network, the second diffusion generation network and the third diffusion generation network share time steps in the diffusion process, so that the three networks can maintain synchronicity when processing data, which helps to improve the overall coordination and consistency of the generated audio-video. Therefore, the data quality of the virtual image audio-video and the coordination and consistency between the audio and the picture can be improved.

[0058] In some disclosed embodiments, before processing the visual feature sequence and the audio feature sequence based on the first diffusion generation network to obtain the target hidden layer representation, the audio-video generation apparatus 30 related to the virtual image further comprises a description generation module (not shown) for generating a scene-action description based on the reference image data and the content to be broadcast; wherein the scene-action description represents a scene description and a corresponding action description matched by the virtual image when broadcasting the content to be broadcast; the first processing module 32 further comprises a first vector module (not shown) for converting the scene-action description into a first embedding vector matching the dimension of the first diffusion generation network; the first processing module 32 further comprises a first fusion module (not shown) for fusing based on the visual feature sequence, the audio feature sequence and the first embedding vector to obtain a first fusion feature; the first processing module 32 further comprises a first processing submodule (not shown) for processing the first fusion feature based on the first diffusion generation network to obtain the target hidden layer representation.

[0059] In some disclosed embodiments, after generating the scene action description based on the reference avatar data and the to-be-broadcast content, the second processing module 33 further includes a second vector module (not shown) for converting the scene action description into a second embedding vector matching the dimension of the second diffusion generative network; the second processing module 33 further includes a second fusion module (not shown) for fusing the target hidden layer representation and the second embedding vector to obtain a second fusion feature; the second processing module 33 further includes a second processing submodule (not shown) for processing the second fusion feature based on the second diffusion generative network to obtain the audio hidden layer representation; and / or, the second processing module 33 further includes a third vector module (not shown) for converting the scene action description into a third embedding vector matching the dimension of the third diffusion generative network; the second processing module 33 further includes a third fusion module (not shown) for fusing the target hidden layer representation and the third embedding vector to obtain a third fusion feature; the second processing module 33 further includes a third processing submodule (not shown) for processing the third fusion feature based on the third diffusion generative network to obtain the video hidden layer representation.

[0060] In some disclosed embodiments, the description generation module (not shown) further includes an instruction construction module (not shown) for constructing a large model instruction based on the reference avatar data and the to-be-broadcast content; wherein the large model instruction is used to instruct the large language model to generate a scene action description text matching the reference avatar data and the to-be-broadcast content; the description generation module (not shown) further includes a model generation module (not shown) for obtaining the output text of the large language model responding to the large model instruction as the scene action description.

[0061] In some disclosed embodiments, before performing feature extraction based on the reference avatar data of the virtual avatar to obtain the visual feature sequence, the audio-video generation apparatus 30 related to the virtual avatar further includes a reference video acquisition module (not shown) for acquiring a reference video indicating audio-video generation; the feature extraction module 31 further includes a first extraction submodule (not shown) for performing feature extraction based on the reference avatar data and the reference video to obtain the visual feature sequence.

[0062] In some disclosed embodiments, before processing the visual feature sequence and the audio feature sequence based on the first diffusion generative network to obtain the target hidden layer representation, the audio-video generation apparatus 30 related to the virtual avatar further includes a reference audio acquisition module (not shown) for acquiring a reference audio indicating audio-video generation and extracting a reference acoustic feature of the reference audio; the first processing module 32 further includes a second extraction submodule (not shown) for processing the visual feature sequence, the audio feature sequence, and the reference acoustic feature based on the first diffusion generative network to obtain the target hidden layer representation.

[0063] In some disclosed embodiments, the second processing module 33 further includes a fourth fusion module (not shown) configured to fuse the target hidden layer representation and the audio feature sequence to obtain fourth fusion features; and the second processing module 33 further includes a fourth processing sub-module (not shown) configured to process the fourth fusion features based on a second diffusion generation network to obtain the audio hidden layer representation.

[0064] In some disclosed embodiments, the target audio-video in the audio-video generation apparatus 30 about the virtual image is generated by a target synthesis system, the target synthesis system including an audio-video generation model, the audio-video generation model including sequentially connected data input module, data processing module, data decoding module and data output module, the data input module including a visual coding network and an audio coding network, the data processing module including a first diffusion generation network and a second diffusion generation network and a third diffusion generation network connected to an output end of the first diffusion generation network, respectively, the data decoding module including an audio decoding network connected to an output end of the second diffusion generation network and a video decoding network connected to an output end of the third diffusion generation network.

[0065] Please refer to Figure 4 , Figure 4 is a schematic diagram of the framework of an embodiment of the electronic device. The electronic device 40 at least includes a memory 41 and a processor 42 coupled with each other, the memory 41 at least stores program instructions, and the processor 42 is configured to execute the program instructions to implement the steps in any of the above-mentioned embodiments of the method for generating audio-video about a virtual image. For details, please refer to the foregoing disclosed embodiments, which will not be repeated here. It should be noted that the specific type of the electronic device 40 is not limited here.

[0066] Specifically, the processor 42 is configured to control itself and the memory 41 to implement the steps in any of the above-mentioned embodiments of the method for generating audio-video about a virtual image. The processor 42 can also be referred to as a CPU (Central Processing Unit, central processing unit). The processor 42 can be an integrated circuit chip with signal processing capability. The processor 42 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware component. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. In addition, the processor 42 can be implemented by an integrated circuit chip together.

[0067] In the above scheme, the electronic device 40 performs feature extraction based on the reference image data of the virtual image to obtain a visual feature sequence, and performs feature extraction based on audio data to be broadcast of the virtual image to obtain an audio feature sequence, processes the visual feature sequence and the audio feature sequence based on a first diffusion generation network to obtain a target hidden layer representation, processes the target hidden layer representation based on a second diffusion generation network to obtain an audio hidden layer representation, and processes the target hidden layer representation based on a third diffusion generation network to obtain a video hidden layer representation, and the first diffusion generation network, the second diffusion generation network and the third diffusion generation network share time steps in the diffusion process, decodes based on the audio hidden layer representation to obtain a target audio sequence, and decodes based on the video hidden layer representation to obtain a target video sequence, and fuses based on the target audio sequence and the target video sequence to obtain a target audio video about the virtual image. On the one hand, the target audio sequence and the target video sequence are generated based on the diffusion network, which can capture as much visual feature and audio feature about the virtual image to be generated as possible, and the visual feature sequence and the audio feature sequence are processed by the first diffusion generation network, realizing data processing of the overall dimension of audio and video to learn the correlation between cross-modal data, which can realize the interaction and alignment between audio data and video data, improve the realism and naturalness of the generated audio and video as a whole, and then generate the audio hidden layer representation and the video hidden layer representation based on the second diffusion generation network and the third diffusion generation network respectively to finely polish the details of different modal data, further improving the detail quality of the audio and the video. On the other hand, the first diffusion generation network, the second diffusion generation network and the third diffusion generation network share time steps in the diffusion process, so that the three networks can maintain synchronicity when processing data, which helps to improve the overall coordination and consistency of the generated audio and video. Therefore, the data quality of the virtual image audio and video and the coordination and consistency between the audio and the picture can be improved.

[0068] Please refer to Figure 5 , Figure 5 is a frame schematic diagram of an embodiment of the computer readable storage medium of the present application. The computer readable storage medium 50 stores program instructions 51 capable of being run by a processor, and the program instructions 51 are used to implement the steps in any of the above embodiments of the method for generating audio and video about a virtual image.

[0069] In the above solution, the computer readable storage medium 50 performs feature extraction based on the reference image data of the virtual image to obtain a visual feature sequence, performs feature extraction based on audio data to be broadcast of the virtual image to obtain an audio feature sequence, processes the visual feature sequence and the audio feature sequence based on a first diffusion generative network to obtain a target hidden layer representation, processes the target hidden layer representation based on a second diffusion generative network to obtain an audio hidden layer representation, and processes the target hidden layer representation based on a third diffusion generative network to obtain a video hidden layer representation, wherein the first diffusion generative network, the second diffusion generative network, and the third diffusion generative network share a time step in the diffusion process, decodes based on the audio hidden layer representation to obtain a target audio sequence, and decodes based on the video hidden layer representation to obtain a target video sequence, and fuses based on the target audio sequence and the target video sequence to obtain a target audio-video about the virtual image. On the one hand, the target audio sequence and the target video sequence are generated based on the diffusion network, which can capture as much visual feature and audio feature about the virtual image to be generated as possible, and the visual feature sequence and the audio feature sequence are processed by the first diffusion generative network, which realizes data processing in the overall dimension of audio-video to learn the correlation between cross-modal data, can realize interaction and alignment between audio data and video data, and improves the realism and naturalness of the generated audio-video as a whole, and then the audio hidden layer representation and the video hidden layer representation are generated based on the second diffusion generative network and the third diffusion generative network respectively to finely polish the details of different modal data respectively, and further improve the detail quality of the audio and the video. On the other hand, the first diffusion generative network, the second diffusion generative network, and the third diffusion generative network share the time step in the diffusion process, so that the three networks can maintain synchronism when processing data, which helps to improve the overall coordination and consistency of the generated audio-video. Therefore, the data quality of the virtual image audio-video and the coordination and consistency between the audio and the picture can be improved.

[0070] In some embodiments, the apparatus provided by the embodiments of the present disclosure has functions or includes modules that can be used to perform the methods described in the above method embodiments, and the specific implementation can refer to the description of the above method embodiments. For brevity, details are not repeated here.

[0071] The above description of various embodiments tends to emphasize the differences between various embodiments, and the same or similar parts can be mutually referred to. For brevity, details are not repeated here.

[0072] In several embodiments provided in the present application, it should be understood that the disclosed methods and apparatuses can be implemented in other manners. For example, the division of the apparatus embodiments described above is merely an example, and the division of the modules or units can be different, for example, multiple modules or units can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0073] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they can be located in one place or distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the present embodiment.

[0074] In addition, the functional units in each embodiment of the present application can be integrated into one processing unit, or each unit can be physically present separately, or two or more units can be integrated into one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0075] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to execute all or part of the steps of the methods of the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program codes that can be stored in the medium.

Claims

1. A method for generating audio and video of a virtual avatar, characterized in that, include: Visual feature sequence is obtained by extracting features from the baseline image data of the virtual avatar, and audio feature sequence is obtained by extracting features from the audio data to be played by the virtual avatar. The visual feature sequence and the audio feature sequence are processed by the first diffusion generation network to obtain the target hidden layer representation; The target hidden layer representation is processed by the second diffusion generation network to obtain the audio hidden layer representation, and the target hidden layer representation is processed by the third diffusion generation network to obtain the video hidden layer representation; wherein the first diffusion generation network, the second diffusion generation network and the third diffusion generation network share the time step during the diffusion process; Decoding is performed based on the audio hidden layer representation to obtain the target audio sequence, and decoding is performed based on the video hidden layer representation to obtain the target video sequence; The target audio sequence and the target video sequence are fused together to obtain the target audio and video of the virtual character.

2. The method according to claim 1, characterized in that, Before processing the visual feature sequence and the audio feature sequence based on the first diffusion generation network to obtain the target hidden layer representation, the method further includes: Based on the baseline image data and the content to be broadcast, a scene action description is generated; wherein, the scene action description represents the scene description and corresponding action description matched by the virtual image when broadcasting the content to be broadcast; The step of processing the visual feature sequence and the audio feature sequence based on the first diffusion generation network to obtain the target hidden layer representation includes: The scene action description is transformed into a first embedding vector that matches the dimension of the first diffusion generation network. The first fused feature is obtained by fusing the visual feature sequence, the audio feature sequence, and the first embedding vector. The target hidden layer representation is obtained by processing the first fused features based on the first diffusion generation network.

3. The method according to claim 2, characterized in that, After generating a scene action description based on the baseline image data and the content to be broadcast, the process of processing the target hidden layer representation using the second diffusion generation network to obtain the audio hidden layer representation includes: The scene action description is transformed into a second embedding vector that matches the dimension of the second diffusion generation network. The second fused feature is obtained by fusing the target hidden layer representation and the second embedding vector. Based on the second diffusion generation network, the second fused feature is processed to obtain the audio hidden layer representation; And / or, the process of processing the target hidden layer representation based on the third diffusion generation network to obtain the video hidden layer representation includes: The scene action description is transformed into a third embedding vector that matches the dimension of the third diffusion generation network; The third fused feature is obtained by fusing the target hidden layer representation and the third embedding vector. The third fusion feature is processed based on the third diffusion generation network to obtain the hidden layer representation of the video.

4. The method according to claim 2, characterized in that, The process of generating scene action descriptions based on the baseline image data and the content to be broadcast includes: Based on the baseline image data and the content to be broadcast, a large model instruction is constructed; wherein, the large model instruction is used to instruct the large language model to generate scene action description text that matches the baseline image data and the content to be broadcast; The output text of the large language model in response to the large model instruction is obtained and used as the scene action description.

5. The method according to claim 1, characterized in that, Before extracting features from the baseline image data based on the virtual avatar to obtain a visual feature sequence, the method further includes: Obtain the reference video for the generated instruction audio and video; The baseline image data based on the virtual avatar is used to extract features to obtain a visual feature sequence, including: Based on the baseline image data and the reference video, feature extraction is performed to obtain a visual feature sequence.

6. The method according to claim 1, characterized in that, Before processing the visual feature sequence and the audio feature sequence based on the first diffusion generation network to obtain the target hidden layer representation, the method further includes: Obtain the reference audio generated by the instruction audio and video, and extract the reference acoustic features of the reference audio; The step of processing the visual feature sequence and the audio feature sequence based on the first diffusion generation network to obtain the target hidden layer representation includes: The target hidden layer representation is obtained by processing the visual feature sequence, the audio feature sequence, and the reference acoustic feature using a first diffusion generation network.

7. The method according to claim 1, characterized in that, The step of processing the target hidden layer representation based on the second diffusion generation network to obtain the audio hidden layer representation includes: The fourth fused feature is obtained by fusing the target hidden layer representation and the audio feature sequence. The fourth fusion feature is processed based on the second diffusion generation network to obtain the audio hidden layer representation.

8. The method according to any one of claims 1 to 7, characterized in that, The target audio and video are generated by a target synthesis system. The target synthesis system includes an audio and video generation model, which includes a data input module, a data processing module, a data decoding module, and a data output module connected in sequence. The data input module includes a visual encoding network and an audio encoding network. The data processing module includes a first diffusion generation network and a second diffusion generation network and a third diffusion generation network respectively connected to the output of the first diffusion generation network. The data decoding module includes an audio decoding network connected to the output of the second diffusion generation network and a video decoding network connected to the output of the third diffusion generation network.

9. An audio-visual generation device for virtual avatars, characterized in that, include: The feature extraction module is used to extract features based on the baseline image data of the virtual image to obtain a visual feature sequence, and to extract features based on the audio data to be played by the virtual image to obtain an audio feature sequence. The first processing module is used to process the visual feature sequence and the audio feature sequence based on the first diffusion generation network to obtain the target hidden layer representation; The second processing module is used to process the target hidden layer representation based on the second diffusion generation network to obtain the audio hidden layer representation, and to process the target hidden layer representation based on the third diffusion generation network to obtain the video hidden layer representation; wherein the first diffusion generation network, the second diffusion generation network and the third diffusion generation network share a time step during the diffusion process; The feature decoding module is used to decode based on the audio hidden layer representation to obtain the target audio sequence, and to decode based on the video hidden layer representation to obtain the target video sequence; The sequence fusion module is used to fuse the target audio sequence and the target video sequence to obtain target audio and video about the virtual image.

10. An electronic device, characterized in that, It includes at least a memory and a processor, wherein the memory stores at least program instructions, and the processor is used to execute the program instructions to implement the audio and video generation method for virtual avatars as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The device stores program instructions that can be executed by a processor, the program instructions being used to implement the audio and video generation method for virtual avatars as described in any one of claims 1 to 8.

Images