Data generation method and apparatus, product, device, and medium

Abstract

Disclosed are a data generation method and apparatus, a product, a device, and a medium. The method comprises: obtaining a feature set, and an input noise image of a diffusion network, the feature set comprising an image feature of a target image and indication features of N pieces of indication data, wherein the target image comprises a first object, and the N pieces of indication data are all used for indicating a target content on which the first object is to simulate language expression; calling the diffusion network to perform temporal noise prediction processing on the input noise image under the constraint of the feature set, so as to generate M noise prediction features; and performing transformation processing on the M noise prediction features on the basis of the input noise image, so as to generate a simulated video of the first object, wherein the simulated video is a video in which the first object simulates language expression on the target content, and each noise prediction feature is used to transform and generate one corresponding video picture frame in the simulated video. By means of using the present application, the generation effect of a simulated video of a first object can be improved.

Description

Data generation methods, apparatus, products, equipment and media

[0001] This application claims priority to Chinese Patent Application No. 2024111760045, filed on August 26, 2024, entitled “Data Generation Method, Apparatus, Product, Equipment and Medium”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence, and more particularly to a data generation method, apparatus, product, device, and medium. Background Technology

[0003] As artificial intelligence technology matures, the range of technological scenarios that can be realized through it is also increasing. For example, artificial intelligence technology can be used to generate videos that simulate users speaking.

[0004] In existing applications, a discriminator network can be trained using samples of spoken facial images to generate videos simulating user speech. However, the discriminator network trained in this way generates each frame of the video separately, which may result in discontinuous frames and cannot meet the requirements for generating spoken facial videos in complex situations, leading to poor quality of the final spoken facial video. Summary of the Invention

[0005] This application provides a data generation method, apparatus, product, device, and medium that can improve the generation effect of analog video of a first object.

[0006] This application provides a data generation method, which includes:

[0007] Obtain the feature set and the input noise map of the diffusion network. The feature set includes the image features of the target image and the indicator features of N indicator data, where N is a positive integer. The target image contains a first object. The N indicator data belong to different data modalities. All N indicator data are used to indicate the target content of the first object to simulate language expression.

[0008] The diffusion network is invoked to perform temporal noise prediction processing on the input noise map under the constraint of the feature set, generating M noise prediction features. The M noise prediction features are temporally correlated, and M is a positive integer.

[0009] Based on the input noise map, M noise prediction features are used to transform the video frame and generate a simulated video of the first object. The simulated video is a video in which the first object simulates the verbal expression of the target content. One noise prediction feature is used to transform and generate a corresponding video frame in the simulated video.

[0010] This application provides a data generation apparatus, which includes:

[0011] The acquisition module is used to acquire the feature set and the input noise map of the diffusion network. The feature set includes the image features of the target image and the indicator features of N indicator data, where N is a positive integer. The target image contains a first object. The N indicator data belong to different data modalities. All N indicator data are used to indicate the target content of the first object to simulate language expression.

[0012] The prediction module is used to call the diffusion network to perform temporal noise prediction processing on the input noise map under the constraints of the feature set, and generate M noise prediction features. The M noise prediction features are temporally correlated, and M is a positive integer.

[0013] The generation module is used to transform video frames based on M noise prediction features from the input noise map to generate a simulated video of the first object. The simulated video is a video in which the first object simulates the verbal expression of the target content. One noise prediction feature is used to transform and generate a corresponding video frame in the simulated video.

[0014] This application provides a computer device, including a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor performs the method of this application.

[0015] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the method described in the above-mentioned aspect.

[0016] According to one aspect of this application, a computer program product is provided, comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the methods provided in the various alternative embodiments described above.

[0017] The method proposed in this application can generate a simulated video of a first object using a diffusion network. This diffusion network facilitates the introduction of multiple modal (e.g., N modal) indication data combined with the image features of the target image to constrain the generation of the simulated video of the first object. Furthermore, during the generation of the simulated video, the diffusion network can perform temporal noise prediction processing on the input noise map, making the generated M noise prediction features temporally correlated. Thus, the video frames in the simulated video generated from the M noise prediction features can also have visual continuity. Therefore, the method of this application can generate a simulated video of the first object with excellent results. Attached Figure Description

[0018] Figure 1 is a schematic diagram of a network architecture provided in an embodiment of this application;

[0019] Figure 2 is a schematic diagram of a scenario for generating a simulated video of a first object according to an embodiment of this application;

[0020] Figure 3 is a flowchart illustrating a data generation method provided in an embodiment of this application;

[0021] Figure 4 is a schematic diagram of another scenario for generating a simulated video of the first object provided in an embodiment of this application;

[0022] Figure 5 is a schematic flowchart of a noise prediction process for an input noise map provided in an embodiment of this application.

[0023] Figure 6 is a schematic diagram of a scenario for generating noise prediction features according to an embodiment of this application;

[0024] Figure 7 is a flowchart illustrating a model training method provided in an embodiment of this application;

[0025] Figure 8 is a schematic diagram of the effect of adding noise to a video image according to an embodiment of this application;

[0026] Figure 9 is a schematic diagram of a model training scenario provided in an embodiment of this application;

[0027] Figure 10 is a schematic diagram of a data generation device provided in an embodiment of this application;

[0028] Figure 11 is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0029] All data collected in this application (such as indication data, target images, sample images, feature sets, sample feature sets, and other related data) are collected with the consent and authorization of the data subject (such as users, institutions, or enterprises), and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant regions.

[0030] Here, the relevant technical concepts involved in this application are explained:

[0031] Latent variables are variables that are not directly observed in the model. They are the most fundamental features in the entire model processing flow and are very important for explaining the observed data and the behavior of the model.

[0032] Talking Head technology is a technique that generates videos of a user's speaking face from their portrait image, and it is a key component of the virtual human (digital human) species.

[0033] Cross attention refers to the cross attention mechanism in a model. This mechanism is typically used to handle the relationships between multiple input sequences, allowing the model to capture important information interactions between different sequences. Cross attention helps the model better understand the relationships between different parts, thus more accurately predicting relevant changes and effects during model processing.

[0034] Self-attention refers to the self-attention mechanism in the model. The role of self-attention is to further integrate all semantic representations in latent features, while making the generated object clearer and improving the network's ability to denoise latent features.

[0035] U-Net is an algorithm network that uses a fully convolutional network for semantic segmentation. It employs a symmetric U-shaped structure that includes compressed and expanded paths.

[0036] Please refer to Figure 1, which is a schematic diagram of a network architecture provided in an embodiment of this application. As shown in Figure 1, the network architecture may include a server 200 and a cluster of terminal devices. The cluster of terminal devices may include one or more terminal devices; the number of terminal devices is not limited here. As shown in Figure 1, the multiple terminal devices may specifically include terminal device 1, terminal device 2, terminal device 3, ..., terminal device n. As shown in Figure 1, terminal device 1, terminal device 2, terminal device 3, ..., terminal device n can all connect to the server 200 via the network, so that each terminal device can interact with the server 200 through the network connection.

[0037] As shown in Figure 1, server 200 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Terminal devices can be smart terminals such as smartphones, tablets, laptops, desktop computers, smart TVs, in-vehicle terminals, and smart home devices. The following description uses the communication between terminal device 1 and server 200 as an example to illustrate the specific implementation of this application.

[0038] The terminal device 1 may have a video client, and the server 200 may be the backend server of the video client. The terminal device 1 may send a video generation request to the server through the video client. The video generation request may be a request to generate a video of the first object simulating language expression. After receiving the video generation request, the server may generate a corresponding video for the first object, as described below.

[0039] Please refer to Figure 2, which is a schematic diagram of a scenario for generating a simulated video of a first object according to an embodiment of this application. The video generation request sent by the terminal device 1 to the server 200 may carry a target image and N indication data. The target image may contain a first object (as shown in Figure 2, the first object may be a girl). The N indication data may belong to different video modalities. The N indication data may be used to indicate that the first object needs to simulate the same expression (which may be called the target content) in language. Here, the N indication data may include audio data and text data.

[0040] Server 200 can generate image features of the target image and indicator features of each of the N indicator data, including indicator feature 1 for audio data and indicator feature 2 for text data. Thus, the image features of the target image, indicator feature 1 for audio data, and indicator feature 2 for text data constitute a feature set.

[0041] Server 200 can perform temporal noise prediction processing on the input noise map under the constraints of the feature set, generating M noise prediction features (including noise prediction feature 1 to noise prediction feature M here) that are temporally correlated. Furthermore, server 200 can transform each of the generated M noise prediction features to generate corresponding video frames (including the first video frame generated from noise prediction feature 1 to the Mth video frame generated from noise prediction feature M). These generated video frames constitute the simulated video of the first object simulating the verbal expression of the target content. For details of this process, please refer to the relevant descriptions in the following embodiments of this application.

[0042] Therefore, the server 200 can return the obtained simulated video to the video client in the terminal device 1, so that the video client can play the simulated video in the client interface for the user to view and use.

[0043] Using the method provided in this application, the generation of the simulated video of the first object by the diffusion network can be indicated by multiple modal indication data (such as N modal indication data when N is greater than 1). In addition, during the generation process, noise prediction processing can be performed on the input noise map in time sequence, so that the video frames in the final generated simulated video can also have coherence, thus ensuring the effect of the simulated video generated for the first object.

[0044] Please refer to Figure 3, which is a flowchart illustrating a data generation method provided in an embodiment of this application. The executing entity in this embodiment can be a data generation device (hereinafter referred to as the generation device). This generation device can be a computer device or a cluster of multiple computer devices. The computer device can be a server, a terminal device, or other devices; there are no limitations on this. As shown in Figure 3, the method may include:

[0045] Step S101: Obtain the feature set and the input noise map of the diffusion network. The feature set includes the image features of the target image and the indicator features of N indicator data, where N is a positive integer. The target image contains a first object. The N indicator data belong to different data modalities. All N indicator data are used to indicate the target content of the first object to simulate language expression.

[0046] Optionally, the generating device can acquire a feature set and an input noise map of a diffusion network. The diffusion network can be a trained network that can be used for video generation. The input noise map can be the input of the diffusion network. The input noise map can be a randomly generated initial noise map. The input noise map can be a feature map of noise (which can be latent). The input noise map can be a noise map for noise prediction by the diffusion network.

[0047] The feature set may include image features of the target image and indicator features of N indicator data, where N is a positive integer. The target image may contain a first object, which can be the object to be used for simulated video generation. In other words, the final video to be generated is a video of the first object simulating speech (which can refer to speaking). The first object can be any real user (including the real user's face) or any anime character (including the anime character's face). In other words, the first object may have a face, and the face may include a mouth for speech (i.e., speaking).

[0048] The N indicator data belong to different data modalities; that is, the N indicator data are indicator data for N different data modalities. One indicator data can correspond to one data modality, such as text modality, audio modality, or video modality. All N indicator data are used to instruct the first object to simulate the target content of language expression. In other words, the N indicator data are used to instruct the first object to simulate the same expression content, which is the target content. This target content is what the first object needs to simulate in language expression, that is, what the first object needs to simulate speaking.

[0049] The aforementioned diffusion network can be trained using indicator data of K modalities, where K is greater than or equal to N. These K modalities can include the N modalities corresponding to the aforementioned N indicator data, and can also include other modalities besides the aforementioned N modalities. In this application, the diffusion network supports the generation of simulated videos using indicator data of any one, any multiple, or all of the K modalities. In other words, the N modalities corresponding to the aforementioned N indicator data can be any one, any multiple, or all of the K modalities. The diffusion network can be constructed using a U-Net or a DiT model (an image generation model).

[0050] The aforementioned K modalities can include text modality, audio modality, and video modality. Therefore, the N indicator data can include at least one (or more) of the following: text data for the text modality, audio data for the audio modality, and video data for the video modality. In other words, the N indicator data can be any one, any two, or all three of the text data, audio data, and video data, depending on the specific application scenario. The video data can be a video of the second object verbally expressing the target content; that is, the video data can be a video of the second object speaking, and the content spoken by the second object in the video data can be the target content.

[0051] The second object can typically be a different object from the first object. The second object can be any object, such as any real user or any anime character.

[0052] Furthermore, both the text content described by the text data and the audio content expressed by the audio data are the aforementioned target content. It can be understood that the content spoken by the second object in the video data is the text content described by the text data, and the content spoken by the voice in the audio data is also the text content described by the text data; both are the target content. That is, the video data is used to represent the visual (e.g., lip movements) expression of the target content, the text data is used to represent the textual expression of the target content, and the audio data is used to represent the audio expression of the target content.

[0053] In this context, both the target image and the N indicator data can be collected, acquired, and sent to the generating device by the video client. For example, the target image could be a photo uploaded by the user of the video client, and the N indicator data could be uploaded by the user within the video client, or selected by the user within the video client. For instance, the video client could contain popular audio or video clips that the user can choose from, and these audio or video clips could be used to express popular content.

[0054] In other words, users can flexibly provide, within the video client, the image of the object to be generated (e.g., the first object) (e.g., the target image) and the instruction data for the target content to be simulated by that object. This instruction data can be any one of the following data modalities: text data (text modality), video data (video modality), and audio data (audio modality), or any two or all three data modalities. The specific data mode can be determined flexibly according to the actual application scenario.

[0055] Optionally, in practical application scenarios, in addition to the text modality, audio modality, and video modality mentioned above, other modalities can be introduced, which can be adaptively determined according to the actual application scenario.

[0056] The generating device can acquire an image encoder (also known as an image encoding network). This image encoder can be any pre-trained network that can be used for feature encoding of an image; that is, the image encoder can be a pre-trained neural network, and this application does not require training. The generating device can call the image encoder to perform feature encoding processing on the target image to generate image features of the target image. These image features can be features in the form of feature maps.

[0057] The generating device can also acquire the feature encoding network and feature mapping network corresponding to the aforementioned N indication data. One indication data point can correspond to one feature encoding network and one feature mapping network, and different modal indication data can correspond to different feature encoding networks and feature mapping networks. The feature encoding network corresponding to one modality of indication data can be a pre-trained network used for feature encoding of the indication data of that modality, and the feature mapping network corresponding to one modality of indication data can be a network used for feature mapping of the encoded features of the indication data of that modality. To enhance the representation of the indication features of each indication data point, the feature mapping networks corresponding to the indication data of various modalities can be trained together with a diffusion network, while the feature encoding networks corresponding to the indication data of various modalities can be pre-trained neural networks, which do not require training in this application.

[0058] Therefore, the generating device can invoke the feature encoding network corresponding to each of the N indicator data to perform feature encoding processing on each of the N indicator data, thereby generating the initial encoded feature of each indicator data. An indicator data can have one initial encoded feature, which is obtained by invoking the feature encoder corresponding to that indicator data to perform feature encoding processing on it.

[0059] Therefore, the generating device can call the feature mapping network corresponding to each indication data and perform feature mapping processing on the initial encoded features of each indication data to generate the indication features of each indication data.

[0060] In this process, the initial encoded features of each of the aforementioned N indicator data are mapped to the same feature dimension by their respective feature mapping networks; that is, the feature dimension of the indicator features for each indicator data can be the same. Furthermore, the initial encoded features of each indicator data are also enhanced through their respective feature mapping networks, ultimately resulting in enhanced indicator features for each indicator data.

[0061] Step S102: The diffusion network is invoked to perform temporal noise prediction processing on the input noise map under the constraint of the feature set, generating M noise prediction features. The M noise prediction features are temporally correlated, and M is a positive integer.

[0062] Optionally, the generating device can invoke a diffusion network to perform temporal noise prediction processing on the input noise map under the constraints of the above feature set, so as to generate M noise prediction features that are temporally correlated. These M noise prediction features can be understood as being generated after they interact with each other in a temporal manner (or feature interaction).

[0063] The specific process of performing time-series noise prediction processing on the input noise map to generate the M noise prediction features can be found in the description of the embodiment corresponding to Figure 5 below.

[0064] In this application, the total number of video frames to be generated can be preset, where M is the total number of frames that the final simulated video needs to contain. Optionally, in this application, the number of video frames to be generated can be adaptively determined by the length of the indicator data. For example, the number of video frames to be generated can be calculated by the data length (such as text length, audio length, or video length) of any of the N indicator data mentioned above, where text length can refer to the number of characters in the text data, audio length can refer to the duration of the audio data, and video length can refer to the duration of the video data.

[0065] It should be noted here that the input noise map mentioned above in this application can be a batch of noise maps, that is, there can be multiple input noise maps, such as 16 (or 32 frames, etc.). The number of input noise maps is the number of video frames that the diffusion network can generate in one batch. One input noise map can be used to generate a corresponding video frame in the process of generating a batch of video frames. The number of input noise maps can be referred to as the target number.

[0066] The generating device can utilize GPU (Graphics Processing Unit) resources (which can be considered computing resources) to generate simulated video. Therefore, optionally, this application can determine the number of input noise maps, i.e., the number of video frames that can be generated in a batch, based on the availability of GPU resources. If the computing resources are abundant (e.g., greater than or equal to a set resource threshold), the target number of input noise maps can be a first number (e.g., 32). If the computing resources are scarce (e.g., less than the resource threshold), the target number of input noise maps can be a second number (e.g., 16), where the second number can be less than the first number. Therefore, the specific number of input noise maps can be determined based on the actual application scenario, and this application does not impose any restrictions on this.

[0067] From the above, it can be understood that if the length of the instruction data is very short, then only one batch of video frames (i.e., video frames with the target number of frames) needs to be generated to obtain the final simulated video; that is, only one generation of video frames is required. In this case, if the length of the instruction data is particularly short, then in the generated 16-frame video frame, the first part of the video frame could be the video frame of the first object speaking, and the second part of the video frame could be the video frame of the first object when it is still (i.e., not speaking), such as the still video frame being the target image itself.

[0068] If the data length of the instruction is long, multiple batches of video frames can be generated, meaning multiple generation of video frames is required. One batch of video frames can be generated at a time (e.g., video frames with the target number of frames). A batch of video frames is generated from a batch of input noise maps. Video frames generated from different batches can overlap. For example, there can be two overlapping frames (or other frame numbers) between adjacent batches. If the target number is 16, the last two frames of the 16-frame video frame generated in the first batch can overlap (i.e., be the same) with the first two frames of the 16-frame video frame generated in the second batch. Similarly, the last two frames of the 16-frame video frame generated in the second batch can overlap (i.e., be the same) with the first two frames of the 16-frame video frame generated in the third batch, and so on.

[0069] By setting overlapping video frames between different batches of video frames, the various video frames in the final simulated video generated from multiple batches can be more coherent. For example, the position and lip movements of the first object in adjacent video frames are consistent, without obvious jumps.

[0070] For example, a reference length can be set, and different modal indication data can correspond to different reference lengths. The number of batches can be obtained by dividing the data length of any of the above N indication data by its corresponding reference length. This number of batches can be the number of batches required to generate video frames. The product of this number of batches and the target number of the above input noise map can be the final total number of video frames M to be generated. Therefore, it can be understood that the value of M can be an integer multiple of the number of video frames generated in a batch (i.e., the target number).

[0071] Step S103: Based on the input noise map, the M noise prediction features are processed to transform the video frame and generate a simulated video of the first object. The simulated video is a video in which the first object simulates the verbal expression of the target content. One noise prediction feature is used to transform and generate a corresponding video frame in the simulated video.

[0072] Optionally, the generating device can use the input noise map to perform video image transformation processing on the M noise prediction features generated above, in order to generate the final simulated video of the first object. One noise prediction feature can be used to transform and generate a corresponding video frame in the simulated video, as described below. Specifically, one noise prediction feature can be obtained by performing noise prediction processing on an input noise map using a feature set during the generation of a batch of video frames; that is, one noise prediction feature can correspond to one input noise map.

[0073] For example, in the process of generating the first batch of video frames, the first noise prediction feature can be obtained by performing noise prediction processing on the first input noise map, the second noise prediction feature can be obtained by performing noise prediction processing on the second input noise map, and so on; similarly, in the process of generating the second batch of video frames, the first noise prediction feature can be obtained by performing noise prediction processing on the first input noise map, the second noise prediction feature can be obtained by performing noise prediction processing on the second input noise map, and so on.

[0074] The generating device can use the M noise prediction features to generate M predicted noise maps from the input noise map. Each noise prediction feature is used to generate a corresponding predicted noise map, which is the noise map predicted from the corresponding input noise map using that noise prediction feature. Therefore, it can be understood that the M generated noise prediction features are the final features used to predict noise.

[0075] The generation device can perform video image transformation processing on the M predicted noise maps respectively by input noise map to generate a simulated video of the first object. The generation device can perform subtraction processing on the input noise map and each predicted noise map to generate M image prediction features. Subtracting a predicted noise map from the input noise map can generate a corresponding image prediction feature (which can be a feature map). The M image prediction features are the features finally used for video image prediction, as described below.

[0076] As described above, there can be a correspondence between the input noise map and the noise prediction feature. Therefore, the generation device can perform subtraction processing on an input noise map and a corresponding prediction noise map. That is, the input noise map can be used to input the prediction noise map to generate a picture prediction feature corresponding to the prediction noise map. By performing subtraction processing on the corresponding input noise map and prediction noise map respectively in this way, M picture prediction features corresponding to M noise prediction features can be generated. The subtraction processing between a noise prediction feature and its corresponding input noise map can obtain a corresponding picture prediction feature.

[0077] The dimensions of the input noise map and each predicted noise map can be the same. Therefore, subtracting the predicted noise map from the input noise map can mean subtracting the values ​​of the elements at the same position (such as the same row and column position) in the input noise map from the values ​​of the elements at the same position in the corresponding predicted noise map.

[0078] This can be understood as the process of performing noise prediction processing on the input noise map, which is the process of predicting noise in the input noise map that does not match the features used for video image prediction. By removing the predicted noise from the input noise map (as described above), the final features used for video image prediction (as described above) can be obtained. Therefore, the process of performing noise prediction processing on the input noise map can also be understood as a process of shaping and learning the features used for video image prediction.

[0079] The generating device can acquire a feature decoder (also called a feature decoding network), which can be a pre-trained neural network used to decode features to generate video images. For example, the feature decoder can be a VAE (Variational Self-Decoder).

[0080] The generating device can use the feature decoder to perform feature decoding processing on the M generated image prediction features, thereby generating M video frames. Each image prediction feature can be used to decode and generate a corresponding video frame, and these M video frames constitute the simulated video of the first object. The video frame conversion processing of the M noise prediction features can include the difference processing between the input noise map and the predicted noise map, as well as the feature decoding processing of the M image prediction features.

[0081] The simulated video is a video in which the generated first object simulates the verbal expression of the target content. That is, the simulated video can be a video of the generated first object speaking, and the content spoken by the first object in the simulated video is the target content.

[0082] It is understandable that the simulated video of the first object speaking to the target content can refer to a highly realistic speaking video generated by the first object simulating the lip movements and / or facial expressions of a human speaking to the target content. That is, the simulated video is a video of the first object simulating human speech generated by deep learning through a neural network (such as a diffusion network), rather than a video of the first object actually speaking. However, the lip movements and / or facial expressions of the first object speaking to the target content in the simulated video can be consistent with (e.g., the same as) the lip movements and / or facial expressions of a human speaking to the target content.

[0083] Please refer to Figure 4, which is a schematic diagram of another scenario for generating a simulated video of the first object according to an embodiment of this application. As shown in Figure 4, the user reference image here can be a target image provided by the user, which may contain the first object, which may be a male. The generating device can call a pre-trained identity encoder (i.e., an image encoder, also known as an image coding network) to perform feature encoding processing on the user reference image, thereby generating the image features of the user reference image.

[0084] The generating device can also acquire an initial noise sequence, which can be a sequence composed of individual input noise maps, and can include a batch of input noise maps.

[0085] Here, the N index data can include indicator data of N modalities, specifically including audio data, text data, and video data. The feature encoding network corresponding to the audio data can be an audio encoder (such as the wave2vec model, which is an unsupervised pre-trained model), the feature encoding network corresponding to the text data can be a text encoder (such as the BERT model, which is a word vector model), and the feature encoding network corresponding to the video data can be a video encoder (such as temporal CLIP, which is a convolutional neural network).

[0086] The generating device can perform feature encoding processing on the audio data through the audio encoder to generate the initial encoded features of the audio data. Then, through the feature mapping network corresponding to the audio data (the feature extractor 1 here), the initial encoded features can be processed to generate the indicator feature z1 corresponding to the audio data.

[0087] Similarly, the generating device can perform feature encoding processing on the text data through the aforementioned text encoder to generate the initial encoded features of the text data. Then, through the feature mapping network corresponding to the text data (the feature extractor 2 here), the initial encoded features can be processed to generate the indicator feature z2 corresponding to the text data.

[0088] Furthermore, the generating device can perform feature encoding processing on the video data using the aforementioned video encoder to generate initial encoded features of the video data. Then, through the feature mapping network corresponding to the video data (the feature extractor 3 here), feature mapping processing is performed on the initial encoded features to generate the indicator feature z3 corresponding to the video data.

[0089] Optionally, the feature mapping network corresponding to the indicator data of various data modalities can be a Transformer network (a neural network based on an attention mechanism). The Transformer network can include self-attention and cross attention. This feature mapping network can be trained together with the aforementioned diffusion network.

[0090] In this context, the initial encoded features generated by the feature encoding network for the indicator data of different data modalities usually have different dimensions. Therefore, by using the feature mapping network corresponding to the indicator data of various data modalities, the initial encoded features of the indicator data of various data modalities can be mapped to the same feature dimension (such as 32×768), thereby obtaining indicator features of the same feature dimension for the indicator data of various data modalities. This allows the subsequent concatenation of various indicator features of the same feature dimension, achieving the goal of sharing the same conditional control network layer for indicator data of multiple data modalities.

[0091] Therefore, the generating device can concatenate the indicator features z1, z2, and z3 to obtain the concatenated features. The generating device can then input the concatenated features, the image features of the user reference image, and the initial noise sequence into a diffusion network (i.e., the diffusion model here). This diffusion network performs noise prediction processing, and the resulting image prediction features are input into the feature decoder to decode and generate a simulated video of the first object. This simulated video is a video of the first object simulating the verbal expression of the target content.

[0092] Furthermore, the generating device can also return the generated simulated video to the video client, allowing the client to play the simulated video on its interface for users to view and use (e.g., download, forward, share, and save). Alternatively, the process of generating the simulated video can be performed by the video client itself, depending on the specific application scenario; this application does not impose any restrictions on this.

[0093] This application can be applied to any scenario that requires the generation of speaking face videos. Several specific application scenarios of this application are described below as examples:

[0094] The solution provided in this application can be applied to the generation of digital human advertising materials. For example, an image (such as a target image) containing an advertising image (e.g., a first object, which can be a real person or an animated character) can be provided, along with the target content that the advertising image needs to express verbally. This target content can be provided through one or more corresponding indicator data (e.g., the aforementioned N indicator data). For example, the target content could be the advertising content that the advertising image needs to express verbally. Thus, using the method provided in this application, a simulated video of the advertising image simulating verbal expression of the advertising content can be generated. The advertising image simulating verbal expression in this simulated video can be understood as an advertising digital human.

[0095] The solution proposed in this application can also be applied to the generation of virtual anchor digital humans during live streaming. For example, when an anchor wants to broadcast live but does not want to show their face, a target image containing a virtual avatar (such as a first object, which can be an animated avatar) can be provided. During the anchor's speech, audio snippets can be captured in real-time (e.g., every 5 seconds). The captured audio can also be converted into corresponding text in real-time. The captured audio and / or converted text can be used as the aforementioned N indicator data. Using these N indicator data and the target image, a simulated video of the virtual avatar verbally expressing the anchor's words can be generated. This simulated video can be played in real-time, achieving the effect that when the anchor broadcasts, the audience sees a video of the anchor broadcasting through this virtual avatar, and the anchor's lip movements can match (e.g., be identical) to the virtual avatar's lip movements. The virtual avatar verbally expressing words in this simulated video can be understood as a virtual anchor digital human.

[0096] Optionally, in the scenario of generating a virtual anchor digital human during the live broadcast, the target image of the virtual image provided by the anchor can be selected by the anchor in the video client (in this scenario, it can be a live broadcast client). In other words, the video client of this application can provide images of some popular virtual images (such as a virtual first object) for users to choose from, thereby simulating language expression through the virtual image.

[0097] This application can obtain a feature set and an input noise map of a diffusion network. The feature set includes image features of a target image and indicator features of N indicator data, where N is a positive integer. The target image contains a first object, and the N indicator data belong to different data modalities. All N indicator data are used to indicate the target content of the first object in simulated language expression. The diffusion network is invoked to perform temporal noise prediction processing on the input noise map under the constraints of the feature set, generating M noise prediction features. The M noise prediction features are temporally correlated, where M is a positive integer. Based on the input noise map, the M noise prediction features are transformed to generate a simulated video of the first object. The simulated video is a video of the first object simulating language expression of the target content. One noise prediction feature is used to transform and generate a corresponding video frame in the simulated video. Therefore, the method proposed in this application can generate a simulated video of the first object through a diffusion network. This diffusion network can easily introduce multiple modal (e.g., N modal) indication data combined with the image features of the target image to constrain the generation of the simulated video of the first object. In the process of generating the simulated video, the diffusion network can perform temporal noise prediction processing on the input noise map, so that the generated M noise prediction features are temporally correlated. Thus, the video frames in the simulated video generated by the M noise prediction features can also have visual continuity. Therefore, the method of this application can generate a simulated video of the first object with excellent effect.

[0098] Please refer to Figure 5, which is a schematic flowchart of a noise prediction process for an input noise map provided in an embodiment of this application. As shown in Figure 5, the process may include:

[0099] Step S201: Invoke the diffusion network to generate object constraint features of the first object under the constraints of the image features of the target image.

[0100] Specifically, the generating device can call the diffusion network to generate object constraint features of the first object under the constraints of the image features of the target image. The object constraint features can be understood as features that fuse the image features of the target image. Therefore, the object constraint features include the features of the first object itself (such as facial features), that is, the object constraint features fuse the features of the first object itself.

[0101] Optionally, the diffusion network may include a self-attention network layer, which can be a network layer with a self-attention mechanism. The diffusion network can first learn self-attention from the network parameters within the diffusion network through this self-attention network layer, thereby generating self-attention features.

[0102] The diffusion network may also include an identity control network layer, which may be a network layer for fusing features of the first object. The identity control network layer may be a network layer that includes a cross attention mechanism. The input of the identity control network layer may include the image features of the target image and the self-attention features.

[0103] Therefore, the identity control network layer can use a cross-attention mechanism to cross-learn the image features of the target image and the self-attention feature, thereby generating the aforementioned object constraint features of the first object. Specifically, the process by which the identity control network layer cross-learns the image features of the target image with other features is the process of generating object constraint features under the constraints of the image features of the target image.

[0104] Step S202: Invoke the diffusion network to generate fused constraint features under the constraints of object constraint features and indicator features of N indicator data.

[0105] Specifically, the diffusion network may include a conditional control network layer. The condition can be a condition that simulates the language expression, that is, the condition can be a condition that provides the content to be expressed in language (such as target content). Therefore, the conditional control network layer can be a network layer for fusing features of the instruction data, or it can be a network layer that includes a cross-attention mechanism.

[0106] Therefore, this application can achieve the fusion of the above-mentioned object constraint features and the indication features of N indication data through the conditional control network layer, as described below.

[0107] The indicator features of the above N indicator data can be features with the same feature dimension. Therefore, the generating device can perform splicing processing on the N indicator features (such as horizontal splicing or vertical splicing, depending on the actual application scenario) to generate spliced ​​features. These spliced ​​features are the features obtained after splicing the N indicator features.

[0108] The input to the conditional control network layer can include the aforementioned object constraint features and the concatenated features. The conditional control network layer can use a cross-attention mechanism to perform cross-learning processing on these object constraint features and the concatenated features, thereby generating fused constraint features. These fused constraint features simultaneously integrate the features of the indicator data and the features of the first object. Specifically, the process by which the conditional control network layer performs cross-learning on the object constraint features and the indicator features of the indicator data is the process of generating fused constraint features under the constraints of these two features.

[0109] Step S203: Based on the fusion constraint features, perform temporal noise prediction processing on the input noise map to generate M noise prediction features.

[0110] Specifically, the diffusion network of this application may also include a temporal network, which can be a network layer capable of processing data in a temporal manner. That is, the temporal network can be a network capable of temporal processing, such as the temporal network being able to interactively learn previously generated features to generate subsequent features.

[0111] The aforementioned fusion constraint features can include M fusion sub-features. These M fusion sub-features can be understood as frame-level features, meaning that these M fusion sub-features can be mapped to the final video frames to be generated, with one fusion sub-feature corresponding to one video frame.

[0112] Therefore, the generating device can call a temporal network to perform feature interaction learning on the M fused sub-features in time to generate M noise prediction features. This feature interaction learning process belongs to the noise prediction processing of the input noise map. This feature interaction learning can be used to make the generated M noise prediction features temporally correlated.

[0113] In other words, the M noise prediction features of this application have interactive information. For example, a noise prediction feature generated later in the M noise prediction features can be generated after interacting with the information of the noise prediction features generated earlier. Therefore, the M noise prediction features are temporally related. For example, the first noise prediction feature interacts with the second noise prediction feature, the second noise prediction feature interacts with the third noise prediction feature, and so on. The M noise prediction features can also be frame-level features. For example, the first noise prediction feature corresponds to the first video frame to be generated, the second noise prediction feature corresponds to the second video frame to be generated, and so on.

[0114] Please refer to Figure 6, which is a schematic diagram of a scenario for generating noise prediction features provided by an embodiment of this application. As shown in Figure 6, the diffusion network may include a self-attention network layer, an identity control network layer, a conditional control network layer, and a temporal network. The self-attention network layer has a self-attention mechanism, and the identity control network layer and the conditional control network layer may include a cross-attention mechanism. The temporal network may be a network capable of performing temporal processing on the data.

[0115] Therefore, the diffusion network can first generate self-attention features through the self-attention network layer, and then input these self-attention features into the identity control network layer, and input the image features of the target image into the identity control network layer. The identity control network layer can perform cross-learning processing on the input self-attention features and the image features of the target image to generate object constraint features.

[0116] The object constraint features can be input into the conditional control network layer, and the concatenated features of the indicator features of the aforementioned N indicator data can also be input into this conditional control network layer. This conditional control network layer can perform cross-learning processing on the input object constraint features and the concatenated features to generate fused constraint features. These fused constraint features are those that fuse both the features of the target image and the features of the N indicator data.

[0117] Finally, the fusion constraint feature can be input into a temporal network, which can perform temporal feature interaction processing on the fusion constraint feature to generate M noise prediction features.

[0118] It's important to clarify that the self-attention network layer can be understood as a network layer used to learn intra-frame features, meaning it can learn the image features of each video frame to be generated. The temporal network, on the other hand, can be understood as a network layer used to learn inter-frame features, meaning it can learn the correlation features (or coherence features) between each video frame to be generated. Therefore, by combining this self-attention network layer and the temporal network, not only intra-frame features but also inter-frame features can be learned, enabling the prediction and generation of highly effective simulated videos.

[0119] It is understandable that the entire data processing process of the diffusion network (i.e. the feature prediction process) is essentially a noise prediction process.

[0120] In this application, a temporal network is introduced to perform noise prediction processing on the input noise map, so that the M noise prediction features generated for noise prediction processing are temporally correlated. This allows the M video frames ultimately predicted by the M noise prediction features to also have visual continuity. For example, the position and lip movements of the first object in adjacent video frames are continuous without significant jumps.

[0121] Please refer to Figure 7, which is a flowchart illustrating a model training method provided in an embodiment of this application. As shown in Figure 7, the method may include:

[0122] Step S301: Obtain the sample feature set and the reference video. The sample feature set includes the image features of the sample image and the indicator features of K sample indicator data, where K is a positive integer and K is greater than or equal to N. The K sample indicator data belong to different data modalities. The sample image contains a third object. The K sample indicator data are all used to indicate the sample content of the third object simulating language expression. The reference video is a video of the third object expressing the sample content in language. The reference video contains L video frames, where L is a positive integer.

[0123] Specifically, the generating device can acquire a sample feature set and a reference video. The sample feature set can include image features of the sample images and indicator features of K sample indicator data, where K is a positive integer and greater than or equal to N. These K sample indicator data also belong to different data modalities. The N data modalities to which the aforementioned N indicator data belong can be included in the K data modalities to which the K sample indicator data belong; that is, the N data modalities can be some or all of the K data modalities.

[0124] The sample image may contain a third object. The K sample indicator data are used to indicate the sample content of the third object simulating language expression. Therefore, it can be understood that the concept of the sample feature set here is similar to the concept of the feature set mentioned above, the concept of the third object here is similar to the concept of the first object mentioned above, the concept of the sample image here is similar to the concept of the target image mentioned above, and the concept of the K sample indicator data here is similar to the concept of the N indicator data mentioned above. The only difference is that the sample feature set, the third object, the sample image, and the K sample indicator data are used during the training process, while the feature set, the first object, the target image, and the N indicator data are used in the application process after the training is completed.

[0125] The reference video can be understood as a labeled video (also called a sample video). This reference video is the target video in which a third party verbally expresses the sample content; that is, the reference video can represent the effect of the final video to be predicted. The reference video can contain L frames, where L is a positive integer. The specific value of L can be determined based on the actual scenario. For example, L can be equal to the number of frames (e.g., 16 or 32 frames) that the initial diffusion network can generate in a batch.

[0126] Step S302: Call the initial diffusion network to add noise to the L frames of video images, generate L frames of noisy video images, and call the initial diffusion network to perform noise prediction processing on the L frames of noisy video images under the constraint of the sample feature set, generate L sample noise prediction features.

[0127] Specifically, the initial diffusion network is a network that needs to be trained. In this application, the initial diffusion network can be used to add noise to the reference video and then denoise it to achieve the training process of the initial diffusion network, as described below.

[0128] The generating device can invoke the initial diffusion network to add noise to L frames of the aforementioned reference video to generate L frames of noisy video. In other words, adding noise to one frame of the reference video yields a corresponding noisy video frame. Optionally, the principle of adding noise to the reference video can be found in the following formula:

[0129] Among them, X t The initial diffusion network (EDN) represents the noisy video frame obtained by adding noise. Here, t is the number of noise addition steps (or the number of times, or time t), X0 is the original reference video frame, α is the hyperparameter used for noise addition, and ε is a random variable following a standard normal distribution, used to simulate the image diffusion process for noise addition. The EDN can progressively perform multi-step noise addition and denoising processing on the original reference video frame; this denoising process is essentially noise prediction. Please refer to Figure 8, which is a schematic diagram illustrating the effect of noise addition processing on a video frame according to an embodiment of this application. As shown in Figure 8, the EDN can progressively perform multi-step (i.e., multiple) noise addition processing on the original reference video frame, thereby allowing for denoising processes that skip steps between arbitrary numbers of steps.

[0130] Therefore, the generating device can call the initial diffusion network to perform noise prediction processing (or temporal noise prediction processing) on ​​the L frames of noisy video under the constraints of the above sample feature set, and can generate L sample noise prediction features. Performing noise prediction processing on a noisy video frame can generate a corresponding sample noise prediction feature, as described below.

[0131] The generating device can select indication features from the indication features of the above K sample indication data for noise prediction processing of L frames of noisy video images. The selected indication features can be called sample indication features. The sample indication features can be some or all of the indication features of the K indication features of the K sample indication data.

[0132] For example, the method of selecting sample indicator features from the indicator features of K sample indicator data may include at least one of the following:

[0133] The generating device can perform partial masking processing on the indicator features of K sample indicator data (e.g., it can perform random masking processing on the indicator features of 1 sample indicator data, or random masking processing on the indicator features of 2 sample indicator data, etc.) to obtain sample indicator features. These sample indicator features can include the indicator features of the K sample indicator data that have not been masked, that is, the indicator features that have been masked will not participate in the subsequent noise prediction processing.

[0134] Alternatively, the generating device can select all the indicator features of the K sample indicator data as sample indicator features.

[0135] Therefore, the generating device can invoke the initial diffusion network to perform noise prediction processing on L frames of noisy video images under the constraints of image features and sample indicator features of the sample images, thereby generating L sample noise prediction features. The principle by which the initial diffusion network performs noise prediction processing on L frames of noisy video images under the constraints of image features and sample indicator features of the sample images to generate L sample noise prediction features is the same as the principle by which the diffusion network performs noise prediction processing on the input noisy map under the constraints of image features of the target image and indicator features of N indicator data to generate M noise prediction features. That is, the role of the selected sample indicator features in the noise prediction processing is similar to the role of the indicator features of the N indicator data in the noise prediction processing.

[0136] In each round of training of the initial diffusion network, sample indicator features for noise prediction processing can be selected according to the above principle. The data modes of the sample indicator data to which the sample indicator features selected in each round belong can be the same or different, so that the initial diffusion network can be trained by combining some or all of the indicator features of the K sample indicator data.

[0137] If K equals 3, then according to the above method, in some training epochs of the initial diffusion network, the selected sample indicator features can be only the indicator features of the sample indicator data of one of the K data modes. That is, in this part of the training epochs of the initial diffusion network, training can be performed only using the indicator features of the sample indicator data of that one data mode. Also, in some training epochs of the initial diffusion network, the selected sample indicator features can be only the indicator features of the sample indicator data of two of the K data modes. That is, in this part of the training epochs of the initial diffusion network, training can be performed only using the indicator features of the sample indicator data of those two data modes (the two data modes in different training epochs can also be the same or different). Furthermore, in some training epochs of the initial diffusion network, the selected sample indicator features can include the indicator features of the sample indicator data of all the K data modes. That is, in this part of the training epochs of the initial diffusion network, training can be performed using the indicator features of the sample indicator data of all the K data modes. In this way, the initial diffusion network can be trained as a whole by combining some data modes from the K data modes and all data modes.

[0138] In other words, during the training of the initial diffusion network, this application can achieve the training of the initial diffusion network through a combination of 1 to K data modalities. That is, different training rounds can train the initial diffusion network using the indicator features of sample indicator data of different tree data modalities.

[0139] Specifically, this application can use sample indicator data of 1 (any 1) data modality, 2 (any 2) data modality, 3 (any 3) data modality, ..., and K data modality in different training rounds of the initial diffusion network to train the initial diffusion network in a combination of different types and numbers of data modalities. This achieves comprehensive training of the initial diffusion network, so that the final trained diffusion network can generate the corresponding simulated video of the object (such as the first object mentioned above) using any number of data modality indicator data from 1 to K.

[0140] Step S303: Using L sample noise prediction features, generate L sample prediction noise maps for L frames of noisy video images.

[0141] Specifically, the generating device can use the L sample noise prediction features generated above to generate L sample predicted noise maps of L frames of noisy video images. This principle is the same as the principle of generating M predicted noise maps by generating M noise prediction features.

[0142] One sample noise prediction feature can be used to predict and generate a corresponding sample prediction noise map. In other words, a sample prediction noise map is a noise map predicted for a corresponding noisy video frame.

[0143] Among them, X t-a This refers to the video frame after denoising (after noise prediction processing). t represents the number of noise addition steps, α is the hyperparameter used for noise addition, ε is a random variable following a standard normal distribution, and a is the number of denoising jumps (i.e., interval steps), which can be used to speed up the denoising process. c represents the denoising conditional information (such as sample indicator features), X t This refers to the noisy video footage that needs to be denoised.

[0144] Step S304: Based on the difference between the noise map predicted by L samples and the noise added in the L frames of noisy video, the network parameters of the initial diffusion network are corrected to obtain the diffusion network.

[0145] Specifically, the generating device can correct the network parameters of the initial diffusion network by predicting the difference between the noise map and the noise added in the L frames of noisy video using L samples. The goal of the correction is to minimize this difference (e.g., to zero), ultimately obtaining the aforementioned diffusion network, which is the trained diffusion network. The noise added to the noisy video is the noise added during the noise processing of the corresponding original video frame in the reference video.

[0146] For example, the generating device can generate the noise prediction loss of the initial diffusion network by using the L sample noise maps generated above and the noise added in the L frames of noisy video images. This noise prediction loss is the prediction bias of the initial diffusion network in noise prediction processing. This noise prediction loss can be used to reflect the difference between the sample noise maps and the noise added in the L frames of noisy video images. The larger the noise prediction loss, the greater the difference; conversely, the smaller the noise prediction loss, the smaller the difference.

[0147] Optionally, the noise prediction loss can be the MSE (mean squared error) loss between the L sample predicted noise maps and the noise added in the L frames of noisy video. There is an MSE loss between a sample predicted noise map and the noise added in the corresponding noisy video. Therefore, the noise prediction loss can be the sum of the MSE losses between each sample predicted noise map and the noise added in the corresponding noisy video (i.e., the summed value).

[0148] Therefore, the generation device can use this noise prediction loss to correct the network parameters of the initial diffusion network to obtain the trained diffusion network in the embodiment corresponding to Figure 3. The goal of correcting the network parameters of the initial diffusion network is to minimize the noise prediction loss (e.g., to 0). The generation device can perform several rounds of iterative training on the initial diffusion network according to the above principle to finally obtain the aforementioned diffusion network.

[0149] Please refer to Figure 9, which is a schematic diagram of a model training scenario provided in an embodiment of this application. As shown in Figure 9, the reference video may contain L frames of video images. The generating device can add noise to these L frames of video images, that is, add noise to these L frames of video images to obtain L frames of noisy video images. Specifically, adding noise to 1 frame of the reference video image can obtain a corresponding 1 frame of noisy video image.

[0150] Therefore, the initial diffusion network can perform noise prediction processing on the L noisy video frames under the constraint of the sample feature set, and can generate L sample prediction noise maps. Specifically, performing noise prediction processing on 1 noisy video frame can generate a corresponding 1 sample prediction noise map.

[0151] By analyzing the difference between the noise map predicted by the L samples and the noise added to the L noisy video frames (i.e., the noise added to the L frames of the reference video respectively), the noise prediction loss of the initial diffusion network for the L noisy video frames can be generated.

[0152] The generating device can backpropagate the noise prediction loss to the initial diffusion network to correct the network parameters of the initial diffusion network, thereby training the aforementioned diffusion network.

[0153] Optionally, the feature mapping network corresponding to the indication data of each of the K modalities in this application can be jointly trained with the initial diffusion network. One feature mapping network can correspond to one modality's indication data, allowing the feature mapping networks corresponding to the indication data of various modalities in this application to better adapt to the needs of the diffusion network and perform more accurate feature mapping processing (such as more accurate feature enhancement processing) on ​​the indication data of various modalities. In other words, the K feature mapping networks corresponding to the indication data of the K modalities in this application can all be trained together with the initial diffusion network using the noise prediction loss of the initial diffusion network.

[0154] In the embodiment corresponding to the N indication data mentioned above, the feature mapping network is included in the K trained feature mapping networks corresponding to the indication data of the K modalities. That is, the K trained feature mapping networks include the N feature mapping networks corresponding to the N indication data described in the embodiment corresponding to the N indication data mentioned above.

[0155] In this application, the initial diffusion network is trained by combining 1 to K data modalities. This enables the final trained diffusion network to support the generation of simulated videos of corresponding objects (such as the first object) simulating language expression using indicator data of any number of data modalities from 1 to K (such as N indicator data of the above N data modalities). This makes the generation of simulated videos more flexible and controllable, and facilitates users to provide corresponding indicator data, thereby improving the user experience.

[0156] This application employs a diffusion network, which can model both spatially (through self-attention network layers) and temporally (through temporal networks), thus effectively addressing the inter-frame jitter problem inherent in traditional methods. Furthermore, this diffusion network unifies multimodal input signals (such as indicator data from various data modalities) as conditions. Inputs can be audio signals (such as the aforementioned audio data, which can be speech signals), video footage of other individuals (such as the aforementioned video data), and / or text information (such as the aforementioned text data). Based on these multimodal signals as control conditions, controllable, expressive, and high-quality speaking face videos can be generated.

[0157] Please refer to Figure 10, which is a schematic diagram of a data generation device provided in an embodiment of this application. As shown in Figure 10, the data generation device 90 may include: an acquisition module 901, a prediction module 902, and a generation module 903.

[0158] The acquisition module 901 is used to acquire the feature set and the input noise map of the diffusion network. The feature set includes the image features of the target image and the indicator features of N indicator data, where N is a positive integer. The target image contains a first object. The N indicator data belong to different data modes. All N indicator data are used to indicate the target content of the first object to simulate language expression.

[0159] The prediction module 902 is used to call the diffusion network to perform temporal noise prediction processing on the input noise map under the constraint of the feature set, and generate M noise prediction features. The M noise prediction features are temporally correlated, and M is a positive integer.

[0160] The generation module 903 is used to perform video image transformation processing on M noise prediction features based on the input noise map to generate a simulated video of the first object. The simulated video is a video in which the first object simulates the verbal expression of the target content. One noise prediction feature is used to transform and generate a corresponding video frame in the simulated video.

[0161] Optionally, the N indicator data include at least one of the following:

[0162] Text data in the text modality, audio data in the audio modality, and video data in the video modality;

[0163] Among them, video data refers to the video of the second object expressing the target content in language, and the text content described by the text data and the audio content expressed by the audio data are both the target content.

[0164] Optionally, the acquisition module 901 described above is used for:

[0165] Obtain the feature encoding network and feature mapping network corresponding to each of the N indicator data;

[0166] Call the feature encoding network corresponding to each of the N indicator data, perform feature encoding processing on each indicator data respectively, and generate the initial encoded features of each indicator data;

[0167] Invoke the feature mapping network corresponding to each indicator data, and perform feature mapping processing on the initial encoded features of each indicator data to generate the indicator features of each indicator data;

[0168] In this context, the initial encoded features of each of the N indicator data are mapped to the same feature dimension by the corresponding feature mapping network.

[0169] Optionally, the prediction module 902 calls the diffusion network to perform temporal noise prediction processing on the input noise map under the constraints of the feature set, generating M noise prediction features in the following ways:

[0170] The diffusion network is invoked to generate object constraint features of the first object under the constraints of the image features of the target image;

[0171] The diffusion network is invoked to generate fused constraint features under the constraints of object constraint features and indicator features of N indicator data;

[0172] Based on the fusion constraint features, the input noise map is processed for temporal noise prediction, generating M noise prediction features.

[0173] Optionally, the indicator features of the N indicator data have the same feature dimension;

[0174] The prediction module 902 invokes the diffusion network to generate fused constraint features under the constraints of object constraint features and indicator features of N indicator data, including:

[0175] The indicator features of N indicator data are concatenated to generate concatenated features;

[0176] The diffusion network is invoked to perform cross-learning processing on the object constraint features and splicing features to generate fused constraint features.

[0177] Optionally, the diffusion network includes a temporal network, and the fusion constraint feature includes M fusion sub-features;

[0178] The prediction module 902 performs temporal noise prediction processing on the input noise map based on fusion constraint features, generating M noise prediction features in the following ways:

[0179] A temporal network is invoked to perform feature interaction learning on M fused sub-features in time sequence, generating M noise prediction features;

[0180] Among them, the feature interaction learning process belongs to the noise prediction processing of the input noise map. Feature interaction learning is used to make the generated M noise prediction features correlated in time.

[0181] Optionally, the generation module 903 can generate a simulated video of the first object by transforming the M noise prediction features based on the input noise map, including:

[0182] Using M noise prediction features, M predicted noise maps are generated from the input noise map. One noise prediction feature is used to generate a corresponding predicted noise map.

[0183] Based on the input noise map, the M predicted noise maps are transformed into video frames to generate a simulated video.

[0184] Optionally, the generation module 903 performs video frame conversion processing on the M predicted noise maps based on the input noise map to generate simulated video, including:

[0185] The input noise map is subtracted from each predicted noise map to generate M image prediction features;

[0186] The M scene prediction features are respectively processed by feature decoding to generate M video frames. One scene prediction feature is used to decode and generate one video frame. The M video frames constitute the analog video.

[0187] The video image conversion process includes difference processing and feature decoding.

[0188] Optionally, the data generation device 90 further includes a training module 904, which is used for:

[0189] Obtain a sample feature set and a reference video. The sample feature set includes the image features of the sample image and the indicator features of K sample indicator data, where K is a positive integer and K is greater than or equal to N. The K sample indicator data belong to different data modalities. The sample image contains a third object. The K sample indicator data are all used to indicate the sample content of the third object simulating language expression. The reference video is a video of the third object expressing the sample content in language. The reference video contains L video frames, where L is a positive integer.

[0190] The initial diffusion network is invoked to add noise to L frames of video images, generating L frames of noisy video images. Then, the initial diffusion network is invoked to perform noise prediction processing on the L frames of noisy video images under the constraints of the sample feature set, generating L sample noise prediction features.

[0191] Using L sample noise prediction features, L sample predicted noise maps are generated for L frames of noisy video footage;

[0192] Based on the difference between the noise map predicted by L samples and the noise added in the L frames of noisy video, the network parameters of the initial diffusion network are corrected to obtain the diffusion network.

[0193] Optionally, the training module 904 above calls the initial diffusion network to perform noise prediction processing on L frames of noisy video under the constraint of the sample feature set, generating L sample noise prediction features in the following ways:

[0194] Select sample indicator features for noise prediction processing from the indicator features of K sample indicator data;

[0195] The initial diffusion network is invoked to perform noise prediction processing on L frames of noisy video images under the constraints of image features and sample indicator features of the sample images, generating L sample noise prediction features.

[0196] Optionally, the methods for selecting sample indicator features from the indicator features of K sample indicator data include:

[0197] Partial masking is performed on the indicator features of K sample indicator data to obtain sample indicator features, which include the indicator features of the K sample indicator data that were not masked; or,

[0198] The indicative features of the K sample indicative data are all used as sample indicative features.

[0199] Optionally, the training module 904 corrects the network parameters of the initial diffusion network based on the difference between the predicted noise map from L samples and the noise added to the L frames of noisy video footage, obtaining the diffusion network in the following ways:

[0200] Based on the L samples of the predicted noise map and the noise added in the L frames of noisy video, the noise prediction loss of the initial diffusion network is generated.

[0201] The network parameters of the initial diffusion network are corrected using noise prediction loss to obtain the diffusion network;

[0202] The noise prediction loss is used to reflect the difference between the noise map predicted by the L samples and the noise added to the L frames of noisy video footage.

[0203] Optionally, the target image and N indicator data are acquired and sent by the video client;

[0204] The data generation device 90 further includes a sending module 905, which is used for:

[0205] The generated simulated video is returned to the video client, allowing the video client to play the simulated video in its client interface.

[0206] According to one embodiment of this application, the steps involved in the data generation method shown in FIG3 can be executed by various modules in the data generation apparatus 90 shown in FIG10. For example, step S101 shown in FIG3 can be executed by the acquisition module 901 in FIG10, step S102 shown in FIG3 can be executed by the prediction module 902 in FIG10, and step S103 shown in FIG3 can be executed by the generation module 903 in FIG10.

[0207] The apparatus proposed in this application can generate a simulated video of a first object through a diffusion network. This diffusion network facilitates the introduction of multiple modal (e.g., N modal) indication data combined with the image features of the target image to constrain the generation of the simulated video of the first object. Furthermore, during the generation of the simulated video, the diffusion network can perform temporal noise prediction processing on the input noise map, making the generated M noise prediction features temporally correlated. Thus, the video frames in the simulated video generated by the M noise prediction features can also have visual continuity. Therefore, the apparatus of this application can generate a simulated video of the first object with excellent results.

[0208] According to one embodiment of this application, the modules in the data generation apparatus 90 shown in FIG10 can be individually or entirely merged into one or more units, or some of the units can be further divided into multiple functionally smaller sub-units to achieve the same operation without affecting the technical effect of the embodiment of this application. The above modules are based on logical function division. In practical applications, the function of one module can also be implemented by multiple units, or the function of multiple modules can be implemented by one unit. In other embodiments of this application, the data generation apparatus 90 may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units.

[0209] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0210] According to one embodiment of this application, a computer program capable of executing the steps involved in the corresponding methods shown in the various embodiments of this application can be run on a general-purpose computer device (which may include processing elements and storage elements such as a central processing unit (CPU), random access storage medium (RAM), and read-only storage medium (ROM)) to construct the data generation apparatus 90 shown in FIG10. The aforementioned computer program can be recorded on a computer-readable recording medium, and can be loaded into and run in the aforementioned computer device via the computer-readable recording medium.

[0211] Please refer to Figure 11, which is a schematic diagram of the structure of a computer device provided in an embodiment of this application. As shown in Figure 11, the computer device 1000 may include a processor 1001, a network interface 1004, and a memory 1005. In some embodiments, the computer device 1000 may also include a user interface 1003 and at least one communication bus 1002. The communication bus 1002 is used to implement communication between these components. The user interface 1003 may include a display screen and a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed RAM or a non-volatile memory, such as at least one disk storage device. Optionally, the memory 1005 may also be at least one storage device located remotely from the aforementioned processor 1001. As shown in Figure 11, the memory 1005, as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a device control application.

[0212] In the computer device 1000 shown in Figure 11, the network interface 1004 provides network communication functions; the user interface 1003 is mainly used to provide an input interface for the user; and the processor 1001 can be used to call the device control application stored in the memory 1005 to achieve:

[0213] Obtain the feature set and the input noise map of the diffusion network. The feature set includes the image features of the target image and the indicator features of N indicator data, where N is a positive integer. The target image contains a first object. The N indicator data belong to different data modalities. All N indicator data are used to indicate the target content of the first object to simulate language expression.

[0214] The diffusion network is invoked to perform temporal noise prediction processing on the input noise map under the constraint of the feature set, generating M noise prediction features. The M noise prediction features are temporally correlated, and M is a positive integer.

[0215] Based on the input noise map, M noise prediction features are used to transform the video frame and generate a simulated video of the first object. The simulated video is a video in which the first object simulates the verbal expression of the target content. One noise prediction feature is used to transform and generate a corresponding video frame in the simulated video.

[0216] It should be understood that the computer device 1000 described in the embodiments of this application can execute the data generation method described in the various embodiments of this application, and can also execute the data generation apparatus 90 described in the embodiment corresponding to FIG10 above, which will not be repeated here. In addition, the beneficial effects of using the same method will not be repeated here.

[0217] Furthermore, it should be noted that this application also provides a computer-readable storage medium storing a computer program. When a processor executes this computer program, it can perform the data generation methods described in the various embodiments of this application; therefore, they will not be repeated here. Additionally, the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the embodiments of the computer storage medium involved in this application, please refer to the description of the method embodiments of this application.

[0218] As an example, the aforementioned computer program can be deployed and executed on a single computer device, or deployed and executed on multiple computer devices located in one location, or executed on multiple computer devices distributed across multiple locations and interconnected via a communication network. These multiple computer devices distributed across multiple locations and interconnected via a communication network can form a blockchain network.

[0219] The aforementioned computer-readable storage medium can be an internal storage unit of the computer device, such as a hard drive or memory. It can also be an external storage device, such as a plug-in hard drive, smart media card (SMC), secure digital (SD) card, or flash card. Furthermore, the computer-readable storage medium can include both internal and external storage units of the computer device. This computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. It can also be used to temporarily store data that has been output or will be output.

[0220] This application provides a computer program product comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the data generation methods described in the embodiments of this application; therefore, these descriptions will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the embodiments of the computer-readable storage medium involved in this application, please refer to the description of the method embodiments of this application.

[0221] The terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the term "comprising," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or device that includes a series of steps or units is not limited to the listed steps or modules, but may optionally include steps or modules not listed, or may optionally include other step units inherent to these processes, methods, apparatuses, products, or devices.

[0222] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0223] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A data generation method, characterized in that, The method includes: Obtain a feature set and an input noise map of a diffusion network. The feature set includes image features of a target image and indicator features of N indicator data, where N is a positive integer. The target image contains a first object. The N indicator data belong to different data modalities. The N indicator data are all used to indicate the target content of the first object in simulating language expression. The diffusion network is invoked to perform temporal noise prediction processing on the input noise map under the constraints of the feature set, generating M noise prediction features. The M noise prediction features are temporally correlated, and M is a positive integer. Based on the input noise map, the M noise prediction features are processed to transform the video frame, generating a simulated video of the first object. The simulated video is a video of the first object simulating the verbal expression of the target content. One of the noise prediction features is used to generate a corresponding video frame in the simulated video.

2. The method as described in claim 1, characterized in that, The N indication data include at least one of the following: Text data in the text modality, audio data in the audio modality, and video data in the video modality; The video data is a video of the second object expressing the target content in language, and the text content described by the text data and the audio content expressed by the audio data are both the target content.

3. The method as described in claim 1 or 2, characterized in that, The method further includes: Obtain the feature encoding network and feature mapping network corresponding to the N indicator data respectively; The feature encoding network corresponding to each of the N indicator data is invoked to perform feature encoding processing on each indicator data to generate the initial encoded features of each indicator data. The feature mapping network corresponding to each indicator data is invoked to perform feature mapping processing on the initial encoded features of each indicator data to generate the indicator features of each indicator data. In this context, the initial encoded features of each of the N indicator data are mapped to the same feature dimension by the corresponding feature mapping network.

4. The method according to any one of claims 1-3, characterized in that, The process involves calling the diffusion network to perform temporal noise prediction processing on the input noise map under the constraints of the feature set, generating M noise prediction features, including: The diffusion network is invoked to generate object constraint features of the first object under the constraints of the image features of the target image; The diffusion network is invoked to generate fused constraint features under the constraints of the object constraint features and the indication features of the N indication data; Based on the fusion constraint features, the input noise map is subjected to temporal noise prediction processing to generate the M noise prediction features.

5. The method according to any one of claims 1-4, characterized in that, The indicator features of the N indicator data have the same feature dimension; The step of calling the diffusion network to generate fused constraint features under the constraints of the object constraint features and the indication features of the N indication data includes: The indication features of the N indication data are concatenated to generate concatenated features; The diffusion network is invoked to perform cross-learning processing on the object constraint features and the spliced ​​features to generate the fused constraint features.

6. The method according to any one of claims 1-5, characterized in that, The diffusion network includes a temporal network, and the fusion constraint feature includes M fusion sub-features; The step of performing temporal noise prediction processing on the input noise map based on the fusion constraint features to generate the M noise prediction features includes: The temporal network is invoked to perform feature interaction learning on the M fused sub-features in a temporal sequence to generate the M noise prediction features; The feature interaction learning process is a noise prediction process for the input noise map, and the feature interaction learning is used to make the generated M noise prediction features temporally correlated.

7. The method according to any one of claims 1-6, characterized in that, The step of converting the M noise prediction features based on the input noise map into a video frame to generate a simulated video of the first object includes: Using the M noise prediction features, M predicted noise maps are generated from the input noise map, where each noise prediction feature is used to generate a corresponding predicted noise map. Based on the input noise map, the M predicted noise maps are converted into video frames to generate the simulated video.

8. The method according to any one of claims 1-7, characterized in that, The step of converting the M predicted noise maps into video frames based on the input noise map to generate the simulated video includes: The input noise map is subtracted from each of the predicted noise maps to generate M image prediction features; The M scene prediction features are respectively subjected to feature decoding processing to generate M video frames. One of the scene prediction features is used to decode and generate one video frame. The M video frames constitute the simulated video. The video image conversion process includes the difference processing and the feature decoding processing.

9. The method according to any one of claims 1-8, characterized in that, The method further includes: A sample feature set and a reference video are obtained. The sample feature set includes image features of sample images and indicator features of K sample indicator data, where K is a positive integer and K is greater than or equal to N. The K sample indicator data belong to different data modalities. The sample images contain a third object. The K sample indicator data are all used to indicate the sample content of the third object simulating language expression. The reference video is a video of the third object expressing the sample content in language. The reference video contains L video frames, where L is a positive integer. The initial diffusion network is invoked to add noise to the L frames of video, generating L frames of noisy video. Then, the initial diffusion network is invoked to perform noise prediction processing on the L frames of noisy video under the constraints of the sample feature set, generating L sample noise prediction features. Using the L sample noise prediction features, L sample predicted noise maps are generated for the L frames of noisy video images; Based on the difference between the predicted noise map of the L samples and the noise added in the L frames of noisy video, the network parameters of the initial diffusion network are corrected to obtain the diffusion network.

10. The method according to any one of claims 1-9, characterized in that, The initial diffusion network, under the constraints of the sample feature set, performs noise prediction processing on the L frames of noisy video footage, generating L sample noise prediction features, including: From the indicator features of the K sample indicator data, sample indicator features for noise prediction processing are selected; The initial diffusion network is invoked to perform noise prediction processing on the L frames of noisy video images under the constraints of the image features and the sample indication features of the sample images, thereby generating the L sample noise prediction features.

11. The method according to any one of claims 1-10, characterized in that, The method for selecting the sample indication features from the indication features of the K sample indication data includes: The indication features of the K sample indication data are partially masked to obtain the sample indication features, which include the indication features of the K sample indication data that were not masked; or, The indicative features of the K sample indicative data are all used as the sample indicative features.

12. The method according to any one of claims 1-11, characterized in that, The process of correcting the network parameters of the initial diffusion network based on the difference between the predicted noise map from the L samples and the noise added to the L frames of noisy video footage to obtain the diffusion network includes: Based on the L sample predicted noise maps and the noise added in the L frames of noisy video images, the noise prediction loss of the initial diffusion network is generated; The noise prediction loss is used to correct the network parameters of the initial diffusion network to obtain the diffusion network; The noise prediction loss is used to reflect the difference between the predicted noise map of the L samples and the noise added to the L frames of noisy video footage.

13. The method according to any one of claims 1-12, characterized in that, The target image and the N indicator data are acquired and sent by the video client; The method further includes: The generated simulated video is returned to the video client, allowing the video client to play the simulated video in its client interface.

14. A data generation apparatus, characterized in that, The device includes: The acquisition module is used to acquire a feature set and an input noise map of a diffusion network. The feature set includes image features of a target image and indicator features of N indicator data, where N is a positive integer. The target image contains a first object. The N indicator data belong to different data modalities. The N indicator data are all used to indicate the target content of the first object in simulating language expression. The prediction module is used to call the diffusion network to perform temporal noise prediction processing on the input noise map under the constraints of the feature set, and generate M noise prediction features. The M noise prediction features are temporally correlated, and M is a positive integer. The generation module is used to perform video image transformation processing on the M noise prediction features based on the input noise map to generate a simulated video of the first object. The simulated video is a video of the first object simulating the verbal expression of the target content. One of the noise prediction features is used to generate a corresponding video frame in the simulated video.

15. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1-13.

16. A computer device, characterized in that, It includes a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the method according to any one of claims 1-13.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded by a processor and to execute the steps of the method according to any one of claims 1-13.

Images