Video generation method and device

Abstract

The embodiment of the invention provides a video generation method and device. The method comprises the steps of obtaining a plurality of face images containing different faces and a first text used for indicating generation of video content; based on each face image, a first feature is obtained through coding and noise adding processing, and the first feature is determined based on a noise feature and a face coding feature of each face in each face image; performing feature extraction on each face image, and determining a face visual feature corresponding to each face; fusing the text feature corresponding to the first text and each face visual feature to obtain a second feature; and based on the first feature, the second feature and the human face visual features corresponding to the human faces, generating a target video including the human faces, the human face visual features corresponding to the human faces being used for determining the positions of the human faces in the target video so as to generate a video including multiple human faces.

Description

Technical Field

[0001] This specification relates to the field of artificial intelligence technology, and in particular to a video generation method and device. Background Art

[0002] Identity-centric controllable video generation is an advanced interactive generation technology that allows users to controllably edit and generate video content. Identity-centricity means that during the video generation process, the facial features of the input image are not altered. For example, detailed facial features, clothing, hairstyle, and other personal attributes are preserved, ensuring the natural and consistent appearance of the subject in the generated video.

[0003] In the aforementioned video generation technology, many methods are used to generate videos for a single person, or a single face. However, there are relatively few methods that can simultaneously generate videos for multiple persons, or multiple different faces. Therefore, how to provide a method for generating videos for multiple persons, or multiple different faces, simultaneously becomes an urgent problem. Summary of the Invention

[0004] One or more embodiments of this specification provide a video generation method and apparatus to generate a video containing multiple faces.

[0005] According to a first aspect, a video generation method is provided, comprising:

[0006] Acquire a plurality of facial images containing different human faces and a first text for instructing to generate video content;

[0007] Based on each facial image, obtaining a first feature through coding and noise processing, wherein the first feature is determined based on the noise feature and the facial coding feature of each face in each facial image;

[0008] Perform feature extraction on each face image respectively to determine the facial visual features corresponding to each face;

[0009] Fusing the text features corresponding to the first text and the visual features of each face to obtain a second feature;

[0010] Based on the first feature, the second feature and the facial visual features corresponding to each face, a target video containing each face is generated, wherein the facial visual features corresponding to each face are used to determine the position of each face in the target video.

[0011] According to a second aspect, there is provided a video generating apparatus, comprising:

[0012] A first acquisition module is configured to acquire a plurality of facial images containing different human faces and a first text for instructing to generate video content;

[0013] a first obtaining module configured to obtain a first feature based on each facial image through coding and noise processing, wherein the first feature is determined based on the noise feature and the facial coding feature of each face in each facial image;

[0014] A first determination module is configured to extract features from each face image and determine the facial visual features corresponding to each face;

[0015] a first fusion module configured to fuse text features corresponding to the first text and visual features of each face to obtain a second feature;

[0016] A generation module is configured to generate a target video containing each of the faces based on the first feature, the second feature and the facial visual features corresponding to each face, wherein the facial visual features corresponding to each face are used to determine the position of each face in the target video.

[0017] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method according to the first aspect.

[0018] According to a fourth aspect, a computing device is provided, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described in the first aspect is implemented.

[0019] According to the video generation method and apparatus provided in the embodiments of this specification, a plurality of facial images containing different faces and a first text for indicating the generated video content are obtained; based on each facial image, a first feature is obtained through coding and noise processing, wherein the first feature is determined based on the noise feature and the facial coding feature of each face in each facial image; feature extraction is performed on each facial image to determine the facial visual feature corresponding to each face; the text feature corresponding to the first text and the facial visual feature of each face are fused to obtain a second feature, so that the second feature includes the aligned and fused text feature and the facial visual feature of each face, so that even if the text changes, the text semantic coherence of each face in the generated video and the identity consistency between the faces can be guaranteed; based on the first feature, the second feature and the facial visual feature corresponding to each face, a target video containing each face is generated, wherein the facial visual feature corresponding to each face is used to determine the position of each face in the target video, thereby providing a video generation method for multiple identities, i.e., multiple different faces, to generate a video containing multiple faces while ensuring the uniqueness and identity consistency of each face in the video and the text semantic coherence of each face. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive effort.

[0021] Figure 1 A schematic diagram of an implementation framework of an embodiment disclosed in this specification;

[0022] Figure 2 A schematic diagram of a structure of a generation model provided in an embodiment;

[0023] Figure 3 A schematic diagram of a video generation process based on a generative model provided in an embodiment;

[0024] Figure 4 A schematic diagram of a flow chart of a generative model training process provided in an embodiment;

[0025] Figure 5 A schematic diagram of a generative model provided in an embodiment;

[0026] Figure 6 A schematic diagram of a flow chart of a video generation method provided in an embodiment;

[0027] Figure 7 A schematic block diagram of a video generation device provided in an embodiment. DETAILED DESCRIPTION

[0028] The technical solutions of the embodiments of this specification will be described in detail below with reference to the accompanying drawings.

[0029] The embodiments of this specification disclose a video generation method and apparatus. The following first introduces the application scenarios and technical concepts of the method, which are as follows:

[0030] As mentioned above, in controllable video generation technology centered around person identities (i.e., faces), there are many methods for generating videos for a single person identity (i.e., a single face), but fewer methods for simultaneously generating videos for multiple person identities (i.e., multiple different faces). Therefore, how to provide a method for simultaneously generating videos for multiple person identities (i.e., multiple different faces) becomes an urgent problem.

[0031] In view of this, the inventor proposes a video generation method. Figure 1A schematic diagram of an implementation scenario according to an embodiment disclosed in this specification is shown. In this implementation scenario, an electronic device obtains multiple facial images containing different faces and a first text for instructing the generation of video content; based on each facial image, a first feature is obtained through coding and noise processing, wherein the first feature is determined based on the noise feature and the facial coding feature of each face in each facial image; feature extraction is performed on each facial image to determine the facial visual feature corresponding to each face; the text feature corresponding to the first text and the facial visual feature of each face are fused to obtain a second feature; based on the first feature, the second feature, and the facial visual feature corresponding to each face, a target video containing each face is generated, wherein the facial visual feature corresponding to each face is used to determine the position of each face in the target video.

[0032] In the above process, a video generation method for multiple character identities, i.e., multiple different faces, is provided to achieve the generation of a video containing multiple faces, wherein the text features corresponding to the first text and the visual features of each face are fused to obtain a second feature, so that the second feature contains the aligned and fused text features and the visual features of each face. In this way, even if the text changes, the text semantics consistency of each face in the generated video and the identity consistency between each face can be guaranteed, and the uniqueness of each face in the video and the text semantics consistency of each face can be guaranteed.

[0033] In some possible implementation scenarios, the electronic device can generate a video using a trained generation model based on multiple facial images containing different faces and text for indicating the generation of video content, so as to obtain a video containing the faces in the aforementioned facial images and ensure the uniqueness of each face in the video.

[0034] In some implementations, the generative model may be a diffusion model. In some specific examples, the generative model may be a diffusion model based on a Transformer architecture. Exemplarily, the Transformer architecture on which the generative model is based may be a 3D Swin Transformer architecture, or other Transformer architectures that incorporate a 3D spatiotemporal attention mechanism.

[0035] Figure 2 An exemplary structural diagram of the generation model is shown as follows: Figure 2 As shown in , the generative model can include an encoding network, an extraction network, a fusion network, and several denoising networks arranged in series. Figure 2 The number of denoising networks shown is M. The plurality of denoising networks arranged in series are used to predict noise in the input and denoise the input to obtain a denoising result, which can then be used to generate a video.

[0036] Among them, the output of any i-th denoising network among several denoising networks arranged in series can be used as the input of the next denoising network. Figure 2 As shown, the input of the first denoising network can include the output of the encoding network, the output of the fusion network, and the output of the extraction network. The input of the non-first denoising network includes the output of the previous denoising network and the output of the extraction network. The value range of i is an integer in [1, M].

[0037] In some specific examples, such as Figure 3 As shown, an exemplary process of an electronic device generating a video through a trained generation model is shown, which may include: the electronic device obtains multiple facial images containing different faces and a first text for indicating the generation of video content; then, based on each facial image, through the encoding network, a first feature is obtained through encoding and noise processing, wherein the first feature is determined based on the noise feature and the facial encoding feature of each face in each facial image; through the extraction network, feature extraction is performed on each facial image respectively to determine the facial visual feature corresponding to each face; through the fusion network, the text feature corresponding to the first text and the facial visual feature of each face are fused to obtain the second feature; through several serially arranged denoising networks, based on the first feature, the second feature and the facial visual feature corresponding to each face, a target video containing each face is generated, wherein the facial visual feature corresponding to each face is used to determine the position of each face in the target video.

[0038] Among them, for the first denoising network among several denoising networks arranged in series, its input includes the first feature, the second feature and the facial visual features corresponding to each face; for the non-first denoising network among several denoising networks arranged in series, its input includes the output of the previous denoising network and the facial visual features corresponding to each face.

[0039] The video generation method provided in this specification is described in detail below with reference to specific embodiments.

[0040] The following first introduces the training process of the aforementioned generative model.

[0041] In the embodiment of this specification, as mentioned above, the generation model includes an encoding network, an extraction network, a fusion network and several denoising networks arranged in series, such as Figure 4 As shown, the training process of the generative model may include the following steps S11-:

[0042] In step S11, a first sample is obtained, wherein the first sample includes: a plurality of sample face images containing different sample faces, a sample text for indicating the generation of video content, and a first sample video, the plurality of sample face images are face images captured from the first sample video, and the sample text corresponds to the video content of the first sample video.

[0043] The plurality of sample face images may include two or more face images. It is understandable that a single sample face image may include a single face.

[0044] In step S12, based on each sample face image, a first sample feature is obtained through coding and noise processing through a coding network, wherein the first sample feature is determined based on the sample noise feature and the sample face coding feature of each sample face in each sample face image.

[0045] Exemplarily, the sample noise feature may be randomly generated Gaussian noise.

[0046] Some possible examples include Figure 5 As shown, the coding network may include a coding layer, a concatenated denoising layer, and a mapping layer.

[0047] exist Figure 5 Based on the structure of the coding network shown, step S12 may include the following steps 121-123:

[0048] In step 121, based on each sample facial image, the aforementioned encoding layer is used to obtain sample facial coding features corresponding to each sample face. In this step, each sample facial image is input into the encoding layer, which processes the input sample facial image to obtain an encoding result for the sample facial image. Based on the encoding result of the sample facial image, the sample facial coding features are obtained, thereby obtaining sample facial coding features corresponding to each sample face.

[0049] In some possible examples, the encoding result of a single sample face image can be expressed as a multi-channel feature map. Exemplarily, the encoding result of a single sample face image can be expressed as (W1, H1, C1), or as (W1, H1, C1, 1), where C1 represents the number of channel dimensions of the encoding result of a single sample face image, W1 and H1 represent the width and height of the feature map in a single channel, respectively, and the dimension where 1 is located represents the dimension of the number of video frames. It can be understood that in image processing, a convolution kernel can correspond to a channel. The multi-channel feature map in the sample face encoding feature can refer to the features obtained by processing different convolution kernels of the encoding layer, that is, the features of different channels.

[0050] After obtaining the encoding results (W1, H1, C1) of each sample facial image, the encoding results of each sample facial image are filled based on the number of frames of the video to be generated, and the encoding results (W1, H1, C1, T) after filling of each sample facial image are obtained, where T represents the number of video frames required to generate the video. The number of video frames T required to generate the video can be a preset parameter. The encoding results (W1, H1, C1, T) after filling of each sample facial image are determined as the sample facial coding features of each sample facial image. Exemplarily, W1 and H1 can be equal. As another example, a specified value, such as 0, can be filled when filling.

[0051] In some possible implementations, the encoding layer may be implemented by a variational autoencoder (VAE), or by other encoders capable of encoding images. The parameters of the encoding layer may be trained parameters.

[0052] In step 122, by splicing the noise layer, each sample face coding feature and the sample noise feature are spliced ​​to obtain a first sample splicing feature. In this step, each sample face coding feature and the sample noise feature can be spliced ​​in a specified dimension to obtain a first sample splicing feature. Exemplarily, the specified dimension can be a channel dimension. Exemplarily, when splicing each sample face coding feature and the sample noise feature, each sample face coding feature and the sample noise feature can be spliced ​​in the channel dimension according to a specified splicing order. Exemplarily, the shape of the first sample splicing feature can be expressed as (W1, H1, C2, T), where C2 is equal to the sum of the number of channel dimensions corresponding to each sample face coding feature and the number of channel dimensions corresponding to the sample noise feature.

[0053] In some cases, the number of channel dimensions corresponding to each sample face coding feature is equal, and the number of channel dimensions corresponding to the sample noise feature may also be equal to the number of channel dimensions corresponding to the sample face coding feature, that is, C2 is equal to 3 times C1.

[0054] In the channel dimension, splicing the facial encoding features and noise features of each sample allows for decoupling of the facial encoding features, ensuring that they contribute equally to the video generation process. This avoids spatial constraints on the positions of each sample face in the generated video and promotes greater content richness and diversity. This splicing method enhances the generative model's ability to produce dynamic and visually coherent results, resulting in more visually coherent dynamic faces in the generated video.

[0055] In step 123, based on the first sample splicing feature, the first sample feature is obtained through the aforementioned mapping layer. In this step, the first sample splicing feature is input into the aforementioned mapping layer, and the first sample splicing feature is processed through the aforementioned mapping layer to obtain the first sample feature, thereby adjusting the dimension of the first sample splicing feature. Exemplarily, the shape of the first sample feature can be [W2, H2, C3, T], where C3 represents the number of channel dimensions of the first sample feature, and W2 and H2 can be the width and height of the feature map in a single channel. In some cases, W2 and H2 can be equal. In some cases, W2 is equal to W1, and H2 is equal to H1, and C3 is equal to the aforementioned C1.

[0056] In some possible examples, the mapping layer can be implemented by a linear layer and a regularization layer.

[0057] In step S13, feature extraction is performed on each sample face image through the aforementioned extraction network to determine the sample face visual features corresponding to each sample face.

[0058] In some possible examples, the extraction network may include a first extraction subnetwork. The first extraction subnetwork may be implemented by a first extraction layer and a first multi-layer perceptron (MLP).

[0059] In some specific examples, the first extraction layer can be implemented using a multimodal network, such as a CLIP (Contrastive Language–Image Pre-training) network. The first extraction layer can better extract semantically relevant features from the image, thereby better representing facial features therein.

[0060] In some other cases, the first extraction layer can also be implemented through a specified facial feature extractor, such as the AreFace extractor, so as to extract more facial features and better characterize the characteristics between each face, so as to better maintain the uniqueness between each sample face.

[0061] During the extraction process, for any nth sample face image among the sample face images, the nth sample face image is input into the first extraction subnetwork, and features of the nth sample face image are extracted through the first extraction layer therein to obtain a first sample face extraction result corresponding to the nth sample face image; then, the first sample face extraction result is input into the first multi-layer perceptron to obtain a first sample visual feature corresponding to the nth sample face image, that is, to obtain a first sample visual feature corresponding to the sample face in the nth sample face image, and the first sample visual feature is determined as the sample face visual feature corresponding to the sample face in the nth sample face image. Similarly, the sample face visual features corresponding to each sample face are obtained.

[0062] The value range of n is an integer in [1, N], where N represents the number of sample face images.

[0063] In some other possible examples, Figure 5 As shown, the extraction network may include a first extraction sub-network, a second extraction sub-network and an integration sub-network.

[0064] The first extraction subnetwork may be implemented by a first extraction layer and a first multilayer perceptron, and the second extraction subnetwork may be implemented by a second extraction layer and a second multilayer perceptron. The first extraction subnetwork and the second extraction subnetwork are provided in parallel. Parameters in the first extraction subnetwork and the second extraction subnetwork may be trained parameters.

[0065] The integration sub-network may be any network capable of integrating features. For example, the integration sub-network may be implemented by a Q-Former network. The Q-Former network may integrate input features by introducing a learnable query vector.

[0066] The first extraction layer can be implemented using a multimodal network, while the second extraction layer can be implemented using a specified facial feature extractor. Combining the first and second extraction subnetworks allows for multi-dimensional feature extraction from sample facial images, yielding information that better and more comprehensively characterizes the sample face, thereby ensuring the uniqueness and consistency of the face in the resulting video.

[0067] During the extraction process, for any nth sample face image in each sample face image, the following steps AC are performed to obtain the sample face visual features corresponding to each sample face:

[0068] In step A: the nth sample face image is input into the first extraction subnetwork, and features of the nth sample face image are extracted through the aforementioned first extraction layer to obtain a first sample face extraction result corresponding to the nth sample face image; then the first sample face extraction result is input into the first multi-layer perceptron to obtain a first sample visual feature corresponding to the nth sample face image.

[0069] In step B: the nth sample face image is input into the second extraction subnetwork, and features of the nth sample face image are extracted through the aforementioned second extraction layer to obtain a second sample face extraction result corresponding to the nth sample face image; then the second sample face extraction result is input into the second multi-layer perceptron to obtain a second sample visual feature corresponding to the nth sample face image.

[0070] Then, in step C: the first sample visual features and the second sample visual features corresponding to the n-th sample face image are spliced ​​in the channel dimension to obtain a splicing result; the splicing result is input into the integration sub-network, so as to process the splicing result through the integration sub-network to obtain the sample face visual features corresponding to the n-th sample face image, that is, the sample face visual features corresponding to the sample face in the n-th sample face image are obtained.

[0071] After obtaining the sample face visual features corresponding to each sample face in the above manner, in step S14, the sample text features corresponding to the sample text and the sample face visual features are fused through the aforementioned fusion network to obtain the second sample features. Figure 5 As shown, the sample text feature corresponding to the sample text may be a feature obtained by encoding the sample text through a text encoder. Exemplarily, the text encoder may be implemented by a T5 encoder.

[0072] In some possible examples, the fusion network may be a network based on a cross-attention mechanism. In still other possible examples, the fusion network may be implemented by instructing the fusion network to perform splicing in a specified dimension, such as a channel dimension, or by instructing the fusion network to add sample text features and visual features of each sample face.

[0073] The following takes the fusion network as an example of a network based on the cross-attention mechanism to introduce the fusion process of the sample text features corresponding to the fusion sample text and the visual features of each sample face.

[0074] The fusion network includes a first query Q weight matrix, a first key K weight matrix and a first value V weight matrix.

[0075] The fusion process may include:

[0076] Based on the first query Q weight matrix and the sample text features, the first query Q matrix Q is determined t For any n-th sample face corresponding to the sample face visual features of each sample face, determine the first key K matrix based on the first key K weight matrix and the sample face visual feature n. Based on the first value V weight matrix and the sample face visual feature n, determine the first value V matrix

[0077] Then, according to the preset attention mechanism formula, based on the first Q matrix Q t , the first K matrix and the first V matrix Determine the attention result corresponding to the sample facial visual feature n; calculate a first sum value of the attention results corresponding to multiple sample facial visual features; then calculate a second sum value of the first sum value and the sample text feature, and use the second sum value as the second sample feature.

[0078] In a specific example, the above fusion process can be expressed by the following formula (1):

[0079]

[0080] in, F t , represents the second sample feature, F t Represents the sample text features, represents the first query Q weight matrix, represents the first key K weight matrix, represents the first value V weight matrix, represents the sample face visual feature n corresponding to the nth sample face. as well as are trainable weight parameters.

[0081] In this process, the fusion network establishes semantic associations between the visual features and text features corresponding to each sample face. This allows alignment and fusion of these features with the text features, providing joint guidance for generating consistent identities for each face. By aligning and fusing the visual features and text features corresponding to each sample face through the fusion network, the semantic coherence of the text and identity consistency of each face in the generated video can be guaranteed, even if the sample text changes.

[0082] After obtaining the first sample feature and the second sample feature, then, in step S15, the aforementioned denoising networks are used to generate a generated video containing each sample face based on the first sample feature, the second sample feature, and the sample face visual features corresponding to each sample face, wherein the sample face visual features corresponding to each sample face are used to determine the position of each sample face in the generated video.

[0083] In some examples, the first sample feature and the second sample feature are spliced ​​together to obtain a second spliced ​​feature; then, the second spliced ​​feature and the sample face visual features corresponding to each sample face are processed through several denoising networks to obtain a final denoising result, and then, the second spliced ​​feature and the sample face visual features corresponding to each sample face are processed through several denoising networks to obtain a final denoising result. Figure 5 The decoding network shown generates a generated video containing each sample face based on the final denoising result. Exemplarily, the decoding network can be implemented as a VAE decoder. The structure of the decoding network can refer to the structure of the decoding network in the related art.

[0084] Among them, the sample face visual features corresponding to each sample face can be used separately to combine the sample intermediate features to determine the position of each corresponding sample face in the generated video, so as to ensure that each identity feature, that is, the feature of each sample face, is local in space, prevent overlap between identity features, and maintain the uniqueness of each identity, that is, the sample face.

[0085] Exemplarily, the second sample feature can exist in the form of a sequence, and its shape can be expressed as (b, length1, C4), where b represents batch_size, length1 represents the sequence length of the second sample feature, and C4 represents the number of channel dimensions corresponding to the second sample feature. This embodiment is described by taking batch_size as 1 as an example. Accordingly, the shape of the second sample feature can also be expressed as (length1, C4). The specific settings of C4 and length1 are related to the structure of the denoising network.

[0086] To facilitate subsequent calculations, a segmentation operation, such as a Patterned operation, needs to be performed on the first sample feature to convert the first sample feature from a feature graph form to a sequence form, thereby obtaining a feature sequence corresponding to the first sample feature. For example, based on a specified segmentation operation, the first sample feature can be segmented from a feature graph form having a shape of [W2, H2, C3, T] into a feature sequence having a shape of (length2, C4). The specified segmentation operation can be pre-set based on the structure of the denoising network and the shape of the second sample feature.

[0087] Accordingly, the aforementioned process of concatenating the first sample feature and the second sample feature may include: concatenating the feature sequence corresponding to the first sample feature and the second sample feature in the sequence length dimension to obtain a second concatenated feature, whose shape can be expressed as (length1+length2, C4). In this example, the shape of the sample facial visual features corresponding to each sample face can be expressed as (length3, C4), where the setting of length3 is related to length1.

[0088] Accordingly, the aforementioned Figure 5 The decoding network shown in FIG1 generates a generated video containing each sample face based on the final denoising result, which may include: first restoring the final denoising result from a sequence form to a feature map form through a specified restoration operation to obtain a restored result, for example, the restored result has a shape of [W2, H2, C3, T], and then, through Figure 5 The decoding network shown generates a generated video containing each sample face based on the restoration result in the form of the feature map. The specified restoration operation corresponds to the specified segmentation operation mentioned above.

[0089] In some possible examples, to prevent identity feature mixing, an identity localization layer is introduced, which provides spatial guidance for multiple identity features. The identity localization layer learns a localization map to predict the valid area of ​​each identity feature, thereby assigning each unique identity to its corresponding spatiotemporal position in the latent space.

[0090] In some possible examples, a processing layer and its corresponding identity localization layer can be set in at least one designated denoising network among the denoising networks, and processing layers can be set in other non-designated denoising networks. The processing layer is used to perform further feature processing on its input, and the identity localization layer is used to determine the position of each sample face in each generated video frame in the generated video.

[0091] In some other possible examples, Figure 5 As shown, each denoising network can include a processing layer and its corresponding identity localization layer. In this example, integrating the identity localization layer into each denoising network ensures that each identity feature is spatially localized, preventing overlap and maintaining the uniqueness of each identity. This allows for dynamic adjustment of facial feature localization during the video generation process. This integration allows the generative model to adaptively refine the spatial distribution of identity features, i.e., facial features, at different stages of the video generation process, ensuring accurate and coherent placement of multiple identities, i.e., faces, in the final output video.

[0092] It can be understood that several denoising networks can be used to predict noise on the input of their processing layers and remove the predicted noise from the input of their processing layers to obtain the final denoising result and then generate a generated video containing each sample face. In the denoising process, the identity positioning layer is combined to predict and remove noise for the positioning restrictions of each sample face.

[0093] The following describes the process of using several denoising networks to predict noise at the input of a processing layer and removing the predicted noise from the input of the processing layer, i.e., the denoising process. The denoising process may include the following steps:

[0094] In the first denoising network, the second splicing feature is processed through its first processing layer 1 to obtain the sample intermediate feature 1; through its first identity positioning layer 1, based on the sample intermediate feature 1 and the sample face visual features corresponding to each sample face, the output corresponding to the first identity positioning layer 1 is obtained; based on the output corresponding to the first identity positioning layer and the sample intermediate feature 1, the output 1 corresponding to the first denoising network is determined, and the output 1 corresponding to the first denoising network is used as the input of the second denoising network.

[0095] It can be understood that the data processing processes from the second denoising network to the last denoising network are similar, and are similar to the data processing process of the first denoising network. The following takes any i-th denoising network as an example to illustrate its data processing process. The data processing process of other denoising networks can refer to the data processing process of the i-th denoising network.

[0096] Specifically, in the i-th denoising network, the output corresponding to the i-1 denoising network is processed through its processing layer i to obtain the sample intermediate feature i; through its identity positioning layer i, based on the sample intermediate feature i and the sample face visual features corresponding to each sample face, the output corresponding to the identity positioning layer i is obtained; based on the output corresponding to the identity positioning layer and the sample intermediate feature i, the output i corresponding to the i-th denoising network is determined, and the output i corresponding to the i-th denoising network is used as the input of the i+1 denoising network.

[0097] This process is repeated until the output corresponding to the last denoising network is obtained, wherein the output includes the predicted noise features x predicted by the multiple denoising networks. Subsequently, the denoising result corresponding to this denoising process can be obtained based on the aforementioned second concatenated features and the predicted noise features x.

[0098] The above process exemplifies a single denoising process performed using multiple denoising networks. In some possible examples, multiple iterations of the denoising process may be required to achieve a final, effective denoising result. This final, effective denoising result can then be used to produce a clear, high-quality video.

[0099] The implementation process of each subsequent non-first denoising process can be referred to the implementation process of the aforementioned denoising process and will not be repeated here. It should be noted that in each non-first denoising process, the input of the denoising network includes: the denoising result corresponding to the previous denoising process and the sample facial visual features corresponding to each sample face.

[0100] After performing multiple denoising processes and obtaining the final denoising result, the Figure 5 The decoding network shown decodes and generates a video based on the final denoising result, and then proceeds to step S16.

[0101] The following introduces the respective structures of the processing layer and the identity positioning layer in the denoising network and their data processing process.

[0102] In some possible examples, the processing layers of each denoising network may include sublayers based on the attention mechanism, such as Figure 5 The "Attention layer" and the feedforward neural network sublayer are shown as Figure 5The structures of the “FFN layer” shown, the attention-based sublayer and the feedforward neural network sublayer, can be found in the structures of the attention-based sublayer and the feedforward neural network sublayer in the related art.

[0103] exist Figure 5 Based on the structure of the processing layer shown, the data processing process of the first processing layer 1 in the first denoising network is introduced as an example. The data processing processes of other processing layers refer to the data processing process of the first processing layer 1.

[0104] In some specific examples, the aforementioned process of processing the second splicing feature through the first processing layer 1 in the first denoising network to obtain the sample intermediate feature 1 may include: in the first denoising network, processing the second splicing feature through the attention mechanism-based sublayer in the first processing layer 1 to obtain the corresponding processing result, and processing the aforementioned corresponding processing result through the feedforward neural network sublayer in the first processing layer 1 to obtain the sample intermediate feature 1.

[0105] In some possible implementations, the attention-based sublayer in the processing layer of the aforementioned denoising network can be implemented as a self-attention-based sublayer, and its data processing process can refer to the data processing process of the self-attention-based sublayer in the related art.

[0106] In some other possible implementations, the attention-based sublayer in the processing layer of the aforementioned denoising network can also be implemented as a cross-attention-based sublayer. In this case, the aforementioned second sample feature can be embedded as a condition, and the second sample feature and the first sample feature, i.e., the feature sequence of the aforementioned first sample feature, are fused through the cross-attention-based sublayer to obtain the processing result corresponding to the attention-based sublayer, and then the processing result corresponding to the attention-based sublayer is processed through the corresponding feedforward neural network sublayer to obtain the sample intermediate feature. At this time, it can be considered that the sample intermediate feature only includes the sample intermediate sub-feature corresponding to the first sample feature.

[0107] The following introduces the structure of the identity localization layer and its data processing process, taking the first identity localization layer in the first denoising network as an example.

[0108] In some possible examples, the aforementioned first identity localization layer 1 may include a localization sublayer and a fusion sublayer. The aforementioned process of obtaining the output corresponding to the first identity localization layer 1 based on the sample intermediate features 1 and the sample facial visual features corresponding to each sample face through the first identity localization layer 1 may include the following steps:

[0109] Through the positioning sublayer of the first identity positioning layer 1, the sample intermediate feature 1 and the sample facial visual features corresponding to each sample face are used to determine the sample mask corresponding to each sample face, wherein the sample mask is used to indicate the position of the corresponding sample face in the generated video; through the fusion sublayer of the first identity positioning layer 1, the sample mask corresponding to each sample face, the sample facial visual features corresponding to each sample face and the aforementioned sample intermediate feature 1 are used to obtain the output corresponding to the first identity positioning layer 1.

[0110] Subsequently, the output corresponding to the first identity positioning layer 1 is used to add to the sample intermediate feature 1 to obtain the output corresponding to the first denoising network, and the subsequent process is performed to obtain the generated video.

[0111] In some examples, the localization sublayer of the identity localization layer of each denoising network may include a first linear sublayer such as Figure 5 The Linear1 and the second linear sublayer shown are as follows Figure 5 The Linear2 and activation functions shown are as follows Figure 5 As shown in the softmax. It can be understood that Figure 5 In the structure shown, the activation function in the identity localization layer is softmax. In some possible examples, the activation function in the identity localization layer can also be other activation functions.

[0112] The fusion sublayers of the identity localization layer include: sublayers using the cross attention mechanism such as Figure 5 "MaskedCross Attention" shown.

[0113] exist Figure 5 Based on the structure shown, the aforementioned process of determining the sample mask corresponding to each sample face may include steps 21-23:

[0114] In step 21, based on the sample face visual features corresponding to each sample face, the first linear sub-layer "Linear1" in the first identity positioning layer 1 is used to obtain the first linear sub-feature corresponding to each sample face. In this step, for any sample face n, the sample face visual features corresponding to the sample face n are obtained. Input the first linear sublayer "Linear1" to generate the sample face visual features corresponding to the sample face n through the first linear sublayer "Linear1" Perform linear processing to obtain the first linear sub-feature corresponding to the sample face n. Similarly, obtain the first linear sub-feature corresponding to each sample face.

[0115] In step 22 , the sample intermediate feature 1 is used to obtain the second linear sub-feature through the second linear sub-layer “Linear2” in the first identity positioning layer 1 .

[0116] In some examples, when the first sample feature and the second sample feature are concatenated in the channel dimension, the sample intermediate feature 1 may include the sample intermediate sub-feature corresponding to the first sample feature and the sample intermediate sub-feature corresponding to the second sample feature. In step 22, the sample intermediate sub-feature corresponding to the first sample feature may be determined from the sample intermediate features, and then the sample intermediate sub-feature corresponding to the first sample feature may be input into the second linear sub-layer "Linear2" to perform linear processing on the sample intermediate sub-feature corresponding to the first sample feature to obtain the second linear sub-feature.

[0117] Next, in step 23, based on the features obtained by fusing the first linear sub-features corresponding to each sample face with the second linear sub-features, the sample mask corresponding to each sample face is obtained through the activation function "Softmax" in the first identity positioning layer.

[0118] In this step, for any sample face n, the first linear sub-feature and the second linear sub-feature corresponding to the sample face n are fused to obtain the fused feature of the two. Then, the fused feature of the first linear sub-feature and the second linear sub-feature corresponding to the sample face n is activated by the activation function "Softmax" to obtain the sample mask n corresponding to the sample face n. And so on, the sample masks corresponding to each sample face are obtained.

[0119] The aforementioned process of fusing the first linear sub-feature and the second linear sub-feature corresponding to the sample face n may include: multiplying the second linear sub-feature by the first linear sub-feature corresponding to the sample face n.

[0120] For example, the sample mask n corresponding to the sample face n can be expressed by the following formula (2):

[0121]

[0122] in, represents the second linear sub-feature, Indicates the first linear sub-feature corresponding to the sample face n, h indicates the sample intermediate sub-feature corresponding to the first sample feature, Mask n It represents the sample mask n corresponding to the sample face n. It can be understood that the shape of the sample intermediate sub-feature h corresponding to the first sample feature is related to, for example, the same as, the shape corresponding to the first sample feature.

[0123] The sample mask n corresponding to the sample face n may include T-frame sample masks n, and the T-frame sample masks n respectively correspond to each generated video frame in the generated video.

[0124] exist Figure 5 Based on the structure shown, the aforementioned process of obtaining the output corresponding to the first identity localization layer 1 by using the sample mask corresponding to each sample face, the sample face visual features corresponding to each sample face, and the aforementioned sample intermediate feature 1 may include step 31:

[0125] In step 31, by adopting the sub-layer "Masked Cross Attention" of the cross attention mechanism, for each sample face, based on its corresponding sample face visual features and sample intermediate features 1, a cross attention mechanism is adopted to obtain the sample face fusion features corresponding to each sample face.

[0126] Specifically, by adopting the sub-layer "Masked Cross Attention" of the cross-attention mechanism, for each sample face, based on its corresponding sample face visual features and the sample intermediate sub-features corresponding to the aforementioned first sample feature in the sample intermediate feature 1, a cross-attention mechanism is adopted to obtain the sample face fusion features corresponding to each sample face.

[0127] In some specific examples, the sublayer "Masked Cross Attention" using the cross attention mechanism may include a second query Q weight matrix, a second key K weight matrix, and a second value V weight matrix. The aforementioned step 31 may specifically include the following steps 311-312:

[0128] In step 311, based on the second Q weight matrix and the sample intermediate sub-features corresponding to the first sample feature in the sample intermediate feature 1, a second Q matrix is ​​obtained; for any sample face n, based on the second value V weight matrix and the sample face visual features corresponding to the sample face n Get the second V matrix corresponding to the sample face n; based on the second key K weight matrix and the sample face visual features corresponding to the sample face n Get the second K matrix corresponding to the sample face n.

[0129] Next, for any sample face n, based on the second Q matrix and the second K matrix and second V matrix corresponding to sample face n, a preset attention mechanism formula is used to obtain the sample face fusion feature n corresponding to sample face n. The preset attention mechanism formula can be found in the setting of the formula based on the attention mechanism in the relevant art.

[0130] Through the above method, the sample face fusion features corresponding to each sample face are obtained.

[0131] Next, in step 33 , the sample mask and sample face fusion features corresponding to each sample face are used respectively to obtain the sample face mask features corresponding to each sample face, so as to obtain the output corresponding to the first identity localization layer 1 .

[0132] In this step, for any sample face n, the sample mask corresponding to sample face n and the sample face fusion feature n are fused to obtain the sample face mask feature corresponding to sample face n. Similarly, the sample face mask features corresponding to each sample face are obtained; the obtained sample face mask features corresponding to each sample face are used as the output corresponding to the first identity positioning layer 1.

[0133] Illustratively, the aforementioned process of fusing the sample mask corresponding to the sample face n and the sample face fusion feature n may include: multiplying the sample mask corresponding to the sample face n and the sample face fusion feature n.

[0134] In some specific examples, the sample face mask feature corresponding to the sample face n is It can be expressed by the following formula (3):

[0135]

[0136] Among them, Attention() can represent the calculation formula corresponding to the sub-layer using the cross attention mechanism.

[0137] Exemplarily, the output corresponding to the aforementioned first identity positioning layer 1 is used to be added to the sample intermediate feature 1, which may refer to: adding the sample face mask features corresponding to each sample face and the sample intermediate sub-features corresponding to the first sample feature in the sample intermediate feature 1 to obtain an addition result, which is used as the input of the subsequent denoising network.

[0138] According to the above implementation, the input is processed in sequence by several denoising networks to obtain the final denoising result, and then the generated video is decoded based on the final denoising result.

[0139] Subsequently, in step S16, based on the difference between the aforementioned first sample video and the generated video, at least the fusion network is trained.

[0140] Each sample video frame of the first sample video has a one-to-one correspondence with each generated video frame of the generated video. In this step, a specified loss function can be used to construct a model loss based on the differences between each sample video frame and the corresponding generated video frame. Subsequently, at least the parameters of the fusion network are adjusted with the goal of minimizing the model loss. In some examples, the specified loss function can include, but is not limited to, mean square error, mean error, and structural similarity index.

[0141] In some possible examples, the respective parameters of the aforementioned encoding network, extraction network, fusion network, and several denoising networks can be jointly adjusted with the goal of minimizing model loss to train the generative model.

[0142] In some possible examples, DDPM (Denoising Diffusion Probabilistic Models) loss can also be combined to construct a noise loss based on the difference between the added noise features and the predicted noise features of the aforementioned denoising networks in multiple denoising processes. Then, the parameters of the generative model can be adjusted with the goal of minimizing the aforementioned model loss and minimizing the noise loss.

[0143] In some possible examples, in order to reduce the interference of other redundant background information besides identity information, i.e., face information, a face bounding box constraint is also introduced to combine the face bounding box constraint and the difference between each sample video frame and the corresponding generated video frame to construct a model loss. Through this type of model loss, the generated model is encouraged to pay more attention to the face area.

[0144] Each sample video frame of the first sample video corresponds to a face bounding box constraint. The following describes the process for determining the face bounding box constraints corresponding to each sample video frame. The process for determining the face bounding box constraints corresponding to each sample video frame is similar. The following describes the process for determining the face bounding box constraints for any sample video frame A from among the sample video frames.

[0145] The process of determining the face bounding box constraints of the sample video frame A includes the following steps 41-42: In step 41, based on the sample video frame A, a face bounding box image corresponding to the sample video frame A is obtained, wherein the face bounding box image is a binary image, wherein the pixel values ​​corresponding to the face area in the face bounding box image are first values, and the pixel values ​​of other areas are second values. For example, the first value can be 1 and the second value can be 0.

[0146] In step 42, the face bounding box corresponding to the sample video frame A is downsampled to obtain a face bounding box having the same width and height as the face coding features of each face, and the face bounding box having the same width and height as the face coding features of each face is used as the face bounding box constraint of the sample video frame A.

[0147] For example, the face bounding box constraint and the difference between each sample video frame and the corresponding generated video frame are combined to construct the model loss, which can be expressed by the following formula (4):

[0148] Le= (α·B)⊙L1; (4)

[0149] Among them, Le represents the model loss, L1 represents the loss constructed based on the difference between each sample video frame and the corresponding generated video frame, B represents the face bounding box constraint corresponding to each sample video frame, and α is a preset hyperparameter used to control the probability of applying the face bounding box constraint.

[0150] It can be understood that in the above-mentioned process of training the generative model, only a first sample is used as an example to illustrate the process of training the generative model. In some examples, the generative model can be trained by iterating the above-mentioned training process multiple times until the generative model reaches the first convergence condition. At this time, the generative model training can be considered completed.

[0151] The first convergence condition may include but is not limited to: the number of training times reaches a first number threshold, the value of the model loss is less than a first loss threshold, and the training time exceeds a preset time.

[0152] In some possible examples, to ensure the video generation capability and training efficiency of the trained generative model, a two-stage training process can be employed. For example, in the first stage of training, the parameters of the encoding network, extraction network, fusion network, and several denoising networks are adjusted with the goal of minimizing model loss until the generative model meets the first convergence condition. The generative model can then be trained in the second stage.

[0153] Among them, during the second stage training process, only the parameters of each identity localization layer in each denoising network in the generation model can be adjusted. In the process of adjusting the parameters of the identity localization layer, the parameters of the identity localization layer are adjusted with the goal of reducing the difference between the sample mask of each sample face determined by the identity localization layer for its input and the corresponding label mask serving as the label.

[0154] In the second phase of training, the samples used to train the generative model can be the same as or different from those used in the first phase of training. Specifically, the second phase of training includes the following steps:

[0155] Step 51, obtain a second sample, wherein the second sample includes: multiple training face images containing different training faces, instruction text for indicating the generation of video content, and a second sample video, the multiple training face images are face images captured from the second sample video, and the instruction text corresponds to the video content of the second sample video.

[0156] In step 52, a first training feature is obtained based on each training face image through a coding network and coding and denoising processing, wherein the first training feature is determined based on the training noise feature and the training coding feature of each training face in each training face image. The sample noise feature may be randomly generated Gaussian noise.

[0157] In step 53, feature extraction is performed on each training face image through the aforementioned extraction network to determine the training face visual features corresponding to each training face.

[0158] In step 54, the instruction text features corresponding to the instruction text and the visual features of each training face are fused through the aforementioned fusion network to obtain a second training feature.

[0159] The implementation principles of steps 51-54 are similar to those of the aforementioned steps S11-S14. For the implementation process thereof, reference may be made to the implementation process of the aforementioned steps S11-S14.

[0160] In step 55, the input of the denoising network i is processed by the processing layer thereof to obtain a training intermediate feature, wherein the input of the processing layer of the denoising network i is the concatenated first training feature and the second training feature, or the output corresponding to the previous denoising network of the denoising network i.

[0161] In step 56, the training mask corresponding to each training face is determined by using the training intermediate features and the training face visual features corresponding to each training face through the positioning sublayer in the identity positioning layer of the denoising network i, wherein the training mask is used to indicate the position of the corresponding training face in the generated video.

[0162] In step 57, the fusion sublayer of the identity localization layer is used to obtain the output result corresponding to the identity localization layer by using the training masks corresponding to each training face, the training face visual features corresponding to each training face, and the aforementioned training intermediate features.

[0163] In step 58, the output corresponding to the identity positioning layer and the training intermediate features are used to obtain the output corresponding to the denoising network i, which is used as the input for the processing of the denoising network i+1. The process returns to step 55 until the output corresponding to the last denoising network is obtained. The output corresponding to the last denoising network includes the predicted noise features y predicted by several denoising networks.

[0164] In step 59 , a denoising result corresponding to this denoising process is obtained based on the predicted noise feature y and the aforementioned spliced ​​first training feature and second training feature.

[0165] In step 510, based on the denoising result corresponding to this denoising process, a current video containing the aforementioned training faces is obtained through a decoding network.

[0166] The implementation process of the aforementioned steps 55-510 can refer to the implementation process of the corresponding steps in the aforementioned first stage training process, and will not be repeated here.

[0167] Subsequently, in step 511, the label mask corresponding to each training face is obtained. In this step, the label mask corresponding to any training face v can be annotated by a specific program or manually based on the area where the training face v is located in each second sample video frame in the second sample video. The label mask corresponding to the training face v includes the mask of the training face n corresponding to each second sample video frame. Exemplarily, in the mask of the training face v corresponding to each second sample video frame, the value at the position corresponding to the training face v is a third numerical value, and the value at the position corresponding to other background areas is a fourth numerical value. Exemplarily, the third numerical value is 1, and the fourth numerical value is 0.

[0168] Next, in step 512, at least each identity localization layer is trained based on the difference between the label mask and the training mask corresponding to each training face. In this step, for any training face v, the face mask loss corresponding to training face v is determined based on the difference between the label mask and the training mask corresponding to training face v. Similarly, the face mask loss corresponding to each training face is determined. With the goal of minimizing the face mask loss corresponding to each training face, the parameters of each identity localization layer are adjusted to train each identity localization layer until the generative model reaches the second convergence condition, indicating that the second stage of training of the generative model is complete.

[0169] The trained generative model can be used to provide users with video generation services for multiple identities, i.e., multiple different faces, at the same time.

[0170] The second convergence condition may include but is not limited to: the number of times the parameters of the identity positioning layer are adjusted reaches a second number threshold, the value of the face mask loss corresponding to each sample face is less than the second loss threshold or the comprehensive value of the face mask loss corresponding to each sample face is less than a third loss threshold, and the time used to adjust the identity positioning layer exceeds the specified time.

[0171] In some possible examples, during the second stage training process, the difference between the current video and the second sample video, as well as the difference between the label mask and the training mask corresponding to each training face can be combined to at least jointly train each identity localization layer.

[0172] In the above embodiment, a new generative model structure and its training method are provided to train a generative model capable of generating videos for multiple character identities, i.e., multiple different faces, and to ensure the uniqueness of each face in the video and reduce the confusion of features between the faces.

[0173] In this generative model, the encoding network splices the sample face encoding features and noise features corresponding to each sample face in the channel dimension. This decouples the encoding features of each sample face, ensuring that they contribute equally to the video generation process. This avoids spatial positional restrictions on each sample face in the generated video and promotes greater content richness and diversity in the generated video. This splicing method enhances the generative model's ability to generate dynamic and visually coherent results, resulting in greater visual coherence of dynamic faces in the generated video.

[0174] A fusion network is constructed to establish semantic associations between the sample facial visual features corresponding to each sample face and the text features. This allows alignment and fusion of the sample facial visual features corresponding to each sample face with the text features, providing joint guidance for generating identity consistency for each face. By aligning and fusing the sample facial visual features corresponding to each sample face with the text features through the fusion network, the semantic coherence of the text and the identity consistency of each face in the generated video can be guaranteed even if the sample text changes.

[0175] An identity localization layer is incorporated into each denoising network to ensure that each identity is spatially localized, preventing overlap and maintaining the uniqueness of each identity. Furthermore, by incorporating an identity localization layer into each denoising network, the localization of facial features can be dynamically adjusted during the video generation process. This integrated approach allows the generative model to adaptively refine the spatial distribution of identity features, i.e., facial features, at different stages of the video generation process, ensuring the precise and coherent placement of multiple identities, i.e., faces, in the final output video.

[0176] Figure 6 The flowchart of the video generation method in one embodiment of the present specification is shown. The method is executed by an electronic device, which can be implemented by any device, equipment, platform, equipment cluster, etc. with computing and processing capabilities. In the video generation process, Figure 6 As shown, the method includes the following steps S610-S650:

[0177] In step S610 , a plurality of facial images including different human faces and a first text for indicating generation of video content are acquired.

[0178] The single facial image may include a single face. The multiple facial images may be any type of facial image input by the user. The first text may be text input by the user to instruct the generation of video content. The first text may at least include text indicating the need to generate video content containing each face in the multiple facial images.

[0179] In step S620, based on each facial image, a first feature is obtained through coding and noise processing, wherein the first feature is determined based on the noise feature and the facial coding feature of each face in each facial image.

[0180] In step S620, the encoding network of the aforementioned generative model can be used to generate the first feature based on each facial image through encoding and denoising. In some possible examples, step S620 may include: obtaining the corresponding facial encoding features of each face through the encoding layer of the encoding network based on each facial image; concatenating the facial encoding features and the noise features through the concatenation and denoising layer of the encoding network to obtain a first concatenated feature; and obtaining the first feature based on the first concatenated feature through the mapping layer of the encoding network. The specific implementation process can be found in the implementation process of step S12 above and is not further described here.

[0181] In some other examples, in step S620, each facial image can be encoded separately based on other trained encoders, and then the encoding results and noise features corresponding to each facial image can be spliced ​​in the channel dimension, and then the trained mapping layer can be used to map the spliced ​​results into a result of a specified shape.

[0182] In step S630, feature extraction is performed on each face image to determine the facial visual features corresponding to each face.

[0183] In some possible examples, the extraction network of the aforementioned generative model can be used to perform feature extraction on each facial image to determine the facial visual features corresponding to each face.

[0184] In some specific examples, step S630 may include: for each facial image, using the first extraction sub-network of the extraction network to obtain the first visual features corresponding to each face; for each facial image, using the second extraction sub-network of the extraction network to obtain the second visual features corresponding to each face; and based on the first and second visual features corresponding to each face, using the integration sub-network of the extraction network to obtain the facial visual features corresponding to each face. The specific implementation process can be found in the specific implementation process of the aforementioned step S13 and is not further described here.

[0185] In step S640, the text feature corresponding to the first text and the visual features of each face are fused to obtain a second feature.

[0186] In some possible examples, the second feature can be obtained by fusing the text features corresponding to the first text and the visual features of each face through the fusion network of the aforementioned generative model.

[0187] In some specific examples, step S640 may include: obtaining the second feature based on the text feature corresponding to the first text and the visual features of each face through a fusion network, wherein the fusion network is a network based on a cross-attention mechanism. The specific implementation process of this process can be found in the specific implementation process of step S14 above and is not repeated here.

[0188] In step S650, a target video containing each face is generated based on the first feature, the second feature, and the facial visual features corresponding to each face, wherein the facial visual features corresponding to each face are used to determine the position of each face in the target video.

[0189] In some possible examples, a target video containing each face can be generated based on the first feature, the second feature, and the facial visual features corresponding to each face using the denoising networks of the aforementioned generative model. The specific implementation process can be found in the specific implementation process of step S15 above and will not be described in detail here.

[0190] In some specific examples, step S650 may include the following steps 61-62:

[0191] In step 61 , based on the guidance of the second feature, the first feature is processed to obtain an intermediate feature.

[0192] In some specific examples, at least one of the plurality of denoising networks includes a processing layer, hereinafter referred to as a first processing layer. Accordingly, step 61 may include: obtaining the intermediate feature based on the first feature and the second feature through an attention-based sublayer and a feedforward neural network sublayer of the first processing layer.

[0193] In step 62, the target video is generated using the intermediate features and the facial visual features corresponding to each face.

[0194] In some specific examples, step 62 may include the following steps 621-622:

[0195] In step 621, the first mask corresponding to each face is determined using the intermediate features and the facial visual features corresponding to each face, wherein the first mask is used to indicate the position of the corresponding face in the generated target video; in step 622, the target video is generated using the first mask corresponding to each face, the facial visual features of each face, and the intermediate features.

[0196] In some specific examples, at least one of the multiple denoising networks includes an identity localization layer corresponding to the aforementioned first processing layer, hereinafter referred to as the first identity localization layer. Accordingly, step 621 may include: obtaining first sub-features corresponding to each face using a first linear sub-layer in the first identity localization layer based on the facial visual features corresponding to each face; obtaining second sub-features using a second linear sub-layer in the first identity localization layer using the intermediate features; and obtaining first masks corresponding to each face using an activation function in the first identity localization layer based on a fusion of the first sub-features corresponding to each face and the second sub-features.

[0197] In some specific examples, step 622 may include: for each face, based on its corresponding facial visual features and the intermediate features, using a cross-attention mechanism to obtain the facial fusion features corresponding to each face; using the first mask and the face fusion features corresponding to each face respectively to obtain the facial mask features corresponding to each face; based on the facial mask features corresponding to each face and the intermediate features, generating the target video.

[0198] To ensure that each identity feature, that is, the feature of each face, is local in space, to prevent overlap between identity features, and to maintain the uniqueness of each identity, that is, face.

[0199] In this embodiment, videos can be generated for multiple person identities, that is, multiple different faces, while ensuring the uniqueness of each face in the video.

[0200] In some possible examples, the aforementioned multiple facial images are facial images extracted from a first sample video; the first text corresponds to the content of the first sample video; and the method further includes:

[0201] Based on the difference between the first sample video and the target video, at least the fusion network is trained. The implementation process of this step can refer to the training process of the aforementioned generative model, which will not be repeated here.

[0202] In some possible examples, the process further includes: obtaining a label mask corresponding to each face; and training at least the first identity localization layer based on the difference between the label mask corresponding to each face and the first mask. The implementation process of this step can be found in the aforementioned generative model training process and will not be further described here.

[0203] In the above process, a video generation method for multiple character identities, i.e., multiple different faces, is provided to achieve the generation of a video containing multiple faces, wherein the text features corresponding to the first text and the visual features of each face are fused to obtain a second feature, so that the second feature contains the aligned and fused text features and the visual features of each face. In this way, even if the text changes, the text semantics consistency of each face in the generated video and the identity consistency between each face can be guaranteed, and the uniqueness of each face in the video and the text semantics consistency of each face can be guaranteed.

[0204] The foregoing description describes specific embodiments of the present disclosure, and other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in a different order than that described in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily need to be performed in the specific order shown or in a sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0205] Corresponding to the above method embodiment, this embodiment of the specification provides a video generating device 700, whose schematic block diagram is as follows: Figure 7 Shown, including:

[0206] A first acquisition module 710 is configured to acquire a plurality of facial images containing different human faces and a first text for instructing to generate video content;

[0207] A first obtaining module 720 is configured to obtain a first feature based on each facial image through coding and noise processing, wherein the first feature is determined based on the noise feature and the facial coding feature of each face in each facial image;

[0208] The first determination module 730 is configured to perform feature extraction on each face image to determine the facial visual features corresponding to each face;

[0209] A first fusion module 740 is configured to fuse the text features corresponding to the first text and the visual features of each face to obtain a second feature;

[0210] The generation module 750 is configured to generate a target video containing each of the faces based on the first feature, the second feature and the facial visual features corresponding to each face, wherein the facial visual features corresponding to each face are used to determine the position of each face in the target video.

[0211] In some possible examples, the generation module 750 includes: a first obtaining unit (not shown in the figure), configured to process the first feature based on the guidance of the second feature to obtain an intermediate feature; a first generating unit (not shown in the figure), configured to use the intermediate feature and the facial visual features corresponding to each face to generate the target video.

[0212] In some possible examples, the first generating unit includes: a first determining submodule (not shown in the figure), configured to determine a first mask corresponding to each face using the intermediate features and the facial visual features corresponding to each face, wherein the first mask is used to indicate the position of the corresponding face in the generated target video;

[0213] The first generation submodule (not shown in the figure) is configured to generate the target video using the first mask corresponding to each face, the visual features of each face and the intermediate features.

[0214] In some possible examples, the first determination submodule is specifically configured to obtain, based on the facial visual features corresponding to each face, the first sub-feature corresponding to each face through the first linear sub-layer in the first identity localization layer;

[0215] Using the intermediate feature, through the second linear sub-layer in the first identity positioning layer, a second sub-feature is obtained;

[0216] Based on the features obtained by fusing the first sub-features corresponding to each face with the second sub-features, a first mask corresponding to each face is obtained through the activation function in the first identity localization layer.

[0217] In some possible examples, the first generation submodule is specifically configured to, for each face, employ a cross attention mechanism based on its corresponding facial visual features and the intermediate features to obtain a facial fusion feature corresponding to each face;

[0218] The first mask and face fusion features corresponding to each face are used respectively to obtain the face mask features corresponding to each face;

[0219] The target video is generated based on the facial mask features corresponding to each face and the intermediate features.

[0220] In some possible examples, the first obtaining module 720 is specifically configured to obtain the facial coding features corresponding to each face based on each facial image through the encoding layer of the encoding network; to obtain the first splicing feature by splicing the facial coding features and the noise features through the splicing and noise adding layer; and to obtain the first feature based on the first splicing feature through the mapping layer of the encoding network.

[0221] In some possible examples, the first determining module 730 is specifically configured to obtain, for each face image, a first visual feature corresponding to each face through a first extraction sub-network of the extraction network;

[0222] For each face image, obtaining a second visual feature corresponding to each face through the second extraction sub-network of the extraction network;

[0223] Based on the first visual features and the second visual features corresponding to each face, the facial visual features corresponding to each face are obtained through the integration sub-network of the extraction network.

[0224] In some possible examples, the first fusion module 740 is specifically configured to obtain the second feature through a fusion network based on the text features corresponding to the first text and the visual features of each face, wherein the fusion network is a network based on a cross-attention mechanism.

[0225] In some possible examples, the first obtaining unit is specifically configured to obtain the intermediate feature based on the first feature and the second feature through the attention mechanism-based sublayer and the feedforward neural network sublayer of the first processing layer.

[0226] In some possible examples, the multiple facial images are facial images extracted from a first sample video; the first text corresponds to the content of the first sample video; and the apparatus further includes:

[0227] The first training module (not shown in the figure) is configured to train at least the fusion network based on the difference between the first sample video and the target video.

[0228] In some possible examples, it also includes: a second acquisition module (not shown in the figure), configured to obtain the label mask corresponding to each face; a second training module (not shown in the figure), configured to train at least the first identity positioning layer based on the difference between the label mask corresponding to each face and the first mask.

[0229] The above-mentioned device embodiments correspond to the method embodiments. For detailed descriptions, please refer to the description of the method embodiments, which will not be repeated here. The device embodiments are obtained based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments. For detailed descriptions, please refer to the corresponding method embodiments.

[0230] The embodiments of this specification also provide a computer-readable storage medium having a computer program stored thereon. When the computer program is executed in a computer, the computer is caused to execute the video generation method provided in this specification.

[0231] An embodiment of the present specification further provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the video generation method provided in the present specification is implemented.

[0232] The various embodiments in this specification are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other, and each embodiment focuses on the differences between the other embodiments. In particular, the storage medium and computing device embodiments are described briefly because they are generally similar to the method embodiments. For relevant portions, refer to the description of the method embodiments.

[0233] Those skilled in the art will appreciate that, in one or more of the above examples, the functions described in the embodiments of the present invention may be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0234] The specific implementation methods described above further illustrate the purpose, technical solutions, and beneficial effects of the embodiments of the present invention. It should be understood that the above description is only a specific implementation method of the embodiments of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solutions of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A video generation method, comprising: Acquire a plurality of facial images containing different human faces and a first text for instructing to generate video content; Based on each facial image, obtaining a first feature through coding and noise processing, wherein the first feature is determined based on the noise feature and the facial coding feature of each face in each facial image; Perform feature extraction on each face image respectively to determine the facial visual features corresponding to each face; Fusing the text features corresponding to the first text and the visual features of each face to obtain a second feature; Based on the first feature, the second feature and the facial visual features corresponding to each face, a target video containing each face is generated, wherein the facial visual features corresponding to each face are used to determine the position of each face in the target video.

2. The method according to claim 1, wherein Generating a target video containing each of the faces includes: Based on the guidance of the second feature, the first feature is processed to obtain an intermediate feature; The target video is generated by utilizing the intermediate features and the facial visual features corresponding to each face.

3. The method according to claim 2, wherein: The generating the target video by using the intermediate features and the facial visual features corresponding to each face includes: Determining a first mask corresponding to each face using the intermediate features and the facial visual features corresponding to each face, wherein the first mask is used to indicate a position of the corresponding face in the generated target video; The target video is generated using the first mask corresponding to each face, the visual features of each face, and the intermediate features.

4. The method according to claim 3, wherein: The determining of the first mask corresponding to each face includes: Based on the facial visual features corresponding to each face, the first sub-feature corresponding to each face is obtained through the first linear sub-layer in the first identity localization layer; Using the intermediate feature, through the second linear sub-layer in the first identity positioning layer, a second sub-feature is obtained; Based on the features obtained by fusing the first sub-features corresponding to each face with the second sub-features, a first mask corresponding to each face is obtained through the activation function in the first identity localization layer.

5. The method according to claim 4, wherein: The generating of the target video comprises: For each face, based on its corresponding facial visual features and the intermediate features, a cross-attention mechanism is used to obtain the corresponding facial fusion features of each face; The first mask and face fusion features corresponding to each face are used respectively to obtain the face mask features corresponding to each face; The target video is generated based on the facial mask features corresponding to each face and the intermediate features.

6. The method of claim 2, wherein: The obtaining of the intermediate features comprises: Based on the first feature and the second feature, the intermediate feature is obtained through the attention mechanism-based sublayer and the feedforward neural network sublayer of the first processing layer.

7. The method according to any one of claims 1 to 6, wherein: The obtaining of the first feature includes: Based on each face image, the facial coding features corresponding to each face are obtained through the encoding layer of the encoding network; By splicing the noise layer, the face coding features and noise features are spliced ​​together to obtain the first splicing feature; The first feature is obtained through a mapping layer of the encoding network based on the first splicing feature.

8. The method according to any one of claims 1 to 6, wherein: Determining the facial visual features corresponding to each face includes: For each face image, the first visual feature corresponding to each face is obtained through the first extraction sub-network of the extraction network; For each face image, obtaining a second visual feature corresponding to each face through the second extraction sub-network of the extraction network; Based on the first visual features and the second visual features corresponding to each face, the facial visual features corresponding to each face are obtained through the integration sub-network of the extraction network.

9. The method according to any one of claims 1 to 6, wherein: The obtaining of the second feature includes: Based on the text features corresponding to the first text and the visual features of each face, the second features are obtained through a fusion network, wherein the fusion network is a network based on a cross-attention mechanism.

10. The method of claim 9, wherein: The multiple facial images are facial images extracted from a first sample video; the first text corresponds to the content of the first sample video; The method further comprises: At least the fusion network is trained based on the difference between the first sample video and the target video.

11. The method of claim 4, further comprising: Get the label mask corresponding to each face; At least the first identity localization layer is trained based on the difference between the label mask corresponding to each face and the first mask.

12. A video generating device, comprising: A first acquisition module is configured to acquire a plurality of facial images containing different human faces and a first text for instructing to generate video content; a first obtaining module configured to obtain a first feature based on each facial image through coding and noise processing, wherein the first feature is determined based on the noise feature and the facial coding feature of each face in each facial image; A first determination module is configured to extract features from each face image and determine the facial visual features corresponding to each face; a first fusion module configured to fuse text features corresponding to the first text and visual features of each face to obtain a second feature; A generation module is configured to generate a target video containing each of the faces based on the first feature, the second feature and the facial visual features corresponding to each face, wherein the facial visual features corresponding to each face are used to determine the position of each face in the target video.

13. A computing device comprising a memory and a processor, wherein: The memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 11 is implemented.

Images