Image generation and model training method, device, equipment and storage medium
By employing an encoder-decoder model with masking and self- and joint attention mechanisms, the problem of poor quality in speech-driven lip-shape image generation was solved, achieving high-quality image generation results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING BAIDU NETCOM SCI & TECH CO LTD
- Filing Date
- 2022-10-14
- Publication Date
- 2026-08-04
AI Technical Summary
In existing technologies, the image generation of speech-driven lip shapes has poor quality and unsatisfactory results, and cannot effectively integrate speech and image features.
A combined model of masking, encoder, and decoder is used to extract image and speech features respectively, and then fusion processing is performed through self-attention and common attention mechanisms to generate high-quality speech-driven lip-shape images.
It improves the quality of image generation, enabling better expression of image and speech information and generating realistic lip-sync animation.
Smart Images
Figure CN115690238B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of artificial intelligence technology, specifically to the fields of computer vision, augmented reality, virtual reality, deep learning, etc., and can be applied to scenarios such as metaverse and virtual digital humans. In particular, it relates to an image generation and model training method, apparatus, device, and storage medium. Background Technology
[0002] Virtual digital humans are a product of the integration of information science and life science, utilizing information science methods to virtually simulate the human body at different levels of form and function. With the development of virtual digital human technology, the virtual images of virtual digital humans are becoming increasingly realistic. Virtual digital humans can be applied to scenarios such as voice broadcasting.
[0003] Voice-driven lip-syncing refers to the process of controlling the lips of a virtual avatar based on the input speech while keeping other facial information unchanged. Summary of the Invention
[0004] This disclosure provides an image generation and model training method, apparatus, device, and storage medium.
[0005] According to one aspect of this disclosure, an image generation method is provided, comprising: masking a lip region of a first image to obtain a second image; encoding the first image to obtain first image features; encoding the second image to obtain second image features; encoding speech to obtain speech features, the speech being used to drive the lip region of the first image; obtaining fusion features based on the first image features, the second image features, and the speech features; and decoding the fusion features to generate a target image, the target image being an image obtained after the speech drives the lip region of the first image.
[0006] According to another aspect of this disclosure, a model training method is provided, the model comprising: a first encoder, a second encoder, a third encoder, a fusion encoder, and a decoder, the method comprising: acquiring training samples, the training samples comprising: a real image, a reference image, and speech, the speech being used to drive the lip region of the real image sample; masking the lip region of the real image to obtain a masked image; encoding the reference image using the first encoder to obtain a first image feature; encoding the masked image using the second encoder to obtain a second image feature; encoding the speech sample using the third encoder to obtain speech features; obtaining fusion features using the fusion encoder based on the first image feature, the second image feature, and the speech feature; decoding the fusion features using the decoder to generate a predicted image; constructing a total loss function based on the real image and the predicted image; and adjusting the model parameters of the first encoder, the second encoder, the third encoder, the fusion encoder, and the decoder based on the total loss function.
[0007] According to another aspect of this disclosure, an image generation apparatus is provided, comprising: a masking module for masking a lip region of a first image to obtain a second image; a first encoding module for encoding the first image to obtain first image features; a second encoding module for encoding the second image to obtain second image features; a third encoding module for encoding speech to obtain speech features, the speech being used to drive the lip region of the first image; a fusion encoding module for obtaining fusion features based on the first image features, the second image features, and the speech features; and a decoding module for decoding the fusion features to generate a target image, the target image being an image obtained after the speech drives the lip region of the first image.
[0008] According to another aspect of this disclosure, a model training apparatus is provided, the model comprising: a first encoder, a second encoder, a third encoder, a fusion encoder, and a decoder. The apparatus includes: an acquisition module for acquiring training samples, the training samples comprising: a real image, a reference image, and speech, the speech being used to drive the lip region of the real image sample; a masking module for masking the lip region of the real image to obtain a masked image; a first encoding module for encoding the reference image using the first encoder to obtain first image features; and a second encoding module for encoding the masked image using the second encoder. The system comprises: a first image encoder, a second image encoder, a third encoder, a third encoder, a third encoder, a third encoder, a third encoder, and ...
[0009] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to said at least one processor; wherein the memory stores instructions executable by said at least one processor, said instructions being executed by said at least one processor to enable said at least one processor to perform the method as described in any of the foregoing aspects.
[0010] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are configured to cause the computer to perform the method according to any of the preceding aspects.
[0011] According to another aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method according to any of the preceding aspects.
[0012] According to the technical solution disclosed herein, image quality can be improved.
[0013] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description
[0014] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:
[0015] Figure 1 This is a schematic diagram based on the first embodiment of the present disclosure;
[0016] Figure 2 This is a schematic diagram illustrating an application scenario applicable to the embodiments of this disclosure;
[0017] Figure 3 This is a schematic diagram of the overall architecture for generating target images based on a model, according to an embodiment of this disclosure.
[0018] Figure 4 This is a schematic diagram according to the second embodiment of the present disclosure;
[0019] Figure 5 This is a schematic diagram of the fourth encoder and the fifth encoder provided according to embodiments of this disclosure;
[0020] Figure 6 This is a schematic diagram of B-TB in the fourth encoder provided according to an embodiment of this disclosure;
[0021] Figure 7 This is a schematic diagram of the CCF-TB in the fifth encoder provided according to an embodiment of this disclosure;
[0022] Figure 8 This is a schematic diagram according to the third embodiment of the present disclosure;
[0023] Figure 9 This is a schematic diagram of the overall architecture of the model training process provided in the embodiments of this disclosure;
[0024] Figure 10 This is a schematic diagram according to the fourth embodiment of the present disclosure;
[0025] Figure 11 This is a schematic diagram according to the fifth embodiment of the present disclosure;
[0026] Figure 12 This is a schematic diagram of an electronic device used to implement the image generation method or model training method of the embodiments of this disclosure. Detailed Implementation
[0027] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.
[0028] In related technologies, for speech-driven lip-syncing, audio features and image features are usually concatenated and then input into a convolutional neural network to generate the final image.
[0029] However, this simple stitching and convolution process results in poor quality of the final image and unsatisfactory results.
[0030] Figure 1 Based on a schematic diagram of the first embodiment of this disclosure, this embodiment provides an image generation method, the method comprising:
[0031] 101. Mask the lip region of the first image to obtain the second image.
[0032] 102. Encode the first image to obtain first image features.
[0033] 103. Encode the second image to obtain the second image features.
[0034] 104. Encode the speech to obtain speech features, the speech being used to drive the lip region of the first image.
[0035] 105. Based on the first image features, the second image features, and the speech features, obtain fused features.
[0036] 106. Decode the fusion features to generate a target image, wherein the target image is the image obtained after the speech drives the lip region of the first image.
[0037] The first image refers to the image to be driven by voice, which is usually a human face image. Understandably, if the virtual avatar is an image other than a human, the first image can also be a face image of another image, such as the face image of an animal.
[0038] The image obtained by masking the lip region of the first image can be called the second image.
[0039] In this case, a preset area of the first image can be used as the lip area. For example, the area at the bottom of the first image with a preset size, such as w*h, can be used as the lip area, where w and h are preset values.
[0040] Masking the lip region can be done by setting the pixel values of the lip region to the pixel values corresponding to black pixels, for example, by setting all pixel values of the lip region to 0.
[0041] After obtaining the first image and the second image, the corresponding image features can be extracted respectively, which are called the first image features and the second image features respectively.
[0042] The first and second images can use the same or different encoders to extract corresponding image features. The encoder can be a pre-trained deep neural network model.
[0043] Speech refers to the speech of the first image to be driven. Speech can be input into the encoder corresponding to the speech, and the corresponding output is the speech features of the speech. The encoder that encodes the speech can also be a pre-trained deep neural network model.
[0044] 102, 103, and 104 have no timing restrictions.
[0045] After obtaining the first image features, the second image features, and the speech features, they can be fused. The feature resulting from the fusion of these three features can be called the fused feature. Fusion processing refers to combining the information from the three features. This fusion processing can be performed using a deep neural network. The specific fusion processing process can be found in subsequent embodiments.
[0046] After obtaining the fusion features, a decoder can be used to decode the fusion features. The output of the decoder is the target image, which is the image obtained after the speech drives the first image, i.e., the final image. The decoder can also be a pre-trained deep neural network model.
[0047] In this embodiment, the target image is generated based on fusion features. Fusion features are obtained by fusing image features and speech features, thus better representing image and speech information and improving the quality of the generated target image. Furthermore, the image features include first image features and second image features, which can better represent image information and further improve the quality of the target image.
[0048] To better understand the embodiments of this disclosure, the application scenarios to which the embodiments of this disclosure are applicable are described below.
[0049] like Figure 2As shown, the user can input the image to be driven and the corresponding voice on the client. The client sends the image and its corresponding voice to the server, which then performs voice-driven lip-syncing based on the image and its corresponding voice to generate the target image. The server can then return the target image to the client. The client can be deployed on a user terminal 201, which can be a personal computer (PC), laptop, mobile device (such as a mobile phone), etc. The server can be deployed on a server 202, which can be a local server or a cloud server, and can be a single server or a server cluster. Furthermore, while the above example uses server-side voice-driven lip-syncing, if the user terminal where the client is located has the corresponding capabilities, voice-driven lip-syncing can be performed locally on the user terminal.
[0050] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0051] like Figure 2 As shown, after the server obtains the image and voice, it can use voice-driven lip-sync technology to process them and generate the target image after voice-driven image processing.
[0052] Voice-driven lip-syncing technology can be applied to video scenarios. For a single frame of the original image, voice at different times can be used to drive the original image to generate the target image at the corresponding time, and then the video can be generated based on the target images at different times.
[0053] Voice-driven lip-syncing technology can be implemented using deep neural network models, such as... Figure 3 As shown, the overall architecture for generating target images based on a model can include a masking module 301 and a generator 302. The first image refers to the image to be driven by speech, and speech is used to drive the lip region of the first image. After speech-driven lip-shape processing, the corresponding target image can be generated.
[0054] The mask model 301 is used to mask the first image to obtain the second image.
[0055] Generator 302 includes: a first encoder, a second encoder, a third encoder, a fourth encoder, a fifth encoder, and a decoder.
[0056] The first encoder is used to encode the first image to obtain the first image features.
[0057] The second encoder is used to encode the second image to obtain the features of the second image.
[0058] The third encoder is used to encode the speech and obtain speech features.
[0059] Generally speaking, in order to reduce the number of parameters, the first encoder and the second encoder can be the same encoder, or an encoder that shares parameters.
[0060] The fourth encoder is used to encode the first image features to obtain the third image features.
[0061] The fourth encoder is a self-attention encoder, which can be selected as an encoder for a transformer model.
[0062] The fifth encoder is used to encode the second image features, the third image features, and the speech features to obtain fused features.
[0063] The fifth encoder is an encoder based on the mutual attention mechanism and fusion processing.
[0064] The decoder is used to decode the fused features to generate the target image.
[0065] In conjunction with the above application scenarios, this disclosure also provides an image generation method.
[0066] Figure 4 Based on a schematic diagram of a second embodiment of this disclosure, this embodiment provides an image generation method, the method comprising:
[0067] 401. Mask the lip region of the first image to obtain the second image.
[0068] The first image is the image to be voice-driven, which users can select according to their needs.
[0069] For example, select a face image of a virtual character as the first image.
[0070] In addition, after selecting a face image, the image can be preprocessed and used as the first image.
[0071] The preprocessing may include: using the center points of the two eyes in the selected face image as the center points of the preprocessed image, and selecting an image of a preset size (such as 256*256) as the preprocessed image.
[0072] The image obtained by masking the lip region of the first image can be called the second image.
[0073] In this case, a preset area of the first image can be used as the lip area. For example, the area at the bottom of the first image with a preset size, such as w*h, can be used as the lip area, where w and h are preset values.
[0074] Masking the lip region can be done by setting the pixel values of the lip region to the pixel values corresponding to black pixels, for example, by setting all pixel values of the lip region to 0.
[0075] 402. The first image is encoded using a first encoder to obtain first image features.
[0076] 403. Use a second encoder to encode the second image to obtain the second image features.
[0077] The first encoder and the second encoder are both encoders used to extract image features, and both can be convolutional neural network models.
[0078] To reduce the number of parameters, the first encoder and the second encoder can be the same encoder. Of course, the first encoder and the second encoder can also be different encoders.
[0079] 404. Use a third encoder to encode the speech to obtain speech features.
[0080] The third encoder is used to extract speech features and can be a ResNet network model.
[0081] Among them, 402 to 404 have no timing restrictions.
[0082] In this embodiment, the first image features, the second image features, and the speech features can be obtained by the first encoder, the second encoder, and the third encoder, respectively, so as to generate a target image based on these features.
[0083] 405. Use a fourth encoder to encode the first image features to obtain the third image features.
[0084] 406. Using a fifth encoder, the second image feature, the third image feature, and the speech feature are encoded to obtain fused features.
[0085] In this embodiment, the first image features are encoded to obtain the third image features, and then the third image features are fused with other features to better represent the features corresponding to the first image, thereby improving the quality of the target image corresponding to the first image.
[0086] For the fourth encoder, it may include:
[0087] The first image features are used as input to the first encoding module included in the fourth encoder;
[0088] The inputs of each encoding module are encoded using at least one encoding module included in the fourth encoder to obtain the outputs of each encoding module, and the outputs of each encoding module are used as the inputs of the next encoding module; and,
[0089] The outputs of each encoding module are used as a set of third image features.
[0090] The fourth encoder can be a self-attention mechanism-based encoder, specifically a transformer network encoder. Correspondingly, its encoding modules can be called Basic Transformer Blocks (B-TB).
[0091] like Figure 5 As shown, the fourth encoder includes M (M is a positive integer, which can be set) B-TBs. The output of the first B-TB serves as the input of the second B-TB, the output of the second B-TB serves as the input of the third B-TB, and so on. The input of the first B-TB is the first image feature output by the first encoder.
[0092] Each B-TB can include: convolutional layers and N (N is a positive integer, which can be set) sub-block layers. For each B-TB, the input of the first sub-block layer is the output of the corresponding B-TB's convolutional layer, the input of the second sub-block layer is the output of the first sub-block layer, the input of the third sub-block layer is the output of the second sub-block layer, and so on. The output of the last sub-block layer of each B-TB is the output of that B-TB.
[0093] In addition, the output of each B-TB also serves as a set of third image features.
[0094] Each sub-block layer includes: a layer normalization (LN) module, a multi-head self-attention (MSA) module, and a multi-layer perceptron (MLP) module.
[0095] like Figure 6 As shown, for the k-th (k = 1, 2, ..., M) B-TB, the input to its l-th (l = 1, 2, ..., N) sub-block layer is F. r(k,l) The output is F r(k,l+1) Where, if k = l = 1, then F r(k,l)If it is the first image feature, then it is the output of the previous sub-block layer.
[0096] In the l-th sub-block layer of the k-th B-TB, the LN module is used to process the input F. r(k,l) LN processing is performed, and the MSA module is used to process the image features after LN processing. The MSA-processed features and the input image features are then added together. The added features are then subjected to LN processing, MLP, and addition again to obtain the corresponding output features F. r(k,l+1) .
[0097] In this embodiment, each group of third image features can be obtained through the various encoding modules included in the fourth encoder.
[0098] For the fifth encoder, it may include:
[0099] Each group of third image features is used as the input to each coding module in at least one coding module of the fifth encoder, the second image feature is used as the input to the first coding module of the fifth encoder, and the speech feature is used as the input to the coding module to be fused; wherein the coding module to be fused is at least a portion of the coding modules in at least one coding module of the fifth encoder.
[0100] The fifth encoder employs each of the encoding modules to encode the inputs of each encoding module, thereby obtaining the outputs of each encoding module, which are then used as the inputs of the next encoding module.
[0101] The output of the last encoding module included in the fifth encoder is used as the fusion feature.
[0102] The fifth encoder can be an encoder based on a common attention mechanism and fusion processing.
[0103] The encoding module of the fifth encoder can be obtained by improving the Transformer block. The encoding module of the fifth encoder can be called the Cross-Modal ContextualFusion Transformer Block (CCF-TB).
[0104] The number of CCF-TBs included in the fifth encoder can be the same as the number of B-TBs included in the fourth encoder, such as... Figure 5 As shown, taking an example where both the number of CCF-TB and the number of B-TB are 5.
[0105] For the fifth encoder, the output of the first CCF-TB serves as the input to the second CCF-TB, the output of the second CCF-TB serves as the input to the third CCF-TB, and so on. The input to the first CCF-TB includes the second image features output from the second encoder. Additionally, each CCF-TB input includes a corresponding set of third image features. Furthermore, for the encoding module to be fused, the input also includes speech features.
[0106] The encoding modules to be fused are part or all of those in CCF-TB. Figure 5 Taking the last three CCF-TB coding modules as an example, the Chinese and Israeli coding modules to be fused are used. Additionally, speech features can be input into the coding modules to be fused through a fully connected network.
[0107] Each CCF-TB can include: convolutional layers and N (N is a positive integer, which can be set) sub-block layers. For each CCF-TB, the input of the first sub-block layer is the output of the corresponding convolutional layer of the CCF-TB, the input of the second sub-block layer is the output of the first sub-block layer, the input of the third sub-block layer is the output of the second sub-block layer, and so on. The output of the last sub-block layer of each CCF-TB is the output of that CCF-TB.
[0108] In addition, the output of the last CCF-TB is used as the fusion feature.
[0109] Each sub-block layer includes: a layer normalization (LN) module, a multi-head self-attention (MSA) module, a multi-head mutual attention (MMA) module, a full connection (FC) module, and a multi-layer perceptron (MLP) module.
[0110] like Figure 7 As shown, for the k-th (k = 1, 2, ..., M) CCF-TB, the input to its l-th (l = 1, 2, ..., N) sub-block layer includes: F t(k,l) F r(k,l) and f a(k,l) The output is F t(k,l+1) Where, if k = l = 1, then F t(k,l) If it is the second image feature, then it is the output of the previous sub-block layer. F r(k,l) It is the third image feature, f a(k,l) It is a speech feature.
[0111] In the l-th sub-block layer of the k-th CCF-TB, the LN module is used to process the input F t(k,l) LN processing is performed, and the MSA module is used to perform MSA processing on the LN-processed image features; the LN module is used to process the input F r(k,l) LN processing is performed, and the MMA module is used to process the image features after LN processing. For non-fusion coding modules (i.e., k=1, 2), the features after MSA and MMA processing are processed by FC (denoted as FC1), and then LN and MLP processing are performed. For coding modules to be fused (i.e., k>2), the features after FC1 processing are added to the speech features, and then FC processing is performed again (denoted as FC2), followed by LN and MLP processing to obtain the corresponding output feature F. r(k,l+1) .
[0112] In this embodiment, the fusion features can be obtained through the encoding module included in the fifth encoder.
[0113] 407. A decoder is used to decode the fused features to generate the target image.
[0114] The decoder is also a pre-trained deep neural network model, specifically a convolutional neural network model. The decoder can generate the corresponding target image based on features.
[0115] In this embodiment, the first image features are encoded using a self-attention mechanism to obtain the third image features, resulting in third image features with better expressive power. By encoding the third image features and the second image features using a common attention mechanism, the first and second image features can be better integrated. Furthermore, by incorporating speech features into at least a portion of the encoding modules of the fifth encoder, the fusion of speech features and image features can be achieved, thereby obtaining fused features that combine image and speech information and improving the quality of the generated target image.
[0116] The above embodiments involve a generator, which can also be called a generative model. The training process of the generative model will be described below.
[0117] Figure 8 This is a schematic diagram based on a third embodiment of the present disclosure. This embodiment provides a model training method, the model including: a first encoder, a second encoder, a third encoder, a fusion encoder, and a decoder. The method includes:
[0118] 801. Obtain training samples, the training samples including: real images, reference images and speech, the speech being used to drive the lip region of the real image samples.
[0119] 802. Mask the lip region of the real image to obtain the masked image.
[0120] 803. Using the first encoder, the reference image is encoded to obtain the first image features.
[0121] 804. Using the second encoder, the masked image is encoded to obtain the second image features.
[0122] 805. The speech sample is encoded using the third encoder to obtain speech features.
[0123] 806. Using the fusion encoder, obtain fusion features based on the first image features, the second image features, and the speech features.
[0124] 807. Using the decoder, the fused features are decoded to generate a predicted image.
[0125] 808. Construct a total loss function based on the real image and the predicted image.
[0126] 809. Based on the total loss function, adjust the model parameters of the first encoder, the second encoder, the third encoder, the fusion encoder, and the decoder.
[0127] During the training process, training samples can be pre-collected. For example, a frame from a video can be used as the real image, and another frame from the same video at a different time can be used as the reference image. Additionally, the audio corresponding to the real image can be collected.
[0128] The training process is similar to the model application process described above. When applying the model, the image itself (the first image) can be used as the reference image. When training the model, an image belonging to the same video as the real image but at a different time can be selected as the reference image.
[0129] refer to Figure 9 The architecture diagram shown represents a video where each frame is represented by the image V = {I1,...,I...}. T Each of these can be used as a real image. For each real image, an image from a different time period can be selected as a reference image. t Additionally, you can select the audio corresponding to each real image, a = {a1,...,a2}. T};
[0130] Figure 9 Taking the example where the first encoder and the second encoder are the same, using E e This indicates that the third encoder uses E. aThis indicates that the decoder uses Net. d The fourth encoder consists of 5 B-TBs, and the fifth encoder consists of 5 CCF-TBs.
[0131] Similar to the model application process, after inputting real images, parametric images, and speech into the generative model, the image output by the decoder can be called the predicted image, denoted by I. t 'Indicates. Generative models in Figure 9 The representation is a transformer-based backbone network, including a reference branch, a backbone branch, and a speech branch.
[0132] After obtaining the predicted image, a total loss function can be constructed based on the real image and the predicted image.
[0133] The total loss function can be constructed based on the first loss function, the second loss function, and the third loss function. For example, the sum of the above three loss functions can be used as the total loss function.
[0134] A first loss function can be constructed based on the real image and the predicted image. The first loss function can also be called a reconstruction loss function, for example, it can be based on the L2 norm between the pixel values of the real image and the pixel values of the predicted image.
[0135] The real image and the predicted image can be input into a feature extraction network to obtain the image features of the real image and the image features of the predicted image, and a second loss function can be constructed based on the image features of the real image and the image features of the predicted image.
[0136] The feature extraction network can be a VGG network; therefore, the second loss function can also be called the VGG loss function, denoted by L. VGG Specifically, it can be the L2 norm constructed from the two image features mentioned above.
[0137] The real image and the predicted image can be input into a discriminative network to obtain a discrimination result, and a third loss function can be constructed based on the discrimination result.
[0138] The third loss function can be called the generative adversarial loss function, denoted by L. GAN This can be represented by the relevant loss function of a generative adversarial network.
[0139] After obtaining the total loss function, the relevant model parameters can be adjusted using the back propagation (BP) algorithm based on the total loss function.
[0140] After the number of adjustments reaches the preset value, the adjusted model parameters can be used as the final model parameters for image generation during the inference stage.
[0141] In this embodiment, predicted images are generated based on fusion features. Fusion features are obtained by fusing image features and speech features, which can better represent image and speech information, thus improving the model's image generation performance. Furthermore, the image features include first image features and second image features, which can better represent image information, further improving the model's performance.
[0142] In some embodiments, the fusion encoder includes a fourth encoder and a fifth encoder;
[0143] The step of using the fusion encoder to obtain fused features based on the first image features, the second image features, and the speech features includes:
[0144] The fourth encoder is used to encode the first image features to obtain the third image features;
[0145] The fifth encoder is used to encode the third image feature, the second image feature, and the speech feature to obtain the fused feature.
[0146] In this embodiment, the first image features are encoded to obtain the third image features, and then the third image features are fused with other features to better represent the features corresponding to the first image, thereby improving the model quality.
[0147] In some embodiments, the step of using the fourth encoder to encode the first image features to obtain the third image features includes:
[0148] The first image feature is used as the input to the first encoding module included in the fourth encoder;
[0149] The inputs of each encoding module are encoded using at least one encoding module included in the fourth encoder to obtain the outputs of each encoding module, and the outputs of each encoding module are used as the inputs of the next encoding module; and,
[0150] The outputs of each encoding module are used as a set of third image features.
[0151] In this embodiment, each group of third image features can be obtained through the various encoding modules included in the fourth encoder.
[0152] In some embodiments, the fifth encoder is used to encode the third image feature, the second image feature, and the speech feature to obtain the fused feature, including:
[0153] Each group of third image features is used as the input to each coding module in at least one coding module of the fifth encoder, the second image feature is used as the input to the first coding module of the fifth encoder, and the speech feature is used as the input to the coding module to be fused; wherein the coding module to be fused is at least a portion of the coding modules in at least one coding module of the fifth encoder.
[0154] The fifth encoder employs each of the encoding modules to encode the inputs of each encoding module, thereby obtaining the outputs of each encoding module, which are then used as the inputs of the next encoding module.
[0155] The output of the last encoding module included in the fifth encoder is used as the fusion feature.
[0156] In this embodiment, the fusion features can be obtained through the encoding module included in the fifth encoder.
[0157] In some embodiments, constructing the total loss function based on the real image and the predicted image includes:
[0158] Based on the real image and the predicted image, a first loss function is constructed;
[0159] The real image and the predicted image are input into a feature extraction network to obtain the image features of the real image and the image features of the predicted image. A second loss function is constructed based on the image features of the real image and the image features of the predicted image.
[0160] The real image and the predicted image are input into the discrimination network to obtain the discrimination result, and a third loss function is constructed based on the discrimination result;
[0161] The total loss function is constructed based on the first loss function, the second loss function, and the third loss function.
[0162] In this embodiment, by constructing a first loss function, a second loss function, and a third loss function, and then constructing a total loss function based on these three loss functions, multiple dimensions of factors can be considered when constructing the total loss function, thereby improving the model performance.
[0163] Figure 10 This is a schematic diagram according to the fourth embodiment of the present disclosure. This embodiment provides an image generation device 1000, which includes: a mask module 1001, a first encoding module 1002, a second encoding module 1003, a third encoding module 1004, a fusion encoding module 1005, and a decoding module 1006.
[0164] A masking module 1001 is used to mask the lip region of a first image to obtain a second image; a first encoding module 1002 is used to encode the first image to obtain first image features; a second encoding module 1003 is used to encode the second image to obtain second image features; a third encoding module 1004 is used to encode speech to obtain speech features, wherein the speech is used to drive the lip region of the first image; a fusion encoding module 1005 is used to obtain fusion features based on the first image features, the second image features, and the speech features; and a decoding module 1006 is used to decode the fusion features to generate a target image, wherein the target image is the image obtained after the speech drives the lip region of the first image.
[0165] In this embodiment, the target image is generated based on fusion features. Fusion features are obtained by fusing image features and speech features, thus better representing image and speech information and improving the quality of the generated target image. Furthermore, the image features include first image features and second image features, which can better represent image information and further improve the quality of the target image.
[0166] In some embodiments, the fusion encoding module 1005 includes a fourth encoding module and a fifth encoding module. The fourth encoding module is used to encode the first image feature to obtain a third image feature; the fifth encoding module is used to encode the third image feature, the second image feature, and the speech feature to obtain the fused feature.
[0167] In this embodiment, the first image features are encoded to obtain the third image features, and then the third image features are fused with other features to better represent the features corresponding to the first image, thereby improving the quality of the target image corresponding to the first image.
[0168] In some embodiments, the fourth encoding module is further configured to: use the first image feature as input to the first encoding module included in the fourth encoder; use each encoding module in at least one encoding module included in the fourth encoder to encode the input of each encoding module to obtain the output of each encoding module, and use the output of each encoding module as input to the next encoding module; and use the output of each encoding module as a set of third image features.
[0169] In this embodiment, each group of third image features can be obtained through the various encoding modules included in the fourth encoder.
[0170] In some embodiments, the fifth encoding module is further configured to: use each group of third image features as input to each encoding module in at least one encoding module included in the fifth encoder; use the second image features as input to the first encoding module included in the fifth encoder; and use the speech features as input to the encoding module to be fused; wherein the encoding module to be fused is at least a portion of the encoding modules in at least one encoding module included in the fifth encoder; use each encoding module included in the fifth encoder to encode the input of each encoding module to obtain the output of each encoding module, and use it as input to the next encoding module; and use the output of the last encoding module included in the fifth encoder as the fused feature.
[0171] In this embodiment, the fusion features can be obtained through the encoding module included in the fifth encoder.
[0172] In some embodiments, the first encoding module is further configured to: encode the first image using a first encoder to obtain first image features; and / or, the second encoding module is further configured to: encode the second image using a second encoder to obtain second image features; and / or, the third encoding module is further configured to: encode speech using a third encoder to obtain speech features.
[0173] In this embodiment, the first image features, the second image features, and the speech features can be obtained by the first encoder, the second encoder, and the third encoder, respectively, so as to generate a target image based on these features.
[0174] Figure 11 The schematic diagram is based on the fifth embodiment of this disclosure. This embodiment provides a model training device. The model includes: a first encoder, a second encoder, a third encoder, a fusion encoder, and a decoder. The device 1100 includes: an acquisition module 1101, a mask module 1102, a first encoding module 1103, a second encoding module 1104, a third encoding module 1105, a fusion encoding module 1106, a decoding module 1107, a construction module 1108, and an adjustment module 1109.
[0175] The acquisition module 1101 is used to acquire training samples, which include: a real image, a reference image, and speech, wherein the speech is used to drive the lip region of the real image sample; the masking module 1102 is used to mask the lip region of the real image to obtain a masked image; the first encoding module 1103 is used to encode the reference image using the first encoder to obtain a first image feature; the second encoding module 1104 is used to encode the masked image using the second encoder to obtain a second image feature; and the third encoding module 1105 is used to encode the third encoder. The speech samples are encoded to obtain speech features; the fusion encoding module 1106 is used to obtain fused features based on the first image features, the second image features, and the speech features using the fusion encoder; the decoding module 1107 is used to decode the fused features using the decoder to generate a predicted image; the construction module 1108 is used to construct a total loss function based on the real image and the predicted image; and the adjustment module 1109 is used to adjust the model parameters of the first encoder, the second encoder, the third encoder, the fusion encoder, and the decoder based on the total loss function.
[0176] In this embodiment, predicted images are generated based on fusion features. Fusion features are obtained by fusing image features and speech features, which can better represent image and speech information, thus improving the model's image generation performance. Furthermore, the image features include first image features and second image features, which can better represent image information, further improving the model's performance.
[0177] In some embodiments, the fusion encoder includes a fourth encoder and a fifth encoder;
[0178] The fusion encoding module includes a fourth encoding module and a fifth encoding module. The fourth encoding module is used to encode the first image feature using the fourth encoder to obtain the third image feature; the fifth encoding module is used to encode the third image feature, the second image feature, and the speech feature using the fifth encoder to obtain the fused feature.
[0179] In this embodiment, the first image features are encoded to obtain the third image features, and then the third image features are fused with other features to better represent the features corresponding to the first image, thereby improving the model quality.
[0180] In some embodiments, the fourth encoding module is further configured to: use the first image feature as input to the first encoding module included in the fourth encoder; use each encoding module in at least one encoding module included in the fourth encoder to encode the input of each encoding module to obtain the output of each encoding module, and use the output of each encoding module as input to the next encoding module; and use the output of each encoding module as a set of third image features.
[0181] In this embodiment, each group of third image features can be obtained through the various encoding modules included in the fourth encoder.
[0182] In some embodiments, the fifth encoding module is further configured to: use each group of third image features as input to each encoding module in at least one encoding module included in the fifth encoder; use the second image features as input to the first encoding module included in the fifth encoder; and use the speech features as input to the encoding module to be fused; wherein the encoding module to be fused is at least a portion of the encoding modules in at least one encoding module included in the fifth encoder; use each encoding module included in the fifth encoder to encode the input of each encoding module to obtain the output of each encoding module, and use it as input to the next encoding module; and use the output of the last encoding module included in the fifth encoder as the fused feature.
[0183] In this embodiment, the fusion features can be obtained through the encoding module included in the fifth encoder.
[0184] In some embodiments, the construction module 1108 is further configured to: construct a first loss function based on the real image and the predicted image; input the real image and the predicted image into a feature extraction network to obtain image features of the real image and image features of the predicted image, and construct a second loss function based on the image features of the real image and image features of the predicted image; input the real image and the predicted image into a discriminator network to obtain a discrimination result, and construct a third loss function based on the discrimination result; and construct the total loss function based on the first loss function, the second loss function, and the third loss function.
[0185] In this embodiment, by constructing a first loss function, a second loss function, and a third loss function, and then constructing a total loss function based on these three loss functions, multiple dimensions of factors can be considered when constructing the total loss function, thereby improving the model performance.
[0186] It is understood that the same or similar content in different embodiments of this disclosure can be referred to each other.
[0187] It is understood that the terms "first" and "second" in the embodiments of this disclosure are only used for distinction and do not indicate the degree of importance or the order of events.
[0188] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0189] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.
[0190] Figure 12 A schematic block diagram of an example electronic device 1200 that can be used to implement embodiments of the present disclosure is shown. Electronic device 1200 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, servers, blade servers, mainframe computers, and other suitable computers. Electronic device 1200 may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.
[0191] like Figure 12 As shown, the electronic device 1200 includes a computing unit 1201, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1202 or a computer program loaded from a storage unit 1208 into a random access memory (RAM) 1203. The RAM 1203 may also store various programs and data required for the operation of the electronic device 1200. The computing unit 1201, ROM 1202, and RAM 1203 are interconnected via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.
[0192] Multiple components in electronic device 1200 are connected to I / O interface 1205, including: input unit 1206, such as keyboard, mouse, etc.; output unit 1207, such as various types of displays, speakers, etc.; storage unit 1208, such as disk, optical disk, etc.; and communication unit 1209, such as network card, modem, wireless transceiver, etc. Communication unit 1209 allows electronic device 1200 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0193] The computing unit 1201 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1201 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1201 performs the various methods and processes described above, such as image generation methods or model training methods. For example, in some embodiments, the image generation method or model training method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1208. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1200 via ROM 1202 and / or communication unit 1209. When the computer program is loaded into RAM 1203 and executed by the computing unit 1201, one or more steps of the image generation method or model training method described above may be performed. Alternatively, in other embodiments, the computing unit 1201 may be configured to perform an image generation method or a model training method by any other suitable means (e.g., by means of firmware).
[0194] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.
[0195] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable retrieval device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.
[0196] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0197] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).
[0198] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.
[0199] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.
[0200] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.
[0201] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.
Claims
1. An image generation method, comprising: The lip region of the first image is masked to obtain the second image; The first image is encoded to obtain first image features; The second image is encoded to obtain its features; The speech is encoded to obtain speech features, which are used to drive the lip region of the first image; Based on the first image feature, the second image feature, and the speech feature, a fusion feature is obtained, comprising: encoding the first image feature to obtain a third image feature; using each group of third image features as input to each encoding module in at least one encoding module of the fifth encoder, using the second image feature as input to the first encoding module of the fifth encoder, and using the speech feature as input to the encoding module to be fused; wherein the encoding module to be fused is at least a portion of the encoding modules in at least one encoding module of the fifth encoder; using each encoding module of the fifth encoder to encode the input of each encoding module to obtain the output of each encoding module, and using it as input to the next encoding module; and using the output of the last encoding module of the fifth encoder as the fusion feature. The fusion features are decoded to generate a target image, which is the image obtained after the speech drives the lip region of the first image.
2. The method according to claim 1, wherein, The process of encoding the first image features to obtain the third image features includes: The first image features are used as input to the first encoding module included in the fourth encoder; The inputs of each encoding module are encoded using at least one encoding module included in the fourth encoder to obtain the outputs of each encoding module, and the outputs of each encoding module are used as the inputs of the next encoding module; and, The outputs of each encoding module are used as a set of third image features.
3. The method according to any one of claims 1-2, wherein, The encoding process of the first image to obtain the first image features includes: The first image is encoded using a first encoder to obtain first image features; and / or, The encoding process for the second image to obtain second image features includes: A second encoder is used to encode the second image to obtain second image features; and / or, The process of encoding the speech to obtain speech features includes: A third encoder is used to encode the speech in order to obtain speech features.
4. A model training method, wherein the model comprises: The method comprises a first encoder, a second encoder, a third encoder, a fusion encoder, and a decoder, wherein the fusion encoder includes a fourth encoder and a fifth encoder, and the method includes: Acquire training samples, which include: real images, reference images, and speech, wherein the speech is used to drive the lip region of the real image samples; The lip region of the real image is masked to obtain the masked image; The reference image is encoded using the first encoder to obtain first image features; The masked image is encoded using the second encoder to obtain second image features; The third encoder is used to encode the speech samples to obtain speech features; Using the fusion encoder, based on the first image feature, the second image feature, and the speech feature, to obtain fused features includes: using the fourth encoder to encode the first image feature to obtain a third image feature; using each group of third image features as input to each encoding module in at least one encoding module included in the fifth encoder, using the second image feature as input to the first encoding module included in the fifth encoder, and using the speech feature as input to the encoding module to be fused; wherein the encoding module to be fused is at least a portion of the encoding modules in at least one encoding module included in the fifth encoder; The inputs of each encoding module included in the fifth encoder are encoded to obtain the outputs of each encoding module, which are then used as the inputs of the next encoding module; the output of the last encoding module included in the fifth encoder is used as the fusion feature. The decoder is used to decode the fused features to generate a predicted image; Based on the real image and the predicted image, a total loss function is constructed; Based on the total loss function, the model parameters of the first encoder, the second encoder, the third encoder, the fusion encoder, and the decoder are adjusted.
5. The method according to claim 4, wherein, The process of encoding the first image features using the fourth encoder to obtain the third image features includes: The first image feature is used as the input to the first encoding module included in the fourth encoder; The inputs of each encoding module are encoded using at least one encoding module included in the fourth encoder to obtain the outputs of each encoding module, and the outputs of each encoding module are used as the inputs of the next encoding module; and, The outputs of each encoding module are used as a set of third image features.
6. The method according to any one of claims 4-5, wherein, The step of constructing a total loss function based on the real image and the predicted image includes: Based on the real image and the predicted image, a first loss function is constructed; The real image and the predicted image are input into a feature extraction network to obtain the image features of the real image and the image features of the predicted image. A second loss function is constructed based on the image features of the real image and the image features of the predicted image. The real image and the predicted image are input into the discrimination network to obtain the discrimination result, and a third loss function is constructed based on the discrimination result; The total loss function is constructed based on the first loss function, the second loss function, and the third loss function.
7. An image generation apparatus, comprising: A masking module is used to mask the lip region of the first image to obtain a second image; A first encoding module is used to encode the first image to obtain first image features; The second encoding module is used to encode the second image to obtain second image features; The third encoding module is used to encode the speech to obtain speech features, and the speech is used to drive the lip region of the first image. A fusion coding module is used to obtain fusion features based on the first image features, the second image features, and the speech features, including: encoding the first image features to obtain third image features; using each group of third image features as input to each coding module in at least one coding module of the fifth encoder, using the second image features as input to the first coding module of the fifth encoder, and using the speech features as input to the coding module to be fused; wherein the coding module to be fused is at least a portion of the coding modules in at least one coding module of the fifth encoder; using each coding module of the fifth encoder to encode the input of each coding module to obtain the output of each coding module, and using it as input to the next coding module; and using the output of the last coding module of the fifth encoder as the fusion feature. A decoding module is used to decode the fused features to generate a target image, which is the image obtained after the speech drives the lip region of the first image.
8. The apparatus according to claim 7, wherein, The fourth encoding module is further used for: The first image features are used as input to the first encoding module included in the fourth encoder; The inputs of each encoding module are encoded using at least one encoding module included in the fourth encoder to obtain the outputs of each encoding module, and the outputs of each encoding module are used as the inputs of the next encoding module; and, The outputs of each encoding module are used as a set of third image features.
9. The apparatus according to any one of claims 7-8, wherein, The first encoding module is further configured to: encode the first image using a first encoder to obtain first image features; and / or, The second encoding module is further configured to: employ a second encoder to encode the second image to obtain second image features; and / or, The third encoding module is further used to: encode the speech using a third encoder to obtain speech features.
10. A model training apparatus, the model comprising: The device comprises a first encoder, a second encoder, a third encoder, a fusion encoder, and a decoder, wherein the fusion encoder includes a fourth encoder and a fifth encoder, and the device includes: An acquisition module is used to acquire training samples, which include: real images, reference images, and speech, wherein the speech is used to drive the lip region of the real image samples; A masking module is used to mask the lip region of the real image to obtain a masked image; A first encoding module is used to encode the reference image using the first encoder to obtain first image features; The second encoding module is used to encode the masked image using the second encoder to obtain second image features; The third encoding module is used to encode the speech samples using the third encoder to obtain speech features; A fusion coding module is used to obtain fused features based on the first image features, the second image features, and the speech features using the fusion encoder. The module includes: encoding the first image features using the fourth encoder to obtain third image features; using each group of third image features as input to each coding module in at least one coding module of the fifth encoder; using the second image features as input to the first coding module of the fifth encoder; and using the speech features as input to the coding module to be fused; wherein the coding module to be fused is at least a portion of the coding modules in at least one coding module of the fifth encoder; encoding the inputs of each coding module using each coding module of the fifth encoder to obtain the outputs of each coding module, and using these outputs as input to the next coding module; and using the output of the last coding module of the fifth encoder as the fused feature. A decoding module is used to decode the fused features using the decoder to generate a predicted image; The construction module is used to construct the total loss function based on the real image and the predicted image; The adjustment module is used to adjust the model parameters of the first encoder, the second encoder, the third encoder, the fusion encoder, and the decoder based on the total loss function.
11. The apparatus according to claim 10, wherein, The fourth encoding module is further used for: The first image feature is used as the input to the first encoding module included in the fourth encoder; The inputs of each encoding module are encoded using at least one encoding module included in the fourth encoder to obtain the outputs of each encoding module, and the outputs of each encoding module are used as the inputs of the next encoding module; and, The outputs of each encoding module are used as a set of third image features.
12. The apparatus according to any one of claims 10-11, wherein, The building module is further used for: Based on the real image and the predicted image, a first loss function is constructed; The real image and the predicted image are input into a feature extraction network to obtain the image features of the real image and the image features of the predicted image. A second loss function is constructed based on the image features of the real image and the image features of the predicted image. The real image and the predicted image are input into the discrimination network to obtain the discrimination result, and a third loss function is constructed based on the discrimination result; The total loss function is constructed based on the first loss function, the second loss function, and the third loss function.
13. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.
14. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-6.
15. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-6.