Training method of image generation model, image generation method, device and equipment

Abstract

The present disclosure provides a training method of an image generation model, an image generation method, device and equipment, relates to the technical field of artificial intelligence, specifically to the technical field of deep learning or computer vision, and can be applied to scenes such as face image processing or face image recognition. The specific implementation scheme is as follows: obtaining a first image of a target person, a first face region image in the first image, a second image and a second face region image in the second image; inputting the first image, the first face region image, the second image and the second face region image into an image generation model to obtain a synthesized image; inputting the first image, the second image and the synthesized image into a pre-trained loss calculation model to calculate a target loss, and adjusting parameters of the image generation model according to the target loss. The embodiment of the present disclosure can improve the accuracy of the image generation model.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, in particular to the technical field of deep learning or computer vision, which can be applied to scenarios such as face image processing or face image recognition, and especially relates to a training method of an image generation model, an image generation method, an apparatus and a device. BACKGROUND

[0002] Face driving has wide applications in scenarios such as film and television editing and virtual people.

[0003] Face driving refers to given an original image and a target image, transferring the identity features in the target image to the original image to obtain a driven image, so that the driven image not only maintains the identity features of the target image, but also has the attribute features such as the facial pose and expression of the original image. SUMMARY

[0004] The present disclosure provides a training method of an image generation model, an image generation method, an apparatus and a device.

[0005] According to an aspect of the present disclosure, a training method of an image generation model is provided, comprising:

[0006] obtaining a first image of a target person, a first facial region image in the first image, a second image and a second facial region image in the second image;

[0007] inputting the first image, the first facial region image, the second image and the second facial region image into an image generation model to obtain a synthesized image;

[0008] calculating a target loss by inputting the first image, the second image and the synthesized image into a pre-trained loss calculation model, and adjusting parameters of the image generation model according to the target loss.

[0009] According to an aspect of the present disclosure, an image generation method is also provided, comprising:

[0010] obtaining a person identity image of a first person, and processing the person identity image to obtain a facial identity image of the person identity image;

[0011] obtaining a pose and expression image of a second person, and processing the pose and expression image to obtain a facial expression image of the pose and expression image;

[0012] inputting the person identity image, the facial identity image, the pose and expression image and the facial expression image into a pre-trained image generation model to obtain a synthesized image; wherein the image generation model is trained according to the training method of the image generation model as described in any one of the embodiments of the present disclosure.

[0013] According to an aspect of the present disclosure, a training device of an image generation model is provided, comprising:

[0014] An image preprocessing module is configured to acquire a first image of a target person, a first facial region image in the first image, a second image, and a second facial region image in the second image.

[0015] A model output acquisition module is configured to input the first image, the first facial region image, the second image, and the second facial region image into an image generation model to obtain a synthesized image.

[0016] A first adjustment parameter module is configured to input the first image, the second image, and the synthesized image into a pre-trained loss calculation model to calculate a target loss, and adjust parameters of the image generation model according to the target loss.

[0017] According to an aspect of the present disclosure, an image generation device is also provided, comprising:

[0018] An identity image acquisition module is configured to acquire a person identity image of a first person, and process the person identity image to obtain a facial identity image of the person identity image.

[0019] A posture expression image acquisition module is configured to acquire a posture expression image of a second person, and process the posture expression image to obtain a facial expression image of the posture expression image.

[0020] An image synthesis module is configured to input the person identity image, the facial identity image, the posture expression image, and the facial expression image into a pre-trained image generation model to obtain a synthesized image, wherein the image generation model is trained according to the training method of the image generation model of any one of the embodiments of the present disclosure.

[0021] According to another aspect of the present disclosure, an electronic device is provided, comprising:

[0022] at least one processor; and

[0023] a memory in communication with the at least one processor; wherein

[0024] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the training method of the image generation model according to any one of the embodiments of the present disclosure, or perform the image generation method according to any one of the embodiments of the present disclosure.

[0025] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, where the computer instructions are used to cause the computer to perform the training method of the image generation model according to any of the embodiments of the present disclosure, or perform the image generation method according to any of the embodiments of the present disclosure.

[0026] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the training method of the image generation model according to any of the embodiments of the present disclosure, or performs the image generation method according to any of the embodiments of the present disclosure.

[0027] The embodiments of the present disclosure can improve the accuracy of the image generation model.

[0028] It should be understood that the content described in this section is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent through the following description. BRIEF DESCRIPTION OF DRAWINGS

[0029] The accompanying drawings are used to better understand the present scheme, and do not limit the present disclosure. Among them:

[0030] Figure 1 is a schematic diagram of a training method of an image generation model according to an embodiment of the present disclosure;

[0031] Figure 2 is a schematic diagram of another training method of an image generation model according to an embodiment of the present disclosure;

[0032] Figure 3 is a schematic diagram of another training method of an image generation model according to an embodiment of the present disclosure;

[0033] Figure 4 is a schematic diagram of an existing image;

[0034] Figure 5 is a schematic diagram of an image used by an embodiment according to an embodiment of the present disclosure;

[0035] Figure 6 is a scenario diagram of a training method of an image generation model according to an embodiment of the present disclosure;

[0036] Figure 7 is a schematic diagram of an image generation method according to an embodiment of the present disclosure;

[0037] Figure 8 is a schematic diagram of a training device of an image generation model according to an embodiment of the present disclosure;

[0038] Figure 9 is a schematic diagram of an image generation device according to an embodiment of the present disclosure;

[0039] Figure 10 is a block diagram of an electronic device for implementing a training method of an image generation model or an image generation method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0040] Exemplary embodiments of the present disclosure are described herein with reference to the accompanying drawings, which are provided to assist in a comprehensive understanding of the present disclosure, and should be considered as merely exemplary. Accordingly, those skilled in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, in the following description, descriptions of well-known functions and constructions are omitted for clarity and conciseness.

[0041] Figure 1 is a flowchart of a training method of an image generation model according to an embodiment of the present disclosure, and the embodiment can be applied to a case of training an image generation model. The embodiment method can be performed by a training device of an image generation model, which can be implemented in software and / or hardware, and specifically configured in an electronic device with certain data operation capability, which can be a client device or a server device, such as a mobile phone, a tablet computer, a vehicle terminal, and a desktop computer.

[0042] S101, obtaining a first image of a target person, a first face region image in the first image, a second image, and a second face region image in the second image.

[0043] The first image and the second image include images of the same person. The first face region image is an image including a face of the target person, and the first image includes the first face region image. The second face region image is an image including a face of the target person, and the second image includes the second face region image. The number of first images is at least one, the number of first face region images is at least one, and the number of first face region images is the same as the number of first images. The identity information of the target person in the first image and the second image is consistent, but the attribute information is different, and the attribute information includes the posture or expression of the target person. The region including the target person can be cut from the image to determine the face region image, which can be understood as eliminating the background information in the image and retaining the key information of the person to obtain the face region image. The first image, the first face region image in the first image, the second image, and the second face region image in the second image are used as image samples to train the image generation model.

[0044] S102, input the first image, the first face region image, the second image and the second face region image into an image generation model to obtain a synthesized image.

[0045] The first image is used to provide identity information of a target person, and the second image is used to provide attribute information of the target person. The synthesized image can be an image of a person whose identity information is displayed according to the attribute information, and the synthesized image contains both the identity information of the target person and the attribute information of the target person. The image generation model is used to synthesize the identity information and the attribute information to generate an image of a person whose identity information is displayed according to the attribute information. The image generation model can include an encoder and a generator (or a decoder).

[0046] S103, by inputting the first image, the second image and the synthesized image into a pre-trained loss calculation model, calculating a target loss, and adjusting parameters of the image generation model according to the target loss.

[0047] The synthesized image is an output result of the image generation model, and the first image and the second image can be used as input images or as true value results. The loss calculation model can be a pre-trained machine learning model used to calculate the loss of the image generation model. The target loss is used to adjust the parameters of the image generation model.

[0048] Specifically, for the first image and the second image of the same target person, the true value result obtained by processing the first image and the second image through the image generation model is an image that contains both the identity information of the target person and the attribute information of the target person, i.e., the second image. The loss can be calculated according to the difference between the second image and the synthesized image. In addition, the identity information in the true value result is the same as the identity information of the first image, i.e., the identity information of the first image can be used as the identity information of the true value result, and the loss can be calculated according to the difference in identity features between the first image and the synthesized image. The target loss is calculated according to the loss between the second image and the synthesized image and the loss between the first image and the synthesized image, for example, the target loss is obtained by accumulation.

[0049] According to the technical solution of the present disclosure, by obtaining the first image of the target person, the first face region image in the first image, the second image and the second face region image in the second image as image samples, inputting them into the image generation model to obtain the synthesized image, and calculating the target loss according to the first image, the second image and the synthesized image, adjusting the parameters of the image generation model, increasing the face details in the image samples, enriching the face contents in the samples, introducing the difference between the information in the synthesized image and the true value information through the additional loss calculation model, so that the image generation model learns more face details of the person, which can improve the ability of the image generation model to learn the face features and improve the image generation accuracy of the image generation model.

[0050] Figure 2 The flowchart of another method for training an image generation model according to an embodiment of the present disclosure is disclosed, which is further optimized and expanded based on the above technical solutions and can be combined with the above various optional embodiments. The inputting of the first image, the first facial region image, the second image and the second facial region image into the image generation model to obtain a synthesized image is specifically: inputting the first image and the first facial region image into an identity image feature extraction network in the image generation model to obtain identity image features output by the identity image feature extraction network; inputting the second image and the second facial region image into a pose expression feature extraction network in the image generation model to obtain pose expression features output by the pose expression feature extraction network; fusing the identity image features and the pose expression features to obtain fused features; and inputting the fused features into a generator in the image generation model to obtain a synthesized image output by the generator.

[0051] S201, obtaining a first image of a target person, a first facial region image in the first image, a second image and a second facial region image in the second image.

[0052] S202, inputting the first image and the first facial region image into an identity image feature extraction network in the image generation model to obtain identity image features output by the identity image feature extraction network.

[0053] The identity image feature extraction network can be a feature extraction network for extracting identity image features. The identity image features are used to describe the identity information of the person in the first image. The first image is used to provide global image information, and the first facial region image is used to provide detailed image information. The global and detailed image information is provided from the first image and the first facial region image, enriching the content of the identity image features. Exemplarily, the identity image feature extraction network can be a network for extracting features, such as an encoder or a convolutional neural network, etc.

[0054] S203, inputting the second image and the second facial region image into a pose expression feature extraction network in the image generation model to obtain pose expression features output by the pose expression feature extraction network.

[0055] The pose expression feature extraction network can be a feature extraction network for extracting pose expression features. The pose expression features are used to describe the attribute information of the character in the second image, specifically the pose expression information of the character. The second image is used to provide global image information, and the second face region image is used to provide detailed image information. The global and detailed image information is provided from the second image and the second face region image, enriching the content of the pose expression features. Exemplarily, the pose expression feature extraction network can be a network for extracting features, such as an encoder or a convolutional neural network.

[0056] S204, the identity image features and the pose expression features are fused to obtain fused features.

[0057] The fused features are used to describe the identity information and the pose expression information of the target character. The fused features can refer to the fusion result of the identity image features and the pose expression features of the target character. The fusion method can include splicing, superimposing the pixel values of corresponding pixels in the image, or multiplying the pixel matrices corresponding to the images.

[0058] Optionally, the fusing of the identity image features and the pose expression features to obtain the fused features includes splicing the identity image features and the pose expression features to obtain the fused features.

[0059] Splicing refers to combining the two features while retaining the information of the identity image features and the pose expression features. For example, the identity image features and the pose expression features are both 10*10 image matrices, and the fused features after splicing are 10*20 or 20*10 image matrices. Splicing refers to combining the two features without changing the features to maintain the original information. In fact, the splicing fusion method is a decoupled fusion method, and in the fused features, the identity image features and the pose expression features do not affect each other.

[0060] By splicing the identity image features and the pose expression features, the identity image features and the pose expression features are decoupled, which can reduce the coupling between the processing process of the identity image features and the processing process of the pose expression features, reduce the mutual influence between the identity image features and the pose expression features, thereby improving the learning ability of the image generation model for the identity image features and the pose expression features respectively, and further improving the image generation accuracy of the image generation model.

[0061] S205, the fused features are input into the generator of the image generation model to obtain a synthesized image output by the generator.

[0062] The generator is configured to process the fusion feature and output a synthetic image. The fusion feature can be directly taken as an input of the generator, and the generator outputs the synthetic image. Alternatively, the image generation model can further include a multi-layer perceptron (MLP), and the generator includes a plurality of concatenated network layers. The fusion feature can be mapped to different network layers through the MLP. The input of the generator includes the fusion feature and the fusion feature mapped through the MLP. Thus, for at least one network layer except the first network layer, the input of the network layer can be the output of the previous concatenated network layer and the fusion feature mapped through the MLP, and the output of the network layer can be taken as the input of the next concatenated network layer.

[0063] In S206, a target loss is calculated by inputting the first image, the second image and the synthetic image into a pre-trained loss calculation model, and parameters of the image generation model are adjusted according to the target loss.

[0064] Optionally, the inputting the first image and the first face region image into the identity image feature extraction network of the image generation model to obtain the identity image feature output by the identity image feature extraction network includes: inputting the first image into a full-image feature extraction network of the image generation model to obtain a full-image feature output by the full-image feature extraction network; inputting the first face region image into an identity feature extraction network of the image generation model to obtain an identity feature output by the identity feature extraction network; and fusing the full-image feature and the identity feature to obtain the identity image feature.

[0065] The identity image feature extraction network includes a full-image feature extraction network and an identity feature extraction network. The full-image feature extraction network and the identity feature extraction network independently extract features. The full-image feature extraction network is configured to extract full-image information from the first image to obtain a full-image feature, and the identity feature extraction network is configured to extract detailed identity information from the first face region image to obtain an identity feature. The full-image feature is used to describe the full-image information in the first image, and the identity feature is used to describe the identity detailed information in the first face region image. Fusing the full-image feature and the identity feature can be splicing the full-image feature and the identity feature, reducing the coupling between the full-image feature and the identity feature, and reducing the mutual influence between the full-image feature and the identity feature, thereby improving the learning ability of the image generation model for the full-image feature and the identity feature respectively, and finally improving the accuracy of the generated image.

[0066] The global image feature is extracted from the first image by a global feature extraction network, the identity information is extracted from the first face region image by an identity feature extraction network, the identity feature is obtained, and the identity image feature is obtained by fusing the global image feature and the identity feature. The identity image feature includes global information and detail information, and the content of the identity image feature is enriched. The identity feature is extracted from the more detailed first face region image, more accurate and more detailed identity information is obtained, the representativeness of the identity feature is improved, and the image generation accuracy of the image generation model is improved.

[0067] Optionally, the inputting the second image and the second face region image into the image generation model to obtain the pose expression feature extracted by the pose expression feature extraction network includes: inputting the second image into a pose feature extraction network in the image generation model to obtain a pose feature output by the pose feature extraction network; inputting the second face region image into an expression feature extraction network in the image generation model to obtain an expression feature output by the expression feature extraction network; and fusing the pose feature and the expression feature to obtain a pose expression feature.

[0068] The pose expression feature extraction network includes a pose feature extraction network and an expression feature extraction network. The pose feature extraction network and the expression feature extraction network independently extract features. The pose feature extraction network is used for extracting global pose information from the second image to obtain a pose feature, and the expression feature extraction network is used for extracting detailed expression information from the second face region image to obtain an expression feature. The pose feature is used for describing the pose information in the second image, and the expression feature is used for describing the expression detail information in the second face region image. The fusion of the pose feature and the expression feature can be splicing the pose feature and the expression feature, reducing the coupling between the pose feature and the expression feature, and reducing the mutual influence between the pose feature and the expression feature, thereby improving the learning ability of the image generation model for the pose feature and the expression feature respectively, and finally improving the accuracy of the generated image.

[0069] The pose information is extracted from the second image by the pose feature extraction network to obtain a pose feature, the expression information is extracted from the second face region image by the expression feature extraction network to obtain an expression feature, and the pose expression feature is obtained by fusing the pose feature and the expression feature. The pose expression feature includes global information and detail information, and the content of the pose expression feature is enriched. The expression feature is extracted from the more detailed second face region image, more accurate and more detailed expression information is obtained, the representativeness of the expression feature is improved, and the image generation accuracy of the image generation model is improved.

[0070] The full-image feature extraction network, the identity feature extraction network, the posture feature extraction network, and the expression feature extraction network are all encoders, and can be any network that can realize feature extraction. The identity feature extraction network can be a feature extraction network in a trained face classification network, for example, a feature extraction network in an Additive Angular Margin Loss for Deep Face Recognition (ArcFace) network. The full-image feature extraction network, the posture feature extraction network, and the expression feature extraction network can be a Residual Neural Network (Resnet).

[0071] According to the technical solution of the present disclosure, by extracting identity image features from the first image and the first face region image, and extracting posture expression features from the second image and the second face region image, the content of the extracted identity image features and posture expression features can be enriched, and the representativeness of the features can be improved, thereby improving the image generation accuracy of the image generation model.

[0072] Figure 3 is a flowchart of another method for training an image generation model according to an embodiment of the present disclosure, which is further optimized and expanded based on the above technical solution and can be combined with the above various optional embodiments. The target loss is calculated by inputting the first image, the second image, and the synthesized image into a pre-trained loss calculation model, which is specifically: inputting the first image and the synthesized image into a first loss feature extraction network to calculate an identity feature loss; inputting the second image and the synthesized image into a second loss feature extraction network to calculate a posture expression feature loss; inputting the second image and the synthesized image into a discriminator to calculate a real feature loss and a real generation loss; and determining the target loss according to the identity feature loss, the posture expression feature loss, the real feature loss, and the real generation loss.

[0073] S301, obtaining a first image of a target person, a first face region image in the first image, a second image, and a second face region image in the second image.

[0074] S302, inputting the first image, the first face region image, the second image, and the second face region image into an image generation model to obtain a synthesized image.

[0075] S303, inputting the first image and the synthesized image into a first loss feature extraction network to calculate an identity feature loss.

[0076] The identity feature loss is used to describe the feature difference between the synthetic image and the first image. The first loss feature extraction network is used to extract the first loss feature. For example, the first loss feature extraction network can have the same structure as the identity feature extraction network, but the first loss feature extraction network is a pre-trained feature extraction network, such as the feature extraction network in the pre-trained ArcFace. In addition, the identity feature extraction network can be a feature extraction network in the ArcFace that is trained but not fully trained. The first loss feature extraction network can be a feature extraction network in the ArcFace that is fully trained.

[0077] The first image can be understood as an image of the true value identity, and the synthetic image can be understood as an image of the output identity. According to the identity feature loss, the parameters of the image generation model are adjusted, which is actually to continuously approach the image of the true value identity from the image of the output identity, so as to improve the ability of the image generation model to learn the identity feature.

[0078] For example, the intermediate features extracted by the first loss feature extraction network are aligned in a high-dimensional information space. Specifically, the identity feature loss L can be calculated based on the following formula ID :

[0079] L ID =||A(Y)-A(X ID )||2

[0080] Wherein, A(Y) is the identity feature extracted from the synthetic image by the first loss feature extraction network, A(X ID ) is the identity feature extracted from the first image by the first loss feature extraction network. ||*||2 represents the L2 norm loss function. A(Y)-A(X ID ) is the sum of feature losses extracted by at least one feature layer in the first loss feature extraction network.

[0081] In fact, the identity feature size output by the same feature layer in the first loss feature extraction network is the same, and the identity feature is actually a feature map, that is, the size of the feature map output by the same feature layer is the same, and the pixel difference can be calculated according to the pixels constituting the feature map to calculate the absolute value of the difference between the images in the pixel dimension as the pixel loss of the feature layer. According to the pixel loss of the feature layer, the feature loss is calculated, which can be specifically: the number of feature layers is one, and the pixel loss is taken as the feature loss; the number of feature layers is at least two, and the sum of the pixel losses is calculated as the feature loss. Any feature layer can be selected, for example, at least one intermediate feature layer is selected, or at least one first and last feature layer can be selected, and the pixel loss between the feature map output for the synthesized image and the feature map output for the first image is calculated for each feature layer, and the sum of the pixel losses of the first feature layer, the second feature layer and the last feature layer is calculated to determine the identity feature loss.

[0082] S304, inputting the second image and the synthesized image into the second loss feature extraction network to calculate the pose-expression feature loss.

[0083] The pose-expression feature loss is used to describe the feature difference between the synthesized image and the second image. The second loss feature extraction network is used to extract the second loss feature, and the second loss feature extraction network can be a deep convolutional neural network (Visual Geometry Group, VGG) for example. The second image can be understood as an image of the true value pose-expression, and the synthesized image can be understood as an image of the output pose-expression. According to the pose-expression feature loss, the parameters of the image generation model are adjusted, which is actually to continuously approach the image of the output pose-expression to the image of the true value pose-expression, so as to improve the ability of the image generation model to learn the pose-expression feature. The second loss feature extraction network can be different from the pose feature extraction network, the expression feature extraction network and the pose-expression feature extraction network.

[0084] The second image can be understood as an image of the true value pose-expression, and the synthesized image can be understood as an image of the output pose-expression. According to the pose-expression feature loss, the parameters of the image generation model are adjusted, which is actually to continuously approach the image of the output pose-expression to the image of the true value pose-expression, so as to improve the ability of the image generation model to learn the pose-expression feature.

[0085] For example, the intermediate features extracted by the second loss feature extraction network are aligned in a high-dimensional information space, and the pose-expression loss L can be calculated based on the following formula pose :

[0086] L pose =||B(Y)-B(X pose)||2

[0087] wherein B(Y) is the pose-expression feature extracted from the synthetic image by the second loss feature extraction network, B(X pose ) is the pose-expression feature extracted from the second image by the second loss feature extraction network. B(Y)-B(X pose ) is the sum of the feature losses of at least one feature layer in the second loss feature extraction network.

[0088] In fact, the pose-expression features output by the same feature layer in the second loss feature extraction network have the same size, and the pose-expression feature is actually a feature map, that is, the feature maps output by the same feature layer have the same size. The pixel difference can be calculated according to the pixels constituting the feature map, and the absolute value of the difference between the images in the pixel dimension can be calculated as the pixel loss of the feature layer. According to the pixel loss of the feature layer, the feature loss can be calculated, which can be: when the number of feature layers is one, the pixel loss is taken as the feature loss; when the number of feature layers is at least two, the sum of the pixel losses is calculated as the feature loss. Any feature layer can be selected, for example, at least one feature layer in the middle or at least one feature layer at the beginning and end can be selected. For example, the first feature layer, the second feature layer and the last feature layer are selected, the pixel loss between the feature map output for the synthetic image and the feature map output for the second image is calculated for each feature layer, and the sum of the pixel losses of the first feature layer, the second feature layer and the last feature layer is calculated to determine the pose-expression feature loss.

[0089] S305, inputting the second image and the synthetic image into the discriminator to calculate the real feature loss and the real generation loss.

[0090] The discriminator is used to classify images to determine whether the images are real human images, specifically to distinguish real human images and model-generated images. The real feature loss is used to describe the difference in real image features between the synthetic image and the second image. The real generation loss is used to describe the difference between the discrimination result of the synthetic image and the discrimination result of the model-generated image, and the difference between the discrimination result of the second image and the discrimination result of the real image.

[0091] The second image can be understood as a real image, and the synthetic image can be understood as an output false image. According to the real feature loss, the parameters of the image generation model are adjusted, which is actually to continuously approach the real image from the output false image to improve the ability of the image generation model to learn the features of the real image. According to the real generation loss, the parameters of the image generation model are adjusted, which is actually to continuously approach the real image from the output false image to improve the ability of the image generation model to learn to generate real images. The second image can be replaced by the first image.

[0092] Exemplarily, the intermediate features extracted by the discriminator are aligned in a high-dimensional information space, and the real feature loss L can be calculated based on the following formula D :

[0093] L D =||D(Y)-D(X pose )||2

[0094] wherein D(Y) is the image feature extracted by the discriminator from the synthetic image, D(X pose ) is the image feature extracted by the discriminator from the second image. D(Y)-D(X pose ) is the sum of feature losses extracted by at least one feature layer in the discriminator.

[0095] In fact, the image features output by the same feature layer in the discriminator have the same size, and the image feature is actually a feature map, that is, the feature maps output by the same feature layer have the same size. The pixel difference can be calculated according to the pixels constituting the feature map, and the absolute value of the difference between the images in the pixel dimension can be calculated as the pixel loss of the feature layer. According to the pixel loss of the feature layer, the feature loss can be calculated, which can be: the number of feature layers is one, and the pixel loss is taken as the feature loss; the number of feature layers is at least two, and the sum of the pixel losses is calculated as the feature loss. Any feature layer can be selected, for example, at least one intermediate feature layer can be selected, or at least one first and last feature layer can be selected. Exemplarily, the first feature layer, the second feature layer and the last feature layer are selected, the pixel loss between the feature map output for the synthetic image and the feature map output for the second image is calculated for each feature layer, and the sum of the pixel losses of the first feature layer, the second feature layer and the last feature layer is calculated to determine the real feature loss.

[0096] The model generated image can be referred to as a fake image, and the real image collected can be referred to as a real image. For the discriminator, the model generated image can be labeled as fake (for example, the value is 0), and the standard real image can be labeled as real (for example, the value is 1). The synthetic image is input into the discriminator, and the expected output of the discriminator should be 0, that is, fake. The actual output of the discriminator is D GAN (X pose ), and part of the loss of the discriminator is determined according to the difference between the expected output and the actual output of the synthetic image, that is, D GAN (X pose )-0. The second image is input into the discriminator, and the expected output of the discriminator should be 1, that is, true. The actual output of the discriminator is D GAN (Y), and the loss of the discriminator can be calculated using mean square error. Part of the loss of the discriminator is determined according to the difference between the expected output and the actual output of the second image or the first image, that is, 1-D GAN(Y), determine. The loss of the discriminator can be calculated using the mean square error. The sum of the two parts of the loss is determined as the real generation loss.

[0097] For example, the adversarial training of the discriminator reduces the artificial traces in the generated image. Specifically, the real generation loss L can be calculated based on the following formula GAN :

[0098] L GAN = E (logD GAN (X pose ))+ E (log (1-D GAN (Y)))

[0099] Where D GAN (X pose ) is the difference between the expected output and the actual output of the synthetic image, and 1-D GAN (Y) is the difference between the expected output and the actual output of the second image or the first image. E (*) represents the mean square error function.

[0100] S306, according to the identity feature loss, pose expression feature loss, real feature loss and real generation loss, determine the target loss, and adjust the parameters of the image generation model according to the target loss.

[0101] Optionally, the first image and the second image of the target person are obtained, comprising: obtaining a video stream of the target person; from the video stream, at least one image is determined as the first image, and an image different from the first image is determined as the second image.

[0102] The video stream can be obtained from the network or a real-time collected video stream. The video stream includes at least one person, and one person can be selected as the target person. The video stream is analyzed to obtain at least two images, and at least one image is selected from each image to determine the first image, and an image different from the first image is selected to determine the second image. The attributes of the target person in the first image and the second image are different, specifically the expression and pose are different. Among them, the attributes of the target person in each first image are different.

[0103] In addition, the images obtained by directly parsing the video stream can also be preprocessed, and the first image and the second image can be screened from the preprocessed images. The determining, from the video stream, of at least one image as the first image and of an image different from the first image as the second image can include: parsing the video stream to obtain at least one image, pre-processing each frame image according to the selected target person to obtain an image including the head of the target person, and screening at least one pre-processed image from the pre-processed images to determine the at least one pre-processed image as the first image, and screening a pre-processed image different from the first image to determine the pre-processed image as the second image. The pre-processing manner can be face key point detection on the image, determining a clipping box aligned with the face key point according to the face key point, and clipping the image according to the clipping box to obtain a forward image including the head of the target person, and the clipped image includes a complete top of the head and a complete edge of the hair, enriching the information of the target person, reducing the loss of image information, thereby enriching the content of the features extracted based on the image, and improving the image quality and the image accuracy.

[0104] By obtaining the video stream of the same target person and screening the first image and the second image, the first image and the second image can be quickly obtained, the cost of sample collection can be reduced, the sample collection efficiency can be improved, and the model training efficiency can be improved.

[0105] According to the technical solution of the present disclosure, the model parameters are adjusted by calculating the identity feature loss to make the output identity image continuously approach the ground truth identity image, to improve the ability of the image generation model to learn the identity feature, the model parameters are adjusted by calculating the pose expression feature loss to make the output pose expression image continuously approach the ground truth pose expression image, to improve the ability of the image generation model to learn the pose expression feature, the model parameters are adjusted by calculating the real feature loss to make the output image continuously approach the real image, to improve the ability of the image generation model to learn the feature of the real image, and the model parameters are adjusted by calculating the real generation loss to make the output image continuously approach the real image, to improve the ability of the image generation model to generate real images, thereby improving the accuracy of the image generation model in generating images.

[0106] Figure 4 is the visualization effect diagram of the existing image. As shown in Figure 4 , the size of the image is 256*256, and the top of the head is incomplete and the bottom edge of the hair is incomplete. The image refers to the first image or the second image. The existing matting model has a low resolution and a size of 256*256 due to the constraints of the data, and there are problems of cutting the top and clipping the shoulder-length hair in the output image, which can cause the synthesized image after face replacement to be different from the identity feature of the target person, for example, the hair is different. Thus, the quality of the synthesized image is poor, and the identity is incorrect.

[0107] Figure 5 is a visualization effect diagram of an image according to an embodiment of the present disclosure. As shown in Figure 5 , the size of the image is 512*512, the proportion of the face region is small, and the top of the head is complete, and the bottom edge of the hair is complete. Among them, the image refers to the first image or the second image. Figure 5 The image shown in Figure 4 is higher in definition than the image shown in Figure 5 , the proportion of the face region in the image shown in Figure 4 is less than Figure 5 , and Figure 5 , the image shown in Figure 5 does not have the problems of truncated top and long hair. Based on the image including complete top of the head and hair edge input into the image generation model, more complete and detailed identity information and posture expression information are retained, and the accuracy of the generated synthetic image is improved.

[0108] Figure 6 is a scene diagram of a training method of an image generation model according to an embodiment of the present disclosure. The image classification model includes a full-image feature extraction network 605, an identity feature extraction network 606, a posture feature extraction network 607, an expression feature extraction network 608, and a generator 613. In addition, the image classification model can also include an MLP and other structures, which are not limited.

[0109] The training of the image generation model can be: parsing the images from the video stream of the target person, and preprocessing the images to obtain images including the head of the target person, and screening a plurality of first images 601 and second images 603 from the images, each first image 601 being different from each other, and each first image 601 and second image 603 being different. The image including the head of the target person can be an image retaining the top of the head and the bottom edge of the hair, and the definition of the image is improved to a resolution of 512.

[0110] The first image 601 is subjected to face cutout to obtain a corresponding first face region image 602; the second image 603 is subjected to face cutout to obtain a corresponding second face region image 604. The plurality of first images 601 is input into a full-image feature extraction network 605 to obtain a plurality of full-image features; the plurality of first face region images 602 is input into an identity feature extraction network 606 to obtain a plurality of identity features; the identity features and the full-image features corresponding to the same first image 601 are spliced, and the plurality of spliced features after splicing is accumulated and averaged to obtain an identity image feature 609. The second image 603 is input into a posture feature extraction network 607 to obtain a posture feature; the second face region image 604 is input into an expression feature extraction network 608 to obtain an expression feature, and the posture feature and the expression feature are spliced to obtain a posture-expression feature 611. The identity image feature 609 and the posture-expression feature 611 are spliced to obtain a fusion feature 612, and the fusion feature 612 is input into a generator 613 to obtain a synthetic image 614. In the case of being configured with an MLP, the fusion feature 612 is input into the generator 613, and the fusion feature 612 is mapped into different depths of the generator 613 for processing through the MLP, and finally the generator 613 outputs the synthetic image 614.

[0111] The first image 601 (or the second image 603) and the synthetic image 614 are input into a first loss feature extraction network to calculate an identity feature loss; the second image 603 and the synthetic image 614 are input into a second loss feature extraction network to calculate a posture-expression feature loss; the first image 601 (or the second image 603) and the synthetic image 614 are input into a discriminator to calculate a real feature loss and a real generation loss; according to the identity feature loss, the posture-expression feature loss, the real feature loss and the real generation loss, a target loss is determined, and the parameters of the image generation model are adjusted according to the target loss, specifically the parameters of the full-image feature extraction network 605, the identity feature extraction network 606, the posture feature extraction network 607, the expression feature extraction network 608 and the generator 613.

[0112] On the basis of acquiring the full-image features, identity features are extracted for the face region image, which can better extract the identity features and extract more abundant identity features, reduce the sample quantity, reduce the k-shot times, reduce the training stage, improve the extraction accuracy of the identity features, and thus improve the image quality and accuracy of the image generation model. In addition, on the basis of acquiring the posture features of the whole person of the target person, the face region accounts for a small proportion, and the posture features have a low degree of fine granularity in encoding the micro-expression, i.e., a lot of detailed information of the micro-expression is lost. Expression features are extracted for the face region image, which can better extract the expression features and extract more abundant expression features, improve the extraction accuracy of the expression features, and thus improve the image quality and accuracy of the image generation model. Thus, according to the technical solution of the present disclosure, a better single-stage driving scheme for micro-expression is proposed, and a large-size image is used to improve the driving result clarity and solve the problem of top cutting and long hair cutting in historical matting, improve the clarity of the synthesized image, and improve the quality of the synthesized image.

[0113] Figure 7 is a flowchart of an image generation method according to an embodiment of the present disclosure. The embodiment can be applied to the case of training an image generation model. The method can be executed by an image generation device, which can be implemented in software and / or hardware and specifically configured in an electronic device with certain data operation capability. The electronic device can be a client device or a server device, such as a mobile phone, a tablet computer, a vehicle terminal, and a desktop computer.

[0114] S701, acquiring a person identity image of a first person, and processing the person identity image to obtain a face identity image of the person identity image.

[0115] The person identity image of the first person is used to provide identity information of the first person. The first person can be photographed in real time to collect at least one person identity image of the first person. The face identity image is used to provide more detailed identity information on the face of the first person. The person identity image is processed to obtain the face identity image of the person identity image, which can specifically be face recognition in the person identity image to obtain the face identity image. Generally, the face identity image is an image including the face of a complete person, such as the complete top of the hair and the bottom edge of the hair.

[0116] S702, acquiring a posture-expression image of a second person, and processing the posture-expression image to obtain a face-expression image of the posture-expression image.

[0117] The pose expression image of the second character is used to provide attribute information such as a pose expression of the second character. The pose expression image can be obtained from a pre-stored local library or a network. The facial expression image is used to provide more detailed expression information of the face of the second character. The pose expression image is processed to obtain a facial expression image of the pose expression image, and specifically, face recognition or face identification can be performed in the pose expression image to obtain the facial expression image.

[0118] S703, input the character identity image, the face identity image, the pose expression image and the facial expression image into a pre-trained image generation model to obtain a synthesized image; wherein the image generation model is trained according to the training method of the image generation model as described in any embodiment of the present disclosure.

[0119] The synthesized image is an image of the first character driven by the expression and pose of the second character. The image generation model is trained according to the training method of the image generation model.

[0120] The pose expression image of a different character or the pose expression image of the same character can be selected and input into the image generation model according to the character identity image of the first character to obtain a synthesized image of the first character in different pose expressions, thereby quickly obtaining images of the first character in different pose expressions, meeting the diversified needs of users and improving user experience.

[0121] According to the technical solution of the present disclosure, by inputting the character identity image, the face identity image, the pose expression image and the facial expression image into a pre-trained image generation model to obtain a synthesized image, the image of the first character in the pose expression of the second character can be accurately generated, face driving is realized, the accuracy of image generation by face driving is improved, the efficiency of image generation by face driving is improved, and the labor cost of image generation by face driving is reduced.

[0122] According to an embodiment of the present disclosure, Figure 8 is a structural diagram of a training device of an image generation model in an embodiment of the present disclosure. The present embodiment is applicable to training an image generation model, wherein the image generation model is used in the case of converting a source domain style word into a target domain style word. The device is implemented by software and / or hardware, and is specifically configured in an electronic device with certain data operation capability.

[0123] As shown in Figure 8 An image generation model training device 800, comprising: an image preprocessing module 801, a model output acquisition module 802 and a first adjustment parameter module 803; wherein,

[0124] The image preprocessing module 801 is configured to acquire a first image of a target person, a first facial region image in the first image, a second image, and a second facial region image in the second image.

[0125] The model output acquisition module 802 is configured to input the first image, the first facial region image, the second image, and the second facial region image into an image generation model to obtain a synthesized image.

[0126] The first adjustment parameter module 803 is configured to input the first image, the second image, and the synthesized image into a pre-trained loss calculation model to calculate a target loss, and adjust parameters of the image generation model according to the target loss.

[0127] According to the technical solution of the present disclosure, by acquiring the first image of the target person, the first facial region image in the first image, the second image, and the second facial region image in the second image as image samples, inputting them into the image generation model to obtain the synthesized image, and calculating the target loss according to the first image, the second image, and the synthesized image, the parameters of the image generation model are adjusted, the face details in the image samples are increased, the face contents in the samples are enriched, the difference between the information in the synthesized image and the true value information is introduced through the additional loss calculation model, so that the image generation model learns more face details of the person, the ability of the image generation model to learn the face features is improved, and the image generation accuracy of the image generation model is improved.

[0128] Further, the model output acquisition module 802 includes: an identity image feature extraction unit configured to input the first image and the first facial region image into an identity image feature extraction network in the image generation model to obtain identity image features output by the identity image feature extraction network; a posture expression feature extraction unit configured to input the second image and the second facial region image into a posture expression feature extraction network in the image generation model to obtain posture expression features output by the posture expression feature extraction network; a first feature fusion unit configured to fuse the identity image features and the posture expression features to obtain fused features; and a synthesized image output unit configured to input the fused features into a generator in the image generation model to obtain a synthesized image output by the generator.

[0129] Further, the first feature fusion unit includes: a feature splicing subunit configured to splice the identity image features and the posture expression features to obtain the fused features.

[0130] Further, the identity image feature extraction unit comprises: a full-image feature extraction subunit, configured to input the first image into a full-image feature extraction network in the image generation model to obtain full-image features output by the full-image feature extraction network; an identity feature extraction subunit, configured to input the first face region image into an identity feature extraction network in the image generation model to obtain identity features output by the identity feature extraction network; and a second feature fusion subunit, configured to fuse the full-image features and the identity features to obtain identity image features.

[0131] Further, the pose-expression feature extraction unit comprises: a pose feature extraction subunit, configured to input the second image into a pose feature extraction network in the image generation model to obtain pose features output by the pose feature extraction network; an expression feature extraction subunit, configured to input the second face region image into an expression feature extraction network in the image generation model to obtain expression features output by the expression feature extraction network; and a third feature fusion subunit, configured to fuse the pose features and the expression features to obtain pose-expression features.

[0132] Further, the first adjustment parameter module 803 comprises: an identity feature loss calculation unit, configured to input the first image and the synthesized image into a first loss feature extraction network to calculate an identity feature loss; a pose-expression feature loss calculation unit, configured to input the second image and the synthesized image into a second loss feature extraction network to calculate a pose-expression feature loss; a ground truth loss calculation unit, configured to input the second image and the synthesized image into a discriminator to calculate a real feature loss and a real generation loss; and a target loss calculation unit, configured to determine a target loss according to the identity feature loss, the pose-expression feature loss, the real feature loss, and the real generation loss.

[0133] Further, the image preprocessing module 801 comprises: a video stream acquisition unit, configured to acquire a video stream of a target person; and an image analysis unit, configured to determine at least one image as the first image and an image different from the first image as the second image from the video stream.

[0134] The training device of the image generation model can execute the training method of the image generation model provided by any embodiment of the present disclosure, and has the corresponding function modules and beneficial effects of executing the training method of the image generation model.

[0135] According to an embodiment of the present disclosure, Figure 9is a structural diagram of an image generation apparatus in the embodiment of the present disclosure, and the embodiment of the present disclosure is applicable to the case of migrating the pose expression of a second person to the face of a first person to obtain a synthesized image in which the first person displays the pose expression of the second person. The apparatus is implemented by software and / or hardware and is specifically configured in an electronic device with certain data operation capability.

[0136] As shown in an image generation apparatus 900, the image generation apparatus 900 comprises an identity image acquisition module 901, a pose expression image acquisition module 902, and an image synthesis module 903. Figure 9

[0137] The identity image acquisition module 901 is configured to acquire a person identity image of a first person, and process the person identity image to obtain a face identity image of the person identity image.

[0138] The pose expression image acquisition module 902 is configured to acquire a pose expression image of a second person, and process the pose expression image to obtain a facial expression image of the pose expression image.

[0139] The image synthesis module 903 is configured to input the person identity image, the face identity image, the pose expression image, and the facial expression image into a pre-trained image generation model to obtain a synthesized image, wherein the image generation model is trained according to the training method of the image generation model as described in any embodiment of the present disclosure.

[0140] According to the technical solution of the present disclosure, by inputting the person identity image, the face identity image, the pose expression image, and the facial expression image into the pre-trained image generation model to obtain the synthesized image, the image of the first person with the pose expression of the second person can be accurately generated, the face driving is realized, the image generation accuracy of the face driving is improved, the image generation efficiency of the face driving is improved, and the image generation labor cost of the face driving is reduced.

[0141] The image generation apparatus described above can execute the image generation method provided by any embodiment of the present disclosure, and has the corresponding function modules and beneficial effects of executing the image generation method.

[0142] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision, and disclosure of user personal information comply with relevant laws and regulations and do not violate public order and good customs.

[0143] According to the embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium, and a computer program product.

[0144] Figure 10 ​A schematic area diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0145] like Figure 10 As shown, device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1002 or a computer program loaded from storage unit 1008 into random access memory (RAM) 1003. The RAM 1003 may also store various programs and data required for the operation of device 1000. The computing unit 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. Input / output (I / O) interface 1005 is also connected to bus 1004.

[0146] Multiple components in device 1000 are connected to I / O interface 1005, including: input unit 1006, such as keyboard, mouse, etc.; output unit 1007, such as various types of monitors, speakers, etc.; storage unit 1008, such as disk, optical disk, etc.; and communication unit 1009, such as network card, modem, wireless transceiver, etc. Communication unit 1009 allows device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0147] The computing unit 1001 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 performs various methods and processes described above, such as the training method of the image generation model or the image generation method. For example, in some embodiments, the training method of the image generation model or the image generation method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 1008. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded onto the RAM 1003 and executed by the computing unit 1001, one or more steps of the training method of the image generation model or the image generation method described above can be performed. Alternatively, in other embodiments, the computing unit 1001 can be configured to perform the training method of the image generation model or the image generation method by any other suitable means, such as by means of firmware.

[0148] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0149] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or the block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0150] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0151] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0152] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0153] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0154] It should be understood that the various forms of flow shown above can be re-ordered, added to, or have steps deleted, using the flow. For example, the steps described in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired results of the technical solutions of the present disclosure can be achieved, which are not limited herein.

[0155] The above detailed description does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for training an image generation model, comprising: obtaining a first image of a target person, a first facial region image in the first image, a second image, and a second facial region image in the second image; inputting the first image, the first facial region image, the second image, and the second facial region image into an image generation model to obtain a synthesized image; wherein the synthesized image is obtained by fusing an identity image feature and a pose-expression feature; the identity image feature is obtained by providing global image information by the first image and detailed image information by the first facial region image; the identity image feature is obtained by fusing a full-image feature extracted from the first image and an identity feature obtained from the first facial region image; the pose-expression feature is obtained by providing global image information by the second image and detailed image information by the second facial region image; the pose-expression feature is obtained by fusing a pose feature extracted from the second image and an expression feature extracted from the second facial region image; calculating a target loss by inputting the first image, the second image, and the synthesized image into a pre-trained loss calculation model, and adjusting parameters of the image generation model according to the target loss; the target loss is determined based on an identity feature loss, a pose-expression feature loss, a real feature loss, and a real generation loss; the real feature loss is used to describe a real image feature difference; the real generation loss is used to describe a difference between a discrimination result of the synthesized image and a discrimination result of a model-generated image, and a difference between a discrimination result of a real image and a discrimination result of a real image.

2. The method of claim 1, wherein, the inputting the first image, the first facial region image, the second image, and the second facial region image into the image generation model to obtain the synthesized image comprises: inputting the first image and the first facial region image into an identity image feature extraction network in the image generation model to obtain an identity image feature output by the identity image feature extraction network; inputting the second image and the second facial region image into a pose-expression feature extraction network in the image generation model to obtain a pose-expression feature output by the pose-expression feature extraction network; fusing the identity image feature and the pose-expression feature to obtain a fused feature; inputting the fused feature into a generator in the image generation model to obtain a synthesized image output by the generator.

3. The method of claim 2, wherein, the fusing the identity image feature and the pose-expression feature to obtain the fused feature comprises: splicing the identity image feature and the pose-expression feature to obtain the fused feature.

4. The method of claim 2, wherein, the inputting the first image and the first facial region image into the identity image feature extraction network in the image generation model to obtain the identity image feature output by the identity image feature extraction network comprises: inputting the first image into a full-image feature extraction network in the image generation model to obtain full-image features output by the full-image feature extraction network; inputting the first face region image into an identity feature extraction network in the image generation model to obtain identity features output by the identity feature extraction network; fusing the full-image features and the identity features to obtain identity image features.

5. The method of claim 2, wherein, The inputting the second image and the second face region image into the pose-expression feature extraction network in the image generation model to obtain pose-expression features output by the pose-expression feature extraction network comprises: inputting the second image into a pose feature extraction network in the image generation model to obtain pose features output by the pose feature extraction network; inputting the second face region image into an expression feature extraction network in the image generation model to obtain expression features output by the expression feature extraction network; fusing the pose features and the expression features to obtain pose-expression features.

6. The method of claim 1, wherein, The calculating a target loss by inputting the first image, the second image and the synthesized image into a pre-trained loss calculation model comprises: inputting the first image and the synthesized image into a first loss feature extraction network to calculate identity feature loss; inputting the second image and the synthesized image into a second loss feature extraction network to calculate pose-expression feature loss; inputting the second image and the synthesized image into a discriminator to calculate real feature loss and real generation loss; determining the target loss according to the identity feature loss, the pose-expression feature loss, the real feature loss and the real generation loss.

7. The method of claim 1, wherein, The obtaining the first image and the second image of the target person comprises: obtaining a video stream of the target person; determining at least one image as the first image and determining an image different from the first image as the second image from the video stream.

8. An image generation method, comprising: obtaining a person identity image of a first person and processing the person identity image to obtain a face identity image of the person identity image; obtaining a pose-expression image of a second person and processing the pose-expression image to obtain a face expression image of the pose-expression image; inputting the person identity image, the face identity image, the pose-expression image and the face expression image into a pre-trained image generation model to obtain a synthesized image; wherein the image generation model is trained according to the training method of the image generation model in any one of claims 1 to 7.

9. A training device of an image generation model, comprising: an image preprocessing module configured to obtain a first image of a target person, a first face region image in the first image, a second image and a second face region image in the second image; The model output obtaining module is configured to input the first image, the first face region image, the second image, and the second face region image into an image generation model to obtain a synthesized image; the synthesized image is obtained by fusing an identity image feature and a posture expression feature; the identity image feature is obtained by providing global image information by the first image and providing detailed image information by the first face region image; the identity image feature is obtained by fusing a full-image feature extracted from the first image and an identity feature obtained from the first face region image; the posture expression feature is obtained by providing global image information by the second image and providing detailed image information by the second face region image; the posture expression feature is obtained by fusing a posture feature extracted from the second image and an expression feature extracted from the second face region image. The first adjustment parameter module is configured to input the first image, the second image, and the synthesized image into a pre-trained loss calculation model to calculate a target loss, and adjust parameters of the image generation model according to the target loss. The target loss is determined based on an identity feature loss, a posture expression feature loss, a real feature loss, and a real generation loss; the real feature loss is used to describe a real image feature difference; the real generation loss is used to describe a difference between a discrimination result of the synthesized image and a discrimination result of a model generated image, and a difference between a discrimination result of a real image and a discrimination result of a real image.

10. The apparatus of claim 9, wherein, The model output obtaining module comprises: The identity image feature extraction unit is configured to input the first image and the first face region image into an identity image feature extraction network in the image generation model to obtain an identity image feature output by the identity image feature extraction network. The posture expression feature extraction unit is configured to input the second image and the second face region image into a posture expression feature extraction network in the image generation model to obtain a posture expression feature output by the posture expression feature extraction network. The first feature fusion unit is configured to fuse the identity image feature and the posture expression feature to obtain a fused feature. The synthesized image output unit is configured to input the fused feature into a generator in the image generation model to obtain a synthesized image output by the generator.

11. The apparatus of claim 10, wherein, The first feature fusion unit comprises: The feature splicing subunit is configured to splice the identity image feature and the posture expression feature to obtain a fused feature.

12. The apparatus of claim 10, wherein, The identity image feature extraction unit comprises: The full-image feature extraction subunit is configured to input the first image into a full-image feature extraction network in the image generation model to obtain a full-image feature output by the full-image feature extraction network. The identity feature extraction subunit is configured to input the first face region image into an identity feature extraction network in the image generation model to obtain an identity feature output by the identity feature extraction network. The second feature fusion subunit is configured to fuse the full-image feature and the identity feature to obtain an identity image feature.

13. The apparatus of claim 10, wherein, The pose-expression feature extraction unit includes: The pose feature extraction subunit is configured to input the second image into a pose feature extraction network in the image generation model to obtain a pose feature output by the pose feature extraction network; The expression feature extraction subunit is configured to input the second face region image into an expression feature extraction network in the image generation model to obtain an expression feature output by the expression feature extraction network; The third feature fusion subunit is configured to fuse the pose feature and the expression feature to obtain a pose-expression feature.

14. The apparatus of claim 9, wherein, The first adjustment parameter module includes: The identity feature loss calculation unit is configured to input the first image and the synthesized image into a first loss feature extraction network to calculate an identity feature loss; The pose-expression feature loss calculation unit is configured to input the second image and the synthesized image into a second loss feature extraction network to calculate a pose-expression feature loss; The ground truth loss calculation unit is configured to input the second image and the synthesized image into a discriminator to calculate a real feature loss and a real generation loss; The target loss calculation unit is configured to determine a target loss according to the identity feature loss, the pose-expression feature loss, the real feature loss, and the real generation loss.

15. The apparatus of claim 9, wherein, The image preprocessing module includes: The video stream acquisition unit is configured to acquire a video stream of a target person; The image analysis unit is configured to determine at least one image as a first image and an image different from the first image as a second image from the video stream.

16. An image generation apparatus, comprising: The identity image acquisition module is configured to acquire a person identity image of a first person, and process the person identity image to obtain a face identity image of the person identity image; The pose-expression image acquisition module is configured to acquire a pose-expression image of a second person, and process the pose-expression image to obtain a face expression image of the pose-expression image; The image synthesis module is configured to input the person identity image, the face identity image, the pose-expression image, and the face expression image into a pre-trained image generation model to obtain a synthesized image; wherein the image generation model is trained according to the training method of the image generation model in any one of claims 1 to 7.

17. An electronic device, comprising: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the training method of the image generation model in any one of claims 1 to 7, or the image generation method in claim 8.

18. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the training method of the image generation model in any one of claims 1 to 7, or the image generation method in claim 8.

19. A computer program product comprising a computer program which, when executed by a processor, implements the method of training an image generation model according to any one of claims 1-7, or the method of image generation according to claim 8.

Images