Driving method of face image and training method and related device of related model

Abstract

This application discloses a face image driving method and related model training method and apparatus. The image generation network model includes an image segmentation network, a structured generation network, and an unstructured generation network. The training method for the image generation network model includes: acquiring a portrait image training set containing a first portrait image and a second portrait image of the same target; obtaining corresponding structured region images and first unstructured region images of the first and second portrait images respectively through an image segmentation network; generating a predicted target image using the structured generation network and the unstructured generation network based on the corresponding structured region images and the first unstructured region images of the first and second portrait images; and adjusting the network parameters in the image generation network model based on the loss function between the second portrait image and the predicted target image. This scheme can improve the face image driving effect.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method for driving a face image and a method for training a related model and a related device. Background Art

[0002] Facial animation is widely used in many fields such as computer animation industry, game industry, remote conferencing, agents and avatars, and has been a hot topic of research at home and abroad in recent years. Based on single-image facial animation, the geometric position of a given image is transformed to produce some interesting visual effects, which is often used to produce stunts in film and television entertainment and advertising design.

[0003] Face animation is produced by deforming the image based on the real face picture, so it has a high sense of reality. However, the image obtained by the existing face picture driving method is not good in the unstructured area. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a driving method for face images and a training method and related devices for related models, which can improve the driving effect of face images.

[0005] In order to solve the above problems, the first aspect of the present application provides a training method for an image generation network model, wherein the image generation network model includes an image segmentation network, a structured generation network and an unstructured generation network; the training method for the image generation network model includes: obtaining a portrait image training set; wherein the portrait image training set includes a first portrait image and a second portrait image of the same target; using an image segmentation network to segment the first portrait image and the second portrait image respectively to obtain a first structured area image and a first unstructured area image corresponding to the first portrait image, and a second structured area image and a second unstructured area image corresponding to the second portrait image; based on the first unstructured area image and the second unstructured area image, using an unstructured generation network to generate a preliminary background image; based on the first structured area image and the second structured area image, using a structured generation network to generate a preliminary face image; based on the preliminary face image and the preliminary background image, generating a predicted target image; obtaining a loss function between the second portrait image and the predicted target image; and adjusting the network parameters in the image generation network model based on the loss function.

[0006] In order to solve the above-mentioned problem, the second aspect of the present application provides a driving method for a face image, which comprises: obtaining a source image and a driving image; inputting the source image and the driving image into an image generation network model to obtain a target image; wherein the image generation network model is trained by the training method of the image generation network model of the first aspect mentioned above.

[0007] In order to solve the above problems, the third aspect of the present application provides a training device for an image generation network model, wherein the image generation network model includes an image segmentation network, a structured generation network and an unstructured generation network; the training device for the image generation network model includes: a sample acquisition module, wherein the sample acquisition module is used to acquire a portrait image training set; wherein the portrait image training set includes a first portrait image and a second portrait image of the same target; an image prediction module, wherein the image prediction module is used to use an image segmentation network to segment the first portrait image and the second portrait image respectively, to obtain a first structured region image and a first unstructured region image corresponding to the first portrait image, and a second portrait image a second structured area image and a second unstructured area image corresponding to the first portrait image and the second unstructured area image; generating a preliminary background image using an unstructured generation network based on the first unstructured area image and the second unstructured area image; generating a preliminary face image using a structured generation network based on the first structured area image and the second structured area image; and generating a predicted target image based on the preliminary face image and the preliminary background image; a loss function determination module, the loss function determination module is used to obtain the loss function between the second portrait image and the predicted target image; a parameter adjustment module, the parameter adjustment module is used to adjust the network parameters in the image generation network model based on the loss function.

[0008] In order to solve the above-mentioned problem, the fourth aspect of the present application provides a driving device for a face image, and the driving device for a face image comprises: an image acquisition module, which is used to acquire a source image and a driving image; an image generation module, which is used to input the source image and the driving image into an image generation network model to obtain a target image; wherein the image generation network model is trained by the training method of the image generation network model of the first aspect mentioned above.

[0009] In order to solve the above problems, the fifth aspect of the present application provides an electronic device, including a memory and a processor coupled to each other, and the processor is used to execute program instructions stored in the memory to implement the training method of the image generation network model of the first aspect above, or the driving method of the face image of the second aspect above.

[0010] In order to solve the above problems, the sixth aspect of the present application provides a computer-readable storage medium on which program instructions are stored. When the program instructions are executed by a processor, the training method of the image generation network model of the first aspect or the driving method of the face image of the second aspect is implemented.

[0011] The beneficial effects of the present invention are as follows: different from the prior art, the image generation network model of the present application includes an image segmentation network, a structured generation network and an unstructured generation network, by obtaining a portrait image training set, wherein the portrait image training set includes a first portrait image and a second portrait image of the same target; then the first portrait image and the second portrait image are segmented by using the image segmentation network respectively to obtain a first structured region image and a first unstructured region image corresponding to the first portrait image, and a second structured region image and a second unstructured region image corresponding to the second portrait image; then based on the first unstructured region image and the second unstructured region image, a preliminary background image is generated by using the unstructured generation network, and based on the first structured region image and the second structured region image, a preliminary face image is generated by using the structured generation network; then based on the preliminary face image and the preliminary background image, a predicted target image is generated; then a loss function between the second portrait image and the predicted target image can be obtained, and based on the loss function, the network parameters in the image generation network model are adjusted. In the present application, after using an image segmentation network to segment the first portrait image and the second portrait image respectively, and obtaining the first structured area image and the first unstructured area image corresponding to the first portrait image, and the second structured area image and the second unstructured area image corresponding to the second portrait image, a preliminary face image can be generated through a structured generation network, and a preliminary background image can be generated through an unstructured generation network, so that the predicted target image generated based on the preliminary face image and the preliminary background image can process the face part and the background part separately, which is conducive to each learning a better action representation and avoiding mutual influence. It can not only accurately control the driving effect of the face key point area, but also cover the area outside the face key points, and can also obtain better driving effect in the unstructured key point area. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a flowchart of an embodiment of a training method for an image generation network model of the present application;

[0013] Figure 2 yes Figure 1 A schematic diagram of a flow chart of an embodiment of step S13;

[0014] Figure 3 yes Figure 1 A schematic diagram of a flow chart of an embodiment of step S14;

[0015] Figure 4 It is a flow chart of an embodiment of a driving method for a face image of the present application;

[0016] Figure 5 yes Figure 4 A schematic diagram of a flow chart of an embodiment of step S52;

[0017] Figure 6 It is a schematic diagram of the framework of an embodiment of a training device for an image generation network model of the present application;

[0018] Figure 7 It is a schematic diagram of the framework of an embodiment of the driving device for the face image of the present application;

[0019] Figure 8 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application;

[0020] Fig. 9 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0021] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0022] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0023] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0024] See also Figure 1 , Figure 1 The figure is a flow chart of an embodiment of a training method for an image generation network model of the present application. The image generation network model includes an image segmentation network, a structured generation network, and an unstructured generation network; specifically, the training method for the image generation network model in the present embodiment may include the following steps:

[0025] Step S11: Acquire a portrait image training set; wherein the portrait image training set includes a first portrait image and a second portrait image of the same target.

[0026] Specifically, two different frames of images having a portrait region of the target can be obtained in a video of the same target, namely a first portrait image and a second portrait image, wherein the first portrait image includes the face of the target, and the expression of the face is the source expression before migration, and the second portrait image also includes the face of the target, and the expression of the face is the target expression after migration. Using the image generation network model of the present application, the expression of the target in the first portrait image can be switched from the source expression to the target expression of the second portrait image.

[0027] Step S12: using an image segmentation network to segment the first portrait image and the second portrait image respectively, to obtain a first structured area image and a first unstructured area image corresponding to the first portrait image, and a second structured area image and a second unstructured area image corresponding to the second portrait image.

[0028] The image segmentation network can divide the image into several specific regions with unique properties and propose targets of interest. For example, different images have different grayscales, and there are generally obvious edges at the boundaries. This feature can be used to segment the image. The edge-based segmentation method is used to detect the places where the grayscale or structure in the image has a sudden change through edge detection, indicating where one area ends and another area begins. Therefore, after the first portrait image is input into the image segmentation network, the first structured area image and the first unstructured area image corresponding to the first portrait image can be output. After the second portrait image is input into the image segmentation network, the second structured area image and the second unstructured area image corresponding to the second portrait image can be output.

[0029] Step S13: Based on the first unstructured region image and the second unstructured region image, a preliminary background image is generated using an unstructured generation network.

[0030] The unstructured generative network is trained in a self-supervised manner. Self-supervised learning mainly uses auxiliary tasks to mine its own supervisory information from large-scale unsupervised data, and trains the network through this constructed supervisory information, so that it can learn representations that are valuable for downstream tasks. In other words, the supervisory information of self-supervised learning is not manually labeled, but the algorithm automatically constructs supervisory information in large-scale unsupervised data to perform supervised learning or training.

[0031] Please combine Figure 2 , Figure 2 yes Figure 1 Schematic diagram of a flow chart of an embodiment of step S13 in FIG. In one embodiment, the unstructured generation network includes a resnet-tiny subnetwork and an optical flow prediction subnetwork; the above step S13 may specifically include:

[0032] Step S131: using the resnet-tiny sub-network to detect the first unstructured area image and the second unstructured area image respectively, to obtain a heat map of the first unstructured key points and a heat map of the second unstructured key points.

[0033] The first unstructured area image and the second unstructured area image are detected by using a resnet-tiny sub-network to obtain a heat map of the first unstructured key point and a heat map of the second unstructured key point; a heat map is an analysis method that uses different marks to mark and present the areas on the image or page according to the degree of attention, and the marking method generally adopts the depth of color, the density of points and the form of presentation weight. The heat map of the first unstructured key point and the heat map of the second unstructured key point of the present application are used to characterize the position information of the unstructured key point in the first portrait image and the second portrait image. Optionally, the unstructured key point includes other areas other than the facial features, such as the hair area. The network structure of the resnet-tiny sub-network can be determined according to the number of key points that need to be predicted. For example, when the number of key points that need to be predicted is 3, a smaller network can be used, and the resnet-tiny sub-network can be composed of 3 downsampling blocks and three upsampling blocks, wherein the number of channels of each block is 128.

[0034] Step S132: inputting the heat map of the first unstructured key point and the heat map of the second unstructured key point into the optical flow prediction subnetwork to generate optical flow data.

[0035] The optical flow prediction subnetwork takes the heat map of the first unstructured key point and the heat map of the second unstructured key point as input information, performs optical flow estimation on the first unstructured key point of the first portrait image and the second unstructured key point of the second portrait image, and obtains optical flow data from the first portrait image to the second portrait image. Specifically, the optical flow prediction subnetwork can be composed of an encoder and a decoder, which inputs the heat map of the six key points (the heat map of the first unstructured key point Figure 3 Zhang, Hotspot of the second unstructured keypoint Figure 3 Finally, the optical flow data is output, and its shape is (64, 64, 2).

[0036] Step S133: deforming the first unstructured region image using the optical flow data to obtain a deformed first unstructured region image as a preliminary background image.

[0037] Specifically, the first unstructured area image corresponding to the first portrait image and the optical flow data from the first portrait image to the second portrait image are mapped, and the optical flow data can be mapped to the first unstructured area image, so that the deformed first unstructured area image obtained contains the optical flow data. Therefore, when the expression of the second portrait image is subsequently migrated to the first portrait image, the generated predicted target image can avoid problems such as distortion, abnormal texture and blur, and the overall quality of the predicted target image can be improved.

[0038] Furthermore, the objective loss function of the optical flow prediction subnetwork training includes a total variation loss function. Specifically, regarding the training of the optical flow prediction subnetwork, the optical flow data can be trained with a total variation loss function (TV loss, TotalVariation loss). In the image fusion process, a little noise on the image may have a very large impact on the generated result. At this time, some regularization terms can be added to the optical flow prediction subnetwork to maintain the smoothness of the image. For example, TV loss is a commonly used regularization term, which helps to avoid the initial background image obtained using the optical flow data due to the clutter of the optical flow data.

[0039] Furthermore, the objective loss function of the unstructured generative network training includes at least one of a distance loss function and an equal constraint loss function.

[0040] Specifically, regarding the training of the unstructured generative network, at the beginning of the training, the network parameters are randomly initialized, which may cause the extracted key points to cluster in the center, or in the later stage of training, the output of the unstructured generative network is the same as the output of the structured generative network, which does not play a role in detecting unstructured key points; therefore, the distance loss function L can be added sep ,Right now:

[0041]

[0042] Among them, σ is the distance standard deviation of the face key points in the training set images. By giving a priori distance standard deviation, we can avoid the distance loss function L sep Too large or too small. By setting the distance loss function L sep , each unstructured key point can be compared with all key points (including face key points and unstructured key points) to make all key points non-repetitive, so as to avoid the unstructured key points detected by the unstructured generation network from being clustered together, or the face key points being detected.

[0043] In addition, during the self-supervised learning training process of the unstructured generative network, it is easy to learn randomly and may output some key points that are irrelevant to the input image; therefore, an equivariance constraint loss function can be added. The central idea is that the key points detected from the image after affine transformation should be the same as those detected from the image that has not undergone affine transformation after the inverse affine transformation. Therefore, by adding the equivariance constraint loss, a certain key point should still be able to be well located when the coordinates are transformed in another image.

[0044] Step S14: Based on the first structured region image and the second structured region image, a preliminary face image is generated using a structured generation network.

[0045] The structured generative network is trained in a supervised manner. Specifically, the structured generative network is a deep neural network. A large number of labeled face images are used to train the structured generative network. Back propagation is performed after the prediction of the structured generative network and the real label are lost. Through continuous learning, the structured generative network can eventually acquire the ability to generate a preliminary face image based on the first structured region image and the second structured region image.

[0046] Please combine Figure 3 , Figure 3 yes Figure 1 Schematic diagram of a flow chart of an embodiment of step S14 in FIG. In one embodiment, the structured generation network includes a face key point sub-network and a HRNet sub-network; the above step S14 may specifically include:

[0047] Step S141: respectively using the facial key point sub-network and the HRNet sub-network to obtain the first facial key points and their Jacobian matrix corresponding to the first structured area image, and the second facial key points and their Jacobian matrix corresponding to the second structured area image.

[0048] The first structured area image and the second structured area image are detected using the facial key point sub-network to obtain the first facial key point corresponding to the first structured area image and the second facial key point corresponding to the second structured area image, and the first facial key point and the second facial key point are rasterized. Rasterization is the process of converting geometric data into pixels after a series of transformations, so as to present them on a display device. The essence of rasterization is coordinate transformation and geometric discretization, so the first facial key point and the second facial key point can be obtained after rasterization. In addition, the Jacobian matrix of the key point can be predicted through the HRNet sub-network. The Jacobian matrix can provide the direction information of the key point. Therefore, the first facial key point Keypoint corresponding to the first structured area image can be obtained using the facial key point sub-network and the HRNet sub-network.src Its Jacobian matrix Jacob src , the second facial key point corresponding to the second structured area image driver Its Jacobian matrix Jacob driver .

[0049] Step S142: generating a facial key point connection graph based on the first facial key points and their Jacobian matrix, and the second facial key points and their Jacobian matrix.

[0050] Step S143: generating the preliminary face image based on the first portrait image and the face key point connection diagram.

[0051] In the application of the actual image generation network model, the face of the second portrait image may be inconsistent with the first portrait image. If the second face key points of the second portrait image are directly used, the generated preliminary face image will be deformed. Therefore, it is necessary to use the Jacobian matrix to adjust the motion information of the second portrait image and then transfer it to the first face key points of the first portrait image to reduce the deformation of the generated preliminary face image. Therefore, the key points are connected to form a face key point connection diagram, that is, based on the first face key point src Its Jacobian matrix Jacob src , Second face key point Keypoint driver Its Jacobian matrix Jacob driver , generate the face key point connection diagram, the formula is as follows:

[0052] kp_new=(Keypoint src -Keypoint driver )(Jaco src Jaco driver -1 )+

[0053] Keypoint src Then connect the first portrait image and the facial key points Figure 1 The encoder and decoder are input together to generate a preliminary face image.

[0054] Step S15: Generate a predicted target image based on the preliminary face image and the preliminary background image.

[0055] After obtaining the preliminary face image and the preliminary background image, the preliminary face image and the preliminary background image may be fused to obtain a predicted target image.

[0056] In one embodiment, the image generation network model further includes a residual generation network; in this case, after generating a face key point connection diagram, the residual generation network can be used to generate a residual image based on the first portrait image and the face key point connection diagram. Therefore, the above step S15 specifically includes: fusing the preliminary face image, the preliminary background image and the residual image to obtain the predicted target image.

[0057] It is understandable that since the image is divided into the face part of the structured area and the background part of the unstructured area for processing, there will be some missing areas during fusion. Therefore, the residual generation network can fill the missing parts to adjust the fusion boundary. Specifically, the first portrait image and the face key point connection diagram are input as input to the encoder and decoder, so that a residual image can be generated; then the preliminary face image, the preliminary background image and the residual image are fused to obtain the final predicted target image.

[0058] Furthermore, the image generation network model also includes a discriminator network. After the above step S15, it can also include: optimizing the predicted target image using the discriminator network so that the discriminator network cannot distinguish between the predicted target image and the second portrait image.

[0059] The discriminator network is used to distinguish whether the input predicted target image is a real second portrait image or a predicted target image generated by the image generation network model; specifically, in the training stage, the entire image generation network model takes the "real second portrait image" as the target and generates a predicted target image similar to the "real second portrait image", and the discriminator network judges the authenticity of the generated predicted target image based on the "real second portrait image". The two compete with each other and finally reach a balance, that is, the image generation network model can obtain a realistic predicted target image, and the discriminator network cannot judge its authenticity.

[0060] Step S16: Obtain a loss function between the second portrait image and the predicted target image.

[0061] Step S17: Based on the loss function, adjust the network parameters in the image generation network model.

[0062] The loss function is used to measure the degree of deviation between the prediction made by the model and the true value (Ground Truth), and usually the objective function needs to be minimized. Specifically, after obtaining the loss function, it can be determined whether the current loss function converges; if so, stop training the image generation network model to obtain the trained image generation network model; if not, use the back propagation algorithm to adjust the network parameters in the image generation network model, and continue to train the image generation network model until the loss function converges.

[0063] Specifically, the loss function may include an L1 loss function, a perceptual loss function, and a style loss function.

[0064] Among them, the L1 loss function (L1 Loss) is also called Mean Absolute Error, or mean absolute error (MAE), which measures the average error amplitude of the distance between the predicted value and the true value, ranging from 0 to positive infinity; while the perceptual loss function (Perceptual Losses) measures the mean square error of the distance between the predicted value and the true value. By minimizing the L1 loss function and the perceptual loss function during the training process, the predicted target image output by the image generation network model can have a better driving effect.

[0065] The style loss function is the key to whether the style of an image can be accurately and effectively transferred to another image. Usually, the style loss function is determined by the difference between the style features of the image. It can be understood that the first portrait image and the second portrait image are two different frames of images obtained from the video of the same target. Since the camera shooting the video may move, the first portrait image and the second portrait image may have changes in the background area in addition to the portrait area. The change in the background area will cause the image generation network model to be unable to learn. The reason is that since the background area is too complex, the image generation network model cannot represent the background area through key points, so the image generation network model does not know how the background area should be generated. Therefore, it is necessary to add a style loss function. When the background area changes, the optimization direction is to generate a predicted target image that is consistent with the style of the first portrait image.

[0066] The above scheme, by using the image segmentation network to segment the first portrait image and the second portrait image respectively, obtains the first structured area image and the first unstructured area image corresponding to the first portrait image, and the second structured area image and the second unstructured area image corresponding to the second portrait image, and then generates a preliminary face image through a structured generation network, and generates a preliminary background image through an unstructured generation network, so that the predicted target image generated based on the preliminary face image and the preliminary background image can process the face part and the background part separately, which is conducive to each learning better action representation and avoiding mutual influence. It can not only accurately control the driving effect of the face key point area, but also cover the area outside the face key points, and can also obtain better driving effect in the unstructured key point area.

[0067] See also Figure 4 , Figure 4 1 is a flow chart of an embodiment of a method for driving a face image of the present application. Specifically, the method for driving a face image in the present embodiment may include the following steps:

[0068] Step S41: Obtain source image and driver image.

[0069] Step S42: Input the source image and the driving image into an image generation network model to obtain a target image.

[0070] Wherein, the image generation network model is trained by the above-mentioned image generation network model training method.

[0071] Specifically, the face included in the source image and the face included in the driving image may have the same identity, that is, the face in the source image and the face in the driving image are the face of the same person. Alternatively, they may have different identities, that is, the face in the source image and the face in the driving image are not the face of the same person. The image generation network model of the present application can output accurate target images for both of the above situations.

[0072] The source image includes the face of the target object, and the expression of the face is the source expression before migration. The driving image includes the face of the driving object, and the expression of the face is the target expression after migration. The target object and the driving object can be the same or different. The target image is generated by the image generation network model, and the expression of the target object in the source image can be switched from the source expression to the target expression of the driving image. It can be understood that the face in the target image generated by the image generation network model is still the face of the target object, only the facial expression of the target object is switched from the source expression to the target expression of the driving image. For example, the facial expression of the target object in the source image is a crying expression, and the facial expression of the driving image is a laughing expression, then the target image generated by the image generation network model is an image containing the laughing expression of the target object.

[0073] Furthermore, the image generation network model includes an image segmentation network, a structured generation network and an unstructured generation network, wherein the unstructured generation network includes a resnet-tiny subnetwork and an optical flow prediction subnetwork, and the structured generation network includes a face key point subnetwork and an HRNet subnetwork. Figure 5 , Figure 5 yes Figure 4 FIG. 1 is a flow chart of an embodiment of step S42 in FIG. 1. In one embodiment, the above step S42 may specifically include:

[0074] Step S421: Use an image segmentation network to segment the source image and the driving image respectively to obtain a first structured area image and a first unstructured area image corresponding to the source image, and a second structured area image and a second unstructured area image corresponding to the driving image.

[0075] Step S422: using the resnet-tiny sub-network to detect the first unstructured area image and the second unstructured area image respectively, to obtain a heat map of the first unstructured key points and a heat map of the second unstructured key points.

[0076] Step S423: inputting the heat map of the first unstructured key point and the heat map of the second unstructured key point into the optical flow prediction subnetwork to generate optical flow data.

[0077] Step S424: deforming the first unstructured region image using the optical flow data to obtain a deformed first unstructured region image as a preliminary background image.

[0078] Step S425: respectively using the facial key point sub-network and the HRNet sub-network to obtain the first facial key points and their Jacobian matrix corresponding to the first structured area image, and the second facial key points and their Jacobian matrix corresponding to the second structured area image.

[0079] Step S426: Generate a facial key point connection graph based on the first facial key points and their Jacobian matrix, and the second facial key points and their Jacobian matrix.

[0080] Step S427: Generate the preliminary face image based on the source image and the face key point connection diagram.

[0081] Step S428: Based on the source image and the facial key point connection diagram, a residual image is generated using the residual generation network.

[0082] Step S429: Fusing the preliminary face image, the preliminary background image and the residual image to obtain the predicted target image.

[0083] The driving method of the face image of the present application is used to process the source image and the driving image. In addition to learning the face key points through the structured generation network, more unstructured key points are learned through the unstructured generation network. The generated predicted target image separates the face part and the background part, which is conducive to each learning better action representation and avoiding mutual influence. It can not only accurately control the driving effect of the face key point area, but also cover the area outside the face key points, and can also obtain better driving effect in the unstructured key point area.

[0084] See also Figure 6 , Figure 6 It is a framework diagram of an embodiment of a training device for an image generation network model of the present application. The image generation network model includes an image segmentation network, a structured generation network, and an unstructured generation network; the training device 60 for the image generation network model includes: a sample acquisition module 600, the sample acquisition module 600 is used to acquire a portrait image training set; wherein the portrait image training set includes a first portrait image and a second portrait image of the same target; an image prediction module 602, the image prediction module 602 is used to use an image segmentation network to segment the first portrait image and the second portrait image respectively, to obtain a first structured region image and a first unstructured region image corresponding to the first portrait image, and a second structured region image and a second unstructured region image corresponding to the second portrait image; structured area image; based on the first unstructured area image and the second unstructured area image, using an unstructured generation network to generate a preliminary background image; based on the first structured area image and the second structured area image, using a structured generation network to generate a preliminary face image; and, based on the preliminary face image and the preliminary background image, generating a predicted target image; a loss function determination module 604, the loss function determination module 604 is used to obtain the loss function between the second portrait image and the predicted target image; a parameter adjustment module 606, the parameter adjustment module 606 is used to adjust the network parameters in the image generation network model based on the loss function.

[0085] In some embodiments, the image prediction module 602 performs the step of generating a preliminary background image based on the first unstructured area image and the second unstructured area image using an unstructured generation network, including: using the resnet-tiny subnetwork to detect the first unstructured area image and the second unstructured area image respectively to obtain a heat map of the first unstructured key points and a heat map of the second unstructured key points; inputting the heat map of the first unstructured key points and the heat map of the second unstructured key points into the optical flow prediction subnetwork to generate optical flow data; using the optical flow data to deform the first unstructured area image to obtain a deformed first unstructured area image as a preliminary background image.

[0086] In some embodiments, the structured generation network includes a face key point subnetwork and an HRNet subnetwork; the image prediction module 602 performs the steps of generating a preliminary face image based on the first structured area image and the second structured area image using the structured generation network, including: respectively using the face key point subnetwork and the HRNet subnetwork to obtain the first face key points and their Jacobian matrix corresponding to the first structured area image, and the second face key points and their Jacobian matrix corresponding to the second structured area image; based on the first face key points and their Jacobian matrix, and the second face key points and their Jacobian matrix, generating a face key point connection diagram; based on the first portrait image and the face key point connection diagram, generating the preliminary face image.

[0087] In some embodiments, the image generation network model also includes a residual generation network; the image prediction module 602 is executed to generate a predicted target image based on the preliminary face image and the preliminary background image, and is also used to generate a residual image based on the first portrait image and the face key point connection diagram using the residual generation network; at this time, the image prediction module 602 executes the step of generating a predicted target image based on the preliminary face image and the preliminary background image, specifically including: fusing the preliminary face image, the preliminary background image and the residual image to obtain the predicted target image.

[0088] In some embodiments, the image generation network model also includes a discriminator network; after the image prediction module 602 performs the step of fusing the preliminary face image, the preliminary background image and the residual image to obtain the predicted target image, the image prediction module 602 is also used to optimize the predicted target image using the discriminator network so that the discriminator network cannot distinguish between the predicted target image and the second portrait image.

[0089] See also Figure 7 , Figure 7 The driving device 70 for human face images of the present application is a schematic diagram of a framework of an embodiment of the driving device for human face images. The driving device 70 for human face images comprises: an image acquisition module 700, which is used to acquire a source image and a driving image; and an image generation module 702, which is used to input the source image and the driving image into an image generation network model to obtain a target image.

[0090] Wherein, the image generation network model is trained by the above-mentioned image generation network model training method.

[0091] See also Figure 8 , Figure 8 80 is a schematic diagram of a framework of an embodiment of an electronic device of the present application. The electronic device 80 includes a memory 801 and a processor 802 coupled to each other, and the processor 802 is used to execute program instructions stored in the memory 801 to implement any of the above-mentioned training methods for image generation network models or steps of the driving method embodiment of face images. In a specific implementation scenario, the electronic device 80 may include but is not limited to: a microcomputer, a server.

[0092] Specifically, the processor 802 is used to control itself and the memory 801 to implement the training method of any of the above-mentioned image generation network models, or the steps of the driving method embodiment of the face picture. The processor 802 can also be called a CPU (Central Processing Unit). The processor 802 may be an integrated circuit chip with signal processing capabilities. The processor 802 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 802 can be implemented by an integrated circuit chip.

[0093] See also Fig. 9 , Fig. 9 The computer-readable storage medium 90 stores program instructions 900 that can be executed by a processor, and the program instructions 900 are used to implement the steps of any of the above-mentioned training methods for image generation network models or driving methods for face images.

[0094] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0095] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0096] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0097] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. A training method for an image generation network model, characterized in that: The image generation network model includes an image segmentation network, a structured generation network and an unstructured generation network; The training method of the image generation network model includes: Acquire a portrait image training set; wherein the portrait image training set includes a first portrait image and a second portrait image of the same target; Using an image segmentation network to segment the first portrait image and the second portrait image respectively, to obtain a first structured region image and a first unstructured region image corresponding to the first portrait image, and a second structured region image and a second unstructured region image corresponding to the second portrait image; Based on the first unstructured area image and the second unstructured area image, a preliminary background image is generated using an unstructured generative network; wherein the unstructured generative network is trained in a self-supervised manner; the target loss function of the unstructured generative network training includes at least one of a distance loss function and an equal constraint loss function; the unstructured generative network includes a resnet-tiny subnetwork and an optical flow prediction subnetwork; the structured generative network includes a face key point subnetwork and an HRNet subnetwork; Based on the first structured region image and the second structured region image, using a structured generation network to generate a preliminary face image; Based on the preliminary face image and the preliminary background image, generating a predicted target image; Obtaining a loss function between the second portrait image and the predicted target image; Based on the loss function, network parameters in the image generation network model are adjusted.

2. The training method of the image generation network model according to claim 1, characterized in that: The step of generating a preliminary background image by using an unstructured generation network based on the first unstructured region image and the second unstructured region image comprises: Using the resnet-tiny sub-network to detect the first unstructured area image and the second unstructured area image respectively, to obtain a heat map of the first unstructured key points and a heat map of the second unstructured key points; Inputting the heat map of the first unstructured key point and the heat map of the second unstructured key point into the optical flow prediction subnetwork to generate optical flow data; The first unstructured region image is deformed using the optical flow data to obtain a deformed first unstructured region image as a preliminary background image.

3. The training method of the image generation network model according to claim 2, characterized in that: The objective loss function of the optical flow prediction sub-network training includes a total variational loss function.

4. The training method of the image generation network model according to claim 1, characterized in that: The step of generating a preliminary face image using a structured generation network based on the first structured region image and the second structured region image comprises: Using the facial key point sub-network and the HRNet sub-network respectively, obtain the first facial key points and the Jacobian matrix thereof corresponding to the first structured region image, and the second facial key points and the Jacobian matrix thereof corresponding to the second structured region image; Generate a facial key point connection graph based on the first facial key points and their Jacobian matrix, and the second facial key points and their Jacobian matrix; The preliminary face image is generated based on the first portrait image and the face key point connection diagram.

5. The training method of the image generation network model according to claim 4, characterized in that: The image generation network model also includes a residual generation network; Before generating a predicted target image based on the preliminary face image and the preliminary background image, the method further includes: Based on the first portrait image and the facial key point connection diagram, using the residual generation network to generate a residual image; The step of generating a predicted target image based on the preliminary face image and the preliminary background image comprises: The preliminary face image, the preliminary background image and the residual image are fused to obtain the predicted target image.

6. The training method of the image generation network model according to claim 5, characterized in that: The image generation network model also includes a discriminator network; After fusing the preliminary face image, the preliminary background image and the residual image to obtain the predicted target image, the method further includes: The predicted target image is optimized using the discriminator network so that the discriminator network cannot distinguish between the predicted target image and the second portrait image.

7. The training method of the image generation network model according to any one of claims 1 to 6, characterized in that: The loss functions include L1 loss function, perceptual loss function and style loss function.

8. A method for driving a face image, characterized in that: The driving method of the face image includes: Get the source image and driver image; Inputting the source image and the driving image into an image generation network model to obtain a target image; Wherein, the image generation network model is trained by the image generation network model training method described in any one of claims 1 to 7.

9. A training device for an image generation network model, characterized in that: The image generation network model includes an image segmentation network, a structured generation network and an unstructured generation network; The training device of the image generation network model includes: A sample acquisition module, the sample acquisition module is used to acquire a portrait image training set; wherein the portrait image training set includes a first portrait image and a second portrait image of the same target; An image prediction module, wherein the image prediction module is used to segment the first portrait image and the second portrait image respectively using an image segmentation network to obtain a first structured region image and a first unstructured region image corresponding to the first portrait image, and a second structured region image and a second unstructured region image corresponding to the second portrait image; based on the first unstructured region image and the second unstructured region image, a preliminary background image is generated using an unstructured generation network; based on the first structured region image and the second structured region image, a preliminary face image is generated using a structured generation network; and, based on the preliminary face image and the preliminary background image, a predicted target image is generated; wherein the unstructured generation network is obtained by training in a self-supervised manner; and the target loss function of the unstructured generation network training includes at least one of a distance loss function and an equal constraint loss function; a loss function determination module, the loss function determination module being used to obtain a loss function between the second portrait image and the predicted target image; A parameter adjustment module, wherein the parameter adjustment module is used to adjust the network parameters in the image generation network model based on the loss function.

10. A driving device for a face image, characterized in that: The driving device of the face picture comprises: An image acquisition module, wherein the image acquisition module is used to acquire a source image and a drive image; An image generation module, wherein the image generation module is used to input the source image and the driving image into an image generation network model to obtain a target image; Wherein, the image generation network model is trained by the image generation network model training method described in any one of claims 1 to 7.

11. An electronic device, characterized in that: It comprises a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement the training method of the image generation network model described in any one of claims 1 to 7, or the driving method of the face image described in claim 8.

12. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by the processor, the training method of the image generation network model described in any one of claims 1 to 7, or the driving method of the face image described in claim 8 is implemented.

Images