Image processing model training method, electronic device and storage medium

Abstract

The present application provides a training method, electronic device and storage medium for an image processing model, and belongs to the field of image processing technology. It obtains a first target interaction loss function based on the identity features of a first sample synthetic face image, the identity features of a second sample synthetic face image and an initial interaction loss function. And obtains a second target interaction loss function based on the attribute features of the first sample synthetic face image, the attribute features of the second sample synthetic face image and the initial interaction loss function. Based on the first target interaction loss function and the second target interaction loss function, the initial attribute priority generator and the initial identity priority generator are complementary trained, and alternately iterated and evolved, which improves the strength of weakly supervised learning, reduces the difficulty of model training, and the image processing effect of the target image processing model obtained after training is better.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a training method for an image processing model, an image processing method, an electronic device, and a storage medium. Background Art

[0002] With the development of artificial intelligence and computer technology, face conversion technology has emerged. Face conversion refers to replacing the identity of the face in the target face image with the person in the source face image, while maintaining the posture, expression, makeup, background and other elements of the target face image. Current face conversion methods usually require a large number of face images of the same identity for model training to achieve the decoupling of identity information and attribute information. However, the model does not have a clear target generated graph as the training truth value, which belongs to weakly supervised learning and is difficult to train. Therefore, how to provide a training method for an image processing model that can improve the training effect of the model has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a training method for an image processing model, an image processing method, an electronic device and a medium, aiming to improve the training effect of the image processing model by reducing the training difficulty of the model.

[0004] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a training method for an image processing model, the training method comprising:

[0005] Obtain a sample source face image and a sample target face image;

[0006] Inputting the sample source face image and the sample target face image into a pre-trained initial image processing model; wherein the initial image processing model comprises: an initial identity priority generator and an initial attribute priority generator;

[0007] Performing face replacement processing on the sample source face image and the sample target face image through the initial identity priority generator to obtain a first sample synthetic face image, and performing face replacement processing on the sample source face image and the sample target face image through the initial attribute priority generator to obtain a second sample synthetic face image;

[0008] Obtaining an initial interaction loss function of the initial image processing model;

[0009] Performing loss calculation according to the identity feature of the first sample synthesized face image, the identity feature of the second sample synthesized face image, and the initial interaction loss function to obtain a first target interaction loss function;

[0010] Performing loss calculation according to the attribute features of the first sample synthesized face image, the attribute features of the second sample synthesized face image, and the initial interaction loss function to obtain a second target interaction loss function;

[0011] Adjusting parameters of the initial attribute priority generator according to the first target interaction loss function to obtain a target attribute priority generator; adjusting parameters of the initial identity priority generator according to the second target interaction loss function to obtain a target identity priority generator;

[0012] A target image processing model is obtained according to the target attribute priority generator and the target identity priority generator.

[0013] In some embodiments, the performing loss calculation according to the identity feature of the first sample synthesized face image, the identity feature of the second sample synthesized face image, and the initial interaction loss function to obtain a first target interaction loss function includes:

[0014] Performing feature extraction on the first sample synthetic face image to obtain a first sample synthetic identity feature;

[0015] performing feature extraction on the second sample synthetic face image to obtain a second sample synthetic identity feature;

[0016] Calculating a cosine distance between the first sample synthetic identity feature and the second sample synthetic identity feature;

[0017] The initial interaction loss function is updated according to the cosine distance to obtain the first target interaction loss function.

[0018] In some embodiments, the performing loss calculation according to the attribute features of the first sample synthesized face image, the attribute features of the second sample synthesized face image, and the initial interaction loss function to obtain a second target interaction loss function includes:

[0019] Calculating the similarity of the attribute features of the first sample synthetic face image and the second sample synthetic face image by using a preset similarity calculation model;

[0020] The initial interaction loss function is updated according to the attribute feature similarity to obtain the second target interaction loss function.

[0021] In some embodiments, before inputting the sample source face image and the sample target face image into a pre-trained initial image processing model, the method further includes:

[0022] Pre-training a preset first neural network model to obtain the initial identity priority generator specifically includes:

[0023] Inputting the sample source face image and the sample target face image into a preset first neural network model; wherein the first neural network model includes a first identity feature extraction module, a first convolution module, an attribute injection module and a first deconvolution module;

[0024] Extracting features from the sample source face image by using the first identity feature extraction module to obtain a first initial sample identity feature;

[0025] Performing convolution processing on the sample target face image through the first convolution module to obtain a first initial sample attribute feature;

[0026] By using the attribute injection module, the first initial sample attribute feature and the first initial sample identity feature are subjected to feature fusion to obtain a first sample mixed feature;

[0027] Performing deconvolution processing on the first sample mixed features through the first deconvolution module to obtain a first sample synthetic face image;

[0028] Obtaining a first target loss function according to the sample source face image, the sample target face image and the first sample synthetic face image;

[0029] The first neural network model is trained according to the first objective loss function to obtain the initial identity priority generator.

[0030] In some embodiments, the attribute injection module includes: an interpolation layer, a first normalization layer, a first convolution layer, and a second convolution layer. The attribute injection module performs feature fusion on the first initial sample attribute feature and the first initial sample identity feature to obtain a first sample mixed feature, including:

[0031] Performing interpolation processing on the first initial sample identity feature through the interpolation layer to obtain a first target sample identity feature having the same size as the first initial sample attribute feature;

[0032] Normalizing the first target sample identity feature by the first normalization layer to obtain a first probability function;

[0033] Performing convolution processing on the first initial sample attribute feature through the first convolution layer to obtain a first mean coefficient;

[0034] Performing convolution processing on the first initial sample attribute feature through the second convolution layer to obtain a first variance coefficient;

[0035] The first probability function, the first mean coefficient and the first variance coefficient are combined to obtain the first sample mixing feature.

[0036] In some embodiments, before inputting the sample source face image and the sample target face image into a pre-trained initial image processing model, the method further includes:

[0037] Pre-training a preset second neural network model to obtain the initial attribute priority generator specifically includes:

[0038] The sample source face image and the sample target face image are input into a preset second neural network model; wherein the second neural network model includes: a second identity feature extraction module, a second convolution module, an identity injection module and a second deconvolution module

[0039] Extracting features from the sample source face image by using the second identity feature extraction module to obtain a second initial sample identity feature;

[0040] Performing convolution processing on the sample target face image through the second convolution module to obtain a second initial sample attribute feature;

[0041] Performing feature fusion on the second initial sample attribute feature and the second initial sample identity feature through the identity injection module to obtain a second sample mixed feature;

[0042] Performing deconvolution processing on the second sample mixed features through the second deconvolution module to obtain a second sample synthetic face image;

[0043] Obtaining a second target loss function according to the sample source face image, the sample target face image and the second sample synthetic face image;

[0044] The second neural network model is trained according to the second objective loss function to obtain the initial attribute priority generator.

[0045] In some embodiments, the identity injection module includes: a convolution layer, a second normalization layer, a first fully connected layer, and a second fully connected layer. The identity injection module performs feature fusion on the second initial sample attribute feature and the second initial sample identity feature to obtain a second sample mixed feature, including:

[0046] Performing convolution processing on the second initial sample attribute feature through the convolution layer to obtain a second target sample attribute feature;

[0047] Normalizing the second target sample attribute feature by the second normalization layer to obtain a second probability function;

[0048] Extracting the identity feature of the second initial sample through the first fully connected layer to obtain a second mean coefficient;

[0049] Extracting the second initial sample identity feature through the second fully connected layer to obtain a second variance coefficient;

[0050] The second probability function, the second mean coefficient and the second variance coefficient are combined to obtain a second sample mixing feature.

[0051] To achieve the above object, a second aspect of an embodiment of the present application provides an image processing method, the method comprising:

[0052] Obtaining a source face image to be processed and a target face image to be processed;

[0053] The source face image to be processed and the target face image to be processed are input into a target image processing model for image processing to obtain a target synthetic face image; wherein the target image processing model is trained according to the training method described in the embodiment of the first aspect of the present invention.

[0054] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory, a processor, a program stored on the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the method described in the first aspect or the method described in the second aspect is realized.

[0055] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium used for computer-readable storage, and the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method described in the first aspect or the method described in the second aspect above.

[0056] The training method, electronic device and storage medium of the image processing model proposed in the present application obtain a first target interaction loss function based on the identity features of the first sample synthetic face image, the identity features of the second sample synthetic face image and the initial interaction loss function. And obtain a second target interaction loss function based on the attribute features of the first sample synthetic face image, the attribute features of the second sample synthetic face image and the initial interaction loss function. Based on the first target interaction loss function and the second target interaction loss function, the initial attribute priority generator and the initial identity priority generator are complementary trained and iteratively evolved alternately, without the need for a large number of sample source face images and sample target face images, thereby improving the strength of weakly supervised learning and reducing the difficulty of model training. The source identity feature consistency and target attribute feature consistency of the target attribute priority generator and the target identity priority generator of this embodiment can be improved, thereby improving the image processing effect of the target image processing model. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 is a flowchart of a training method for an image processing model provided in an embodiment of the present application;

[0058] Figure 2 yes Figure 1 Flow chart of step S105 in FIG.

[0059] Figure 3 yes Figure 1 Flow chart of step S106 in FIG.

[0060] Figure 4 is another flow chart of the training method of the image processing model provided in the embodiment of the present application;

[0061] Figure 5 yes Figure 4 Flow chart of step S405 in FIG.

[0062] Figure 6 is another flow chart of the training method of the image processing model provided in the embodiment of the present application;

[0063] Figure 7 yes Figure 6 Flow chart of step S605 in FIG.

[0064] Figure 8 is a flowchart of an image processing method provided in an embodiment of the present application;

[0065] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0067] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0068] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0069] First, some nouns involved in this application are analyzed:

[0070] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, language recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0071] Face swapping is a technology in the field of computer vision. Its purpose is to transfer the identity features of a person in a source image to the face of a target image while keeping other attributes of the target image (such as expression, posture, lighting, background, etc.) unchanged. Face swapping technology has a wide range of applications, such as synthesizing high-fidelity images for privacy protection, and has huge application space in the entertainment industry, including film production.

[0072] Interpolation: Interpolation is a method of changing the number of pixels by redistributing the pixels of the original image without generating new pixels. During the image enlargement process, the number of pixels increases accordingly. The process of increase is the process of "interpolation". The "interpolation" program automatically selects pixels with better information as the space to increase and fill the blank pixels, rather than just using adjacent pixels. Therefore, when the image is enlarged, the image will look smoother and cleaner. However, it should be noted that interpolation cannot increase image information. Although the image size becomes larger, the effect is relatively blurred.

[0073] Normalization: There are two forms of normalization methods, one is to convert the number into a decimal between (0, 1), and the other is to convert the dimensional expression into a dimensionless expression. It is mainly proposed for the convenience of data processing. Mapping the data to the range of 0 to 1 for processing is more convenient and faster, and should be classified as part of digital signal processing.

[0074] Convolutional layer (CL): Each convolutional layer in a convolutional neural network consists of several convolutional units, and the parameters of each convolutional unit are optimized through the back-propagation algorithm. The purpose of the convolution operation is to extract different features of the input. The first convolutional layer may only extract some low-level features such as edges, lines, and corners. More layers of the network can iteratively extract more complex features from low-level features.

[0075] Fully connected layer (FC): Each node of the fully connected layer is connected to all nodes of the previous layer, which is used to integrate the features extracted previously. Due to its fully connected characteristics, the fully connected layer generally has the most parameters. It plays the role of a "classifier" in the entire convolutional neural network. If the operations of the convolution layer, pooling layer, and activation function layer are to map the original data to the hidden layer feature space, the fully connected layer plays the role of mapping the learned "distributed feature representation" to the sample label space. In actual use, the fully connected layer can be implemented by convolution operations: the fully connected layer with a fully connected previous layer can be converted into a convolution with a convolution kernel of 1×1; and the fully connected layer with a convolution layer as the previous layer can be converted into a global convolution with a convolution kernel of H×W, where H and W are the height and width of the previous convolution result, respectively.

[0076] Back propagation: The general principle of back propagation is: input the training set data into the input layer of the neural network, pass through the hidden layer of the neural network, and finally reach the output layer of the neural network and output the result; since there is an error between the output result of the neural network and the actual result, the error between the estimated value and the actual value is calculated, and the error is backpropagated from the output layer to the hidden layer until it propagates to the input layer; in the process of back propagation, the values ​​of various parameters are adjusted according to the error; the above process is continuously iterated until convergence.

[0077] Current facial image replacement methods usually require a large number of facial images of the same identity for model training in order to achieve the decoupling of identity information and attribute information. However, the model does not have a clear target generated graph as the training truth value, which belongs to weakly supervised learning and is difficult to train. Therefore, how to provide a training method for an image processing model that can reduce the difficulty of model training and improve the generation quality of synthetic facial images has become a technical problem that needs to be solved urgently.

[0078] Based on this, the embodiments of the present application provide a training method for an image processing model, an image processing method, an electronic device and a storage medium, aiming to reduce the training difficulty of the target image processing model and improve the generation quality of the target synthetic face image.

[0079] The training method of the image processing model, the image processing method, the electronic device and the storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the training method of the image processing model in the embodiments of the present application is described.

[0080] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0081] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0082] The training method of the image processing model provided in the embodiment of the present application relates to the field of artificial intelligence technology. The training method of the image processing model provided in the embodiment of the present application can be applied to the terminal, can also be applied to the server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or it can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the image processing method, etc., but is not limited to the above forms.

[0083] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0084] Figure 1 is an optional flowchart of the training method of the image processing model provided in the embodiment of the present application, Figure 1 The method may include but is not limited to steps S101 to S108.

[0085] Step S101, obtaining a sample source face image and a sample target face image.

[0086] In step S101 of some embodiments, a sample source face image and a sample target face image may be obtained from an existing face image database.

[0087] Step S102, inputting the sample source face image and the sample target face image into a pre-trained initial image processing model; wherein the initial image processing model includes: an initial identity priority generator and an initial attribute priority generator.

[0088] Step S103, performing face replacement processing on the sample source face image and the sample target face image through the initial identity priority generator to obtain a first sample synthetic face image, and performing face replacement processing on the sample source face image and the sample target face image through the initial attribute priority generator to obtain a second sample synthetic face image.

[0089] Step S104, obtaining an initial interaction loss function of the initial image processing model.

[0090] In step S104 of some embodiments, the initial interaction loss function includes: a discriminator loss function, an identity loss function, an attribute loss function, a posture loss function, etc. Among them, the discriminator loss: GAN Loss, the discriminator loss in generative adversarial learning is used to make the first sample synthetic face image or the second sample synthetic face image look more realistic. It should be noted that how to obtain the discriminator loss is a common open source technology. Identity loss: Id Loss = cosine_similary(S id ,Out id ), using the identity feature S of the source sample face image id And the identity features of the sample synthesized face image Out id Attribute loss: The attribute loss is characterized by the cosine distance between the attribute features of the source sample face image and the attribute features of the sample synthetic face image, which is specifically expressed as Attr loss =∑‖T atti -Out atti ‖2. Where T atti is the attribute feature of the sample target face image, is Out atti Attribute characteristics of sample synthetic face images; Out atti The sample synthetic face image is input into the identity priority generator, where i represents the feature level. loss =∑‖T atti -Out atti ‖2,T pose and Out pose Respectively represent the facial postures of the sample target face image and the sample synthetic face image. It should be noted that the facial posture can be obtained by an open source detector such as OpenCV. In one embodiment, the initial interaction loss function: Total loss =w0*GAN Loss +w1*ID Loss +w2*Attr loss +w3*Pose loss , w0, w1, w2, w3 are empirical parameters.

[0091] Step S105, performing loss calculation based on the identity features of the first sample synthetic face image, the identity features of the second sample synthetic face image, and the initial interaction loss function to obtain a first target interaction loss function.

[0092] Step S106, performing loss calculation according to the attribute features of the first sample synthesized face image, the attribute features of the second sample synthesized face image and the initial interaction loss function to obtain a second target interaction loss function.

[0093] Step S107, adjusting the parameters of the initial attribute priority generator according to the first target interaction loss function to obtain a target attribute priority generator; adjusting the parameters of the initial identity priority generator according to the second target interaction loss function to obtain a target identity priority generator.

[0094] Step S108, obtaining a target image processing model according to the target attribute priority generator and the target identity priority generator.

[0095] In the steps S101 to S108 shown in the embodiment of the present application, the identity features of the first sample synthetic face image, the identity features of the second sample synthetic face image and the initial interaction loss function are used to calculate the loss, thereby obtaining the first target interaction loss function, and then the parameters of the initial attribute priority generator are adjusted through the first target interaction function to obtain a target attribute priority generator with more optimized identity features. The loss is calculated through the attribute features of the first sample synthetic face image, the attribute features of the second sample synthetic face image and the initial interaction loss function to obtain the second target interaction loss function, and then the parameters of the initial identity priority generator are adjusted through the second target interaction function to obtain a target identity priority generator with more optimized attribute features. It should be noted that if the initial identity priority generator and the initial attribute priority generator are trained separately with the initial interaction loss function, since there is no strict result image as the training target in the separate training process, it belongs to a weakly supervised learning process. It is precisely because of the lack of a clear target image that weakly supervised learning is difficult to obtain satisfactory learning results, so the intensity of supervision needs to be strengthened. Therefore, in an embodiment of the present application, a first target interaction loss function and a second target interaction loss function are obtained based on the initial interaction loss function. This embodiment performs complementary training on the initial attribute priority generator and the initial identity priority generator based on the above-mentioned first target interaction loss function and the second target interaction loss function, and alternately iterates and evolves. It does not require a large number of sample source face images and sample target face images, which improves the strength of weakly supervised learning and reduces the difficulty of model training. The source identity feature consistency and target attribute feature consistency of the target attribute priority generator and the target identity priority generator of this embodiment can be improved, thereby improving the image processing effect of the target image processing model. It should be noted that the target image processing model can select one of the target attribute priority generator and the target identity priority generator.

[0096] See also Figure 2 In some embodiments, step S105 may include but is not limited to steps S201 to S204.

[0097] Step S201, extracting features from the first sample synthetic face image to obtain first sample synthetic identity features;

[0098] Step S202: extract features from the second sample synthetic face image to obtain second sample synthetic identity features.

[0099] It should be noted that the first sample synthetic face image is subjected to feature extraction by the identity feature extractor to obtain the first sample synthetic identity feature. The second sample synthetic face image is subjected to feature extraction by the identity feature extractor to obtain the second sample synthetic identity feature. It should be noted that the identity feature extraction module is composed of a common face recognition feature extractor, which belongs to the prior art and will not be described in detail here.

[0100] Step S203, calculating the cosine distance between the first sample synthetic identity feature and the second sample synthetic identity feature.

[0101] Step S204: updating the initial interaction loss function according to the cosine distance to obtain a first target interaction loss function.

[0102] Specifically, since the identity feature often also contains the attribute information of the source face image’s posture and illumination, the identity feature S id And the identity features of the sample synthesized face image Out id The cosine distance between the two is used to characterize the identity loss, and then the initial interaction loss function obtained according to the identity loss will make the second sample synthesized face image also carry the attribute characteristics of the source sample face image, affecting the synthesis quality of the second sample synthesized face image. Therefore, in steps S201 to S204 of an embodiment, the following method is used to improve: the identity loss calculation method in the initial interaction loss function is changed to Thereby determining the first objective interaction loss function. is the identity feature of the second sample synthetic face image obtained by the initial attribute priority generator, is the identity feature of the first sample synthetic face image of the initial identity priority generator. After the optimization in the previous step, The attribute features of the sample source face image will be gradually replaced by the attribute features of the target sample face image, and the identity features of the sample source face image will be maintained, realizing the decoupling of the source identity features and the source attribute features, and improving the synthesis quality of the second sample synthetic face image.

[0103] See also Figure 3 In some embodiments, step S106 may include but is not limited to steps S301 to S302.

[0104] Step S301, calculating the attribute feature similarity between the first sample synthetic face image and the second sample synthetic face image by using a preset similarity calculation model;

[0105] Step S302, updating the initial interaction loss function according to the attribute feature similarity to obtain a second target interaction loss function.

[0106] Specifically, the main identity features relied on by the identity priority generator come from the sample target face image, so the first sample synthetic face image often has a higher identity feature consistency, but it will be lacking in maintaining the attribute features of the sample target face image. In order to enhance the ability to maintain attribute features, in steps S301 to S302 of an embodiment, the following methods are used to improve:

[0107] In Total loss Add L on the basis lpips =LPIPS(I out ,A out ), where the LPIPS loss algorithm is a public technology, mainly used to measure the similarity between two images. out is the first sample synthetic face image output by the identity-first generator, A out L is the second sample synthetic face image output by the attribute priority generator. lpips The attribute characteristics of the first sample synthetic face image are made as close as possible to the attribute characteristics of the second sample synthetic face image, so as to achieve the preservation of the attribute characteristics. In one embodiment, the second objective interaction loss function: Gid loss =Total loss +w4*L lpips , w4 is an empirical parameter.

[0108] See also Figure 4 In some embodiments, before step S102, it also includes pre-training a preset first neural network model to obtain an initial identity priority generator. The training of the first neural network model specifically includes but is not limited to steps S401 to S407.

[0109] Step S401, inputting the sample source face image and the sample target face image into a preset first neural network model; wherein the first neural network model includes a first identity feature extraction module, a first convolution module, an attribute injection module and a first deconvolution module.

[0110] It should be noted that the first identity feature extraction module is composed of a common face recognition feature extractor, which belongs to the prior art and will not be described in detail here. The first convolution module may include three convolution layers, the convolution kernel size of the three convolution layers is 4, the step size is 1, and the number of channels is 32, 64, and 128 respectively. The first deconvolution module may include three deconvolution layers, the convolution kernel size of the three deconvolution layers is 4, the step size is 1, and the number of channels is 128, 64, and 32 respectively. The attribute injection module can modulate the identity feature based on the attribute feature to obtain a mixed feature.

[0111] Step S402: extracting features from the sample source face image through a first identity feature extraction module to obtain a first initial sample identity feature.

[0112] Step S403: performing convolution processing on the sample target face image through a first convolution module to obtain a first initial sample attribute feature.

[0113] Step S404: The attribute injection module is used to fuse the first initial sample attribute feature and the first initial sample identity feature to obtain a first sample mixed feature.

[0114] Step S405: performing deconvolution processing on the first sample mixed features through a first deconvolution module to obtain a first sample synthetic face image.

[0115] Specifically, taking the first convolution module as three convolution layers and the first deconvolution module as three deconvolution layers as an example, through step S401 to step S405, the sample target face image T is subjected to convolution processing and downsampling processing of the three-level convolution modules conv0, conv1, and conv2 respectively, and then outputs the three-level first initial sample attribute features (T att0 , T att1 , T att2 ). The sample source face image is processed by the first identity feature extraction module to obtain the first initial sample identity feature, the first initial sample identity feature and the three-level first initial sample attribute feature are input into the attribute injection module, the attribute injection module modulates the first initial sample identity feature based on the first initial sample attribute feature to output the first sample mixed feature, the first sample mixed feature is continuously deconvolved and up-sampled by the three-level deconvolution modules DCnv0, DCnv1, and DCnv2 to obtain the first sample synthetic face image.

[0116] Step S406, obtaining a first target loss function according to the sample source face image, the sample target face image and the first sample synthetic face image.

[0117] It should be noted that the determination of the first objective loss function in step S406 may refer to the specific calculation process of the initial interaction loss function in the above embodiment, which will not be repeated here.

[0118] Step S407, training the first neural network model according to the first objective loss function to obtain an initial identity priority generator.

[0119] In step S407, back propagation can be performed according to the first objective loss function to update the first neural network model by optimizing the first objective loss function, mainly to update the internal parameters (i.e., loss parameters) of the first neural network model. It is understandable that the back propagation principle can apply the conventional back propagation principle, which is not limited in the embodiment of the present application. By continuously repeating the above process until the first objective loss function meets the preset threshold condition, or the number of iterations meets the preset number of times, the first objective loss function of the first neural network model is optimized to obtain an initial identity priority generator.

[0120] See also Figure 5 In some embodiments, the attribute injection module includes: an interpolation layer, a first normalization layer, a first convolution layer, and a second convolution layer. Step S405 may include but is not limited to steps S501 to S505.

[0121] Step S501 : interpolating the first initial sample identity feature through an interpolation layer to obtain a first target sample identity feature having the same size as the first initial sample attribute feature.

[0122] Step S502, normalizing the identity feature of the first target sample through a first normalization layer to obtain a first probability function.

[0123] Step S503: performing convolution processing on the first initial sample attribute feature through a first convolution layer to obtain a first mean coefficient.

[0124] Step S504, performing convolution processing on the first initial sample attribute feature through the second convolution layer to obtain a first variance coefficient, which will not be described in detail here.

[0125] It should be noted that in step S503 to step S504, the parameters of the first convolution layer and the second convolution layer are the same. In a specific example, the convolution kernel, step size and number of channels of the first convolution layer are 3, 1, and 256 respectively. The convolution kernel, step size and number of channels of the second convolution layer are 3, 1, and 256 respectively. However, the weight coefficients in the two convolution layers are obtained by training and learning, so the outputs of the two convolution layers have their own semantics, and the obtained outputs are different. Therefore, the first mean coefficient can be obtained through the first convolution layer, and the first variance coefficient can be obtained through the second convolution layer.

[0126] Step S505: Combine the first probability function, the first mean coefficient and the first variance coefficient to obtain a first sample mixture feature.

[0127] Specifically, in step S501 to step S505, the first initial sample identity feature is interpolated by the interpolation layer to obtain the first target sample identity feature of the same size as the first initial sample attribute feature. If the input first target sample identity feature is regarded as a probability distribution, the normalization layer normalizes the first target sample identity feature to a normal distribution with a zero mean. The normal distribution has two adjustable parameters, namely the mean and the variance. Adjusting these two parameters can change the distribution. The first initial sample attribute feature is respectively obtained through two convolutional layers to obtain the first mean coefficient and the first variance coefficient, which are respectively used to adjust the variance and the mean of the first probability function. That is, the first probability function, the first mean coefficient and the first variance coefficient are combined to obtain a new probability function, thereby obtaining the first sample mixed feature.

[0128] See also Figure 6 In some embodiments, before step S102, it also includes pre-training a preset second neural network model to obtain an initial attribute priority generator. The training of the second neural network model specifically includes but is not limited to steps S601 to S607.

[0129] Step S601, inputting the sample source face image and the sample target face image into a preset second neural network model; wherein the second neural network model includes: a second identity feature extraction module, a second convolution module, an identity injection module and a second deconvolution module.

[0130] Step S602: extracting features from the sample source face image through a second identity feature extraction module to obtain a second initial sample identity feature.

[0131] Step S603, performing convolution processing on the sample target face image through a second convolution module to obtain a second initial sample attribute feature.

[0132] Step S604: The identity injection module is used to fuse the second initial sample attribute feature and the second initial sample identity feature to obtain a second sample mixed feature.

[0133] Step S605: Perform deconvolution processing on the second sample mixed features through a second deconvolution module to obtain a second sample synthetic face image.

[0134] Specifically, taking the second convolution module as three convolution layers and the second deconvolution module as three deconvolution layers as an example, through steps S601 to S605, the sample target face image T is processed by the three-level convolution modules conv0, conv1, and conv2 for continuous convolution and downsampling to output the second initial sample attribute feature. The sample source face image is processed by the second identity feature extraction module to obtain the second initial sample identity feature, and the second initial sample identity feature and the second initial sample attribute feature are input into the identity injection module. The identity injection module modulates the second initial sample attribute feature based on the second initial sample identity feature to output the second sample mixed feature. The second sample mixed feature is processed by the three-level deconvolution modules DConv0, DConv1, and DConv2 for continuous deconvolution and upsampling to obtain the second sample synthetic face image.

[0135] Step S606, obtaining a second target loss function according to the sample source face image, the sample target face image and the second sample synthetic face image.

[0136] It should be noted that the determination of the second objective loss function in step S606 may refer to the specific calculation process of the initial interaction loss function in the above embodiment, which will not be repeated here.

[0137] Step S607, training the second neural network model according to the second objective loss function to obtain an initial attribute priority generator.

[0138] In step S607, back propagation can be performed according to the second objective loss function to update the second neural network model by optimizing the second objective loss function, mainly to update the internal parameters (i.e., loss parameters) of the second neural network model. It is understandable that the back propagation principle can apply the conventional back propagation principle, which is not limited in the embodiment of the present application. By continuously repeating the above process until the second objective loss function meets the preset threshold condition, or the number of iterations meets the preset number of times, the second objective loss function of the second neural network model is optimized to obtain an initial attribute priority generator.

[0139] See also Figure 7 In some embodiments, the identity injection module includes: a convolutional layer, a second normalization layer, a first fully connected layer, and a second fully connected layer. Step S605 may include but is not limited to steps S701 to S705.

[0140] Step S701, performing convolution processing on the second initial sample attribute feature through a convolution layer to obtain a second target sample attribute feature.

[0141] Step S702: normalize the attribute features of the second target sample through a second normalization layer to obtain a second probability function.

[0142] Step S703: extract the identity feature of the second initial sample through the first fully connected layer to obtain a second mean coefficient.

[0143] Step S704: extract the second initial sample identity feature through the second fully connected layer to obtain a second variance coefficient.

[0144] It should be noted that in step S703 to step S704, the number of channels of the first fully connected layer and the second fully connected layer is the same, for example, both are 256. However, the weight coefficients in the two fully connected layers are obtained by training and learning, so the outputs of the two fully connected layers have their own semantics and the obtained outputs are different. Therefore, the second mean coefficient can be obtained through the first fully connected layer, and the second variance coefficient can be obtained through the second fully connected layer.

[0145] Step S705, combining the second probability function, the second mean coefficient and the second variance coefficient to obtain a second sample mixing feature.

[0146] Specifically, in step S701 to step S705, the second initial sample attribute feature is convolved through the convolution layer to obtain the second target sample attribute feature. If the input second target sample attribute feature is regarded as a probability distribution, the normalization layer normalizes the second target sample identity feature to a normal distribution with a zero mean. The normal distribution has two adjustable parameters, namely the mean and the variance. Adjusting these two parameters can change the distribution. The second initial sample identity feature obtains the second mean coefficient and the second variance coefficient through two fully connected layers, which are used to adjust the variance and mean of the second probability function respectively. That is, the second probability function, the second mean coefficient and the second variance coefficient are combined to obtain a new probability function, thereby obtaining the second sample mixed feature.

[0147] The training method of the image processing model of the embodiment of the present application uses the first target interaction loss function and the second target interaction loss function to complementarily train the initial attribute priority generator and the initial identity priority generator, and iterates and evolves alternately. It does not require a large number of sample source face images and sample target face images, which improves the strength of weakly supervised learning and reduces the difficulty of model training. The consistency of the source identity features and the consistency of the target attribute features of the target attribute priority generator and the target identity priority generator of this embodiment can be improved, thereby improving the image processing effect of the target image processing model.

[0148] See also Figure 8 The present application also provides an image processing method, which includes but is not limited to steps S801 to S802:

[0149] Step S801, obtaining a source face image to be processed and a target face image to be processed;

[0150] Step S802, input the source face image to be processed and the target face image to be processed into a target image processing model for image processing to obtain a target synthetic face image; wherein the target image processing model is trained according to the training method of the first aspect embodiment.

[0151] In step S801 of some embodiments, the source face image to be processed and the target face image to be processed may be obtained from electronic devices such as computers, mobile phones, and cameras.

[0152] In step S802 of some embodiments, the source face image to be processed and the target face image to be processed are input into the target image processing model for image processing. If the target image processing model is an identity-first generator, the source face image to be processed is subjected to feature extraction through the first identity feature extraction module to obtain a first initial identity feature; the target face image to be processed is subjected to convolution processing through the first convolution module to obtain a first initial attribute feature; the first initial attribute feature and the first initial identity feature are subjected to feature fusion through the attribute injection module to obtain a first mixed feature; the first mixed feature is subjected to deconvolution processing through the first deconvolution module to obtain a first synthetic face image. If the target image processing model is an attribute-first generator, the source face image to be processed is subjected to feature extraction through the second identity feature extraction module to obtain a second initial identity feature; the target face image to be processed is subjected to convolution processing through the second convolution module to obtain a second initial attribute feature; the second initial attribute feature and the second initial identity feature are subjected to feature fusion through the identity injection module to obtain a second sample mixed feature; the second mixed feature is subjected to deconvolution processing through the second deconvolution module to obtain a second synthetic face image.

[0153] The image processing method of the embodiment of the present application performs image processing based on a trained target image processing model, thereby achieving effective decoupling of attribute features and identity features, and the obtained synthetic face image is of high quality.

[0154] The embodiment of the present application also provides an electronic device, the electronic device comprising: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program is executed by the processor to realize the training method of the above-mentioned image processing model. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0155] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0156] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0157] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the training method or image processing method of the image processing model in the embodiment of this application;

[0158] Input / output interface 903, used to implement information input and output;

[0159] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0160] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0161] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0162] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium used for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned image processing model training method or image processing method.

[0163] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0164] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0165] It can be understood by those skilled in the art that Figure 1-8 The technical solutions shown in the figure do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown in the figure, or a combination of certain steps, or different steps.

[0166] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0167] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0168] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0169] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0170] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0171] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0172] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0173] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0174] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A training method for an image processing model, It is characterized in that The training method comprises: Obtain a sample source face image and a sample target face image; Inputting the sample source face image and the sample target face image into a pre-trained initial image processing model; wherein the initial image processing model comprises: an initial identity priority generator and an initial attribute priority generator; Performing face replacement processing on the sample source face image and the sample target face image through the initial identity priority generator to obtain a first sample synthetic face image, and performing face replacement processing on the sample source face image and the sample target face image through the initial attribute priority generator to obtain a second sample synthetic face image; Obtaining an initial interaction loss function of the initial image processing model; Performing loss calculation according to the identity feature of the first sample synthesized face image, the identity feature of the second sample synthesized face image, and the initial interaction loss function to obtain a first target interaction loss function; Performing loss calculation according to the attribute features of the first sample synthesized face image, the attribute features of the second sample synthesized face image, and the initial interaction loss function to obtain a second target interaction loss function; Adjusting parameters of the initial attribute priority generator according to the first target interaction loss function to obtain a target attribute priority generator; adjusting parameters of the initial identity priority generator according to the second target interaction loss function to obtain a target identity priority generator; Obtaining a target image processing model according to the target attribute priority generator and the target identity priority generator; Before inputting the sample source face image and the sample target face image into the pre-trained initial image processing model, the method further includes: Pre-training a preset first neural network model to obtain the initial identity priority generator specifically includes: Inputting the sample source face image and the sample target face image into a preset first neural network model; wherein the first neural network model includes a first identity feature extraction module, a first convolution module, an attribute injection module and a first deconvolution module, and the attribute injection module includes: an interpolation layer, a first normalization layer, a first convolution layer, and a second convolution layer; Extracting features from the sample source face image by using the first identity feature extraction module to obtain a first initial sample identity feature; Performing convolution processing on the sample target face image through the first convolution module to obtain a first initial sample attribute feature; The first initial sample attribute feature and the first initial sample identity feature are subjected to feature fusion through the attribute injection module to obtain a first sample mixed feature, including: interpolating the first initial sample identity feature through the interpolation layer to obtain a first target sample identity feature having the same size as the first initial sample attribute feature; normalizing the first target sample identity feature through the first normalization layer to obtain a first probability function; convolving the first initial sample attribute feature through the first convolution layer to obtain a first mean coefficient; convolving the first initial sample attribute feature through the second convolution layer to obtain a first variance coefficient; combining the first probability function, the first mean coefficient and the first variance coefficient to obtain the first sample mixed feature; Performing deconvolution processing on the first sample mixed features through the first deconvolution module to obtain a first sample synthetic face image; Obtaining a first target loss function according to the sample source face image, the sample target face image and the first sample synthetic face image; Training the first neural network model according to the first objective loss function to obtain the initial identity priority generator; Before inputting the sample source face image and the sample target face image into the pre-trained initial image processing model, the method further includes: Pre-training a preset second neural network model to obtain the initial attribute priority generator specifically includes: Inputting the sample source face image and the sample target face image into a preset second neural network model; wherein the second neural network model comprises: a second identity feature extraction module, a second convolution module, an identity injection module and a second deconvolution module, and the identity injection module comprises: a convolution layer, a second normalization layer, a first fully connected layer, and a second fully connected layer; Extracting features from the sample source face image by using the second identity feature extraction module to obtain a second initial sample identity feature; Performing convolution processing on the sample target face image through the second convolution module to obtain a second initial sample attribute feature; The second initial sample attribute feature and the second initial sample identity feature are subjected to feature fusion through the identity injection module to obtain a second sample mixed feature, including: performing convolution processing on the second initial sample attribute feature through the convolution layer to obtain a second target sample attribute feature; performing normalization processing on the second target sample attribute feature through the second normalization layer to obtain a second probability function; performing feature extraction on the second initial sample identity feature through the first fully connected layer to obtain a second mean coefficient; performing feature extraction on the second initial sample identity feature through the second fully connected layer to obtain a second variance coefficient; and combining the second probability function, the second mean coefficient and the second variance coefficient to obtain the second sample mixed feature; Performing deconvolution processing on the second sample mixed features through the second deconvolution module to obtain a second sample synthetic face image; Obtaining a second target loss function according to the sample source face image, the sample target face image and the second sample synthetic face image; The second neural network model is trained according to the second objective loss function to obtain the initial attribute priority generator.

2. The training method according to claim 1, It is characterized in that The performing loss calculation according to the identity feature of the first sample synthesized face image, the identity feature of the second sample synthesized face image, and the initial interaction loss function to obtain a first target interaction loss function includes: Performing feature extraction on the first sample synthetic face image to obtain a first sample synthetic identity feature; performing feature extraction on the second sample synthetic face image to obtain a second sample synthetic identity feature; Calculating a cosine distance between the first sample synthetic identity feature and the second sample synthetic identity feature; The initial interaction loss function is updated according to the cosine distance to obtain the first target interaction loss function.

3. The training method according to claim 1, It is characterized in that The loss calculation is performed according to the attribute features of the first sample synthesized face image, the attribute features of the second sample synthesized face image, and the initial interaction loss function to obtain a second target interaction loss function, including: Calculate the similarity of the attribute features of the first sample synthetic face image and the second sample synthetic face image by using a preset similarity calculation model; The initial interaction loss function is updated according to the attribute feature similarity to obtain the second target interaction loss function.

4. An image processing method, It is characterized in that The method comprises: Obtaining a source face image to be processed and a target face image to be processed; The source face image to be processed and the target face image to be processed are input into a target image processing model for image processing to obtain a target synthetic face image; wherein the target image processing model is trained according to the training method according to any one of claims 1 to 3.

5. An electronic device, It is characterized in that The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, wherein when the program is executed by the processor: The method according to any one of claims 1 to 3; Or, the method as claimed in claim 4.

6. A storage medium, the storage medium being a computer-readable storage medium, for computer-readable storage, It is characterized in that The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement: The method according to any one of claims 1 to 3; Or, the method as claimed in claim 4.

Images