Image processing method, system, terminal device and storage medium

Abstract

The present invention provides an image processing method, system, terminal device and storage medium, the method comprising: inputting a to-be-processed image and a target image into a pre-trained skin quality migration model for feature extraction, obtaining a first extracted feature and a second extracted feature; performing feature fusion on the first image feature and the second skin quality feature according to the pre-trained skin quality migration model to obtain an expected image feature, and performing image generation according to the expected image feature to obtain a first migration image; performing image scaling on the first migration image and the to-be-processed image respectively to obtain a first image pyramid and a second image pyramid, and performing skin quality migration on the to-be-processed image according to the first image pyramid and the second image pyramid to obtain an output image. The first migration image in the present invention can effectively retain skin quality-irrelevant features such as contours, facial features, and background in the to-be-processed image and skin quality features in the target image, thereby improving the accuracy of skin quality migration.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image processing method, system, terminal device and storage medium. Background Art

[0002] As smartphones become increasingly powerful, more and more users use their phones' camera functions to record their daily lives. Compared with portraits taken with professional imaging equipment, mobile devices such as mobile phones are subject to more restrictions, such as imaging size, device performance, and shooting methods. The portraits taken lack sufficient portrait details, resulting in a lack of "texture" in the images. Therefore, how to add appropriate texture to the skin of the image to make the portrait more textured is becoming more and more important.

[0003] The purpose of the image processing operation of skin texture migration is to allow the user to migrate the skin texture of a target portrait to the image to be processed, so that the image to be processed presents a similar skin detail effect.

[0004] In the existing image processing process, the portrait style transfer method is used to achieve the image processing effect of skin texture transfer. However, the portrait style transfer method can only convert the image to be processed into an image with a fixed style and cannot change according to the target portrait, resulting in low accuracy of skin texture transfer. Summary of the invention

[0005] The purpose of the embodiments of the present invention is to provide an image processing method, system, terminal device and storage medium, aiming to solve the problem of low accuracy of skin quality migration in the existing image processing process.

[0006] The embodiment of the present invention is implemented as follows: an image processing method, the method comprising:

[0007] Inputting the image to be processed and the target image into the pre-trained skin quality migration model for feature extraction, respectively, to obtain a first extracted feature and a second extracted feature, wherein the first extracted feature includes a first image feature and a first skin quality feature, and the second extracted feature includes a second image feature and a second skin quality feature;

[0008] Performing feature fusion on the first image feature and the second skin quality feature according to the pre-trained skin quality migration model to obtain expected image features, and performing image generation according to the expected image features to obtain a first migration image;

[0009] The first migration image and the image to be processed are respectively scaled to obtain a first image pyramid and a second image pyramid, and skin quality migration is performed on the image to be processed according to the first image pyramid and the second image pyramid to obtain an output image

[0010] Furthermore, before the image to be processed and the target image are respectively input into the pre-trained skin quality migration model for feature extraction, the method further includes:

[0011] Acquire a training sample image, and perform sample division on the training sample image to obtain a reference sample image and a target sample image;

[0012] Inputting the reference sample image and the target sample image into an encoder in a skin quality migration model for feature extraction to obtain reference features and target features, wherein the reference features include reference image features and reference skin quality features, and the target features include target image features and target skin quality features;

[0013] The image features of the reference image and the skin quality features of the reference image are fused and sent to the generator to obtain a reconstructed reference image, and the target image features and the target skin quality features are fused and sent to the generator to obtain a reconstructed target image;

[0014] exchanging the skin quality features of the reference sample image and the target sample image, and using the generator to generate an image to obtain a migration map,

[0015] The reconstructed reference image and the reconstructed target image are respectively subjected to loss calculation with the reference image and the target image to obtain a reconstruction loss;

[0016] Sending the migration map and the corresponding sample image into the discriminator to obtain the migration loss;

[0017] The parameters of the encoder and the generator are updated according to the reconstruction loss and the migration loss, and based on the encoder and the generator after the parameter update, the step of performing sample division on the training sample image is returned to be executed, until the encoder and the generator after the parameter update meet the preset iteration condition, and the skin quality migration model is output to obtain the pre-trained skin quality migration model.

[0018] Furthermore, the model loss value includes a reconstruction loss value, a perception loss value and a discrimination loss value;

[0019] The formula used to calculate the reconstruction loss value is:

[0020]

[0021] in:

[0022] tar indicates that the relevant information comes from the target graph.

[0023] ref indicates that the relevant information comes from the reference graph.

[0024] G is the generator

[0025] F represents feature map (feature-map)

[0026] I represents image input

[0027] img represents image features

[0028] Skin indicates skin characteristics

[0029] The formula used to calculate the perceived loss value is:

[0030] L pec =||σ k (G(F img , F skin ))-σ k (I)|| 2

[0031] Among them, σ k It's VG G The kth layer in the model;

[0032] The formula used to calculate the discrimination loss value is:

[0033]

[0034]

[0035]

[0036] Among them, D f It is the discriminator in the skin quality migration model.

[0037] Furthermore, the target image feature and the reference skin quality feature are fused to obtain a fused feature, including:

[0038] Performing convolution processing on the target image features, and performing image filtering on the target image features after the convolution processing;

[0039] At multiple different feature map scales, the reference skin quality features are scaled and mapped into two components, “magnification” and “offset”, through convolution operations.

[0040] The two components of magnification and offset are applied to the skin quality features to obtain the fusion features.

[0041] Furthermore, the model loss value also includes a skin loss value, a low-frequency loss value, a skin quality loss value and a feature loss value. The migration map and the sample image are subjected to loss calculation to obtain a migration loss value, which also includes:

[0042] Combining the target image features with the reference skin quality features, and sending the results to a generator to obtain a first migration map, and combining the target skin quality features with the reference image features to obtain a second migration map;

[0043] The skin loss value, the low-frequency loss value, the skin quality loss value and the characteristic loss value of the skin quality migration model are calculated respectively according to the migration map.

[0044] Furthermore, the formula used to calculate the skin loss value is:

[0045]

[0046] Among them, τ is the pre-trained portrait skin segmentation model, O k，j is the first migration map, O j，k is the second migration map;

[0047] The formula used to calculate the low frequency loss value is:

[0048]

[0049] Among them, δ is the low-frequency signal extraction function;

[0050] The formula used to calculate the skin quality loss value is:

[0051]

[0052]

[0053]

[0054] Among them, η is the high-frequency signal extraction function, M i i∈t, r is the face skin mask, patchify is the screenshot function, D skin is the region discriminator;

[0055] The formula used to calculate the feature loss value is:

[0056]

[0057]

[0058] The formula used to calculate the loss value of the model is:

[0059] L=λ 1 L rec +λ 2 L im9 +λ 3 L freq +λ4 L pec +λ 5 L dis +λ 6 L skin +λ 7 L cons .

[0060] Furthermore, before the image to be processed and the target image are respectively input into the pre-trained skin quality migration model for feature extraction, the method further includes:

[0061] Extracting facial key points from the image to be processed and the target image respectively to obtain a first facial point set and a second facial point set;

[0062] Determine a cropping rectangle for facial key points in the to-be-processed image according to the first facial point set to obtain a first cropping rectangle, and determine a cropping rectangle for facial key points in the target image according to the second facial point set to obtain a second cropping rectangle;

[0063] Determining a rotation angle of a face in the to-be-processed image according to the first cropping rectangle to obtain a first rotation angle, and determining a rotation angle of a face in the target image according to the second cropping rectangle to obtain a second rotation angle;

[0064] Performing face correction on the image to be processed according to the first rotation angle, and performing face correction on the target image according to the second rotation angle;

[0065] Performing face cropping on the image to be processed after face correction according to the first cropping rectangle to obtain a first face image, and performing face cropping on the target image after face correction according to the second cropping rectangle to obtain a second face image;

[0066] The first face image and the second face image are downsampled respectively, and the downsampled first face image is set as the image to be processed, and the downsampled second face image is set as the target image.

[0067] Another object of an embodiment of the present invention is to provide an image processing system, the system comprising:

[0068] A feature extraction unit, used for inputting the image to be processed and the target image into the pre-trained skin quality migration model for feature extraction, to obtain a first extracted feature and a second extracted feature, wherein the first extracted feature includes a first image feature and a first skin quality feature, and the second extracted feature includes a second image feature and a second skin quality feature;

[0069] A feature fusion unit, configured to perform feature fusion on the first image feature and the second skin quality feature according to the pre-trained skin quality migration model to obtain an expected image feature, and perform image generation according to the expected image feature to obtain a first migration image;

[0070] The skin quality migration unit is used to perform image scaling on the first migration image and the image to be processed respectively to obtain a first image pyramid and a second image pyramid, and perform skin quality migration on the image to be processed according to the first image pyramid and the second image pyramid to obtain an output image.

[0071] Another object of an embodiment of the present invention is to provide a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0072] Another object of an embodiment of the present invention is to provide a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0073] In the embodiment of the present invention, the image to be processed and the target image are respectively input into a pre-trained skin quality migration model for feature extraction, so as to decouple the portrait features in the image to be processed and the target image into image features and skin quality features respectively, and the expected image features are obtained by performing feature fusion on the first image features and the second skin quality features, and an image is generated based on the expected image features to obtain a first migration image, so that the first migration image can effectively retain the contour, facial features, background and other skin quality-independent features in the image to be processed and the skin quality features in the target image, thereby improving the accuracy of skin quality migration, and the first migration image and the image to be processed are respectively scaled to obtain the first image pyramid and the second image pyramid, and skin quality migration is performed on the image to be processed according to the first image pyramid and the second image pyramid, so that the low-frequency information in the first migration image can be effectively stripped, and the high-frequency information can be migrated to the image to be processed, so as to obtain the output image after skin quality migration, thereby further improving the accuracy of skin quality migration. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 is a flowchart of an image processing method provided by a first embodiment of the present invention;

[0075] Figure 2 is a schematic diagram of a feature extraction process provided by the first embodiment of the present invention;

[0076] Figure 3 is a schematic diagram of a first image pyramid and a second image pyramid provided by a first embodiment of the present invention;

[0077] Figure 4 is a flowchart of an image processing method provided by a second embodiment of the present invention;

[0078] Figure 5 is a schematic structural diagram of a skin quality migration model provided by a second embodiment of the present invention;

[0079] Figure 6 is a schematic diagram of the structure of an encoder provided by a second embodiment of the present invention;

[0080] Figure 7 is a schematic structural diagram of a residual module provided by a second embodiment of the present invention;

[0081] Figure 8 is a schematic diagram of the structure of a generator provided by a second embodiment of the present invention;

[0082] Fig. 9 is a schematic diagram of the structure of an image processing system provided by a third embodiment of the present invention;

[0083] Fig.10 It is a schematic diagram of the structure of a terminal device provided in the fourth embodiment of the present invention. DETAILED DESCRIPTION

[0084] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0085] In order to illustrate the technical solution of the present invention, a specific embodiment is provided below for illustration.

[0086] Embodiment 1

[0087] See also Figure 1 , is a flowchart of an image processing method provided by a first embodiment of the present invention. The image processing method can be applied to any terminal device or system. The image processing method comprises the steps of:

[0088] Step S10, inputting the image to be processed and the target image into the pre-trained skin quality migration model for feature extraction to obtain a first extracted feature and a second extracted feature;

[0089] The image to be processed is an image selected by the user for skin quality migration, and the target image is an image selected by the user for skin quality migration, that is, the user needs to migrate the skin quality effect in the target image to the image to be processed. In this step, the first extracted feature includes the first image feature and the first skin quality feature, and the second extracted feature includes the second image feature and the second skin quality feature. The image feature is a skin quality-independent feature, and the image feature includes features such as the face contour, facial features, and image background in the image;

[0090] Specifically, in this step, see Figure 2 , the image to be processed I t Input the encoder in the pre-trained skin quality transfer model for feature extraction to obtain the first image feature And the first skin characteristics The target image I r Input the encoder in the pre-trained skin quality transfer model for feature extraction to obtain the second image feature and second skin characteristics

[0091] Optionally, in this step, before the image to be processed and the target image are respectively input into the pre-trained skin quality migration model for feature extraction, the step further includes:

[0092] Extracting facial key points from the image to be processed and the target image respectively to obtain a first facial point set and a second facial point set;

[0093] Among them, face detection or face alignment can be used to extract face key points in the image to be processed and the target image;

[0094] Determine a cropping rectangle for facial key points in the to-be-processed image according to the first facial point set to obtain a first cropping rectangle, and determine a cropping rectangle for facial key points in the target image according to the second facial point set to obtain a second cropping rectangle;

[0095] Among them, the circumscribed rectangle formed by the facial key points in the image to be processed is calculated according to the first facial point set to obtain a first circumscribed rectangle, and the first circumscribed rectangle is expanded outward in the image to be processed to obtain the first cropping rectangle. It can be understood that for the target image, the circumscribed rectangle formed by the facial key points in the target image is calculated according to the second facial point set to obtain a second circumscribed rectangle, and the second circumscribed rectangle is expanded outward in the target image to obtain the second cropping rectangle;

[0096] Determining a rotation angle of a face in the to-be-processed image according to the first cropping rectangle to obtain a first rotation angle, and determining a rotation angle of a face in the target image according to the second cropping rectangle to obtain a second rotation angle;

[0097] Performing face correction on the image to be processed according to the first rotation angle, and performing face correction on the target image according to the second rotation angle;

[0098] The face of the image to be processed is corrected by the first rotation angle, and the face of the marked image is corrected by the second rotation angle, so as to achieve the effect of straightening the faces in the image to be processed and the target image, thereby improving the accuracy of subsequent face cropping;

[0099] Performing face cropping on the image to be processed after face correction according to the first cropping rectangle to obtain a first face image, and performing face cropping on the target image after face correction according to the second cropping rectangle to obtain a second face image;

[0100] The first face image and the second face image are downsampled respectively, and the downsampled first face image is set as the image to be processed, and the downsampled second face image is set as the target image.

[0101] Step S20, performing feature fusion on the first image feature and the second skin quality feature according to the pre-trained skin quality migration model to obtain expected image features, and performing image generation according to the expected image features to obtain a first migration image;

[0102] The encoder in the pre-trained skin quality transfer model fuses the first image feature and the second skin quality feature to obtain the expected image feature F p , and the image expected feature F p Input the generator in the pre-trained skin quality migration model to generate an image, and obtain the first migration image O first ;

[0103] Step S30, performing image scaling on the first migration image and the image to be processed respectively to obtain a first image pyramid and a second image pyramid, and performing skin quality migration on the image to be processed according to the first image pyramid and the second image pyramid to obtain an output image;

[0104] Among them, see Figure 3 , the first migration map and the image to be processed are scaled step by step, and an image pyramid is constructed to obtain the first image pyramid P first and the second image pyramid P t , and performing image restoration (skin quality migration) on the image to be processed according to the first image pyramid and the second image pyramid to obtain an output image;

[0105] In this embodiment, the image to be processed and the target image are respectively input into the pre-trained skin quality migration model for feature extraction, so as to decouple the portrait features in the image to be processed and the target image into image features and skin quality features, respectively; the first image features and the second skin quality features are feature fused to obtain expected image features; image generation is performed based on the expected image features to obtain a first migration image, so that the first migration image can effectively retain the contour, facial features, background and other skin quality-irrelevant features in the image to be processed and the skin quality features in the target image, thereby improving the accuracy of skin quality migration; the first migration image and the image to be processed are respectively scaled to obtain the first image pyramid and the second image pyramid; and skin quality migration is performed on the image to be processed according to the first image pyramid and the second image pyramid, so that low-frequency information in the first migration image can be effectively stripped, and high-frequency information can be migrated to the image to be processed, thereby obtaining an output image after skin quality migration, thereby further improving the accuracy of skin quality migration.

[0106] Example two

[0107] See also Figure 4 , is a flow chart of an image processing method provided by a second embodiment of the present invention, and this embodiment is used to further refine the steps before step S10, including the steps of:

[0108] Step S40, obtaining a training sample image, and performing sample division on the training sample image to obtain a reference sample image and a target sample image;

[0109] The training sample image can be divided into samples by random division to obtain a reference sample image and a target sample image;

[0110] Optionally, in this step, for any group of N portrait inputs I, the encoder will obtain two groups of feature outputs:

[0111]

[0112] Dividing these two sets of features into two equal parts, we can get four sets of feature vectors:

[0113]

[0114] The inputs corresponding to the above four features are I k and I j , where each group F img and a group of F skin A new portrait feature can be obtained by fusion, and the generated results of the feature can be supervised until convergence to obtain the required encoder and generator;

[0115] Step S50, respectively inputting the reference sample image and the target sample image into an encoder in a skin quality migration model for feature extraction to obtain reference features and target features;

[0116] The reference features include reference image features and reference skin quality features, and the target features include target image features and target skin quality features; see Figure 5 ,The skin quality transfer model adopts the encoder-generator structure, the input is a 3-channel RGB image, and the output is a 3-channel RGB image. The skin quality feature is used to characterize the high-frequency distribution characteristics of the portrait skin;

[0117] See also Figure 6 to Figure 7 The first few layers of the encoder use fast Fourier convolution. Through fast Fourier transform, the model can obtain a large receptive field while saving performance overhead as much as possible. At the same time, after the output layer of skin quality features, several convolution and residual modules are stacked to further extract the abstract features of the image. The residual module structure consists of 3x3 convolution, 1x1 convolution and ReLu layers. The 1x1 convolution only exists during model training. During inference, its parameters will be superimposed on the 3x3 convolution to further reduce the amount of calculation.

[0118] FFT-Conv:

[0119] FCONV(input,kernel)=IFFT(FFT(input)*FFT(kernel))

[0120] FFT is fast Fourier transform, IFFT is inverse Fourier transform, kernel is convolution kernel

[0121] Formally, the encoder can be expressed as:

[0122] (F img , F skin )=E(I)

[0123] Step S60, fusing the target image feature and the reference skin quality feature to obtain a fused feature, sending the fused feature to the generator to obtain a first migration map, fusing the target image feature and the target skin quality feature and sending them to the generator to obtain a reconstructed map;

[0124] See also Figure 8 ,The generator takes skin quality irrelevant features as the bottom layer input, and continuously incorporates skin quality features in the step-by-step decoding and amplification process, and obtains the final result;

[0125] Optionally, in this step, the step of fusing the target image feature and the reference skin quality feature to obtain a fused feature comprises: fusing the target image feature and the reference skin quality feature according to the encoder to obtain a fused feature, comprising:

[0126] At multiple different feature map scales, the reference skin quality features are scaled and mapped into two components, “magnification” and “offset”, through convolution operations.

[0127] Apply the two components of magnification and offset to the skin quality feature to obtain the fusion feature;

[0128] Wherein, the calculation method used for fusing the target image feature and the reference skin quality feature is:

[0129] F fused =F img +M(Conυ(F skin ))

[0130] Among them, Conv is a convolution operation that maps the input feature map to a specific size, and M is a filtering operation:

[0131]

[0132] Step S70, performing loss calculation on the first migration graph and the reference graph to obtain a model loss value, and performing loss calculation on the reconstructed graph and the target graph to obtain a reconstruction loss;

[0133] Optionally, in this step, the model loss value includes a reconstruction loss value, a perception loss value and a discrimination loss value;

[0134] The formula used to calculate the reconstruction loss value is:

[0135]

[0136] in:

[0137] tar indicates that the relevant information comes from the target graph.

[0138] ref indicates that the relevant information comes from the reference graph.

[0139] G is the generator

[0140] F represents feature map (feature-map)

[0141] I represents image input

[0142] img represents image features

[0143] Skin indicates skin characteristics

[0144] The formula used to calculate the perceived loss value is:

[0145] L pec =||σ k (G(F img , F skin ))-σ k (I)|| 2

[0146] Among them, σ k It's VG G The kth layer in the model;

[0147] The formula used to calculate the discrimination loss value is:

[0148]

[0149]

[0150]

[0151] Among them, D f It is the discriminator in the skin quality migration model.

[0152] Furthermore, in this step, the model loss value also includes a skin loss value, a low-frequency loss value, a skin quality loss value and a feature loss value, and the loss calculation is performed based on the migration map and the original map, and also includes:

[0153] Combining the image features of the target image with the skin quality features of the reference image to obtain a first migration map, and combining the skin quality features of the reference image with the image features of the target image to obtain a second migration map;

[0154] Calculate the skin loss value, the low-frequency loss value, the skin quality loss value and the characteristic loss value of the skin quality migration model respectively according to the migration graph;

[0155] Different combinations of features can produce different results. The above combination results include: Get a group of k portraits with the same skin quality as j portraits, Get a group of j portraits with the same skin quality as k portraits, and use O k，j and O j，k Respectively represent the above two mixed results, where:

[0156]

[0157]

[0158] Furthermore, in this step, the formula used to calculate the skin loss value is:

[0159]

[0160] Among them, τ is the pre-trained portrait skin segmentation model, O k，j is the first migration map, O j，k is the second migration map;

[0161] The formula used to calculate the low frequency loss value is:

[0162]

[0163] Among them, δ is the low-frequency signal extraction function;

[0164] The formula used to calculate the skin quality loss value is:

[0165]

[0166]

[0167]

[0168] Among them, η is the high-frequency signal extraction function, M i i∈t, r is the face skin mask, patchify is the screenshot function, D skin is the region discriminator;

[0169] The formula used to calculate the feature loss value is:

[0170]

[0171]

[0172] The formula used to calculate the loss value of the model is:

[0173] L=λ 1 L rcc +λ 2 L img +λ 3 L freq +λ 4 L pec +λ 5 L dis +λ 6 L skin +λ 7 L cons

[0174] Step S80, updating the parameters of the encoder and the generator according to the reconstruction loss and the migration loss, and returning to the step of performing sample division on the training sample image according to the encoder and the generator after the parameter update, until the encoder and the generator after the parameter update meet the preset iteration condition, outputting the skin quality migration model, and obtaining the pre-trained skin quality migration model;

[0175] In this embodiment, the skin quality migration of the portrait is achieved by decoupling and recombining the skin quality features and image features. Since the portrait features contain a large amount of information irrelevant to the skin quality, it is necessary to add an additional encoding and decoding module after fusion, and the intermediate results are repaired and fused multiple times to achieve the final migration purpose. In fact, the portrait features are regarded as the result of the fusion of "target features (skin quality features)" and "target irrelevant features (image features)". By separating these two types of features and recombining them, the migration task can be completed more conveniently, thereby simplifying the migration process. The skin quality of the portrait can also be adjusted by editing the "target features". Shielding portrait differences. There are huge differences between different portraits. Continuing to use the supervision method based on the whole image will introduce a lot of skin quality irrelevant information, resulting in task failure. A reasonable way is needed to shield these differences and let the model focus on the skin quality of the portrait. Therefore, this embodiment uses a slicing method to intercept multiple different patches from the reference portrait for loss calculation to shield the huge differences between portraits. A main feature of skin quality migration that distinguishes it from other (style, makeup) migrations is that only high-frequency signals need to be migrated. Through a reasonable loss calculation function, the model is guided to decouple the "target" features from the "irrelevant to the target" features. Directly calculating the loss on the image is actually mixing the high-frequency signal with the low-frequency signal. Indiscriminate use will bring many difficulties to the task. In order to decouple the skin quality features and the irrelevant to the skin quality features, the high-frequency signal and the low-frequency signal of the image in this embodiment are separated, and different supervision signals are applied to different branches to guide the model to decouple the above two types of features. For the training of the skin quality migration model, the migration method of directly fusing multiple portrait features is abandoned. Its essence is to decouple the "skin quality" features and "non-skin quality" features of the portrait, and does not care about the specific attributes of the image (reference image / target image). Therefore, all training sample images can be either reference images or target images, which greatly reduces the collection cost of sample images.

[0176] Embodiment 3

[0177] See also Fig. 9 , is a schematic diagram of the structure of an image processing system 100 provided in the third embodiment of the present invention, comprising: a feature extraction unit 10, a feature fusion unit 11 and a skin quality migration unit 12, wherein:

[0178] The feature extraction unit 10 is used to input the image to be processed and the target image into the pre-trained skin quality migration model for feature extraction, and obtain a first extracted feature and a second extracted feature. The first extracted feature includes a first image feature and a first skin quality feature, and the second extracted feature includes a second image feature and a second skin quality feature.

[0179] The feature extraction unit 10 is further used to: extract facial key points in the image to be processed and the target image respectively to obtain a first facial point set and a second facial point set;

[0180] Determine a cropping rectangle for facial key points in the to-be-processed image according to the first facial point set to obtain a first cropping rectangle, and determine a cropping rectangle for facial key points in the target image according to the second facial point set to obtain a second cropping rectangle;

[0181] Determining a rotation angle of a face in the to-be-processed image according to the first cropping rectangle to obtain a first rotation angle, and determining a rotation angle of a face in the target image according to the second cropping rectangle to obtain a second rotation angle;

[0182] Performing face correction on the image to be processed according to the first rotation angle, and performing face correction on the target image according to the second rotation angle;

[0183] Performing face cropping on the image to be processed after face correction according to the first cropping rectangle to obtain a first face image, and performing face cropping on the target image after face correction according to the second cropping rectangle to obtain a second face image;

[0184] The first face image and the second face image are downsampled respectively, and the downsampled first face image is set as the image to be processed, and the downsampled second face image is set as the target image.

[0185] The feature fusion unit 11 is used to perform feature fusion on the first image feature and the second skin quality feature according to the pre-trained skin quality migration model to obtain expected image features, and generate an image according to the expected image features to obtain a first migration image.

[0186] The skin quality migration unit 12 is used to perform image scaling on the first migration image and the image to be processed respectively to obtain a first image pyramid and a second image pyramid, and perform skin quality migration on the image to be processed according to the first image pyramid and the second image pyramid to obtain an output image.

[0187] The model training unit 13 is used to obtain a training sample image and perform sample division on the training sample image to obtain a reference sample image and a target sample image;

[0188] Inputting the reference sample image and the target sample image into an encoder in a skin quality migration model for feature extraction to obtain reference features and target features, wherein the reference features include reference image features and reference skin quality features, and the target features include target image features and target skin quality features;

[0189] The target image feature and the reference skin quality feature are fused to obtain a fused feature, the fused feature is sent to a generator to obtain a first migration map, the target image feature and the target skin quality feature are fused and sent to the generator to obtain a reconstructed map;

[0190] Performing loss calculation on the first migration map and the reference sample image to obtain a model loss value, and performing loss calculation on the reconstructed map and the target sample image to obtain a reconstruction loss;

[0191] Exchanging the skin quality features of the reference sample image and the target sample image, and using the generator to generate an image to obtain a migration map, and sending the migration map and the reference sample image to a discriminator to obtain a migration loss;

[0192] The parameters of the encoder and the generator are updated according to the reconstruction loss and the migration loss, and based on the encoder and the generator after the parameter update, the step of performing sample division on the training sample image is returned to be executed, until the encoder and the generator after the parameter update meet the preset iteration condition, and the skin quality migration model is output to obtain the pre-trained skin quality migration model.

[0193] The model training unit 13 is also used to: scale the reference skin quality features at multiple different feature map scales and map them into two components of "multiplier" and "offset" through convolution operation;

[0194] The two components of magnification and offset are applied to the skin quality features to obtain the fusion features.

[0195] Optionally, the model loss value further includes a skin loss value, a low-frequency loss value, a skin quality loss value and a feature loss value, and the model training unit 13 is further used to: perform feature combination of the reference image feature and the target skin quality feature to obtain a first supervised feature, and perform feature combination of the reference skin quality feature and the target image feature to obtain a second supervised feature;

[0196] The skin loss value, the low-frequency loss value, the skin quality loss value and the feature loss value of the skin quality migration model are calculated respectively according to the second supervisory feature.

[0197] In this embodiment, the image to be processed and the target image are respectively input into the pre-trained skin quality migration model for feature extraction, so as to decouple the portrait features in the image to be processed and the target image into image features and skin quality features respectively, and the expected image features are obtained by feature fusion of the first image features and the second skin quality features, and the image is generated based on the expected image features to obtain the first migration image, so that the contour, facial features, background and other skin quality-irrelevant features in the image to be processed and the skin quality features in the target image can be effectively retained in the first migration image, thereby improving the accuracy of skin quality migration, and the first image pyramid and the second image pyramid are obtained by respectively scaling the first migration image and the image to be processed, and the skin quality migration is performed on the image to be processed according to the first image pyramid and the second image pyramid, so that the low-frequency information in the first migration image can be effectively stripped, and the high-frequency information can be migrated to the image to be processed, so as to obtain the output image after the skin quality migration, thereby further improving the accuracy of the skin quality migration.

[0198] Embodiment 4

[0199] Fig.10 2 is a block diagram of a terminal device 2 provided in the fourth embodiment of the present application. Fig.10 As shown, the terminal device 2 of this embodiment includes: a processor 20, a memory 21, and a computer program 22 stored in the memory 21 and executable on the processor 20, such as a program of the image processing method. When the processor 20 executes the computer program 22, the steps in each embodiment of the above-mentioned image processing method are implemented, such as Figure 1 S10 to S30 as shown, or Figure 4 Alternatively, the processor 20 implements the above when executing the computer program 22 Fig. 9 The functions of each unit in the corresponding embodiment are, for example, Fig. 9 For details on the functions of units 10 to 13, please refer to Fig. 9 The relevant descriptions in the corresponding embodiments are not repeated here.

[0200] Exemplarily, the computer program 22 can be divided into one or more units, and the one or more units are stored in the memory 21 and executed by the processor 20 to complete the present application. The one or more units can be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program 22 in the terminal device 2. For example, the computer program 22 can be divided into a feature extraction unit 10, a feature fusion unit 11, a skin quality migration unit 12 and a model training unit 13, and the specific functions of each unit are as described above.

[0201] The terminal device may include, but is not limited to, a processor 20 and a memory 21. Those skilled in the art will appreciate that Fig. 9 It is only an example of the terminal device 2 and does not constitute a limitation of the terminal device 2. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal device may also include input and output devices, network access devices, buses, etc.

[0202] The processor 20 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0203] The memory 21 may be an internal storage unit of the terminal device 2, such as a hard disk or memory of the terminal device 2. The memory 21 may also be an external storage device of the terminal device 2, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 2. Further, the memory 21 may also include both an internal storage unit and an external storage device of the terminal device 2. The memory 21 is used to store the computer program and other programs and data required by the terminal device. The memory 21 may also be used to temporarily store data that has been output or is to be output.

[0204] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0205] If the integrated module is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Among them, the computer-readable storage medium can be non-volatile or volatile. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in computer-readable storage media can be appropriately increased or decreased according to the requirements of legislation and patent practices in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practices, computer-readable storage media do not include electrical carrier signals and telecommunications signals.

[0206] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Inputting the image to be processed and the target image into the pre-trained skin quality migration model for feature extraction, respectively, to obtain a first extracted feature and a second extracted feature, wherein the first extracted feature includes a first image feature and a first skin quality feature, and the second extracted feature includes a second image feature and a second skin quality feature; Performing feature fusion on the first image feature and the second skin quality feature according to the pre-trained skin quality migration model to obtain expected image features, and performing image generation according to the expected image features to obtain a first migration image; The first migration image and the image to be processed are respectively scaled to obtain a first image pyramid and a second image pyramid, and skin quality migration is performed on the image to be processed according to the first image pyramid and the second image pyramid to obtain an output image.

2. The image processing method according to claim 1, characterized in that: Before the image to be processed and the target image are respectively input into the pre-trained skin quality migration model for feature extraction, the method further includes: Acquire a training sample image, and perform sample division on the training sample image to obtain a reference sample image and a target sample image; Inputting the reference sample image and the target sample image into an encoder in a skin quality migration model for feature extraction to obtain reference features and target features, wherein the reference features include reference image features and reference skin quality features, and the target features include target image features and target skin quality features; The image features of the reference image and the skin quality features of the reference image are fused and sent to a generator to obtain a reconstructed reference image, and the target image features and the target skin quality features are fused and sent to the generator to obtain a reconstructed target image; exchanging the skin quality features of the reference sample image and the target sample image, and using the generator to generate an image to obtain a migration map, The reconstructed reference image and the reconstructed target image are respectively subjected to loss calculation with the reference image and the target image to obtain a reconstruction loss; Sending the migration map and the corresponding sample image into the discriminator to obtain the migration loss; The parameters of the encoder and the generator are updated according to the reconstruction loss and the migration loss, and based on the encoder and the generator after the parameter update, the step of performing sample division on the training sample image is returned to be executed, until the encoder and the generator after the parameter update meet the preset iteration condition, and the skin quality migration model is output to obtain the pre-trained skin quality migration model.

3. The image processing method according to claim 2, characterized in that: The model loss value includes a reconstruction loss value, a perception loss value and a discrimination loss value; The formula used to calculate the reconstruction loss value is: in: tar indicates that the relevant information comes from the target graph. ref indicates that the relevant information comes from the reference graph. G is the generator F represents feature map (feature-map) I represents image input img represents image features Skin indicates skin characteristics The formula used to calculate the perceived loss value is: L pec =||σ k (G(F img , F skin ))-σ k (I)||2where, σ k is the kth layer in the VGG model; The formula used to calculate the discrimination loss value is: Among them, D f It is the discriminator in the skin quality migration model.

4. The image processing method according to claim 2, wherein: The step of fusing the target image feature and the reference skin quality feature to obtain a fused feature includes: At multiple different feature map scales, the reference skin quality features are scaled and mapped into two components, "magnification" and "offset", through convolution operations; The two components of magnification and offset are applied to the skin quality features to obtain the fusion features.

5. The image processing method according to claim 3, characterized in that: The model loss value also includes skin area loss value, low frequency loss value, skin quality loss value and feature loss value. The migration map and the corresponding image are subjected to loss calculation to obtain the migration loss value, which also includes: Combining the image features of the target image with the skin quality features of the reference image to obtain a first migration map, and combining the image features of the reference image with the skin quality features of the target image to obtain a second migration map; The skin loss value, the low-frequency loss value, the skin quality loss value and the characteristic loss value of the skin quality migration model are calculated respectively according to the migration map.

6. The image processing method according to claim 5, characterized in that: The formula used to calculate the skin loss value is: Among them, τ is the pre-trained portrait skin segmentation model, O k，j is the first type of migration diagram, O j，k It is the second type of migration diagram; The formula used to calculate the low frequency loss value is: Among them, δ is the low-frequency signal extraction function; The formula used to calculate the skin quality loss value is: Among them, η is the high-frequency signal extraction function, M i i∈t, r is the face skin mask, patchify is the screenshot function, D skin is the region discriminator; The formula used to calculate the feature loss value is: The formula used to calculate the loss value of the model is: L=λ1L rec +λ2L img +λ3L freq +λ4L pec +λ5L dis +λ6L skin +λ7L cons 。 7. The image processing method according to any one of claims 1 to 6, characterized in that: Before the image to be processed and the target image are respectively input into the pre-trained skin quality migration model for feature extraction, the method further includes: Extracting facial key points from the image to be processed and the target image respectively to obtain a first facial point set and a second facial point set; Determine a cropping rectangle for facial key points in the to-be-processed image according to the first facial point set to obtain a first cropping rectangle, and determine a cropping rectangle for facial key points in the target image according to the second facial point set to obtain a second cropping rectangle; Determining a rotation angle of a face in the to-be-processed image according to the first cropping rectangle to obtain a first rotation angle, and determining a rotation angle of a face in the target image according to the second cropping rectangle to obtain a second rotation angle; Performing face correction on the image to be processed according to the first rotation angle, and performing face correction on the target image according to the second rotation angle; Performing face cropping on the image to be processed after face correction according to the first cropping rectangle to obtain a first face image, and performing face cropping on the target image after face correction according to the second cropping rectangle to obtain a second face image; The first face image and the second face image are downsampled respectively, and the downsampled first face image is set as the image to be processed, and the downsampled second face image is set as the target image.

8. An image processing system, characterized in that: The system comprises: A feature extraction unit, used for inputting the image to be processed and the target image into the pre-trained skin quality migration model for feature extraction, to obtain a first extracted feature and a second extracted feature, wherein the first extracted feature includes a first image feature and a first skin quality feature, and the second extracted feature includes a second image feature and a second skin quality feature; A feature fusion unit, configured to perform feature fusion on the first image feature and the second skin quality feature according to the pre-trained skin quality migration model to obtain an expected image feature, and perform image generation according to the expected image feature to obtain a first migration image; The skin quality migration unit is used to perform image scaling on the first migration image and the image to be processed respectively to obtain a first image pyramid and a second image pyramid, and perform skin quality migration on the image to be processed according to the first image pyramid and the second image pyramid to obtain an output image.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images