Methods, devices and computer-readable storage media for profile correction

Abstract

This application discloses a method, apparatus, and computer-readable storage medium for profile correction. The method includes: acquiring a target image, which includes a target face; extracting features from the target image to obtain frontal features of the target face and background features of the background in which the target face is located; fusing the frontal features and background features to obtain fused features; and generating a corrected image based on the fused features, wherein the face in the corrected image is the frontal face corresponding to the target face. The method provided by this application can retain the background information of the input target image during profile correction.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, and in particular relates to a method, device and computer-readable storage medium for profile correction. Background Technology

[0002] The completeness of facial information is crucial for fields such as face recognition and facial attribute analysis. Under normal circumstances, the frontal face contains the richest and most complete facial information. However, due to the angle of the acquisition device and the freedom of human movement, most facial images acquired by these devices are profile images containing some angle. Furthermore, profile images lose facial information to varying degrees due to non-linear changes caused by facial rotation and self-occlusion effects, thus affecting subsequent face-related operations. The goal of profile correction is to generate a corresponding frontal face image containing complete facial information based on the input profile image with missing facial information, thereby significantly improving the effectiveness of face analysis by supplementing the facial information.

[0003] However, when performing profile correction, the background information in the input image is easily ignored, resulting in the generated frontal image failing to reproduce the background in the input image. Summary of the Invention

[0004] This application provides a method, device, and computer-readable storage medium for profile correction, which can retain background information of the input target image during profile correction.

[0005] A first aspect of this application provides a method for profile correction, the method comprising: acquiring a target image, the target image including a target face; extracting features from the target image to obtain frontal features of the target face and background features of the background in which the target face is located; fusing the frontal features and the background features to obtain fused features; and generating a corrected image based on the fused features, wherein the face in the corrected image is a frontal face corresponding to the target face.

[0006] A second aspect of this application provides a correction device, which includes a processor, a memory, and a communication circuit. The processor is coupled to the memory and the communication circuit, respectively. The memory stores program data, and the processor executes the program data in the memory to implement the steps in the above method.

[0007] A third aspect of this application provides a computer-readable storage medium storing a computer program that can be executed by a processor to implement the steps in the above-described method.

[0008] The beneficial effect is that this application generates a corrected image based on the fusion features. Since the fusion features combine frontal features and background features, the generated corrected image can retain the background information in the input target image, and ultimately ensures that there will be no sense of incongruity when the corrected image is placed into the original image. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort, wherein:

[0010] Figure 1 This is a flowchart illustrating one implementation method of the profile correction method of this application;

[0011] Figure 2 yes Figure 1 A flowchart illustrating step S120;

[0012] Figure 3 yes Figure 2 A flowchart illustrating step S121;

[0013] Figure 4 This is a partial flowchart illustrating another embodiment of the profile correction method of this application;

[0014] Figure 5 yes Figure 4 A flowchart illustrating step S270;

[0015] Figure 6 yes Figure 4 A flowchart illustrating step S280;

[0016] Figure 7 yes Figure 4 A flowchart illustrating step S210;

[0017] Figure 8 This is a schematic diagram of one embodiment of the correction device of this application;

[0018] Figure 9 This is a schematic diagram of another embodiment of the correction device of this application;

[0019] Figure 10 This is a schematic diagram of one embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0021] See Figure 1 , Figure 1 This is a flowchart illustrating one embodiment of the profile correction method of this application, which includes:

[0022] S110: Acquire the target image, which includes the target human face.

[0023] Specifically, the target image contains only one face, which is defined as the target face. The target face can be either the frontal or side profile of the target object. The purpose of this application is to correct the side profile of the target image to a frontal face, that is, to make the face in the final corrected image a frontal face.

[0024] Specifically, the target image is cropped from the original image. The original image contains at least one face. After target recognition is performed on the original image, a detection box is generated corresponding to each face. Each detection box selects the corresponding face, and then the image within each detection box is cropped to obtain at least one target image. It is understood that when there are multiple target images, subsequent steps S120 to S140 are performed for each target image.

[0025] After obtaining the corrected image corresponding to the target image through subsequent steps, it is also necessary to place the corrected image corresponding to the target image into the corresponding position in the original image.

[0026] S120: Extract features from the target image to obtain the frontal features of the target face and the background features of the background in which the target face is located.

[0027] Specifically, the frontal features of the target face carry frontal information about the target face, while the background features of the background surrounding the target face carry background information about the target face.

[0028] See Figure 2 In this embodiment, step S120 specifically includes:

[0029] S121: Separate the face image of the target person and the background image of the background where the target face is located from the target image.

[0030] Specifically, the target face and background in the target image are separated to obtain a face image and a background image. The size of the face image and the background image are the same as the size of the target image. The face image can be understood as being obtained by filling the background area of ​​the target image with a solid color (e.g., black), and the background image can be understood as being obtained by filling the face area of ​​the target image with a solid color (e.g., black).

[0031] S122: Extract frontal face features from the face image to obtain frontal face features, and extract background features from the background image to obtain background features.

[0032] Specifically, when extracting features from a face image, the frontal features of the target face are extracted, not all the features of the entire target face. Therefore, the obtained frontal features carry the frontal information of the target face.

[0033] In this embodiment, the face image and background image are separated from the target image, and then the frontal face features and background features are extracted. Compared with directly extracting the frontal face features and background features from the target image, the influence of the background when extracting the frontal face features and the influence of the target face when extracting the background features can be reduced.

[0034] However, in other implementations, it is also possible to extract frontal and background features directly from the target image without separating the face image and background image.

[0035] See Figure 3 In this embodiment, step S121 specifically includes:

[0036] S1211: Generate a 3D face model of the target face based on the target image.

[0037] Specifically, a pre-trained 3D face estimation model is used to estimate the 3D face model of the target face in the target image. The 3D face estimation model is pre-trained and can generate the 3D face model of the face in the received image.

[0038] S1212: Generate a face mask of the target face on the target image based on the 3D face model.

[0039] Specifically, step S1212, which generates a face mask, can be understood as follows: the angle and position of the 3D face model are adjusted to correspond to the target face in the target image, and then the 3D face model is projected onto a mask image that is exactly the same size as the target image. Then, the pixel values ​​of the pixels in the orthographic projection area of ​​the 3D face model on the mask image are set to the first pixel value, and the pixel values ​​of the pixels in other areas are set to the second pixel value that is different from the first pixel value. The final mask image is the face mask.

[0040] S1213: Based on the face mask, separate the face image and the background image from the target image.

[0041] Specifically, the position of the target face in the target image can be determined based on the face mask, thereby separating the face image and the background image from the target image.

[0042] In other embodiments, step S121 may specifically involve: using methods such as edge detection to determine the outline of the target face in the target image, and then separating the face image and the background target from the target image based on the outline of the target face.

[0043] Understandably, when determining the outline of a target face, if the target face and the background color are similar, determining the outline becomes more difficult and less accurate. Therefore, if the face image and background target are separated from the target image based on the outline of the target face, the accuracy of the separation is easily affected by the colors of the target face and the background.

[0044] However, in this embodiment, a three-dimensional face estimation model is used to generate a three-dimensional face model. Since the three-dimensional face estimation model is pre-trained, the process of generating the three-dimensional face model does not require determining the outline of the target face. Therefore, this embodiment uses a three-dimensional face model to separate the face image from the background image with high accuracy and low difficulty.

[0045] The above describes the specific process of obtaining frontal and background features in step S120. The following describes the steps after step S120.

[0046] S130: The frontal face features and background features are fused to obtain the fused features.

[0047] Specifically, any existing fusion technique can be used to fuse the frontal face features and background features. For example, in one application scenario, the frontal face features have a dimension of 256, the background features have a dimension of 256, and after fusion processing, a fused feature with a dimension of 512 is obtained.

[0048] S140: Generate a corrected image based on the fusion features, wherein the face in the corrected image is a frontal face corresponding to the target face.

[0049] The corrected image and the target image have the same size; for example, both the corrected image and the target image are 128×128.

[0050] In this context, the face in the corrected image is the frontal face corresponding to the target face; that is, the corrected image is the image after the target face in the target image has been corrected to a frontal face.

[0051] In this embodiment, a corrected image is generated based on the fusion features. Since the fusion features combine frontal features and background features, the generated corrected image can retain the background information in the input target image, thus ensuring that no incongruity occurs when the corrected image is placed into the original image.

[0052] In this embodiment, step S120 specifically includes: extracting features from the target image using the encoder in the pre-trained profile correction model to obtain frontal face features and background features; step S140 specifically includes: inputting the fused features into the generator in the profile correction model to generate a corrected image.

[0053] Specifically, in order to improve the speed and accuracy of profile correction, a pre-trained profile correction model is used in the profile correction process.

[0054] In one application scenario, in order to improve processing speed and efficiency, the target image is preprocessed before being input into the profile correction model. The preprocessing process may be as follows: normalize the target image, then perform data augmentation on the normalized target image, and finally scale the augmented target image to a preset size, such as 128×128.

[0055] The normalization process can be performed by dividing the pixel value of each pixel in the target image by 255, so that the pixel value of each pixel after normalization is within the range [0, 1].

[0056] In other application scenarios, the target image may not need to be preprocessed before being input into the profile correction model.

[0057] In one application scenario, the profile correction model has only one encoder. In this case, the encoder directly extracts features from the target image to obtain frontal face features and background features. That is, when the target image is input into the encoder, the encoder outputs frontal face features and background features.

[0058] In another application scenario, the profile correction model uses two encoders, designated as the first encoder and the second encoder. After separating the target face image and the background image from the target image, the first encoder extracts frontal features from the face image, and the second encoder extracts background features from the background image. In other words, the face image is input into the first encoder to obtain frontal features, and the background image is input into the second encoder to obtain background features.

[0059] Regardless of whether the profile correction model has one or two encoders, each encoder can include three ConvBlocks. Each ConvBlock includes two Con2D layers, one ResBlock, one BatchNorm2D layer, and one LeakyReLU layer. Each ResBlock contains three Con2D layers and one LeakyReLU layer connected in a skip-layer manner. This application does not limit the specific structure of the encoder in the profile correction model.

[0060] In a Generative Adversarial Network (GAN), the generator is the component that makes the network work. A GAN consists of a generator and a discriminator. During training, the generator aims to produce realistic images to deceive the discriminator, while the discriminator tries to distinguish the generated images from real images. This creates a dynamic game between the generator and the discriminator. In other words, the generator's task is to generate fake images that closely resemble real images, while the discriminator's task is to determine whether a received image is real or fake after receiving both real and generated fake images.

[0061] Therefore, after training the generator in the profile correction model, the face in the corrected image generated by the generator is a frontal face or very close to a frontal face, thereby achieving the purpose of profile correction.

[0062] In one application scenario, the generator consists of three UpConvBlocks, each containing two Con2D layers, one PixelShuffle layer, one ResBlock, one BatchNorm2D layer, and one LeakyReLU layer. When the fused feature dimension is 512, the resolution of the input fused feature doubles with each UpConvBlock pass, but the feature dimension decreases to one-quarter of its original value. After three UpConvBlock passes, a 128×128 corrected image is obtained.

[0063] This application does not impose any restrictions on the specific structure of the generator.

[0064] The following describes the training process for the profile correction model. (See attached document.) Figure 4 The process of training a profile correction model includes:

[0065] S210: Acquire multiple first sample images, each of which includes a first sample face.

[0066] Specifically, the first sample image includes a face, which is defined as the first sample face. The first sample face in the first sample image can be a frontal face or a side face. The first sample faces in different first sample images can be the faces of the same sample object or the faces of different sample objects.

[0067] S220: Use the encoder to extract features from each first sample image to obtain the first sample frontal face features of the first sample face and the first sample background features of the background where the first sample face is located in each first sample image.

[0068] Before inputting the first sample image into the profile correction model, the first sample image is also preprocessed. This process is similar to the preprocessing of the target image described above. For details, please refer to the relevant content above, which will not be repeated here.

[0069] Specifically, the process of extracting features from the first sample image is the same as the process of extracting features from the target image described above, and will not be repeated here.

[0070] After step S220, each first sample image corresponds to a first sample frontal face feature and a first sample background feature.

[0071] S230: Use the encoder to extract features from the corresponding second sample image of each first sample image to obtain the second sample frontal face features of the second sample face in each second sample image. The first sample face in the first sample image and the second sample face in the corresponding second sample image correspond to the same sample object. The background in the first sample image and the background in the corresponding second sample image correspond to the same background. The second sample face in each second sample image is a frontal face.

[0072] Specifically, each first sample image corresponds to a second sample image. The second sample image corresponding to the first sample image refers to a second sample face in the second sample image that is a frontal face, and the second sample face and the first sample face in the first sample image are the same person's face. Additionally, the background in the second sample image and the background in the first sample image are the same background. To facilitate understanding, a specific example is provided: assuming the first sample image includes a profile of person A, then the face in the second sample image corresponding to the first sample image must be a frontal face of person A, and the background in the second sample image must be the same background as the background in the first sample image.

[0073] It is understandable that the second sample images corresponding to different first sample images may be the same image or they may not be the same image.

[0074] The process of extracting the frontal facial features of the second sample image from the second sample image is the same as the process of extracting the frontal facial features of the first sample image from the first sample image, as described above.

[0075] S240: The frontal face features and background features of the first sample corresponding to each first sample image are fused to obtain the sample fusion features corresponding to each first sample image.

[0076] Specifically, the process of fusing the frontal face features and the background features of the first sample is the same as the process of fusing the frontal face features and the background features described above. For details, please refer to the relevant content above.

[0077] It is understandable that after step S240, each first sample image corresponds to a sample fusion feature.

[0078] S250: Input the fusion features of each sample into the generator to obtain the sample correction image corresponding to each first sample image.

[0079] Specifically, after inputting the sample fusion features corresponding to the first sample image into the generator, the generator outputs a sample-corrected image corresponding to the first sample image. This sample-corrected image can be understood as the image after correcting the face of the first person sample in the first sample image to a frontal face. However, since it is still in the training stage, the correction effect of the generator has not yet reached the ideal state. Only after continuous training will the face in the sample-corrected image generated by the generator be or close to the frontal face of the sample object corresponding to the first sample face.

[0080] S260: Determine the first loss value based on multiple first sample frontal face features and the corresponding second sample frontal face features of each first sample frontal face feature.

[0081] Specifically, the second sample frontal face feature corresponding to the first sample frontal face feature refers to the second sample frontal face feature corresponding to the second sample image corresponding to the first sample image corresponding to the first sample frontal face feature.

[0082] The first loss value represents the accuracy of the encoder in extracting the first sample frontal face feature of the first sample face. One of the purposes of training the side face correction model is to make the first sample frontal face feature extracted by the encoder either or close to the frontal face feature of the sample object corresponding to the first sample face, that is, to make the first sample frontal face feature either or close to the second sample frontal face feature corresponding to the first sample frontal face feature.

[0083] Specifically, step S260 can be performed as follows: based on each first sample frontal face feature and the second sample frontal face feature corresponding to the first sample frontal face feature, determine the frontal face sub-loss value corresponding to each first sample frontal face feature; then, based on the frontal face sub-loss value corresponding to each first sample frontal face feature, determine the first loss value.

[0084] Specifically, the L1 loss function can be used to determine the frontal face sub-loss value corresponding to each frontal face feature of the first sample, and the frontal face sub-loss value corresponding to each frontal face feature of the first sample can be summed to obtain the first loss value, or the average value of the frontal face loss values ​​can be obtained to obtain the first loss value.

[0085] In summary, this application does not impose any restrictions on the specific process for determining the first loss value, as long as the first loss value can characterize the accuracy of the encoder in extracting frontal face features.

[0086] S270: Determine the second loss value based on multiple sample corrected images and the second sample image corresponding to the sample corrected image.

[0087] Specifically, the second sample image corresponding to the sample-corrected image refers to the second sample image corresponding to the first sample image corresponding to the sample-corrected image.

[0088] The second loss value represents the accuracy of the generator in generating the sample correction image corresponding to the first sample image. One of the purposes of training the profile correction model is to make the face in the sample correction image generated by the generator be the same as or close to the face in the second sample image corresponding to the first sample image, that is, to make the correction image close to or the same as the corresponding second sample image.

[0089] See Figure 5 In one application scenario, step S270 specifically includes:

[0090] S271: Determine the first sub-loss value based on the pixel values ​​of the multiple sample corrected images and the pixel values ​​of the second sample image corresponding to each sample corrected image.

[0091] Specifically, based on the pixel values ​​of each sample corrected image and the pixel values ​​of the corresponding second sample image, the pixel sub-loss value of each sample corrected image is determined, and then the first sub-loss value is determined based on the pixel sub-loss value of each sample corrected image.

[0092] In determining the pixel sub-loss value corresponding to each sample corrected image, the L1 loss function can be used for calculation. Specifically, the L1 loss function is used to calculate the pixel value error between the sample corrected image and the corresponding second sample image pixel by pixel, thus obtaining the pixel sub-loss value corresponding to the sample corrected image. The L1 loss function is also called the absolute value loss function, and its purpose is to minimize the sum of the absolute differences between the target value and the estimated value. For more information on the L1 loss function, please refer to existing technologies; it will not be described in detail here.

[0093] In this process, the pixel sub-loss value corresponding to each sample corrected image is calculated by performing processes such as summation and averaging to obtain the first sub-loss value.

[0094] As can be seen from the above process, the first sub-loss value represents the pixel-level loss between the sample corrected image and the corresponding second sample image.

[0095] S272: Determine the second sub-loss value based on the facial features of multiple sample corrected images and the facial features of the second sample image corresponding to each sample corrected image.

[0096] Specifically, the facial features of the sample corrected image and the facial features of the second sample image both include frontal and side information of the corresponding face.

[0097] For each corrected image and each second image, a pre-trained face feature extraction network can be used to extract face features. In one application scenario, the pre-trained face feature extraction network is based on a VGG19 network pre-trained on the MS_Celeb_1M dataset. The specific structure of the face feature extraction network is not limited.

[0098] Specifically, step S272 can be: determining the perceptron loss value corresponding to each sample correction image based on the facial features of each sample correction image and the facial features of the corresponding second sample image, and then determining the second sub-loss value based on the perceptron loss value corresponding to each sample correction image.

[0099] The distance between the facial features of the sample-corrected image and the facial features of the corresponding second sample image can be calculated using the L1 loss function to obtain the perceptron loss value corresponding to the sample-corrected image.

[0100] In this process, the perceptron loss value corresponding to each sample corrected image is calculated by performing operations such as summation and averaging to obtain the second sub-loss value.

[0101] As can be seen from the above process, the second sub-loss value represents the perceptual loss between the sample corrected image and the corresponding second sample image.

[0102] S273: Determine the third sub-loss value based on the first part features of at least one target part on the face in multiple sample corrected images and the second part features of at least one target part on the face in the second sample image corresponding to each sample corrected image.

[0103] Specifically, for each sample correction image, the target parts of the face are cropped from the sample correction image using techniques such as key point recognition. The target parts include at least one of the eyes, nose, and mouth. Then, feature extraction is performed on each target part to obtain the corresponding first part features.

[0104] Similarly, for each second sample image, the target part of the face is cropped from the second sample image, and then feature extraction is performed on each target part to obtain the corresponding second part features.

[0105] When extracting features from the target area, a network such as VGG19 can be used.

[0106] In this case, at least one target part corresponding to the sample correction image corresponds one-to-one with at least one target part corresponding to the second sample image. For example, if the feature extraction for the sample correction image is for the nose and eyes, then the feature extraction for the second sample image is also for the nose and eyes.

[0107] When the target region is a single region, such as the nose, each corrected image corresponds to a first region feature, which is the nose region feature. Similarly, each second image corresponds to a second region feature, also a nose region feature. In this case, step S273 specifically includes: determining the nose region sub-loss value for each corrected image based on the first region feature (nose region feature) and the second region feature (nose region feature) of each corrected image; and then determining the third sub-loss value based on the region sub-loss value. For example, the region sub-loss value for each corrected image can be calculated by performing operations such as summation or averaging to obtain the third sub-loss value.

[0108] When there are multiple target areas, such as the nose, left eye, and right eye, the above method is used to determine the sub-loss values ​​for the nose, left eye, and right eye for each sample corrected image. Then, based on these sub-loss values, a third sub-loss value is determined. For example, the nose sub-loss value is obtained by summing the nose sub-loss values ​​for each sample corrected image; the left eye sub-loss value is obtained by summing the left eye sub-loss values ​​for each sample corrected image; the right eye sub-loss value is obtained by summing the right eye sub-loss values ​​for each sample corrected image; finally, the nose loss value, left eye loss value, and right eye loss value are processed, for example, by weighted summation or averaging, to obtain the third sub-loss value.

[0109] Specifically, the L1 loss function can be used to calculate the part-loss value corresponding to the nose, the part-loss value corresponding to the left eye, and the part-loss value corresponding to the right eye for each sample corrected image.

[0110] As can be seen from the above process, the third sub-loss value represents the loss of facial details between the sample corrected image and the corresponding second sample image.

[0111] S274: Determine the second loss value based on at least one of the first sub-loss value, the second sub-loss value, and the third sub-loss value.

[0112] The first, second, or third sub-loss value can be directly determined as the second loss value, or the second loss value can be determined by combining two or three of the first, second, and third sub-loss values.

[0113] For example, in one application scenario, the first sub-loss value, the second sub-loss value, and the third sub-loss value are directly summed to obtain the second loss value.

[0114] For example, in another application scenario, the first sub-loss value, the second sub-loss value, and the third sub-loss value are weighted and summed to obtain the second loss value.

[0115] In obtaining the second loss value using a weighted summation method, considering that pixel-level loss is better at reflecting the similarity between the sample-corrected image and the second sample image than perceptual loss and loss of facial details, and that perceptual loss is better at reflecting the similarity between the sample-corrected image and the second sample image than loss of facial details, the weights of the first sub-loss value, the second sub-loss value, and the third sub-loss value are set to decrease sequentially.

[0116] It should be noted that this application does not limit the weights of the first sub-loss value, the second sub-loss value, and the third sub-loss value; designers can flexibly set them according to actual needs.

[0117] When the second loss value is determined by combining the first, second, and third sub-loss values, compared to directly determining the first sub-loss value as the total loss value, the second sub-loss value is related to facial features and the third sub-loss value is related to facial details. Therefore, on the one hand, it can ensure that the corrected image generated by the side face correction model contains facial feature information and avoids the corrected image being too smooth. On the other hand, it can also ensure that the corrected image can restore the detailed information of the target face, such as expression information.

[0118] S280: Determine the total loss value based on the first loss value and the second loss value.

[0119] Specifically, after obtaining the total loss value based on the first loss value and the second loss value, the profile correction model is trained based on the total loss value. The specific process is to adjust the parameters of the profile correction model based on the total loss value so that the profile correction model develops in the direction of convergence.

[0120] The total loss value can be obtained by directly summing the first and second loss values, or by weighted summing of the first and second loss values.

[0121] In the process of weighted summation of the first loss value and the second loss value to obtain the total loss value, it is considered that the key to the side face correction of the first sample face in the first sample image is to be able to extract the front face features of the first sample face. At the same time, the encoder has a greater influence on the final correction result than the generator. Therefore, the weight of the first loss value is set to be greater than the weight of the second loss value.

[0122] Of course, the weights corresponding to the first and second loss values ​​can be flexibly set by the designers according to actual needs, and there are no restrictions here.

[0123] See Figure 6 In one application scenario, step S280 includes:

[0124] S281: Input multiple sample corrected images and the second sample image corresponding to each sample corrected image into the discriminator in the side face correction model for discrimination.

[0125] S282: Generate a third loss value based on the discriminator's judgment result.

[0126] Specifically, the discriminator in the profile correction model is the discriminator in a generative adversarial network, as detailed above.

[0127] For each corrected image sample, perform the following steps:

[0128] The sample-corrected image and the corresponding second sample image (i.e., the second sample image corresponding to the first sample image corresponding to the sample-corrected image) are input into the discriminator for discrimination. Based on the discrimination result output by the discriminator, the discriminant sub-loss value corresponding to each sample-corrected image is determined.

[0129] After performing the above steps for each sample corrected image, the discriminant loss value corresponding to each sample corrected image can be obtained. Then, the discriminant loss value corresponding to each sample corrected image is calculated by, for example, summing or averaging, to obtain the third sub-loss value.

[0130] S283: Determine the total loss value based on the first loss value, the second loss value, and the third loss value.

[0131] In one application scenario, the first loss value, the second loss value, and the third loss value are directly summed to obtain the total loss value.

[0132] In another application scenario, the first loss value, the second loss value, and the third loss value are weighted and summed to obtain the total loss value.

[0133] In determining the total loss value using the weighted summation method, considering that the encoder's influence on the final correction result is greater than the generator's influence on the final correction result, and the generator's influence on the final correction result is greater than the discriminator's influence on the final correction result, the weights corresponding to the first loss value, the second loss value, and the third loss value are set to decrease sequentially.

[0134] In one application scenario, when the second loss value is obtained by weighted summation of the first, second, and third sub-loss values, the total loss value L can be determined according to the following formula. total :

[0135] L total =λ1×L1+λ2×L2+λ3×L3+λ4×L4+λ5×L5;

[0136] Where L1 is the first loss value, L2 is the first sub-loss value, L3 is the second sub-loss value, L4 is the third sub-loss value, L5 is the third loss value, λ1 is the weight corresponding to the first loss value, λ2 is the weight corresponding to the first sub-loss value, λ3 is the weight corresponding to the second sub-loss value, λ4 is the weight corresponding to the third sub-loss value, and λ5 is the weight corresponding to the third loss value. At this time, the values ​​of λ1, λ2, λ3, λ4 and λ5 decrease in sequence. For example, λ1 = 15, λ2 = 5, λ3 = 2, λ4 = 1 and λ5 = 0.1.

[0137] S290: Train the profile correction model based on the total loss value.

[0138] Specifically, step S290 involves training the encoder and generator in the profile correction model based on the total loss value. When there are two encoders in the profile correction model, step S290 trains both encoders and the generator in the profile correction model.

[0139] The total loss value represents both the accuracy of the encoder in extracting frontal facial features and the accuracy of the generator in generating the corrected image. Therefore, training the profile correction model based on the total loss value allows the encoder and generator in the model to gradually improve, ultimately leading to convergence. Furthermore, determining the second loss value based on the first, second, and third sub-loss values ​​ensures that the generated corrected image is rich in feature information, avoiding excessive smoothness, and also guarantees that the generated corrected image can reproduce the detailed information of the target face.

[0140] In step S290, when training the profile correction model, the stochastic gradient descent algorithm can be used to train the encoder and generator. For example, the initial learning rate can be set to 0.001, the number of iterations (epochs) can be set to 100, and the learning rate can be reduced to 80% after every 5 epochs.

[0141] During the training phase, the discriminator in the profile correction model also needs to be trained. However, the discriminator is not trained based on the total loss value, but rather on the discriminator's judgment results. Any existing method can be used to train the discriminator in the profile correction model; the process will not be detailed here.

[0142] See Figure 7 In this embodiment, step S210, which involves acquiring multiple first sample images, includes:

[0143] S211: Obtain the second sample image.

[0144] Specifically, any image that includes a frontal face can be used as a second sample image.

[0145] S212: Based on the second sample image, generate a 3D face model of the second sample face in the second sample image.

[0146] Specifically, the process of generating the 3D face model of the second sample face is the same as the process of generating the 3D face model of the target face described above. For details, please refer to the relevant content above.

[0147] S213: Extract the original color information of the second sample face from the second sample image and add the original color information to the 3D face model.

[0148] Specifically, by adding the original color information of the second sample face to the 3D face model, the 3D face model has color and texture information, making it closer to a real face.

[0149] S214: After rotating the 3D face model at different angles, render it onto a 2D plane to obtain multiple first sample images corresponding to the second sample image.

[0150] Specifically, the 3D face model with added original color information is randomly rotated at multiple angles. After each rotation, the 3D face model with color and texture information is rendered onto a 2D plane using a rasterization method to obtain multiple side face images. The background of these multiple side face images is the same as the background of the second sample image. These multiple side face images are the multiple first sample images corresponding to the second sample image.

[0151] In existing technologies, training sets mostly consist of face images at specific angles corresponding to yaw angles, such as {15°, 30°, 45°, 60°, 75°, 90°}, where the pitch and roll angles corresponding to these images are both 0°.

[0152] This implementation method uses a rotated 3D face model, which can randomly rotate the yaw and pitch angles, so that the rotated yaw angle is distributed in the interval [-90°, 90°] and the rotated pitch angle is distributed in the interval [-45°, 45°]. This increases the richness of the first sample face pose in the first sample image of the training set, improves the diversity of the training data, and makes the trained side face correction model have good generalization ability.

[0153] See Figure 8 , Figure 8This is a schematic diagram of one embodiment of the correction device of this application. The correction device 200 includes a processor 210, a memory 220, and a communication circuit 230. The processor 210 is coupled to the memory 220 and the communication circuit 230 respectively. The memory 220 stores program data. The processor 210 executes the program data in the memory 220 to implement the method steps in any of the above embodiments. The detailed steps can be found in the above embodiments and will not be repeated here.

[0154] The correction device 200 can be any device with image processing capabilities, such as a computer or mobile phone, and there are no restrictions on its use.

[0155] See Figure 9 , Figure 9 This is a schematic diagram of another embodiment of the correction device of this application. The correction device 300 includes an acquisition module 310, an extraction module 320, a fusion module 330, and a correction module 340.

[0156] The acquisition module 310 is used to acquire the target image, which includes the target human face.

[0157] The extraction module 320 is connected to the acquisition module 310 and is used to extract features from the target image to obtain the frontal features of the target face and the background features of the background where the target face is located.

[0158] The fusion module 330 is connected to the extraction module 320 and is used to fuse the frontal face features and background features to obtain fused features.

[0159] The correction module 340 is connected to the fusion module 330 and is used to generate a corrected image based on the fusion features, wherein the face in the corrected image is a frontal face corresponding to the target face.

[0160] When the correction device 300 is in operation, it performs the method steps of any of the above embodiments. For detailed steps, please refer to the relevant content above, which will not be repeated here.

[0161] The correction device 300 can be any electronic device with image processing capabilities, such as a mobile phone or a computer, and there are no restrictions on its use.

[0162] See Figure 10 , Figure 10 This is a schematic diagram of one embodiment of the computer-readable storage medium of this application. The computer-readable storage medium 400 stores a computer program 410, which can be executed by a processor to implement the steps in any of the above methods.

[0163] Specifically, the computer-readable storage medium 400 can be a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, or a device that can store the computer program 410. Alternatively, it can be a server that stores the computer program 410, which can send the stored computer program 410 to other devices for execution, or it can run the stored computer program 410 itself.

[0164] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for profile correction, characterized in that, The method includes: Acquire a target image, wherein the target image includes a target human face; Features are extracted from the target image to obtain the frontal facial features of the target face and the background features of the background in which the target face is located; The frontal features and the background features are fused together to obtain the fused features; Based on the fusion features, a corrected image is generated, wherein the face in the corrected image is a frontal face corresponding to the target face; The step of extracting features from the target image to obtain the frontal facial features of the target face and the background features of the background in which the target face is located includes: The encoder in the pre-trained profile correction model extracts features from the target image to obtain the frontal face features and the background features; The step of generating a corrected image based on the fusion features includes: The fused features are input into the generator in the profile correction model to generate the corrected image.

2. The method according to claim 1, characterized in that, The step of extracting features from the target image to obtain the frontal facial features of the target face and the background features of the background in which the target face is located includes: Separate the face image of the target human face and the background image of the background in which the target face is located from the target image; The face image is subjected to frontal face feature extraction to obtain the frontal face feature, and the background image is subjected to feature extraction to obtain the background feature.

3. The method according to claim 2, characterized in that, The step of separating the face image of the target person and the background image of the background where the target face is located from the target image includes: Based on the target image, generate a three-dimensional face model of the target face; Based on the three-dimensional face model, a face mask of the target face on the target image is generated; Based on the face mask, the face image and the background image are separated from the target image.

4. The method according to claim 1, characterized in that, Before acquiring the target image, the following is also included: Acquire multiple first sample images, each of which includes a first sample face; The encoder is used to extract features from each of the first sample images to obtain the first sample frontal face features of the first sample face and the first sample background features of the background where the first sample face is located in each of the first sample images; The encoder is used to extract features from the second sample image corresponding to each first sample image to obtain the second sample frontal face features of the second sample face in each second sample image. The first sample face in the first sample image and the second sample face in the second sample image corresponding to the first sample image correspond to the same sample object. The background in the first sample image and the background in the second sample image corresponding to the first sample image correspond to the same background. The second sample face in each second sample image is a frontal face. The face features and background features of the first sample corresponding to each first sample image are fused to obtain the sample fusion features corresponding to each first sample image. Each of the sample fusion features is input into the generator to obtain the sample correction image corresponding to each of the first sample images; A first loss value is determined based on multiple first sample frontal face features and the second sample frontal face features corresponding to each of the first sample frontal face features; A second loss value is determined based on the plurality of the sample corrected images and the second sample image corresponding to the sample corrected images; The total loss value is determined based on the first loss value and the second loss value; The profile correction model is trained based on the total loss value.

5. The method according to claim 4, characterized in that, The step of determining the second loss value based on the plurality of sample corrected images and the second sample image corresponding to the sample corrected images includes: A first sub-loss value is determined based on the pixel values ​​of the plurality of the sample corrected images and the pixel values ​​of the second sample image corresponding to each of the sample corrected images; The second sub-loss value is determined based on the facial features of the multiple corrected sample images and the facial features of the second sample image corresponding to each corrected sample image; A third sub-loss value is determined based on the first part features of at least one target part on the face in the plurality of sample corrected images and the second part features of at least one target part on the face in the second sample image corresponding to each sample corrected image; The second loss value is determined based on at least one of the first sub-loss value, the second sub-loss value, and the third sub-loss value.

6. The method according to claim 5, characterized in that, The step of determining the second loss value based on at least one of the first sub-loss value, the second sub-loss value, and the third sub-loss value includes: The first sub-loss value, the second sub-loss value, and the third sub-loss value are weighted and summed to obtain the second loss value.

7. The method according to claim 6, characterized in that, The weights corresponding to the first sub-loss value, the second sub-loss value, and the third sub-loss value decrease sequentially.

8. The method according to claim 4, characterized in that, The step of determining the total loss value based on the first loss value and the second loss value includes: The multiple corrected sample images and the second sample image corresponding to each corrected sample image are respectively input into the discriminator in the profile correction model for discrimination; A third loss value is generated based on the discrimination result of the discriminator; The total loss value is determined based on the first loss value, the second loss value, and the third loss value.

9. The method according to claim 8, characterized in that, The step of determining the total loss value based on the first loss value, the second loss value, and the third loss value includes: The first loss value, the second loss value, and the third loss value are weighted and summed to obtain the total loss value.

10. The method according to claim 9, characterized in that, The weights corresponding to the first loss value, the second loss value, and the third loss value decrease sequentially.

11. The method according to claim 4, characterized in that, The step of acquiring multiple first sample images includes: Obtain the second sample image; Based on the second sample image, generate a three-dimensional face model of the second sample face in the second sample image; Extract the original color information of the second sample face from the second sample image, and add the original color information to the three-dimensional face model; The three-dimensional face model is rotated at different angles and then rendered onto a two-dimensional plane to obtain multiple first sample images corresponding to the second sample image.

12. A corrective device, characterized in that, The correction device includes a processor, a memory, and a communication circuit. The processor is coupled to the memory and the communication circuit. The memory stores program data. The processor executes the program data in the memory to implement the steps of the method as described in any one of claims 1-11.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that can be executed by a processor to implement the steps of the method as described in any one of claims 1-11.

Images