Image processing model training method, image processing method and device

Abstract

The application discloses an image processing model training method, an image processing method and device, and belongs to the technical field of image processing. The model training method comprises the following steps: acquiring a training data pair of image training data, the training data pair comprising a first image processing parameter, a first original image, a second original image, a first reference image and a second reference image; wherein the first reference image is obtained by processing the first original image based on the first image processing parameter, the second reference image is obtained by processing the second original image based on the first image processing parameter, and the first original image and the second original image are face images corresponding to a same shooting object; performing encoding processing on the training data pair to obtain a first face feature, a first makeup feature and a first instruction feature; and training an initial image processing model based on the first face feature, the first makeup feature and the first instruction feature to obtain a target image processing model.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, specifically relating to an image processing model training method, an image processing method, and an apparatus. Background Technology

[0002] With the rapid development of mobile terminal image processing technology, beautification functions have become one of the core needs of users in their daily photography.

[0003] Currently, facial image beautification typically utilizes one-click beautification functions, which may distort the user's facial proportions and fail to match the user's unique facial features. Alternatively, users can manually adjust beautification parameters such as skin smoothing and face slimming, which relies heavily on user experience and has a high operational threshold. If the parameters are not adjusted properly, the final beautification effect will be unnatural. Summary of the Invention

[0004] The purpose of this application is to provide an image processing model training method, an image processing method, and an apparatus that can obtain images with better beautification effects that match the user's unique facial features, and improve image beautification efficiency without requiring the user to manually adjust beautification parameters.

[0005] In a first aspect, embodiments of this application provide an image processing model training method, the method comprising: A training data pair for image training data is obtained, comprising a first image processing parameter, a first original image, a second original image, a first reference image, and a second reference image; wherein the first reference image is obtained by processing the first original image based on the first image processing parameter, and the second reference image is obtained by processing the second original image based on the first image processing parameter, and the first original image and the second original image are face images corresponding to the same subject; the training data pair is encoded to obtain a first face feature, a first makeup feature, and a first instruction feature; an initial image processing model is trained based on the first face feature, the first makeup feature, and the first instruction feature to obtain a target image processing model.

[0006] Secondly, embodiments of this application provide an image processing method, the method comprising: A pair of data to be processed is obtained, comprising an image to be repaired, a reference image corresponding to the image to be repaired, and second image processing parameters for processing the image to be repaired. The image to be repaired and the reference image are facial images corresponding to the same subject. The pair of data to be processed is encoded to obtain a second facial feature, a second beauty feature, and a second instruction feature. The second facial feature, the second beauty feature, and the second instruction feature are input into the target image processing model to obtain image latent variables. The image latent variables are decoded to obtain the target image.

[0007] Thirdly, embodiments of this application provide an image processing model training apparatus, the apparatus comprising: The acquisition module is used to acquire training data pairs of image training data. The training data pairs include first image processing parameters, a first original image, a second original image, a first reference image, and a second reference image. The first reference image is obtained by processing the first original image based on the first image processing parameters, and the second reference image is obtained by processing the second original image based on the first image processing parameters. The first original image and the second original image are face images corresponding to the same subject. The encoding module is used to encode the training data pairs to obtain the first facial feature, the first makeup feature, and the first instruction feature; The model training module is used to train the initial image processing model based on the first facial features, the first makeup features, and the first instruction features to obtain the target image processing model.

[0008] Fourthly, embodiments of this application provide an image processing model training apparatus, applied to the target image processing model described in the third aspect, the apparatus comprising: The acquisition module is used to acquire a pair of data to be processed, which includes an image to be repaired, a reference image corresponding to the image to be repaired, and second image processing parameters for processing the image to be repaired. The image to be repaired and the reference image are face images corresponding to the same subject. The encoding module is used to encode the data pair to be processed to obtain the second facial feature, the second beauty feature, and the second instruction feature. The determination module is used to input the second facial feature, the second makeup feature, and the second instruction feature into the target image processing model to obtain image latent variables; The decoding module is used to decode the latent variables of the image to obtain the target image.

[0009] Fifthly, embodiments of this application provide an electronic device, the electronic device including a processor and a memory, the memory storing programs or instructions executable on the processor, the programs or instructions being executed by the processor to implement the method as described in the first aspect, and / or the method as described in the second aspect.

[0010] Sixthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the method described in the first aspect, and / or the method described in the second aspect.

[0011] In a seventh aspect, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect, and / or the method as described in the second aspect.

[0012] Eighthly, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method described in the first aspect and / or the method described in the second aspect.

[0013] In this embodiment, by encoding a first image processing parameter, a first original image, a second original image, a first reference image obtained by processing the first original image based on the first image processing parameter, and a second reference image obtained by processing the second original image based on the first image processing parameter, the first facial features and first makeup features of the subject in the first and second original images are obtained, as well as the first instruction features for beautifying the subject in the first and second original images. Then, the initial image processing model is processed based on the first facial features, the first makeup features, and the first instruction features to obtain the target image processing model. In this way, it can be ensured that the initial image processing model retains the facial features of the subject to the greatest extent while transferring the makeup feature information of the second reference image, so that the beautification effect is realistic, natural, and highly faithful. This effectively solves the problem of the user's facial features being out of proportion and difficult to match the user's unique facial features caused by traditional one-click beautification solutions. Furthermore, this application embodiment trains an initial image processing model to obtain a target image processing model. Subsequently, the image to be edited can be automatically beautified based on the target image processing model. Users do not need to manually adjust complex beautification parameters to obtain a highly personalized beautification effect that is highly consistent with the style of the desired reference image, which greatly reduces the operation threshold. Attached Figure Description

[0014] Figure 1 This is a flowchart illustrating an image processing model training method provided in some embodiments of this application; Figure 2 This is a flowchart illustrating an image processing model training method provided in some embodiments of this application; Figure 3 This is a schematic flowchart of an image processing method provided in some embodiments of this application; Figure 4 This is a schematic diagram of an image processing system framework for implementing image processing methods according to some embodiments of this application; Figure 5 This is a schematic flowchart of an image processing method provided in some embodiments of this application; Figure 6 These are schematic diagrams illustrating the structure of an image processing model training apparatus according to some embodiments of this application; Figure 7 These are schematic diagrams illustrating the structure of an image processing apparatus according to some embodiments of this application; Figure 8 These are schematic diagrams illustrating the structure of an electronic device according to some embodiments of this application; Figure 9 These are schematic diagrams illustrating the hardware structure of an electronic device according to some embodiments of this application. Detailed Implementation

[0015] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0016] The terms "first," "second," etc., used in this application's specification are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, the subject of the photograph can be one or N objects. Furthermore, in the specification, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0017] The terminology used in the embodiments of this invention will be explained below.

[0018] Diffusion generative models are deep learning-based generative models that generate data by progressively adding and removing noise. The core idea is to simulate the physical diffusion process, gradually "diffusing" data from its original distribution to a noisy distribution, and then recovering the original data from the noise through a reverse process. Diffusion generative models consist of two key processes: the first is the forward process, or diffusion process, where Gaussian noise is progressively added to the data at multiple time steps, eventually transforming the data into pure noise. This process follows a fixed Markov chain, with the noise intensity at each time step controlled by a predefined scheduling strategy. The second process is the reverse process, or denoising process, where a neural network is trained to learn how to progressively recover the original data from the noise. The diffusion generative model predicts the current noise or data at each time step and generates high-quality samples through iterative denoising.

[0019] The technical solution of this application embodiment can be applied to scenarios involving beautifying faces in images. For example, Xiaohong's mobile phone album application contains 5 photos, namely Photo 1, Photo 2, Photo 3, Photo 4, and Photo 5. Photo 1 is a portrait photo taken by Xiaohong at a photo studio. When taking Photo 1, Xiaohong wore makeup, and the photo studio also performed some retouching on the original photo of Photo 1, thus obtaining Photo 1. Photos 2, 3, and 4 are selfies that Xiaohong took at home previously. Photo 3 is the original selfie taken by Xiaohong. After taking Photo 3, Xiaohong was very satisfied and therefore did not retouch it. Xiaohong herself retouched the original photo 21 of Photo 2 and the original photo 41 of Photo 4, thus obtaining Photos 2 and 4. Among them, Photo 21 is a photo taken by Xiaohong in her bedroom at home, holding her cat. Photo 5 is a selfie that Xiaohong just took at home. The user is not very satisfied with Photo 5 and wants to beautify the face in Photo 5.

[0020] The image processing model training method and image processing method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0021] Figure 1 This is a flowchart illustrating an image processing model training method provided in an embodiment of this application. The subject executing the image processing model training method can be an electronic device, which can be, but is not limited to, a personal computer (PC), a smartphone, a tablet computer, or a personal digital assistant (PDA).

[0022] like Figure 1 As shown, the image model training method provided in this application embodiment may include steps 110-130.

[0023] Step 110: Obtain training data pairs for image training data.

[0024] The training data pair can be a data pair used to train the initial image processing model. There can be multiple sets of data pairs for training the initial image processing model. One set of training data pairs can include a first image processing parameter, a first original image, a second original image, a first reference image, and a second reference image.

[0025] The first image processing parameter mentioned above can be a parameter used to process an image, specifically a beautification parameter used to beautify an image. For example, the first image processing parameter can be: skin smoothing: 13, whitening: 45, skin tone: 55, short face: 37.

[0026] The first original image can be the image to be beautified, such as the original photo 21 of photo 2 in the example above, or the original photo 41 of photo 4.

[0027] The second original image can be an image that has undergone some beautification of the first original image, but the beautification effect may not be very good. For example, if the first original image is photo 21 in the example above, the second original image is photo 2; if the first original image is photo 41 in the example above, the second original image is photo 4.

[0028] The first and second original images mentioned above are facial images of the same subject. For example, in the above example, both Photo 21 and Photo 2 are facial images of Xiaohong obtained by taking pictures of Xiaohong.

[0029] The first reference image can be obtained by processing the first original image based on the first image processing parameters. The second reference image is obtained by processing the second original image based on the first image processing parameters. The second reference image is further beautified to a certain extent based on the first image processing parameters on top of the original beautification effect of the second original image. Thus, the beautification effect of the second reference image is very good and reaches the ideal standard.

[0030] Step 120: Encode the training data pairs to obtain the first facial feature, the first makeup feature, and the first instruction feature.

[0031] The first facial feature can be the facial features of the subject extracted from the training data pair, such as the subject's facial features and face shape. Specific facial features can include the distance between the subject's eyebrows and eyes, the width of the mouth, and the distance between the mouth and nose. Face shape features can include round face, square face, rectangular face, oval face, etc.

[0032] The first beauty feature can be the beauty features of the subject extracted from the training data pair, such as the beauty parameters of the subject. Specifically, it can be the beauty parameters used in the second original image, such as skin smoothing 10 and whitening 5.

[0033] The first instruction feature can be an instruction for beautification parameters obtained by encoding training data pairs, specifically, it can be a feature obtained by encoding the first image processing parameters.

[0034] In some embodiments of this application, where the facial features include the facial features and face shape of the subject being photographed, step 120 may specifically include: The facial features and face shape features of the subject in the second original image are encoded using a face shape encoding model to obtain the first face features; The makeup information of the subject in the second original image is encoded using a makeup coding model to obtain the first beauty feature. The first image processing parameters are encoded using an instruction encoding model to obtain the first instruction features.

[0035] The face coding model can be a model used to encode the facial features and face shape features of the subject in the second original image. The face coding model can be a pre-trained model used to encode the facial features and face shape features of the subject in the second original image. For example, the face coding model can be a neural network model based on deep learning, a support vector machine model, or a decision tree model.

[0036] The makeup encoding model can be a module used to encode the makeup information of the subject in the second original image. The face encoding model can be a pre-trained model used to encode the makeup information of the subject in the second original image. For example, the face encoding model can be a neural network model based on deep learning, a support vector machine model, or a decision tree model.

[0037] The instruction encoding model can be a module used to encode the first image processing parameters. The instruction encoding model can be a pre-trained model used to encode the first image processing parameters, such as a neural network model based on deep learning, a support vector machine model, or a decision tree model.

[0038] It should be noted that the dimensions of the above-mentioned facial features, face shape features, makeup features, and instruction features are the same, for example, they can all be vectors of 512*768.

[0039] In some embodiments of this application, the facial features and face shape features of the subject in the second original image can be encoded using a face shape encoding model, and the makeup information of the subject in the second original image can be encoded using a makeup encoding model.

[0040] In the embodiments of this application, a multi-branch feature coding model is designed to encode the facial features and face shape features of the subject in the second original image based on a face shape coding model, encode the makeup information of the subject in the second original image based on a makeup coding model, and encode the first image processing parameters based on an instruction coding model. This achieves decoupling control of the subject's identity features and beauty attributes, and does not extract the subject's facial features and makeup features manually, thus improving the accuracy and efficiency of extracting the subject's facial features, makeup features, and beauty instruction features.

[0041] Step 130: Train the initial image processing model based on the first face feature, the first beauty feature, and the first instruction feature to obtain the target image processing model.

[0042] The initial image processing model can be an image processing model to be trained. This initial image processing model can be a neural network model, a support vector machine model, or a decision tree model, etc. In this embodiment, the initial image processing model can be a diffusion generation model.

[0043] The target image processing model can be an image processing model obtained by training an initial image processing model based on the first face feature, the first beauty feature, and the first instruction feature.

[0044] In some embodiments of this application, prior to step 130, the aforementioned scheme may further include: The first original image is encoded using an image compression model, and random noise is added to the encoded first original image to obtain the first random noise feature vector. The background information of the object in the first original image is encoded using an image compression model to obtain the first background feature; Step 130 may specifically include: The initial image processing model is trained based on the first face feature, the first beauty feature, the first instruction feature, the first random noise feature vector, and the first background feature to obtain the target image processing model.

[0045] The image compression model can be a model used to encode a first original image and add random noise to the encoded first original image. The image compression model can be, but is not limited to, a neural network model, a support vector machine model, or a decision tree model.

[0046] The first random noise feature vector can be the feature vector obtained by adding random noise to the first original image after encoding processing.

[0047] It should be noted that the first random noise feature vector mentioned above can be a pure noise feature vector with the same size as the encoding vector of the first original image.

[0048] The first background feature can be a feature vector obtained by encoding the background information of the subject in the first original image through an image compression model. For example, in the above example, the background of photo 21 is Xiaohong's bedroom and the cat that Xiaohong is holding, so the background features are the features of Xiaohong's bedroom and the features of the cat she is holding.

[0049] In some embodiments of this application, the first original image can be first feature-encoded to obtain the encoding vector of the first original image. Specifically, the first original image can be feature-encoded based on an image compression model to obtain the encoding vector of the first original image. Then, a pure random noise vector with the same size as the encoding vector of the first original image is generated, i.e., the first random noise feature vector.

[0050] Then, the background features of the subject can be extracted from the first original image using a background constraint model to obtain the first background features. This background constraint model can be a pre-trained model used to extract the background features of the subject from the first original image; for example, it could be a deep learning-based neural network model, a support vector machine model, or a decision tree model.

[0051] Then, the first background feature, the first face feature, the first makeup feature, the first instruction feature, and the first random noise feature vector are input into the initial image processing model. The initial image processing model is then trained to obtain the target image processing model.

[0052] In the embodiments of this application, by extracting the background features of the subject from the first original image, the background features of the subject that are not related to the face area are accurately locked. This ensures that the background area maintains its original pixel-level appearance during the beautification process, avoiding the background blurring, distortion or color distortion that often occurs in traditional beautification methods. This achieves the precise editing effect of "changing the face without changing the background" and improves the robustness of the initial image processing model.

[0053] In some embodiments of this application, training the initial image processing model based on the first face feature, the first makeup feature, the first instruction feature, the first random noise feature vector, and the first background feature to obtain the target image processing model may specifically include: The first face feature, the first makeup feature, the first instruction feature, the first random noise feature vector, and the first background feature are input into the initial image processing model to obtain the first predicted noise vector; The loss function value of the initial image processing model is determined based on the mean square error of the first predicted noise vector and the first random noise feature vector. The target image processing model is obtained when the loss function value satisfies the training stopping condition.

[0054] The first predicted noise vector can be the noise vector added to the first original image by inputting the first face feature, the first makeup feature, the first instruction feature, the first random noise feature vector, and the first background feature into the initial image processing model, and then performing a forward process through the initial image processing model.

[0055] The training stopping condition can be a pre-set stopping condition for the initial image processing model whose training stops when the loss function value is less than a certain threshold, such as when the loss function value is less than 0.2.

[0056] In the embodiments of this application, the loss function of the initial image processing model is obtained by predicting the noise added to the first original image using the initial image processing model and comparing the noise with the previous first random noise feature vector. Instead of comparing the image after the initial image processing model beautifies the first original image with the first reference image to determine the loss function value of the initial image processing model, the initial image processing model provided in this application provides a deterministic regression task, that is, given a noisy image, the noise to be predicted is uniquely determined. This avoids the complex "two-player game" in traditional adversarial networks, and the training process is very stable, easy to converge, and will not have mode collapse problems.

[0057] In some embodiments of this application, after step 120, the method described above may further include: Obtain the random probability value of the first feature taking effect; If the random probability of the first feature taking effect is less than the preset probability of the first feature taking effect, the preset feature will be used as the first feature.

[0058] The first feature can be any one of the first facial feature, the first beauty feature, and the first instruction feature.

[0059] The random activation probability value can be the activation probability value of a randomly given first feature. The preset activation probability value can be the activation probability value of the first feature that is set in advance.

[0060] The preset feature can be a feature that is set in advance to replace the first feature. The preset feature can be a feature in which all elements are 0, that is, the value of all elements in the preset feature is 0.

[0061] In some embodiments of this application, the aforementioned first original image, second original image, first reference image, and second reference image can be understood as follows: the first original image is a completely unedited image or an image with a very good beautification effect; the second original image is an image with a better beautification effect than the first original image. If a user wants to achieve the beautification effect corresponding to the first image processing parameters on top of the beautification effect of the second original image, the first image processing parameters can be used to process the first original image and the second original image respectively to obtain the first reference image and the second reference image. That is, during the initial image processing model training process, there may only be the first original image and the second original image, without the first image processing parameters; that is, only the beautification features required for the first original image are available, without other beautification instructions. Alternatively, there may only be the first image processing parameters, but the second original image is missing; that is, the beautification features required for the first original image are not available.

[0062] Therefore, given the aforementioned first image processing parameters, before inputting the first background feature, first face feature, first makeup feature, first instruction feature, and first random noise feature vector into the initial image processing model, the first face feature, first makeup feature, and first instruction feature can be merged to obtain a merged feature. Then, the merged feature, the first background feature, and the first random noise feature vector can be input into the initial image processing model.

[0063] It should be noted that when processing the first background feature, first face feature, first beauty feature, first instruction feature, and first random noise feature vector using the initial image processing model, it is desirable to have individual response capabilities for the first face feature, first beauty feature, and first instruction feature. This is to ensure that the model can still function correctly even when other features are missing. For example, if only the second original image or only the first instruction feature is available, the initial image processing model can still respond correctly. Therefore, when merging the first face feature, first beauty feature, and first instruction feature, if any one of these features is missing, it can be represented by a feature vector consisting entirely of zeros. The size of this zero-based feature vector is consistent with the size of the other features. For example, if the user did not input a beauty instruction, the first instruction feature will not exist. The size of both the first face feature and the first beauty feature is 512*768, so a vector with all zeros and a size of 512*768 can be constructed.

[0064] Specifically, when merging the first face feature, the first makeup feature, and the first command feature, a feature activation probability can be set for each of the first face feature, the first makeup feature, and the first command feature in advance, such as (0.5, 0.4, 0.7). Then, for each of the first face feature, the first makeup feature, and the first command feature, a number is randomly selected between (0 and 1). If the randomly selected value corresponding to the feature is less than the activation probability corresponding to the feature, then the feature is replaced with a feature of the same size with all zeros.

[0065] In the embodiments of this application, if the random probability of at least one of the first facial feature, the first makeup feature, and the first instruction feature is less than its corresponding preset probability value, the feature can be set to a feature vector of all zeros. This enables the initial image processing model to have individual response capabilities to the first facial feature, the first makeup feature, and the first instruction feature, so that the initial image processing model can still make a correct response when other features are missing, thereby improving the training accuracy of the initial image processing model.

[0066] To better understand the solutions of the embodiments of this application, the image processing model training method provided by the embodiments of this application is described below with specific scenarios.

[0067] like Figure 2 As shown, the image processing model training method provided in this application embodiment may include steps 201-207. Figure 2 Specifically, this refers to the training process of the initial image processing model.

[0068] Step 201: Obtain training data pairs for image training data.

[0069] Step 201 is the same as step 110 in the above embodiment, and will not be described again here.

[0070] Step 202: Encode the facial features and face shape information of the subject in the second original image using a face shape encoding model to obtain the first face features.

[0071] Step 203: Encode the makeup information of the subject in the second original image using a makeup coding model to obtain the first beauty feature.

[0072] Step 204: Encode the first image processing parameters using the instruction encoding model to obtain the first instruction features.

[0073] Steps 202-204 above are consistent with the process described in the above embodiment of encoding the facial features and face shape information of the subject in the second original image using a face shape encoding model to obtain the first face feature; encoding the makeup information of the subject in the second original image using a makeup encoding model to obtain the first beauty feature; and encoding the first image processing parameters using an instruction encoding model to obtain the first instruction feature. These steps will not be repeated here.

[0074] Step 205: Encode the first original image using an image compression model, and add random noise to the encoded first original image to obtain the first random noise feature vector.

[0075] Step 206: Encode the background information of the object in the first original image using an image compression model to obtain the first background feature.

[0076] Step 207: Train the initial image processing model based on the first face feature, the first makeup feature, the first instruction feature, the first random noise feature vector, and the first background feature to obtain the target image processing model.

[0077] Steps 205-207 above are consistent with the process described in the above embodiment of encoding the first original image using an image compression model, adding random noise to the encoded first original image to obtain a first random noise feature vector; encoding the background information of the subject in the first original image using an image compression model to obtain a first background feature; and training the initial image processing model based on the first face feature, the first makeup feature, the first instruction feature, the first random noise feature vector, and the first background feature to obtain the target image processing model. These steps will not be repeated here.

[0078] The following describes the process of using the target image processing model.

[0079] like Figure 3 As shown, the image processing method provided in this application embodiment may include steps 310-340.

[0080] Step 310: Obtain the data pairs to be processed.

[0081] The data pair to be processed can be a data pair to be processed, which may include the image to be repaired, a reference image corresponding to the image to be repaired, and a second image processing parameter for processing the image to be repaired.

[0082] The image to be repaired can be an image that needs to be modified, such as photo 5 in the example above.

[0083] The reference image corresponding to the image to be repaired can be an image of a beauty style referenced when retouching the image to be repaired. For example, the reference image can be any one of the images in the above example, such as photo 1, photo 2, photo 3 and photo 4.

[0084] In the example above, Xiaohong wants to edit photo 5. When editing photo 5, Xiaohong thinks that the makeup in photos 1, 2, 3 and 4 is also good. She wants to select one photo from photos 1, 2, 3 and 4, for example, photo 2. In this way, when editing photo 5 later, Xiaohong's makeup in photo 2 can be used as a reference. Therefore, photo 5 is the image to be edited, and photo 2 is the reference image.

[0085] The image to be repaired and the reference image mentioned above are facial images corresponding to the same subject. The subject can be the object to be repaired, such as a person, like Xiaohong in the example above.

[0086] The second image processing parameter can be the parameter that the user wants to process the image to be edited. For example, the second image processing parameter can be a beautification parameter. In the example above, in addition to wanting to refer to the makeup in photo 2, Xiaohong also wants to adjust the skin tone to be whiter. So when the user edits photo 5, she inputs the beautification command: whitening 45.

[0087] In some embodiments of this application, in order to significantly shorten the image retouching time, step 310 may specifically include: Obtain the image to be repaired; Select a reference image from the reference image library that corresponds to the image to be repaired; Receives second image processing parameters input by the user for processing the image to be repaired.

[0088] The reference image library can be a library of reference images selected from those corresponding to the image to be repaired. The construction of this reference image library will be described in more detail in later embodiments.

[0089] In some embodiments of this application, the image to be repaired can be obtained first, then a reference image corresponding to the image to be repaired can be selected from the reference image library, and then the second image processing parameters input by the user for processing the image to be repaired can be received.

[0090] In the embodiments of this application, a reference image corresponding to the image to be repaired is obtained from a reference image library. This reference image can serve as a visual template, helping to quickly determine the direction and style of the image to be repaired, avoiding repeated trial and error, and significantly shortening the repair time. Furthermore, it can receive second image processing parameters input by the user for processing the image to be repaired. Thus, when repairing the image to be repaired, if the user has other repair needs besides referencing the makeup in the reference image, these other needs can be addressed simultaneously, allowing for personalized image repair tailored to the user's requirements.

[0091] In some embodiments of this application, in order to improve the diversity of recommended reference images, the method described above may further include the following before selecting a reference image corresponding to the image to be repaired from the reference image library based on the image to be repaired: Extract all images containing the photographed subject from the photo album application to obtain N candidate images; Cluster the N candidate images according to at least one clustering dimension to obtain at least one image set corresponding to each different clustering dimension; Based on image quality, each of the N candidate images is scored, resulting in a score for each of the N candidate images. The scores corresponding to the N candidate images are sorted in descending order to obtain the mass order map set; At least one image set and mass sequence set corresponding to different clustering dimensions are used as reference image libraries.

[0092] Here, the N candidate images can be any images containing the subject extracted from the photo album application, where N can be a positive integer. In other words, all images containing the subject can be extracted from the photo album application and used as candidate images for reference retouching of the image to be edited. For example, photos 1, 2, 3, and 4 in the above example are all candidate images.

[0093] Clustering dimension can be a pre-defined dimension for clustering all images containing the photographed subject in the photo album application.

[0094] In some embodiments of this application, at least one image set is formed by different clustering dimensions. For at least one image set corresponding to a certain clustering dimension, the images in the same image set have the same clustering attribute. The aforementioned clustering attribute may include at least one of the following: shooting time, shooting location, shooting background, clothing of the subject, and makeup of the subject. That is, for at least one image set corresponding to a certain clustering dimension, the images in an image set have the same clustering attribute.

[0095] For example, in at least one image set formed by clustering N candidate images according to the shooting time dimension, the images in each image set were shot at the same time. Similarly, in at least one image set formed by clustering N candidate images according to the shooting location dimension, the images in each image set were shot at the same location.

[0096] In some embodiments of this application, N candidate images can be clustered according to at least one clustering dimension to obtain at least one image set corresponding to different clustering dimensions.

[0097] A quality-ordered image set can be obtained by scoring N candidate images according to their image quality, obtaining the scores corresponding to the N candidate images, and then sorting the scores of the N candidate images in descending order.

[0098] In some embodiments of this application, for N candidate images, the image quality can be evaluated based on factors such as aesthetics and clarity of each candidate image. The image quality of the N candidate images is scored to obtain scores for each candidate image. Then, the scores are sorted in descending order to obtain the mass order map set. At least one image set and the mass order map set corresponding to different clustering dimensions are then used as a reference image library.

[0099] In the embodiments of this application, by extracting all images containing the photographed object from the photo album application and clustering them according to at least one clustering dimension, at least one image set with different clustering dimensions can be obtained. Furthermore, the image quality of N candidate images can be scored according to image quality to obtain a quality order set. Thus, when a user wants to select a reference image for retouching an image, an image with good image quality that corresponds to the type of the image to be retouched can be selected from at least one image set with different clustering dimensions, thereby improving the diversity of recommended reference images.

[0100] In some embodiments of this application, to improve the accuracy of the reference image push, the step of clustering N candidate images according to at least one clustering dimension to obtain at least one image set corresponding to different clustering dimensions may specifically include: Based on the shooting time of N candidate images, each candidate image is clustered according to the shooting time to obtain at least one time image set corresponding to different shooting times; Based on the shooting locations of N candidate images, each image is clustered according to its shooting location to obtain at least one set of location images corresponding to different shooting locations; Based on the shooting background in N candidate images, each image is clustered according to the shooting background to obtain at least one set of background images corresponding to different shooting backgrounds; Based on the clothing of the subjects in the N candidate images, each image is clustered according to the clothing of the subjects to obtain at least one clothing image set corresponding to different clothing. Based on the makeup of the subject in N candidate images, each image is clustered according to the makeup of the subject to obtain at least one set of makeup images corresponding to different makeup styles.

[0101] At least one time image set can be obtained by clustering N candidate images according to different shooting times, and the images in the same time image set are shot at the same time.

[0102] At least one location image set can be obtained by clustering N candidate images according to different shooting locations, and the images in the same location image set are taken at the same shooting location.

[0103] At least one background image set can be obtained by clustering N candidate images according to different shooting backgrounds, and the images in the same background image set have the same shooting background.

[0104] At least one clothing image set can be obtained by clustering N candidate images according to the different clothing of the subject. In the images of the same clothing image set, the clothing of the subject is the same.

[0105] At least one makeup image set can be obtained by clustering N candidate images according to different makeup looks of the subject. In the images of the same makeup image set, the makeup of the subject is the same.

[0106] In some embodiments of this application, the N candidate images can be clustered according to their shooting time, thus obtaining at least one time image set corresponding to different shooting times.

[0107] In the example above, photo 1 was taken on December 1, 2025, and photos 2, 3, and 4 were all taken on December 15, 2025. Therefore, photos 1, 2, 3, and 4 can be clustered according to their taking time to obtain two time image sets, namely time image set 1 and time image set 2. Time image set 1 contains photo 1, and time image set 2 contains photos 2, 3, and 4.

[0108] In some embodiments of this application, the N candidate images can be clustered according to their shooting locations, thus obtaining at least one set of location images corresponding to different shooting locations.

[0109] In the example above, photo 1 was taken at a photo studio, while photos 2, 3, and 4 were all taken at Xiaohong's home. Therefore, photos 1, 2, 3, and 4 can be clustered according to their shooting locations to obtain two location image sets, namely location image set 1 and location image set 2. Location image set 1 contains photo 1, and location image set 2 contains photos 2, 3, and 4.

[0110] In some embodiments of this application, the N candidate images can be clustered according to their shooting backgrounds to obtain at least one set of background images corresponding to different shooting backgrounds.

[0111] In the example above, the background of photo 1 is the flowers and plants arranged by the photo studio, the background of photo 2 is Xiaohong's own bedroom, and the background of photos 3 and 4 is the playground in Xiaohong's own neighborhood. Therefore, photos 1, 2, 3 and 4 can be clustered according to their background to obtain three background image sets, namely background image set 1, background image set 2 and background image set 3. Background image set 1 contains photo 1, background image set 2 contains photo 2, and background image set 3 contains photos 3 and 4.

[0112] In some embodiments of this application, the N candidate images can be clustered according to the clothing of the photographed object in the N candidate images, so as to obtain at least one clothing image set corresponding to different clothing.

[0113] In the example above, Xiaohong is wearing a pink dress in photo 1, a white T-shirt and blue jeans in photo 2, and a brown shirt and black skinny jeans in photos 3 and 4. Photos 1, 2, 3, and 4 can be clustered according to Xiaohong's clothing to obtain three clothing image sets: Clothing Image Set 1, Clothing Image Set 2, and Clothing Image Set 3. Clothing Image Set 1 contains photo 1, Clothing Image Set 2 contains photo 2, and Clothing Image Set 3 contains photos 3 and 4.

[0114] In some embodiments of this application, the N candidate images can be clustered according to the makeup of the subject in the N candidate images, so as to obtain at least one set of makeup images corresponding to different makeup.

[0115] As in the example above, in photo 1, Xiaohong has a delicate makeup look, specifically wearing false eyelashes, eyeshadow, mauve lipstick, blush, and eyebrows, and her hair is styled in an updo. In photos 2, 3, and 4, Xiaohong has her hair in a bun and only wears mauve lipstick. Therefore, photos 1, 2, 3, and 4 can be clustered according to Xiaohong's makeup to obtain two makeup image sets, namely Makeup Image Set 1 and Makeup Image Set 2. Makeup Image Set 1 contains photo 1, and Makeup Image Set 2 contains photos 2, 3, and 4.

[0116] In the embodiments of this application, each image is clustered according to the clustering dimensions of shooting time, shooting location, shooting background, clothing of the subject, and makeup of the subject, to obtain image sets corresponding to different clustering dimensions. In this way, when selecting reference images based on shooting time, shooting location, shooting background, clothing of the subject, and makeup of the subject, the dimensions of the recommended reference images can be more matched with those of the images to be repaired, thereby improving the accuracy of the reference image push.

[0117] In some embodiments of this application, to improve the accuracy of the recommended reference images, the step of selecting a reference image corresponding to the image to be repaired from a reference image library may specifically include: Based on at least one clustering dimension of the image to be repaired, select M reference images that match the image to be repaired from at least one image set corresponding to different clustering dimensions. Based on the order of the M reference images in the mass sequence set, the reference image with the highest score is selected from the M reference images as the reference image corresponding to the image to be repaired.

[0118] The M reference images can be selected from at least one image set corresponding to different clustering dimensions of the image to be repaired, based on at least one clustering dimension of the image to be repaired. Here, M can be a positive integer and M < N.

[0119] In some embodiments of this application, M reference images matching the image to be repaired can be selected from at least one image set corresponding to different clustering dimensions, based on at least one clustering dimension of the image to be repaired.

[0120] In the example above, Xiaohong took photo 5 on December 26, 2025. When photo 5 was taken, Xiaohong had her hair in a bun, wore lipstick, a khaki short-sleeved shirt, and black skinny jeans. The background was Xiaohong's living room. Based on the time of photo 5, an image matching photo 5 could be selected from time image set 1 and time image set 2. However, since there is no image matching the time of photo 5 in time image set 1 and time image set 2, no image was selected from time image set 1 and time image set 2.

[0121] Then, based on the shooting location of photo 5, images matching photo 5 are selected from location image set 1 and location image set 2. Since the shooting locations of photos 2, 3, and 4 match the shooting location of photo 5, photos 2, 3, and 4 are selected from location image set 2.

[0122] Then, based on the shooting background of photo 5, select images that match photo 5 from background image set 1, background image set 2 and background image set 3. Since the shooting background of photo 2 matches the shooting background of photo 5, both being the decorations in Xiaohong's home, photo 2 is selected from background image set 2.

[0123] Then, based on Xiaohong's clothing in photo 5, images matching photo 5 are selected from clothing image set 1, clothing image set 2, and clothing image set 3. Since Xiaohong's clothing in photos 3 and 4 matches Xiaohong's clothing in photo 5, photos 3 and 4 are selected from clothing image set 3.

[0124] Then, based on Xiaohong's makeup in photo 5, images matching photo 5 are selected from makeup image set 1 and makeup image set 2. Since Xiaohong's makeup in photos 2, 3 and 4 matches Xiaohong's makeup in photo 5, photos 2, 3 and 4 are selected from makeup image set 2.

[0125] In summary, three photos matching photo 5 were selected from at least one image set corresponding to different clustering dimensions, namely photo 2, photo 3, and photo 4.

[0126] Then, according to the order of the M reference images in the quality sequence set, the reference image with the highest score is selected from the M reference images as the reference image corresponding to the image to be repaired. In the example above, if the image quality scores of photos 2, 3, and 4 are from largest to smallest as photos 3, 4, and 2, then the photo with the highest image quality score among photos 2, 3, and 4 is photo 3, and therefore photo 3 can be used as the reference image for repairing photo 5.

[0127] In the embodiments of this application, when a user selects a reference image for retouching an image, M reference images corresponding to the type of the image to be retouched are first selected from at least one image set with different clustering dimensions. Then, the image with the best image quality is selected from the M reference images as the reference image, thereby improving the accuracy of the reference image recommendation.

[0128] Step 320: Encode the data pair to be processed to obtain the second facial feature, the second beauty feature, and the second instruction feature.

[0129] The second facial feature can be a feature of the subject's face extracted from the data pair to be processed. This could include the subject's facial features and face shape. Specific facial features could include the distance between the subject's eyebrows and eyes, the width of the mouth, and the distance between the mouth and nose. Face shape features could include round, square, rectangular, or oval faces. The extraction method for the second facial feature is the same as the extraction method for the first facial feature in step 120 of the above embodiment, and will not be repeated here.

[0130] The second beauty feature can be the beauty features of the subject extracted from the data pair to be processed. For example, it can be the beauty parameters of the subject, specifically the beauty parameters used in the reference image, such as skin smoothing 10 and whitening 5.

[0131] The second instruction feature can be an instruction to encode the data pair to be processed to obtain beautification parameters. Specifically, it can be a feature obtained by encoding the second image processing parameters.

[0132] In some embodiments of this application, reference is made to Figure 4 The facial features and face shape features of the subject in the reference image can be encoded by the face shape encoding model 401 to obtain the second face features. The makeup information of the subject in the reference image can be encoded by the makeup encoding model 402 to obtain the second beauty features. The second image processing parameters can be encoded by the instruction encoding model 403 to obtain the second instruction features.

[0133] Step 330: Input the second facial features, the second makeup features, and the second instruction features into the target image processing model to obtain the image latent variables.

[0134] The target image processing model can be the target image processing model obtained after training the initial image processing model in the above embodiments.

[0135] Image latent variables can be the latent variables of the retouched image obtained based on the target image processing model. The specific method for determining image latent variables will be described in detail in subsequent embodiments.

[0136] In some embodiments of this application, prior to step 330, the method described above may further include: The image to be repaired is encoded using an image compression model, and random noise is added to the encoded image to obtain a second random noise feature vector. The background information of the photographed object in the image to be repaired is encoded by an image compression model to obtain the second background feature; Step 330 may specifically include: The second facial feature, second makeup feature, second instruction feature, second random noise feature vector, and second background feature are input into the target image processing model to obtain the image latent variables.

[0137] The second random noise feature vector can be a feature vector obtained by adding random noise to the encoded image to be repaired. The method of obtaining the second random noise feature vector is the same as the method of obtaining the first random noise feature vector in the above embodiment, and will not be repeated here.

[0138] The second background feature can be a feature vector obtained by encoding the background information of the subject in the image to be edited using an image compression model. For example, in the above example, the background of photo 5 is Xiaohong's living room, so the background feature is the feature of Xiaohong's living room.

[0139] In some embodiments of this application, feature encoding can be performed on the image to be repaired to obtain the encoded vector of the image to be repaired. For details, please refer to... Figure 4 Based on the image compression model 404, feature encoding is performed on the image to be repaired to obtain the encoded vector of the image to be repaired. Then, a pure random noise vector with the same size as the encoded vector of the image to be repaired is generated, which is the second random noise feature vector.

[0140] Then continue to refer to Figure 4 The background features of the subject can be extracted from the image to be repaired using the background constraint model 406 to obtain the second background features. The background constraint model 406 can be a pre-trained model for extracting the background features of the subject from the image to be repaired. For example, the background constraint model can be a neural network model based on deep learning, a support vector machine model, or a decision tree model.

[0141] Then, the second background feature, the second face feature, the second makeup feature, the second instruction feature, and the second random noise feature vector are input into the target image processing model 405 to obtain the image latent variables.

[0142] In the embodiments of this application, by extracting the background features of the subject from the image to be edited, the background features of the subject that are not related to the face area are accurately locked. This ensures that the background area maintains its original pixel-level appearance during the beautification process, avoiding the background blurring, distortion or color distortion that often occurs in traditional beautification methods. This achieves the precise editing effect of "changing the face without changing the background" and improves the robustness of the target image processing model.

[0143] Step 340: Decode the latent variables of the image to obtain the target image.

[0144] The target image can be the image after the image to be edited has been edited.

[0145] In some embodiments of this application, the latent variables of the image can be decoded to obtain the target image; specifically, this can be referred to... Figure 4 The target image can be obtained by decoding the latent variables of the image based on the image decompression model 407.

[0146] In some embodiments of this application, step 340 may specifically include: The second background feature, second face feature, second makeup feature, second instruction feature, and second random noise feature vector are input into the target image processing model to obtain the second predicted noise vector. The second predicted noise vector is removed from the second random noise feature vector by the target image processing model to obtain the image latent variable.

[0147] The second predicted noise vector can be the noise vector added to the image to be repaired that is predicted by the target image processing model after the second background feature, second face feature, second makeup feature, second instruction feature, and second random noise feature vector are input into the target image processing model and the model performs a forward process.

[0148] In some embodiments of this application, the second background feature, the second face feature, the second makeup feature, the second instruction feature, and the second random noise feature vector can be input into the target image processing model. The target image processing model performs a forward process to obtain the second predicted noise vector. Then, the target image processing model performs a denoising process, that is, removes the second predicted noise vector from the second random noise feature vector, thereby obtaining the image latent variable.

[0149] It should be noted that the target image processing model can first predict a noise vector based on the second background feature, the second face feature, the second makeup feature, the second instruction feature, and the second random noise feature vector. Then, the predicted noise vector is removed from the second random noise feature vector. In this way, some noise can be removed from the image to be repaired. Then, the above forward process and denoising process are repeated until the removal of the second random noise feature vector is completed.

[0150] In the embodiments of this application, the reverse process of "adding noise first and then removing noise" of the target image processing model enables the target image processing model to gradually recover high-quality content from pure noise, thereby improving the accuracy of obtaining image latent variables.

[0151] It should be noted that, as in the above embodiments, when the above-mentioned second image processing parameters exist, before inputting the second background feature, the second face feature, the second makeup feature, the second instruction feature, and the second random noise feature vector into the target image processing model, the second face feature, the second makeup feature, and the second instruction feature can be merged to obtain a merged feature, and then the merged feature, the second background feature, and the second random noise feature vector can be input into the target image processing model.

[0152] When processing the second background feature, second face feature, second beauty feature, second instruction feature, and second random noise feature vector using the target image processing model, it is desirable to have independent response capabilities for the second face feature, second beauty feature, and second instruction feature. This is to ensure that the model can still function correctly even when other features are missing. For example, if only the image to be repaired or only the second instruction feature is available, the target image processing model should still be able to respond correctly. Therefore, when merging the second face feature, second beauty feature, and second instruction feature, if any one of these features is missing, it can be represented by a feature vector consisting entirely of zeros. The size of this zero-based feature vector should be consistent with the size of the other features. For example, if the user has not input a beauty instruction, the second instruction feature will not exist. The size of both the second face feature and the second beauty feature is 512*768, so a vector with all zeros and a size of 512*768 can be constructed.

[0153] Specifically, when merging the second face feature, the second makeup feature, and the second command feature, a feature activation probability can be set in advance for each of the second face feature, the second makeup feature, and the second command feature, such as (0.5, 0.4, 0.7). Then, for each of the second face feature, the second makeup feature, and the second command feature, a number is randomly selected between (0 and 1). If the randomly selected value corresponding to the feature is less than the activation probability corresponding to the feature, then the feature is replaced with a feature of the same size with all zeros.

[0154] To better understand the solutions of the embodiments of this application, the image processing method provided by the embodiments of this application will be described below with specific scenarios.

[0155] like Figure 5 As shown, the image processing method provided in this application embodiment may include steps 501-513. Figure 5 Specifically, it refers to the process of using the target image processing model.

[0156] Step 501: Obtain the image to be repaired.

[0157] Step 501 is the same as the process of obtaining the image to be repaired in the above embodiment, and will not be described again here.

[0158] Step 502: Extract all images containing the photographed subject from the photo album application to obtain N candidate images.

[0159] Step 502 is consistent with the process in the above embodiment of extracting all images containing the photographed object from the photo album application to obtain N candidate images, and will not be described again here. N is a positive integer.

[0160] Step 503: Cluster the N candidate images according to at least one clustering dimension to obtain at least one image set corresponding to each different clustering dimension.

[0161] In step 503, at least one image set corresponding to a clustering dimension has the same clustering attribute. The clustering attribute includes at least one of the following: shooting time, shooting location, shooting background, clothing of the subject, and makeup of the subject.

[0162] Step 504: Score the N candidate images according to their image quality to obtain the scores corresponding to the N candidate images.

[0163] Step 505: Sort the scores corresponding to the N candidate images in descending order to obtain the mass order image set.

[0164] Step 506: Use at least one image set and mass sequence set corresponding to different clustering dimensions as a reference image library.

[0165] Steps 502-506 above are consistent with the process described in the above embodiment of extracting all images containing the photographed object from the photo album application to obtain N candidate images; clustering the N candidate images according to at least one clustering dimension to obtain at least one image set corresponding to each different clustering dimension; scoring the N candidate images according to image quality to obtain scores corresponding to each of the N candidate images; sorting the scores corresponding to each of the N candidate images in descending order to obtain a mass order image set; and using the at least one image set corresponding to each different clustering dimension and the mass order image set as a reference image library, which will not be described again here.

[0166] Step 507: Based on at least one clustering dimension of the image to be repaired, select M reference images that match the image to be repaired from at least one image set corresponding to different clustering dimensions.

[0167] In step 507 above, M is a positive integer, and M < N.

[0168] Step 508: Based on the order of the M reference images in the mass sequence image set, select the reference image with the highest score from the M reference images as the reference image corresponding to the image to be repaired.

[0169] Steps 507-508 above are consistent with the process in the above embodiment of selecting M reference images that match the image to be repaired from at least one image set corresponding to different clustering dimensions based on at least one clustering dimension of the image to be repaired; and selecting the reference image with the highest score from the M reference images as the reference image corresponding to the image to be repaired based on the order of the M reference images in the mass sequence set. Therefore, they will not be described again here.

[0170] Step 509: Extract the second facial features and second makeup features of the subject from the reference image.

[0171] Step 510: Extract the second background features of the photographed object from the image to be repaired.

[0172] Steps 509-510 above are consistent with the process of obtaining the second facial features, the second makeup features and the second background features in the above embodiments, and will not be repeated here.

[0173] Step 511: Upon receiving the second image processing parameters input by the user, the second image processing parameters are encoded to obtain the second instruction feature.

[0174] In step 511, the method of obtaining the second instruction feature is the same as that of obtaining the second instruction feature in the above embodiment, and will not be repeated here.

[0175] Step 512: Input the second background feature, second face feature, second makeup feature, second instruction feature, and second random noise feature vector into the target image processing model to obtain the image latent variables.

[0176] In step 512 above, the second random noise feature vector is determined based on the encoding vector of the image to be repaired. The process in step 512 is consistent with the process in the above embodiment of inputting the second background feature, the second face feature, the second makeup feature, the second instruction feature, and the second random noise feature vector into the target image processing model to obtain the image latent variables, and will not be repeated here.

[0177] Step 513: Decode the latent variables of the image to obtain the target image.

[0178] Step 513 above is the same as step 340 in the above embodiment, and will not be repeated here.

[0179] In the embodiments of this application, by extracting facial feature information and makeup feature information of the subject from the reference image corresponding to the image to be repaired, fine-grained beauty attributes such as makeup can be decoupled and transferred from the reference image while preserving the user's original facial identity features. This results in a realistic, natural, and highly faithful beauty effect, effectively solving the problem of disproportionate facial features and difficulty in matching the user's unique facial features caused by traditional one-click beauty solutions. Furthermore, by extracting the background feature information of the subject from the image to be repaired, the original content of the unedited background area can be ensured, avoiding background blurring, distortion, or color distortion often caused by traditional beauty methods. Thus, beauty reference images can be intelligently recommended based on the user's album images, achieving personalized and refined beauty effects.

[0180] The image processing model training method provided in this application can be executed by an image processing model training device. This application uses an image processing model training device executing the image processing method as an example to illustrate the image processing model training device provided in this application.

[0181] Figure 6 This is a schematic diagram illustrating the structure of an image processing model training device according to an exemplary embodiment. Figure 6 As shown, the image processing model training device 600 may include: The acquisition module 610 is used to acquire training data pairs of image training data. The training data pairs include first image processing parameters, a first original image, a second original image, a first reference image, and a second reference image. The first reference image is obtained by processing the first original image based on the first image processing parameters, and the second reference image is obtained by processing the second original image based on the first image processing parameters. The first original image and the second original image are face images corresponding to the same subject. The encoding module 620 is used to encode the training data pair to obtain the first face feature, the first makeup feature, and the first instruction feature; The model training module 630 is used to train the initial image processing model based on the first facial features, the first makeup features and the first instruction features to obtain the target image processing model.

[0182] In this embodiment, by encoding a first image processing parameter, a first original image, a second original image, a first reference image obtained by processing the first original image based on the first image processing parameter, and a second reference image obtained by processing the second original image based on the first image processing parameter, the first facial features and first makeup features of the subject in the first and second original images are obtained, as well as the first instruction features for beautifying the subject in the first and second original images. Then, the initial image processing model is processed based on the first facial features, the first makeup features, and the first instruction features to obtain the target image processing model. In this way, it can be ensured that the initial image processing model retains the facial features of the subject to the greatest extent while transferring the makeup feature information of the second reference image, so that the beautification effect is realistic, natural, and highly faithful. This effectively solves the problem of the user's facial features being out of proportion and difficult to match the user's unique facial features caused by traditional one-click beautification solutions. Furthermore, this application embodiment trains an initial image processing model to obtain a target image processing model. Subsequently, the image to be edited can be automatically beautified based on the target image processing model. Users do not need to manually adjust complex beautification parameters to obtain a highly personalized beautification effect that is highly consistent with the style of the desired reference image, which greatly reduces the operation threshold.

[0183] In some embodiments of this application, the encoding module is specifically used to: encode the facial features and face shape information of the subject in the second original image using a face shape encoding model to obtain a first face feature; encode the makeup information of the subject in the second original image using a makeup encoding model to obtain a first beauty feature; and encode the first image processing parameters using an instruction encoding model to obtain a first instruction feature.

[0184] In some embodiments of this application, the apparatus further includes: a noise addition module, used to encode the first original image through an image compression model and add random noise to the encoded first original image before training the initial image processing model based on the first face feature, the first makeup feature and the first instruction feature to obtain the target image processing model, thereby obtaining a first random noise feature vector. The encoding module is further configured to encode the background information of the photographed object in the first original image using the image compression model to obtain the first background feature; the model training module is further configured to train the initial image processing model based on the first face feature, the first makeup feature, the first instruction feature, the first random noise feature vector and the first background feature to obtain the target image processing model.

[0185] In some embodiments of this application, the model training module is specifically used to: input the first face feature, the first makeup feature, the first instruction feature, the first random noise feature vector, and the first background feature into an initial image processing model to obtain a first predicted noise vector; determine the loss function value of the initial image processing model based on the mean square error of the first predicted noise vector and the first random noise feature vector; and obtain the target image processing model when the loss function value satisfies the training stopping condition.

[0186] In some embodiments of this application, the acquisition module is further configured to acquire a random probability value of the first feature after encoding the training data pair to obtain the first face feature, the first makeup feature, and the first instruction feature, wherein the first feature is any one of the first face feature, the first makeup feature, and the first instruction feature; The device further includes: a setting module, used to set a preset feature as the first feature when the random activation probability value of the first feature is less than the preset activation probability value corresponding to the first feature, wherein the preset feature is a feature in which all elements are 0.

[0187] The image processing method provided in this application can be executed by an image processing device. This application uses an image processing device executing the image processing method as an example to illustrate the image processing device provided in this application.

[0188] Figure 7 This is a schematic diagram illustrating the structure of an image processing apparatus according to an exemplary embodiment. The image processing apparatus is applied to the target image processing model provided in the above embodiment, such as... Figure 7 As shown, the image processing apparatus 700 may include: The acquisition module 710 is used to acquire a pair of data to be processed, the pair of data to be processed including an image to be repaired, a reference image corresponding to the image to be repaired, and second image processing parameters for processing the image to be repaired, wherein the image to be repaired and the reference image are face images corresponding to the same subject being photographed. The encoding module 720 is used to encode the data pair to be processed to obtain the second facial feature, the second beauty feature, and the second instruction feature. The determining module 730 is used to input the second facial feature, the second makeup feature, and the second instruction feature into the target image processing model to obtain image latent variables; The decoding module 740 is used to decode the latent variables of the image to obtain the target image.

[0189] In this embodiment, during the process of beautifying the image to be repaired to obtain the target image, facial feature information and makeup feature information of the subject are extracted from the reference image corresponding to the image to be repaired. This ensures that the generated target image retains the facial feature information of the subject to the greatest extent possible while migrating the makeup feature information of the reference image. This results in a realistic, natural, and highly faithful beautification effect, effectively solving the problem of disproportionate facial features and difficulty in matching the unique facial features of users caused by traditional one-click beautification solutions. Furthermore, the solution in this embodiment inputs the second facial feature, the second makeup feature, and the second instruction feature into the target image processing model, and obtains the target image based on the target image processing model. This eliminates the need for users to manually adjust complex beautification parameters, allowing for a highly personalized beautification effect that closely matches the desired reference image style, significantly reducing the operational threshold.

[0190] In some embodiments of this application, the acquisition module 710 is specifically used for: acquiring the image to be repaired; selecting a reference image corresponding to the image to be repaired from a reference image library; and receiving second image processing parameters input by the user for processing the image to be repaired.

[0191] In some embodiments of this application, the encoding module is further configured to encode the image to be repaired using an image compression model before inputting the second facial feature, the second makeup feature, and the second instruction feature into the target image processing model to obtain image latent variables, and to add random noise to the encoded image to obtain a second random noise feature vector; and to encode the background information of the photographed object in the image to be repaired using the image compression model to obtain a second background feature; The determining module 730 is specifically used for: The second facial feature, the second makeup feature, the second instruction feature, the second random noise feature vector, and the second background feature are input into the target image processing model to obtain image latent variables.

[0192] In some embodiments of this application, the determining module 730 is specifically used to: input the second face feature, the second makeup feature, the second instruction feature, the second random noise feature vector, and the second background feature into the target image processing model to obtain a second predicted noise vector; and remove the second predicted noise vector from the second random noise feature vector through the target image processing model to obtain the image latent variable.

[0193] The image processing model training device and / or image processing device in the embodiments of this application can be an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, handheld computer, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM or self-service machine, etc. The embodiments of this application do not specifically limit the scope.

[0194] The image processing model training device and / or image processing device in the embodiments of this application can be a device with an operating system. The operating system can be Android, iOS, or other possible operating systems, and this application embodiment does not specifically limit it.

[0195] The image processing model training device and / or image processing device provided in the embodiments of this application can achieve... Figure 1 and / or Figure 3 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0196] Optionally, such as Figure 8 As shown, this application embodiment also provides an electronic device 800, including a processor 801 and a memory 802. The memory 802 stores a program or instructions that can run on the processor 801. When the program or instructions are executed by the processor 801, they implement the various steps of the above-described image processing model training method and / or image processing method embodiments, and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0197] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0198] Figure 9 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0199] The electronic device 900 includes, but is not limited to, components such as: radio frequency unit 901, network module 902, audio output unit 903, input unit 904, sensor 905, display unit 906, user input unit 907, interface unit 908, memory 909, and processor 910.

[0200] Those skilled in the art will understand that the electronic device 900 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 910 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 9 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0201] When the electronic device is used as a training device for an image processing model: The processor 910 is configured to acquire training data pairs for image training data, the training data pairs including first image processing parameters, a first original image, a second original image, a first reference image, and a second reference image; wherein the first reference image is obtained by processing the first original image based on the first image processing parameters, and the second reference image is obtained by processing the second original image based on the first image processing parameters, and the first original image and the second original image are face images corresponding to the same subject; the processor 910 encodes the training data pairs to obtain a first face feature, a first makeup feature, and a first instruction feature; and trains an initial image processing model based on the first face feature, the first makeup feature, and the first instruction feature to obtain a target image processing model.

[0202] Thus, by encoding the first image processing parameters, the first original image, the second original image, the first reference image obtained by processing the first original image based on the first image processing parameters, and the second reference image obtained by processing the second original image based on the first image processing parameters, the first facial features and the first makeup features of the subject in the first and second original images, as well as the first instruction features for beautifying the subject in the first and second original images, are obtained. Then, the initial image processing model is processed based on the first facial features, the first makeup features, and the first instruction features to obtain the target image processing model. In this way, it can be ensured that the initial image processing model retains the facial features of the subject to the greatest extent while transferring the makeup feature information of the second reference image, so that the beautification effect is realistic, natural, and highly faithful. This effectively solves the problem of the user's facial proportion being out of proportion and difficult to match the user's unique facial features caused by traditional one-click beautification solutions. Furthermore, this application embodiment trains an initial image processing model to obtain a target image processing model. Subsequently, the image to be edited can be automatically beautified based on the target image processing model. Users do not need to manually adjust complex beautification parameters to obtain a highly personalized beautification effect that is highly consistent with the style of the desired reference image, which greatly reduces the operation threshold.

[0203] Optionally, the processor 910 is further configured to encode the facial features and face shape information of the subject in the second original image using a face shape encoding model to obtain a first face feature; encode the makeup information of the subject in the second original image using a makeup encoding model to obtain a first beauty feature; and encode the first image processing parameters using an instruction encoding model to obtain a first instruction feature.

[0204] Optionally, the processor 910 is further configured to encode the first original image using an image compression model, and add random noise to the encoded first original image to obtain a first random noise feature vector; encode the background information of the photographed object in the first original image using the image compression model to obtain a first background feature; and train an initial image processing model based on the first face feature, the first makeup feature, the first instruction feature, the first random noise feature vector, and the first background feature to obtain a target image processing model.

[0205] Optionally, the processor 910 is further configured to input the first face feature, the first makeup feature, the first instruction feature, the first random noise feature vector, and the first background feature into an initial image processing model to obtain a first predicted noise vector; determine the loss function value of the initial image processing model based on the mean square error of the first predicted noise vector and the first random noise feature vector; and obtain the target image processing model when the loss function value satisfies the training stopping condition.

[0206] Optionally, the processor 910 is further configured to obtain a random probability value of the first feature, wherein the first feature is any one of the first facial feature, the first beauty feature, and the first instruction feature; if the random probability value of the first feature is less than the preset probability value of the first feature, the preset feature is used as the first feature, wherein the preset feature is a feature in which all elements are 0.

[0207] When the electronic device is used as a training device for an image processing model: The processor 910 is configured to acquire a pair of data to be processed, the pair of data to be processed including an image to be repaired, a reference image corresponding to the image to be repaired, and second image processing parameters for processing the image to be repaired, wherein the image to be repaired and the reference image are facial images corresponding to the same subject; to encode the pair of data to be processed to obtain a second facial feature, a second beauty feature, and a second instruction feature; to input the second facial feature, the second beauty feature, and the second instruction feature into the target image processing model to obtain image latent variables; and to decode the image latent variables to obtain the target image.

[0208] Thus, in the process of beautifying the image to be repaired and obtaining the target image, by extracting the facial feature information and makeup feature information of the subject from the reference image corresponding to the image to be repaired, it is possible to ensure that the generated target image retains the facial feature information of the subject to the greatest extent while transferring the makeup feature information of the reference image. This results in a realistic, natural, and highly faithful beautification effect, effectively solving the problem of disproportionate facial features and difficulty in matching the unique facial features of users caused by traditional one-click beautification solutions. Furthermore, the solution of this application's embodiment inputs the second facial feature, the second makeup feature, and the second instruction feature into the target image processing model, and obtains the target image based on the target image processing model. In this way, users can obtain a highly personalized beautification effect that is highly consistent with the style of the desired reference image without manually adjusting complex beautification parameters, greatly reducing the operational threshold.

[0209] Optionally, the processor 910 is further configured to acquire the image to be repaired; select a reference image corresponding to the image to be repaired from a reference image library; and receive second image processing parameters input by the user for processing the image to be repaired.

[0210] Optionally, the processor 910 is further configured to encode the image to be repaired using an image compression model, and add random noise to the encoded image to obtain a second random noise feature vector; encode the background information of the photographed object in the image to be repaired using the image compression model to obtain a second background feature; and input the second face feature, the second makeup feature, the second instruction feature, the second random noise feature vector, and the second background feature into the target image processing model to obtain the image latent variable.

[0211] Optionally, the processor 910 is further configured to input the second face feature, the second makeup feature, the second instruction feature, the second random noise feature vector, and the second background feature into the target image processing model to obtain the second predicted noise vector; and to remove the second predicted noise vector from the second random noise feature vector through the target image processing model to obtain the image latent variable.

[0212] It should be understood that, in this embodiment, the input unit 904 may include a graphics processing unit (GPU) 9041 and a microphone 9042. The GPU 9041 processes image data of still images or videos obtained by an image capture device (such as a color camera) in video capture mode or image capture mode. The display unit 906 may include a display panel 9061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 907 includes at least one of a touch panel 9071 and other input devices 9072. The touch panel 9071 is also called a touch screen. The touch panel 9071 may include a touch detection device and a touch controller. Other input devices 9072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0213] The memory 909 can be used to store software programs and various data. The memory 909 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 909 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 909 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0214] Processor 910 may include one or more processing units; optionally, processor 910 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 910.

[0215] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described image processing model training method and / or image processing method embodiments, and achieve the same technical effect. To avoid repetition, further details are omitted here.

[0216] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0217] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described image processing model training method and / or image processing method embodiments, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0218] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0219] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the above-described image processing model training method and / or image processing method embodiments, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0220] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0221] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0222] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for training an image processing model, characterized in that, The method includes: The training data pair for obtaining image training data includes a first image processing parameter, a first original image, a second original image, a first reference image, and a second reference image; wherein, the first reference image is obtained by processing the first original image based on the first image processing parameter, the second reference image is obtained by processing the second original image based on the first image processing parameter, and the first original image and the second original image are face images corresponding to the same subject. The training data pairs are encoded to obtain the first facial feature, the first makeup feature, and the first instruction feature; The initial image processing model is trained based on the first facial features, the first makeup features, and the first instruction features to obtain the target image processing model.

2. The method according to claim 1, characterized in that, The encoding process of the training data pairs to obtain the first facial feature, the first makeup feature, and the first instruction feature includes: The facial features and face shape information of the subject in the second original image are encoded using a face shape encoding model to obtain the first face feature; The makeup information of the subject in the second original image is encoded using a makeup coding model to obtain the first beauty feature; The first image processing parameters are encoded using an instruction encoding model to obtain the first instruction feature.

3. The method according to claim 1, characterized in that, Before training the initial image processing model based on the first facial features, the first makeup features, and the first instruction features to obtain the target image processing model, the method further includes: The first original image is encoded using an image compression model, and random noise is added to the encoded first original image to obtain a first random noise feature vector. The background information of the photographed object in the first original image is encoded using the image compression model to obtain the first background feature; The step of training the initial image processing model based on the first facial features, the first makeup features, and the first instruction features to obtain the target image processing model includes: The initial image processing model is trained based on the first facial feature, the first makeup feature, the first instruction feature, the first random noise feature vector, and the first background feature to obtain the target image processing model.

4. The method according to claim 3, characterized in that, The step of training an initial image processing model based on the first facial features, the first makeup features, the first instruction features, the first random noise feature vector, and the first background features to obtain a target image processing model includes: The first face feature, the first makeup feature, the first instruction feature, the first random noise feature vector, and the first background feature are input into the initial image processing model to obtain the first predicted noise vector. The loss function value of the initial image processing model is determined based on the mean square error of the first predicted noise vector and the first random noise feature vector. The target image processing model is obtained when the loss function value satisfies the training stopping condition.

5. The method according to claim 1, characterized in that, After encoding the training data pairs to obtain the first facial feature, the first makeup feature, and the first instruction feature, the method further includes: Obtain the random probability value of the first feature, wherein the first feature is any one of the first facial feature, the first beauty feature, and the first instruction feature; If the random probability of the first feature taking effect is less than the preset probability of the first feature taking effect, the preset feature is used as the first feature, and the preset feature is a feature in which all elements are 0.

6. An image processing method, characterized in that, The method, applied to the target image processing model according to any one of claims 1-5, comprises: Acquire a pair of data to be processed, the pair of data to be processed including an image to be repaired, a reference image corresponding to the image to be repaired, and second image processing parameters for processing the image to be repaired, wherein the image to be repaired and the reference image are face images corresponding to the same subject being photographed. The data pairs to be processed are encoded to obtain the second facial feature, the second makeup feature, and the second instruction feature; The second facial feature, the second makeup feature, and the second instruction feature are input into the target image processing model to obtain image latent variables; The latent variables of the image are decoded to obtain the target image.

7. The method according to claim 6, characterized in that, The acquisition of the data pair to be processed includes: Obtain the image to be repaired; Select a reference image from the reference image library that corresponds to the image to be repaired; Receive second image processing parameters input by the user for processing the image to be repaired.

8. The method according to claim 6, characterized in that, Before inputting the second facial feature, the second makeup feature, and the second instruction feature into the target image processing model to obtain the image latent variables, the method further includes: The image to be repaired is encoded using an image compression model, and random noise is added to the encoded image to obtain a second random noise feature vector. The background information of the photographed object in the image to be repaired is encoded using the image compression model to obtain the second background feature. The step of inputting the second facial feature, the second makeup feature, and the second instruction feature into the target image processing model to obtain image latent variables includes: The second facial feature, the second makeup feature, the second instruction feature, the second random noise feature vector, and the second background feature are input into the target image processing model to obtain the image latent variables.

9. The method according to claim 8, characterized in that, The step of inputting the second facial feature, the second makeup feature, the second instruction feature, the second random noise feature vector, and the second background feature into the target image processing model to obtain the image latent variables includes: The second face feature, the second makeup feature, the second instruction feature, the second random noise feature vector, and the second background feature are input into the target image processing model to obtain the second predicted noise vector; The image latent variable is obtained by removing the second predicted noise vector from the second random noise feature vector using the target image processing model.

10. An image processing model training device, characterized in that, The device includes: The acquisition module is used to acquire training data pairs of image training data. The training data pairs include first image processing parameters, a first original image, a second original image, a first reference image, and a second reference image. The first reference image is obtained by processing the first original image based on the first image processing parameters, and the second reference image is obtained by processing the second original image based on the first image processing parameters. The first original image and the second original image are face images corresponding to the same subject. The encoding module is used to encode the training data pairs to obtain the first facial feature, the first makeup feature, and the first instruction feature; The model training module is used to train the initial image processing model based on the first facial features, the first makeup features, and the first instruction features to obtain the target image processing model.

11. An image processing apparatus, characterized in that, The apparatus, applied to the target image processing model of claim 10, comprises: The acquisition module is used to acquire a pair of data to be processed, which includes an image to be repaired, a reference image corresponding to the image to be repaired, and second image processing parameters for processing the image to be repaired. The image to be repaired and the reference image are face images corresponding to the same subject. The encoding module is used to encode the data pair to be processed to obtain the second facial feature, the second beauty feature, and the second instruction feature. The determination module is used to input the second facial feature, the second makeup feature, and the second instruction feature into the target image processing model to obtain image latent variables; The decoding module is used to decode the latent variables of the image to obtain the target image.

Images