Image processing method, device, electronic device and storage medium

Abstract

The present disclosure discloses an image processing method, apparatus, electronic device, and storage medium, wherein the method comprises: inputting a facial image to be processed into a preset model; and outputting, through the preset model, a target facial image with a preset object removed from the facial image to be processed; wherein the preset model is generated based on attention map training of the preset object. Because the preset model is trained based on the attention map of the preset object, it is possible to first generate an attention map of the preset object based on unpaired data training, and then use the attention map of the preset object to train and remove the preset object from the facial image. This allows for a model for removing preset objects from facial images to be trained based on unpaired data, reducing training costs.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of image processing technology, and in particular to an image processing method, apparatus, electronic device, and storage medium. Background Art

[0002] Nowadays, facial image beautification functions have been widely used in various application software, such as image / video editing software, shooting software, etc.

[0003] Existing technologies typically require the collection of large amounts of paired data (i.e., images of the same person in different states) for model training. The trained model is then used to implement the transitions between different facial states within the facial images. The drawbacks of existing technologies include the difficulty of collecting and acquiring paired data, which increases training costs. Summary of the Invention

[0004] The embodiments of the present disclosure provide an image processing method, apparatus, electronic device, and storage medium, which can implement a model for removing preset objects from facial images based on unpaired data training, thereby reducing training costs.

[0005] In a first aspect, an embodiment of the present disclosure provides an image processing method, comprising:

[0006] Input the face image to be processed into the preset model;

[0007] Outputting a target face image by using the preset model to remove a preset object from the face image to be processed;

[0008] The preset model is generated based on the attention map training of the preset object.

[0009] In a second aspect, the embodiments of the present disclosure further provide an image processing device, including:

[0010] An input module, used to input the face image to be processed into a preset model;

[0011] An output module, configured to output a target face image with a preset object removed from the face image to be processed through the preset model;

[0012] The preset model is generated based on the attention map training of the preset object.

[0013] In a third aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising:

[0014] one or more processors;

[0015] a storage device for storing one or more programs,

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the image processing method as described in any one of the embodiments of the present disclosure.

[0017] In a fourth aspect, an embodiment of the present disclosure further provides a storage medium comprising computer-executable instructions, which, when executed by a computer processor, are used to execute the image processing method as described in any one of the embodiments of the present disclosure.

[0018] The technical solution of the disclosed embodiments involves inputting a facial image to be processed into a preset model; the preset model then outputs a target facial image with a preset object removed from the facial image to be processed; the preset model is generated based on an attention map trained on the preset object. Because the preset model is trained on an attention map of the preset object, it is possible to first generate an attention map of the preset object based on unpaired data training, and then use the attention map of the preset object to train the removal of the preset object from the facial image. This allows for training a model for removing the preset object from facial images based on unpaired data, reducing training costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that the originals and elements are not necessarily drawn to scale.

[0020] Figure 1 A flowchart of an image processing method provided in the first embodiment of the present disclosure;

[0021] Figure 2 A flowchart of a preset model training step in an image processing method provided in the first embodiment of the present disclosure;

[0022] Figure 3 A schematic diagram of a preset model training step in an image processing method provided in the second embodiment of the present disclosure;

[0023] Figure 4 A schematic flow chart of a preset model training step in an image processing method provided in the third embodiment of the present disclosure;

[0024] Figure 5 A schematic diagram of a preset model training step in an image processing method provided in the third embodiment of the present disclosure;

[0025] Figure 6 A schematic structural diagram of an image processing device provided in a fourth embodiment of the present disclosure;

[0026] Figure 7This is a structural diagram of an electronic device provided in Example 5 of the present disclosure. DETAILED DESCRIPTION

[0027] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0028] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0029] As used herein, the term "including" and its variations are open-ended, i.e., "including but not limited to." The term "based on" means "based, at least in part, on." The term "one embodiment" means "at least one embodiment," the term "another embodiment" means "at least one additional embodiment," and the term "some embodiments" means "at least some embodiments." Other terms are defined in the following description.

[0030] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0031] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0032] Example 1

[0033] Figure 1 This is a flowchart of an image processing method provided in Embodiment 1 of the present disclosure. This embodiment of the present disclosure is applicable to removing predetermined objects from an image using a predetermined model trained with unpaired data. The method can be performed by an image processing device, which can be implemented in software and / or hardware.

[0034] like Figure 1 As shown, the image processing method provided by this embodiment includes:

[0035] S110, inputting the face image to be processed into a preset model;

[0036] S120 , outputting a target facial image with a preset object removed from the facial image to be processed through a preset model.

[0037] The device for executing the image processing method provided in the embodiments of the present disclosure can be integrated into application software that supports image processing functions and can be installed along with the application software on electronic devices such as servers, mobile phones, and computers. The application software can be multimedia application software involving images / videos, such as image / video editing software, shooting software, multimedia sharing software, and multimedia communication software, etc., and the examples are not exhaustive.

[0038] When an electronic device runs these applications, the application can call upon the electronic device's image acquisition function to capture a facial image to be processed, or it can read the facial image to be processed from the electronic device's storage space. The facial image to be processed can be a facial image containing a preset object. The preset object can be understood as an object in the facial image that is desired to be removed, including, but not limited to, facial hair, bangs, or eye bags.

[0039] The application software can also receive processing instructions for a facial image to be processed. In response to the processing instructions, the application can invoke an image processing device to input the facial image to be processed into a preset model. The preset model then outputs a target facial image with the preset object removed from the facial image to be processed. By utilizing the preset model for facial image processing, special effects such as removing the preset object from the image can be achieved, enhancing the user experience.

[0040] The preset model can be considered a machine learning model, such as a neural network model, and can be generated based on attention maps of preset objects. The training process of the preset model can include first generating an attention map of the preset object based on unpaired data training, and then using the attention map of the preset object to train and remove the preset object from the face image.

[0041] If the images in one set of images can be paired with the images in the other set of images, the two sets of images can be considered paired data; conversely, if there are images that cannot be paired, the two sets of images can be considered unpaired data. If the difference between the two images is the presence or absence of a preset object, and other preset factors are consistent (for example, belonging to the same individual, consistent ambient lighting, consistent position and posture, etc.), the two images can be considered paired. In this embodiment, two sets of images belonging to any individual, with and without the preset object, can be considered unpaired data.

[0042] In the field of machine learning, the attention mechanism may include a saliency-based attention mechanism. In this embodiment, the attention map of the preset object can be generated based on the saliency-based attention mechanism training. For example, it may include: based on the significant difference of the input unpaired data (such as whether the preset object exists), the data with the difference is selected from the input data for key processing, and other data is ignored to determine the attention map of the preset object. Moreover, after determining the attention map of the preset object, the preset object in the corresponding original image can be removed according to the position of the preset object in the attention map of the preset object.

[0043] Because the preset model is trained based on the attention map of the preset object, it is possible to first generate the attention map of the preset object based on unpaired data training, and then use the attention map of the preset object to train and remove the preset object from the face image. This allows a model for removing the preset object from the face image to be trained based on unpaired data, reducing training costs.

[0044] For example, Figure 2 This is a flow chart of the preset model training steps in the image processing method provided in the first embodiment of the present disclosure. Figure 2 In some optional implementations, the preset model may be generated based on the following steps:

[0045] S210: Training a first model based on a first facial image containing a preset object and a second facial image not containing the preset object.

[0046] The first facial image contains a preset object, while the second facial image does not, and the first and second facial images may be unpaired facial images. The first and second facial images may be acquired by at least one of the following methods: acquiring with the authorization of the user to whom the facial images belong; rendering a virtual character; or automatically generating the image using a network. Furthermore, the first and second facial images may be acquired by other methods, which are not exhaustive here. Theoretically, the more first and second facial images acquired, and the more diverse the angles and lighting of the characters in the images, the more conducive it is to learning and training the first model.

[0047] Among them, the first model can be considered as a machine learning model, such as a neural network model. Since the first face image and the second face image are classified according to whether they contain a preset object, the preset object can be considered to be a significant difference between the two types of images. The attention map of the preset object can be considered as an image output by the first model after highlighting the preset object contained in the face image under the attention mechanism based on saliency. The method of highlighting the preset object can be to set the pixel value of the pixel point of the preset object to be different from the pixel value of other pixel points, for example, setting the pixel point of the preset object to the pixel value corresponding to white, and setting the pixel value of other pixel points to the pixel value corresponding to black.

[0048] The first model can be trained to automatically distinguish between the first and second facial images, thereby locating a preset object. Training the first model based on unpaired first and second facial images can include: predicting significant differences between the two images through weakly supervised learning, while the first model only determines the classification of the first and second facial images; and, during the learning process, modifying the parameters of each layer of the network in the first model with the goal of improving prediction accuracy, thereby training the first model.

[0049] S220: Output an attention map of a preset object in the first face image through the trained first model.

[0050] Among them, the trained first model can realize the positioning of the preset object and highlight the pixel points of the preset object to obtain the attention map of the preset object.

[0051] S230: Training a second model based on the first face image and the attention map.

[0052] The second model can also be considered a machine learning model, such as a neural network model. Typically, the attention map has the same resolution as the first facial image. The second model can process the corresponding preset object in the first facial image based on the location of the preset object in the attention map, and output an image of the first facial image after removing the preset object. Furthermore, the second model can be trained based on the image output by the second model and the second facial image, so that the image output by the second model can approximate the image without the preset object.

[0053] S240: Generate a preset model based on the trained first model and the trained second model.

[0054] By using an unpaired first and second facial image to perform a first phase of training on the first model, the first model can automatically distinguish the differences between the first and second facial images, locate the preset object, and output an attention map for the preset object. By training the second model based on the attention map and the first facial image, the second model can achieve the effect of removing the preset object.

[0055] After the first model and the second model are trained, a preset model is generated using the first model and the second model, which may include but is not limited to: combining the first model and the second model as sub-models to obtain a preset model; or processing unpaired data into paired data according to the first model and the second model, and performing model training based on the paired data to obtain a preset model.

[0056] The technical solution of the disclosed embodiments involves inputting a facial image to be processed into a preset model; the preset model then outputs a target facial image with a preset object removed from the facial image to be processed; the preset model is generated based on an attention map trained on the preset object. Because the preset model is trained on an attention map of the preset object, it is possible to first generate an attention map of the preset object based on unpaired data training, and then use the attention map of the preset object to train the removal of the preset object from the facial image. This allows for training a model for removing the preset object from facial images based on unpaired data, reducing training costs.

[0057] Example 2

[0058] The embodiments of the present disclosure can be combined with the various optional solutions for presetting model training in the image processing method provided in the above embodiments. This embodiment describes in detail the training steps of the first model and the second model.

[0059] In the embodiment of the present disclosure, the first model can be trained based on the following steps:

[0060] The image labels of the first face image and the second face image are set differently; the first face image, the second face image and the image labels corresponding to each image are input into the first model; the candidate objects are determined by the first model, and the predicted labels of the first face image and the second face image are output based on the candidate objects; the first model is trained based on the predicted labels and the set image labels, and the candidate objects determined by the trained first model are used as preset objects.

[0061] Image labels can be set for the first and second facial images, respectively, and the two image labels can uniquely correspond to the two types of images. The image labels can be composed of at least one element such as text, letters, numbers, and special symbols. For example, the image label of the first facial image can be set as a "positive sample," and the image label of the second facial image can be set as a "negative sample."

[0062] By inputting a first facial image, a second facial image, and the image labels of each image into a first model, the first model can determine the classification of the first facial image and the second facial image. After determining the classification of the facial image, the first model can cyclically determine candidate objects and reclassify the input facial images based on the candidate objects determined in the current cycle to obtain predicted labels for each facial image. The first model can be trained with the goal of ensuring that the predicted labels are consistent with the set image labels, so that the first model can be trained to accurately distinguish the significant differences between the first facial image and the second facial image, that is, to locate the preset objects.

[0063] For example, Figure 3 This is a schematic diagram of a preset model training step in an image processing method provided in the second embodiment of the present disclosure. Figure 3 A large number of unpaired first face images A' and second face images B can be input into the first model M1, so that the first model M1 can automatically locate the preset objects (such as beards) that are different in the first face images A' and the second face images B through training, and output the attention map a of the preset objects.

[0064] In the embodiment of the present disclosure, the second model can be trained based on the following steps:

[0065] The first face image and the attention map are input into the second model, and the second model outputs a third face image with the preset object removed from the first face image; the second face image and the third face image are input into the first discriminator, and the second model is trained according to the discrimination result of the first discriminator.

[0066] The attention map is the attention map of the preset object output upon completion of training for the first model. This attention map is not updated during training for the second model. Because the preset object is highlighted in the attention map, the second model can determine the location of the preset object in the first facial image based on the attention map and remove the preset object from the first facial image based on this location to obtain a third facial image.

[0067] See again Figure 3 The first face image A' input to the first model M1 and the attention map a output by the first model M1 can be used as the input to the second model M2, so that the second model M2 outputs a third face image A with the preset object removed from the first face image A'.

[0068] The second model and the first discriminator can form a generative adversarial network and can be trained along with the generative adversarial network. The training process can also include two stages: first, training the first discriminator, i.e., with the parameters of the second model fixed, using the second model to output a third facial image; then inputting the third facial image and the second facial image into the first discriminator to determine whether the facial images are real facial images excluding a predetermined object or facial images obtained by the second model with the predetermined object removed. Furthermore, the first discriminator can be trained with the goal of improving the accuracy of the discrimination results. Second, training the second model, i.e., with the parameters of the first discriminator fixed, training the second model with the goal of ensuring that the first discriminator cannot distinguish between the second and third facial images. This two-stage training process of the generative adversarial network can be cyclical, with either the second model or the first discriminator trained first. Training can be terminated when a preset condition is met. The preset conditions may include, but are not limited to, a predetermined number of cycles, the accuracy of the first discriminator reaching a certain level, etc., which are not exhaustive here.

[0069] See also Figure 3 The second facial image B and the third facial image A can be input into the first discriminator D1 to determine whether these facial images are real facial images that do not contain the preset object, or facial images that have had the preset object removed by the second model M2. Furthermore, the first discriminator D1 can be trained with the goal of improving the accuracy of the discrimination results. After the first discriminator D1 is trained, the parameters of the first discriminator D1 can be fixed, and the second model M2 can be trained with the goal of ensuring that the first discriminator D1 cannot distinguish between the second facial image B and the third facial image A.

[0070] By training the second model against the first discriminator, the second model's effectiveness at removing pre-set objects can be improved. Training the model for removing pre-set objects from facial images based on unpaired data significantly reduces the model training process's reliance on paired data, thereby reducing costs.

[0071] In some optional implementations, outputting a third facial image with a preset object removed from the first facial image through a second model may include: processing the pixel points corresponding to the preset object in the first facial image according to the attention map through the second model, and outputting the third facial image with the preset object removed.

[0072] In these optional implementations, the second model can determine the coordinate position of the pixel point of the preset object in the first facial image based on the attention map, and can process the preset object in the first facial image pixel by pixel based on the coordinate position to obtain a third facial image with the preset object removed. Particularly, processing the preset object in the first facial image pixel by pixel can include changing the pixel value of the preset object pixel by pixel. Particularly, changing the pixel value of the preset object can include changing the pixel value of the pixel point of the preset object by using the pixel value of the pixel points adjacent to the pixel point of the preset object, or using the pixel value of the pixel points at other positions of the facial tissue, so as to achieve the effect of removing the preset object.

[0073] In some further implementations, processing pixels corresponding to a preset object in the first facial image includes: copying and transferring pixels in the first facial image that are not marked by the attention map to the locations of pixels marked by the attention map; wherein the pixels marked by the attention map belong to the preset object. In these further implementations, the preset object can be marked pixel by pixel in the attention map. Accordingly, the second model can copy and transfer the unmarked pixels to the locations of the marked pixels to cover the original marked preset object, thereby achieving the purpose of removing the preset object.

[0074] Furthermore, before outputting the third facial image with the preset object removed, the method may further include performing a preset adjustment process on the third facial image. The preset adjustment process may include fine-tuning the image, such as smoothing or filtering the image pixel values, to make the rough result after removing the preset object according to the attention map more realistic and natural.

[0075] The technical solution of the embodiment of the present disclosure describes in detail the training steps of the first model and the second model. By determining the candidate objects through the first model loop, and reclassifying the input first / second face image according to the candidate objects determined in the current loop, the predicted label of each face image can be obtained; and the first model is trained with the goal of consistency between the predicted label and the set image label, so that the trained first model can accurately distinguish the significant difference between the first face image and the second face image, that is, locate the preset object. The effect of the second model in removing the preset object can be improved by training the mutual game between the second model and the first discriminator. By training the model for removing the preset object in the face image based on unpaired data, the dependence of the model training process on paired data can be significantly reduced, thereby reducing costs.

[0076] The model training scheme in the image processing method provided in the embodiment of the present disclosure and the model training scheme in the image processing method provided in the above embodiment belong to the same disclosed concept. Technical details not fully described in this embodiment can be found in the above embodiment, and the same technical features have the same beneficial effects in this embodiment and the above embodiment.

[0077] Example 3

[0078] The embodiments of the present disclosure can be combined with the various optional solutions for training the preset model in the image processing method provided in the above embodiments. This embodiment describes in detail the steps of generating the preset model based on the trained first model and the trained second model.

[0079] In some optional implementations, a preset model is generated based on the trained first model and the trained second model, including: establishing a connection between the output layer of the trained first model and the input layer of the trained second model, and integrating them to obtain the preset model.

[0080] In these optional implementations, the first model and the second model can be integrated as sub-models to obtain a model for removing preset objects from an image. The connection between the output layer of the trained first model and the input layer of the trained second model can include: using the output layer of the attention map output by the first model as the input layer of the second model, and establishing a connectivity relationship between the input layer of the first model inputting the first face image and the input layer of the second model. The integrated model can be trained in two stages using unpaired face images, i.e., the first model is trained in the first stage using the first face image and the second face image, and the second model is trained in the second stage using the attention map output by the first model, the first face image, and the second face image. The integrated preset model after training can achieve the removal of preset objects from an image.

[0081] In addition, in some other methods of generating preset models, the following can also be done: first, the third model can be preliminarily trained by using a small amount of collected paired data (i.e., the fourth / fifth facial image). At this time, due to the small amount of training data, the effect of the third model obtained by training on removing the preset object is poor. Then, by using the third model to process a large number of sixth facial images containing the preset object, a large number of images with unclean preset objects removed can be obtained, and they are used as the first facial images containing the preset object. Then, by using the trained first model and the trained second model, the third facial image with the preset object removed can be obtained, thereby obtaining a large amount of paired data (i.e., the third / sixth facial image). Finally, the third model can be optimized and trained using a large amount of paired images, and the optimized third model can be used as the preset model. In this way, it is possible to train the third model using a small amount of paired data and a large amount of unpaired data, so that the third model has a good effect of removing the preset object.

[0082] For example, Figure 4 This is a flow chart of the preset model training steps in an image processing method provided in the third embodiment of the present disclosure. Figure 4 As shown, in some optional implementations, the preset model can be generated based on the following steps:

[0083] S410: Acquire a first number of fourth facial images containing a preset object, fifth facial images corresponding to the fourth facial images but not containing the preset object, and a second number of sixth facial images containing the preset object.

[0084] In this embodiment, the fourth facial image contains a preset object, while the fifth facial image does not. Furthermore, the fourth and fifth facial images may form a pair of facial images. The sixth facial image also contains a preset object, and the fourth and sixth facial images may overlap. The methods for acquiring the fourth, fifth, and sixth facial images can refer to other embodiments of this disclosure and are not further elaborated here.

[0085] Because paired image acquisition is relatively difficult, a first number of paired fourth / fifth facial images can be acquired within a certain acquisition cost range. However, acquiring only the sixth facial image containing the preset object is less difficult, allowing for a large number of sixth facial images to be acquired. While controlling acquisition costs, the second number can be greater than the first number, and can be significantly greater.

[0086] S420: Preliminarily train a third model based on the fourth face image and the fifth face image.

[0087] In this embodiment, the third model can also be considered a machine learning model, such as a neural network model. Training the third model based on the fourth facial image and the fifth facial image may include: using the third model to generate an image with a predetermined object removed from the fourth facial image, and training the third model with the goal of achieving a predetermined degree of similarity between the output image and the fifth facial image.

[0088] In some optional implementations, the third model can be preliminarily trained based on the following steps: inputting the fourth facial image into the third model so that the third model outputs the seventh facial image; inputting the fifth facial image and the seventh facial image into the second discriminator, and preliminarily training the third model based on the discrimination results of the second discriminator.

[0089] In these optional implementations, the third model and the second discriminator can also form a generative adversarial network and can be trained with the generative adversarial network. This training process can also include two stages: first, training the second discriminator, i.e., using the third model to output the seventh facial image while keeping the third model parameters fixed; then inputting the fifth and seventh facial images into the second discriminator to determine whether these facial images are real facial images excluding the preset object or facial images obtained by removing the preset object using the third model. Furthermore, the second discriminator can be trained with the goal of improving the accuracy of the discrimination results. Second, training the third model, i.e., using the second discriminator parameters fixed and the goal of ensuring that the second discriminator cannot distinguish between the fifth and seventh facial images, can be trained. This two-stage training process can be cyclical, with either the third model or the second discriminator trained first, or it can be terminated by looping until a preset condition is met. The preset conditions may include, but are not limited to, a predetermined number of loops and the second discriminator reaching a certain accuracy, etc., which are not exhaustive here.

[0090] The small amount of paired data collected (i.e., the fourth / fifth face image) can be used to perform preliminary training on the third model. At this time, due to the small amount of training data, the effect of the third model obtained by preliminary training on removing the preset objects is poor.

[0091] S430: Input the sixth facial image into the preliminarily trained third model, use the image output by the preliminarily trained third model as the first facial image, and obtain a second facial image that does not include the preset object.

[0092] By using the preliminarily trained third model to process a large number of sixth facial images containing the preset object, a large number of facial images with poor preset object removal can be obtained. Since the preset object cannot be completely removed from the images output by the third model, these facial images can be used as the first facial images and continue the preset object removal process. Furthermore, the obtained second and fifth facial images may overlap.

[0093] S440: Train a first model based on a first facial image containing a preset object and a second facial image not containing the preset object.

[0094] S450: Output an attention map of a preset object in the first face image through the trained first model.

[0095] S460: Train a second model based on the first face image and the attention map.

[0096] S470: Process the first facial image with the trained first model and the trained second model to obtain a third facial image with the preset object removed from the first facial image.

[0097] After the first model and the second model are trained, the first facial image can be processed by the first model to obtain a corresponding attention map, and the first facial image and the attention map can be processed by the second model to obtain a third facial image in which the preset object is removed from the first facial image. Thus, based on the trained first and second models, a large number of paired facial images (i.e., a large number of sixth facial images containing the preset object and corresponding third facial images not containing the preset object) can be obtained from a large number of unpaired facial images (i.e., the first facial image and the second facial image).

[0098] S480: Optimize and train a third model based on the third facial image and the sixth facial image, and use the optimized and trained third model as a preset model.

[0099] By optimizing and training the third model based on a large number of paired facial images, the third model can achieve superior results in removing pre-set objects. This allows the third model to be trained using a small amount of paired data and a large amount of unpaired data, resulting in excellent beard removal performance. Furthermore, compared to traditional methods of collecting paired facial images, generating a large number of paired facial images based on unpaired facial images using the trained first and second models significantly reduces data acquisition costs.

[0100] For example, Figure 5 This is a schematic diagram of a preset model training step in an image processing method provided in the third embodiment of the present disclosure. Figure 5The first number of paired fourth face images C' and fifth face images C can be used to perform preliminary training on the third model M3. At this time, due to the small amount of paired data, the effect of removing the preset object (such as beard) obtained by the preliminary training of the third model is poor. The third model obtained by the preliminary training can be used to process the second number of sixth face images E' to obtain a large number of first face images A'. The process of training the first model M1 and the second model M2 based on the first face image A' and the second face image B can be seen in Figure 2 The trained second model can output a third face image A with the preset object removed, thereby obtaining a large amount of paired data, namely, the third face image A and the sixth face image E. The third face image A and the sixth face image E' can be used to optimize the training of the third model M3 so that the third model has a better effect of removing the preset object.

[0101] In this embodiment, the optimized and trained third model can be used to remove preset objects from facial images. The integrated model of the trained first and second models can also be used to remove preset objects from facial images. The optimized and trained third model or integrated model can be integrated into an electronic device, such as a server or client, to perform online special effects to remove preset objects from facial images, thereby achieving facial image beautification.

[0102] Furthermore, compared to the integrated model, the third model is typically smaller, making it easier to integrate into electronic devices with limited storage / computing resources, such as client devices. Furthermore, compared to the steps involved in training the third model, the steps involved in training the integrated model are more streamlined, making it easier to quickly train a model that effectively removes the preset object. Therefore, depending on the specific needs of the actual application scenario, one can selectively train either the integrated model or the third model; both can effectively remove the preset object.

[0103] The technical solution of the embodiment of the present disclosure can integrate the trained first model and the second model to obtain a preset model, or use the third model after optimization training as the preset model. The integrated model or the third model can be selectively trained according to the specific needs of the actual application scenario, enriching the generation method of the preset model. The model training scheme in the image processing method provided by the embodiment of the present disclosure and the model training scheme in the image processing method provided by the above embodiment belong to the same public concept. The technical details not fully described in this embodiment can be referred to the above embodiment, and the same technical features have the same beneficial effects in this embodiment and the above embodiment.

[0104] Example 4

[0105] Figure 6This is a schematic diagram of the structure of an image processing device provided by the fourth embodiment of the present disclosure. The model training device provided in this embodiment is suitable for removing preset objects in an image using a preset model obtained by training with unpaired data.

[0106] like Figure 6 As shown, the image processing device provided in this embodiment may include:

[0107] Input module 610, used to input the face image to be processed into a preset model;

[0108] An output module 620 is configured to output a target face image with a preset object removed from the face image to be processed using a preset model;

[0109] Among them, the preset model is generated based on the attention map training of the preset object.

[0110] In some optional implementations, the preset objects include beard, bangs, or eye bags.

[0111] In some optional implementations, the image processing apparatus may further include:

[0112] The model training module is used to generate a preset model based on the following steps:

[0113] Training a first model based on a first facial image containing a preset object and a second facial image not containing the preset object;

[0114] Outputting an attention map of a preset object in the first face image through the trained first model;

[0115] Training a second model based on the first face image and the attention map;

[0116] A preset model is generated based on the trained first model and the trained second model.

[0117] In some optional implementations, the model training module may include:

[0118] The first model training unit may be configured to train the first model based on the following steps:

[0119] Differentiating image labels of the first face image and the second face image;

[0120] Inputting the first face image, the second face image, and the image labels corresponding to the images into the first model;

[0121] Determine a candidate object through the first model, and output predicted labels for the first face image and the second face image based on the candidate object;

[0122] The first model is trained according to the predicted label and the set image label, and the candidate object determined by the trained first model is used as the preset object.

[0123] In some optional implementations, the model training module may further include:

[0124] The second model training unit can be used to train the second model based on the following steps:

[0125] Inputting the first face image and the attention map into a second model, and outputting a third face image with the preset object removed from the first face image through the second model;

[0126] The second face image and the third face image are input into the first discriminator, and the second model is trained according to the discrimination result of the first discriminator.

[0127] In some optional implementations, the second model training unit may be specifically configured to:

[0128] The second model processes the pixel points corresponding to the preset object in the first face image according to the attention map, and outputs a third face image without the preset object.

[0129] In some optional implementations, the second model training unit may be specifically configured to:

[0130] Copy and transfer the pixels in the first face image that are not marked by the attention map to the positions of the pixels marked by the attention map;

[0131] Among them, the pixels marked by the attention map belong to the preset objects.

[0132] In some optional implementations, the second model training unit may be further specifically configured to: perform preset adjustment processing on the third facial image before outputting the third facial image with the preset object removed.

[0133] In some optional implementations, the model training module can be specifically used to:

[0134] The output layer of the trained first model is connected to the input layer of the trained second model to obtain a preset model.

[0135] In some optional implementations, the model training module may include:

[0136] The first face acquisition unit may be configured to acquire a first face image based on the following steps:

[0137] Acquire a first number of fourth facial images containing a preset object, a fifth facial image corresponding to the fourth facial image but not containing the preset object, and a second number of sixth facial images containing the preset object; wherein the second number is greater than the first number;

[0138] preliminarily training a third model based on the fourth face image and the fifth face image;

[0139] Inputting the sixth facial image into the preliminarily trained third model, and using the image output by the preliminarily trained third model as the first facial image;

[0140] Accordingly, the model training module can be used specifically for:

[0141] Processing the first facial image with the trained first model and the trained second model to obtain a third facial image in which a preset object is removed from the first facial image;

[0142] The third model is optimized and trained according to the third face image and the sixth face image, and the optimized and trained third model is used as the preset model.

[0143] In some optional implementations, the model training module may include:

[0144] The third model training unit may be used to preliminarily train the third model based on the following steps:

[0145] inputting the fourth facial image into the third model so that the third model outputs a seventh facial image;

[0146] The fifth face image and the seventh face image are input into the second discriminator, and the third model is preliminarily trained according to the discrimination result of the second discriminator.

[0147] The image processing device provided by the embodiments of the present disclosure can execute the image processing method provided by any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0148] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present disclosure.

[0149] Example 5

[0150] Reference below Figure 7 , which shows an electronic device (eg Figure 7The terminal device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (such as in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0151] like Figure 7 As shown, the electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. Various programs and data required for the operation of the electronic device 700 are also stored in the RAM 703. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0152] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0153] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 709, or installed from the storage device 708, or installed from the ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the image processing method of the embodiment of the present disclosure are performed.

[0154] The electronic device provided by the embodiment of the present disclosure and the image processing method provided by the above embodiment belong to the same disclosed concept. For technical details not fully described in this embodiment, please refer to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0155] Example 6

[0156] An embodiment of the present disclosure provides a computer storage medium having a computer program stored thereon. When the program is executed by a processor, the image processing method provided by the above embodiment is implemented.

[0157] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor image processing method, apparatus or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM) or flash memory (FLASH), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with instruction execution, an image processing method, apparatus or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or convey a program for use by or in conjunction with an instruction execution, image processing method, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), or the like, or any suitable combination thereof.

[0158] In some embodiments, the client and server can communicate using any currently known or future developed network protocol, such as HTTP (Hypertext Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0159] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0160] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device:

[0161] The face image to be processed is input into a preset model; the target face image with the preset object removed from the face image to be processed is output by the preset model; wherein the preset model is generated based on the attention map training of the preset object.

[0162] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0163] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based implementation that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0164] The units involved in the embodiments described in this disclosure may be implemented in software or hardware, wherein the names of the units and modules do not, in certain circumstances, limit the units and modules themselves.

[0165] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: Field Programmable Gate Array (FPGA), Application Specific Integrated Circuit (ASIC), Application Specific Standard Parts (ASSP), System on Chip (SOC), Complex Programmable Logic Device (CPLD), and the like.

[0166] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use with or in conjunction with an instruction execution, an image processing method, a device or an apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor, an image processing method, a device or an apparatus, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0167] According to one or more embodiments of the present disclosure, [Example 1] provides an image processing method, the method comprising:

[0168] Input the face image to be processed into the preset model;

[0169] Outputting a target face image by using the preset model to remove a preset object from the face image to be processed;

[0170] The preset model is generated based on the attention map training of the preset object.

[0171] According to one or more embodiments of the present disclosure, [Example 2] provides an image processing method, further comprising:

[0172] In some optional implementations, the preset objects include beard, bangs, or eye bags.

[0173] According to one or more embodiments of the present disclosure, [Example 3] provides an image processing method, further comprising:

[0174] In some optional implementations, the preset model is generated based on the following steps:

[0175] Training a first model based on a first facial image containing the preset object and a second facial image not containing the preset object;

[0176] Outputting an attention map of a preset object in the first face image through the trained first model;

[0177] Training a second model based on the first face image and the attention map;

[0178] A preset model is generated based on the trained first model and the trained second model.

[0179] According to one or more embodiments of the present disclosure, [Example 4] provides an image processing method, further comprising:

[0180] In some optional implementations, the first model is trained based on the following steps:

[0181] Differentiating image labels of the first facial image and the second facial image;

[0182] Inputting the first face image, the second face image, and the image labels corresponding to the images into a first model;

[0183] Determining candidate objects using the first model, and outputting predicted labels for the first facial image and the second facial image based on the candidate objects;

[0184] The first model is trained according to the predicted label and the set image label, and the candidate object determined by the trained first model is used as the preset object.

[0185] According to one or more embodiments of the present disclosure, [Example 5] provides an image processing method, further comprising:

[0186] In some optional implementations, the second model is trained based on the following steps:

[0187] Inputting the first facial image and the attention map into a second model, and outputting a third facial image by the second model without the preset object in the first facial image;

[0188] The second face image and the third face image are input into a first discriminator, and the second model is trained according to the discrimination result of the first discriminator.

[0189] According to one or more embodiments of the present disclosure, [Example 6] provides an image processing method, further comprising:

[0190] In some optional implementations, outputting, by the second model, a third facial image in which the preset object is removed from the first facial image includes:

[0191] The second model processes the pixel points corresponding to the preset object in the first facial image according to the attention map, and outputs a third facial image without the preset object.

[0192] According to one or more embodiments of the present disclosure, [Example 7] provides an image processing method, further comprising:

[0193] In some optional implementations, the processing the pixels corresponding to the preset object in the first facial image includes:

[0194] Copying and transferring the pixels in the first face image that are not marked by the attention map to the positions of the pixels marked by the attention map;

[0195] Among them, the pixel points marked in the attention map belong to the preset object.

[0196] According to one or more embodiments of the present disclosure, [Example 8] provides an image processing method, including:

[0197] In some optional implementations, before outputting the third facial image with the preset object removed, the method further includes: performing a preset adjustment process on the third facial image.

[0198] According to one or more embodiments of the present disclosure, [Example 9] provides an image processing method, including:

[0199] In some optional implementations, generating a preset model based on the trained first model and the trained second model includes:

[0200] The output layer of the trained first model is connected to the input layer of the trained second model to obtain a preset model.

[0201] According to one or more embodiments of the present disclosure, [Example 10] provides an image processing method, including:

[0202] In some optional implementations, the first facial image is obtained based on the following steps:

[0203] Acquire a first number of fourth facial images containing a preset object, a fifth facial image corresponding to the fourth facial image but not containing the preset object, and a second number of sixth facial images containing the preset object; wherein the second number is greater than the first number;

[0204] Preliminarily training a third model based on the fourth facial image and the fifth facial image;

[0205] Inputting the sixth facial image into the preliminarily trained third model, and using the image output by the preliminarily trained third model as the first facial image;

[0206] Accordingly, generating a preset model based on the trained first model and the trained second model includes:

[0207] Processing the first facial image with the trained first model and the trained second model to obtain a third facial image in which the preset object is removed from the first facial image;

[0208] The third model is optimized and trained according to the third facial image and the sixth facial image, and the optimized and trained third model is used as the preset model.

[0209] According to one or more embodiments of the present disclosure, [Example 11] provides an image processing method, including:

[0210] In some optional implementations, the third model is preliminarily trained based on the following steps:

[0211] inputting the fourth facial image into a third model so that the third model outputs a seventh facial image;

[0212] The fifth face image and the seventh face image are input into a second discriminator, and the third model is preliminarily trained according to the discrimination result of the second discriminator.

[0213] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0214] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0215] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. An image processing method, characterized in that: include: Input the face image to be processed into the preset model; Outputting a target face image by using the preset model to remove a preset object from the face image to be processed; Wherein, the preset model is generated based on the attention map training of the preset object; The preset model is generated based on the following steps: Training a first model based on a first facial image containing the preset object and a second facial image not containing the preset object; Outputting an attention map of a preset object in the first face image through the trained first model; Training a second model based on the first face image and the attention map; A preset model is generated based on the trained first model and the trained second model.

2. The image processing method according to claim 1, wherein: The preset objects include beard, bangs or eye bags.

3. The image processing method according to claim 1, wherein: The first model is trained based on the following steps: Differentiating image labels of the first facial image and the second facial image; Inputting the first face image, the second face image, and the image labels corresponding to the images into a first model; Determining candidate objects using the first model, and outputting predicted labels for the first facial image and the second facial image based on the candidate objects; The first model is trained according to the predicted label and the set image label, and the candidate object determined by the trained first model is used as the preset object.

4. The image processing method according to claim 1, wherein: The second model is trained based on the following steps: Inputting the first facial image and the attention map into a second model, and outputting a third facial image by the second model without the preset object in the first facial image; The second face image and the third face image are input into a first discriminator, and the second model is trained according to the discrimination result of the first discriminator.

5. The image processing method according to claim 4, wherein: Outputting the third face image by using the second model and removing the preset object from the first face image includes: The second model processes the pixel points corresponding to the preset object in the first facial image according to the attention map, and outputs a third facial image without the preset object.

6. The image processing method according to claim 5, characterized in that The processing of the pixel points corresponding to the preset object in the first facial image includes: Copying and transferring the pixels in the first face image that are not marked by the attention map to the positions of the pixels marked by the attention map; Among them, the pixel points marked in the attention map belong to the preset object.

7. The method according to claim 5, characterized in that Before outputting the third facial image with the preset object removed, the method further includes: performing a preset adjustment process on the third facial image.

8. The image processing method according to claim 1, wherein: The generating of a preset model based on the trained first model and the trained second model includes: The output layer of the trained first model is connected to the input layer of the trained second model to obtain a preset model.

9. The method according to claim 1, characterized in that The first face image is obtained based on the following steps: Acquire a first number of fourth facial images containing a preset object, a fifth facial image corresponding to the fourth facial image but not containing the preset object, and a second number of sixth facial images containing the preset object; wherein the second number is greater than the first number; Preliminarily training a third model based on the fourth facial image and the fifth facial image; Inputting the sixth facial image into the preliminarily trained third model, and using the image output by the preliminarily trained third model as the first facial image; Accordingly, generating a preset model based on the trained first model and the trained second model includes: Processing the first facial image with the trained first model and the trained second model to obtain a third facial image in which the preset object is removed from the first facial image; The third model is optimized and trained according to the third facial image and the sixth facial image, and the optimized and trained third model is used as the preset model.

10. The method according to claim 9, characterized in that The third model is initially trained based on the following steps: inputting the fourth facial image into a third model so that the third model outputs a seventh facial image; The fifth face image and the seventh face image are input into a second discriminator, and the third model is preliminarily trained according to the discrimination result of the second discriminator.

11. An image processing device, characterized in that: include: An input module, used to input the face image to be processed into a preset model; An output module, configured to output a target face image with a preset object removed from the face image to be processed through the preset model; Wherein, the preset model is generated based on the attention map training of the preset object; The preset model is generated based on the following steps: Training a first model based on a first facial image containing the preset object and a second facial image not containing the preset object; Outputting an attention map of a preset object in the first face image through the trained first model; Training a second model based on the first face image and the attention map; A preset model is generated based on the trained first model and the trained second model.

12. An electronic device, characterized in that: The electronic device comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the image processing method according to any one of claims 1 to 10.

13. A storage medium comprising computer executable instructions, wherein the computer executable instructions are used to perform the image processing method according to any one of claims 1 to 10 when executed by a computer processor.

Images