Image processing method, device, storage medium, and program product

Abstract

The present application relates to the technical field of image processing, and discloses an image processing method, a device, a storage medium, and a program product. The image processing method comprises: if a portrait to be processed that meets a processing condition (for example, having a feature that affects image aesthetics, such as closed eyes) is present in an image to be processed, acquiring a facial feature of said portrait, and selecting, from an image library, a reference image that has high similarity to the facial feature of said portrait and compensates for said image defect (for example, a reference portrait having the same facial appearance as said portrait and having open eyes); and then performing image restoration on said image on the basis of the reference image to obtain a target image that have overcome said image defect, for example, replacing the part of said image that affects image aesthetics with the corresponding part in the reference image. In this way, the overall aesthetics of the image can be improved, facilitating the improvement of visual experience of a user viewing the target image.

Description

Image processing method, device, storage medium and program product

[0001] The present application claims priority to the Chinese patent application No. 202411135292.X, filed on August 15, 2024, and titled "Image processing method, device, storage medium and program product", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the technical field of image processing, and in particular to an image processing method, device, storage medium and program product. BACKGROUND

[0003] When a user takes a photo, the user usually blinks, frowns, or has a nervous expression, which leads to a decrease in image quality and affects the overall aesthetics of the image and the visual experience of the user when viewing the image. For example, as shown in FIG. 1, when user A and other users use the camera function of a mobile phone 10 to take a photo, if user A blinks when the camera collects an image, the image of user A in the captured image 101 will appear to have closed eyes, which affects the overall aesthetics of the image 101. SUMMARY

[0004] To solve the above problems, the present application provides an image processing method, device, storage medium and program product.

[0005] In a first aspect, the present application provides an image processing method, which includes: obtaining a first image to be processed; detecting that a first image feature that does not satisfy a preset condition exists in a first region in the first image, wherein the first region includes a first face region, and the first image feature includes a first face feature; selecting a first reference image from a gallery, wherein the similarity between a second region of the first reference image and the first region is greater than a similarity threshold, and a second image feature of the second region satisfies the preset condition; processing the first image feature in the first image based on the second image feature to obtain a second image that satisfies the preset condition.

[0006] In the present application, the first image can be a to-be-processed image mentioned in the present application; the first region can be a region where a face of the first to-be-processed portrait mentioned in the present application is located, wherein the region where the face is located at least includes a first face region; the first face region can be a face region of the first to-be-processed portrait mentioned in the present application; the first image feature can be an image feature in the region where the face of the first to-be-processed portrait mentioned in the present application is located, wherein the first image feature at least includes a first face feature; the first face feature can be a face feature of the first to-be-processed portrait mentioned in the present application, for example, an eye contour, a face contour, and the like; the first reference image can be a reference image corresponding to the first image feature mentioned in the present application; the second region can be a face region in the first reference image mentioned in the present application; the second image feature can be an image feature in the face region of the first reference image mentioned in the present application; and the second image can be a target image mentioned in the present application.

[0007] In some implementations, the presence of the first image feature in the first region that does not satisfy the preset condition can indicate that the to-be-processed portrait in the first region meets the image processing condition, and the to-be-processed portrait can be subjected to image processing.

[0008] In some implementations, after determining the to-be-processed portrait and the first image feature of the to-be-processed portrait, the electronic device can further acquire second image features (for example, face features) of other images in the gallery, and select an image that has a greater similarity to the first image feature and overcomes the image defect as the first reference image. For example, if the to-be-processed portrait that meets the processing condition is a closed-eye portrait, the electronic device can select a reference image that is open-eyed and has the same appearance as the to-be-processed portrait from other images. Then, the electronic device can process the to-be-processed image based on the reference image, for example, can cut out the part of the to-be-processed image that needs to be processed, then copy and paste the same part of the reference image to the part cut out in the to-be-processed image, and adjust the position of the pasted edge and the texture, color difference, and the like, so that the target image after processing achieves a harmonious overall effect.

[0009] In this way, by using the above image processing method, the electronic device can perform image beautification on the to-be-processed image, so that the final image obtained can overcome the image defect, thereby improving the overall visual effect of the image.

[0010] In a possible implementation of the first aspect, the method further includes: detecting an image editing operation of the user on the first image, obtaining the first image to be processed, and displaying a first image editing interface corresponding to the first image, the first image editing interface including an image processing button; in a case where the first image feature exists in the first image and the first reference image exists in the image gallery, the image processing button is in a state capable of responding to a user operation instruction; in a case where the first image feature does not exist in the first image, or the first image feature exists in the first image and the first reference image does not exist in the image gallery, the image processing button is in a state incapable of responding to the user operation instruction.

[0011] In the present application, the first image editing interface can be the image editing interface displayed when the user clicks the image editing button, as mentioned in the present application. For example, the first image editing interface can be the editing interface 202 shown in (b) of FIG. 2.

[0012] In some implementations, during the process in which the user views the image to be processed (for example, views the image to be processed in the album), when the electronic device detects the operation of the user clicking the image editing button, it can first determine whether the image to be processed contains a portrait to be processed that meets the processing condition, and in a case where the portrait to be processed exists, further determine whether the reference image that has high similarity with the image feature of the portrait to be processed and makes up for the image defect exists in each image stored by the electronic device. In a case where the portrait to be processed does not exist, or the portrait to be processed exists but the reference image does not exist, the image processing button is in a state incapable of responding to the user operation instruction, for example, in a gray state. In a case where the portrait to be processed exists in the image to be processed and the reference image exists in the image gallery, the image processing button is in a state capable of responding to the user operation instruction, for example, in a highlighted state. In a case where the image processing button is in the state capable of responding to the user operation instruction, when the electronic device detects the operation of the user clicking the image processing button, the electronic device can perform image repair on the image to be processed based on the reference image to obtain the target image.

[0013] In this way, compared with the image processing button being directly in a clickable state (for example, in a highlighted state), the phenomenon that the image processing button does not respond when the user clicks it can occur. The method of determining that the portrait to be processed exists and the reference image exists, and then making the image processing button in the clickable state, enables the electronic device to detect the clicking operation of the user every time the user clicks the image processing button, so that the image to be processed can be processed to obtain the target image, thereby improving the user experience.

[0014] In a possible implementation of the first aspect, in response to the image processing button being in a state capable of responding to a user operation instruction, the first image features in the first image are processed based on the second image features to obtain the second image satisfying the preset condition, including: in a case where it is detected that the user clicks the image processing button, the first image features in the first image are processed based on the second image features in the first reference image to obtain the second image.

[0015] In some implementations, when the electronic device detects the user operation of clicking the image processing button, the electronic device can trigger image repair of the image features corresponding to the portrait to be processed based on the determined image features of the reference image, for example, replace the eyes of the portrait to be processed with the eyes of the corresponding portrait in the reference image. In addition, the electronic device can also perform texture, brightness, and other detail adjustments on the repaired portrait image.

[0016] In this way, the electronic device can respond to the user operation instruction to perform image beautification processing on the portrait to be processed in the image to be processed, thereby obtaining the target image satisfying the preset condition.

[0017] In a possible implementation of the first aspect, the method further includes: detecting that the first image features not satisfying the preset condition exist in the first region in the first image, and the third image features not satisfying the preset condition exist in the third region, where the first region includes a first face region, the first image features include first face features, the third region includes a second face region, and the third image features include second face features; selecting the first reference image and the second reference image from the gallery, where the similarity between the second region of the first reference image and the first region is greater than a similarity threshold, the second image features of the second region satisfy the preset condition, the similarity between the fourth region of the second reference image and the third region is greater than the similarity threshold, and the fourth image features of the fourth region satisfy the preset condition; processing the first image features in the first image based on the second image features, and processing the third image features in the first image based on the fourth image features to obtain the second image satisfying the preset condition.

[0018] In this application, the third region can be the face region of the second portrait to be processed mentioned in this application, where the face region at least includes a face region; the second face region can be the face region of the second portrait to be processed mentioned in this application; the third image features can be the image features in the face region of the second portrait to be processed mentioned in this application, where the third image features at least include the second face features; the second face features can be the face features of the second portrait to be processed mentioned in this application; and the fourth region can be the face region in the second reference image mentioned in this application; and the fourth image features can be the face features in the second reference image mentioned in this application.

[0019] In some implementations, the to-be-processed image can include a plurality of to-be-processed portraits satisfying the processing condition, for example, a first to-be-processed portrait and a second to-be-processed portrait. The electronic device can select a first reference image similar to the first image feature and overcoming the image defect from the gallery, and select a second reference image similar to the third image feature and overcoming the image defect, process the second image feature in the first image based on the second image feature of the first reference image, process the fourth image feature in the first image based on the fourth image feature in the second reference image, and obtain a target image meeting the preset condition.

[0020] In this way, when there are a plurality of to-be-processed portraits in the image, each to-be-processed portrait can still be processed by the image processing method mentioned in the present application to obtain a target image overcoming the image defect.

[0021] In a possible implementation of the first aspect, the method includes: detecting an image editing operation of a user on the first image, obtaining the to-be-processed first image, and displaying a second image editing interface corresponding to the first image, the second image editing interface including an image processing button; corresponding to the first image feature and the third image feature existing in the first image and the first reference image and the second reference image existing in the gallery, the image processing button is in a state capable of responding to a user operation instruction; corresponding to the first image feature and the third image feature not existing in the first image or the first reference image and the second reference image not existing in the gallery, the image processing button is in a state incapable of responding to the user operation instruction.

[0022] In the present application, the second image editing interface can be the image editing interface displayed by the electronic device when detecting the operation of the user clicking the image editing button. For example, the second image editing interface can be the editing interface 301 shown in (a) of FIG. 3.

[0023] In some implementations, when the electronic device detects that there is no to-be-processed portrait in the to-be-processed image or no reference image in the gallery, the image processing button is in a state incapable of responding to the user operation instruction. When there is a to-be-processed portrait in the to-be-processed image and there is a reference image in each stored image, the image processing button is in a state capable of responding to the user operation instruction. When the image processing button is in a state capable of responding to the user operation instruction, the electronic device can obtain a target image by performing image restoration on the corresponding to-be-processed portrait based on the reference image when detecting the operation of the user clicking the image processing button.

[0024] Therefore, compared with the image processing button being directly in a clickable state (for example, in a highlighted state), the phenomenon that the user clicks the image processing button without response can occur. The method of determining that there is a to-be-processed portrait and there is a reference image and then making the image processing button in a clickable state can make the electronic device detect the click operation of the user each time the user clicks the image processing button, so that the to-be-processed image is processed to obtain the target image, and the use experience of the user is improved.

[0025] In a possible implementation of the first aspect, in response to the image processing button being in a state capable of responding to a user operation instruction, the first image feature in the first image is processed based on the second image feature, and the third image feature in the first image is processed based on the fourth image feature to obtain the second image satisfying the preset condition, including: detecting that the user clicks the image processing button, displaying a third image editing interface corresponding to the first image, the third image editing interface including a first image identifier corresponding to the first image feature and a second image identifier corresponding to the third image feature; detecting that the user selects the first image identifier, processing the first image feature in the first image based on the second image feature; and detecting that the user selects the second image identifier, processing the third image feature in the first image based on the fourth image feature.

[0026] In this application, the third image editing interface can be an image editing interface displayed by the electronic device when detecting that the user clicks the image processing button. For example, the third image editing interface can be the editing interface 302 shown in (b) of FIG. 3; the first image identifier can be an image identifier corresponding to the first to-be-processed portrait mentioned in this application, for example, the first image identifier can be the image identifier 3021 corresponding to the first to-be-processed portrait 3012 shown in (b) of FIG. 3; and the second image identifier can be an image identifier corresponding to the second to-be-processed portrait mentioned in this application, for example, the second image identifier can be the image identifier 3022 corresponding to the second to-be-processed portrait 3013 shown in (b) of FIG. 3.

[0027] In some embodiments, the electronic device can display the image identifier corresponding to each to-be-processed portrait. Then, the to-be-processed portrait corresponding to the image identifier selected by the user is processed. In this way, when processing the to-be-processed image, the electronic device can process only part of the to-be-processed portraits based on the user's intention, so that the target image overcomes the image defects while meeting the user's needs.

[0028] In a possible implementation of the first aspect, the operation of detecting that the user selects the first image identifier, and processing the first image feature in the first image based on the second image feature, includes: detecting that the user selects the first image identifier, displaying a fourth image editing interface corresponding to the first image, the fourth image editing interface including at least one image identifier corresponding to at least one reference image corresponding to the first image feature, where the similarity between the at least one second region corresponding to the at least one reference image and the first region is greater than the similarity threshold, and the second image feature of the second region meets the preset condition; detecting that the user selects a third image identifier from the at least one image identifier, and processing the first image feature based on the second image feature in the first reference image corresponding to the third image identifier.

[0029] In this application, the fourth image editing interface can be the image editing interface displayed by the application when detecting that the user clicks the image identifier corresponding to the portrait to be processed. For example, the fourth image editing interface can be the editing interface 303 shown in (c) of FIG. 3; and the third image identifier can be the image identifier corresponding to the first reference image.

[0030] In some implementations, when determining the corresponding reference image for each portrait to be processed, the electronic device can also determine a plurality of reference images similar to the portrait to be processed and overcoming the image defects, and then select which reference image to process the portrait to be processed by the user. In this way, when processing the portrait to be processed, the electronic device can determine the reference image for processing the portrait to be processed based on the user's intention, so that the target image overcomes the image defects while meeting the user's demand.

[0031] In a possible implementation of the first aspect, the preset condition includes that the face score is greater than a score threshold; and the face score is determined based on at least one of face rotation angle data, face occlusion degree data, face smiling degree data, and eye opening amplitude data.

[0032] In some implementations, the presence of the first image feature in the first region that does not meet the preset condition can indicate that the portrait to be processed in the first region meets the image processing condition and can be processed. The preset condition can be set arbitrarily. For example, the preset condition can be that the eyes are open and the face is in a smiling state. Alternatively, the preset condition can be that the face score is higher than a score threshold, where the face score can be determined based on at least one of face rotation angle data, face occlusion degree data, face smiling degree data, and eye opening amplitude data. The application does not limit this.

[0033] In a possible implementation of the first aspect, the detection of the first image feature that does not satisfy the preset condition in the first region in the first image includes: detection of at least one face region in the first image; obtaining of at least one face score corresponding to the at least one face region; and, in a case where the face score corresponding to the first face region is less than or equal to a score threshold, detection of the first image feature that does not satisfy the preset condition in the first region in the first image.

[0034] In some implementations, the electronic device can first detect whether there is a face region in the image to be processed, for example, at least one face region is detected in the first image. Then, the electronic device can determine the face score corresponding to each face region. When the electronic device determines that the face score corresponding to the first face region is less than or equal to the score threshold, it means that the electronic device detects the first image feature that does not satisfy the preset condition in the first region.

[0035] In a possible implementation of the first aspect, the selection of the first reference image from the gallery includes: in the gallery, selection of N adjacent images adjacent to the first image and having a shooting time earlier than the shooting time of the first image, and selection of N adjacent images adjacent to the first image and having a shooting time later than the shooting time of the first image, to obtain a first image set, where N is a positive integer; from each image in the first image set, selection of an image having a similarity between an image feature vector and a feature vector of the first image greater than a similarity threshold, to obtain a second image set; from each image in the second image set, selection of an image having a similarity between the second region and the first region greater than the similarity threshold, to obtain a third image set; determination of a face score of the second region in each image in the third image set; and, from each image in the third image set, selection of an image having a maximum face score of the second region as the first reference image.

[0036] In this application, the first image set can be a set of images obtained by selecting N consecutive images forward and backward from the position of the image to be processed in the album; the second image set can be a set of images composed of images similar to the first image selected from the first image set; and the third image set can be a set of images composed of images having a face region similar to the face region of the image to be processed selected from the second image set.

[0037] In some implementations, the electronic device can first select N images (e.g., 5 images before and after) from the position of the to-be-processed image in the album to obtain a first image set. Next, the electronic device can calculate the similarity between the image feature vector of each image in the first image set and the image feature vector of the to-be-processed image, and select images with a similarity greater than a similarity threshold (e.g., 95%) to obtain a second image set. Next, the electronic device can traverse the images in the second image set, select images with a face region similar to the face region of the to-be-processed portrait, and obtain a third image set. Finally, the electronic device can perform face scoring on the face region similar to the to-be-processed portrait in each image in the third image set, and select the image with the largest face score as the reference image corresponding to the to-be-processed portrait. In this way, the reference image corresponding to the to-be-processed portrait can be selected from the gallery.

[0038] In a possible implementation of the first aspect, the first image feature in the first image is processed based on the second image feature to obtain the second image satisfying the preset condition, including: in response to the face occlusion degree in the first region being less than a first occlusion threshold and greater than a second occlusion threshold, the head image feature in the first image feature is replaced with the head image feature in the second image feature to obtain the second image, the first occlusion threshold being greater than the second occlusion threshold; in response to the face occlusion degree in the first region being less than the second occlusion threshold and the face rotation angle being greater than an angle threshold, the head image feature in the first image feature is replaced with the head image feature in the second image feature to obtain the second image; and in response to the face occlusion degree in the first region being less than the second occlusion threshold and the face rotation angle being less than the angle threshold, at least part of the facial feature in the first image feature that does not satisfy the preset condition is replaced with at least part of the facial feature in the second image feature to obtain the second image.

[0039] In this application, the first occlusion threshold can be the maximum occlusion threshold mentioned in this application; the second occlusion threshold can be the minimum occlusion threshold mentioned in this application.

[0040] In some implementations, the repair strategy for the to-be-processed portrait can include replacing the head image feature and replacing the facial feature. When the electronic device determines that the face occlusion degree of the to-be-processed portrait is large or the face rotation angle is large, the repair strategy of replacing the head image feature can be performed; when the electronic device determines that the face occlusion degree of the to-be-processed portrait is small and the face rotation angle is small, the repair strategy of replacing part of the facial feature can be performed. In this way, the portrait repair for the to-be-processed portrait can be completed, and the aesthetic appearance of the image can be improved.

[0041] In a possible implementation of the first aspect, the replacing the head image feature in the first image feature with the head image feature in the second image feature to obtain the second image comprises: replacing the head image feature in the first image feature with the head image feature in the second image feature to obtain a third image, and adjusting one or more of a texture feature, a color feature, and a brightness feature in the head image feature in the third image to obtain the second image satisfying the preset condition.

[0042] In the present application, the third image can be an image obtained by replacing the head image feature mentioned in the present application.

[0043] In some implementations, the electronic device can further adjust one or more details of the texture feature, the color feature, and the brightness feature in the third image, so as to improve the overall harmony and visual effect of the target image.

[0044] In a possible implementation of the first aspect, the replacing the at least part of the facial feature in the first image feature that does not satisfy the preset condition with the corresponding at least part of the facial feature in the second image feature to obtain the second image comprises: replacing the at least part of the facial feature in the first image feature that does not satisfy the preset condition with the corresponding at least part of the facial feature in the second image feature to obtain a fourth image, and adjusting one or more of a texture feature, a color feature, and a brightness feature in the facial feature in the fourth image to obtain the second image satisfying the preset condition.

[0045] In the present application, the fourth image feature can be an image obtained by replacing the facial image feature mentioned in the present application.

[0046] In some implementations, the electronic device can further adjust one or more details of the texture feature, the color feature, and the brightness feature in the fourth image, so as to improve the overall harmony and visual effect of the target image.

[0047] In a second aspect, the present application provides an electronic device, comprising: a memory and a processor, the memory is used to store instructions executed by one or more processors of the electronic device, and the processor is one of the one or more processors of the electronic device and is used to execute the image processing method mentioned in the present application.

[0048] In a third aspect, the present application provides a readable storage medium, and the readable storage medium stores instructions, and the instructions are executed on the electronic device to make the electronic device execute the image processing method mentioned in the present application.

[0049] In a fourth aspect, the present application provides a computer program product, comprising computer instructions, which, when executed on an electronic device, cause the electronic device to perform the image processing method mentioned in the present application.

[0050] The advantages of the second aspect to the fourth aspect above can be referred to the description of the first aspect and various possible implementations of the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0051] FIG. 1 shows a schematic diagram of a scenario in which the user blinks and causes the image quality to decrease, according to some embodiments of the present application;

[0052] FIG. 2 shows a schematic diagram of a first interface for image processing of a to-be-processed image, according to some embodiments of the present application;

[0053] FIG. 3 shows a schematic diagram of a second interface for image processing of a to-be-processed image, according to some embodiments of the present application;

[0054] FIG. 4 shows a schematic diagram of a system structure for an image processing method, according to some embodiments of the present application;

[0055] FIG. 5 shows a schematic diagram of a flow of an image processing method, according to some embodiments of the present application;

[0056] FIG. 6A shows a schematic diagram of a flow of displaying an image beautification entry, according to some embodiments of the present application;

[0057] FIG. 6B shows a schematic diagram of interaction between various modules when displaying an image beautification entry, according to some embodiments of the present application;

[0058] FIG. 7A shows a schematic diagram of a flow of directly processing a to-be-processed image based on a reference image, according to some embodiments of the present application;

[0059] FIG. 7B shows a schematic diagram of interaction between various modules when directly processing a to-be-processed image based on a reference image, according to some embodiments of the present application;

[0060] FIG. 8A shows a schematic diagram of a flow of processing a to-be-processed image based on a reference image selected by a user, according to some embodiments of the present application;

[0061] FIG. 8B shows a schematic diagram of a flow of processing a to-be-processed image based on a reference image selected by a user, according to some embodiments of the present application;

[0062] FIG. 8C shows a schematic diagram of interaction between various modules when processing a to-be-processed image based on a reference image selected by a user, according to some embodiments of the present application;

[0063] FIG. 9 shows a schematic diagram of a hardware structure of an electronic device according to some embodiments of the present application. DETAILED DESCRIPTION

[0064] Illustrative embodiments of the present application include, but are not limited to, an image processing method, an electronic device, a storage medium, and a program product.

[0065] The following provides a brief description of the background of the image processing method provided by the embodiments of the present application.

[0066] As described above, when a user takes a photo, the user usually blinks or frowns, which results in insufficient overall aesthetic quality of the captured image and affects the user's visual experience. For example, as shown in FIG. 1, when a user A takes a photo with other users, if the user A blinks when the camera captures the image, the image of the user A in the captured image 101 will appear to have closed eyes, which affects the overall aesthetic quality of the image 101.

[0067] To solve the above problem, the present application provides an image processing method. In the method, if there is a to-be-processed target (for example, a person image with closed eyes) in a to-be-processed image that meets a processing condition (for example, has a feature that affects the aesthetic quality of the image, such as closed eyes), an electronic device can obtain an image feature (for example, a face feature) of the to-be-processed target, and select a reference image (for example, a reference image with the same face as the to-be-processed person image and open eyes) from each image stored in the electronic device, which has a high similarity to the image feature of the to-be-processed target and remedies the above image defects. And based on the reference image, the to-be-processed image is image repaired to obtain a target image that overcomes the above image defects. The image repair method can be to replace the part of the image in the to-be-processed image that affects the aesthetic quality of the image with the corresponding part of the image in the reference image. For example, for the to-be-processed target being the above-mentioned person image with closed eyes, the eyes of the to-be-processed person image can be removed and replaced with the eyes of the reference image, thereby obtaining the target image. Or based on the features of the non-eye part of the to-be-processed person image and the features of the eye part of the reference image, a new open-eyed person image can be generated, and the generated open-eyed person image can be used to replace the initial closed-eyed person image in the to-be-processed image, thereby obtaining the target image. In this way, the overall aesthetic quality of the image can be improved, which is conducive to improving the user's visual experience when viewing the target image.

[0068] In some embodiments, after the electronic device receives a user's instruction to capture an image, the captured image can be directly used as a to-be-processed image. By performing the above image processing method on the captured to-be-processed image, a target image overcoming image defects can be directly displayed. For example, after the electronic device captures the to-be-processed image, it can first determine whether there is a to-be-processed target in the to-be-processed image that meets the processing condition. If there is no to-be-processed target, the electronic device can directly display the captured to-be-processed image to the user. If there is a to-be-processed target, the electronic device can select a reference image from the stored images that has high similarity to the image features (e.g., facial features) of the to-be-processed target and overcomes the above image defects, and perform image inpainting on the to-be-processed image based on the reference image to obtain a target image. In this way, when a user captures an image through an electronic device, the electronic device can directly perform image processing on the captured image to generate and display a target image overcoming image defects.

[0069] In other embodiments, a user can view a certain image stored in the electronic device, and during the user's viewing of the image, if the electronic device detects an image processing instruction, for example, detects a user's operation of clicking an image editing button corresponding to the image, the electronic device can use the image viewed by the user as a to-be-processed image, and perform the above image processing method on the to-be-processed image to generate and display a target image overcoming image defects. For example, after the electronic device receives an image processing instruction, it can first determine whether there is a to-be-processed target in the to-be-processed image that meets the processing condition. If there is a to-be-processed target, the electronic device further determines whether there is a reference image in the stored images that has high similarity to the image features (e.g., facial features) of the to-be-processed target and overcomes the above image defects. If there is no to-be-processed target, or there is a to-be-processed target but no reference image, the user is prompted that the to-be-processed image cannot be processed. In the case where there is a to-be-processed target in the to-be-processed image and there is a reference image in the stored images, the electronic device can perform image inpainting on the to-be-processed image based on the reference image to obtain a target image. In this way, when the electronic device detects a user's image processing instruction for a to-be-processed image, it can directly perform image processing on the to-be-processed image to generate and display a target image overcoming image defects.

[0070] In other embodiments, during the process of a user viewing an image to be processed (e.g., viewing an image to be processed in a photo album), when the electronic device detects an image processing instruction, such as detecting a user clicking the image's edit button, it can first determine whether there is a target image in the image to be processed that meets the processing conditions. If a target image exists, it further determines whether there is a reference image among the images stored by the electronic device that has a high similarity to the image features (e.g., facial features) of the target image and compensates for the aforementioned image defects. If there is no target image, or if there is a target image but no reference image, the image processing button may be in a state where it cannot respond to user operation instructions. If there is a target image in the image to be processed and a reference image exists among the stored images, the image processing button is in a state where it can respond to user operation instructions. When the image processing button is in a state where it can respond to user operation instructions, and the electronic device detects a user clicking the image processing button, the electronic device can perform image repair on the image to be processed based on the reference image to obtain the target image. Thus, compared to the image processing button being directly in a clickable state that can respond to user operation commands (e.g., in a highlighted state), which may result in no response when the user clicks the image processing button, the above method, which makes the image processing button clickable only after determining that there is a target to be processed and a reference image, ensures that the electronic device can detect the user's click operation every time the user clicks the image processing button, thereby processing the image to be processed to obtain the target image and improving the user experience.

[0071] Therefore, it is evident that in this application, the timing of executing the image processing method on the image to be processed is not limited. To facilitate the description of the image processing method provided in this application, the following embodiments will use the example of setting the image processing button to a state responsive to user operation commands only after determining that a target to be processed and a reference image exist.

[0072] Furthermore, in some embodiments, the image to be processed can be any image, such as images of people, landscapes, or animals. This application does not limit the image content of the image to be processed.

[0073] In other embodiments, the target to be processed can be any object in the image that meets the processing conditions. For example, the target to be processed can be any object in the image, such as a person, animal, or plant, and this application does not limit this.

[0074] In order to better introduce the image processing method provided in the present application, in the following embodiments, the image to be processed is taken as a portrait image, the target to be processed is taken as a portrait to be processed in the portrait image, and the description of the portrait in the reference image corresponding to the target to be processed is taken as a reference portrait, and the image processing method provided in the present application is introduced.

[0075] As described above, if there is no portrait to be processed in the image to be processed, or if there is a portrait to be processed but there is no reference image in each image stored by the electronic device, the electronic device will not display the image processing entry of the image to be processed. For example, the image processing button corresponding to the image processing entry is in a state of being unable to respond to the user operation instruction (for example, in a gray state). When the electronic device determines that there is a portrait to be processed in the image to be processed, and there is a reference image in each image stored by the electronic device, the electronic device will display the image processing entry. For example, the image processing button corresponding to the image processing entry is in a state of being able to respond to the user operation instruction (for example, in a highlighted state).

[0076] As an example, the image processing method provided in the present application is briefly introduced below in combination with the interface schematic diagram shown in FIG. 2, taking the portrait to be processed as a closed-eye portrait in a group photo and the reference portrait as an open-eye portrait as an example. In FIG. 2, the image processing button mentioned above can be described as an "image beautification button".

[0077] As shown in (a) of FIG. 2, the user can view the image 2011 in the album in the display interface 201 of the mobile phone 10. The display interface 201 can include an edit button 2012, a share button 2013, a delete button 2014 and the like, so that the user can perform various operations on the image 2011. For example, the user can send the image 2011 to others by clicking the share button 2013, and delete the image 2011 by clicking the delete button 2014. In addition, when the mobile phone 10 detects that the user clicks the edit button 2012, the mobile phone 10 displays the edit interface 202, as shown in (b) of FIG. 2, so that the user can adjust the filter and size of the image 2011 through the multiple editing controls in the edit interface 202.

[0078] When the mobile phone 10 detects that the user clicks the edit button 2012, the mobile phone 10 triggers the operation of detecting whether there is a to-be-processed portrait (e.g., a portrait with closed eyes) in the image 2011 that meets the processing condition. When the mobile phone 10 detects that there is a to-be-processed portrait 2021 in the image 2011, the mobile phone 10 further determines whether there is a reference portrait in the photo album that has the same face as the to-be-processed portrait 2021 and overcomes the image defect (e.g., open eyes). If there is no to-be-processed portrait or there is a to-be-processed portrait (e.g., the closed-eye portrait 2021) but there is no reference portrait in the photo album, the image beautification button 2022 in the edit interface 202 is in a state of being unable to respond to the user's operation instruction (e.g., in a gray state). If there is a to-be-processed portrait (e.g., the closed-eye portrait 2021) and there is a reference portrait in the photo album, the image beautification button 2022 in the edit interface 202 is in a state of being able to respond to the user's operation instruction (e.g., in a highlighted state). In addition, the edit interface 202 shown in FIG. 2(b) can include the image beautification button 2022, the filter button 2023, the cropping button 2024, and the like, so that the user can perform various editing operations on the image 2011. For example, the user can enter a filter adjustment interface by clicking the filter button 2023 to adjust the filter (e.g., a yellow filter, a gray filter, or the like) of the image 2011; the user can cause the mobile phone 10 to perform the image processing method of the present application on the image 2011 by clicking the image beautification button 2022; and the user can also crop the image 2011 by clicking the cropping button 2024.

[0079] When the image beautification button 2022 is in a state of being able to respond to the user's operation instruction (e.g., in a highlighted state), when the mobile phone 10 detects that the user clicks the image beautification button 2022, the mobile phone 10 triggers the image repair of the corresponding image feature part in the to-be-processed portrait 2021 based on the image features of the determined reference portrait, so as to obtain a target image that overcomes the image defect. For example, the eyes of the to-be-processed portrait 2021 are replaced with the eyes in the reference portrait. After the mobile phone 10 completes the image processing on the image 2011, the mobile phone 10 displays a display interface 203 as shown in FIG. 2(c). In the display interface 203, the target image 2031 obtained after the image processing on the to-be-processed image 2011 is displayed, and each portrait in the target image 2031 is in an open-eye state. In addition, a prompt box 2032 can be displayed in the display interface 203 shown in FIG. 2(c), prompting the user that the image beautification is completed. In addition, when the mobile phone 10 detects that the user clicks the complete button 2033, the mobile phone 10 can save the processed target image 2031 and exit the display interface 203; or when the mobile phone 10 detects that the user clicks the cancel button 2034, the mobile phone 10 does not save the processed image 2031 and directly exits the display interface 203.

[0080] In some embodiments, the image to be processed can further include a plurality of portraits to be processed. For example, the first region in the image to be processed is detected to include a first portrait to be processed that does not satisfy the preset condition, and the third region in the image to be processed is detected to include a second portrait to be processed that does not satisfy the preset condition. In this case, the electronic device can obtain a first image feature of a face region of the first portrait to be processed and a second image feature of a face region of the second portrait to be processed. In addition, the electronic device can select, from the stored images, a first reference image that has a high similarity between at least a partial region and the face region of the first portrait to be processed and satisfies the preset condition, and a second reference image that has a high similarity between at least a partial region and the face region of the second portrait to be processed and satisfies the preset condition. Finally, the electronic device can process corresponding partial image features of the first portrait to be processed based on partial image features of the first reference image, and process corresponding partial image features of the second portrait to be processed based on partial image features of the second reference image, so as to obtain a target image that satisfies the preset condition.

[0081] In some embodiments, when the corresponding reference portrait is determined for each portrait to be processed, the electronic device can further determine a plurality of reference portraits that are similar to the portrait to be processed and overcome the image defect, and then select, by the user, which reference portrait to be used to process the portrait to be processed.

[0082] As an example, the image processing method provided by the present application is briefly introduced below by taking the portrait to be processed as a closed-eye and non-smiling portrait in a group photo and the reference portrait as a smiling portrait with eyes open, in combination with the interface schematic diagram shown in FIG. 3. In FIG. 3, the image processing button mentioned above can still be described as an “image beautification button”.

[0083] The user can view the image 3011 in the album through the mobile phone 10, and when the user views the image 3011, if the mobile phone 10 detects that the user clicks the image editing button (for details, refer to the description of (a) in FIG. 2 above), the mobile phone 10 displays an editing interface 301 as shown in (a) of FIG. 3. The editing interface 301 can include an image beautification button 3014, a filter button 3015, a cropping button 3016, and a plurality of editing controls, so that the user can perform various editing operations on the image 3011. For example, the user can enter the editing interface by clicking the image beautification button 3014 to perform image beautification on the image 3011 by using the image processing method mentioned in the present application; the user can enter the filter adjustment interface by clicking the filter button 3015 to adjust the filter (such as yellow filter, gray filter, etc.) of the image 3011; and the user can also cut the image 3011 by clicking the cropping button 3016.

[0084] When the mobile phone 10 detects that the user clicks the edit button, the mobile phone 10 detects whether there is a to-be-processed portrait in the to-be-processed image 3011 and whether there is a reference portrait corresponding to the to-be-processed portrait in the gallery. When the mobile phone 10 detects that there is a to-be-processed portrait, for example, the first to-be-processed portrait 3012 (closed-eye portrait) and the second to-be-processed portrait 3013 (no-smile portrait), and determines that there is a first reference portrait corresponding to the first to-be-processed portrait 3012 and a second reference portrait corresponding to the second to-be-processed portrait 3013 in the gallery, the image beautification button 3014 in the edit interface 301 is in a state capable of responding to the user's operation instruction (for example, in a highlighted state). When the mobile phone 10 detects that there is no to-be-processed portrait in the to-be-processed image 3011 or no reference image in the gallery, for example, there is no first to-be-processed portrait and no second to-be-processed portrait, or there is no first reference portrait and no second reference portrait in the gallery, the image beautification button 3014 in the edit interface 301 is in a state incapable of responding to the user's operation instruction (for example, in a gray state).

[0085] When the image beautification button 3012 is in a state capable of responding to the user's operation instruction, when the mobile phone 10 detects that the user clicks the image beautification button 3014, as shown in (b) of FIG. 3, the mobile phone 10 displays an edit interface 302. In the edit interface 302, an image identifier corresponding to the to-be-processed portrait is displayed. For example, the first image identifier 3021 corresponding to the first to-be-processed portrait 3012 and the second image identifier 3022 corresponding to the second to-be-processed portrait 3013 are displayed in the edit interface 302, so that the user can process the to-be-processed portrait by clicking the image identifier. For example, when the mobile phone 10 detects that the user selects the first image identifier 3021, the mobile phone 10 can process part of the image features corresponding to the first to-be-processed portrait 3012 based on part of the image features in the first reference image. Alternatively, when the mobile phone 10 detects that the user selects the second image identifier 3022, the mobile phone 10 can process part of the image features corresponding to the second to-be-processed portrait 3013 based on part of the image features in the second reference image.

[0086] In some embodiments, when the mobile phone 10 detects that the user clicks the first image identifier 3021, the mobile phone 10 can also display an editing interface 303 as shown in (c) of FIG. 3. In the editing interface 303, there are displayed image identifiers corresponding to one or more reference images corresponding to the first to-be-processed image 3012 of the first image identifier 3021, and the face regions of each reference image are the same as the face region of the first to-be-processed image 3012 and are smiling. The user can select an image identifier from the one or more image identifiers, so that the mobile phone 10 can process the first to-be-processed image 3012 based on the reference image corresponding to the selected image identifier. For example, if the mobile phone 10 detects that the user clicks the image identifier 3031, the mobile phone 10 can process the first to-be-processed image 3012 based on the reference image corresponding to the image identifier 3031 selected by the user. After the mobile phone 10 processes the first to-be-processed image 3012, the mobile phone 10 displays an editing interface 304 as shown in (d) of FIG. 3. In the editing interface 304, there is displayed a target image 3041 obtained by processing the first to-be-processed image 3012 in the image 3011.

[0087] Similarly, in the editing interface 304, the user can also click the second image identifier 3022 to process the second to-be-processed image 3013 corresponding to the second image identifier 3022, and the mobile phone 10 can display image identifiers of one or more reference images corresponding to the second to-be-processed image 3013, and the user can select a reference image based on which to process the to-be-processed image 3013. Details are not described herein. In addition, when the mobile phone 10 detects that the user clicks the complete button 3042, the mobile phone 10 can save the processed target image 3041 and exit the editing interface 304; or when the mobile phone 10 detects that the user clicks the cancel button 3043, the mobile phone 10 does not save the processed target image 3041 and directly exits the editing interface 304.

[0088] In this way, the electronic device can process a to-be-processed image by the image processing method provided in the present application to beautify the image, so that the final image obtained can overcome image defects and improve the overall visual effect of the image.

[0089] Before describing the image processing method provided in the embodiments of the present application in detail, first, the algorithm modules in the system structure used by the image processing method provided in the present application are briefly described in conjunction with the system structure schematic diagram shown in FIG. 4.

[0090] The system structure applicable to the image processing method provided in the application can include a cloud-side device and an end-side device. The cloud-side device can be a physical server or a server cluster, used to deploy related algorithms, codes, parameters, and other data of the image processing method provided in the application. The end-side device can be any electronic device such as a mobile phone, a tablet, a notebook, a computer, etc., used to store an image to be processed. The end-side device can send the image to be processed and an image processing request to the cloud-side device, the cloud-side device can perform image processing on the received image to be processed, and send a target image obtained after processing to the end-side device for display.

[0091] As shown in FIG. 4, the end-side device can include a media information library, a gallery, a visual middle platform software development kit (SDK), and a visual middle platform Android application package (APK), and the cloud-side device can include a smart cloud-micro service module, an algorithm service module, a cloud service module, and an object storage service (OBS) module. TM The application package (APK) of the cloud-side device can include a smart cloud-micro service module, an algorithm service module, a cloud service module, and an object storage service (OBS) module. Through the interaction between the modules in the end-side device and the cloud-side device, the image processing method provided in the application can be implemented.

[0092] The gallery in the end-side device can be an image database such as a photo album in the electronic device (e.g., a mobile phone 10) mentioned above, and the image to be processed is an image in the gallery. The end-side device can send an image processing request to the cloud-side device to perform data analysis and image beautification on the image to be processed.

[0093] The content of data analysis can include but is not limited to human body information detection for detecting whether there is a human image in the image to be processed, face information detection for detecting the state of the human image, determination of whether there is a human image to be processed based on the state of the human image, and label classification of the human image to be processed to determine whether the human image to be processed is eyes closed or no smile, image feature extraction of the human image to be processed and other images in the gallery, and clustering processing for judging whether there is a reference human image with the same appearance as the human image to be processed in the gallery. In addition, the process of image beautification can include but is not limited to similar picture screening for preliminarily screening images with high similarity to the image features of the human image to be processed, face comparison for matching each human image to be processed and the corresponding reference human image based on face features when there are multiple human images to be processed and multiple human images in the screened similar pictures, face / head replacement processing for replacing the face, head, or single feature in the human image to be processed with the face, head, or single feature in the reference human image, and face enhancement for adjusting the texture and other details of the face of the human image to be processed after face replacement.

[0094] As shown in FIG. 4, the end-side device can further include a media information library for storing face-related data or for storing image-related data. The data stored in the media information library can be data obtained by processing a to-be-processed image by using an image processing algorithm. When the same processing procedure is performed on the same to-be-processed image, the data stored in the media information library can be directly called, and the image processing algorithm does not need to be run again. For example, if data B1 is obtained by processing a to-be-processed image B by using an algorithm A, the data B1 can be stored in the media database. When the to-be-processed image B needs to be processed in the same way, the data B1 stored in the media database can be directly called, and the algorithm A does not need to be run again.

[0095] It can be understood that, in order to apply the image processing method provided in the present application to the end-side device, the relevant program code package (visual platform SDK and visual platform APK) needs to be installed in the end-side device. When a user needs to process a to-be-processed image, the end-side device can run the code in the relevant program code package. In the process of running the code in the relevant program code package, the end-side device can call the cloud computing resource in the cloud-side device, and process the to-be-processed image by using the algorithm, parameter and other data in the cloud-side device.

[0096] The visual platform SDK can include a plurality of application programming interfaces (APIs) to realize the connection between the gallery and the visual platform APK. For example, the visual platform SDK can include a repair strategy judgment API, a face information detection API, an image feature extraction API, a portrait generation API, a face generation (FaceGenerate) plug-in, a face enhancement API, and the like. By using the algorithm interface provided by each API, the corresponding algorithm in the visual platform APK can be called.

[0097] The visual middle platform APK can include a face detection APK, a human body detection APK, a portrait generation APK, and a portrait enhancement APK. The face detection APK can include a face frame detection APK for detecting whether a face exists, a face information detection APK for detecting attributes such as face feature positions, an image feature extraction APK for extracting portrait features, and a face clustering APK for determining a reference portrait that is the same as a face of a portrait to be processed. The human body detection APK can include a human body key point detection APK for detecting human body key points, and a face-human body mapping APK for matching a face and a corresponding human body. The portrait generation APK can include a repair strategy judgment APK and a portrait generation APK. The portrait enhancement APK can include a face quality detection APK to determine whether a generated target portrait overcomes image defects, a face analysis APK for extracting face feature points, and a face enhancement APK for adjusting details such as textures of a face. Among them, when each algorithm in the visual middle platform APK receives a call request sent by a corresponding API, the visual middle platform APK can send an image processing request to a cloud computing management platform (such as the smart cloud-micro service module in FIG. 4) to implement image processing by using cloud computing resources and reduce resource consumption of an end-side device.

[0098] It can be understood that the related algorithms of the image processing method provided in the present application can be run and deployed by using cloud computing resources to improve data processing efficiency. As shown in FIG. 4, the smart cloud-micro service module in the cloud-side device can be a cloud computing management platform for interacting with the end-side device, and can also call related algorithms in the algorithm service module to process the image to be processed. Among them, the algorithm service module can include a repair strategy judgment algorithm and a portrait repair algorithm. The repair strategy judgment algorithm can implement the following functions: judging a face rotation angle, judging whether a face is blocked. The portrait repair algorithm can implement the following functions: generating a processed portrait, adjusting details such as textures of a processed face to achieve face enhancement, and processing color differences in an image.

[0099] The intelligent cloud-micro service module also has a plurality of APIs, and the APIs provide an operation interface for calling each service algorithm in the algorithm service module. For example, when the intelligent cloud-micro service module receives the to-be-processed image and the image processing request sent by the terminal-side device, the intelligent cloud-micro service module can run a start service API, indicating that the algorithm in the algorithm service module is started to be called. Then, the intelligent cloud-micro service module can determine the repair strategy by calling the repair strategy judgment algorithm in the algorithm service module through the algorithm interface provided by the repair strategy API and store the repair strategy, for example, it can be determined that the to-be-processed portrait in the to-be-processed image needs to replace a certain facial feature or replace the whole head. The intelligent cloud-micro service module can also call the portrait repair algorithm in the algorithm service module through the algorithm interface provided by the portrait repair API, and repair the to-be-processed portrait based on the repair strategy determined above.

[0100] After the portrait is repaired based on the repair strategy, the repaired portrait can also be stored in the OBS module of the cloud-side module, so as to directly call the portrait picture, portrait information and other data in the OBS module for processing in the subsequent process, or return the data in the OBS module to the gallery in the terminal-side device for display.

[0101] When the intelligent cloud-micro service module runs the end service API, the stored data can be deleted, and the execution of the image processing method provided in the present application is ended.

[0102] In addition, the intelligent cloud-micro service module can also call the risk control algorithm in the cloud service module to realize the risk control processing of the image data, so as to determine whether the image data is safe or whether there is a risk of leakage.

[0103] In this way, through the interaction between the above-mentioned modules and the running of the codes of the modules, the image processing method provided in the present application can be realized, the to-be-processed image can be beautified, and the final image obtained can overcome the image defects, thereby improving the overall visual effect of the image.

[0104] It can be understood that the image processing method provided in the embodiments of the present application can be applied to electronic devices or three-party applications such as beauty photo albums. The electronic device to which the image processing method is applied can be any terminal device such as a mobile phone, a computer, a tablet computer, a notebook computer, a netbook, an augmented reality (AR) device, a virtual reality (VR) device, a wearable device, a vehicle-mounted intelligent terminal, and the like. The electronic device can also be a physical server or a cloud device, such as an X86 server, an ARM server, and the like; or a virtual machine (VM) based on a general-purpose physical server combined with network function virtualization (NFV) technology, which refers to a complete computer system that is simulated by software, runs in a completely isolated environment, and has complete hardware system functions.

[0105] The image processing method provided in the embodiments of the present application will be briefly introduced below based on the flowchart shown in FIG. 5. The image processing method can be applied to an electronic device, such as the mobile phone 10 shown in FIG. 1. As shown in FIG. 5, specifically, the method includes the following steps:

[0106] S501: Obtain a first image to be processed.

[0107] In the present application, the first image can be the image to be processed mentioned in the present application.

[0108] In some embodiments, when the electronic device receives a shooting instruction of a user to shoot an image, the electronic device can directly take the shot image as the first image to be processed, and then perform the subsequent steps S502 to S504 on the first image, so as to directly display the target image overcoming the image defect.

[0109] In other embodiments, the user can view a certain image stored in the electronic device. During the process of viewing the image, if the electronic device receives an image processing instruction, for example, detects that the user clicks an image editing button, the electronic device can take the image viewed by the user as the first image to be processed, and then perform the subsequent steps S502 to S504 on the first image, so as to generate and display the target image overcoming the image defect.

[0110] S502: Detect that there is a first image feature in the first region of the first image that does not satisfy a preset condition, wherein the first region includes a first face region, and the first image feature includes a first face feature.

[0111] In some embodiments, the first region can be a region where a face of the first to-be-processed portrait mentioned in the present application is located, wherein the region where the face is located at least includes a face region; the first face region can be a face region of the first to-be-processed portrait mentioned in the present application; the first image feature can be an image feature in the region where the face of the first to-be-processed portrait mentioned in the present application is located, wherein the first image feature at least includes a first face feature; and the first face feature can be a face feature of the first to-be-processed portrait mentioned in the present application, such as an eye contour, a face contour, and the like.

[0112] In some embodiments, the presence of the first image feature in the first region that does not meet the preset condition can indicate that the to-be-processed portrait in the first region meets the image processing condition, and the to-be-processed portrait can be subjected to image processing. The preset condition can be set arbitrarily. For example, the preset condition can be that the eyes are open, the face is in a smiling state, or the like. Alternatively, the preset condition can be that the face score is higher than a score threshold, wherein the face score can be determined based on at least one of face rotation angle data, face occlusion degree data, face smiling degree data, and eye opening amplitude data.

[0113] In some embodiments, the electronic device can first detect whether there is a face region in the to-be-processed image, for example, at least one face region is detected in the first image, and then the electronic device can determine at least one face score corresponding to the at least one face region. When the electronic device determines that the face score corresponding to the first face region is less than or equal to the score threshold, it indicates that the electronic device detects that there is a first image feature in the first region that does not meet the preset condition.

[0114] S503: Select a first reference image from the gallery, wherein the similarity between the second region of the first reference image and the first region is greater than a similarity threshold, and the second image feature of the second region meets the preset condition.

[0115] In some embodiments, the first reference image can be a reference image corresponding to the first image feature mentioned in the present application; the second region can be a face region similar to the face region of the to-be-processed portrait in the first reference image mentioned in the present application; and the second image feature can be a face feature in the second region of the first reference image mentioned in the present application.

[0116] In some embodiments, after determining the to-be-processed portrait and the first image feature of the to-be-processed portrait through the above step S502, the electronic device can further acquire the second image feature (for example, the facial feature) of each image in the gallery, and select an image with a greater similarity to the first image feature and overcoming the image defect as the first reference image. For example, if the to-be-processed portrait is a closed-eye portrait, the electronic device can select a reference portrait with open eyes and the same appearance as the to-be-processed portrait from the other images, and then process the to-be-processed portrait based on the reference portrait. The similarity threshold can be set arbitrarily, and the present application does not limit this.

[0117] In some embodiments, after determining the to-be-processed portrait through the above step S502 and determining the reference image through the step S503, the electronic device can directly execute the subsequent step S504 to process the first image based on the reference image to obtain the second image overcoming the image defect.

[0118] In some embodiments, after determining the to-be-processed portrait through the above step S502 and determining the reference image through the step S503, the electronic device can display the image processing entry. When the electronic device detects that the user enters the image processing entry, for example, detects that the user clicks the image processing button corresponding to the image processing entry, the electronic device will execute the subsequent step S504 to process the first image based on the reference image to obtain the second image overcoming the image defect. If the electronic device does not detect that the user enters the image processing entry, for example, does not detect that the user clicks the image processing button corresponding to the image processing entry, the electronic device will not execute the subsequent step S504.

[0119] S504: processing the first image feature in the first image based on the second image feature to obtain a second image satisfying a preset condition.

[0120] In some embodiments, the second image can be the target image mentioned in the present application.

[0121] In some embodiments, the electronic device can perform image repair on the to-be-processed image based on the selected reference image to obtain the target image. For example, the part of the to-be-processed portrait that needs to be processed can be cut off, and then the same part of the reference image is copied and pasted to the part cut off in the to-be-processed image, and the paste edge position and texture, color difference, etc. are adjusted, so that the processed target image achieves the effect of overall harmony.

[0122] In this way, the electronic device can perform image beautification on the to-be-processed image through the image processing method provided by the present application, so that the final image obtained can overcome the image defect, thereby improving the overall visual effect of the image.

[0123] It can be understood that when the region affecting the image aesthetics in the image to be processed is the region of the portrait to be processed, the part to be processed can be the face (facial features and facial state, etc.), the head (face and hair, etc.), the limbs, etc. For ease of description, the following embodiments all take the face of the portrait to be processed as an example to introduce the image processing method provided by the present application in detail.

[0124] In some embodiments, as shown in (a) and (b) of FIG. 2, when implementing the image processing method provided by the present application, the electronic device can first determine whether there is a reference portrait similar to the face of the portrait to be processed and overcoming the image defects in the gallery. If there is, the image processing button will be in a state capable of responding to the user's operation instruction; if there is not, the image processing button will be in a state incapable of responding to the user's operation instruction.

[0125] For example, when the electronic device detects that the user edits the image to be processed (such as a group photo), the electronic device can first select N images (such as 5 images before and after) from the position of the image to be processed in the album to obtain a first image set. Next, the electronic device can calculate the similarity between the image feature vector of each image in the first image set and the image feature vector of the image to be processed, and select the images with a similarity greater than a similarity threshold (such as 95%) to obtain a second image set. Next, the electronic device can traverse the images in the second image set, select the images with a face region similar to the portrait to be processed, and obtain a third image set. Finally, the electronic device can perform face scoring on the face region similar to the portrait to be processed in each image in the third image set, and select the image with the largest face score as the reference image corresponding to the portrait to be processed. When the reference image does not exist, it means that there is no reference portrait with the same face as the portrait to be processed and overcoming the image defects before and after the image to be processed, and the image processing button is in a state incapable of responding to the user's operation instruction; if there is a reference image, it means that there is a reference portrait with the same face as the portrait to be processed and overcoming the image defects before and after the image to be processed, and the image processing button is in a state capable of responding to the user's operation instruction.

[0126] Specifically, referring to the flowchart shown in FIG. 6A, the execution process of the electronic device determining that the image processing button is in a state capable of responding to the user's operation instruction is briefly described. Specifically, the method comprises the following steps:

[0127] 601: detecting the user's operation of editing the image A.

[0128] In some embodiments, the image editing operation can be performed when the user views the image A. At this time, the electronic device can detect an operation of the user clicking the image editing button corresponding to the image A to perform the image processing method provided by the present application on the image A to be processed. For example, as shown in (a) of FIG. 2 described above, the mobile phone 10 can detect an operation of the user clicking the editing button 2012.

[0129] 602: Determine whether there is a face in the image A. If yes, go to step 603; if no, go to step 611.

[0130] In some embodiments, after detecting the operation of the user editing the image A, the electronic device can first call a face information detection algorithm to detect whether there is a face in the image A to be processed. For example, the electronic device can determine whether there is a face in the image A to be processed by determining whether there are eye contours, nose contours, and other image features in the image A to be processed. If there is no face, it is determined that the image processing button needs to be in a state of being unable to respond to the user's operation instruction, and the image A to be processed cannot be processed. If there is a face, the image feature F1 of the image A to be processed can be further extracted.

[0131] 603: Extract the image feature F1 in the image A.

[0132] In some embodiments, when there is a face in the image A to be processed, more detailed face image features F1 of the face region in the image A can be further extracted. For example, the image features such as eye shape, size, pupil color, etc. can be obtained.

[0133] 604: Obtain an image set B by selecting N images before and after the image A.

[0134] In some embodiments, the electronic device can continuously obtain N images before the position of the image A to be processed in the album, and continuously obtain N images after the position of the image A to be processed in the album, thereby obtaining the image set B. The number of images obtained can be set arbitrarily, and the present application does not limit this.

[0135] 605: Determine whether the image set B is empty. If yes, go to step 611; if no, go to step 606.

[0136] In some embodiments, the electronic device can further determine whether the image set B is empty after selecting N images before and after the image A to be processed. If the image set B is empty, it means that there is only the image A to be processed in the gallery, and the face in the image A to be processed cannot be processed by other images. Therefore, it is determined that the image processing button needs to be in a state of being unable to respond to the user's operation instruction. If the image set B is not empty, the electronic device can traverse each image in the image set B to select an image with a larger similarity to the image A to be processed through step 606.

[0137] 606: Traversing each image in the image set B to select an image with a larger similarity to the image A to be processed to obtain an image set C.

[0138] In some embodiments, since the images in the image set B selected in step 604 are not compared with the image A to be processed, there can be images in the image set B that are completely irrelevant to the image A to be processed. Therefore, the electronic device needs to further filter each image in the image set B to select an image with a larger similarity to the image A to be processed to obtain an image set C through steps 6061 to 6065.

[0139] In some embodiments, the filtering process of determining an image with a larger similarity to the image A to be processed in the image set B can refer to steps 6061 to 6065.

[0140] 6061: Obtaining the image features Fi of each image.

[0141] In some embodiments, the electronic device can obtain the image features of the face region of each image in the image set B, such as the image features of the eye shape, size, pupil color, etc.

[0142] 6062: Calculating the similarity between each image feature Fi and the image feature F1 of the image A.

[0143] In some embodiments, after obtaining the image feature Fi of a certain image in the image set B, the similarity between the image feature Fi and the image feature F1 can be calculated. The similarity can be determined by the Euclidean distance, cosine similarity, etc. The present application does not make any limitation.

[0144] 6063: Determining whether the similarity is greater than a similarity threshold. If not, go to step 6064; if yes, go to step 6065.

[0145] In some embodiments, the similarity threshold can be set arbitrarily, and the present application does not make any limitation.

[0146] 6064: Traversing the next image in the image set B.

[0147] In some embodiments, if it is determined through step 6063 that the similarity between the image feature Fi of the face region of the image and the image feature F1 of the face region of the image to be processed A is less than the similarity threshold, it indicates that the image is irrelevant to the image to be processed A, and the next image in the image set B is traversed. The electronic device can continue to perform steps 6061-6063 on the next image in the image set B.

[0148] 6065: Add the image to the image set C.

[0149] In some embodiments, if it is determined through step 6063 that the similarity between the image feature Fi of the face region of the image and the image feature F1 of the face region of the image to be processed A is greater than the similarity threshold, it indicates that the image is relevant to the image to be processed A, and it can be added to the image set C.

[0150] In some embodiments, the electronic device can repeatedly perform steps 6061-6065 to determine whether the similarity between the face region of each image in the image set B and the face region of the image A is greater than the similarity threshold, and add the images greater than the similarity threshold to the image set C, thereby obtaining the image set C.

[0151] 607: Determine whether the image set C is empty. If yes, perform step 611; if no, perform step 608.

[0152] In some embodiments, the electronic device can determine whether the image set C is empty. If it is empty, it indicates that there is no image similar to the image to be processed A in the image gallery, and the image processing button needs to be in a state that cannot respond to user operation instructions; if the image set C is not empty, each image in the image set C needs to be traversed to match each face in the image with each face in the image to be processed.

[0153] 608: Traverse each image in the image set C to match each face in the image set C with the image to be processed A.

[0154] In some embodiments, if there are multiple faces (such as face 11 and face 12) in the image A' in the image set C, and there are multiple faces (such as face 31 and face 32) in the image to be processed A, the electronic device needs to match the face to be processed in the image A with the face in the image A'. For example, the face to be processed 31 in the image to be processed A is the same as the face 11 in the image A', and the matched face image (face 11) can be added to the image set D.

[0155] The matching of each face in each image in the image set C with the face to be processed can refer to steps 6081 to 6083.

[0156] 6081: Face information extraction is performed on each face in the image.

[0157] In some embodiments, when there are multiple faces (e.g., face 11, face 12, etc.) in an image A' in the image set C, the electronic device needs to extract face information of the multiple faces respectively.

[0158] 6082: Matching is performed between each face and the face to be processed in the image A.

[0159] In some embodiments, the multiple faces (e.g., face 11, face 12, etc.) in the image A' can be matched with the face to be processed by using various clustering algorithms. The present application does not limit this. For example, k pixel points can be randomly selected from face 11 and face 12 respectively as clustering centers to form two clusters. Then, some pixel points of the face to be processed are distributed in the cluster where the nearest clustering center is located. Next, the mean value of all pixel point data in each cluster is calculated as the new clustering center of the cluster, and the remaining pixel points of the face to be processed are distributed in the cluster where the nearest clustering center is located. The above process is repeated, if each pixel point of the face to be processed falls in the cluster where face 11 is located, it indicates that face 11 and the face to be processed are the same person; if each pixel point of the face to be processed does not fall in the cluster where face 11 or face 12 is located, it indicates that face 11 and face 12 are not the same person as the face to be processed, i.e., there is no face in the image A' that matches the face to be processed.

[0160] 6083: The matched face image is added to the image set D.

[0161] In some embodiments, if face 11 in the image A' and the face to be processed are the same person, the face image corresponding to face 11 is added to the image set D.

[0162] In some embodiments, the electronic device can repeatedly perform the above steps 6081 to 6083 to match each face in each image in the image set C with the face to be processed in the image A.

[0163] 609: It is determined whether the image set D is empty. If not, step 610 is entered; if yes, step 611 is entered.

[0164] In some embodiments, the electronic device can determine whether the reference image set D is empty. If it is empty, it is determined that the image processing button needs to be in a state that cannot respond to user operation instructions; if the reference image set D is not empty, it is determined that the image processing button needs to be in a state that can respond to user operation instructions.

[0165] 610: Display the image beautification entry.

[0166] In some embodiments, when the electronic device displays the image beautification entry, the image processing button (e.g., the image beautification button 2022 shown in (b) of FIG. 2) can be in a clickable state that can respond to user operation instructions, such as a highlighted state.

[0167] 611: Do not display the image beautification entry.

[0168] In some embodiments, when the electronic device does not display the image beautification entry, the image processing button (e.g., the image beautification button 2022 shown in (b) of FIG. 2) can be in an unclickable state that cannot respond to user operation instructions, such as a grayed-out state.

[0169] In this way, when the electronic device detects that the user edits the to-be-processed image, the electronic device can determine whether to display the image beautification entry based on the above process. If the image beautification entry does not need to be displayed, it means that the to-be-processed image cannot be processed by the image processing method provided in the present application; if the image beautification entry needs to be displayed, it means that when the electronic device detects that the user clicks the button corresponding to the image beautification entry, the to-be-processed image can be processed by the image processing method provided in the present application.

[0170] The following describes the interaction process of displaying the image beautification entry based on the timing interaction flowchart shown in FIG. 6B in combination with the system structure diagram of the image processing method shown in FIG. 4. As shown in FIG. 6B, the interaction process of displaying the image beautification entry can be implemented through the gallery, the media information library, the visual platform SDK, the face information detection APK, the image feature extraction APK, the face clustering APK, and the interaction between users. Specifically, the interaction flowchart shown in FIG. 6B includes the following steps:

[0171] 620: The user performs an editing operation on the to-be-processed image in the gallery.

[0172] In some embodiments, the user can edit the to-be-processed image when viewing it. At this time, the electronic device can detect the user's editing operation on the to-be-processed image, thereby triggering the electronic device to perform the image processing method provided in the present application on the to-be-processed image. For example, referring to (a) of FIG. 2, when the user is viewing the to-be-processed image 2011, the electronic device detects that the user clicks the edit button 2012, and can display the edit interface 202 shown in (b) of FIG. 2.

[0173] 621: The media information library detects whether to store face data in the to-be-processed image.

[0174] In some embodiments, the media information library can be used to store data obtained by image processing related algorithms processing the to-be-processed image. For example, if the face data in the to-be-processed image A is extracted by the face information detection related algorithm, the extracted face data can be stored in the media information library. If the face data of the to-be-processed image A needs to be obtained again, the face data stored in the media information library can be directly read, avoiding the electronic device to run the face information detection related algorithm again, and saving the computing resources. Therefore, when the user edits the to-be-processed image, the electronic device can first detect whether the face data in the to-be-processed image is stored in the media information library. If the face data is stored, the face data can be directly used for subsequent processing; if the face data is not stored, steps 6231 to 6237 need to be performed to extract the face data in the to-be-processed image and store it.

[0175] 622: The media information library sends the face data to the gallery.

[0176] In some embodiments, if the face data in the to-be-processed image is pre-stored in the media information library, the detected face data can be sent to the gallery. Then step 624 is performed, and N images before and after the to-be-processed image are selected to determine whether there are reference images similar to the face data of the to-be-processed image and can make up for the defects of the to-be-processed image.

[0177] 6231: The gallery determines that the media information library does not store the face data.

[0178] In some embodiments, if the media information library does not send the face data of the to-be-processed image to the gallery, it can be determined that the media information library does not pre-store the face data of the to-be-processed image. In this way, the face data can be obtained through subsequent steps 6232 to 6237 and stored in the media information library.

[0179] 6232: The gallery does not exhibit the image beautification entry to the user.

[0180] In some embodiments, if it is determined that the media information library does not store the face data, the image beautification entry, for example, the image beautification button, can be temporarily not exhibited to the user in a gray state that cannot respond to the user's operation instruction. When it is determined through subsequent steps that the to-be-processed image has face data, and there are reference images similar to the face data of the to-be-processed image and can make up for the defects of the to-be-processed image before and after the to-be-processed image, the image beautification entry, for example, the image processing button, is exhibited in a highlighted state that can respond to the user's operation instruction.

[0181] 6233: The gallery calls the face information detection API in the visual middle platform SDK.

[0182] In some embodiments, the API is a data interface. When the gallery calls the face information detection API, the face information detection algorithm is called.

[0183] 6234: The visual platform SDK sends a start instruction to the face information detection APK.

[0184] In some embodiments, when the visual platform SDK detects that the gallery calls the face information detection API in the visual platform SDK, a start instruction is sent to the face information detection APK to run the face information detection algorithm to obtain face data.

[0185] 6235: The face information detection APK obtains face data.

[0186] In some embodiments, the electronic device can run the face information detection algorithm in the face information detection APK to obtain face data. For example, key point data of the eyes, mouth, etc. of the face; data such as face rotation angle. Among them, the feature data of the face can be obtained by any image processing algorithm. For example, the edge pixel data, color data, texture data, position and shape data of the five organs of the image can be extracted by a convolutional neural network. The present application does not make any limitation.

[0187] 6236: The face information detection APK returns the face data to the gallery.

[0188] In some embodiments, after the face information detection APK obtains the face data in the to-be-processed image, the face data can be sent to the gallery.

[0189] 6237: The gallery stores the face data to the media information library.

[0190] In some embodiments, after the gallery obtains the face data sent by the face information detection APK, it is stored in the media information library. If the face data of the to-be-processed image needs to be applied again, the stored face data can be directly read from the media information library, without the need to execute the above steps 6233 to 6236 again.

[0191] In some embodiments, in the above step 622, if the gallery receives the face data sent by the media information library, it does not need to execute the above steps 6231 to 6237, and can directly use the face data sent by the media information library for image processing; if the gallery does not receive the face data sent by the media information library, it needs to obtain the face data by the above steps 6231 to 6237 and store the face data in the media information library.

[0192] 624: The gallery selects N images before and after the to-be-processed image.

[0193] In some embodiments, the gallery can also acquire N images in front of the position of the to-be-processed image in the gallery in sequence; and acquire N images behind the position of the to-be-processed image in the gallery in sequence. Wherein, N is a positive integer, and the number of acquired images can be set arbitrarily, which is not limited in the present application.

[0194] 625: The media information library detects whether the feature vectors of the selected images are stored.

[0195] In some embodiments, the electronic device can first detect whether the feature vectors of the selected images are stored in the media information library. If the feature vectors are stored, the feature vectors can be directly used for subsequent processing; if the feature vectors are not stored, steps 6271 to 6276 need to be performed to extract and store the feature vectors of the selected images.

[0196] 626: The media information library sends the feature vectors to the gallery.

[0197] In some embodiments, if the feature vectors of the selected images are pre-stored in the media information library, the detected feature vectors can be sent to the gallery. Then step 628 is performed to calculate the similarity between the selected images and the to-be-processed image.

[0198] 6271: The gallery determines that the feature vectors are not stored in the media information library.

[0199] In some embodiments, if the media information library does not send the feature vectors to the gallery, the gallery can determine that the feature vectors are not pre-stored in the media information library. Then subsequent steps 6272 to 6276 need to be performed to acquire and store the feature vectors in the media information library.

[0200] 6272: The gallery calls the image feature extraction API in the visual platform SDK.

[0201] In some embodiments, the API is a data interface. When the gallery calls the image feature extraction API, the image feature extraction algorithm can be called.

[0202] 6273: The visual platform SDK sends a start instruction to the image feature extraction APK.

[0203] In some embodiments, when the visual platform SDK detects that the gallery calls the image feature extraction API in the visual platform SDK, a start instruction can be sent to the image feature extraction APK to run the image feature extraction algorithm to acquire the feature vectors of the selected images.

[0204] 6274: The image feature extraction APK acquires the feature vectors.

[0205] In some embodiments, the image feature vector is a mathematical representation for describing the image content. It usually contains information of color, texture, shape, etc. in the image. The feature vector of the selected image can be obtained by any image processing algorithm. For example, edge pixel data, color data, texture data, facial feature position and shape data, etc. of the image can be extracted by a convolutional neural network. The present application does not limit this.

[0206] 6275: The image feature extraction APK returns the feature vector to the gallery.

[0207] In some embodiments, after the image feature extraction APK obtains the feature vector of each selected image, the feature vector is sent to the gallery.

[0208] 6276: The gallery stores the feature vector in the media information library.

[0209] In some embodiments, after the gallery obtains the feature vector sent by the image feature extraction APK, it is stored in the media information library. If the feature vector of the image needs to be applied again, the stored feature vector can be directly read from the media information library, without the need to perform the above steps 6272 to 6275 again.

[0210] In some embodiments, in the above step 626, if the gallery receives the feature vector sent by the media information library, it does not need to perform the above steps 6271 to 6276, and can directly use the feature vector for image processing; if the gallery does not receive the feature vector sent by the media information library, it needs to obtain the feature vector and store it in the media information library through the above steps 6271 to 6276.

[0211] 628: The gallery calculates the similarity between the selected image and the image to be processed.

[0212] In some embodiments, the electronic device can calculate the similarity between the selected image and the image to be processed by any method. For example, the distance between the feature vectors of the selected image and the image to be processed can be obtained by Euclidean distance, cosine similarity, etc. The present application does not limit this.

[0213] 629: The gallery adds the image with high similarity to the image set.

[0214] In some embodiments, if the similarity is lower than the similarity threshold, it means that the face in the selected image and the face in the image to be processed are not the same person, and the image to be processed cannot be processed by the selected image, and the next image needs to be traversed.

[0215] In some embodiments, if the similarity is higher than the similarity threshold, the selected image can be added to the image set.

[0216] In some embodiments, the electronic device can repeat the above steps 625 to 629 to traverse each of the N images before and after the image to be processed, to determine whether each of the selected 2N images is similar to the image to be processed.

[0217] 630: The gallery calls the clustering algorithm in the face clustering APK.

[0218] In some embodiments, clustering the images means matching the multiple faces in the selected images with the faces in the image to be processed. For example, if there are faces A and B in the image to be processed, and there are faces A1 and B1 in the selected images, the clustering algorithm is used to match the faces. For example, if face A and face A1 match, it means that face A and face A1 are the same person.

[0219] In some embodiments, the multiple faces in the selected images can be matched with the faces in the image to be processed by various clustering algorithms. The present application does not limit this. For example, if there is a face A in the image to be processed, and there are faces A1 and B1 in the selected images, k pixel points can be randomly selected from face A1 and face B1 as clustering centers to form two clusters. Then, some of the pixel points of face A are assigned to the cluster closest to the clustering center. Next, the mean value of all pixel point data in each cluster is calculated as the new clustering center of the cluster, and the remaining pixel points of face A are assigned to the cluster closest to the clustering center. Repeat the above process, if each pixel point of face A falls in the cluster where face A1 is located, it means that face A1 and face A are the same person; if each pixel point of face A does not fall in the cluster where face A1 or face B1 is located, it means that face A1 and face B1 are not the same person as face A, i.e., there is no face in the selected images that matches the face in the image to be processed.

[0220] In some embodiments, the gallery can also first detect whether the clustering result is stored in the media information library. If it is stored, the clustering result is directly called. If it is not stored, the clustering algorithm is called in this step 630.

[0221] 631: The face clustering APK returns the clustering result to the gallery.

[0222] In some embodiments, after the face clustering APK performs clustering processing, the clustering result can be returned to the gallery.

[0223] 632: If the clustering result is empty, the gallery does not show the image beautification entry to the user.

[0224] In some embodiments, if the gallery determines that the clustering result is empty, it means that the faces in the selected images cannot match the faces in the to-be-processed image. That is, there is no reference image identical to the face in the to-be-processed image, and the image beautification entry is not displayed, for example, the image processing button is in a state of being unable to respond to user operation instructions.

[0225] 633: If the clustering result is not empty, the gallery displays the image beautification entry to the user.

[0226] In some embodiments, if the gallery determines that the clustering result is not empty, it means that the faces in the selected images can match the faces in the to-be-processed image. That is, there is a reference image identical to the face in the to-be-processed image, and the image beautification entry is displayed, for example, the image processing button is in a state of being able to respond to user operation instructions.

[0227] In this way, when the user edits the to-be-processed image, the electronic device can determine whether the image beautification entry needs to be displayed based on the above interaction process. If the image beautification entry does not need to be displayed, it means that the to-be-processed image cannot be processed by the image processing method provided in the present application; if the image beautification entry needs to be displayed, it means that when the electronic device detects that the user clicks the image processing button corresponding to the image beautification entry, the to-be-processed image can be processed by the image processing method provided in the present application.

[0228] In some embodiments, when the electronic device detects that the user clicks the image processing button corresponding to the image beautification entry, for example, the user can click the image beautification button 2022 shown in (b) of FIG. 2, the electronic device can directly process the to-be-processed image based on the reference image to generate a target image (for example, the target image 2031 shown in (c) of FIG. 2), and display the target image to the user.

[0229] Specifically, taking the face as an example of the part of the to-be-processed portrait that needs to be processed, the execution process of the electronic device processing the to-be-processed image based on the reference image to generate a target image is described in brief in combination with the flow diagram shown in FIG. 7A. Specifically, the method includes the following steps:

[0230] 701: Detecting that the user clicks the image processing button.

[0231] In some embodiments, when the electronic device displays the image processing entry, the user can click the image processing button corresponding to the image processing entry. At this time, the electronic device can detect that the user clicks the image processing button, and then perform image repair on the to-be-processed image based on the reference image. For example, as shown in (b) of FIG. 2, the mobile phone 10 can detect that the user clicks the image beautification button 2022.

[0232] 702: Traverse the to-be-processed image and the reference image to obtain an expanded image set.

[0233] In some embodiments, the electronic device can not obtain the entire portrait data when processing the to-be-processed image and the reference image, but only obtain the image data of the face region for processing.

[0234] The process of expanding each face frame to obtain the face region image data can refer to steps 7021 to 70222.

[0235] 7021: Determine a portrait list in each image.

[0236] In some embodiments, the electronic device can first determine the portrait list of each image, for example, determine the number of portraits in the to-be-processed image and the position of each portrait.

[0237] 7022: Traverse each portrait in the portrait list to obtain an expanded image set.

[0238] In some embodiments, if an image includes multiple portraits, each portrait can be traversed, and the expanded region of the face frame of each portrait can be added to the expanded image set.

[0239] The process of traversing each portrait in the portrait list can refer to steps 70221 to 70222.

[0240] 70221: Regionally expand the face frame of the portrait.

[0241] In some embodiments, the electronic device can first regionally expand the face frame of the portrait, for example, the selected expanded region can be twice as wide and twice as high as the face frame.

[0242] 70222: Extract the image in the expanded region and add the image to the expanded image set.

[0243] In some embodiments, for the expanded region obtained in step 70221, the image in the expanded region can be extracted, and the extracted image can be added to the expanded image set for subsequent processing.

[0244] In some embodiments, for multiple portraits of an image, steps 70221 and 70222 can be repeatedly executed to obtain the image in the expanded region of each portrait.

[0245] In some embodiments, steps 7021-70222 can be repeatedly performed for the plurality of reference images and the image to be processed, so as to obtain images in the outer expansion region of each portrait in each image. Thus, when processing the portrait, the entire portrait data can not be obtained, and only the image data in the outer expansion region (the face region) is obtained for processing.

[0246] 703: Traverse each face information in the expanded image set to obtain a face to be processed and a corresponding reference face.

[0247] In some embodiments, after traversing each image in the set of images to be processed and reference images and obtaining the expanded image set, the electronic device can further traverse each face image in the expanded image set to determine a replacement face.

[0248] Specifically, the face to be processed and the corresponding reference face can be determined with reference to steps 7031-7032.

[0249] 7031: Sort the same faces based on face scores.

[0250] In some embodiments, the electronic device can sort the same faces from high to low or from low to high based on the face scores. The face scores can be determined based on the eye opening amplitude, the face rotation angle, the smile amplitude of the mouth corner, etc.

[0251] 7032: Determine the face to be processed with a lower score and the corresponding face with a higher score.

[0252] In some embodiments, in the sorted groups of faces, the face to be processed with a lower score in the image to be processed is obtained, and the reference face corresponding to the face to be processed with the highest score is obtained. For example, there are faces A and B in the image to be processed, there are faces A1 and B1 in the reference image 1, and there are faces A2 and B2 in the reference image 2. The electronic device can sort the same faces through step 7031 (for example, the sorted order can be AA1A2, B1BB2), and then select the face to be processed with a lower score (for example, face A) in the image to be processed through step 7032, and obtain the reference face corresponding to the face to be processed with the highest score (for example, face A2).

[0253] In some embodiments, the electronic device can repeatedly perform steps 7031 and 7032, and by traversing each face information in the expanded image set, the face to be processed with a lower score in the image to be processed and the corresponding reference face that can be replaced can be determined.

[0254] 704: Determine whether the reference face is empty. If yes, go to step 705; if no, go to step 706.

[0255] In some embodiments, the reference face is the face with a higher score determined in step 7032.

[0256] In some embodiments, if it is determined that the reference face is empty, it means that the image beautification cannot be performed on the to-be-processed face. If the reference face is not empty, the to-be-processed face can be beautified based on the reference face.

[0257] 705: It is determined that the image beautification cannot be performed on the to-be-processed face.

[0258] In some embodiments, if it is determined that the reference face is empty, it means that the image beautification cannot be performed on the to-be-processed face.

[0259] 706: The to-be-processed face is beautified based on the reference face.

[0260] In some embodiments, if the reference face is not empty, the to-be-processed face can be beautified based on the reference face. For example, the face part of the to-be-processed portrait can be cut out, and then the reference face is copied and pasted to the face position of the to-be-processed portrait.

[0261] 707: The image after face beautification is adjusted to obtain a target image and display the target image.

[0262] In some embodiments, after the to-be-processed face is beautified through step 706, the electronic device can also adjust the paste edge position and texture, color difference, etc., so that the processed target image achieves a harmonious overall effect.

[0263] In this way, when the user edits the to-be-processed image, the electronic device can beautify the to-be-processed image based on the above process to obtain a target image.

[0264] Next, based on the timing interaction flowchart shown in FIG. 7B, the interactive process of beautifying the to-be-processed image to obtain a target image is described in detail. As shown in FIG. 7B, the interactive process of beautifying the to-be-processed image to obtain a target image can be realized through a gallery, a face generation (FaceGenerate) plug-in, a smart cloud-micro service module, a repair strategy judgment algorithm service module, a portrait repair algorithm service module, an OBS module, and interaction between users. Specifically, the interactive process shown in FIG. 7B includes the following steps:

[0265] 720: The user clicks the image processing button of the to-be-processed image in the gallery.

[0266] In some embodiments, as shown in (b) of FIG. 2, the user can click the image beautification button 2022. At this time, the electronic device can detect the operation of the user clicking the image beautification button 2022, thereby triggering the electronic device to perform the subsequent step of repairing the image 2011, thereby generating the target image 2031 shown in (c) of FIG. 2.

[0267] 721: The gallery sends a service initialization instruction to the face generation plug-in.

[0268] In some embodiments, the electronic device can first send a service initialization instruction to the face generation plug-in installed in the electronic device, so that the face generation plug-in can send a service initialization instruction to the wisdom cloud-micro service module in the cloud computing resource, so that the wisdom cloud-micro service module can perform operations such as loading image processing related algorithms, setting parameters, establishing a connection with the electronic device, and the like.

[0269] 722: The face generation plug-in sends a service initialization instruction to the wisdom cloud-micro service module.

[0270] In some embodiments, after the face generation plug-in receives the service initialization instruction sent by the gallery, the face generation plug-in sends the instruction to the wisdom cloud-micro service module.

[0271] 723: The wisdom cloud-micro service module loads related resources.

[0272] In some embodiments, after the wisdom cloud-micro service module receives the service initialization instruction sent by the face generation plug-in, the wisdom cloud-micro service module performs initialization, loads image processing related algorithms and parameters, and the like.

[0273] 724: The wisdom cloud-micro service module returns an initialization result to the gallery.

[0274] In some embodiments, after the wisdom cloud-micro service module loads the related resources, the wisdom cloud-micro service module returns the loading result to the gallery through the face generation plug-in, to notify the gallery whether the service initialization is completed.

[0275] 725: The gallery selects a list of to-be-repaired portraits and corresponding reference portraits.

[0276] In some embodiments, the electronic device can determine whether there is a to-be-repaired portrait in the to-be-processed image that meets the processing condition, and one or more reference portraits that overcome image defects and have a high similarity with the to-be-repaired portrait. If there is no to-be-repaired portrait, or there is a to-be-repaired portrait but no corresponding reference portrait, the execution of the related algorithm is ended. If there is a to-be-repaired portrait and a corresponding reference portrait, the electronic device can call a repair strategy algorithm service module to perform portrait repair.

[0277] 726: The gallery sends a call repair strategy judgment algorithm service module instruction to the face generation plug-in.

[0278] In some embodiments, when the electronic device calls the repair strategy judgment algorithm service module in the cloud computing resource to determine the portrait repair strategy, it first needs to send a call instruction to the face generation plug-in, and then the face generation plug-in sends the call instruction to the repair strategy judgment algorithm service module through the smart cloud-micro service module.

[0279] 727: The face generation plug-in sets an image identifier (sessionid).

[0280] In some embodiments, before calling the repair strategy judgment algorithm service module to determine the portrait repair strategy, the face generation plug-in can also mark the image to be processed first, for example, the image to be processed can be numbered by a number, so that in the subsequent processing process, the image to be processed can be called directly by calling the image identifier.

[0281] 728: The face generation plug-in sends a repair strategy judgment request to the smart cloud-micro service module.

[0282] In some embodiments, when calling the repair strategy judgment algorithm service module in the cloud computing resource to determine the portrait repair strategy, the electronic device can first send a repair strategy judgment request to the smart cloud-micro service module in the cloud computing resource, and then send the request to the repair strategy judgment algorithm service module through the smart cloud-micro service module.

[0283] 729: The smart cloud-micro service module sends a repair strategy judgment request to the repair strategy judgment algorithm service module.

[0284] In some embodiments, after the smart cloud-micro service module receives the repair strategy judgment request, it can send the request to the repair strategy judgment algorithm service module.

[0285] 730: The repair strategy judgment algorithm service module performs face analysis and other operations, and determines the repair strategy according to angle information, occlusion information, etc.

[0286] In some embodiments, the repair strategy determination algorithm service module can obtain the face rotation angle, face occlusion condition and other data of the portrait to be processed, and then determine the repair strategy according to the angle information and the occlusion information. For example, if the repair strategy determination algorithm service module determines that the face rotation angle is greater than a preset angle threshold, such as the face of the portrait to be processed is facing two sides, the repair strategy determination algorithm service module can determine that the portrait to be processed cannot be processed, and directly sends the result of the unprocessed to the gallery to display to the user. For another example, if the repair strategy determination algorithm service module determines that the face rotation angle is less than the preset angle threshold, and the face is occluded, the corresponding repair strategy can be to replace the entire face part of the portrait to be processed. For another example, if the repair strategy determination algorithm service module determines that the face rotation angle is less than the preset angle threshold, the face is not occluded, and a single feature meets the processing condition (such as closed eyes), the corresponding repair strategy can be to replace a certain feature part (such as replacing the eyes) of the portrait to be processed. The present application does not limit this.

[0287] In other embodiments, the repair strategy determination algorithm service module can also select the image with the highest face score from the reference portrait list as the reference portrait based on the face score, to repair the portrait to be processed based on the reference portrait. For example, the repair strategy determination algorithm service module can score each face in the reference portrait list based on the face rotation angle, eye opening degree, face smiling amplitude and other data, and then select the portrait with the highest score as the reference portrait.

[0288] 731: The repair strategy determination algorithm service module stores data to the OBS library.

[0289] In some embodiments, the repair strategy determination algorithm service module can directly store the determined repair strategy to the database of the cloud computing resource, and if the repair strategy needs to be called again, the repair strategy can be directly read from the OBS library without running the repair strategy determination algorithm again.

[0290] 732: The repair strategy determination algorithm service module sends the repair strategy to the gallery.

[0291] In some embodiments, the repair strategy determination algorithm service module can send the repair strategy to the gallery, if the repair strategy is that the face of the portrait to be processed cannot be repaired, the gallery will display the strategy to the user; if the repair strategy is that the face of the portrait to be processed can be repaired, the gallery can call the portrait repair algorithm service module for repair.

[0292] 733: The gallery prompts the user that the face rotation angle of the portrait to be processed is large or occluded.

[0293] In some embodiments, when the gallery receives the repair strategy, if the repair strategy is that the face rotation angle of the to-be-processed portrait is large or the occlusion is more, so that the face of the to-be-processed portrait cannot be repaired, the gallery can display the repair strategy to the user.

[0294] 734: The gallery sends a call portrait repair algorithm service module instruction to the face generation plug-in.

[0295] In some embodiments, when the electronic device calls the portrait repair algorithm service module in the cloud computing resource for portrait repair, it first needs to send an instruction to the face generation plug-in, and then the face generation plug-in sends the instruction to the portrait repair algorithm service module through the smart cloud-micro service module.

[0296] 735: The face generation plug-in sends a portrait repair request to the smart cloud-micro service module.

[0297] In some embodiments, when the portrait repair algorithm service module in the cloud computing resource is used for portrait repair, the electronic device can first send a portrait repair request to the smart cloud-micro service module in the cloud computing resource, and then send the request to the portrait repair algorithm service module through the smart cloud-micro service module.

[0298] 736: The smart cloud-micro service module obtains data from the OBS library based on the image identifier.

[0299] In some embodiments, before the smart cloud-micro service module sends a portrait repair request to the portrait repair algorithm service module, it can first read the repair strategy corresponding to the image identifier of the to-be-processed image from the OBS library based on the image identifier set in step 727 above, and then the smart cloud-micro service module sends the repair strategy and the portrait repair request to the portrait repair algorithm service module.

[0300] 737: The smart cloud-micro service module sends a portrait repair request to the portrait repair algorithm service module.

[0301] In some embodiments, when the smart cloud-micro service module sends a portrait repair request to the portrait repair algorithm service module, it can also send the repair strategy read from the OBS library at the same time.

[0302] 738: The portrait repair algorithm service module performs data analysis, portrait repair, portrait enhancement, and other operations.

[0303] In some embodiments, upon receiving the repair strategy and the portrait repair request, the portrait repair algorithm service module first performs data analysis, and then performs portrait repair based on the analyzed repair strategy. For example, the face part of the to-be-processed portrait is replaced with the face part of the reference portrait. Then, the portrait repair algorithm service module can also perform portrait enhancement processing on the repaired portrait, such as adjusting texture, color difference, and the like, so that the processed target image achieves a harmonious overall effect.

[0304] 739: The portrait repair algorithm service module sends the repaired target image to the gallery.

[0305] In some embodiments, after the portrait repair algorithm service module repairs the to-be-processed portrait to obtain a target image, the target image can be sent to the gallery for display to the user.

[0306] 740: The gallery displays the target image to the user.

[0307] In some embodiments, after the gallery receives the repaired target image, the target image can be displayed to the user.

[0308] In this way, by using the image processing method provided in the present application, the to-be-processed image can be beautified, and the obtained final image can overcome image defects, thereby improving the overall visual effect of the image.

[0309] In the above-mentioned schematic diagrams of FIGS. 7A and 7B, the target image is obtained by directly repairing the portrait in the to-be-processed image based on a reference portrait. For example, a plurality of portraits with the same appearance as the to-be-processed portrait can be scored based on data such as face rotation angle, eye opening degree, and face smiling amplitude, and then the portrait with the highest score is selected as the reference portrait, and the to-be-processed portrait is repaired based on the reference portrait.

[0310] In other embodiments, the electronic device can also display a plurality of portraits with the same appearance as the to-be-processed portrait and capable of making up for image defects, and the user can select a reference portrait from the plurality of portraits, and the to-be-processed portrait is repaired based on the reference portrait selected by the user. As shown in (c) of FIG. 3, the editing interface 303 can display a plurality of image identifiers corresponding to portraits with the same appearance as the to-be-processed portrait 3012 and with open eyes, and when the mobile phone 10 detects that the user clicks the image identifier 3031, the electronic device can repair the to-be-processed portrait 3012 based on the reference portrait corresponding to the image identifier 3031.

[0311] Specifically, the following takes the part to be processed in the portrait to be processed as the face as an example, and a flowchart shown in FIG. 8A is combined to briefly describe the execution process of obtaining the reference image in combination with the user interaction to generate the target image by processing the portrait to be processed. Specifically, the process can include the following steps:

[0312] 801: detecting that the user selects the operation of outputting the reference portrait.

[0313] In some embodiments, when the portrait to be processed corresponds to multiple reference portraits, the user can arbitrarily select one of the reference portraits. At this time, the electronic device can detect the selection operation of the user, and beautify the portrait to be processed based on the reference portrait selected by the user. For example, as shown in (c) of FIG. 3 described above, the electronic device can beautify the portrait to be processed 3012 based on the reference portrait corresponding to the image identifier 3031 selected by the user.

[0314] 802: obtaining the occlusion information and the rotation angle of the portrait to be processed.

[0315] In some embodiments, the electronic device can determine the occlusion degree of the facial features of the portrait to be processed, and the angle of the facial features to the lens.

[0316] 803: determining whether the occlusion degree is greater than the maximum occlusion threshold. If yes, go to step 804; if no, go to step 806.

[0317] In some embodiments, when the occlusion degree of the facial features is greater than the maximum occlusion threshold, it indicates that the portrait to be processed cannot be processed. Conversely, it indicates that the portrait to be processed can be processed by face or head replacement.

[0318] 804: determining that the portrait to be processed cannot be processed.

[0319] In some embodiments, when it is determined through step 803 that the occlusion degree of the facial features is greater than the maximum occlusion threshold, it indicates that the portrait to be processed cannot be processed.

[0320] 805: end the current processing flow.

[0321] In some embodiments, when the portrait to be processed cannot be processed, the electronic device can end the current cycle to end the current processing flow, and then start the next cycle to continue processing the next portrait to be processed.

[0322] 806: determining whether the occlusion degree is greater than the minimum occlusion threshold. If yes, go to step 807; if no, go to step 811.

[0323] In some embodiments, when it is determined through step 803 that the occlusion degree is less than the maximum occlusion threshold, it is further determined whether the occlusion degree is greater than the minimum occlusion threshold. If yes, the head replacement processing is required to be performed on the image to be processed; if no, it is further determined whether the face rotation angle is greater than the angle threshold.

[0324] 807: Determine to perform the head replacement processing.

[0325] In some embodiments, if it is determined through step 803 that the face occlusion degree of the portrait to be processed is less than the maximum occlusion threshold, and it is determined through step 806 that the face occlusion degree is greater than the minimum occlusion threshold, it can be determined that the corresponding repair strategy is to replace the head image feature in the portrait to be processed with the head image feature of the corresponding reference image, so as to obtain a target image overcoming the image defect. For details, refer to steps 808 to 810.

[0326] 808: Align and post the head.

[0327] In some embodiments, when the electronic device performs the head replacement operation on the portrait to be processed, it can perform the image processing operation of aligning and posting the head of the reference portrait with the head of the portrait to be processed through the method shown in step 808, so as to replace the head region of the portrait to be processed with the head region in the reference portrait.

[0328] In other embodiments, when the electronic device performs the head replacement operation on the portrait to be processed, it can not perform the method shown in step 808, but perform the head replacement operation on the portrait to be processed through an artificial intelligence (AI) generation method. For example, if the portrait to be processed is a closed-eye portrait and the reference portrait is an open-eye portrait, the face features of the portrait to be processed and the face features of the reference portrait can be input into an AI generation model (such as a stable diffusion model), and the AI generation model can generate an open-eye target portrait based on the input feature data (such as the eye feature data of the reference portrait and the other facial feature data of the portrait to be processed).

[0329] It should be understood that the present application can perform the head replacement operation on the portrait to be processed through any image processing method, which is not limited in the present application.

[0330] 809: Perform fusion processing on the head.

[0331] In some embodiments, the electronic device can fuse the obtained target head region with other regions in the image to be processed, so as to obtain a target image.

[0332] 810: Mark the replacement mark.

[0333] In some embodiments, a replacement mark can be marked after the head or face replacement processing of the to-be-processed portrait, so as to determine whether the head or face replacement operation is performed on the to-be-processed portrait based on the replacement mark.

[0334] 811: Determine whether the rotation angle is greater than the angle threshold. If yes, go to step 807; if no, go to step 812.

[0335] In some embodiments, after the electronic device determines that the face occlusion degree of the to-be-processed portrait is less than the minimum occlusion threshold through step 806, the electronic device further needs to determine whether the face rotation angle of the to-be-processed portrait is greater than the angle threshold. If the face rotation angle is greater than the angle threshold, go to step 807 to perform the head replacement operation, and replace the head image feature in the to-be-processed portrait with the head image feature corresponding to the reference image, so as to obtain the target image overcoming the image defect. If the face occlusion degree is less than the minimum occlusion threshold and the face rotation angle is less than the angle threshold, go to step 812 to determine that the repair strategy is to replace at least part of the facial feature in the to-be-processed portrait that does not meet the preset condition with at least part of the facial feature in the corresponding reference image, so as to obtain the target image overcoming the image defect.

[0336] 812: Determine to perform facial feature processing.

[0337] In some embodiments, if it is determined through step 806 that the occlusion degree is less than the minimum occlusion threshold, and it is determined through step 811 that the face rotation angle is less than the angle threshold, it is determined that the replacement of a certain facial feature of the to-be-processed portrait can be performed. For details, refer to steps 813 to 814.

[0338] 813: Align and post the face.

[0339] In some embodiments, when the electronic device performs the replacement of the facial feature area (such as, replacement of the eyes) of the to-be-processed portrait, the electronic device can perform the image processing operation of aligning and posting the facial feature area of the reference portrait and the facial feature area of the to-be-processed portrait through the method shown in step 813, so as to replace part of the facial feature area of the to-be-processed portrait with part of the facial feature area of the reference portrait.

[0340] In some embodiments, the electronic device can perform the method shown in step 813 to replace the facial feature of the target portrait. For example, if the target portrait is a closed-eye portrait and the reference portrait is an open-eye portrait, the electronic device can input the facial feature (e.g., eye feature) of the reference portrait into an AI generation model (e.g., a stable diffusion model) or input the facial feature of the reference portrait and the facial feature of the target portrait into the AI generation model at the same time, and the AI generation model can generate a target facial feature image based on the input feature data (e.g., eye feature data of the reference portrait).

[0341] It should be understood that the application can replace the facial feature of the target portrait by any image processing method, and the application does not limit this.

[0342] 814: Perform fusion processing on the face.

[0343] In some embodiments, the electronic device can fuse the obtained target facial feature with other regions in the target image to obtain a target image. In addition, after performing the face replacement operation on the target portrait, the electronic device can proceed to step 810 to mark the replacement mark.

[0344] 815: Determine whether the replacement mark exists. If yes, proceed to step 817; if no, proceed to step 816.

[0345] In some embodiments, the electronic device can determine whether the face replacement or head replacement operation is performed based on whether the replacement mark exists in the target image. If the replacement mark exists, the electronic device can update the target image as the processed image and display the processed image to the user; if the replacement mark does not exist, the electronic device can determine that the beautification processing on the target image cannot be performed.

[0346] 816: Determine that the beautification processing on the target image cannot be performed.

[0347] In some embodiments, if it is determined through step 815 that the replacement mark does not exist in the target image, it means that the target image is still the original target image and the face replacement or head replacement operation is not performed, and it can be determined that the beautification processing on the target image cannot be performed.

[0348] 817: Replace the target image with the processed image.

[0349] In some embodiments, if it is determined through step 815 that the replacement mark exists in the target image, it means that the target image is not the original target image and the face replacement or head replacement operation is performed on the target portrait in the target image, and the electronic device can replace the target image with the processed image and display the processed image to the user.

[0350] 818: display the target image to the user.

[0351] In some embodiments, the electronic device can replace the to-be-processed image with the target image and display it to the user.

[0352] In this way, the to-be-processed image can be processed based on the reference image selected by the user through the above process, so that the target image can overcome image defects and improve the overall aesthetic appearance of the target image.

[0353] The specific process of the image processing method based on the reference image selected by the user to process the to-be-processed image will be further described below in conjunction with the flowchart shown in FIG. 8B. The method can be applied to an electronic device, such as the mobile phone 10 shown in FIG. 1. Compared with the flowchart shown in FIG. 8A, the flowchart shown in FIG. 8B adds a judgment on whether the to-be-processed image is a single-person photo. Specifically, the method shown in FIG. 8B can include the following steps:

[0354] S820: Obtain a set of portraits in the to-be-processed image.

[0355] In some embodiments, the electronic device can first obtain each portrait in the to-be-processed image. For example, for a single-person photo, the electronic device can obtain a single portrait of the single-person photo; for a group photo, the electronic device can obtain multiple portraits in the group photo.

[0356] S821: Determine a set of reference portraits corresponding to a to-be-processed portrait in the set of portraits.

[0357] In some embodiments, when there are multiple portraits in the set of portraits of the to-be-processed image, each portrait can be processed separately. Specifically, when processing a portrait, a set of reference portraits similar in appearance to the portrait is first selected from the gallery.

[0358] S822: Sort each portrait in the set of reference portraits.

[0359] In some embodiments, after obtaining multiple portraits similar in appearance to the to-be-processed portrait, each portrait can be sorted based on the face score of each portrait, and each sorted portrait can be displayed to the user. For example, each portrait can be scored based on data such as eye opening amplitude, face rotation angle, and smile amplitude of the mouth, and the scores can be arranged in descending order to display each portrait to the user, so that the user can select the final reference portrait from the multiple portraits to replace the to-be-processed portrait.

[0360] S823: Detect that the user has selected a reference portrait from the set of reference portraits.

[0361] In some embodiments, the user can select a final reference portrait from the ranked plurality of reference portraits to replace the to-be-processed portrait. At this time, the electronic device can detect the selection operation of the user, and process the to-be-processed image based on the reference image selected by the user.

[0362] S824: Obtain face data of the to-be-processed portrait, such as face rotation angle or face occlusion information.

[0363] In some embodiments, the electronic device can obtain face rotation angle data of the to-be-processed portrait, for example, the rotation angle of the face facing the user can be set to 0, and then the included angle between the face of the to-be-processed portrait and the direction facing the user is obtained.

[0364] In other embodiments, the electronic device can also obtain face occlusion information of the to-be-processed portrait, for example, it can be detected that the face is occluded by x%.

[0365] S825: Portrait repair strategy determination based on face data.

[0366] In some embodiments, the electronic device can determine the portrait repair strategy based on the face data of the to-be-processed portrait, such as the face rotation angle or the face occlusion information.

[0367] In some embodiments, if the face occlusion is serious (for example, 80% of the face part of the to-be-processed portrait is occluded) or the face rotation angle is greater than the angle threshold, the portrait repair strategy is that the to-be-processed portrait cannot be processed. Then go to step S8261 to determine whether the to-be-processed image is a single portrait, so as to process other portraits in the to-be-processed image.

[0368] In some embodiments, if the face occlusion of the to-be-processed portrait is not serious, for example, the occlusion area is less than the occlusion threshold, or the face rotation angle is small, the portrait repair strategy is to replace the entire head of the to-be-processed portrait. Then go to step S8271 to perform face analysis on the to-be-processed portrait and the final reference portrait selected by the user.

[0369] In some embodiments, if the face is not occluded and the face is facing the user, only the facial feature part meets the processing condition to affect the image aesthetics, the portrait repair strategy is to replace a certain facial feature part of the to-be-processed portrait. Then go to step S8281 to analyze the facial feature region of the to-be-processed portrait and the final reference portrait selected by the user.

[0370] S8261: Determine whether the to-be-processed image is a single portrait? If not, go to step S8262; if yes, go to step S8263.

[0371] In some embodiments, in step S825, when the portrait repair strategy is determined based on the face data such as the face rotation angle or the face occlusion information of the portrait to be processed, if the face occlusion is serious or the face rotation angle is greater than the angle threshold, the portrait repair strategy is that the portrait to be processed cannot be processed. At this time, the electronic device needs to determine whether there is only one portrait in the image to be processed. If yes, go to step S8263, prompt the user that the image to be processed cannot be processed; if no, go to step S8262, mark the portrait to be processed as unprocessable, and then process the next portrait in the image to be processed.

[0372] S8262: Mark the portrait to be processed as unchangeable.

[0373] In some embodiments, in step S825, when the portrait repair strategy is determined based on the face data such as the face rotation angle or the face occlusion information of the portrait to be processed, if the face occlusion is serious or the face rotation angle is greater than the angle threshold, the portrait repair strategy is that the portrait to be processed cannot be processed. And in step S8261, when it is determined that the image to be processed is not a single-person photo, the portrait to be processed is marked as unchangeable, and then go to step S821 to obtain a reference portrait set corresponding to the next portrait to be processed in the image to be processed.

[0374] S8263: Prompt the user that the image to be processed cannot be processed.

[0375] In some embodiments, in step S825, when the portrait repair strategy is determined based on the face data such as the face rotation angle or the face occlusion information of the portrait to be processed, if the face occlusion is serious or the face rotation angle is greater than the angle threshold, the portrait repair strategy is that the portrait to be processed cannot be processed. And in step S8261, when it is determined that the image to be processed is a single-person photo, it is determined that the image to be processed cannot be processed, and the user is prompted that the image to be processed cannot be processed through a pop-up window or the like.

[0376] S8264: End the execution of the image processing method.

[0377] In some embodiments, after prompting the user that the image to be processed cannot be processed, the electronic device can end the execution of the image processing method provided in the present application.

[0378] S8271: Perform face analysis on the portrait to be processed.

[0379] In some embodiments, if the face of the to-be-processed portrait is not seriously occluded, for example, the occluded area is less than the occlusion threshold, or the face rotation angle is small, the portrait repair strategy determined in step S825 is to replace the entire head of the to-be-processed portrait. Wherein, when replacing the head, it is necessary to first perform face analysis on the to-be-processed portrait selected by the user, for example, the edge features of the face can be obtained so as to completely cut out the head region of the person.

[0380] S8272: Cut out the head region of the to-be-processed portrait.

[0381] In some embodiments, after the face edge features and other data of the to-be-processed portrait are determined, the head region of the to-be-processed portrait can be completely cut out.

[0382] S8273: Align and post the head region of the to-be-processed portrait.

[0383] In some embodiments, the head region of the reference portrait can be aligned with the head region of the to-be-processed portrait, and the head region of the to-be-processed portrait can be replaced with the head region of the reference portrait.

[0384] S8274: Perform head fusion on the to-be-processed portrait.

[0385] In some embodiments, after the head region of the to-be-processed portrait is replaced with the head region of the reference portrait through step S8273, the connection position of the head region and the body region can be adjusted to make the connection position more smooth and improve the authenticity of the processed portrait. In this way, through the above steps S8271 to S8274, the head replacement operation on the to-be-processed portrait can be completed.

[0386] S8281: Analyze the facial feature part of the to-be-processed portrait.

[0387] In some embodiments, if the face is not occluded and faces the user, and only the facial feature part meets the processing condition to affect the image aesthetics, the portrait repair strategy determined in step S825 is to replace a certain facial feature part of the to-be-processed portrait. Wherein, when replacing the facial feature part, it is necessary to first analyze the facial feature part of the to-be-processed portrait selected by the user, for example, the position of the facial feature part in the face and the edge features of the facial feature part can be obtained so as to completely cut out a certain facial feature region.

[0388] S8282: Cut out the facial feature region of the to-be-processed portrait.

[0389] In some embodiments, after the facial feature edge features and other data of the to-be-processed portrait are determined, the facial feature part (such as eyes) that needs to be adjusted in the to-be-processed portrait can be completely cut out.

[0390] S8283: Aligning and pasting the facial feature regions of the to-be-processed portrait.

[0391] In some embodiments, the facial feature regions of the reference portrait can be aligned with the facial feature regions of the to-be-processed portrait, and the facial feature regions that need to be adjusted in the to-be-processed portrait can be replaced with the facial feature regions of the reference portrait.

[0392] S8284: Face fusion on the to-be-processed portrait.

[0393] In some embodiments, after the facial feature regions of the to-be-processed portrait are replaced with the facial feature regions of the reference portrait through step S8283, the connection position of the replaced facial feature regions and the face region can be adjusted to make the connection position smoother and improve the authenticity of the processed portrait. In this way, the replacement of facial features on the to-be-processed portrait can be completed through the above steps S8281 to S8284.

[0394] S829: Adjusting the brightness and other data of the processed portrait.

[0395] In some embodiments, after the head replacement processing on the to-be-processed portrait through steps S8271 to S8274, or the replacement of facial features on the to-be-processed portrait through steps S8281 to S8284, the brightness, texture, color difference, and other details of the processed portrait can be adjusted to make the overall portrait have higher uniformity.

[0396] S830: Scoring the processed portrait.

[0397] In some embodiments, after processing the to-be-processed portrait, the processed portrait can be scored based on the data such as the eye opening amplitude, face rotation angle, and smile amplitude of the mouth corner of each portrait.

[0398] In some embodiments, the electronic device can also repeatedly execute the above steps S821 to S830 to process multiple to-be-processed portraits in the to-be-processed image. That is, when processing the first to-be-processed portrait, the above steps S821 to S830 can be executed to obtain a processed portrait; then, steps S821 to S830 can be repeatedly executed to process the next to-be-processed portrait.

[0399] S831: Displaying the processed target image.

[0400] In some embodiments, the electronic device can display the obtained target image to the user.

[0401] S832: Ending the execution of the image processing algorithm.

[0402] In some embodiments, the electronic device ends executing the image processing algorithm after presenting the target image to the user.

[0403] In this way, the target image can overcome image defects and improve overall aesthetics by processing the target image based on the reference image selected by the user through the above process.

[0404] The following describes the interactive process of processing the target image based on the reference image selected by the user in combination with the system structure diagram of the image processing method shown in FIG. 4 and the timing interaction flow diagram shown in FIG. 8C. As shown in FIG. 8C, the interactive process of processing the target image based on the reference image selected by the user can be implemented through the gallery, the face generation (FaceGenerate) plug-in, the smart cloud-micro service module, the repair strategy judgment algorithm service module, the portrait repair algorithm service module, and the interaction between users. Specifically, the interactive process shown in FIG. 8C includes the following steps:

[0405] 850: The user selects a reference portrait in the editing interface of the gallery.

[0406] In some embodiments, the electronic device can display a plurality of portraits with the same appearance as the target portrait, and the user can select any one of the portraits as the reference portrait.

[0407] 851: The gallery determines the target portrait and the reference portrait.

[0408] In some embodiments, after the user selects the reference portrait, the gallery can determine the target portrait and the corresponding reference portrait.

[0409] 852: The gallery sends an instruction to call the repair strategy judgment algorithm service module to the face generation plug-in.

[0410] In some embodiments, when the electronic device calls the repair strategy judgment algorithm service module in the cloud computing resource to determine the portrait repair strategy, it first needs to send a calling instruction to the face generation plug-in, and then the face generation plug-in sends the calling instruction to the repair strategy judgment algorithm service module through the smart cloud-micro service module.

[0411] 853: The face generation plug-in performs image size processing.

[0412] In some embodiments, before calling the repair strategy judgment algorithm service module to determine the portrait repair strategy, the face generation plug-in can first perform image size processing on the target image to adjust the image size to the input size that can be received by the portrait repair strategy algorithm.

[0413] 854: The face generation plug-in sends a repair strategy judgment request to the smart cloud-micro service module.

[0414] In some embodiments, when determining the portrait repair strategy through the repair strategy judgment algorithm service module in the cloud computing resource, the electronic device can first send a repair strategy judgment request to the smart cloud-micro service module in the cloud computing resource, and then send the request to the repair strategy judgment algorithm service module through the smart cloud-micro service module.

[0415] 855: The smart cloud-micro service module sends a repair strategy judgment request to the repair strategy judgment algorithm service module.

[0416] In some embodiments, after receiving the repair strategy judgment request, the smart cloud-micro service module can send the request to the repair strategy judgment algorithm service module.

[0417] 856: The repair strategy judgment algorithm service module performs face analysis and other operations, and determines the repair strategy according to angle information, occlusion information, etc.

[0418] In some embodiments, the repair strategy judgment algorithm service module can obtain the face rotation angle, face occlusion degree, and other data of the portrait to be processed, and then determine the repair strategy according to the angle information and the occlusion information. For example, if the repair strategy judgment algorithm service module determines that the face rotation angle is greater than a preset angle threshold, such as the face of the portrait to be processed is facing two sides, the repair strategy judgment algorithm service module can determine that the portrait to be processed cannot be processed, and directly sends the result of being unable to process to the gallery to display to the user. For another example, if the repair strategy judgment algorithm service module determines that the face rotation angle is less than a preset angle threshold, and the face is occluded, the corresponding repair strategy can be to replace the entire head region of the portrait to be processed. For another example, if the repair strategy judgment algorithm service module determines that the face rotation angle is less than a preset angle threshold, the face is not occluded, and a single feature satisfies the processing condition (such as closed eyes), the corresponding repair strategy can be to replace a certain feature part (such as replacing the eyes) of the portrait to be processed. The present application does not limit this.

[0419] 857: The repair strategy judgment algorithm service module sends the repair strategy to the smart cloud-micro service module.

[0420] In some embodiments, after determining the repair strategy, the repair strategy judgment algorithm service module can send the repair strategy to the smart cloud-micro service module, and the smart cloud-micro service module determines whether to send to the gallery or call the portrait repair algorithm service module for portrait repair.

[0421] 8581: The smart cloud-micro service module sends the repair strategy to the gallery.

[0422] In some embodiments, if the repair strategy is that the portrait to be processed cannot be repaired, the intelligent cloud-micro service module can send the repair strategy to the gallery, and the gallery can display the repair strategy to the user.

[0423] 8582: The gallery prompts the user that the face rotation angle of the portrait to be processed is large or there is occlusion.

[0424] In some embodiments, when the gallery receives the repair strategy, if the repair strategy is that the face rotation angle of the portrait to be processed is large or there is too much occlusion, so that the portrait to be processed cannot be repaired, the gallery can display the repair strategy to the user.

[0425] 859: The intelligent cloud-micro service module sends a portrait repair request to the portrait repair algorithm service module.

[0426] In some embodiments, if the repair strategy can be repaired, the intelligent cloud-micro service module can send a portrait repair request to the portrait repair algorithm service module, and can also send the repair strategy at the same time.

[0427] 860: The portrait repair algorithm service module performs data analysis, portrait repair, portrait enhancement, and the like.

[0428] In some embodiments, when the portrait repair algorithm service module receives the repair strategy and the portrait repair request, it first performs data analysis, and then performs portrait repair based on the analyzed repair strategy. For example, the head region of the portrait to be processed is replaced with the head region of the reference portrait. Then, the portrait repair algorithm service module can also perform portrait enhancement processing on the repaired portrait, such as adjusting the texture, color difference, and the like, so that the processed target image achieves a harmonious overall effect.

[0429] 861: The portrait repair algorithm service module sends the repaired target image to the gallery.

[0430] In some embodiments, after the portrait repair algorithm service module repairs the portrait to be processed to obtain a target image, the target image can be sent to the gallery, and the gallery can display the target image to the user.

[0431] 862: The gallery displays the target image to the user.

[0432] In some embodiments, after the gallery receives the repaired target image, the gallery can display the target image to the user.

[0433] In this way, by using the image processing method provided in the present application, the portrait to be processed can be processed based on the reference image selected by the user, so that the target image can overcome image defects and improve the overall aesthetic appearance of the target image.

[0434] The embodiment of the present application provides a readable storage medium, the readable storage medium stores instructions, and the instructions are executed on an electronic device to enable the electronic device to execute the image processing method mentioned in the present application.

[0435] The embodiment of the present application provides a computer program product, comprising: computer instructions, when the computer instructions are run on an electronic device, enabling the electronic device to execute the image processing method mentioned in the present application.

[0436] The embodiment of the present application further provides an electronic device, comprising: a memory and a processor, the memory is used to store instructions executed by one or more processors of the electronic device, and the processor is one of the one or more processors of the electronic device, and is used to execute the image processing method mentioned in the present application.

[0437] The image processing method mentioned in the present application can be executed by an electronic device, wherein the electronic device can be any electronic device such as a mobile phone, a computer, a tablet computer, an augmented reality (AR) device, a notebook computer and the like. The form of the electronic device is not limited in the embodiment of the present application.

[0438] As shown in FIG. 9, taking the mobile phone 10 as an example, the hardware structure schematic diagram of the electronic device of one embodiment of the present application is exemplarily illustrated.

[0439] As shown in FIG. 9, the mobile phone 10 can include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charge management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a loudspeaker 170A, a receiver 170B, a microphone 170C, a headset interface 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 can include a gyroscope sensor 180A, a magnetic sensor 180B, an acceleration sensor 180C, an infrared sensor 180D, a temperature sensor 180E, a touch sensor 180F, an ambient light sensor 180G, a light sensor 180H, etc.

[0440] It can be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the mobile phone 10. In other embodiments of the present application, the mobile phone 10 can include more or fewer components than illustrated, or combine certain components, or split certain components, or different component arrangements. The illustrated components can be implemented in hardware, software, or a combination of software and hardware.

[0441] The processor 110 can include one or more processing units, for example: the processor 110 can include a central processing unit (CPU), a graphics processing unit (GPU), an embedded neural processing unit (NPU), an application processor (AP), a modem processor, a controller, a video codec, a digital signal processor (DSP), etc. Different processing units can be independent devices, or can be integrated into one or more processors.

[0442] In the embodiments of the present application, the processor 110 can be used to execute the image processing method provided by the present application. For example, the processor 110 can acquire the first image feature (for example, the face feature) of the to-be-processed image, and acquire the second image feature (for example, the face feature) of each image in the image library, select the reference image with high similarity to the image feature of the to-be-processed target and capable of making up for the image defect from each image, and finally perform image inpainting on the to-be-processed image based on the selected reference image to obtain the target image.

[0443] The memory in the processor 110 can also be configured to store instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory can save instructions or data that the processor 110 has just used or repeatedly uses. If the processor 110 needs to use the instructions or data again, it can directly call from the memory. This avoids repeated access and reduces the waiting time of the processor 110, thereby improving the efficiency of the system.

[0444] The USB interface 130 can be used to connect a charger to charge the mobile phone 10, and can also be used to transmit data between the mobile phone 10 and peripheral devices. It can also be used to connect earphones to play audio through the earphones. The interface can also be used to connect other electronic devices, such as AR devices, etc.

[0445] The charging management module 140 is configured to receive charging input from a charger. The charger can be a wireless charger or a wired charger. The charging management module 140 charges the battery 142 and also supplies power to the electronic device via the power management module 141.

[0446] The power management module 141 is configured to connect the battery 142 and the charging management module 140 to the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to supply power to the processor 110, the internal memory 121, the screen 194, the camera 193, and the wireless communication module 160. In some embodiments, the power management module 141 can be disposed in the processor 110. In some other embodiments, the power management module 141 and the charging management module 140 can be disposed in the same device.

[0447] The wireless communication function of the mobile phone 10 can be implemented by the antennas 1 and 2, the mobile communication module 150, the wireless communication module 160, and the modem processor. The antennas 1 and 2 are configured to transmit and receive electromagnetic wave signals. Each antenna in the mobile phone 10 can be configured to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization of the antennas. For example, the antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antenna 1 can be used in combination with a tuning switch.

[0448] The mobile communication module 150 can provide a solution for wireless communication, including 2G / 3G / 4G / 5G, applied to the mobile phone 10.

[0449] The screen 194 is configured to display images, videos, and the like. The screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), or the like. In some embodiments, the mobile phone 10 can include one or N screens 194, where N is a positive integer greater than 1.

[0450] In some embodiments, the screen 194 can be configured to display a to-be-processed image and a target image, and also display an image editing interface.

[0451] The camera 193 is used to capture still images or videos. An object projects an optical image through a lens onto a photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. In some embodiments, the mobile phone 10 can include one or N cameras 193, where N is a positive integer greater than one.

[0452] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 can include a program storage area and a data storage area. The program storage area can store an operating system, at least one application program required for a function (such as a sound playing function, an image playing function, etc.), and the like. The data storage area can store data created during use of the mobile phone 10 (such as audio data, a phone book, etc.), and the like. In addition, the internal memory 121 can include a high-speed random access memory, and can further include a non-volatile memory such as at least one magnetic disk storage device, a flash memory device, a universal flash storage (UFS), and the like. The processor 110 executes various function applications and data processing of the mobile phone 10 by running instructions stored in the internal memory 121 and / or instructions stored in a memory disposed in the processor.

[0453] The mobile phone 10 can implement audio functions through an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, an application processor, and the like. For example, music playing, voice recording, and the like.

[0454] The gyroscope sensor 180A can be used to determine the motion posture of the mobile phone 10. In some embodiments, the angular velocity of the mobile phone 10 around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor 180A.

[0455] The magnetic sensor 180B can detect the strength of a magnetic field.

[0456] The acceleration sensor 180C can detect the magnitude of acceleration of the mobile phone 10 in various directions (generally three axes). When the mobile phone 10 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device, and applied to landscape / portrait screen switching, pedometer applications, and the like.

[0457] The infrared sensor 180D is used to detect, receive, and analyze infrared radiation signals.

[0458] The temperature sensor 180E is configured to detect temperature. In some embodiments, the phone 10 performs a temperature handling strategy based on the temperature detected by the temperature sensor 180E. For example, when the temperature reported by the temperature sensor 180E exceeds a threshold, the phone 10 performs a performance reduction of a processor located near the temperature sensor 180E to reduce power consumption and implement thermal protection.

[0459] The touch sensor 180F is also referred to as a "touch device". The touch sensor 180F can be disposed on the screen 194, and the touch sensor 180F and the screen 194 form a touch screen, also referred to as a "touch panel". The touch sensor 180F is configured to detect a touch operation applied thereon or in the vicinity thereof.

[0460] The ambient light sensor 180G can be configured to sense ambient light brightness.

[0461] The light sensor 180H can be configured to detect light intensity, and has high sensitivity and fast response speed, and can be applied to various lighting environments and provide real-time feedback on light conditions.

[0462] The keys 190 include a power-on key, a volume key, and the like. The keys 190 can be mechanical keys. Alternatively, the keys 190 can be touch keys. The phone 10 can receive key input and generate key signal input related to user settings and function control of the phone 10.

[0463] The motor 191 can generate a vibration prompt. The motor 191 can be configured to provide a call vibration prompt, and can also be configured to provide touch vibration feedback.

[0464] The indicator 192 can be an indicator light, and can be configured to indicate a charging state, a power change, and can also be configured to indicate a message, a missed call, a notification, and the like.

[0465] The SIM card interface 195 is configured to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to achieve contact and separation with the phone 10. The phone 10 can support one or N SIM card interfaces, where N is a positive integer greater than 1.

[0466] Embodiments disclosed in the present application can be implemented in hardware, software, firmware, or a combination thereof. Embodiments of the present application can be implemented as computer programs or program codes executed on a programmable system, which includes at least one processor, a storage system (including volatile and non-volatile memories and / or storage elements), at least one input device, and at least one output device.

[0467] The program code can be applied to input instructions to perform the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a processing system includes any system that has a processor, such as for example a digital signal processor, microcontroller, application specific integrated circuit, or microprocessor.

[0468] The program code can be implemented in a high-level programming language or an object-oriented programming language to communicate with a processing system. When necessary, the program code can also be implemented in assembly or machine language. In fact, the mechanisms described in this application are not limited to any particular programming language. In any case, the language can be a compiled or interpreted language.

[0469] In the drawings, some structural or methodological features can be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order can not be required. Rather, in some embodiments, these features can be arranged in a manner different from that shown in the illustrative drawings. In addition, the inclusion of structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features can not be included or can be combined with other features.

[0470] It should be noted that each unit / module mentioned in the device embodiments of the present application is a logical unit / module. Physically, a logical unit / module can be a physical unit / module, a part of a physical unit / module, or a combination of multiple physical units / modules. The physical implementation of these logical units / modules is not the most important, and the combination of the functions implemented by these logical units / modules is the key to solving the technical problems proposed in the present application. In addition, in order to highlight the innovative part of the present application, the above-mentioned device embodiments of the present application do not introduce units / modules that are not closely related to solving the technical problems proposed in the present application, which does not mean that the above-mentioned device embodiments do not have other units / modules.

[0471] It is to be understood that the phrases such as "first" and "second", and the like, used herein are used only to distinguish one from another entity or action, and do not require or imply these entities or actions to be in any particular sequence or order. Also, the terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0472] While the application has been illustrated and described in relation to certain exemplary embodiments thereof, it will be appreciated that various changes in form and detail can be made therein without departing from the scope of the application.

Claims

An image processing method, characterized by, The method comprises: acquiring a first image to be processed; detecting that a first image feature that does not satisfy a preset condition exists in a first region in the first image, wherein the first region comprises a first face region, and the first image feature comprises a first face feature; selecting a first reference image from a gallery, wherein a similarity between a second region of the first reference image and the first region is greater than a similarity threshold, and a second image feature of the second region satisfies the preset condition; processing the first image feature in the first image based on the second image feature to obtain a second image that satisfies the preset condition. The method of claim 1, wherein The method comprises: detecting an image editing operation of a user on the first image, acquiring the first image to be processed, and displaying a first image editing interface corresponding to the first image, wherein the first image editing interface comprises an image processing button; corresponding to the existence of the first image feature in the first image and the existence of the first reference image in the gallery, the image processing button is in a state capable of responding to a user operation instruction; corresponding to the non-existence of the first image feature in the first image, or the existence of the first image feature in the first image and the non-existence of the first reference image in the gallery, the image processing button is in a state incapable of responding to a user operation instruction. The method according to claim 2, characterized in that corresponding to the image processing button being in a state capable of responding to a user operation instruction, processing the first image feature in the first image based on the second image feature to obtain a second image that satisfies the preset condition comprises: in a case where it is detected that the user clicks the image processing button, processing the first image feature in the first image based on the second image feature in the first reference image to obtain the second image. The method of claim 1, wherein The method further comprises: detecting that the first image feature that does not satisfy the preset condition exists in the first region in the first image, and a third image feature that does not satisfy the preset condition exists in a third region, wherein the first region comprises the first face region, the first image feature comprises the first face feature, the third region comprises a second face region, and the third image feature comprises a second face feature; selecting the first reference image and a second reference image from the gallery, wherein a similarity between the second region of the first reference image and the first region is greater than the similarity threshold, the second image feature of the second region satisfies the preset condition, a similarity between a fourth region of the second reference image and the third region is greater than the similarity threshold, and a fourth image feature of the fourth region satisfies the preset condition; processing the first image feature in the first image based on the second image feature, and processing the third image feature in the first image based on the fourth image feature to obtain a second image that satisfies the preset condition. The method according to claim 4, characterized in that The method comprises: detecting a user image editing operation on the first image, obtaining the first image to be processed, and displaying a second image editing interface corresponding to the first image, the second image editing interface including an image processing button; corresponding to the presence of the first image feature and the third image feature in the first image, and the presence of the first reference image and the second reference image in the gallery, the image processing button is in a state capable of responding to a user operation instruction; corresponding to the absence of the first image feature and the third image feature in the first image, or the absence of the first reference image and the second reference image in the gallery, the image processing button is in a state incapable of responding to a user operation instruction. The method according to claim 5, characterized in that corresponding to the image processing button being in a state capable of responding to a user operation instruction, processing the first image feature in the first image based on the second image feature, and processing the third image feature in the first image based on the fourth image feature to obtain a second image satisfying the preset condition, including: detecting a user operation of clicking the image processing button, displaying a third image editing interface corresponding to the first image, the third image editing interface including a first image identifier corresponding to the first image feature and a second image identifier corresponding to the third image feature; detecting a user operation of selecting the first image identifier, processing the first image feature in the first image based on the second image feature; detecting a user operation of selecting the second image identifier, processing the third image feature in the first image based on the fourth image feature. The method according to claim 6, characterized in that detecting a user operation of selecting the first image identifier, processing the first image feature in the first image based on the second image feature, including: detecting a user operation of selecting the first image identifier, displaying a fourth image editing interface corresponding to the first image, the fourth image editing interface including at least one image identifier corresponding to at least one reference image corresponding to the first image feature, wherein the similarity between the at least one reference image corresponding to the at least one image identifier and the first region is greater than the similarity threshold, and the second image feature of the second region satisfies the preset condition; detecting a user operation of selecting a third image identifier from the at least one image identifier, processing the first image feature based on the second image feature in the first reference image corresponding to the third image identifier. The method of claim 1, wherein The preset condition includes a face score greater than a score threshold; wherein The face score is determined based on at least one of face rotation angle data, face occlusion degree data, face smile degree data, and eye opening amplitude data. The method of claim 8, wherein The detection of the presence of the first image feature in the first region of the first image that does not satisfy the preset condition includes: detecting the presence of at least one face region in the first image; obtaining at least one face score corresponding to the at least one face region; In a case where the face score corresponding to the first face region is less than or equal to the score threshold, it is detected that the first image feature that does not satisfy the preset condition exists in the first region in the first image. The method of claim 9, wherein The selecting the first reference image from the gallery comprises: In the gallery, N images adjacent to the first image and taken earlier than the first image are selected, and N images adjacent to the first image and taken later than the first image are selected, to obtain a first image set, wherein N is a positive integer. From each image in the first image set, an image whose similarity between an image feature vector and a feature vector of the first image is greater than a similarity threshold is selected to obtain a second image set. From each image in the second image set, an image whose similarity between a second region and the first region is greater than the similarity threshold is selected to obtain a third image set. Face scores of the second region in each image in the third image set are determined. From each image in the third image set, an image with the maximum face score of the second region is selected as the first reference image. The method of claim 10, wherein The processing of the first image feature in the first image based on the second image feature to obtain the second image that satisfies the preset condition comprises: In a case where the face occlusion degree in the first region is less than a first occlusion threshold and greater than a second occlusion threshold, the head image feature in the first image feature is replaced by the head image feature in the second image feature to obtain the second image, and the first occlusion threshold is greater than the second occlusion threshold. In a case where the face occlusion degree in the first region is less than the second occlusion threshold and the face rotation angle is greater than an angle threshold, the head image feature in the first image feature is replaced by the head image feature in the second image feature to obtain the second image. In a case where the face occlusion degree in the first region is less than the second occlusion threshold and the face rotation angle is less than the angle threshold, at least part of the facial feature in the first image feature that does not satisfy the preset condition is replaced by at least part of the facial feature in the second image feature to obtain the second image. The method of claim 11, wherein The replacing the head image feature in the first image feature by the head image feature in the second image feature to obtain the second image comprises: The head image feature in the first image feature is replaced by the head image feature in the second image feature to obtain a third image, and one or more of a texture feature, a color feature, and a brightness feature in the head image feature in the third image is adjusted to obtain the second image that satisfies the preset condition. The method of claim 11, wherein The replacing at least part of the facial feature in the first image feature that does not satisfy the preset condition by at least part of the facial feature in the second image feature to obtain the second image comprises: Replace at least part of the facial feature image features in the first image features that do not meet the preset condition with corresponding at least part of the facial feature image features in the second image features to obtain a fourth image, and adjust one or more of texture features, color features, and brightness features in the facial feature image features in the fourth image to obtain the second image that meets the preset condition. An electronic device, characterized by comprising: Comprising: A memory for storing instructions for one or more processors of the electronic device to execute, and a processor that is one of the one or more processors of the electronic device, configured to execute the image processing method of any one of claims 1-13. A readable storage medium characterized by, The readable storage medium has instructions stored thereon, and the instructions, when executed on an electronic device, cause the electronic device to execute the image processing method of any one of claims 1-13. A computer program product, characterized in that Comprising: Computer instructions that, when executed on an electronic device, cause the electronic device to execute the image processing method of any one of claims 1-13.

Images