Image processing method and device, storage medium and program product

Abstract

The invention relates to the technical field of image processing, and discloses an image processing method and device, a storage medium and a program product. The image processing method comprises the following steps: if a to-be-processed portrait meeting a processing condition (for example, the to-be-processed portrait has the characteristics of eye closure and the like which influence the beauty of the image) exists in the to-be-processed image, obtaining the face characteristics of the to-be-processed portrait; and selecting a reference image (such as a reference portrait with the same appearance as the to-be-processed portrait and open eyes) which has high similarity with the face features of the to-be-processed portrait and makes up for the image defects from the image library. And carrying out image restoration on the to-be-processed image based on the reference image to obtain a target image for overcoming the image defects, for example, replacing a part of image influencing the image beauty in the to-be-processed image with a corresponding part of image in the reference image. In this way, the overall attractiveness of the image can be improved, and the visual experience of a user watching the target image can be improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, apparatus, storage medium and program product. Background Technology

[0002] When users take photos, they often close their eyes, frown, or show signs of tension, which degrades image quality, affects the overall aesthetics of the image, and reduces the user's visual experience. For example, Figure 1 As shown, when user A and other users take photos using the camera function of mobile phone 10, if user A blinks while the camera is capturing the image, the image of user A in the captured image 101 will have a visual effect of closed eyes, thus affecting the overall aesthetics of image 101. Summary of the Invention

[0003] To address the aforementioned problems, this application provides an image processing method, apparatus, storage medium, and program product.

[0004] In a first aspect, this application provides an image processing method, the method comprising: acquiring a first image to be processed; detecting that a first image feature in a first region of the first image does not meet a preset condition, wherein the first region includes a first face region and the first image feature includes a first face feature; selecting a first reference image from an image library, wherein the similarity between a second region of the first reference image and the first region is greater than a similarity threshold, and the second image feature of the second region meets the preset condition; and processing the first image feature in the first image based on the second image feature to obtain a second image that meets the preset condition.

[0005] In this application, the first image can be the image to be processed mentioned in this application; the first region can be the region where the face of the first image to be processed mentioned in this application is located, wherein the region where the face is located includes at least the first face region; the first face region can be the face region of the first image to be processed mentioned in this application; the first image feature can be the image feature in the region where the face of the first image to be processed mentioned in this application is located, wherein the first image feature includes at least the first face feature; the first face feature can be the face feature of the first image to be processed mentioned in this application, such as eye contours, face contours, etc.; the first reference image can be the reference image corresponding to the first image feature mentioned in this application; the second region can be the face region in the first reference image mentioned in this application; the second image feature can be the image feature within the face region of the first reference image mentioned in this application; the second image can be the target image mentioned in this application.

[0006] In some implementations, the presence of a first image feature in the first region that does not meet the preset conditions can indicate that the image to be processed in the first region meets the image processing conditions, and image processing can be performed on the image to be processed.

[0007] In some implementations, after determining the image to be processed and its first image features, the electronic device can also acquire second image features (e.g., facial features) from other images in the image library, selecting an image with a high similarity to the first image features and that overcomes image defects as the first reference image. For example, if the image to be processed that meets the processing conditions is an image with closed eyes, the electronic device can select a reference image with open eyes that has the same facial features as the image to be processed from other images. Then, the electronic device can process the image to be processed based on the reference image. For example, it can remove the part of the image to be processed, copy and paste the same part of the reference image to the removed part of the image to be processed, and adjust the pasted edge position, texture, color difference, etc., so that the processed target image achieves an overall harmonious effect.

[0008] Thus, through the above image processing method, electronic devices can enhance the image to be processed, so that the final image can overcome image defects and improve the overall visual effect of the image.

[0009] In one possible implementation of the first aspect described above, the method further includes: detecting a user's image editing operation on a first image, obtaining the first image to be processed, and displaying a first image editing interface corresponding to the first image, the first image editing interface including an image processing button; corresponding to the presence of a first image feature in the first image and the presence of a first reference image in the image library, the image processing button is in a state that can respond to user operation instructions; corresponding to the absence of a first image feature in the first image, or the presence of a first image feature in the first image and the absence of a first reference image in the image library, the image processing button is in a state that cannot respond to user operation instructions.

[0010] In this application, the first image editing interface can be the image editing interface mentioned in this application that is displayed when the user clicks the image editing button. For example, the first image editing interface can be... Figure 2 The editing interface 202 is shown in Figure (b).

[0011] In some implementations, when a user views an image to be processed (e.g., in a photo album), and the electronic device detects the user clicking the image editing button, it first determines whether a portrait of the subject exists in the image that meets the processing conditions. If a portrait exists, it further determines whether a reference image exists in the images stored by the electronic device that has high image feature similarity to the portrait and compensates for the aforementioned image defects. If no portrait exists, or if a portrait exists but no reference image exists, the image processing button is unresponsive to user commands, for example, it is grayed out. If a portrait exists in the image and a reference image exists in the gallery, the image processing button is responsive to user commands, for example, it is highlighted. When the image processing button is responsive, and the electronic device detects the user clicking the image processing button, it can perform image restoration on the image to be processed based on the reference image to obtain the target image.

[0012] Therefore, compared to the image processing button being directly clickable (e.g., highlighted), which may result in no response when the user clicks the image processing button, the method described above, which makes the image processing button clickable only after determining that there is a portrait to be processed and a reference image, ensures that the electronic device can detect the user's click operation every time the user clicks the image processing button, thereby processing the image to be processed to obtain the target image and improving the user experience.

[0013] In one possible implementation of the first aspect above, corresponding to the image processing button being in a state capable of responding to user operation instructions, processing the first image features in the first image based on the second image features to obtain a second image that meets preset conditions includes: when an operation of clicking the image processing button by the user is detected, processing the first image features in the first image based on the second image features in the first reference image to obtain a second image.

[0014] In some implementations, when the electronic device detects a user clicking the image processing button, it triggers the device to perform image restoration based on the image features of the portrait to be processed, corresponding to the image features of the portrait in the reference image. For example, the eyes of the portrait to be processed can be replaced with the eyes of the corresponding portrait in the reference image. Furthermore, the electronic device can also make detailed adjustments to the texture, brightness, and other aspects of the restored portrait image.

[0015] In this way, the electronic device can respond to the user's operation command and perform image enhancement processing on the portrait to be processed in the image to obtain the target image that meets the preset conditions.

[0016] In one possible implementation of the first aspect described above, the method further includes: detecting that a first image feature in a first region of a first image does not meet a preset condition, and a third image feature in a third region does not meet a preset condition, wherein the first region includes a first face region, the first image feature includes a first face feature, the third region includes a second face region, and the third image feature includes a second face feature; selecting a first reference image and a second reference image from a library, wherein the similarity between the second region and the first region of the first reference image is greater than a similarity threshold, the second image feature of the second region meets the preset condition, and the similarity between the fourth region and the third region of the second reference image is greater than a similarity threshold, the fourth image feature of the fourth region meets the preset condition; processing the first image feature in the first image based on the second image feature, and processing the third image feature in the first image based on the fourth image feature, to obtain a second image that meets the preset condition.

[0017] In this application, the third region can be the area where the face of the second image to be processed mentioned in this application is located, wherein the area where the face is located includes at least a face region; the second face region can be the face region of the second image to be processed mentioned in this application; the third image feature can be the image feature in the area where the face of the second image to be processed mentioned in this application is located, wherein the third image feature includes at least a second face feature; the second face feature can be the face feature of the second image to be processed mentioned in this application; the fourth region can be the face region in the second reference image mentioned in this application; the fourth image feature can be the face feature in the second reference image mentioned in this application.

[0018] In some implementations, the image to be processed may include multiple portraits that meet the processing conditions, such as a first portrait and a second portrait. The electronic device can select a first reference image from a library that has a high similarity to the first image features and overcomes image defects, and a second reference image that has a high similarity to the third image features and overcomes image defects. Based on the second image features of the first reference image, the second image features in the first image are processed; based on the fourth image features of the second reference image, the fourth image features in the first image are processed, thereby obtaining a target image that meets preset conditions.

[0019] Thus, when there are multiple human figures to be processed in an image, each human figure can still be processed separately using the image processing method mentioned in this application, thereby obtaining a target image that overcomes image defects.

[0020] In one possible implementation of the first aspect above, the method includes: detecting a user's image editing operation on a first image, obtaining the first image to be processed, and displaying a second image editing interface corresponding to the first image, the second image editing interface including an image processing button; corresponding to the presence of a first image feature and a third image feature in the first image, and the presence of a first reference image and a second reference image in the image library, the image processing button is in a state that can respond to user operation instructions; corresponding to the absence of a first image feature and a third image feature in the first image, or the absence of a first reference image and a second reference image in the image library, the image processing button is in a state that cannot respond to user operation instructions.

[0021] In this application, the second image editing interface can be the image editing interface mentioned in this application that is displayed when the electronic device detects a user clicking the image editing button. For example, the second image editing interface can be... Figure 3 The editing interface 301 is shown in Figure (a).

[0022] In some implementations, when the electronic device detects that the image to be processed does not contain the subject image, or that there is no reference image in the image library, the image processing button is in a state where it cannot respond to user commands. When the image to be processed contains the subject image, and a reference image exists among the stored images, the image processing button is in a state where it can respond to user commands. When the image processing button is in a state where it can respond to user commands, when the electronic device detects that the user has clicked the image processing button, the electronic device can perform image restoration on the corresponding subject image based on the reference image to obtain the target image.

[0023] Therefore, compared to the image processing button being directly clickable (e.g., highlighted), which may result in no response when the user clicks the image processing button, the method described above, which makes the image processing button clickable only after determining that there is a portrait to be processed and a reference image, ensures that the electronic device can detect the user's click operation every time the user clicks the image processing button, thereby processing the image to be processed to obtain the target image and improving the user experience.

[0024] In one possible implementation of the first aspect described above, corresponding to the image processing button being in a state capable of responding to user operation commands, processing the first image features in the first image based on the second image features, and processing the third image features in the first image based on the fourth image features to obtain a second image that meets preset conditions, includes: detecting the user's click on the image processing button and displaying a third image editing interface corresponding to the first image, the third image editing interface including a first image identifier corresponding to the first image features and a second image identifier corresponding to the third image features; detecting the user's selection of the first image identifier and processing the first image features in the first image based on the second image features; detecting the user's selection of the second image identifier and processing the third image features in the first image based on the fourth image features.

[0025] In this application, the third image editing interface can be the image editing interface mentioned in this application that is displayed when the electronic device detects a user clicking the image processing button. For example, the third image editing interface can be... Figure 3 The editing interface 302 shown in Figure (b) is as follows; the first image identifier can be the image identifier corresponding to the first image to be processed mentioned in this application, for example, the first image identifier can be... Figure 3 The image identifier 3021 corresponding to the first image to be processed 3012 shown in Figure (b) is used; the second image identifier can be the image identifier corresponding to the second image to be processed mentioned in this application, for example, the second image identifier can be... Figure 3 The image identifier 3022 corresponding to the second human image 3013 shown in Figure (b) is shown.

[0026] In some implementations, the electronic device can display image identifiers corresponding to each image to be processed. Then, based on the image identifier selected by the user, the image to be processed corresponding to that image identifier is processed. In this way, when processing the image to be processed, the electronic device can process only a portion of the image to be processed based on the user's wishes, thereby overcoming image defects while also meeting the user's needs.

[0027] In one possible implementation of the first aspect described above, detecting a user's selection of a first image identifier and processing the first image features in the first image based on second image features includes: detecting a user's selection of a first image identifier and displaying a fourth image editing interface corresponding to the first image, the fourth image editing interface including at least one image identifier corresponding to at least one reference image corresponding to the first image features, wherein the similarity between at least one second region corresponding to at least one reference image and the first region is greater than a similarity threshold, and the second image features of the second region satisfy a preset condition; detecting a user's selection of a third image identifier from at least one image identifier and processing the first image features based on the second image features in the first reference image corresponding to the third image identifier.

[0028] In this application, the fourth image editing interface can be the image editing interface mentioned in this application that is displayed when the user clicks on the image icon corresponding to the portrait to be processed. For example, the fourth image editing interface can be... Figure 3 The editing interface 303 shown in Figure (c) is shown in the figure; the third image identifier may be the image identifier corresponding to the first reference image mentioned in this application.

[0029] In some implementations, when determining the corresponding reference image for each portrait to be processed, the electronic device can also determine multiple reference images that are similar to the portrait to be processed and overcome image defects. The user then selects which reference image to use for processing. In this way, when processing the portrait to be processed, the electronic device can determine the reference image based on the user's wishes, thus enabling the target image to overcome image defects while also meeting the user's needs.

[0030] In one possible implementation of the first aspect above, the preset condition includes that the face score is greater than the score threshold; wherein the face score is determined based on at least one of face rotation angle data, face occlusion degree data, face smile degree data, and eye opening amplitude data.

[0031] In some implementations, the presence of a first image feature in the first region that does not meet preset conditions indicates that the image to be processed within the first region meets the image processing conditions, allowing image processing to be performed on the image. The preset conditions can be arbitrarily set. For example, preset conditions could include eyes open, a smiling face, etc. Alternatively, preset conditions could include a face score higher than a scoring threshold, where the face score can be determined based on at least one of face rotation angle data, face occlusion degree data, face smiling degree data, and eye opening amplitude data. This application does not limit this.

[0032] In one possible implementation of the first aspect described above, detecting the presence of a first image feature in a first region of the first image that does not meet a preset condition includes: detecting the presence of at least one face region in the first image; obtaining at least one face score corresponding to the at least one face region; and detecting the presence of a first image feature in a first region of the first image that does not meet a preset condition when the face score corresponding to the first face region is less than or equal to a score threshold.

[0033] In some implementations, the electronic device first detects whether a face region exists in the image to be processed; for example, it detects at least one face region in a first image. Then, the electronic device determines the face score corresponding to each face region. When the electronic device determines that the face score corresponding to the first face region is less than or equal to a score threshold, it indicates that the electronic device has detected a first image feature in the first region that does not meet a preset condition.

[0034] In one possible implementation of the first aspect described above, selecting a first reference image from the image library includes: selecting N adjacent images from the image library that are adjacent to the first image and were captured earlier than the capture time of the first image, and selecting N adjacent images that are adjacent to the first image and were captured later than the capture time of the first image, to obtain a first image set, where N is a positive integer; selecting images from the images in the first image set whose image feature vectors have a similarity greater than a similarity threshold with the feature vectors of the first image, to obtain a second image set; selecting images from the images in the second image set whose second region has a similarity greater than a similarity threshold with the first region, to obtain a third image set; determining the face score of the second region in each image in the third image set; and selecting the image with the highest face score in the second region from each image in the third image set as the first reference image.

[0035] In this application, the first image set can be the image set mentioned in this application, which is obtained by selecting N consecutive images forward and backward from the position of the image to be processed in the album; the second image set can be the image set mentioned in this application, which is composed of images similar to the first image selected from the first image set; the third image set can be the image set mentioned in this application, which is composed of images with face regions similar to the face regions of the image to be processed selected from the second image set.

[0036] In some implementations, the electronic device first selects N consecutive images (e.g., 5 images forward and 5 images backward) from the position of the image to be processed in the album to obtain a first image set. Next, the electronic device calculates the similarity between the feature vectors of each image in the first image set and the feature vector of the image to be processed, and selects images with a similarity greater than a similarity threshold (e.g., 95%) to obtain a second image set. Then, the electronic device iterates through the images in the second image set, selecting images where the face regions are similar to the face region of the image to be processed, to obtain a third image set. Finally, the electronic device performs face scoring on the face regions similar to the image to be processed in each image in the third image set, and selects the image with the highest face score as the reference image corresponding to the image to be processed. In this way, the reference image corresponding to the image to be processed can be selected from the image library.

[0037] In one possible implementation of the first aspect described above, processing the first image features in the first image based on the second image features to obtain a second image that meets preset conditions includes: replacing the head image features in the first image features with the head image features in the second image features when the degree of face occlusion in the first region is less than a first occlusion threshold and greater than a second occlusion threshold, thereby obtaining a second image, wherein the first occlusion threshold is greater than the second occlusion threshold; replacing the head image features in the first image features with the head image features in the second image features when the degree of face occlusion in the first region is less than the second occlusion threshold and the face rotation angle is greater than an angle threshold, thereby obtaining a second image; and replacing at least some facial feature image features in the first image features that do not meet the preset conditions with the corresponding at least some facial feature image features in the second image features when the degree of face occlusion in the first region is less than the second occlusion threshold and the face rotation angle is less than an angle threshold, thereby obtaining a second image.

[0038] In this application, the first occlusion threshold can be the maximum occlusion threshold mentioned in this application; the second occlusion threshold can be the minimum occlusion threshold mentioned in this application.

[0039] In some implementations, the restoration strategy for the portrait to be processed can include: replacing head image features and replacing facial feature images. When the electronic device determines that the degree of face occlusion in the portrait to be processed is large or the face rotation angle is large, it can execute the restoration strategy of replacing head image features; when the electronic device determines that the degree of face occlusion in the portrait to be processed is small and the face rotation angle is small, it can execute the restoration strategy of replacing some facial feature images. In this way, the portrait to be processed can be restored, improving the aesthetics of the image.

[0040] In one possible implementation of the first aspect above, replacing the head image features in the first image features with the head image features in the second image features to obtain the second image includes: replacing the head image features in the first image features with the head image features in the second image features to obtain the third image, and adjusting one or more of the texture features, color features, and brightness features in the head image features of the third image to obtain the second image that meets the preset conditions.

[0041] In this application, the third image can be an image obtained by replacing the head image features mentioned in this application.

[0042] In some implementations, electronic devices can also adjust one or more details of texture features, color features, and brightness features in the third image, thereby improving the overall harmony and visual effect of the target image.

[0043] In one possible implementation of the first aspect above, replacing at least some facial feature image features in the first image features that do not meet the preset conditions with corresponding at least some facial feature image features in the second image features to obtain a second image includes: replacing at least some facial feature image features in the first image features that do not meet the preset conditions with corresponding at least some facial feature image features in the second image features to obtain a fourth image; and adjusting one or more of the texture features, color features, and brightness features in the facial feature image features of the fourth image to obtain a second image that meets the preset conditions.

[0044] In this application, the fourth image feature can be the image obtained by replacing the facial features mentioned in this application.

[0045] In some implementations, the electronic device can also adjust one or more details of the texture features, color features, and brightness features in the fourth image, thereby improving the overall harmony and visual effect of the target image.

[0046] Secondly, this application provides an electronic device, including: a memory and a processor, wherein the memory is used to store instructions executed by one or more processors of the electronic device, and the processor is one of one or more processors of the electronic device, for executing the image processing method mentioned in this application.

[0047] Thirdly, this application provides a readable storage medium storing instructions that, when executed on an electronic device, cause the electronic device to perform the image processing method mentioned in this application.

[0048] Fourthly, this application provides a computer program product, including: computer instructions, which, when executed on an electronic device, cause the electronic device to perform the image processing method mentioned in this application.

[0049] The beneficial effects of the second to fourth aspects mentioned above can be referred to the relevant descriptions in the first aspect and various possible implementations of the first aspect, which will not be repeated here. Attached Figure Description

[0050] Figure 1 According to some embodiments of this application, a schematic diagram of a scenario in which a user blinking causes a decrease in image quality is shown;

[0051] Figure 2 According to some embodiments of this application, a schematic diagram of a first interface for image processing of an image to be processed is shown;

[0052] Figure 3 According to some embodiments of this application, a schematic diagram of a second interface for image processing of an image to be processed is shown;

[0053] Figure 4 According to some embodiments of this application, a schematic diagram of the system structure used in an image processing method is shown;

[0054] Figure 5 According to some embodiments of this application, a schematic flowchart of an image processing method is shown;

[0055] Figure 6A According to some embodiments of this application, a flowchart illustrating the display of an image enhancement entry point is shown;

[0056] Figure 6B According to some embodiments of this application, a schematic diagram of the interaction of various modules for displaying an image enhancement entry is shown;

[0057] Figure 7A According to some embodiments of this application, a schematic diagram of a process for processing an image to be processed directly based on a reference image is shown;

[0058] Figure 7B According to some embodiments of this application, a schematic diagram of the interaction between modules is shown when processing an image to be processed directly based on a reference image.

[0059] Figure 8A According to some embodiments of this application, a schematic diagram of a first process for processing an image to be processed based on a user-selected reference image is shown;

[0060] Figure 8BAccording to some embodiments of this application, a schematic diagram of a second process for processing an image to be processed based on a user-selected reference image is shown;

[0061] Figure 8C According to some embodiments of this application, a schematic diagram of the interaction between modules is shown when processing an image to be processed based on a user-selected reference image.

[0062] Figure 9 According to some embodiments of this application, a schematic diagram of the hardware structure of an electronic device is shown. Detailed Implementation

[0063] The illustrative embodiments of this application include, but are not limited to, an image processing method, apparatus, storage medium, and program product.

[0064] The background of the image processing method provided in the embodiments of this application is briefly described below.

[0065] As mentioned earlier, users often close their eyes or frown when taking photos, resulting in less aesthetically pleasing images and negatively impacting the user's visual experience. For example, ... Figure 1 As shown, when user A takes a group photo with other users, if user A blinks while the camera is capturing the image, it will cause user A's image in the captured image 101 to appear with closed eyes, thus affecting the overall aesthetics of image 101.

[0066] To address the aforementioned problems, this application provides an image processing method. In this method, if the image to be processed contains a target image (e.g., a portrait with closed eyes) that meets processing conditions (e.g., features affecting image aesthetics such as closed eyes), the electronic device can acquire the image features (e.g., facial features) of the target image and select a reference image (e.g., a reference image with the same facial features as the portrait to be processed but with open eyes) from images stored in the electronic device. Based on the reference image, image restoration is performed on the image to be processed to obtain a target image that overcomes the aforementioned image defects. The image restoration method can be to replace the portion of the image to be processed that affects aesthetics with the corresponding portion of the reference image. For example, for the target image being a portrait with closed eyes, the eyes of the portrait to be processed can be removed and replaced with the eyes of the reference image to obtain the target image. Alternatively, based on the features of the non-eye portion of the portrait to be processed and the features of the eye portion of the reference image, an open-eyed portrait can be regenerated, and the generated open-eyed portrait can replace the initial closed-eyed portrait in the image to be processed to obtain the target image. This can improve the overall aesthetics of the image and enhance the user's visual experience when viewing the target image.

[0067] In some embodiments, after an electronic device receives a user's shooting instruction and captures an image, it can directly use the captured image as the image to be processed. By performing the aforementioned image processing method on the captured image to be processed, a target image that overcomes image defects can be directly displayed. For example, after capturing the image to be processed, the electronic device can first determine whether there is a target image in the image that meets the processing conditions. If there is no target image, the electronic device can directly display the captured image to the user. If there is a target image, the electronic device can select a reference image from the stored images that has a high similarity to the image features (e.g., facial features) of the target image and compensates for the aforementioned image defects, and perform image restoration on the image to be processed based on the reference image to obtain the target image. Thus, when a user captures an image using the electronic device, the electronic device can directly perform image processing on the captured image to generate and display a target image that overcomes image defects.

[0068] In other embodiments, a user can view an image stored in an electronic device. During this viewing, if the electronic device detects an image processing instruction, such as clicking an image editing button corresponding to the image, the device can use the viewed image as the image to be processed and perform the aforementioned image processing method to generate and display a target image that overcomes the image defects. For example, upon receiving an image processing instruction, the electronic device can first determine if there is a target image in the image to be processed that meets the processing conditions. If a target image exists, it further determines if there is a reference image in the stored images that has a high similarity to the image features (e.g., facial features) of the target image and compensates for the aforementioned image defects. If there is no target image, or if there is a target image but no reference image, the user is prompted that image processing cannot be performed on the image to be processed. If a target image exists in the image to be processed and a reference image exists in the stored images, the electronic device can perform image repair on the image to be processed based on the reference image to obtain the target image. Thus, when the electronic device detects an image processing instruction from the user for an image to be processed, it can directly perform image processing on the image to be processed, generating and displaying a target image that overcomes the image defects.

[0069] In other embodiments, during the process of a user viewing an image to be processed (e.g., viewing an image to be processed in a photo album), when the electronic device detects an image processing instruction, such as detecting a user clicking the image's edit button, it can first determine whether there is a target image in the image to be processed that meets the processing conditions. If a target image exists, it further determines whether there is a reference image among the images stored by the electronic device that has a high similarity to the image features (e.g., facial features) of the target image and compensates for the aforementioned image defects. If there is no target image, or if there is a target image but no reference image, the image processing button may be in a state where it cannot respond to user operation instructions. If there is a target image in the image to be processed and a reference image exists among the stored images, the image processing button is in a state where it can respond to user operation instructions. When the image processing button is in a state where it can respond to user operation instructions, and the electronic device detects a user clicking the image processing button, the electronic device can perform image repair on the image to be processed based on the reference image to obtain the target image. Thus, compared to the image processing button being directly in a clickable state that can respond to user operation commands (e.g., in a highlighted state), which may result in no response when the user clicks the image processing button, the above method, which makes the image processing button clickable only after determining that there is a target to be processed and a reference image, ensures that the electronic device can detect the user's click operation every time the user clicks the image processing button, thereby processing the image to be processed to obtain the target image and improving the user experience.

[0070] Therefore, it is evident that in this application, the timing of executing the image processing method on the image to be processed is not limited. To facilitate the description of the image processing method provided in this application, the following embodiments will use the example of setting the image processing button to a state responsive to user operation commands only after determining that a target to be processed and a reference image exist.

[0071] Furthermore, in some embodiments, the image to be processed can be any image, such as images of people, landscapes, or animals. This application does not limit the image content of the image to be processed.

[0072] In other embodiments, the target to be processed can be any object in the image that meets the processing conditions. For example, the target to be processed can be any object in the image, such as a person, animal, or plant, and this application does not limit this.

[0073] To better illustrate the image processing method provided in this application, in the following embodiments, the image to be processed is a person image, the target to be processed is the person image to be processed in the person image, and the person image in the reference image corresponding to the target to be processed is described as a reference person image, in order to illustrate the image processing method provided in this application.

[0074] As described above, if the image to be processed does not contain the image of the person to be processed, or if the image of the person to be processed exists but there is no reference image among the images stored in the electronic device, the electronic device will not display the image processing entry for that image. For example, the image processing button corresponding to the image processing entry will be in a state where it cannot respond to user operation commands (e.g., it will be grayed out). When the electronic device determines that the image to be processed contains the image of the person to be processed and that there is a reference image among the images stored in the electronic device, the electronic device will display the image processing entry. For example, the image processing button corresponding to the image processing entry will be in a state where it can respond to user operation commands (e.g., it will be highlighted).

[0075] As an example, the following combines Figure 2 The illustrated interface, using a closed-eye person in a group photo as the subject of processing and an open-eye person as the reference image, provides a brief introduction to the image processing method provided in this application. Figure 2 In this context, the aforementioned image processing button can be described as an "image enhancement button".

[0076] like Figure 2 As shown in Figure (a), a user can view images 2011 in the photo album on the display interface 201 of the mobile phone 10. The display interface 201 may include multiple controls such as an edit button 2012, a share button 2013, and a delete button 2014, allowing the user to perform various operations on the images 2011. For example, the user can send the image 2011 to others by clicking the share button 2013; or delete the image 2011 by clicking the delete button 2014. Furthermore, when the mobile phone 10 detects that the user has clicked the edit button 2012, such as... Figure 2 As shown in Figure (b), the mobile phone 10 displays an editing interface 202, allowing the user to adjust the filter, size, etc. of the image 2011 using multiple editing controls in the editing interface 202.

[0077] Specifically, when the mobile phone 10 detects that the user clicks the edit button 2012, it triggers the operation of "detecting whether there is a portrait to be processed (such as a portrait with closed eyes) in the image 2011 that meets the processing conditions." Furthermore, when the mobile phone 10 detects that there is a portrait to be processed 2021 in the image 2011, it also determines whether there is a reference portrait in the gallery that has the same appearance as the portrait to be processed 2021 and overcomes image defects (such as open eyes). If there is no portrait to be processed, or if there is a portrait to be processed (such as a portrait with closed eyes 2021) but no reference portrait in the album, the image beautification button 2022 in the editing interface 202 is in a state where it cannot respond to user operation commands (e.g., in a grayed-out state). If there is a portrait to be processed (such as a portrait with closed eyes 2021) and a reference portrait in the album, the image beautification button 2022 in the editing interface 202 is in a state where it can respond to user operation commands (e.g., in a highlighted state). In addition, in Figure 2 The editing interface 202 shown in Figure (b) may include multiple editing controls such as an image enhancement button 2022, a filter button 2023, and a crop button 2024, so that the user can perform various editing operations on the image 2011. For example, the user can enter the filter adjustment interface by clicking the filter button 2023 to adjust the filter of the image 2011 (e.g., yellow filter, gray filter, etc.); the user can click the image enhancement button 2022 to make the mobile phone 10 perform the image processing method mentioned in this application on the image 2011; the user can also crop the image 2011 by clicking the crop button 2024.

[0078] When the image enhancement button 2022 is in a state that can respond to user operation commands, for example, when it is highlighted, the mobile phone 10 can trigger image restoration based on the image features of the reference portrait and the corresponding image feature parts of the portrait to be processed 2021, thereby obtaining a target image that overcomes image defects. For example, the eyes of the portrait to be processed 2021 can be replaced with the eyes of the reference portrait. After the mobile phone 10 completes image processing on image 2011, such as... Figure 2 As shown in (c), the mobile phone 10 displays a display interface 203. In this display interface 203, a target image 2031 is displayed, obtained after image processing of the image 2011 to be processed. All figures in the target image 2031 have their eyes open. Furthermore, in... Figure 2In the display interface 203 shown in Figure (c), a prompt box 2032 can also be displayed to prompt the user that "image enhancement is complete". In addition, when the mobile phone 10 detects that the user clicks the finish button 2033, the mobile phone 10 can save the processed target image 2031 and exit the display interface 203; or, when the mobile phone 10 detects that the user clicks the cancel button 2034, the mobile phone 10 does not save the processed image 2031 and directly exits the display interface 203.

[0079] In other embodiments, the image to be processed may further include multiple images of people to be processed. For example, if a first image of a person to be processed that does not meet a preset condition is detected in a first region of the image to be processed, and a second image of a person to be processed that does not meet a preset condition is detected in a third region, then the electronic device can acquire the first image features of the face region of the first image to be processed and the second image features of the face region of the second image to be processed, respectively. Furthermore, the electronic device can select, from the stored images, a first reference image whose at least some regions have a high similarity to the face region of the first image to be processed and meet the preset condition, and a second reference image whose at least some regions have a high similarity to the face region of the second image to be processed and meet the preset condition. Finally, the electronic device can process the corresponding image features of the first image to be processed based on the partial image features of the first reference image, and process the corresponding image features of the second image to be processed based on the partial image features of the second reference image, thereby obtaining a target image that meets the preset condition.

[0080] In other embodiments, when determining the corresponding reference image for each image to be processed, the electronic device can also determine multiple reference images that are similar to the image to be processed and overcome image defects, and then the user can select which reference image to process the image to be processed.

[0081] As an example, the following combines Figure 3 The illustrated interface, using a group photo of a person with closed eyes and no smile as the subject of processing and a smiling person with open eyes as the reference image, provides a brief introduction to the image processing method provided in this application. Specifically, in... Figure 3 In the text, the image processing button mentioned above can still be described as an "image enhancement button".

[0082] Users can view images 3011 in the photo album using mobile phone 10. Furthermore, while the user is viewing image 3011, if mobile phone 10 detects that the user has clicked the image editing button (see above for details),... Figure 2 When performing the operation shown in Figure (a), as described in Figure (a), Figure 3As shown in Figure (a), the mobile phone 10 displays an editing interface 301. The editing interface 301 may include multiple editing controls such as an image enhancement button 3014, a filter button 3015, and a crop button 3016, allowing the user to perform various editing operations on the image 3011. For example, the user can enter the editing interface by clicking the image enhancement button 3014 to enhance the image 3011 using the image processing methods mentioned in this application; the user can enter the filter adjustment interface by clicking the filter button 3015 to adjust the filter of the image 3011 (e.g., yellow filter, gray filter, etc.); and the user can also crop the image 3011 by clicking the crop button 3016.

[0083] When the mobile phone 10 detects that the user clicks the edit button, it can trigger the mobile phone 10 to detect whether there is a portrait to be processed in the image to be processed 3011 that meets the processing conditions, and whether there is a reference portrait corresponding to the portrait to be processed in the image library. When the mobile phone 10 detects that there is a portrait to be processed in the image to be processed 3011, for example, the first portrait to be processed 3012 (a portrait with closed eyes) and the second portrait to be processed 3013 (a portrait without a smile), and determines that there is a first reference portrait corresponding to the first portrait to be processed 3012 and a second reference portrait corresponding to the second portrait to be processed 3012 in the image library, the image beautification button 3014 in the editing interface 301 is in a state that can respond to the user's operation command (e.g., in a highlighted state). When the mobile phone 10 detects that there is no portrait to be processed in the image to be processed 3011 or no reference image in the gallery, such as no first portrait to be processed and no second portrait to be processed, or no first reference portrait and no second reference portrait in the gallery, the image beautification button 3014 in the editing interface 301 is in a state of not responding to user operation commands (e.g., in a grayed-out state).

[0084] Specifically, when the image enhancement button 3012 is in a state capable of responding to user operation commands, when the mobile phone 10 detects that the user has clicked the image enhancement button 3014, such as... Figure 3As shown in Figure (b), the mobile phone 10 displays an editing interface 302. The editing interface 302 displays image identifiers corresponding to the images to be processed. For example, the editing interface 302 displays a first image identifier 3021 corresponding to the first image to be processed 3012 and a second image identifier 3022 corresponding to the second image to be processed 3013, so that the user can perform image processing on the corresponding image by clicking the image identifier. For example, when the mobile phone 10 detects the user's selection of the first image identifier 3021, the mobile phone 10 can process the partial image features corresponding to the first image to be processed 3012 based on partial image features in the first reference image. Alternatively, when the mobile phone 10 detects the user's selection of the second image identifier 3022, the mobile phone 10 can process the partial image features corresponding to the second image to be processed 3013 based on partial image features in the second reference image.

[0085] In some embodiments, when the mobile phone 10 detects that the user has tapped the first image identifier 3021, such as... Figure 3 As shown in Figure (c), the mobile phone 10 can also display an editing interface 303. The editing interface 303 displays image identifiers corresponding to one or more reference portraits corresponding to the first image to be processed 3012 corresponding to the first image identifier 3021. The face areas of each reference portrait are the same as the face area of ​​the first image to be processed 3012, and the reference portraits are smiling with their eyes open. The user can arbitrarily select one image identifier from the one or more image identifiers so that the mobile phone 10 can process the first image to be processed 3012 based on the reference portrait corresponding to the selected image identifier. For example, if the mobile phone 10 detects that the user clicks on image identifier 3031, the mobile phone 10 can process the first image to be processed 3012 using the reference portrait corresponding to the image identifier 3031 selected by the user. After the mobile phone 10 processes the first image to be processed 3012, as... Figure 3 As shown in Figure (d), the mobile phone 10 displays an editing interface 304. In the editing interface 304, the target image 3041 obtained after image processing of the first portrait 3012 to be processed in the image 3011 is displayed.

[0086] Similarly, in the editing interface 304, the user can also click the second image identifier 3022 to process the second image to be processed 3013 corresponding to the second image identifier 3022. The mobile phone 10 can display image identifiers of one or more reference images corresponding to the second image to be processed 3013, allowing the user to select which reference image to use for processing the image to be processed 3013. Further details are omitted here. Furthermore, when the mobile phone 10 detects that the user clicks the complete button 3042, the mobile phone 10 can save the processed target image 3041 and exit the editing interface 304; or, when the mobile phone 10 detects that the user clicks the cancel button 3043, the mobile phone 10 will not save the processed target image 3041 and will directly exit the editing interface 304.

[0087] Thus, electronic devices can use the image processing method provided in this application to enhance the image to be processed, so that the final image obtained can overcome image defects and improve the overall visual effect of the image.

[0088] Before detailing the image processing method provided in the embodiments of this application, we will first combine it with... Figure 4 The system structure diagram shown provides a brief introduction to the algorithm modules in the system structure used in the image processing method provided in this application.

[0089] The system architecture applicable to the image processing method provided in this application may include cloud-side devices and edge devices. The cloud-side device may be a physical server or server cluster, used to deploy the relevant algorithms, code, parameters, and other data of the image processing method provided in this application. The edge device may be any electronic device such as a mobile phone, tablet, laptop, or computer, used to store the image to be processed. The edge device can send the image to be processed and an image processing request to the cloud-side device. The cloud-side device can perform image processing on the received image and send the processed target image to the edge device for display.

[0090] Among them, such as Figure 4 As shown, the edge device may include a media information library, a gallery, a visual middleware software development kit (SDK), and a visual middleware Android platform. TM The application package (APK) can include a smart cloud microservice module, an algorithm service module, a cloud service module, and an object storage service (OBS) module. The image processing method provided in this application can be implemented through the interaction between the various modules in the terminal device and the cloud device.

[0091] The image library in the edge device can be an image database such as the photo album of the aforementioned electronic device (e.g., mobile phone 10), and the image to be processed is an image from the image library. The edge device can send an image processing request to the cloud device to perform data analysis and image enhancement on the image to be processed.

[0092] The data analysis may include, but is not limited to: human body detection to detect the presence of a human figure in the image to be processed; facial information detection to detect the state of facial features; determining the presence of a human figure to be processed based on the state of facial features, and classifying the human figure to be processed by labeling it to determine whether the human figure to be processed has open eyes or is not smiling; image feature extraction of the human figure to be processed and other images in the image library; and clustering to determine whether there are reference human figures in the image library with the same appearance as the human figure to be processed. Furthermore, the image enhancement process may include, but is not limited to: similar image screening to initially screen images with high image feature similarity to the human figure to be processed; face comparison, when there are multiple human figures to be processed and multiple similar images are also selected, matching each human figure to be processed and its corresponding reference human figure based on facial features; face / head swapping: replacing the face, head, or individual facial features in the human figure to be processed with the face, head, or individual facial features in the reference human figure; and face enhancement: adjusting the texture, skin tone, and other details of the face of the human figure to be processed after face swapping.

[0093] like Figure 4 As shown, the edge device may also include a media information library for storing face-related data or image-related data. The data stored in the media information library can be data obtained by processing images to be processed using image processing algorithms. When performing the same processing procedure on the same image, the data stored in the media information library can be directly accessed without running the image processing algorithms again. For example, if data B1 is obtained by processing image B using algorithm A, data B1 can be stored in the media database. If the same processing is needed for image B, data B1 stored in the media database can be directly accessed without running algorithm A again.

[0094] It is understood that to apply the image processing method provided in this application to an edge device, it is also necessary to install relevant program code packages (Visual Platform SDK and Visual Platform APK) on the edge device. When a user needs to process the image to be processed, the edge device can run the code in the relevant program code packages. During the execution of the code in the relevant program code packages, the edge device will call the cloud computing resources in the cloud device and perform image processing on the image to be processed using the algorithms, parameters, and other data in the cloud device.

[0095] The visual middleware SDK can include multiple application programming interfaces (APIs) to connect the image library and the visual middleware APK. For example, it may include APIs for repair strategy judgment, face information detection, image feature extraction, portrait generation, face generation plugins, and face enhancement. Through the algorithm interfaces provided by each API, the corresponding algorithms in the visual middleware APK can be called.

[0096] The visual platform APK can include a face detection APK, a human body detection APK, a portrait generation APK, and a portrait enhancement APK. The face detection APK can include: a face bounding box detection APK for detecting the presence of faces; a face information detection APK for detecting facial features and other attributes; an image feature extraction APK for extracting portrait features; and a face clustering APK for identifying reference portraits that match the portrait to be processed. The human body detection APK can include a human keypoint detection APK for detecting human keypoints; and a face-body mapping APK for matching faces with corresponding human bodies. The portrait generation APK can include a repair strategy judgment APK and a portrait generation APK. The portrait enhancement APK can include a face quality detection APK to determine whether the generated target portrait overcomes image defects; it can also include a face parsing APK for extracting facial feature points; and it can also include a face enhancement APK for adjusting facial texture and other details. When each algorithm in the visual platform APK receives a call request from the corresponding API, the visual platform APK can send the request to a cloud computing management platform (such as...). Figure 4 The Zhongzhi Cloud-Microservices module sends image processing requests to utilize cloud computing resources for image processing, thereby reducing resource consumption on the edge devices.

[0097] It is understood that cloud computing resources can be used to run and deploy the related algorithms of the image processing method provided in this application to improve data processing efficiency. For example, Figure 4 As shown, the smart cloud-microservice module in the cloud-side device can be a cloud computing management platform used to interact with the edge device. It can also call relevant algorithms in the algorithm service module to process the image to be processed. The algorithm service module can include a restoration strategy judgment algorithm and a portrait restoration algorithm. The restoration strategy judgment algorithm can perform the following functions: determine the face rotation angle and determine whether the face is occluded. The portrait restoration algorithm can perform the following functions: generate the processed portrait, adjust the texture and other details of the processed face to achieve face enhancement, and process color differences in the image.

[0098] The Smart Cloud-Microservices module includes multiple APIs that provide interfaces for calling various service algorithms within the Algorithm Service module. For example, when the Smart Cloud-Microservices module receives an image to be processed and an image processing request from the edge device, it can run the Startup Service API, indicating the initiation of algorithm calls within the Algorithm Service module. Then, the Smart Cloud-Microservices module can use the algorithm interface provided by the Repair Strategy API to call the repair strategy judgment algorithm in the Algorithm Service module to determine and store the repair strategy. For instance, it can determine whether the portrait in the image to be processed needs a specific facial feature replacement or the entire head replacement. The Smart Cloud-Microservices module can also use the algorithm interface provided by the Portrait Repair API to call the portrait repair algorithm in the Algorithm Service module and repair the portrait based on the determined repair strategy.

[0099] Among them, after the portrait to be processed is repaired based on the repair strategy, the repaired portrait can also be stored in the OBS module of the cloud-side module, so that the portrait images, portrait information and other data in the OBS module can be directly called for processing in the future, or the data in the OBS module can be returned to the image library in the end device for display.

[0100] Specifically, when the Smart Cloud-Microservices module finishes running its service API, the stored data can be deleted, and the execution of the image processing method provided in this application can be terminated.

[0101] In addition, the Smart Cloud-Microservices module can also call the risk control algorithm in the cloud service module to perform risk control processing on image data, thereby determining whether the image data is safe or whether there is a risk of leakage.

[0102] Thus, through the interaction between the above modules and the execution of the module code, the image processing method provided in this application can be realized to beautify the image to be processed, so that the final image obtained can overcome image defects and thus improve the overall visual effect of the image.

[0103] It is understood that the image processing method provided in this application embodiment can be applied to electronic devices or third-party applications such as beauty photo albums. The applicable electronic devices can be any terminal device such as mobile phones, computers, tablets, laptops, netbooks, augmented reality (AR) devices, virtual reality (VR) devices, wearable devices, and in-vehicle smart terminals. The electronic devices can also be physical servers or cloud devices, such as x86 servers, ARM servers, etc.; or they can be virtual machines (VMs) implemented based on general-purpose physical servers combined with network functions virtualization (NFV) technology. A virtual machine refers to a complete computer system simulated by software, possessing complete hardware system functions and running in a completely isolated environment. This application does not impose any restrictions on the specific type of electronic device.

[0104] The following is based on Figure 5 The flowchart shown provides a brief introduction to the image processing method mentioned in the embodiments of this application. This image processing method can be applied to electronic devices, as described above. Figure 1 The mobile phone 10 shown in the image. Figure 5 As shown, specifically, the method includes the following steps:

[0105] S501: Obtain the first image to be processed.

[0106] In this application, the first image may be the image to be processed mentioned in this application.

[0107] In some embodiments, after the electronic device receives a user's shooting instruction to capture an image, it can directly use the captured image as the first image to be processed, and then perform subsequent steps S502 to S504 on the first image so as to directly display the target image that overcomes the image defects.

[0108] In other embodiments, a user can view an image stored in an electronic device. While the user is viewing the image, if the electronic device receives an image processing instruction, for example, detecting that the user clicks an image editing button, the electronic device can use the image viewed by the user as the first image to be processed, and then perform subsequent steps S502 to S504 on the first image to generate and display a target image that overcomes the image defects.

[0109] S502: A first image feature that does not meet a preset condition is detected in a first region of the first image, wherein the first region includes a first face region and the first image feature includes a first face feature.

[0110] In some embodiments, the first region may be the region where the face of the first image to be processed mentioned in this application is located, wherein the region where the face is located includes at least a face region; the first face region may be the face region of the first image to be processed mentioned in this application; the first image feature may be the image feature in the region where the face of the first image to be processed mentioned in this application is located, wherein the first image feature includes at least a first face feature; the first face feature may be the face feature of the first image to be processed mentioned in this application, such as features like eye contours and face contours.

[0111] In some embodiments, the presence of a first image feature in the first region that does not meet preset conditions indicates that the image to be processed in the first region meets the image processing conditions, and image processing can be performed on the image to be processed. The preset conditions can be arbitrarily set. For example, preset conditions could be eyes open, a smiling face, etc. Alternatively, preset conditions could be a face score higher than a scoring threshold, wherein the face score can be determined based on at least one of face rotation angle data, face occlusion degree data, face smiling degree data, and eye opening amplitude data.

[0112] In some embodiments, the electronic device can first detect whether a face region exists in the image to be processed. For example, at least one face region is detected in a first image. Then, the electronic device can determine at least one face score corresponding to the at least one face region. When the electronic device determines that the face score corresponding to the first face region is less than or equal to a score threshold, it indicates that the electronic device has detected a first image feature in the first region that does not meet a preset condition.

[0113] S503: Select a first reference image from the image library, wherein the similarity between the second region of the first reference image and the first region is greater than a similarity threshold, and the second image features of the second region meet preset conditions.

[0114] In some embodiments, the first reference image may be a reference image corresponding to the first image feature mentioned in this application; the second region may be a face region in the first reference image mentioned in this application that is similar to the face region of the portrait to be processed; and the second image feature may be a face feature in the second region of the first reference image mentioned in this application.

[0115] In some embodiments, after determining the portrait to be processed and its first image features through step S502, the electronic device can further acquire the second image features (e.g., facial features) of other images in the image library, and select an image with a high similarity to the first image features that overcomes image defects as a first reference image. For example, if the portrait to be processed is a portrait with closed eyes, the electronic device can select a reference portrait with open eyes that has the same facial features as the portrait to be processed from other images, and then process the portrait to be processed based on the reference portrait. The similarity threshold can be set arbitrarily, and this application does not limit it.

[0116] In some embodiments, after the electronic device determines the human image to be processed through step S502 and the reference image through step S503, the electronic device can directly execute the subsequent step S504 to process the first image based on the reference image to obtain a second image that overcomes the image defects.

[0117] In other embodiments, after the electronic device determines the image to be processed through step S502 and the reference image through step S503, it can display an image processing entry point. The electronic device will only execute the subsequent step S504, processing the first image based on the reference image to obtain a second image that overcomes image defects, when it detects that the user has entered the image processing entry point—for example, by detecting that the user has clicked the image processing button corresponding to the entry point. If the electronic device does not detect that the user has entered the image processing entry point—for example, by not detecting that the user has clicked the image processing button corresponding to the entry point—then the electronic device will not execute the subsequent step S504.

[0118] S504: Process the first image features in the first image based on the second image features to obtain a second image that meets the preset conditions.

[0119] In some embodiments, the second image may be the target image mentioned in this application.

[0120] In some embodiments, the electronic device can perform image restoration on the image to be processed based on a selected reference image to obtain a target image. For example, the part that needs to be processed in the portrait to be processed can be removed, and then the same part of the reference image can be copied and pasted onto the removed part of the image to be processed. The position of the pasted edge, as well as the texture, color difference, etc., can be adjusted so that the processed target image achieves an overall harmonious effect.

[0121] Thus, electronic devices can use the image processing method provided in this application to enhance the image to be processed, so that the final image obtained can overcome image defects and improve the overall visual effect of the image.

[0122] It is understandable that when the area affecting the aesthetics of the image to be processed is the area of ​​the human figure to be processed, the parts that need to be processed may be the face (facial features and facial structure, etc.), the head (face and hair, etc.), limbs, etc. For ease of description, the following embodiments all use the face of the human figure to be processed as an example to describe the image processing method provided in this application in detail.

[0123] In some embodiments, as described above Figure 2 As shown in Figures (a) and (b), when implementing the image processing method provided in this application, the electronic device can first determine whether there exists a reference image in the image library that is similar to the face of the person to be processed and overcomes image defects. If such a reference image exists, the image processing button will be in a state that can respond to user operation commands; if not, the image processing button will be in a state that cannot respond to user operation commands.

[0124] For example, when an electronic device detects that a user is editing an image (e.g., a group photo), it first selects N consecutive images (e.g., 5 images forward and 5 images backward) from the image in the album to obtain a first image set. Next, the electronic device calculates the similarity between the feature vectors of each image in the first image set and the feature vector of the image to be processed, and selects images with a similarity greater than a similarity threshold (e.g., 95%) to obtain a second image set. Then, the electronic device iterates through the images in the second image set, selecting images where facial regions are similar to the image to be processed, thus obtaining a third image set. Finally, the electronic device performs face scoring on the facial regions similar to the image to be processed in each image in the third image set, selecting the image with the highest face score as the reference image corresponding to the image to be processed. When a reference image is not present, it means that there is no reference image with the same face as the person being processed that overcomes the image defects before and after the image to be processed, and the image processing button is in a state where it cannot respond to user operation commands; if a reference image is present, it means that there is a reference image with the same face as the person being processed that overcomes the image defects before and after the image to be processed, and the image processing button is in a state where it can respond to user operation commands.

[0125] Specifically, refer to Figure 6A The flowchart shown provides a simplified description of the process by which an electronic device determines that an image processing button is in a state capable of responding to user commands. Specifically, the method includes the following steps:

[0126] 601: User editing of image A was detected.

[0127] In some embodiments, when a user views image A, an image editing operation can be performed. At this time, the electronic device can detect the user clicking the image editing button corresponding to image A, and then execute the image processing method provided in this application on image A to be processed. For example, as described above. Figure 2 As shown in Figure (a), mobile phone 10 can detect the user's action of clicking the edit button 2012.

[0128] 602: Determine whether a human face exists in image A. If yes, proceed to step 603; otherwise, proceed to step 611.

[0129] In some embodiments, after detecting a user's editing operation on image A, the electronic device can first invoke a face detection algorithm to detect whether a face exists in image A. For example, the electronic device can determine whether a face exists in image A by determining whether image features such as eye contours and nose contours are present. If no face exists, the image processing button must be in a state where it cannot respond to user operation commands, and image processing of image A cannot be performed. If a face exists, image features F1 of image A can be further extracted.

[0130] 603: Extract image features F1 from image A.

[0131] In some embodiments, when a face is present in the image A to be processed, more detailed facial image features F1 of the face region in image A can be extracted. For example, image features such as eye shape, size, and pupil color can be obtained.

[0132] 604: Select N photos before and after image A to obtain image set B.

[0133] In some embodiments, the electronic device can continuously acquire N images forward from the position of the image to be processed A in the album; and continuously acquire N images backward from the position of the image to be processed A in the album, thereby obtaining an image set B. The number of images acquired can be arbitrarily set, and this application does not limit this.

[0134] 605: Determine if image set B is empty. If yes, proceed to step 611; otherwise, proceed to step 606.

[0135] In some embodiments, after selecting N photos before and after the image A to be processed, the electronic device can further determine whether the image set B is empty. If it is empty, it means that the image library only contains the image A to be processed, and the face in the image A to be processed cannot be processed by other images. Therefore, it is determined that the image processing button needs to be in a state where it cannot respond to user operation commands. If the image set B is not empty, it can traverse each image in the image set B through step 606 and select the image with a high similarity to the image A to be processed.

[0136] 606: Traverse each image in image set B and select the image with the highest similarity to image A to obtain image set C.

[0137] In some embodiments, since the images in image set B selected in step 604 are not compared with the image to be processed A, there may be images in image set B that are completely unrelated to image A. Therefore, it is necessary to further filter the images in image set B through steps 6061 to 6065 to select images with a high similarity to image A to obtain image set C.

[0138] The process of identifying images in image set B that have a high similarity to image A can be referred to as steps 6061 to 6065:

[0139] 6061: Obtain the image features Fi of each image.

[0140] In some embodiments, the electronic device can acquire image features of the face region of each image in the image set B, such as image features of eye shape, size, pupil color, etc.

[0141] 6062: Calculate the similarity between each image feature Fi and the image feature F1 of image A.

[0142] In some embodiments, after obtaining the image feature Fi of an image in the image set B, the similarity between the image feature Fi and the image feature F1 can be calculated. The similarity can be determined using Euclidean distance, cosine similarity, etc. This application does not limit this method.

[0143] 6063: Determine if the similarity is greater than the similarity threshold. If not, proceed to step 6064; if yes, proceed to step 6065.

[0144] In some embodiments, the similarity threshold can be set arbitrarily, and this application does not limit it.

[0145] 6064: Iterate through the next image in image set B.

[0146] In some embodiments, if it is determined through step 6063 above that the similarity between the image feature Fi of the face region of the image and the image feature F1 of the face region of the image to be processed A is less than a similarity threshold, it indicates that the image is unrelated to the image to be processed A, and the process continues to traverse the next image in image set B. The electronic device can continue to execute steps 6061 to 6063 above on the next image in image set B.

[0147] 6065: Add the image to image set C.

[0148] In some embodiments, if it is determined through step 6063 above that the similarity between the face region image feature Fi of the image and the face region image feature F1 of the image to be processed A is greater than the similarity threshold, it indicates that the image is related to the image to be processed A and can be added to the image set C.

[0149] In some embodiments, the electronic device may repeatedly execute steps 6061 to 6065 to determine whether the similarity between the face region of each image in image set B and the face region of image A is greater than a similarity threshold, and add the images with similarity greater than the similarity threshold to image set C, thereby obtaining image set C.

[0150] 607: Determine if image set C is empty. If yes, proceed to step 611; otherwise, proceed to step 608.

[0151] In some embodiments, the electronic device can determine whether the image set C is empty. If it is empty, it means that there is no image in the image library similar to the portrait to be processed in the image A to be processed. Therefore, it is impossible to process the portrait to be processed in image A using other images, and it is determined that the image processing button needs to be in a state where it cannot respond to user operation commands. If the image set C is not empty, it is necessary to traverse each image in the image set C and match each face in the image with each face in the image to be processed.

[0152] 608: Traverse each image in image set C and match each face in each image in image set C with the face to be processed in image A.

[0153] In some embodiments, if an image A' in image set C contains multiple faces (e.g., face 11, face 12), and the image to be processed A also contains multiple faces (e.g., face 31, face 32), the electronic device needs to match the face to be processed in image A with the face in image A'. For example, if the face to be processed in image A is the same as face 11 in image A', the matched face image (face 11) can be added to image set D.

[0154] The process of matching each face in each image in image set C with the face to be processed can be referred to steps 6081 to 6083.

[0155] 6081: Extract facial information from each face in the image.

[0156] In some embodiments, when an image A' in the image set C contains multiple faces (such as face 11, face 12, etc.), the electronic device needs to extract the facial information of each of the multiple faces.

[0157] 6082: Match each face with the face to be processed in image A.

[0158] In some embodiments, multiple clustering algorithms can be used to match multiple faces (such as face 11, face 12, etc.) in image A' with the face to be processed. This application does not limit this. For example, k pixels can be randomly selected from face 11 and face 12 respectively as cluster centers to form two clusters. Then, some pixels of the face to be processed are assigned to the cluster containing the nearest cluster center. Next, the mean of all pixel data in each cluster is calculated as the new cluster center of that cluster, and the remaining pixels of the face to be processed are again assigned to the cluster containing the nearest cluster center. The above process is repeated. If all pixels of the face to be processed fall in the cluster containing face 11, it means that face 11 and the face to be processed are the same person; if none of the pixels of the face to be processed fall in the cluster containing face 11 or face 12, it means that face 11, face 12 and the face to be processed are not the same person, that is, there is no face in image A' that matches the face to be processed.

[0159] 6083: Add the matched face image to image set D.

[0160] In some embodiments, if the face 11 in image A' is the same person as the face to be processed, then the face image corresponding to face 11 is added to image set D.

[0161] In some embodiments, the electronic device may repeatedly perform steps 6081 to 6083 to match each face in each image in the image set C with the face to be processed in image A.

[0162] 609: Determine if image set D is empty. If not, proceed to step 610; if yes, proceed to step 611.

[0163] In some embodiments, the electronic device can determine whether the reference image set D is empty. If it is empty, it is determined that the image processing button needs to be in a state where it cannot respond to user operation commands; if the reference image set D is not empty, it is determined that the image processing button needs to be in a state where it can respond to user operation commands.

[0164] 610: Displays the image enhancement entry point.

[0165] In some embodiments, when an electronic device displays an image enhancement entry point, it may be an image processing button (as described above). Figure 2 The image enhancement button (2022) shown in Figure (b) is in a clickable state that can respond to user operation commands, such as being in a highlighted state.

[0166] 611: The image enhancement entry is not supported.

[0167] In some embodiments, when the electronic device does not support displaying an image enhancement entry point, it may be an image processing button (as described above). Figure 2 The image enhancement button (2022) shown in Figure (b) is in an unclickable state that cannot respond to user operation commands, such as being grayed out.

[0168] Thus, when the electronic device detects that a user is editing an image to be processed, it can determine whether to display the image enhancement entry based on the above process. If the image enhancement entry is not required to be displayed, it means that the image to be processed cannot be processed using the image processing method provided in this application; if the image enhancement entry is required to be displayed, it means that when the electronic device detects that a user clicks the button corresponding to the image enhancement entry, the image to be processed can be processed using the image processing method provided in this application.

[0169] The following is in conjunction with the above. Figure 4 The system architecture diagram of the image processing method shown is based on Figure 6B The illustrated time-series interaction flowchart describes in detail the interaction process of the aforementioned image enhancement entry point. For example, Figure 6B As shown, the interactive process of displaying the image enhancement entry can be implemented through the image library, media information library, visual middleware SDK, face information detection APK, image feature extraction APK, face clustering APK, and user-to-user interaction. Specifically, Figure 6B The interactive process shown includes the following steps:

[0170] 620: The user performs editing operations on the images to be processed in the gallery.

[0171] In some embodiments, a user can edit an image while viewing it. In this case, the electronic device can detect the user's editing operation on the image, thereby triggering the electronic device to execute the image processing method provided in this application on the image. For example, refer to the above... Figure 2 As shown in Figure (a), while the user is viewing the image to be processed 2011, the electronic device detects that the user clicks the edit button 2012, which can then be displayed. Figure 2 The editing interface 202 is shown in Figure (b).

[0172] 621: The media information database detects whether facial data in the image to be processed is stored.

[0173] In some embodiments, the media information library can be used to store data obtained by image processing algorithms from processing images to be processed. For example, if face data is extracted from image A to be processed using a face detection algorithm, the extracted face data can be stored in the media information library. If face data from image A to be processed needs to be retrieved again, the face data stored in the media information library can be read directly, avoiding the electronic device from running the face detection algorithm again and saving computing resources. Therefore, when a user edits an image to be processed, the electronic device can first detect whether face data from the image to be processed is stored in the media information library. If face data is stored, it can be directly used for subsequent processing; if face data is not stored, steps 6231 to 6237 need to be executed to extract and store the face data from the image to be processed.

[0174] 622: The media information database sends facial data to the image database.

[0175] In some embodiments, if the media information library pre-stores facial data in the image to be processed, the detected facial data can be sent to the image library. Then, step 624 is performed, selecting N images before and after the image to be processed to determine whether there are reference images before and after the image to be processed that are similar to the facial data of the image to be processed and can compensate for image defects.

[0176] 6231: The image library has determined that no facial data is stored in the media information database.

[0177] In some embodiments, if the media information library does not send the face data of the image to be processed to the image library, it can be determined that the media information library does not have the face data of the image to be processed pre-stored. Thus, the face data can be obtained through subsequent steps 6232 to 6237 and stored in the media information library.

[0178] 6232: The image gallery does not display the image enhancement entry to the user.

[0179] In some embodiments, if it is determined that no facial data is stored in the media information library, the image enhancement entry point may be temporarily not displayed to the user. For example, the image enhancement button may be grayed out and unable to respond to user commands. The image enhancement entry point will only be displayed when subsequent steps determine that facial data exists in the image to be processed, and that there are reference images before and after the image to be processed that are similar to the facial data of the image to be processed and can compensate for image defects. For example, the image processing button may be highlighted and able to respond to user commands.

[0180] 6233: The image library calls the face detection API in the visual middleware SDK.

[0181] In some embodiments, an API is a data interface. When a gallery calls the face detection API, it can invoke the face detection algorithm.

[0182] 6234: The visual middleware SDK sends a start command to the face information detection APK.

[0183] In some embodiments, when the visual middleware SDK detects that the image library calls the face information detection API in the visual middleware SDK, it can send a start command to the face information detection APK to run the face information detection algorithm to obtain face data.

[0184] 6235: Face Information Detection APK obtains face data.

[0185] In some embodiments, the electronic device can run a face detection algorithm in the face detection APK to acquire face data. For example, it can acquire key point data such as the eyes and mouth of the face; and data such as the face rotation angle. The feature data of the face can be acquired using any image processing algorithm. For example, a convolutional neural network can be used to extract edge pixel data, color data, texture data, and facial feature position and shape data of the image. This application does not limit this.

[0186] 6236: The face information detection APK returns face data to the image library.

[0187] In some embodiments, after the face information detection APK obtains the face data in the image to be processed, it can send the face data to the image library.

[0188] 6237: The image library stores facial data to the media information database.

[0189] In some embodiments, after the image library obtains the face data sent by the face information detection APK, it stores it in the media information library. If the face data of the image to be processed needs to be applied again, the stored face data can be read directly from the media information library without having to perform steps 6233 to 6236 again.

[0190] In some embodiments, if the image library receives face data sent by the media information library in step 622, then steps 6231 to 6237 do not need to be executed, and the face data sent by the media information library can be used directly for image processing; if the image library does not receive face data sent by the media information library, then it needs to obtain the face data and store the face data in the media information library through steps 6231 to 6237.

[0191] 624: The image library selects N images before and after the image to be processed.

[0192] In some embodiments, the image library may also acquire N consecutive images forward from the position of the image to be processed in the image library; and acquire N consecutive images backward from the position of the image to be processed in the image library. Here, N is a positive integer, and the number of images acquired can be arbitrarily set, which is not limited in this application.

[0193] 625: The media information database checks whether the feature vectors of the selected images are stored.

[0194] In some embodiments, the electronic device can first detect whether the feature vectors of the selected images are stored in the media information library. If the feature vectors are stored, they can be directly used for subsequent processing; if the feature vectors are not stored, steps 6271 to 6276 need to be executed to extract and store the feature vectors of the selected images.

[0195] 626: The media information database sends feature vectors to the image library.

[0196] In some embodiments, if the feature vectors of the selected images are pre-stored in the media information library, the detected feature vectors can be sent to the library. Then, step 628 is performed to calculate the similarity between the selected image and the image to be processed.

[0197] 6271: The image library has determined that no feature vectors are stored in the media information library.

[0198] In some embodiments, if the media information library does not send the feature vector to the image library, the image library can determine that the media information library does not have the feature vector pre-stored. Then, subsequent steps 6272 to 6276 need to be performed to obtain the feature vector and store it in the media information library.

[0199] 6272: The image library calls the image feature extraction API in the Visual Platform SDK.

[0200] In some embodiments, an API is a data interface. When a gallery calls the image feature extraction API, it can invoke the image feature extraction algorithm.

[0201] 6273: The Visual Platform SDK sends a startup command to the Image Feature Extraction APK.

[0202] In some embodiments, when the Visual Platform SDK detects that the image library calls the image feature extraction API in the Visual Platform SDK, it can send a start command to the image feature extraction APK to run the image feature extraction algorithm to obtain the feature vectors of each selected image.

[0203] 6274: Image Feature Extraction APK obtains feature vectors.

[0204] In some embodiments, an image feature vector is a mathematical representation used to describe the content of an image. It typically contains information such as color, texture, and shape in the image. The feature vector of a selected image can be obtained using any image processing algorithm. For example, a convolutional neural network can be used to extract edge pixel data, color data, texture data, facial feature positions, and shape data. This application does not limit this approach.

[0205] 6275: Image feature extraction APK returns feature vectors to the image library.

[0206] In some embodiments, after the image feature extraction APK obtains the feature vectors of each selected image, it sends the feature vectors to the image library.

[0207] 6276: The image library stores feature vectors to the media information library.

[0208] In some embodiments, after the image library obtains the feature vector sent by the image feature extraction APK, it stores it in the media information library. If the feature vector of the image needs to be applied again, the stored feature vector can be read directly from the media information library without having to perform steps 6272 to 6275 again.

[0209] In some embodiments, if the image library receives a feature vector sent by the media information library in step 626, then there is no need to execute steps 6271 to 6276, and the feature vector can be used directly for image processing; if the image library does not receive a feature vector sent by the media information library, then it is necessary to obtain the feature vector and store it in the media information library through steps 6271 to 6276.

[0210] 628: The image library calculates the similarity between the selected image and the image to be processed.

[0211] In some embodiments, the electronic device can calculate the similarity between the selected image and the image to be processed using any method. For example, the distance between the feature vectors of the selected image and the feature vectors of the image to be processed can be obtained using Euclidean distance, cosine similarity, etc. This application does not limit this.

[0212] 629: The image library adds images with high similarity to the image set.

[0213] In some embodiments, if the similarity is lower than the similarity threshold, it means that the face in the selected image is not the same person as the face in the image to be processed. Therefore, the selected image cannot be used to process the image to be processed, and it is necessary to continue to traverse the next image.

[0214] In some embodiments, if the similarity is higher than a similarity threshold, the selected image can be added to the image set.

[0215] In some embodiments, the electronic device may repeatedly execute steps 625 to 629 above to traverse each of the N images before and after the image to be processed, and determine whether each of the selected 2N images is similar to the image to be processed.

[0216] 630: The image library calls the clustering algorithm in the face clustering APK.

[0217] In some embodiments, image clustering refers to matching multiple faces in a selected image with faces in the image to be processed. For example, if faces A and B exist in the image to be processed, and faces A1 and B1 exist in the selected image, then a clustering calculation is needed to match each face. For example, if face A and face A1 match, it means that face A and face A1 are the same person.

[0218] In some embodiments, multiple clustering algorithms can be used to match multiple faces in the selected image with faces in the image to be processed. This application does not limit this. For example, face A exists in the image to be processed, and faces A1 and B1 exist in the selected image. Then, k pixels can be randomly selected from faces A1 and B1 respectively as cluster centers to form two clusters. Then, some pixels of face A are assigned to the cluster containing the nearest cluster center. Next, the mean of all pixel data in each cluster is calculated as the new cluster center of that cluster, and the remaining pixels of face A are again assigned to the cluster containing the nearest cluster center. The above process is repeated. If all pixels of face A fall in the cluster containing face A1, it means that face A1 and face A are the same person; if none of the pixels of face A fall in the cluster containing face A1 or face B1, it means that face A1, B1 and face A are not the same person, that is, there is no face in the selected image that matches the face in the image to be processed.

[0219] In some embodiments, the gallery may first detect whether clustering results are stored in the media information database. If stored, the clustering results are directly invoked. If not stored, step 630 is executed to invoke the clustering algorithm.

[0220] 631: The face clustering APK returns the clustering results to the image library.

[0221] In some embodiments, after the face clustering APK performs clustering processing, the clustering results can be returned to the image library.

[0222] 632: If the clustering result is empty, the image library will not display the image enhancement entry to the user.

[0223] In some embodiments, if the image library determines that the clustering result is empty, it means that the faces in the selected images cannot be matched with the faces in the image to be processed. That is, it means that there is no reference image with the same face as the image to be processed, and the image beautification entry is not displayed. For example, the image processing button is in a state where it cannot respond to user operation commands.

[0224] 633: If the clustering result is not empty, the image library will display the image enhancement entry to the user.

[0225] In some embodiments, if the image library determines that the clustering result is not empty, it means that the faces in the selected images can be matched with the faces in the image to be processed. That is, it indicates that there is a reference image with the same face as the image to be processed, and then the image beautification entry is displayed, for example, the image processing button is in a state that can respond to user operation commands.

[0226] Thus, when a user edits an image to be processed, the electronic device can determine whether to display the image enhancement entry based on the above interaction process. If the image enhancement entry is not required to be displayed, it means that the image to be processed cannot be processed using the image processing method provided in this application; if the image enhancement entry is required to be displayed, it means that when the electronic device detects that the user clicks the image processing button corresponding to the image enhancement entry, the image to be processed can be processed using the image processing method provided in this application.

[0227] In some embodiments, when the electronic device detects that a user has clicked the image processing button corresponding to the image enhancement entry, for example, the user can click the aforementioned... Figure 2 The image enhancement button 2022 shown in Figure (b) allows the electronic device to directly process the image to be processed based on the reference image to generate the target image (as described above). Figure 2 The target image 2031 shown in Figure (c) is displayed to the user.

[0228] Specifically, the following example uses the face as the part of the portrait to be processed, combined with... Figure 7A The flowchart shown provides a simplified description of the process by which an electronic device processes an image to be processed based on a reference image to generate a target image. Specifically, the method includes the following steps:

[0229] 701: User clicked the image processing button.

[0230] In some embodiments, when an electronic device displays an image processing entry point, the user can click the image processing button corresponding to that entry point. At this time, the electronic device can detect the user's click on the image processing button and then perform image restoration on the image to be processed based on a reference image. For example, as described above... Figure 2As shown in Figure (b), mobile phone 10 can detect the user's action of clicking the image enhancement button 2022.

[0231] 702: Traverse the image to be processed and the reference image to obtain the extended image set.

[0232] In some embodiments, when processing the image to be processed and the reference image, the electronic device may not acquire the entire portrait data, but only acquire the image data of the face region for processing.

[0233] The process of expanding each face frame to obtain face region image data can be referred to steps 7021 to 70222.

[0234] 7021: Determine the list of people in each image.

[0235] In some embodiments, the electronic device may first determine a list of people in each image, for example, determining the number of people in the image to be processed and the position of each person.

[0236] 7022: Iterate through each portrait in the portrait list to obtain the expanded image set.

[0237] In some embodiments, if an image includes multiple portraits, each portrait can be traversed separately, and the outer region of the face frame of each portrait can be added to the outer image set.

[0238] The process of traversing each portrait in the portrait list can be referred to in steps 70221 to 70222.

[0239] 70221: Expand the area of ​​the face frame of the portrait.

[0240] In some embodiments, the electronic device can first expand the area of ​​the face frame of the portrait. For example, the selected expansion area can be twice as wide and twice as high as the face frame.

[0241] 70222: Extract the image within the expanded region and add the image to the expanded image set.

[0242] In some embodiments, for the extended region obtained in step 70221 above, the image within the extended region can be extracted and added to the extended image set for subsequent processing.

[0243] In some embodiments, for multiple portraits in an image, steps 70221 and 70222 can be repeated to obtain images within the outer region of each portrait.

[0244] In some embodiments, for multiple reference images and images to be processed, steps 7021 to 70222 can be repeatedly executed to obtain images within the outer region of each portrait in each image. Thus, when processing portraits, it is not necessary to acquire the entire portrait data, but only the image data within the outer region (face region) can be acquired for processing.

[0245] 703: Traverse the face information in the extended image set to obtain the face to be processed and the corresponding reference face.

[0246] In some embodiments, after traversing the images in the image to be processed and the reference image set to obtain the extended image set, the electronic device may also traverse each face image in the extended image set to determine the replacement face.

[0247] Specifically, steps 7031 to 7032 can be used to determine the face to be processed and the corresponding reference face.

[0248] 7031: Sort the same face based on face score.

[0249] In some embodiments, the electronic device can sort the same face from high to low or from low to high based on individual face scores. These face scores can be determined based on factors such as the degree of eye opening, the angle of facial rotation, and the degree of smile.

[0250] 7032: Identify the faces to be processed with lower scores and the corresponding faces with higher scores.

[0251] In some embodiments, among the sorted groups of faces, the face with the lower score in the image to be processed is selected, and the reference face with the highest score corresponding to that face is selected. For example, faces A and B exist in the image to be processed, faces A1 and B1 exist in reference image 1, and faces A2 and B2 exist in reference image 2. The electronic device can sort the same faces in step 7031 (e.g., after sorting, they can be AA1A2, B1BB2), and then select the face with the lower score in the image to be processed (e.g., face A) in step 7032, and select the reference face with the highest score corresponding to that face (e.g., face A2).

[0252] In some embodiments, the electronic device may repeatedly execute steps 7031 and 7032 to determine the face to be processed with a low face score and the corresponding reference face that can be replaced in the image to be processed by traversing the face information in the extended image set.

[0253] 704: Determine if the reference face is empty. If yes, proceed to step 705; otherwise, proceed to step 706.

[0254] In some embodiments, the reference face is the face with a higher score determined in step 7032 above.

[0255] In some embodiments, if the reference face is determined to be empty, it means that image beautification of the face to be processed cannot be performed. If the reference face is not empty, beautification of the face to be processed can be performed based on the reference face.

[0256] 705: It has been determined that image enhancement cannot be performed on the face being processed.

[0257] In some embodiments, if the reference face is determined to be empty, it means that image enhancement of the face to be processed cannot be performed.

[0258] 706: Enhance the face to be processed based on the reference face.

[0259] In some embodiments, if the reference face is not empty, the face to be processed can be beautified based on the reference face. For example, the face portion of the image to be processed can be cut out, and then the reference face can be copied and pasted onto the face position of the image to be processed.

[0260] 707: Adjust the image after facial beautification to obtain the target image and display it.

[0261] In some embodiments, after beautifying the face to be processed through the above step 706, the electronic device can also adjust the position of the pasted edge, as well as the texture, color difference, etc., so that the processed target image achieves an overall harmonious effect.

[0262] In this way, when the user edits the image to be processed, the electronic device can enhance the image to obtain the target image based on the above process.

[0263] The following is in conjunction with the above. Figure 4 The system architecture diagram of the image processing method shown is based on Figure 7B The illustrated time-series interaction flowchart describes in detail the interactive process of enhancing the image to obtain the target image. For example, ... Figure 7B As shown, the interactive process of beautifying the image to obtain the target image can be achieved through an image library, a FaceGenerate plugin, a smart cloud microservice module, a restoration strategy judgment algorithm service module, a portrait restoration algorithm service module, an OBS module, and user-to-user interaction. Specifically, Figure 7B The interactive process shown includes the following steps:

[0264] 720: The user clicks the image processing button on the image to be processed in the gallery.

[0265] In some embodiments, as described above Figure 2 As shown in Figure (b), the user can click the image enhancement button 2022. At this time, the electronic device can detect the user's click on the image enhancement button 2022, thereby triggering the electronic device to perform subsequent steps to repair image 2011, thus generating... Figure 2 The target image 2031 is shown in Figure (c).

[0266] 721: The image library sends a service initialization command to the face generation plugin.

[0267] In some embodiments, the electronic device can first send a service initialization command to the face generation plugin installed in the electronic device, so that the face generation plugin can send a service initialization command to the smart cloud-microservice module in the cloud computing resources, so that the smart cloud-microservice module can perform operations such as loading image processing related algorithms, setting parameters, and establishing a connection with the electronic device.

[0268] 722: The face generation plugin sends a service initialization command to the Smart Cloud-Microservices module.

[0269] In some embodiments, after receiving the service initialization instruction sent by the image library, the face generation plugin will send the instruction to the Smart Cloud-Microservices module.

[0270] 723: Smart Cloud - Microservices module loads related resources.

[0271] In some embodiments, after receiving the service initialization instruction sent by the face generation plugin, the Smart Cloud-Microservices module will initialize and load resources such as image processing related algorithms and parameters.

[0272] 724: The Smart Cloud - Microservices module returns the initialization result to the gallery.

[0273] In some embodiments, after the Smart Cloud-Microservices module loads the relevant resources, it will return the loading result to the image library through the face generation plugin to notify the image library whether the service initialization has been completed.

[0274] 725: Select the portrait to be restored and the corresponding list of reference portraits from the image library.

[0275] In some embodiments, the electronic device can determine whether there exists a portrait to be repaired in the image to be processed that meets the processing conditions, and one or more reference portraits that overcome image defects and have a high similarity to the portrait to be repaired. If there is no portrait to be repaired, or if there is a portrait to be repaired but no corresponding reference portrait, the execution of the relevant algorithm ends. If there is a portrait to be repaired and a corresponding reference portrait, the electronic device can call the repair strategy algorithm service module to perform portrait repair.

[0276] 726: The image library sends a command to the face generation plugin to call the repair strategy judgment algorithm service module.

[0277] In some embodiments, when an electronic device calls the repair strategy judgment algorithm service module in cloud computing resources to determine the portrait repair strategy, it first needs to send a call instruction to the face generation plugin, and then the face generation plugin sends the call instruction to the repair strategy judgment algorithm service module through the smart cloud microservice module.

[0278] 727: Face generation plugin sets image identifier (sessionid).

[0279] In some embodiments, before calling the restoration strategy judgment algorithm service module to determine the portrait restoration strategy, the face generation plugin can also mark the image to be processed. For example, the image to be processed can be numbered so that the image to be processed can be directly called by calling the image identifier in the subsequent processing.

[0280] 728: The face generation plugin sends a repair strategy judgment request to the Smart Cloud-Microservices module.

[0281] In some embodiments, when the repair strategy judgment algorithm service module in the cloud computing resources is invoked to determine the portrait repair strategy, the electronic device can first send a repair strategy judgment request to the smart cloud-microservice module in the cloud computing resources, and then send the request to the repair strategy judgment algorithm service module through the smart cloud-microservice module.

[0282] 729: The Smart Cloud-Microservices module sends a repair strategy judgment request to the repair strategy judgment algorithm service module.

[0283] In some embodiments, after receiving a repair strategy judgment request, the Smart Cloud-Microservices module can send the request to the Repair Strategy Judgment Algorithm Service Module.

[0284] 730: The repair strategy judgment algorithm service module performs face analysis and other operations, and determines the repair strategy based on angle information, occlusion information, etc.

[0285] In some embodiments, the repair strategy judgment algorithm service module can acquire data such as the face rotation angle and face occlusion status of the portrait to be processed, and then determine the repair strategy based on the angle information and occlusion information. For example, if the repair strategy judgment algorithm service module determines that the face rotation angle is greater than a preset angle threshold, such as when the face of the portrait to be processed is facing to the sides, the repair strategy judgment algorithm service module can determine that the portrait to be processed cannot be processed, and directly send the result of not being able to process it to the image library for display to the user. As another example, if the repair strategy judgment algorithm service module determines that the face rotation angle is less than a preset angle threshold, and the face is occluded, the corresponding repair strategy can be to replace the entire face portion of the portrait to be processed. As yet another example, if the repair strategy judgment algorithm service module determines that the face rotation angle is less than a preset angle threshold, the face is not occluded, and a single facial feature meets the processing conditions (e.g., eyes are closed), the corresponding repair strategy can be to replace a certain facial feature of the portrait to be processed (e.g., replace the eyes). This application does not limit this.

[0286] In other embodiments, the repair strategy judgment algorithm service module can also select the image with the highest face score from the reference image list as the reference image based on face scoring, so as to repair the image to be processed based on the reference image. For example, the repair strategy judgment algorithm service module can score each face in the reference image list based on data such as face rotation angle, eye opening degree, and face smile amplitude, and then select the image with the highest score as the reference image.

[0287] 731: Fixed the storage of data from the strategy judgment algorithm service module to the OBS library.

[0288] In some embodiments, the repair strategy determination algorithm service module can directly store the determined repair strategy in the cloud computing resource database. If the repair strategy needs to be called again, it can be directly read from the OBS library without running the repair strategy determination algorithm again.

[0289] 732: The repair strategy judgment algorithm service module sends the repair strategy to the image library.

[0290] In some embodiments, the repair strategy judgment algorithm service module can send the repair strategy to the image library. If the repair strategy is that the face of the person to be processed cannot be repaired, the image library will display the strategy to the user. If the repair strategy is that the face of the person to be processed can be repaired, the image library can call the image repair algorithm service module to perform the repair.

[0291] 733: The image library indicates to the user that the face of the person to be processed is rotated at a large angle or is obscured.

[0292] In some embodiments, when the image library receives a repair strategy, if the repair strategy is that the face of the portrait to be processed has a large rotation angle or is heavily obscured, making it impossible to repair the face of the portrait to be processed, the image library can display the repair strategy to the user.

[0293] 734: The image library sends a command to the face generation plugin to invoke the portrait restoration algorithm service module.

[0294] In some embodiments, when an electronic device calls the portrait restoration algorithm service module in the cloud computing resources to perform portrait restoration, it first needs to send an instruction to the face generation plugin, and then the face generation plugin sends the instruction to the portrait restoration algorithm service module through the smart cloud-microservice module.

[0295] 735: The face generation plugin sends a face restoration request to the Smart Cloud-Microservices module.

[0296] In some embodiments, when performing portrait restoration through the portrait restoration algorithm service module in the cloud computing resources, the electronic device can first send a portrait restoration request to the smart cloud-microservice module in the cloud computing resources, and then send the request to the portrait restoration algorithm service module through the smart cloud-microservice module.

[0297] 736: The Smart Cloud - Microservices module retrieves data from the OBS library based on image identification.

[0298] In some embodiments, before the Smart Cloud-Microservices module sends a portrait restoration request to the portrait restoration algorithm service module, it can first read the restoration strategy corresponding to the image identifier of the image to be processed set in step 727 above from the OBS library. Then, the Smart Cloud-Microservices module sends the restoration strategy and the portrait restoration request together to the portrait restoration algorithm service module.

[0299] 737: The Smart Cloud-Microservices module sends a portrait restoration request to the portrait restoration algorithm service module.

[0300] In some embodiments, when the Smart Cloud-Microservices module sends a portrait restoration request to the portrait restoration algorithm service module, it can also simultaneously send the restoration strategy read from the OBS library.

[0301] 738: The portrait restoration algorithm service module performs data parsing, portrait restoration, and portrait enhancement operations.

[0302] In some embodiments, when the portrait restoration algorithm service module receives a restoration strategy and a portrait restoration request, it first performs data parsing and then performs portrait restoration based on the parsed restoration strategy. For example, it replaces the face of the portrait to be processed with the face of a reference portrait. Then, the portrait restoration algorithm service module can also perform portrait enhancement processing on the restored portrait, such as adjusting texture and color difference to achieve an overall harmonious effect in the processed target image.

[0303] 739: The portrait restoration algorithm service module sends the restored target image to the image library.

[0304] In some embodiments, after the portrait restoration algorithm service module restores the portrait to be processed and obtains the target image, it can send the target image to the image library, which will then display it to the user.

[0305] 740: The gallery displays the target image to the user.

[0306] In some embodiments, after receiving the repaired target image, the image library can display the target image to the user.

[0307] Thus, the image processing method provided in this application can beautify the image to be processed, so that the final image can overcome image defects and improve the overall visual effect of the image.

[0308] Among them, the above Figure 7A and Figure 7B The illustrations shown all demonstrate how to directly repair the image of the person in the image to be processed based on a reference portrait. For example, multiple portraits with the same appearance as the image to be processed can be scored based on data such as facial rotation angle, eye opening degree, and facial smile amplitude. Then, the portrait with the highest score is selected as the reference portrait, and the image to be processed is repaired based on this reference portrait.

[0309] In other embodiments, the electronic device can also display multiple portraits that resemble the image to be processed and can compensate for image defects. The user selects a reference portrait from these multiple portraits, and the image to be processed is repaired based on the user-selected reference portrait. As described above. Figure 3 As shown in Figure (c), the editing interface 303 can display multiple image icons corresponding to portraits with the same appearance as the portrait to be processed 3012 and with open eyes. When the mobile phone 10 detects that the user clicks on the image icon 3031, the electronic device can repair the portrait to be processed 3012 based on the reference portrait corresponding to the image icon 3031.

[0310] Specifically, the following example uses the face as the part of the portrait to be processed, combined with... Figure 8AThe flowchart shown provides a simplified description of the execution process of obtaining a reference image through user interaction to process the image to be processed and generate the target image. Specifically, the process may include the following steps:

[0311] 801: The user selected a reference portrait.

[0312] In some embodiments, when the portrait to be processed corresponds to multiple reference portraits, the user can arbitrarily select one of the reference portraits. In this case, the electronic device can detect the user's selection and enhance the portrait to be processed based on the selected reference portrait. For example, as described above... Figure 3 As shown in Figure (c), the electronic device can beautify the portrait to be processed 3012 based on the reference portrait corresponding to the image identifier 3031 selected by the user.

[0313] 802: Obtain the occlusion information and rotation angle of the portrait to be processed.

[0314] In some embodiments, the electronic device can determine the degree of occlusion of the facial features of the person being processed, as well as the angle of the facial features toward the camera.

[0315] 803: Determine if the degree of occlusion is greater than the maximum occlusion threshold. If yes, proceed to step 804; otherwise, proceed to step 806.

[0316] In some embodiments, when the degree of facial feature occlusion exceeds the maximum occlusion threshold, it indicates that the portrait to be processed cannot be processed. Conversely, it indicates that the portrait to be processed can be swapped or have its face replaced.

[0317] 804: It has been determined that the image to be processed cannot be processed.

[0318] In some embodiments, when it is determined in step 803 that the degree of facial feature occlusion is greater than the maximum occlusion threshold, it means that the portrait to be processed cannot be processed.

[0319] 805: End of this processing flow.

[0320] In some embodiments, when the image to be processed cannot be processed, the electronic device can end the current loop to terminate the current processing flow, and then start the next loop to continue processing the next image to be processed.

[0321] 806: Determine if the degree of occlusion is greater than the minimum occlusion threshold. If yes, proceed to step 807; otherwise, proceed to step 811.

[0322] In some embodiments, when it is determined in step 803 that the occlusion degree is less than the maximum occlusion threshold, it is also necessary to determine whether the occlusion degree is greater than the minimum occlusion threshold. If yes, then the image to be processed needs to undergo head swapping; if no, then it is necessary to further determine whether the face rotation angle is greater than the angle threshold.

[0323] 807: Head replacement process confirmed.

[0324] In some embodiments, if step 803 determines that the degree of face occlusion in the image to be processed is less than the maximum occlusion threshold, and step 806 determines that the degree of face occlusion is greater than the minimum occlusion threshold, then the corresponding repair strategy can be determined as follows: replace the head image features in the image to be processed with the head image features of the corresponding reference image, thereby obtaining a target image that overcomes the image defects. See steps 808 to 810 for details.

[0325] 808: Align the header and reply.

[0326] In some embodiments, when the electronic device performs a head-swapping operation on the image to be processed, it can use the method shown in step 808 to perform image processing operations such as aligning and pasting the head of the reference image with the head of the image to be processed, thereby replacing the head region of the image to be processed with the head region of the reference image.

[0327] In other embodiments, when performing head-swapping on the image to be processed, the electronic device may not execute the method shown in step 808, but instead use an artificial intelligence (AI) generation method to perform the head-swapping operation on the image to be processed. For example, if the image to be processed is a closed-eye image and the reference image is an open-eye image, the facial features of the image to be processed and the facial features of the reference image can be input into an AI generation model (e.g., a stable diffusion model). The AI ​​generation model can generate an open-eye target image based on the input feature data (e.g., the eye feature data of the reference image and other facial feature data of the image to be processed).

[0328] It should be understood that this application can perform head swapping on the image to be processed using any image processing method, and this application does not limit it in this regard.

[0329] 809: Perform fusion processing on the head.

[0330] In some embodiments, the electronic device can fuse the obtained target head region with other regions in the image to be processed to obtain the target image.

[0331] 810: Indicates a replacement mark.

[0332] In some embodiments, after a head or face swap is performed on the image to be processed, a replacement mark can be marked so as to determine whether a head or face swap operation has been performed on the image to be processed.

[0333] 811: Determine if the rotation angle is greater than the angle threshold. If yes, proceed to step 807; otherwise, proceed to step 812.

[0334] In some embodiments, after determining in step 806 that the degree of face occlusion in the image to be processed is less than the minimum occlusion threshold, the electronic device also needs to determine whether the face rotation angle of the image to be processed is greater than the angle threshold. If the face rotation angle is greater than the angle threshold, the process proceeds to step 807 to perform a head-swapping operation, replacing the head image features in the image to be processed with the head image features of the corresponding reference image, thereby obtaining a target image that overcomes the image defect. If the degree of face occlusion is less than the minimum occlusion threshold and the face rotation angle is less than the angle threshold, the process proceeds to step 812, where the repair strategy is determined to be: replacing at least some facial feature image features in the image to be processed that do not meet the preset conditions with at least some facial feature head image features in the corresponding reference image, thereby obtaining a target image that overcomes the image defect.

[0335] 812: It is determined that facial features treatment will be performed.

[0336] In some embodiments, if step 806 determines that the degree of occlusion is less than the minimum occlusion threshold, and step 811 determines that the face rotation angle is less than the angle threshold, then it is determined that a certain facial feature of the portrait to be processed can be replaced. See steps 813 to 814 for details.

[0337] 813: Align and reply to faces.

[0338] In some embodiments, when the electronic device performs the operation of replacing facial features (e.g., replacing eyes) on the image to be processed, it can perform image processing operations such as aligning and pasting the facial features of the reference image and the facial features of the image to be processed through the method shown in step 813, thereby replacing part of the facial features of the image to be processed with part of the facial features of the reference image.

[0339] In other embodiments, when the electronic device performs the operation of replacing facial features (such as replacing eyes) on the image to be processed, it may not execute the method shown in step 813. Instead, it may perform the operation of replacing facial features on the image to be processed through an AI generation method. For example, if the image to be processed is a closed-eye image and the reference image is an open-eye image, the facial features of the reference image (such as eye features) can be input into the AI ​​generation model (such as a stable diffusion model). Alternatively, the facial features of the reference image and the facial features of the image to be processed can be input into the AI ​​generation model simultaneously. The AI ​​generation model can then generate the target facial feature image based on the input feature data (such as the eye feature data of the reference image).

[0340] It should be understood that this application can perform facial feature replacement on the image to be processed using any image processing method, and this application does not limit this.

[0341] 814: Perform face merging processing.

[0342] In some embodiments, the electronic device can fuse the obtained target facial features region with other regions in the image to be processed to obtain the target image. Furthermore, after performing a face-swapping operation on the portrait to be processed, the process can proceed to step 810 to mark the replacement.

[0343] 815: Determine if a replacement mark exists. If yes, proceed to step 817; otherwise, proceed to step 816.

[0344] In some embodiments, the electronic device can determine whether a face-swapping or head-swapping operation has been performed based on the presence of a replacement marker in the target image. If a replacement marker exists, the image to be processed is updated to the target image and displayed to the user; if no replacement marker exists, it is determined that the image to be processed cannot be beautified.

[0345] 816: It was determined that the image to be processed could not be beautified.

[0346] In some embodiments, if step 815 determines that there is no replacement mark in the target image, it means that the target image is still the original image to be processed, and no face swapping or head swapping operation has been performed. Therefore, it can be determined that the image to be processed cannot be beautified.

[0347] 817: Replace the image to be processed with the target image.

[0348] In some embodiments, if step 815 determines that there is a replacement mark in the target image, it means that the target image is not the original image to be processed, and the face or head of the person to be processed in the image to be processed has been swapped or replaced. Then the image to be processed can be replaced with the target image and displayed to the user.

[0349] 818: Display the target image to the user.

[0350] In some embodiments, the electronic device may replace the image to be processed with the target image and display it to the user.

[0351] Thus, through the above process, the image to be processed can be processed based on the reference image selected by the user, so that the target image can overcome image defects and improve the overall aesthetics of the target image.

[0352] The following is combined Figure 8B The flowchart shown further describes the specific process of an image processing method that processes the image to be processed based on a user-selected reference image. This method can be applied to electronic devices, as described above. Figure 1 The mobile phone 10 shown in the image. Among them, and Figure 8A Compared to the flowchart shown, Figure 8B The flowchart shown includes an added check to determine whether the image to be processed is a single-person photograph. Specifically, Figure 8B The method shown may include the following steps:

[0353] S820: Obtain the set of human figures in the image to be processed.

[0354] In some embodiments, the electronic device can first acquire individual portraits in the image to be processed. For example, for a single person photo, the electronic device can acquire the single portrait of the person in the single person photo; for a group photo, the electronic device can acquire multiple portraits of the people in the group photo.

[0355] S821: Determine the reference set of portraits corresponding to a portrait to be processed in the portrait set.

[0356] In some embodiments, when there are multiple portraits in the set of portraits to be processed, each portrait can be processed separately. Specifically, when processing one of the portraits, it is first necessary to select multiple portraits with similar appearances from the image library to obtain a reference portrait set.

[0357] S822: Sort the portraits in the reference portrait set.

[0358] In some embodiments, after acquiring multiple portraits that resemble the image to be processed, they can be sorted based on facial scores and displayed to the user. For example, each portrait can be scored based on data such as the degree of eye opening, facial rotation angle, and the degree of smile at the corners of the mouth, and the scores can be displayed to the user in descending order, so that the user can select the final reference portrait to replace the image to be processed from among the multiple portraits.

[0359] S823: The user has selected a reference portrait from the set of reference portraits.

[0360] In some embodiments, a user can select the final reference image to replace the image to be processed from a sorted set of reference images. The electronic device can then detect the user's selection and process the image to be processed based on the selected reference image.

[0361] S824: Obtain facial data such as the face rotation angle or face occlusion information of the portrait to be processed.

[0362] In some embodiments, the electronic device can acquire face rotation angle data of the image to be processed. For example, the rotation angle when the face is facing the user can be set to 0, and then the angle between the face of the image to be processed and the direction facing the user can be acquired.

[0363] In other embodiments, the electronic device can also acquire facial occlusion information of the image to be processed, for example, it can detect that the face is occluded by x%.

[0364] S825: Determine portrait restoration strategies based on facial data.

[0365] In some embodiments, the electronic device can determine a portrait restoration strategy based on facial data such as the face rotation angle or face occlusion information of the portrait to be processed.

[0366] In some embodiments, if the face is severely occluded (e.g., 80% of the face in the image to be processed is obscured) or the face rotation angle is greater than an angle threshold, the image restoration strategy is that the image to be processed cannot be processed. Then, the process proceeds to step S8261 to determine whether the image to be processed is a single-person photo, so that other images in the image to be processed can be processed.

[0367] In some embodiments, if the face occlusion of the image to be processed is not severe, for example, the occlusion area is smaller than the occlusion threshold, or the face rotation angle is small, the image restoration strategy is to replace the entire head of the image to be processed. Then, it is necessary to proceed to step S8271 to perform face analysis on the image to be processed and the final reference image selected by the user.

[0368] In some embodiments, if the face is not obscured and is facing the user, and only the facial features meet the processing conditions affecting the image aesthetics, then the portrait restoration strategy is to replace a certain facial feature of the portrait to be processed. Then, it is necessary to proceed to step S8281 to perform facial feature region analysis on the portrait to be processed and the final reference portrait selected by the user.

[0369] S8261: Determine if the image to be processed is a single person photo. If not, proceed to step S8262; if yes, proceed to step S8263.

[0370] In some embodiments, during step S825, when determining the portrait restoration strategy based on facial data such as the face rotation angle or face occlusion information of the portrait to be processed, if the face occlusion is severe or the face rotation angle is greater than the angle threshold, the portrait restoration strategy is that the portrait to be processed cannot be processed. At this time, the electronic device needs to determine whether there is only one portrait in the image to be processed. If yes, proceed to step S8263 and prompt the user that the image to be processed cannot be processed; if no, proceed to step S8262, mark the portrait to be processed as unprocessable, and then process the next portrait in the image to be processed.

[0371] S8262: Mark the portrait to be processed as unchangeable.

[0372] In some embodiments, during step S825, when determining the portrait restoration strategy based on facial data such as the face rotation angle or face occlusion information of the portrait to be processed, if the face occlusion is severe or the face rotation angle is greater than the angle threshold, the portrait restoration strategy is that the portrait to be processed cannot be processed. Furthermore, if it is determined in step S8261 that the image to be processed is not a single-person photo, the portrait to be processed is marked as unchangeable, and then the process proceeds to step S821 to obtain the reference portrait set corresponding to the next portrait to be processed in the image to be processed.

[0373] S8263: The user is prompted that the image to be processed cannot be processed.

[0374] In some embodiments, during step S825, when determining the portrait restoration strategy based on facial data such as the face rotation angle or face occlusion information of the portrait to be processed, if the face occlusion is severe or the face rotation angle is greater than the angle threshold, the portrait restoration strategy is that the portrait to be processed cannot be processed. Furthermore, if it is determined in step S8261 that the image to be processed is a single-person photo, it is determined that the image to be processed cannot be processed, and the user is notified via a pop-up window or similar means that the image to be processed cannot be processed.

[0375] S8264: End execution of image processing method.

[0376] In some embodiments, the electronic device may terminate the execution of the image processing method provided in this application after notifying the user that the image to be processed cannot be processed.

[0377] S8271: Perform face analysis on the image to be processed.

[0378] In some embodiments, if the face occlusion of the portrait to be processed is not severe, for example, the occlusion area is smaller than the occlusion threshold, or the face rotation angle is small, then the portrait restoration strategy determined in step S825 is to replace the entire head of the portrait to be processed. Specifically, during head replacement, the user-selected portrait to be processed first needs to undergo face analysis; for example, the edge features of the face can be obtained to completely extract the head region.

[0379] S8272: Extract the head area of ​​the portrait to be processed.

[0380] In some embodiments, after determining data such as the facial edge features of the portrait to be processed, the head region of the portrait to be processed can be completely extracted.

[0381] S8273: Align and repost the head area of ​​the portrait to be processed.

[0382] In some embodiments, the head region of the reference portrait can be aligned with the head region of the portrait to be processed, and the head region of the portrait to be processed can be replaced with the head region of the reference portrait.

[0383] S8274: Perform head fusion on the portrait to be processed.

[0384] In some embodiments, after replacing the head region of the portrait to be processed with the head region of the reference portrait in step S8273, the connection position between the head region and the body region can be adjusted to make the connection smoother and improve the realism of the processed portrait. Thus, through the above steps S8271 to S8274, the head replacement operation on the portrait to be processed can be completed.

[0385] S8281: Perform facial feature analysis on the portrait being processed.

[0386] In some embodiments, if the face is not obscured and is facing the user, and only the facial features meet the processing conditions affecting the image's aesthetics, then the portrait restoration strategy determined in step S825 is to replace a specific facial feature of the portrait to be processed. Specifically, when replacing facial features, it is first necessary to analyze the facial features of the user-selected portrait. For example, the position of the facial features within the face and their edge features can be obtained to completely extract a specific facial feature area.

[0387] S8282: Extract the facial features of the person being processed.

[0388] In some embodiments, after determining the data such as the edge features of the facial features of the portrait to be processed, the facial features that need to be adjusted (such as the eyes) in the portrait to be processed can be completely extracted.

[0389] S8283: Aligns and reposts the facial features of the portrait being processed.

[0390] In some embodiments, the facial features of the reference portrait can be aligned with the facial features of the portrait to be processed, and the facial features that need to be adjusted in the portrait to be processed can be replaced with the facial features of the reference portrait.

[0391] S8284: Perform facial blending on the portrait being processed.

[0392] In some embodiments, after replacing the facial features of the portrait to be processed with the facial features of the reference portrait in step S8283, the connection position between the replaced facial features and the face region can be adjusted to make the connection smoother and improve the realism of the processed portrait. Thus, through the above steps S8281 to S8284, the facial feature replacement operation can be completed on the portrait to be processed.

[0393] S829: Adjust the brightness and other data of the processed portrait.

[0394] In some embodiments, after the head replacement processing of the portrait to be processed is carried out through steps S8271 to S8274, or after the facial features of the image to be processed are replaced through steps S8281 to S8284, the brightness, texture, color difference and other details of the processed portrait can be adjusted to make the overall portrait have a high degree of uniformity.

[0395] S830: Scoring the processed portrait.

[0396] In some embodiments, after processing the portrait to be processed, the processed portrait can also be scored based on data such as the degree of eye opening, the angle of facial rotation, and the degree of smile at the corners of the mouth.

[0397] In some embodiments, the electronic device may also repeatedly execute steps S821 to S830 to process multiple images of people to be processed in the image to be processed. That is, when processing the first image of people to be processed, steps S821 to S830 may be executed to obtain the processed image; then, steps S821 to S830 may be repeated to process the next image of people to be processed.

[0398] S831: Displays the processed target image.

[0399] In some embodiments, the electronic device can display the acquired target image to the user.

[0400] S832: End execution of image processing algorithm.

[0401] In some embodiments, the electronic device can end the execution of the image processing algorithm after displaying the target image to the user.

[0402] Thus, through the above process, the image to be processed can be processed based on the reference image selected by the user, so that the target image can overcome image defects and improve the overall aesthetics of the target image.

[0403] The following is in conjunction with the above. Figure 4 The system architecture diagram of the image processing method shown is based on Figure 8C The illustrated time-series interaction flowchart describes in detail the interaction process of processing the image to be processed based on the reference image selected by the user. For example, Figure 8C As shown, the interactive process of processing the image to be processed based on the user-selected reference image can be implemented through a gallery, a FaceGenerate plugin, a smart cloud microservice module, a restoration strategy judgment algorithm service module, a portrait restoration algorithm service module, and user-to-user interaction. Specifically, Figure 8C The interactive process shown includes the following steps:

[0404] 850: The user selects a reference portrait in the gallery's editing interface.

[0405] In some embodiments, the electronic device may display multiple portraits that have the same appearance as the portrait to be processed, and the user can select any one of these portraits as a reference portrait.

[0406] 851: The image library has identified the portrait to be processed and the reference portrait.

[0407] In some embodiments, after the user selects a reference portrait, the image library can determine the portrait to be processed and the corresponding reference portrait.

[0408] 852: The image library sends a command to the face generation plugin to call the repair strategy judgment algorithm service module.

[0409] In some embodiments, when an electronic device calls the repair strategy judgment algorithm service module in cloud computing resources to determine the portrait repair strategy, it first needs to send a call instruction to the face generation plugin, and then the face generation plugin sends the call instruction to the repair strategy judgment algorithm service module through the smart cloud microservice module.

[0410] 853: The face generation plugin performs image size processing.

[0411] In some embodiments, before calling the restoration strategy judgment algorithm service module to determine the portrait restoration strategy, the face generation plugin may first perform image size processing on the image to be processed so as to adjust the image size to the input size that the portrait restoration strategy algorithm can accept.

[0412] 854: The face generation plugin sends a repair strategy judgment request to the Smart Cloud-Microservices module.

[0413] In some embodiments, when determining the portrait restoration strategy through the restoration strategy judgment algorithm service module in the cloud computing resources, the electronic device can first send a restoration strategy judgment request to the smart cloud-microservice module in the cloud computing resources, and then send the request to the restoration strategy judgment algorithm service module through the smart cloud-microservice module.

[0414] 855: The Smart Cloud-Microservices module sends a repair strategy judgment request to the repair strategy judgment algorithm service module.

[0415] In some embodiments, after receiving a repair strategy judgment request, the Smart Cloud-Microservices module can send the request to the Repair Strategy Judgment Algorithm Service Module.

[0416] 856: The repair strategy judgment algorithm service module performs face analysis and other operations, and determines the repair strategy based on angle information, occlusion information, etc.

[0417] In some embodiments, the repair strategy judgment algorithm service module can acquire data such as the face rotation angle and the degree of face occlusion of the portrait to be processed, and then determine the repair strategy based on the angle information and occlusion information. For example, if the repair strategy judgment algorithm service module determines that the face rotation angle is greater than a preset angle threshold, such as when the face of the portrait to be processed is facing to the sides, the repair strategy judgment algorithm service module can determine that the portrait to be processed cannot be processed, and directly send the result of not being able to process it to the image library for display to the user. As another example, if the repair strategy judgment algorithm service module determines that the face rotation angle is less than a preset angle threshold, and the face is occluded, the corresponding repair strategy can be to replace the entire head area of ​​the portrait to be processed. As yet another example, if the repair strategy judgment algorithm service module determines that the face rotation angle is less than a preset angle threshold, the face is not occluded, and a single facial feature meets the processing conditions (e.g., eyes are closed), the corresponding repair strategy can be to replace a certain facial feature of the portrait to be processed (e.g., replace the eyes). This application does not limit this.

[0418] 857: The repair strategy judgment algorithm service module sends the repair strategy to the Smart Cloud-Microservice module.

[0419] In some embodiments, after determining the repair strategy, the repair strategy judgment algorithm service module can send the repair strategy to the Smart Cloud-Microservice module, which will then determine whether to send it to the image library or call the portrait repair algorithm service module for portrait repair.

[0420] 8581: Smart Cloud - Microservices module sends repair strategy to gallery.

[0421] In some embodiments, if the repair strategy is that the portrait to be repaired cannot be repaired, the Smart Cloud-Microservices module can send the repair strategy to the gallery and display the strategy to the user through the gallery.

[0422] 8582: The image library prompts the user that the face of the person to be processed is rotated at a large angle or is obscured.

[0423] In some embodiments, when the image library receives a repair strategy, if the repair strategy is that the face of the person to be processed has a large rotation angle or is heavily obscured, making it impossible to repair the person to be processed, the image library can display the repair strategy to the user.

[0424] 859: The Smart Cloud-Microservices module sends a portrait restoration request to the portrait restoration algorithm service module.

[0425] In some embodiments, if the repair strategy is that repair is possible, the Smart Cloud-Microservices module can send a portrait repair request to the portrait repair algorithm service module, and can also send the repair strategy at the same time.

[0426] 860: The portrait restoration algorithm service module performs data parsing, portrait restoration, and portrait enhancement operations.

[0427] In some embodiments, when the portrait restoration algorithm service module receives a restoration strategy and a portrait restoration request, it first parses the data and then performs portrait restoration based on the parsed restoration strategy. For example, it replaces the head region of the portrait to be processed with the head region of a reference portrait. Then, the portrait restoration algorithm service module can also perform portrait enhancement processing on the restored portrait, such as adjusting texture and color difference to achieve an overall harmonious effect in the processed target image.

[0428] 861: The portrait restoration algorithm service module sends the restored target image to the image library.

[0429] In some embodiments, after the portrait restoration algorithm service module restores the portrait to be processed and obtains the target image, it can send the target image to the image library, which will then display it to the user.

[0430] 862: The gallery displays the target image to the user.

[0431] In some embodiments, after receiving the repaired target image, the image library can display the target image to the user.

[0432] Thus, the image processing method provided in this application can process the image to be processed based on the reference image selected by the user, so that the target image can overcome image defects and improve the overall aesthetics of the target image.

[0433] This application provides a readable storage medium storing instructions that, when executed on an electronic device, cause the electronic device to perform the image processing method mentioned in this application.

[0434] This application provides a computer program product, including: computer instructions, which, when executed on an electronic device, cause the electronic device to perform the image processing method mentioned in this application.

[0435] This application also provides an electronic device, which includes a memory and a processor. The memory stores instructions executed by one or more processors of the electronic device, and the processor is one of the one or more processors of the electronic device, used to execute the image processing method mentioned in this application.

[0436] The image processing method described in this application can be executed by an electronic device, which can be any electronic device such as a mobile phone, computer, tablet computer, augmented reality (AR) device, or laptop computer. The form of the electronic device is not specifically limited in the embodiments of this application.

[0437] like Figure 9 As shown, taking the aforementioned mobile phone 10 as an example, this paper illustrates a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application.

[0438] like Figure 9 As shown, the mobile phone 10 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, antenna 1, antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a gyroscope sensor 180A, a magnetic sensor 180B, an accelerometer sensor 180C, an infrared sensor 180D, a temperature sensor 180E, a touch sensor 180F, an ambient light sensor 180G, a light sensor 180H, etc.

[0439] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the mobile phone 10. In other embodiments of this application, the mobile phone 10 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0440] Processor 110 may include one or more processing units, such as a central processing unit (CPU), a graphics processing unit (GPU), an embedded neural network processing unit (NPU), an application processor (AP), a modem processor, a controller, a video codec, a digital signal processor (DSP), etc. These different processing units may be independent devices or integrated into one or more processors.

[0441] In this embodiment, the processor 110 can be used to execute the image processing method provided in this application. For example, the processor 110 can acquire the first image features (e.g., facial features) of the image to be processed, and acquire the second image features (e.g., facial features) of other images in the image library, select a reference image from the images that has a high similarity to the image features of the target to be processed and can compensate for image defects, and finally perform image restoration on the image to be processed based on the selected reference image to obtain the target image.

[0442] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0443] USB port 130 can be used to connect a charger to charge phone 10, and can also be used for data transfer between phone 10 and peripheral devices. It can also be used to connect headphones for audio playback. This port can also be used to connect other electronic devices, such as AR devices.

[0444] The charging management module 140 receives charging input from a charger, which can be a wireless charger or a wired charger. While charging the battery 142, the charging management module 140 can also supply power to the electronic device via the power management module 141.

[0445] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to power the processor 110, internal memory 121, screen 194, camera 193, and wireless communication module 160, etc. In some other embodiments, the power management module 141 may also be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be located in the same device.

[0446] The wireless communication function of mobile phone 10 can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, and modem processor, etc. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in mobile phone 10 can be used to cover one or more communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, antenna 1 can be used in conjunction with a tuning switch.

[0447] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, for use in the mobile phone 10.

[0448] Screen 194 is used to display images, videos, etc. Screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), or the like. In some embodiments, mobile phone 10 may include one or N screens 194, where N is a positive integer greater than 1.

[0449] In some embodiments, screen 194 can be used to display the image to be processed and the target image, and can also be used to display an image editing interface.

[0450] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element may be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. In some embodiments, mobile phone 10 may include one or N cameras 193, where N is a positive integer greater than 1.

[0451] The internal memory 121 can be used to store computer executable program code, which includes instructions. The internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of the mobile phone 10 (such as audio data, phonebook, etc.). Furthermore, the internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the mobile phone 10 by running instructions stored in the internal memory 121 and / or instructions stored in memory located in the processor.

[0452] The phone 10 can perform audio functions, such as music playback and recording, through an audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0453] The gyroscope sensor 180A can be used to determine the motion attitude of the mobile phone 10. In some embodiments, the angular velocity of the mobile phone 10 about three axes (i.e., the x, y, and z axes) can be determined by the gyroscope sensor 180A.

[0454] The Magnetic Sensor 180B can detect magnetic field strength.

[0455] The 180C accelerometer can detect the magnitude of acceleration in various directions (typically three axes) of the mobile phone 10. When the mobile phone 10 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the posture of electronic devices and applied to applications such as screen orientation switching and pedometers.

[0456] The 180D infrared sensor is used to detect, receive, and analyze infrared radiation signals.

[0457] Temperature sensor 180E is used to detect temperature. In some embodiments, mobile phone 10 uses the temperature detected by temperature sensor 180E to execute a temperature processing strategy. For example, when the temperature reported by temperature sensor 180E exceeds a threshold, mobile phone 10 reduces the performance of the processor located near temperature sensor 180E to reduce power consumption and implement thermal protection.

[0458] Touch sensor 180F, also known as a "touch device," can be placed on screen 194. The touch sensor 180F and screen 194 together form a touch screen, also known as a "touchscreen." Touch sensor 180F is used to detect touch operations applied to or near it.

[0459] The 180G ambient light sensor can be used to sense ambient light intensity.

[0460] The 180H light sensor can be used to detect light intensity. It has high sensitivity and fast response speed, and can be used in various lighting environments to provide real-time feedback on light conditions.

[0461] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch buttons. Mobile phone 10 can receive button input and generate key signal inputs related to user settings and function control of mobile phone 10.

[0462] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback.

[0463] Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0464] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to make contact with or separate from the mobile phone 10. The mobile phone 10 can support one or N SIM card interfaces, where N is a positive integer greater than 1.

[0465] The embodiments disclosed in this application can be implemented in hardware, software, firmware, or a combination of these implementation methods. Embodiments of this application can be implemented as computer programs or program code executable on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0466] Program code can be applied to input instructions to execute the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a digital signal processor, a microcontroller, an application-specific integrated circuit, or a microprocessor.

[0467] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. In fact, the mechanisms described in this application are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0468] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. Furthermore, including structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0469] It should be noted that all units / modules mentioned in the device embodiments of this application are logical units / modules. Physically, a logical unit / module can be a physical unit / module, a part of a physical unit / module, or a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important factor; the combination of functions implemented by these logical units / modules is the key to solving the technical problem proposed in this application. Furthermore, in order to highlight the innovative aspects of this application, the above-described device embodiments of this application have not introduced units / modules that are not closely related to solving the technical problem proposed in this application. This does not mean that the above-described device embodiments do not contain other units / modules.

[0470] It should be noted that in the examples and description of this application, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0471] Although this application has been illustrated and described with reference to certain preferred embodiments thereof, those skilled in the art will understand that various changes in form and detail may be made thereto without departing from the scope of this application.

Claims

1. An image processing method, characterized by, The method comprises: acquiring a first image to be processed; detecting that a first image feature that does not satisfy a preset condition exists in a first region in the first image, wherein the first region comprises a first face region, and the first image feature comprises a first face feature; selecting a first reference image from a gallery, wherein a similarity between a second region of the first reference image and the first region is greater than a similarity threshold, and a second image feature of the second region satisfies the preset condition; processing the first image feature in the first image based on the second image feature to obtain a second image that satisfies the preset condition.

2. The method of claim 1, wherein, The method comprises: detecting an image editing operation of a user on the first image, acquiring the first image to be processed, and displaying a first image editing interface corresponding to the first image, wherein the first image editing interface comprises an image processing button; corresponding to the existence of the first image feature in the first image and the existence of the first reference image in the gallery, the image processing button is in a state capable of responding to a user operation instruction; corresponding to the non-existence of the first image feature in the first image, or the existence of the first image feature in the first image and the non-existence of the first reference image in the gallery, the image processing button is in a state incapable of responding to a user operation instruction.

3. The method of claim 2, wherein, corresponding to the image processing button being in a state capable of responding to a user operation instruction, processing the first image feature in the first image based on the second image feature to obtain a second image that satisfies the preset condition comprises: in a case where it is detected that the user clicks the image processing button, processing the first image feature in the first image based on the second image feature in the first reference image to obtain the second image.

4. The method of claim 1, wherein, The method further comprises: detecting that the first image feature that does not satisfy the preset condition exists in the first region in the first image, and a third image feature that does not satisfy the preset condition exists in a third region, wherein the first region comprises the first face region, the first image feature comprises the first face feature, the third region comprises a second face region, and the third image feature comprises a second face feature; selecting the first reference image and a second reference image from the gallery, wherein a similarity between the second region of the first reference image and the first region is greater than the similarity threshold, the second image feature of the second region satisfies the preset condition, a similarity between a fourth region of the second reference image and the third region is greater than the similarity threshold, and a fourth image feature of the fourth region satisfies the preset condition; processing the first image feature in the first image based on the second image feature, and processing the third image feature in the first image based on the fourth image feature to obtain a second image that satisfies the preset condition.

5. The method of claim 4, wherein, The method comprises: detecting a user image editing operation on the first image, obtaining the first image to be processed, and displaying a second image editing interface corresponding to the first image, the second image editing interface including an image processing button; corresponding to the presence of the first image feature and the third image feature in the first image, and the presence of the first reference image and the second reference image in the gallery, the image processing button is in a state capable of responding to a user operation instruction; corresponding to the absence of the first image feature and the third image feature in the first image, or the absence of the first reference image and the second reference image in the gallery, the image processing button is in a state incapable of responding to a user operation instruction.

6. The method of claim 5, wherein, corresponding to the image processing button being in a state capable of responding to a user operation instruction, processing the first image feature in the first image based on the second image feature, and processing the third image feature in the first image based on the fourth image feature to obtain a second image satisfying the preset condition, including: detecting a user operation of clicking the image processing button, displaying a third image editing interface corresponding to the first image, the third image editing interface including a first image identifier corresponding to the first image feature and a second image identifier corresponding to the third image feature; detecting a user operation of selecting the first image identifier, processing the first image feature in the first image based on the second image feature; detecting a user operation of selecting the second image identifier, processing the third image feature in the first image based on the fourth image feature.

7. The method of claim 6, wherein, detecting a user operation of selecting the first image identifier, processing the first image feature in the first image based on the second image feature, including: detecting a user operation of selecting the first image identifier, displaying a fourth image editing interface corresponding to the first image, the fourth image editing interface including at least one image identifier corresponding to at least one reference image corresponding to the first image feature, wherein the similarity between the at least one reference image corresponding to the at least one image identifier and the first region is greater than the similarity threshold, and the second image feature of the second region satisfies the preset condition; detecting a user operation of selecting a third image identifier from the at least one image identifier, processing the first image feature based on the second image feature in the first reference image corresponding to the third image identifier.

8. The method of claim 1, wherein, The preset condition includes a face score greater than a score threshold; wherein The face score is determined based on at least one of face rotation angle data, face occlusion degree data, face smile degree data, and eye opening amplitude data.

9. The method of claim 8, wherein, The detection of the presence of the first image feature in the first region of the first image that does not satisfy the preset condition includes: detecting the presence of at least one face region in the first image; obtaining at least one face score corresponding to the at least one face region; In a case where the face score corresponding to the first face region is less than or equal to the score threshold, it is detected that the first image feature that does not satisfy the preset condition exists in the first region in the first image.

10. The method of claim 9, wherein, The selecting the first reference image from the gallery comprises: In the gallery, N images adjacent to the first image and taken earlier than the first image are selected, and N images adjacent to the first image and taken later than the first image are selected, to obtain a first image set, wherein N is a positive integer. From each image in the first image set, an image whose similarity between an image feature vector and a feature vector of the first image is greater than a similarity threshold is selected to obtain a second image set. From each image in the second image set, an image whose similarity between a second region and the first region is greater than the similarity threshold is selected to obtain a third image set. Face scores of the second region in each image in the third image set are determined. From each image in the third image set, an image with the maximum face score of the second region is selected as the first reference image.

11. The method of claim 10, wherein, The processing of the first image feature in the first image based on the second image feature to obtain the second image that satisfies the preset condition comprises: In a case where the face occlusion degree in the first region is less than a first occlusion threshold and greater than a second occlusion threshold, the head image feature in the first image feature is replaced by the head image feature in the second image feature to obtain the second image, and the first occlusion threshold is greater than the second occlusion threshold. In a case where the face occlusion degree in the first region is less than the second occlusion threshold and the face rotation angle is greater than an angle threshold, the head image feature in the first image feature is replaced by the head image feature in the second image feature to obtain the second image. In a case where the face occlusion degree in the first region is less than the second occlusion threshold and the face rotation angle is less than the angle threshold, at least part of the facial feature in the first image feature that does not satisfy the preset condition is replaced by at least part of the facial feature in the second image feature to obtain the second image.

12. The method of claim 11, wherein, The replacing the head image feature in the first image feature by the head image feature in the second image feature to obtain the second image comprises: The head image feature in the first image feature is replaced by the head image feature in the second image feature to obtain a third image, and one or more of a texture feature, a color feature, and a brightness feature in the head image feature in the third image is adjusted to obtain the second image that satisfies the preset condition.

13. The method of claim 11, wherein, The replacing at least part of the facial feature in the first image feature that does not satisfy the preset condition by at least part of the facial feature in the second image feature to obtain the second image comprises: Replace at least part of the facial feature image features in the first image features that do not meet the preset condition with corresponding at least part of the facial feature image features in the second image features to obtain a fourth image, and adjust one or more of texture features, color features, and brightness features in the facial feature image features in the fourth image to obtain the second image that meets the preset condition.

14. An electronic device, comprising: Comprising: A memory for storing instructions for one or more processors of the electronic device to execute, and a processor that is one of the one or more processors of the electronic device, configured to execute the image processing method of any one of claims 1-13.

15. A readable storage medium, characterized by, The readable storage medium has instructions stored thereon, and the instructions, when executed on an electronic device, cause the electronic device to execute the image processing method of any one of claims 1-13.

16. A computer program product, characterised in that, Comprising: Computer instructions that, when executed on an electronic device, cause the electronic device to execute the image processing method of any one of claims 1-13.

Images