Image processing method and device and electronic equipment

Abstract

The embodiment of the invention discloses an image processing method and device and electronic equipment. The method comprises the following steps: acquiring a first feature vector of a first face image, wherein the eyes of a target face in the first face image are in an eye closing state; editing the first feature vector based on an eye-opening editing vector to obtain a second feature vector, the second feature vector representing that the eyes of the target face are in an eye-opening state, and the eye-opening editing vector representing the difference between the eyes in an eye-closing state and the eye-opening state; obtaining a second face image through the second feature vector, wherein the eyes of the target face in the second face image are in an eye opening state; and replacing the human eyes of the target human face in the first human face image with the human eyes in the second human face image to obtain a target image. In this way, the eyes in the closed state of the first face image can be converted into the eyes in the open state, so that the image quality of the first face image is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and more specifically, to an image processing method, device, and electronic device. Background Art

[0002] With the continuous advancement of photography technology in mobile phones and other electronic devices, selfies and group photos have become a daily pleasure for many users. However, due to various uncontrollable factors during the shooting process, the eyes of the people in the photos taken by electronic devices may be closed, thereby reducing the image quality. Summary of the Invention

[0003] In view of the above problems, the present application proposes an image processing method, device and electronic device to improve the above problems.

[0004] In a first aspect, the present application provides an image processing method, comprising: obtaining a first eigenvector of a first facial image, wherein the eyes of a target face in the first facial image are in a closed state; editing the first eigenvector based on an eye-open editing vector to obtain a second eigenvector, wherein the second eigenvector represents that the eyes of the target face are in an open state, and the eye-open editing vector represents the difference between eyes in a closed state and eyes in an open state; obtaining a second facial image through the second eigenvector, wherein the eyes of the target face in the second facial image are in an open state; replacing the eyes of the target face in the first facial image with the eyes in the second facial image to obtain a target image.

[0005] In a second aspect, the present application provides an image processing device, the method comprising: a feature vector acquisition unit for acquiring a first feature vector of a first facial image, wherein the eyes of a target face in the first facial image are in a closed state; a vector editing unit for editing the first feature vector based on an eye-open editing vector to obtain a second feature vector, wherein the second feature vector represents that the eyes of the target face are in an open state, and the eye-open editing vector represents the difference between eyes being in a closed state and eyes being in an open state; an image generation unit for obtaining a second facial image through the second feature vector, wherein the eyes of the target face in the second facial image are in an open state; and an image processing unit for replacing the eyes of the target face in the first facial image with the eyes in the second facial image to obtain a target image.

[0006] In a third aspect, the present application provides an electronic device, which includes at least a processor and a memory; one or more programs are stored in the memory and configured to be executed by the processor to implement the above method.

[0007] In a fourth aspect, the present application provides a computer readable storage medium, wherein the computer readable storage medium stores program codes, and when the program codes are run by a processor, the above method is executed.

[0008] The present application provides an image processing method and device and electronic equipment. After a first feature vector of a first face image is obtained, the first feature vector is edited based on an open-eye editing vector to obtain a second feature vector. Then, a second face image is obtained through the second feature vector. In the second face image, the eyes of a target face are in an open-eye state. Then, the eyes of the target face in the first face image are replaced with the eyes in the second face image to obtain a target image. In this way, the first feature vector of the first face image is edited to obtain a second feature vector representing the eyes of the target face in an open-eye state. Then, the second face image for eye replacement is obtained through the second feature vector. Then, the eyes in the first face image in a closed-eye state are replaced with the eyes in the second face image in an open-eye state. In this way, the eyes in the first face image in a closed-eye state can be converted to an open-eye state, thereby improving the image quality of the first face image. BRIEF DESCRIPTION OF DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort.

[0010] Figure 1 A schematic diagram of an application scenario proposed by an embodiment of the present application is shown;

[0011] Figure 2 A schematic diagram of another application scenario proposed by an embodiment of the present application is shown;

[0012] Figure 3 A flowchart of an image processing method provided by an embodiment of the present application is shown;

[0013] Figure 4 A schematic diagram of eyes in an open-eye state in an embodiment of the present application is shown;

[0014] Figure 5 A schematic diagram of eyes in a closed-eye state in an embodiment of the present application is shown;

[0015] Figure 6 A flowchart of an image processing method provided by another embodiment of the present application is shown;

[0016] Figure 7 A flow chart of an image processing method according to an embodiment of the present application is shown;

[0017] Figure 8 A flow chart of an image processing method according to an embodiment of the present application is shown;

[0018] Figure 9 A structural block diagram of an image processing device according to an embodiment of the present application is shown;

[0019] Figure 10 A structural block diagram of an electronic device for performing an image processing method according to an embodiment of the present application is shown;

[0020] Figure 11 A storage unit for storing or carrying program codes for implementing an image processing method according to an embodiment of the present application. DETAILED DESCRIPTION

[0021] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0022] With the rapid development of technology, the photography technology of electronic devices such as mobile phones has made significant progress, making photography more convenient and efficient. Today, whether walking on the streets or being in natural scenery, selfie and group photo have become an important way for people to record life and share happiness. These photos not only carry our memories, but also show our personality and aesthetics.

[0023] However, while enjoying the fun of photography, we also face some challenges. One common problem is that the eyes of the people in the photos may be in a closed state. This situation is often caused by various uncontrollable factors, such as blinking at the moment of shooting, light stimulation or sand invasion, etc. Although these seemingly small details, however, will have a significant impact on the overall quality of the photo.

[0024] Therefore, the inventors propose the image processing method, device and electronic equipment in the present application. In the method, after obtaining a first feature vector of a first face image, the first feature vector can be edited based on an open-eye editing vector to obtain a second feature vector, then a second face image is obtained through the second feature vector, a person's eye of a target face in the second face image is in an open-eye state, and then the person's eye of the target face in the first face image is replaced with the person's eye in the second face image to obtain a target image. Thus, through the above manner, the first feature vector of the first face image can be edited to obtain the second feature vector representing that the person's eye of the target face is in the open-eye state, then the second face image for performing the person's eye replacement is obtained through the second feature vector, and then the person's eye in the first face image in the closed-eye state is replaced with the person's eye in the second face image in the open-eye state, so that the person's eye in the first face image in the closed-eye state can be converted to the open-eye state, thereby improving the image quality of the first face image.

[0025] Before the embodiments of the present application are further described in detail, an application environment involved in the embodiments of the present application is introduced.

[0026] First, an application scenario involved in the embodiments of the present application is introduced.

[0027] In the embodiments of the present application, the image processing method provided can be executed by an electronic device. In this way executed by the electronic device, all steps in the image processing method provided by the embodiments of the present application can be executed by the electronic device. For example, as shown in Figure 1 In the case that all steps in the image processing method provided by the embodiments of the present application can be executed by the electronic device, all steps can be executed by the processor of the electronic device 100.

[0028] Further, the image processing method provided by the embodiments of the present application can also be executed by a server (cloud). Correspondingly, in this manner of execution by the server, the server can start to execute the steps in the image processing method provided by the embodiments of the present application in response to a trigger instruction. The trigger instruction can be sent by an electronic device used by a user, or can be triggered locally by the server in response to some automatic event. For example, in the case where the electronic device uploads an image to the server, the server can start to execute the image processing method in response to receiving the image. The image uploaded by the electronic device can be only the first face image, or the image uploaded by the electronic device can include the first face image and the third face image. For example, in the case where the first electronic device acquires the first face image, if no third face image is detected, the first face image can be uploaded to the server only. For another example, in the case where the first electronic device acquires the first face image and also detects a third face image, the first face image and the third face image can be uploaded to the server together.

[0029] In addition, as shown in Figure 2 , the image processing method provided by the embodiments of the present application can also be executed by the electronic device and the server cooperatively. In this manner of cooperative execution by the electronic device and the server, part of the steps in the image processing method provided by the embodiments of the present application are executed by the electronic device, and the other part of the steps are executed by the server. For example, the electronic device 100 can execute the step of uploading the first face image to the server included in the image processing method. Then the subsequent steps are executed by the server 200 to obtain the target image, and the target image is returned to the electronic device 100, so that the electronic device 100 displays the target image, or uses the target image to replace the first face image stored locally. It should be noted that in this manner of cooperative execution by the electronic device and the server, the steps executed by the electronic device and the server are not limited to the manner described in the above examples, and in actual applications, the steps executed by the electronic device and the server can be adjusted dynamically according to actual conditions.

[0030] It should be noted that according to different steps required to be executed by the electronic device, the electronic device 100 can be not only the smart phone shown in Figure 1 and Figure 2 , but also a tablet computer, a smart watch, smart glasses and the like. The server 200 can be a physical server, or a server cluster or a distributed system composed of multiple physical servers. In the case where the image processing method provided by the embodiments of the present application is executed by the server cluster or the distributed system composed of multiple physical servers, different steps in the image processing method can be executed by different physical servers respectively, or can be executed by the server based on the distributed system in a distributed manner.

[0031] The embodiments of the present application will be described in detail below with reference to the drawings.

[0032] Please refer to Figure 3 The embodiments of the present application provide an image processing method, which comprises the following steps.

[0033] S110: obtaining a first feature vector of a first face image, wherein the eyes of a target face in the first face image are in a closed eye state.

[0034] In the embodiments of the present application, the face image can be understood as an image including a face. The eyes of the target face in the first face image are in a closed eye state. Therefore, the first face image can be understood as an image to be processed for eye replacement. The target face can be understood as the face of a person to be processed for eye replacement.

[0035] In the embodiments of the present application, the state of the eyes can be determined by the distance between the upper eyelid and the lower eyelid of the eyes. As a way, a first distance and a second distance can be obtained, wherein the first distance is the distance between a first key point in the upper eyelid and a second key point in the lower eyelid, and the second distance is the width of the eyes. For example, as shown in Figure 4 , the key points (such as the circular marks) of the eyes in the face image (for example, the first face image) are obtained by recognition. The first key point can be the key point at the middle position of the upper eyelid, for example, the key point P1 in Figure 4 . The second key point can be the key point at the middle position of the lower eyelid, for example, the key point P2 in Figure 4 . The width of the eyes can be the distance between the two corners of the eyes. For example, the width of the eyes can be the distance between the key point P3 and the key point P4. Figure 4 After obtaining the first distance and the second distance, the first distance can be compared with the second distance. If the ratio of the first distance to the second distance is less than a first proportion threshold, it is determined that the eyes are in a closed eye state. For example, as shown in

[0036] , it is a schematic diagram of the eyes in a closed eye state. Figure 5

[0037] ​Optionally, if the ratio of the first distance and the second distance is greater than a first proportion threshold, it is determined that the human eye is in an open-eye state. Optionally, if the ratio of the first distance and the second distance is greater than a second proportion threshold, it is determined that the human eye is in an open-eye state, the second proportion threshold being greater than the first proportion threshold. It should be noted that in the case where the ratio of the first distance and the second distance is greater than the first proportion threshold, the human eye can be in a semi-open-eye state or the degree of opening of the eye is not high. Therefore, by using the second proportion threshold to determine whether the human eye is in an open-eye state, the degree of opening of the human eye determined to be in an open-eye state is higher.

[0038] The feature vector is mainly used to represent some column features in data (for example, an image), which can describe the structure and correlation of the data. In the embodiment of the present application, the first feature vector can be understood as a feature vector used to represent the image content of the first face image. Optionally, after the first face image is obtained, the first face image can be encoded to map the first face image to a hidden space to obtain the feature vector of the first face image as the first feature vector.

[0039] S120: editing the first feature vector based on the open-eye editing vector to obtain a second feature vector, the second feature vector representing that the human eye of the target face is in an open-eye state, the open-eye editing vector representing the difference between the human eye being in a closed-eye state and being in an open-eye state.

[0040] In the embodiment of the present application, the open-eye editing vector is a vector representing the difference between the human eye being in a closed-eye state and being in an open-eye state.

[0041] As one way, an open-eye and closed-eye data set can be established in advance, and the open-eye and closed-eye data set can include a plurality of open-eye face images and a plurality of closed-eye face images. The human eyes in the open-eye face images are in an open-eye state, and the human eyes in the closed-eye face images are in a closed-eye state. By inputting the images in the open-eye and closed-eye data set into the encoding model, the feature vectors of each face image in the open-eye and closed-eye data set are obtained, and then the open-eye feature vectors and the closed-eye feature vectors are distinguished by the classifier, so as to obtain the difference vector between the two categories (open-eye feature vector set and closed-eye feature vector) as the open-eye editing vector. The classifier can be a SVM (Support Vector Machine) classifier.

[0042] In the case where the open-eye editing vector can represent the difference between the human eye being in a closed-eye state and being in an open-eye state, editing the first feature vector based on the open-eye editing vector can be understood as changing the content of the first feature vector towards the direction representing the human eye being in an open-eye state. Further, the second feature vector obtained by editing represents that the human eye of the target face is in an open-eye state.

[0043] S130: obtaining a second face image through the second feature vector, wherein the eyes of the target face in the second face image are in an open-eye state.

[0044] In the case of obtaining the second feature vector, the second face image can be obtained through a face generation model. The face generation model has the ability to obtain a corresponding image according to a feature vector. After obtaining the second feature vector, the second feature vector can be input into the face generation model to obtain the second face image output by the face generation model.

[0045] As one way, in the process of obtaining the open-eye editing vector, the open-eye editing vector can be further adjusted, so that the adjusted open-eye editing vector not only represents the difference between the closed-eye state and the open-eye state of the eyes, but also carries more information about eye beautification, so that the obtained second feature vector not only represents that the eyes of the target face are in an open-eye state, but also makes the second face image generated by the second feature vector have decorations. The decorations can include eye shadow or eyelashes, etc.

[0046] S140: replacing the eyes of the target face in the first face image with the eyes in the second face image to obtain a target image.

[0047] It should be noted that although the eyes in the generated second face image are in an open-eye state, because the second face image is a model-generated image, the realism of the image content of the second face image will be different from that of the image captured by the camera (for example, the first face image). Therefore, by replacing the eyes of the target face in the first face image with the eyes in the second face image, the eyes of the target face in the first face image can be switched to an open-eye state, and the obtained target image can also maintain realism.

[0048] It should be noted that after the replacement, artifacts may appear. In the image field, artifacts refer to image features or deformations that do not match the original image information and are not expected to appear during image processing or display. These artifacts can be caused by physical limitations of imaging devices, defects in image acquisition process, deficiencies in image processing algorithms, or characteristics of display devices. In order to eliminate artifacts, as one way, the eyes of the target face in the first face image can be replaced with the eyes in the second face image to obtain a second face image to be processed, and the target generation model is used to eliminate artifacts in the second face image to be processed to obtain a target image.

[0049] The target generation model is a model for artifact removal of an image. In the embodiments of the present application, the target generation model can be implemented by training an encoder-decoder model. The training data of the target generation model is data with strong artifact feeling and corresponding data without artifact feeling. The training target is to restore the data without artifact feeling from the data with strong artifact feeling. For example, real human face data can be taken, and then a simulated artifact feeling method can be used, such as blackening the eyelid area, to construct data with strong artifact feeling. Or directly take the replaced human face as the input of model training, and use the corresponding real open-eye person as the target to train.

[0050] The embodiments of the present application provide an image processing method, so that the first feature vector of the first human face image can be edited to obtain the second feature vector representing that the human eye of the target human face is in an open-eye state, and then the second human face image for human eye replacement is obtained through the second feature vector, and then the human eye in the closed-eye state in the first human face image is replaced with the human eye in the open-eye state in the second human face image, so that the human eye in the closed-eye state in the first human face image can be converted into the open-eye state, and the image quality of the first human face image is improved.

[0051] Please refer to Figure 6 The embodiments of the present application provide an image processing method, which comprises the following steps.

[0052] S210: In response to obtaining the first human face image, it is detected whether there is a third human face image.

[0053] In the embodiments of the present application, the human eye of the target human face in the first human face image is in a closed-eye state. The third human face image comprises the target human face, and the human eye of the target human face is in an open-eye state. Alternatively, it can be understood that the first human face image and the third human face image both comprise the target human face, but the human eye included in the target human face in the first human face image is in a closed-eye state, and the human eye included in the target human face in the third human face image is in an open-eye state.

[0054] The first face image can be understood as being displayed. For example, if the electronic device displays the first face image, it can be understood that the first face image is acquired. The electronic device can display the first face image in full screen. For example, in one case, the electronic device can display images in an album. In this case, the displayed images can all be displayed in the form of a thumbnail. If an image in the displayed images is switched to full screen display by user touch, it can be detected whether the image switched to full screen display includes a person's eye in a closed eye state. If so, the image switched to full screen display can be used as the first face image, and the face including the person's eye in a closed eye state can be used as the target face.

[0055] The third face image can be detected within a certain range. The certain range can include at least one of a storage range, a location range, and a time range.

[0056] The storage range can be understood as a range in which the image is stored in the electronic device. For example, as one way, the electronic device can detect whether the third face image exists in an album.

[0057] The location range can be a range corresponding to a collection location of the image. For example, as one way, the collection location of the first face image can be acquired as a target location, and then it can be detected whether the third face image exists in an image collected at the target location. Alternatively, it can be detected whether the third face image exists in an image collected at a location close to the target location. The location close to the target location can be understood as having a distance to the target location less than a target distance threshold. The target distance threshold can be 1 meter, 10 meters, or 100 meters.

[0058] The time range can be understood as a range corresponding to a collection time of the image. For example, the collection time of the first face image can be acquired as a target time, and then it can be detected whether the third face image exists in an image collected at a time close to the target time. The time close to the target time can include an interval between the collection time and the target time less than a target time length. The target time length is not limited in the embodiments of the present application, for example, it can be 5 seconds, 10 seconds, or 15 seconds.

[0059] For example, when the photo album of the electronic device is opened to one picture, the electronic device can call the face detection and clustering algorithm to scan the images within a fixed time length, which can be a predefined time, for example, 5s. After obtaining N pictures, all the faces can be recognized and clustered into multiple ID classes, each of which has multiple faces of the ID. For example, the current user browses the photos that include not only the face Al of a certain ID, but also the faces A2, A3, … A N-1 If the eyes of the face Al are in a closed state, the images in which the eyes of the face Al are in an open state can be found from the other N-1 photos.

[0060] In the foregoing case where there are multiple implementations of the certain range (time range, location range, and storage range), the current specific range in which the third face image is detected can be determined according to the actual situation.

[0061] As one way, the range in which the third face image is detected can be determined according to the image category of the first face image. Optionally, if the first face image is a travel category image, the third face image can be detected according to the location range and / or the time range. Optionally, if the first face image is a certificate photo category image, the third face image can be detected according to the storage range.

[0062] As another way, the detection range of the third face image can be determined according to the user's settings. In this way, a setting interface can be configured, so that the user can configure the required detection range through the setting interface.

[0063] S220: If the third face image is obtained, the eyes of the target face in the first face image are replaced with the eyes of the target face in the third face image to obtain a target image.

[0064] In the case where the first face image and the third face image are obtained, the first face image and the third face image can be aligned through the face key points. Then, the eye region can be identified to obtain the corresponding mask region, and the mapping replacement is performed to replace the eyes of the target face in the first face image with the eyes of the target face in the third face image. The mask region can be obtained by segmenting the eye region (for example, through a feature model to segment the eye region), or can be obtained by expanding the region surrounded by the eye key points.

[0065] Optionally, after the human eye replacement, a blending operation can also be performed to reduce the boundary obvious problem caused by the replacement. The blending operation is a way of superimposing color values of different graphic elements on each other to achieve transparency, semi-transparency and color mixing effects. By adjusting the factors of the source pixels and the target pixels, the mixing behavior can be controlled to make the final output value of the color of the two pixels more natural and coordinated.

[0066] It should be noted that when the first face image and the third face image are collected, the surrounding environment of the person in the image can be different. For example, when the first face image is collected, the weather is cloudy, and when the third face image is collected, the weather is sunny, in which case the brightness of the eyes in the two images will be different. In order to improve the brightness difference after replacement, as a way, the eyes of the target face in the first face image are replaced with the eyes of the target face in the third face image to obtain a first to-be-processed face image, and the eyes of the target face in the first to-be-processed face image are aligned in brightness based on a target brightness difference value to obtain a target image, wherein the brightness alignment is used to reduce the brightness difference between the eyes of the target face and the surrounding area in the target image.

[0067] The target brightness difference value is obtained based on the first brightness difference value and / or the second brightness difference value, the first brightness difference value is the brightness difference value of the left eye of the target face in the first face image and in the third face image, and the second brightness difference value is the brightness difference value of the right eye of the target face in the first face image and in the third face image.

[0068] S230: If the third face image is not obtained, a first feature vector of the first face image is obtained, and the eyes of the target face in the first face image are in a closed eye state.

[0069] S240: The first feature vector is edited based on an open eye editing vector to obtain a second feature vector, the second feature vector represents that the eyes of the target face are in an open eye state, and the open eye editing vector represents the difference between the closed eye state and the open eye state.

[0070] S250: A second face image is obtained through the second feature vector, and the eyes of the target face in the second face image are in an open eye state.

[0071] S260: The eyes of the target face in the first face image are replaced with the eyes in the second face image to obtain a target image.

[0072] This embodiment proposes an image processing method, which makes it possible to convert the eyes of a person in a first facial image that are in a closed state into an open state, thereby improving the image quality of the first facial image. In addition, in this embodiment, when the first facial image is obtained, it is further detected whether there is a third facial image corresponding to the first facial image. If the third facial image is not detected, a second facial image is generated based on the first facial image. If the third facial image is detected, the eyes in the third facial image can be directly used to replace the eyes in the second facial image to obtain the target image, thereby further improving the flexibility and adaptability of this application.

[0073] See also Figure 7 , an image processing method provided in an embodiment of the present application is applied to a server, and the method includes:

[0074] S310: If facial images sent by the user terminal are acquired, the number of facial images sent by the user terminal is detected.

[0075] The user terminal can be Figure 1 or Figure 2 The electronic device shown in .

[0076] The number of facial images sent by the user terminal is determined by whether it detects a third facial image. If the user terminal captures the first facial image but does not detect the third facial image, only the first facial image may be uploaded to the server. In this case, the number of images uploaded by the user terminal is one. If the first electronic device captures the first facial image and also detects the third facial image, both the first and third facial images may be uploaded to the server. In this case, the number of images uploaded by the user terminal is two.

[0077] S320: If the user terminal sends only one facial image, determine that the third facial image is not acquired, and use the facial image sent by the user terminal as the first facial image.

[0078] S321: If the third facial image is not obtained, obtain a first feature vector of the first facial image, wherein the third facial image includes the target face, and eyes of the target face are in an open state.

[0079] S322: Edit the first feature vector based on the eyes-open editing vector to obtain a second feature vector, wherein the second feature vector represents that the eyes of the target face are in an eyes-open state, and the eyes-open editing vector represents the difference between the eyes being in a closed state and in an eyes-open state.

[0080] S323: obtaining a second face image through the second feature vector, wherein the eyes of the target face in the second face image are in an open eye state.

[0081] S324: replacing the eyes of the target face in the first face image with the eyes of the target face in the second face image to obtain a target image.

[0082] S331: if the face image sent by the user terminal is two, determining that the third face image is obtained, and taking the image including the eyes in a closed eye state as the first face image.

[0083] S332: if the third face image is obtained, replacing the eyes of the target face in the first face image with the eyes of the target face in the third face image to obtain a target image.

[0084] Before the eyes of the first face image are replaced, the first face image and the third face image can be aligned first. For example, the face key points in the first face image and the face key points in the third face image can be aligned. Optionally, the face key points in the first face image and the face key points in the third face image can be obtained locally by the server, or can be obtained by the user terminal and then uploaded to the server.

[0085] The embodiment provides an image processing method, so that the eyes in a closed eye state in the first face image can be converted to eyes in an open eye state, so as to improve the image quality of the first face image. In the embodiment, the user terminal can upload different numbers of face images to the server according to different detection conditions. Correspondingly, the server can adopt different ways to obtain a target image according to the number of face images uploaded by the user terminal, so as to improve the flexibility and adaptability of the image processing method provided by the application.

[0086] Referring to Figure 8 The image processing method provided by the embodiment of the application comprises:

[0087] S410: if there are eyes in a closed eye state in the currently displayed image, taking the face where the eyes are as a target face, and taking the currently displayed image as a first face image.

[0088] As a way, the electronic device can automatically trigger detection of whether there are eyes in a closed eye state in the currently displayed object when it is detected that there is image display (for example, full-screen image display). Alternatively, as another way, the electronic device can start detecting whether there are eyes in a closed eye state in the currently displayed object in response to a user operation.

[0089] S420: obtain a first feature vector of the first face image, wherein the eyes of the target face in the first face image are in a closed eye state.

[0090] As one way, if no fourth face image is detected, a first feature vector of the first face image is obtained, wherein the interval time between the collection time of the fourth face image and the collection time of the currently displayed image is less than a time threshold, and the eyes of the target face in the fourth face image are in an open eye state. If the fourth face image is detected, the eyes of the target face in the first face image are replaced by the eyes in the fourth face image to obtain a target image.

[0091] S430: editing the first feature vector based on an open eye editing vector to obtain a second feature vector, wherein the second feature vector represents that the eyes of the target face are in an open eye state, and the open eye editing vector represents the difference between the closed eye state and the open eye state.

[0092] S440: obtaining a second face image through the second feature vector, wherein the eyes of the target face in the second face image are in an open eye state.

[0093] S450: replacing the eyes of the target face in the first face image with the eyes in the second face image to obtain a target image.

[0094] As one way, after obtaining the target image, the currently displayed image can be automatically replaced by the target image, so that the switching of the eye state (from the closed eye state to the open eye state) can be automatically performed according to the image display operation triggered by the user.

[0095] The embodiment provides an image processing method, so that the eyes in a closed eye state in a first face image can be converted to an open eye state, so as to improve the image quality of the first face image. In the embodiment, the currently displayed image can be used as the first face image, so that the eye replacement can be triggered by the display of the image. After obtaining the target image, the currently displayed image (the first face image) can be replaced by the obtained target image, so that after the user triggers the display of the first face image, the user can replace the eyes in the currently displayed face image without additional operations, thereby improving the user convenience.

[0096] Referring to Figure 9 The embodiment of the present application provides an image processing device 500, and the method comprises the following steps:

[0097] The feature vector obtaining unit 510 is configured to obtain a first feature vector of a first face image, wherein eyes of a target face in the first face image are in a closed eye state.

[0098] The vector editing unit 520 is configured to edit the first feature vector based on an open eye editing vector to obtain a second feature vector, wherein the second feature vector represents that the eyes of the target face are in an open eye state, and the open eye editing vector represents a difference between the closed eye state and the open eye state.

[0099] The image generating unit 530 is configured to obtain a second face image through the second feature vector, wherein the eyes of the target face in the second face image are in the open eye state.

[0100] The image processing unit 540 is configured to replace the eyes of the target face in the first face image with the eyes of the target face in the second face image to obtain a target image.

[0101] As a manner, the feature vector obtaining unit 510 is specifically configured to obtain the first feature vector of the first face image if a third face image is not obtained, wherein the third face image includes the target face, and the eyes of the target face are in the open eye state; and the image processing unit 540 is specifically configured to replace the eyes of the target face in the first face image with the eyes of the target face in the third face image to obtain the target image if the third face image is obtained.

[0102] Optionally, the image processing unit 540 is specifically configured to replace the eyes of the target face in the first face image with the eyes of the target face in the third face image to obtain a first face image to be processed; and perform brightness alignment on the eyes of the target face in the first face image to be processed based on a target brightness difference value to obtain the target image, wherein the brightness alignment is used to reduce a brightness difference between the eyes of the target face and a surrounding area in the target image; and the target brightness difference value is obtained based on a first brightness difference value and / or a second brightness difference value, the first brightness difference value is a brightness difference value of a left eye of the target face in the first face image and in the third face image, and the second brightness difference value is a brightness difference value of a right eye of the target face in the first face image and in the third face image.

[0103] As a manner, the feature vector acquisition unit 510 is specifically configured to: if the face image sent by the user terminal is acquired, detect the number of the face image sent by the user terminal; if the face image sent by the user terminal is one, determine that the third face image is not acquired, and take the face image sent by the user terminal as the first face image; if the face image sent by the user terminal is two, determine that the third face image is acquired, and take the image including the eye in the closed state in the two face images as the first face image.

[0104] As a manner, the feature vector acquisition unit 510 is specifically configured to: if the eye in the closed state is in the currently displayed image, take the face where the eye is located as the target face, and take the currently displayed image as the first face image. The feature vector acquisition unit 510 is further specifically configured to: if it is detected that there is no fourth face image, acquire the first feature vector of the first face image, wherein the interval time length between the acquisition time of the fourth face image and the acquisition time of the currently displayed image is less than the time length threshold, and the eye in the target face in the fourth face image is in the open state.

[0105] As a manner, the image processing unit 540 is specifically configured to: replace the eye of the target face in the first face image with the eye in the second face image to obtain a second to-be-processed face image; and perform artifact elimination on the second to-be-processed face image through the target generation model to obtain a target image.

[0106] The embodiment provides an image processing device, so that the first feature vector of the first face image can be edited through the image processing device to obtain a second feature vector representing that the eye of the target face is in the open state, and then the second face image for performing eye replacement is obtained through the second feature vector, and then the eye in the closed state in the first face image is replaced with the eye in the open state in the second face image, so that the eye in the closed state in the first face image can be converted to the open state, and the image quality of the first face image is improved.

[0107] The following will be combined with the specific implementation of the embodiment to explain the technical solutions of the present application. Figure 10 An electronic device provided by the present application is described.

[0108] Please refer to Figure 10, based on the image processing method and device described above, the electronic device 1000 provided by the embodiment of the present application can execute the image processing method described above. The electronic device 1000 includes one or more (only one is shown in the figure) processors 105, a memory 104, an audio playing module 106, and an audio acquisition device 108 which are coupled to each other. The memory 104 stores programs that can execute the contents in the foregoing embodiments, and the processor 105 can execute the programs stored in the memory 104.

[0109] The processor 105 can include one or more processing cores. The processor 105 connects various parts in the entire electronic device 1000 by various interfaces and lines, executes various functions of the electronic device 1000 and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 104, and calling data stored in the memory 104. Optionally, the processor 105 can be implemented in at least one of the hardware forms of a digital signal processing (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor 105 can be integrated with a combination of one or more of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used for processing wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor 105, but can be realized by a separate communication chip.

[0110] The memory 104 can include a random access memory (RAM) and a read-only memory (ROM). The memory 104 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 104 can include a program storage area and a data storage area, wherein the program storage area can store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playing function, an image playing function, etc.), instructions for implementing each of the methods described below, etc.

[0111] In addition, the electronic device 1000 can further include a network module 110 and a sensor module 112 in addition to the devices shown in the foregoing.

[0112] The network module 110 is configured to realize information interaction between the electronic device 1000 and other devices. For example, the network module 110 can establish a connection with other audio playback devices or other electronic devices, and perform information interaction based on the established connection. As one way, the network module 110 of the electronic device 1000 is a radio frequency module configured to receive and send electromagnetic waves, realize mutual conversion between electromagnetic waves and electrical signals, and thus communicate with a communication network or other devices. The radio frequency module can include various existing circuit elements for performing these functions, such as an antenna, a radio frequency transceiver, a digital signal processor, an encryption / decryption chip, a subscriber identity module (SIM) card, a memory, and the like. For example, the radio frequency module can interact with external devices through transmitted or received electromagnetic waves.

[0113] The sensor module 112 can include at least one sensor. Specifically, the sensor module 112 can include, but is not limited to, a pressure sensor, a motion sensor, an acceleration sensor, and other sensors.

[0114] The pressure sensor can detect pressure generated by pressing on the electronic device 1000. That is, the pressure sensor detects pressure generated by contact or pressing between a user and the electronic device 1000, such as pressure generated by contact or pressing between the user's ear and the electronic device 1000. Therefore, the pressure sensor can be used to determine whether contact or pressing occurs between the user and the electronic device 1000, and the magnitude of the pressure.

[0115] The acceleration sensor can detect the magnitude of acceleration in each direction (generally three axes), and when at rest, can detect the magnitude and direction of gravity, and can be used for applications that identify the posture of the electronic device 1000 (such as switching between landscape and portrait screens, related games, magnetometer posture calibration), vibration recognition related functions (such as a pedometer, tapping), and the like. In addition, the electronic device 1000 can also be configured with a gyroscope, a barometer, a hygrometer, a thermometer, and other sensors, which will not be described here.

[0116] The audio acquisition device 108 is configured to acquire an audio signal. Optionally, the audio acquisition device 108 includes a plurality of audio acquisition devices, which can be microphones.

[0117] Please refer to Figure 11 which shows a structural block diagram of a computer readable storage medium provided by an embodiment of the present application. The computer readable medium 800 stores program code, which can be called by a processor to execute the method described in the above method embodiment.

[0118] The computer-readable storage medium 800 can be an electronic memory such as a flash memory, EEPROM (electrically erasable programmable read-only memory), EPROM, hard disk or ROM. Optionally, the computer-readable storage medium 800 comprises a non-volatile computer-readable medium. The computer-readable storage medium 800 has storage space for program code 810 to perform any of the method steps described above. These program codes can be read from or written to one or more computer program products. The program code 810 can be compressed, for example, in a suitable form.

[0119] To sum up, the present application proposes an image processing method and device and electronic equipment. After obtaining a first feature vector of a first face image, the first feature vector is edited based on an open-eye editing vector to obtain a second feature vector. Then, a second face image is obtained through the second feature vector. The eyes of a target face in the second face image are in an open-eye state. Further, the eyes of the target face in the first face image are replaced with the eyes in the second face image to obtain a target image. In this way, the first feature vector of the first face image is edited to obtain a second feature vector representing the eyes of the target face in an open-eye state. Then, the second face image for eye replacement is obtained through the second feature vector. Finally, the eyes in the first face image in a closed-eye state are replaced with the eyes in the second face image in an open-eye state. Thus, the eyes in the first face image in a closed-eye state can be converted to an open-eye state, thereby improving the image quality of the first face image.

[0120] Moreover, the embodiments of the present application have corresponding processing methods for two situations that may occur in actual application, i.e., for the same target face, there are both eyes in a closed-eye state and eyes in an open-eye state, and for the same target face, there are only eyes in a closed-eye state. For example, if there is an open-eye state picture, a replacement fusion method is used. If there is no open-eye state picture, a generation plus replacement fusion method is used to achieve the purpose of opening the eyes in the target face, thereby ensuring the authenticity of the open-eye image as much as possible.

[0121] Finally, it should be noted that the above examples are only used to illustrate the technical solutions of the present application, and are not intended to limit the same; although the present application has been described in detail with reference to the foregoing examples, those of ordinary skill in the art will understand that they can still modify the technical solutions described in the foregoing examples, or make equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An image processing method, characterized by, The method comprises: obtaining a first feature vector of a first face image, wherein eyes of a target face in the first face image are in a closed eye state; editing the first feature vector based on an open eye editing vector to obtain a second feature vector, wherein the second feature vector represents that the eyes of the target face are in an open eye state, and the open eye editing vector represents a difference between the eyes being in the closed eye state and the open eye state; obtaining a second face image through the second feature vector, wherein the eyes of the target face in the second face image are in the open eye state; replacing the eyes of the target face in the first face image with the eyes of the target face in the second face image to obtain a target image.

2. The method of claim 1, wherein, The method comprises: if a third face image is not obtained, obtaining a first feature vector of a first face image, wherein the third face image comprises the target face, and the eyes of the target face are in an open eye state.

3. The method of claim 2, wherein, The method further comprises: if the third face image is obtained, replacing the eyes of the target face in the first face image with the eyes of the target face in the third face image to obtain a target image.

4. The method of claim 3, wherein, The method further comprises: replacing the eyes of the target face in the first face image with the eyes of the target face in the third face image to obtain a target image, comprising: replacing the eyes of the target face in the first face image with the eyes of the target face in the third face image to obtain a first face image to be processed; performing brightness alignment on the eyes of the target face in the first face image to be processed based on a target brightness difference value, so as to obtain a target image, wherein the brightness alignment is used to reduce a brightness difference between the eyes of the target face and a surrounding area in the target image; 5. The method according to any of claims 2-4, characterized by, wherein the target brightness difference value is obtained based on a first brightness difference value and / or a second brightness difference value, the first brightness difference value is a brightness difference value of a left eye of the target face in the first face image and in the third face image, and the second brightness difference value is a brightness difference value of a right eye of the target face in the first face image and in the third face image. The method is applied to a server, and the method further comprises: if a face image sent by a user terminal is obtained, detecting a number of face images sent by the user terminal; if the face image sent by the user terminal is one, determining that a third face image is not obtained, and taking the face image sent by the user terminal as a first face image; 6. The method of claim 1, wherein, if the face image sent by the user terminal is two, determining that the third face image is obtained, and taking an image comprising eyes in a closed eye state from the two face images as the first face image. The method further comprises:

7. The method of claim 6, wherein, if there are eyes in a closed eye state in a currently displayed image, taking a face where the eyes are as a target face, and taking the currently displayed image as a first face image. The method further comprises: If it is detected that there is no fourth face image, a first feature vector of the first face image is obtained, wherein an interval time between a collection time of the fourth face image and a collection time of the currently displayed image is less than a time threshold, and in the target face in the fourth face image, the human eyes are in an open eye state.

8. The method of claim 1, wherein, The method further comprises: obtaining a first distance, the first distance being a distance between a first key point in an upper eyelid of the human eye and a second key point in a lower eyelid of the human eye; obtaining a second distance, the second distance being a width of the human eye; if a ratio of the first distance to the second distance is less than a first proportion threshold, determining that the human eyes are in a closed eye state.

9. The method of claim 8, wherein, The method further comprises: if the ratio of the first distance to the second distance is greater than a second proportion threshold, determining that the human eyes are in an open eye state, the second proportion threshold being greater than the first proportion threshold.

10. The method of claim 1, wherein, The replacing the human eyes of the target face in the first face image with the human eyes in the second face image to obtain a target image comprises: replacing the human eyes of the target face in the first face image with the human eyes in the second face image to obtain a second face image to be processed; performing artifact elimination on the second face image to be processed through a target generation model to obtain a target image.

11. An image processing apparatus characterized by comprising: The method comprises: a feature vector obtaining unit configured to obtain a first feature vector of a first face image, wherein human eyes of a target face in the first face image are in a closed eye state; a vector editing unit configured to edit the first feature vector based on an open eye editing vector to obtain a second feature vector, wherein the second feature vector represents that the human eyes of the target face are in an open eye state, and the open eye editing vector represents a difference between a closed eye state and an open eye state of the human eyes; an image generation unit configured to obtain a second face image through the second feature vector, wherein human eyes of a target face in the second face image are in an open eye state; an image processing unit configured to replace the human eyes of the target face in the first face image with the human eyes in the second face image to obtain a target image.

12. An electronic device, comprising: The apparatus comprises a processor and a memory, one or more programs are stored in the memory and configured to be executed by the processor to implement the method of any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer readable storage medium stores program codes, wherein when the program codes are run by a processor, the method of any one of claims 1-10 is executed.

Images