Image processing method and device

Abstract

The invention discloses an image processing method and device, and belongs to the technical field of electronics. The method comprises the steps that first information of a human eye part in a first image is acquired, the first information comprises at least one of a first value and a second value, the first value represents the state of the human eye part, and the second value represents the image definition corresponding to the human eye part; under the condition that the first information meets the first condition, the first model is adopted to process the first image to obtain the second image, the second model is adopted to process the first image to obtain the third image, and the image enhancement capability of the first model is higher than that of the second model; the image enhancement capability is used for representing the amplitude of adjusting the parameters of the image when the model processes the image; and performing image fusion processing based on the second image and the third image to obtain a fourth image.

Description

Technical Field

[0001] The present application belongs to the field of electronic technology, and specifically relates to an image processing method and a device thereof. Background Art

[0002] When watching live performances such as concerts, music festivals, and dramas, users often need to use electronic devices to take pictures of the performers. However, since the stage is far away and the overall brightness is dim, the clarity of the faces in the photos taken by the electronic devices is poor.

[0003] The current method for improving the clarity of face photography is mainly as follows: electronic devices use face enhancement algorithms to enhance the details of faces in photographs to improve facial clarity.

[0004] However, if the subject in the photo has closed eyes or the eye area is blurred, the eye area in the image processed by the face enhancement algorithm will be very different from the eye area in the original photo, resulting in problems of fake eyes or abnormal eyes, thus causing poor image quality obtained by the electronic device. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide an image processing method and device thereof, which can improve the quality of images processed by electronic devices.

[0006] In a first aspect, an embodiment of the present application provides an image processing method, the method comprising: obtaining first information of a human eye part in a first image, the first information comprising at least one of the following: a first value and a second value, the first value representing a state of the human eye part, and the second value representing an image clarity corresponding to the human eye part; when the first information satisfies a first condition, using a first model to process the first image to obtain a second image, and using a second model to process the first image to obtain a third image, the image enhancement capability of the first model being higher than that of the second model, and the image enhancement capability being used to represent the amplitude of adjustment of image parameters when the model processes the image; performing image fusion processing based on the second image and the third image to obtain a fourth image.

[0007] In a second aspect, an embodiment of the present application provides an image processing device, which includes: an acquisition module and a processing module. The acquisition module is used to acquire first information about a human eye part in a first image, and the first information includes at least one of the following: a first value and a second value, the first value representing the state of the human eye part, and the second value representing the image clarity corresponding to the human eye part. The processing module is used to process the first image using a first model to obtain a second image, and to process the first image using a second model to obtain a third image when the first information acquired by the acquisition module meets a first condition, and the image enhancement capability of the first model is higher than the image enhancement capability of the second model, and the image enhancement capability is used to characterize the amplitude of adjustment of the parameters of the image when the model processes the image. The processing module is also used to perform image fusion processing based on the second image and the third image to obtain a fourth image.

[0008] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0009] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0010] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0011] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0012] In an embodiment of the present application, first information of a human eye part in a first image can be obtained, and the first information includes at least one of the following: a first value and a second value, wherein the first value represents a state of the human eye part, and the second value represents an image clarity corresponding to the human eye part. When the first information satisfies a first condition, the first image is processed using a first model to obtain a second image, and the first image is processed using a second model with lower image enhancement capability than the first model to obtain a third image, and then image fusion processing is performed based on the second image and the third image to obtain a fourth image. In the present solution, since the state of the human eye part, that is, whether the human eye part is in a closed state, can be determined by the first value, or the image clarity corresponding to the human eye part can be determined by the second value, when the human eye part is in a closed state or the image clarity corresponding to the human eye part is low, a second image with a stronger enhancement effect and a third image with a weaker enhancement effect are generated, and image fusion processing is performed based on the second image and the third image, that is, the image area corresponding to the human eye part in the fourth image obtained by the image fusion processing is obtained by fusion of the image area corresponding to the human eye part with a strong enhancement effect and the image area corresponding to the human eye part with a weak enhancement effect, so that the enhancement effect of the image area corresponding to the human eye part in the fourth image reaches a balance, thereby avoiding the problem of false human eyes or abnormal human eyes caused by strong enhancement effect on the closed eye part or the low-definition human eye part, thereby improving the image quality processed by the electronic device. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is one of the flow charts of the image processing method provided in the embodiment of the present application;

[0014] Figure 2 is a schematic diagram of a human eye provided in an embodiment of the present application;

[0015] Figure 3 This is the second flowchart of the image processing method provided in the embodiment of the present application;

[0016] Figure 4 is a schematic diagram of characteristic points provided in an embodiment of the present application;

[0017] Figure 5 This is the third flowchart of the image processing method provided in the embodiment of the present application;

[0018] Figure 6 It is a schematic diagram of the vertical direction and the horizontal direction provided by the embodiment of the present application;

[0019] Figure 7 is one of the schematic diagrams of the first distance and the second distance provided in the embodiment of the present application;

[0020] Figure 8This is the second schematic diagram of the first distance and the second distance provided in the embodiment of the present application;

[0021] Fig. 9 This is the fourth flowchart of the image processing method provided in the embodiment of the present application;

[0022] Fig.10 is a schematic diagram of a circumscribed rectangle provided in an embodiment of the present application;

[0023] Fig.11 This is the fifth flowchart of the image processing method provided in the embodiment of the present application;

[0024] Fig.12 This is the sixth flowchart of the image processing method provided in the embodiment of the present application;

[0025] Fig.13 is a schematic diagram of a first blank image provided in an embodiment of the present application;

[0026] Fig.14 is one of the schematic diagrams of the image processing device provided in the embodiment of the present application;

[0027] Fig.15 This is a second schematic diagram of the image processing device provided in an embodiment of the present application;

[0028] Fig.16 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0029] Fig.17 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0031] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0032] The terms "at least one (item)", "at least one of" and the like in this application refer to any one, any two or a combination of more than two of the objects included therein. For example, at least one (item) of a, b, and c can be represented by: "a", "b", "c", "a and b", "a and c", "b and c" and "a, b and c", where a, b, and c can be single or multiple. Similarly, "at least two (items)" refers to two or more, and its meaning is similar to that of "at least one (item)".

[0033] The image processing method and device provided by the embodiments of the present application are described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0034] The embodiments of the present application can be applied to scenarios where electronic devices process images containing human faces, such as processing facial images collected in performance activity scenes, security monitoring scenes, news reporting scenes, artistic creation scenes, or ID photo production scenes.

[0035] For example, for performance activities:

[0036] The embodiments of the present application can be applied to scenes where the audience uses electronic devices to shoot performers in live performances such as concerts, music festivals, and dramas. For example, at a rock music festival, the audience wants to shoot a singer who is singing passionately on the stage, but due to the flickering stage lights and the distance from the stage, the singer's face is a little blurry in the photo. At this time, the electronic device automatically determines that the clarity of the human eye part in the photo is low, so the strong enhancement model and the weak enhancement model are used to enhance the photo respectively, and the other areas of the photo processed by the strong enhancement model except the human eye part are merged with the area of ​​the human eye part in the photo processed by the weak enhancement model into one photo, avoiding the problem of false human eyes or abnormal human eyes caused by the strong enhancement model for processing the low-definition human eye part in the photo, so that the audience can get a photo with a clear face of the singer without false human eyes or abnormal human eyes, so as to better preserve the memory.

[0037] For another example, for a news report scenario:

[0038] The embodiments of the present application can be applied to scenarios where electronic devices are used for news editing and processing news picture materials. For example, when a news editor is processing a live picture of an international conference, an important diplomat in the picture happens to be blinking. When the electronic device performs enhancement processing on the entire picture, it detects that the human eye part of the diplomat's face is in a closed state, and then the electronic device uses a strong enhancement model and a weak enhancement model to enhance the picture respectively, and then merges the other areas of the picture except the human eye part obtained by the strong enhancement model with the human eye part area of ​​the picture obtained by the weak enhancement model into one picture, avoiding the problem of false human eyes or abnormal human eyes caused by the strong enhancement model for processing the closed human eye part, so that the face of the diplomat in this picture is clear and the eye part is normal, which can be better used for news reporting.

[0039] The embodiment of the present application provides a picture processing method and device thereof. Since the state of a human eye part, that is, whether the human eye part is in a closed state, can be determined by a first value, or the clarity of the image corresponding to the human eye part can be determined by a second value, when the human eye part is in a closed state or the clarity of the image corresponding to the human eye part is low, the electronic device generates a second image with a stronger enhancement effect and a third image with a weaker enhancement effect, and performs image fusion processing based on the second image and the third image, that is, other areas outside the human eye part in the second image are fused with the area of ​​the human eye part in the third image into one image, thereby avoiding the problem of false human eyes or abnormal human eyes caused by strong enhancement effect on human eye parts in a closed state or low-definition human eye parts, thereby improving the image quality processed by the electronic device.

[0040] The image processing method provided in the embodiment of the present application may be executed by an image processing device, which may be an electronic device, or a functional module or functional entity in an electronic device. The technical solution provided in the embodiment of the present application is described below using an electronic device as an example.

[0041] Figure 1 A flowchart of an image processing method provided by an embodiment of the present application is shown. Figure 1 As shown, the image processing method provided in the embodiment of the present application may include the following steps 201 to 203.

[0042] Step 201: The electronic device obtains first information of a human eye region in a first image.

[0043] In an embodiment of the present application, the first information includes at least one of the following: a first value and a second value, the first value representing the state of the human eye part, and the second value representing the image clarity corresponding to the human eye part.

[0044] Optionally, in an embodiment of the present application, the first image may be an image containing a face, which may be referred to as a face image.

[0045] For example, a mobile phone is used as an example. Figure 2 As shown, the camera application interface 10 includes a face image 11 , and the face image 11 includes an eye area 12 .

[0046] Optionally, in an embodiment of the present application, the above-mentioned first image can be an image taken by a user through a camera of the electronic device, or an image selected by a user from a photo album of the electronic device, or an image imported by a user from other storage devices to the electronic device, or an image downloaded by the electronic device from the Internet. The specific image can be determined based on actual usage requirements and is not limited in the embodiment of the present application.

[0047] For example, the first image is an image captured by a user through a camera of an electronic device. When a user attends a concert, in order to record the wonderful moment of the singer's singing, the user uses the shooting function of the electronic device to shoot the singer on the stage. The electronic device can process the captured image as the first image.

[0048] As another example, the first image is an image selected by the user from the photo album of the electronic device. Assuming that the user's photo album contains a selfie of the user standing in front of an ancient castle taken during a trip, and the user selects the selfie, the electronic device can process the selfie as the first image.

[0049] As another example, the first image is an image that the user imports from another storage device into the electronic device. Assume that the user has an external hard disk that stores photos of activities taken at work. The user imports a group photo of an activity site from the external hard disk into the electronic device, and the electronic device can process the group photo as the first image.

[0050] As another example, the first image is an image downloaded from the Internet by the electronic device. Assume that the user is collecting information for a medical report, and when browsing a medical website, the user downloads a very rare patient photo from the website interface to the electronic device, and the electronic device can use the patient photo as the first image.

[0051] In the embodiment of the present application, the above-mentioned human eye part is the eye of the person in the first image, that is, the human eye contained in the first image.

[0052] Optionally, in an embodiment of the present application, the human eye part in the above-mentioned first image may be a left eye part or a right eye part.

[0053] Optionally, in the embodiment of the present application, the state of the above-mentioned human eye part can be any one of the following two states: an open state and a closed state.

[0054] Optionally, in the embodiment of the present application, combined with Figure 1 ,like Figure 3 As shown, before the above step 201, the image processing method provided in the embodiment of the present application further includes the following step 301, and the above step 201 can be implemented through the following step 302.

[0055] Step 301: The electronic device determines coordinate information of feature points of a human eye from a first image.

[0056] In the embodiment of the present application, the above-mentioned characteristic points of the human eye area are points used to characterize the area of ​​the human eye area in the first image.

[0057] Optionally, in an embodiment of the present application, the electronic device may use a convolutional neural network to extract features from the first image, and detect feature points of the human eye using a regression method. It may also use a template matching-based method to detect feature points of the human eye by comparing a pre-trained template with a facial image. The specific method used may be determined based on actual usage requirements and is not limited in the embodiment of the present application.

[0058] Optionally, in the embodiment of the present application, the number of feature points in the above human eye part is greater than or equal to 4, for example, it can be 4, 8, 12, etc. The embodiment of the present application takes 8 feature points as an example for explanation. The specific number can be determined according to actual use requirements, and the embodiment of the present application does not limit it.

[0059] For example, Figure 4 As shown, assuming Figure 4 For the self-portrait image 14 taken by the user, taking the left eye as an example, the electronic device extracts 8 feature points from the left eye part 15 of the self-portrait image 14. The feature points of the left eye part 15 are P0 to P7, among which the leftmost feature point is P0, the rightmost feature point is P4, the top feature point is P2, and the bottom feature point is P6. The coordinates of the feature points are P0(25,60), P1(40,68), P2(50,70), P3(60,68), P4(75,60), P5(60,52), P6(50,50), and P7(40,52).

[0060] Step 302: The electronic device calculates first information based on the coordinate information.

[0061] In this way, the electronic device can extract the coordinate information of the feature points of the human eye part, and thus can determine the state and clarity of the human eye part in the first image according to the coordinate information.

[0062] Optionally, in the embodiment of the present application, the first information includes a first value. Figure 3 ,like Figure 5 As shown, the above step 302 can be implemented through the following steps 302a and 302b.

[0063] Step 302a: The electronic device calculates at least one first distance and a second distance based on at least four feature points of the human eye.

[0064] In the embodiment of the present application, the first distance is the distance between the two feature points in the vertical direction, and the second distance is the distance between the two feature points in the horizontal direction.

[0065] In the embodiment of the present application, the vertical direction is: the direction from the upper eyelid edge to the lower eyelid edge, or from the lower eyelid edge to the upper eyelid edge. The horizontal direction is: the direction from the outer corner of the eye to the inner corner of the eye, or from the inner corner of the eye to the outer corner of the eye.

[0066] It can be understood that the above vertical direction and horizontal direction are the vertical direction and horizontal direction relative to the human eye, rather than the vertical direction and horizontal direction relative to the first image.

[0067] For example, Figure 6 As shown in (a), the left eye is used as an example for explanation. With the pupil 16 of the left eye 15 as the center and the direction from the outer corner of the eye 17 to the inner corner of the eye 18 as the x-axis, a rectangular coordinate system is established, and the vertical direction is the direction of the y-axis or the negative direction of the y-axis. For example, the three straight lines x=-1, x=0 or x=1 point to the vertical direction. Figure 6 As shown in (b), the horizontal direction is the direction of the x-axis or the negative direction of the x-axis. For example, the directions pointed by the three straight lines y=-1, y=0 or y=1 are all horizontal directions.

[0068] It can be understood that since the upper eyelid edge and the lower eyelid edge of the human eye are two curves with length, at least one feature point can be selected from the upper eyelid edge and the lower eyelid edge, so the number of the first distance can be at least one. The outer corner of the eye and the inner corner of the eye are two points, so these two points can be directly selected as feature points, so the number of the second distance is one.

[0069] Optionally, in the embodiment of the present application, the number of the first distances is determined by the number of feature points. For example, when the number of feature points is 4, the number of the first distances is 1, and when the number of feature points is 8, the number of the first distances is 3. The specific number can be determined according to actual use requirements and is not limited in the embodiment of the present application.

[0070] It should be noted that the first distance and the second distance are the Euclidean distance between the feature points, that is, the L2 norm, and the specific formula is as follows:

[0071]

[0072] Where d is the first distance or the second distance, x 1 and 1 is the horizontal and vertical coordinates of one of the feature points in the first distance or the second distance, x 2 and 2 are the horizontal and vertical coordinates of another feature point in the first distance or the second distance.

[0073] For example, when the number of feature points is 4, the 4 feature points may be the top feature point, the bottom feature point, the leftmost feature point and the rightmost feature point of the human eye. In this case, the first distance is the distance between the top feature point and the bottom feature point, and the second distance is the distance between the leftmost feature point and the rightmost feature point. Figure 4 ,like Figure 7 As shown, assuming that the four feature points are P0, P2, P4, and P6, the first distance is the distance between P2 and P6, that is, pixels, the second distance is the distance between P0 and P4, that is Pixels.

[0074] For example, when the number of feature points is 8, the 8 feature points may be the top feature point, the bottom feature point, the leftmost feature point, the rightmost feature point, the top left feature point, the bottom left feature point, the top right feature point and the bottom right feature point of the human eye. In this case, the first distance is the distance between the top feature point and the bottom feature point, and the second distance is the distance between the leftmost feature point and the rightmost feature point. Figure 4 ,like Figure 8 As shown, assuming that the eight feature points are P0, P1, P2, P3, P4, P5, P6, and P7, there are three first distances, namely the distance between P2 and P6, the distance between P1 and P7, and the distance between P3 and P5. The distance between P2 and P6 is pixels, the distance between P1 and P7 is pixels, the distance between P3 and P5 is pixels, the second distance is the distance between P0 and P4, that is Pixels.

[0075] Step 302b: The electronic device calculates a first value based on at least one first distance and a second distance.

[0076] In the embodiment of the present application, the first value is the ratio of the lengths of the human eye part in the vertical direction and the horizontal direction.

[0077] It can be understood that since the first value represents the state of the human eye, that is, it is used to determine whether the human eye is in an open or closed state, and the length from the inner corner of the eye to the outer corner of the eye is equal in the open and closed states, and only the degree of opening and closing of the upper eyelid edge and the lower eyelid edge is different, the first distance representing the distance between the upper eyelid edge and the lower eyelid edge is used as the numerator,

[0078] In the embodiment of the present application, the first value is calculated by: the sum of the at least one first distance divided by the product of the number of first distances and the second distance. The specific formula is as follows: Formula (2):

[0079]

[0080] Where EAR is the first value, n is the number of first distances, d 1 to d n is the n first distances, d 0 is the second distance.

[0081] Optionally, in the embodiment of the present application, when the number of feature points is 4, that is, when the number of first distances is 1, the above formula can be simplified to the following formula (3):

[0082]

[0083] Among them, EAR is the first value, d 1 is the first distance, d 0 is the second distance.

[0084] Optionally, in the embodiment of the present application, when the number of feature points is 8, that is, when the number of first distances is 3, the above formula can be simplified to the following formula (4):

[0085]

[0086] Among them, EAR is the first value, d 1 d 2 d 3 They are the three first distances, d 0 is the second distance.

[0087] It should be noted that the first value can be calculated using 4 feature points, i.e., 1 first distance and 1 second distance. However, due to individual differences, the morphology of the human eye may be different. In order to take into account various forms of human eye parts and improve the calculation accuracy of the first value, the image processing method provided in the embodiment of the present application usually uses 8 feature points, i.e., 3 first distances to calculate the first value, thereby greatly improving the calculation accuracy of the first value while keeping the calculation amount as low as possible.

[0088] For example, Figure 8 As shown, assuming that the eight feature points are P0, P1, P2, P3, P4, P5, P6, and P7, and there are three first distances, namely, the distance between P2 and P6, the distance between P1 and P7, and the distance between P3 and P5, the calculation formula of the first value can be the following formula (5):

[0089]

[0090] Among them, EAR is the first value, || represents the L2 norm, that is, the Euclidean distance. The meaning of the above formula is that the first value is equal to the sum of the distances between P1 and P7, P2 and P6, and P3 and P5, divided by 3 times the distance between P0 and P4, that is, the first value is: About 0.3467.

[0091] Optionally, in an embodiment of the present application, when the first image includes both the left eye and the right eye of the same person, the electronic device can calculate the ratio of the left eye and the ratio of the right eye respectively, and take the average of the two ratios as the first value.

[0092] In this way, the electronic device can calculate the length ratio of the human eye in the vertical direction and the horizontal direction through the feature points, and thus can determine the state of the human eye according to the ratio.

[0093] Optionally, in the embodiment of the present application, the first information includes a second value. Figure 3 ,like Fig. 9 As shown, the above step 302 can be implemented through the following steps 302c to 302e.

[0094] Step 302c: The electronic device establishes a circumscribed rectangle based on the coordinate information.

[0095] In the embodiment of the present application, the circumscribed rectangle includes the image area corresponding to the human eye.

[0096] In an embodiment of the present application, the electronic device can determine the four boundaries of the circumscribed rectangle based on the coordinates of the above-mentioned at least one feature point. The specific method is: compare the horizontal coordinate of the above-mentioned at least one feature point, determine the minimum value Vl and the maximum value Vr, determine the straight line x=Vl as the left boundary of the circumscribed rectangle, and determine the straight line x=Vr as the right boundary of the circumscribed rectangle, and compare the vertical coordinate of the above-mentioned at least one feature point to determine the minimum value Vt and the maximum value Vb, determine the straight line y=Vb as the upper boundary of the circumscribed rectangle, and determine the straight line y=Vt as the lower boundary of the circumscribed rectangle. The area enclosed by the four straight lines x=Vl, x=Vr, y=Vt, and y=Vb is the above-mentioned circumscribed rectangle, that is, a rectangle whose four vertex coordinates are (Vl, Vt), (Vl, Vb), (Vr, Vt), and (Vr, Vb) respectively.

[0097] For example, in combination Figure 4 ,like Fig.10 As shown, the coordinates of the feature points in the left eye area 15 are P0 (25, 60), P1 (40, 68), P2 (50, 70), P3 (60, 68), P4 (75, 60), P5 (60, 52), P6 (50, 50), and P7 (40, 52). The electronic device can compare the horizontal coordinates of all the feature points and determine that the minimum value is 25 and the maximum value is 75, so that x=25 is used as the left boundary 19 of the circumscribed rectangle and x=75 is used as The right boundary 20 of the circumscribed rectangle is determined by comparing the vertical coordinates of all feature points to determine that the minimum value is 50 and the maximum value is 70, so that y=70 is used as the upper boundary 21 of the circumscribed rectangle, and y=50 is used as the lower boundary 22 of the circumscribed rectangle. The area 23 enclosed by the four straight lines x=25, x=75, y=70, and y=50 is the circumscribed rectangle, that is, a rectangle with four vertex coordinates of (25, 70), (25, 50), (75, 70), and (70, 50) respectively.

[0098] Step 302d: The electronic device performs convolution calculation on the grayscale value of each pixel in the circumscribed rectangle and the filter template to obtain at least one convolution value.

[0099] In the embodiment of the present application, the electronic device can convert the image area where the external matrix in the first image is located into a grayscale image.

[0100] It should be noted that color images usually use the Red-Green-Blue (RGB) color model, where each pixel consists of three color channels, and the value range of each channel is usually 0 to 255. For example, the pixel value (also called RGB value) of a pixel is (255,0,0), which represents pure red. The above grayscale value is used to represent the brightness of the pixel in the image. For example, the grayscale value range of an 8-bit grayscale image is 0-255, where 0 represents the lowest brightness black and 255 represents the highest brightness white. For example, the grayscale value of a certain pixel is 128, which means that the color of this pixel is medium-bright gray. To convert a color image to a grayscale image, the weighted average method is usually used, that is, the RGB value of each pixel in the color image is weighted averaged according to a certain weight, and then the corresponding grayscale value is obtained.

[0101] Exemplarily, the red channel weight can be 0.299, the green channel weight can be 0.587, and the blue channel weight can be 0.114. Then for each pixel in the image, the grayscale value can be 0.299×R+0.587×G+0.114×B, where R is the value of the red channel of the pixel, G is the value of the green channel of the pixel, and B is the value of the blue channel of the pixel. Assuming that the RGB value of pixel A in the first image is (100, 200, 50), the RGB value of pixel B is (255, 128, 0), and the RGB value of pixel C is (0, 0, 255), the electronic device can calculate that the grayscale value of pixel A is 0.299×100+0.587×200+0.114×50=153, the grayscale value of pixel B is 0.299×255+0.587×128+0.114×0=151.381, which is approximately 151, and the grayscale value of pixel C is 0.299×0+0.587×0+0.114×255=29.07, which is approximately 29.

[0102] In the embodiment of the present application, after the electronic device converts the image area where the circumscribed matrix in the first image is located into a grayscale image, the electronic device can perform a convolution operation on the grayscale value of each pixel in the circumscribed matrix and a filter template of a discrete Laplacian operator, where the filter template is the following matrix:

[0103]

[0104] Among them, after the convolution operation is performed on each pixel, a convolution value will be obtained, and the convolution value represents the change of the grayscale value around the pixel. If the convolution value is large, it means that the grayscale value around the pixel changes greatly, and the pixel may be the edge or outline of an object or person in the first image.

[0105] For example, it is assumed that after the image area where the circumscribed matrix is ​​located is converted into a grayscale image, the grayscale value of the pixel D in the circumscribed matrix is ​​70, and the pixel D and its surrounding eight pixels form a 3×3 (i.e., 3 rows and 3 columns) pixel area. The grayscale value matrix of the pixel area is as follows:

[0106]

[0107] The electronic device can then perform a convolution operation on the grayscale value matrix and the filter template of the discrete Laplace operator to obtain 50×0+60×1+50×0+60×1+70×(-4)+60×1+50×0+60×1+50×0=-40, and the convolution value of the pixel D after the convolution operation is -40.

[0108] Step 302e: The electronic device calculates the variance of at least one convolution value to obtain a second value.

[0109] For example, assuming that the electronic device calculates 5 convolution values, namely 10, 15, 20, 25, and 30, the electronic device can calculate the mean of the 5 convolution values, which is (10+15+20+25+30)÷5=20, and then calculate the square of the difference between each convolution value and the mean. The square of the difference between the convolution value 10 and the mean is (10-20). 2 =(-10) 2 =100, the square of the difference between convolution 15 and the mean is (15-20) 2 =(-5) 2 =25, the square of the difference between convolution 20 and the mean is (20-20) 2 =(0) 2 =0, the square of the difference between the convolution 25 and the mean is (25-20) 2 =(5) 2 =25, the square of the difference between convolution 30 and the mean is (30-20) 2 =(10) 2 =100, and finally calculate the mean of the squares of these differences, that is, the variance, which is (100+25+0+25+100)÷5=50.

[0110] In this way, the electronic device can calculate the variance of the convolution value of all pixel values ​​in the image area where the human eye is located in the first image and the filter template, so as to judge the clarity of the image area where the human eye is located according to the variance.

[0111] Optionally, in an embodiment of the present application, when the first information includes the first value and the second value, the electronic device may perform steps 302c to 302e, as well as steps 302a and 302b. It should be noted that the embodiment of the present application does not limit the execution order between steps 302c to 302e and steps 302a and 302b. For example, steps 302c to 302e may be performed before steps 302a and 302b, or steps 302c to 302e may be performed after steps 302a and 302b.

[0112] Step 202: When the first information satisfies the first condition, the electronic device uses the first model to process the first image to obtain the second image, and uses the second model to process the first image to obtain the third image.

[0113] In an embodiment of the present application, the image enhancement capability of the first model is higher than that of the second model, and the image enhancement capability is used to characterize the extent to which the parameters of the image are adjusted when the model processes the image.

[0114] In the embodiment of the present application, the first model and the second model are face enhancement models, for example, artificial intelligence (AI) models constructed using a generative adversarial network (GAN) architecture. The specific training process of the AI ​​model is:

[0115] (1) A high-resolution face dataset is obtained in advance, and images in the face dataset are degraded to obtain low-resolution images.

[0116] (2) The high-definition image and the corresponding low-definition image are combined into a data pair and input into the constructed AI model.

[0117] (3) Through multiple iterations of training, the AI ​​model continuously improves the low-definition image until the difference between its output and the high-definition image meets the requirements.

[0118] In the embodiment of the present application, the parameters of the above-mentioned image include but are not limited to at least one of the following: contrast, sharpness factor, color saturation, hue, brightness, etc.

[0119] In the embodiment of the present application, the above-mentioned first model has a strong image enhancement capability, and the amplitude of adjustment of the image parameters is large when processing the image, which can well enhance the image details of the first image and improve the clarity.

[0120] In the embodiment of the present application, the image enhancement capability of the second model is relatively weak, and the amplitude of adjustment of the image parameters when processing the image is relatively small, that is, the enhanced image details are less than the image details enhanced by the first model, and the improved clarity is less than the improved clarity of the first image, but the image processed by the second model is closer to the original image than the image processed by the first model. In this way, even if the human eyes in the first image are closed and / or the clarity of the image area where the human eyes are located is relatively low, the problem of false human eyes or abnormal human eyes can be avoided, and an enhanced image with no abnormal enhancement effect can be obtained.

[0121] Optionally, in the embodiment of the present application, the first information satisfies the first condition including at least one of the following:

[0122] The first value is less than or equal to a first threshold;

[0123] The second value is less than or equal to the second threshold.

[0124] In the embodiment of the present application, the electronic device can determine that the human eye is in an open state when the first value is greater than the first threshold, and determine that the human eye is in a closed state when the first value is less than or equal to the first threshold.

[0125] In the embodiment of the present application, the first value is usually in the range of [0, 0.4]. When the eye is closed, the first value is 0. When the eye is open, the first value fluctuates around a certain value, such as about 0.35. The first value is not sensitive to the posture of the person and the head, and the difference in the first value between different individuals when the eye is open is small. In this way, a reasonable first threshold can be set to determine the open or closed state of the eyes. For example, the first threshold can be 0.25, or it can also be 0.3.

[0126] In an embodiment of the present application, the electronic device can determine that the image area where the human eye is located in the first image has a higher clarity when the second value is greater than the second threshold, and determine that the image area where the human eye is located in the first image has a lower clarity when the second value is less than or equal to the second threshold.

[0127] In an embodiment of the present application, to determine the first threshold and the second threshold, it is necessary to input the image of the portrait data set into the first model, screen out the output results of the first model with abnormal human eyes, calculate the first value and the second value corresponding to the input image, take the maximum value of the first value as the first threshold, and take the maximum value of the second value as the second threshold.

[0128] In the embodiment of the present application, the electronic device can determine whether the first value is greater than the first threshold value, and determine whether the second value is greater than the second threshold value, which is divided into four cases, namely:

[0129] (1) When the first value is greater than the first threshold value and the second value is greater than the second threshold value, the electronic device can determine that the human eye in the first image is in an open state and the image area where the human eye is located has a high clarity. It is considered that using a model with a strong image enhancement capability to enhance the first image will not cause problems of false human eyes or abnormal human eyes. The electronic device can directly use the first model to process the first image and use the processed image as the final output result.

[0130] (2) When the first value is greater than the first threshold and the second value is less than or equal to the second threshold, the electronic device can determine that the human eye in the first image is in an open state, but the image area where the human eye is located has low clarity. It is believed that using a model with stronger image enhancement capability to enhance the first image may result in problems with false human eyes or abnormal human eyes. The electronic device can use the first model to process the first image to obtain a second image, and use the second model to process the first image to obtain a third image, and execute subsequent steps.

[0131] (3) When the first value is less than or equal to the first threshold value, and the second value is greater than the second threshold value, the electronic device can determine that the human eye in the first image is in a closed state, and the image area where the human eye is located has a high clarity. It is considered that using a model with a strong image enhancement capability to enhance the first image may result in problems of false human eyes or abnormal human eyes. The electronic device can use the first model to process the first image to obtain a second image, and use the second model to process the first image to obtain a third image, and execute subsequent steps.

[0132] (4) When the first value is less than or equal to the first threshold value, and the second value is less than or equal to the second threshold value, the electronic device can determine that the human eye in the first image is in a closed state and the image area where the human eye is located has a low clarity. It is considered that using a model with a strong image enhancement capability to enhance the first image may result in problems of false human eyes or abnormal human eyes. The electronic device can use the first model to process the first image to obtain the second image, and use the second model to process the first image to obtain the third image, and execute subsequent steps.

[0133] Step 203: The electronic device performs image fusion processing based on the second image and the third image to obtain a fourth image.

[0134] Optionally, in the embodiment of the present application, combined with Figure 1 ,like Fig.11 As shown, the above step 203 can be implemented through the following steps 203a and 203b.

[0135] Step 203a, the electronic device multiplies the pixel value of each pixel in the second image with the mark value of the corresponding pixel in the first blank image to obtain the fifth image, and multiplies the pixel value of each pixel in the third image with the mark value of the corresponding pixel in the second blank image to obtain the sixth image.

[0136] In an embodiment of the present application, the marking values ​​of the pixel points in the first area in the above-mentioned first blank image are different from those in other areas, the marking values ​​of the pixel points in the first area in the above-mentioned second blank image are the same as those in other areas in the first blank image, and the marking values ​​of the pixel points in other areas in the above-mentioned second blank image are the same as those in the first area in the first blank image; the first area is the image area corresponding to the above-mentioned human eye part, and the other areas are the image areas other than the above-mentioned human eye part.

[0137] In the embodiment of the present application, the value range of the above-mentioned mark value is [0,1], such as 0.1, 0.2, etc.

[0138] In the embodiment of the present application, in the first blank image, each pixel has a label value, which is used as a coefficient of the pixel value of the corresponding pixel in the second image. In the second blank image, each pixel has a label value, which is used as a coefficient of the pixel value of the corresponding pixel in the third image.

[0139] In the embodiment of the present application, since the second image is obtained by processing using the first model, the image details are relatively fine and the clarity is relatively high. The other areas in the second image can be marked with a higher marking value, such as 0.9, 1, etc., and the first area can be marked with a lower marking value, such as 0.1, 0, etc., through the marking value of the first blank image. Therefore, when performing image fusion, the proportion of other areas in the second image occupied can be higher, and the proportion of the first area occupied can be lower.

[0140] Similarly, since the third image is obtained by processing using the second model, there will be no problems with false human eyes or abnormal human eyes. The first area in the third image can be marked with a higher marking value, such as 0.9, 1, etc., through the marking value of the second blank image, and other areas in the third image can be marked with higher marking values, such as 0.1, 0, etc., so that when performing image fusion, the first area in the third image can be occupied at a higher proportion, and the proportions occupied by other areas can be lower.

[0141] Exemplarily, assuming that the second image is a 3×3 pixel region, the pixel value matrix of the pixel region is as follows:

[0142]

[0143] Assume that the label value of each pixel value in the first blank image forms a 3×3 label value matrix as follows:

[0144]

[0145] Then the electronic device can multiply each pixel value in the pixel value matrix with the tag value at the corresponding position in the tag value matrix. For example, for the pixel value (10, 20, 30) at the upper left in the pixel value matrix, its pixel value can be calculated as (10×0.8, 20×0.8, 30×0.8)=(8, 16, 24). Similarly, other pixel values ​​can be calculated to obtain a pixel value matrix multiplied by the tag value at the corresponding position in the tag value matrix. That is, the fifth image can be represented by the pixel value matrix:

[0146]

[0147] In the embodiment of the present application, the manner in which the marking values ​​of corresponding pixels in the third image and the second blank image are multiplied to obtain the fifth image is similar to that for the second image and the first blank image, and will not be repeated here.

[0148] Step 203b: The electronic device adds the pixel value of each pixel in the fifth image to the pixel value of the corresponding pixel in the sixth image to obtain a fourth image.

[0149] Exemplarily, it is assumed that both the fifth image and the sixth image are 3×3 pixel regions, wherein the pixel value matrix of the fifth image is as follows:

[0150]

[0151] The pixel value matrix of the sixth image is as follows:

[0152]

[0153] Then the electronic device can add the two pixel value matrices. For example, for the pixel value (8, 16, 24) at the upper left in the pixel value matrix of the fifth image and the pixel value (15, 25, 35) at the upper left in the pixel value matrix of the sixth image, the addition can obtain (8+15, 16+25, 24+35)=(23, 41, 59). Similarly, other pixel values ​​can be calculated to obtain a pixel value matrix, that is, the fourth image can be represented by the pixel value matrix:

[0154]

[0155] In this way, the electronic device can merge the other areas in the second image with the first area in the third image into one image, thereby improving the clarity of the image area other than the human eye area while avoiding the influence of false human eye or abnormal human eye problems.

[0156] Optionally, in the embodiment of the present application, combined with Fig.11,like Fig.12 As shown, before the above step 203a, the image processing method provided in the embodiment of the present application also includes the following steps 401 to 403.

[0157] Step 401: The electronic device creates a first blank image and a second blank image.

[0158] In the embodiment of the present application, the first blank image and the second blank image are blank images having the same resolution as the first image.

[0159] Step 402: The electronic device marks the pixel points in the first area of ​​the first blank image as m, and marks the pixel points in other areas of the first blank image as 1-m.

[0160] In the embodiment of the present application, the above m∈[0,1].

[0161] Optionally, in the embodiment of the present application, the value of m can be 0, that is, when the second image and the third image are fused, the first area in the second image is completely removed, and the other areas in the third image are completely removed, that is, the other areas in the second image and the first area in the third image are cropped out and spliced ​​into the fourth image. Alternatively, the value of m can also be non-0, that is, the ratio of the second image and the third image when they are fused can be adjusted by changing the value of m.

[0162] For example, in combination Figure 4 ,like Fig.13 As shown, the selfie image 14 includes a first area 24 and other areas 25, then the electronic device can create a first blank image, and mark the pixels of the first area 24 in the first blank image as 0, and mark the pixels of the other areas 25 in the first blank image as 1.

[0163] Step 403: The electronic device marks the pixel points in the first area of ​​the second blank image as 1-m, and marks the pixel points in other areas of the second blank image as m.

[0164] It can be understood that after executing the above steps 402 and 403, the electronic device can cover the image area corresponding to the human eye part in the third image (weak enhancement effect) with the image area corresponding to the human eye part in the second image (strong enhancement effect) by executing the above steps 203a and 203b, and in the case of m=0, it can be regarded as the image area corresponding to the human eye part in the second image is replaced by the image area corresponding to the human eye part in the third image; and, cover other areas in the second image (strong enhancement effect) with other areas in the third image (weak enhancement effect), and in the case of m=0, it can be regarded as the other areas in the third image are replaced by other areas in the second image. The enhancement effect of the image area corresponding to the human eye part in the fourth image obtained in this way is weak, and the enhancement effect of other areas is strong, so that the problem of false human eyes or abnormal human eyes in the image area corresponding to the human eye part in the fourth image is avoided while ensuring the high clarity of other areas in the fourth image.

[0165] It should be noted that the embodiment of the present application does not limit the execution order between the above step 402 and the above step 403. For example, the above step 402 can be executed before the above step 403, or the above step 402 can be executed after the above step 403, or the above step 402 and step 403 can also be executed at the same time.

[0166] In this way, the electronic device can create a first blank image and a second blank image and add corresponding label values ​​to the pixels in the images, so that the first blank image and the second blank image can be used as coefficients of the second image and the third image to control the values ​​of the pixels in each area of ​​the second image and the third image, thereby achieving the fusion of the second image and the third image.

[0167] The embodiment of the present application provides a picture processing method and device thereof. Since the state of a human eye part, that is, whether the human eye part is in a closed state, can be determined by a first value, or the clarity of the image corresponding to the human eye part can be determined by a second value, when the human eye part is in a closed state or the clarity of the image corresponding to the human eye part is low, the electronic device generates a second image with a stronger enhancement effect and a third image with a weaker enhancement effect, and performs image fusion processing based on the second image and the third image, that is, other areas outside the human eye part in the second image are fused with the area of ​​the human eye part in the third image into one image, thereby avoiding the problem of false human eyes or abnormal human eyes caused by strong enhancement effect on human eye parts in a closed state or low-definition human eye parts, thereby improving the image quality processed by the electronic device.

[0168] Each of the above-mentioned method embodiments, or various possible implementation methods in each method embodiment can be executed separately, or any two or more of them can be executed in combination with each other. The specific implementation method can be determined according to actual usage requirements, and the embodiments of the present application do not limit this.

[0169] The image processing method provided in the embodiment of the present application can be executed by an image processing device. In the embodiment of the present application, an image processing device executing the image processing method is taken as an example to illustrate the image processing device provided in the embodiment of the present application.

[0170] Fig.14 A possible structural diagram of an image processing device involved in some embodiments of the present application is shown. Fig.14 As shown, the image processing device 70 may include: an acquisition module 71 and a processing module 72 .

[0171] The acquisition module 71 is used to acquire first information of the human eye part in the first image, and the first information includes at least one of the following: a first value and a second value, the first value represents the state of the human eye part, and the second value represents the image clarity corresponding to the human eye part.

[0172] The above-mentioned processing module 72 is used to use the first model to process the first image to obtain the second image, and use the second model to process the first image to obtain the third image when the first information obtained by the acquisition module 71 meets the first condition, and the image enhancement capability of the first model is higher than that of the second model; and perform image fusion processing based on the second image and the third image to obtain the fourth image.

[0173] In one possible implementation, combining Fig.14 ,like Fig.15 As shown, the image processing device provided in the embodiment of the present application further includes: a determination module 73. The determination module 73 is used to determine the coordinate information of the feature points of the human eye part from the first image before obtaining the first information of the human eye part in the first image. The acquisition module 71 is specifically used to calculate the first information based on the coordinate information determined by the determination module 73.

[0174] In a possible implementation, the first information includes a first value. The acquisition module 71 is specifically used to: calculate at least one first distance and a second distance based on at least four feature points of the human eye determined by the determination module 73, the first distance being the distance between the two feature points in the vertical direction, the second distance being the distance between the two feature points in the horizontal direction, and; calculate a first value based on at least one first distance and the second distance, the first value being the length ratio of the human eye in the vertical direction and the horizontal direction.

[0175] In a possible implementation, the first information includes the second value. The acquisition module 71 is specifically used to: establish a bounding rectangle based on the coordinate information determined by the determination module 73, the bounding rectangle including the image area corresponding to the human eye part; and convolve the gray value of each pixel point in the bounding rectangle with the filter template to obtain at least one convolution value; and calculate the variance of at least one convolution value to obtain the second value.

[0176] In a possible implementation, the first information satisfies the first condition including at least one of the following:

[0177] The first value is less than or equal to a first threshold;

[0178] The second value is less than or equal to the second threshold.

[0179] In one possible implementation, the processing module 72 is specifically used to: multiply the pixel value of each pixel in the second image by the mark value of the corresponding pixel in the first blank image to obtain the fifth image, and multiply the pixel value of each pixel in the third image by the mark value of the corresponding pixel in the second blank image to obtain the sixth image; and add the pixel value of each pixel in the fifth image to the pixel value of the corresponding pixel in the sixth image to obtain the fourth image; wherein, the mark values ​​of the pixels in the first area and other areas in the first blank image are different, the mark values ​​of the pixels in the first area in the second blank image and other areas in the first blank image are the same, and the mark values ​​of the pixels in other areas in the second blank image and the first area in the first blank image are the same; the first area is the image area corresponding to the human eye part, and the other areas are the image areas other than the human eye part.

[0180] In one possible implementation, the processing module 72 is further used to: create a first blank image and a second blank image before performing image fusion processing based on the second image and the third image to obtain a fourth image; and mark the pixel points in the first area of ​​the first blank image as m, and mark the pixel points in other areas of the first blank image as 1-m; and mark the pixel points in the first area of ​​the second blank image as 1-m, and mark the pixel points in other areas of the second blank image as m; wherein m∈[0,1].

[0181] In an embodiment of the present application, an image processing device is provided. Since the state of a human eye part, that is, whether the human eye part is in a closed state, can be determined by a first value, or the clarity of the image corresponding to the human eye part can be determined by a second value, when the human eye part is in a closed state or the clarity of the image corresponding to the human eye part is low, the image processing device generates a second image with a stronger enhancement effect and a third image with a weaker enhancement effect, and performs image fusion processing based on the second image and the third image, that is, other areas outside the human eye part in the second image are fused with the area of ​​the human eye part in the third image into one image, thereby avoiding the problem of false human eyes or abnormal human eyes caused by strong enhancement effect on human eye parts in a closed state or low-definition human eye parts, thereby improving the quality of images processed by the electronic device.

[0182] The image processing device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0183] The image processing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0184] The image processing device provided in the embodiment of the present application can implement each process implemented in the above method embodiment, and will not be described again here to avoid repetition.

[0185] Alternatively, if Fig.16As shown, an embodiment of the present application further provides an electronic device 1000, including a processor 1001 and a memory 1002, wherein the memory 1002 stores programs or instructions that can be executed on the processor 1001, and when the program or instructions are executed by the processor 1001, the various steps of the above-mentioned image processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0186] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0187] Fig.17 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0188] The electronic device 100 includes but is not limited to components such as a radio frequency unit 101, a network module 102, an audio output unit 103, an input unit 104, a sensor 105, a display unit 106, a user input unit 107, an interface unit 108, a memory 109, and a processor 110.

[0189] Those skilled in the art will appreciate that the electronic device 100 may also include a power source (such as a battery) for supplying power to various components, and the power source may be logically connected to the processor 110 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.17 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0190] Among them, the above-mentioned processor 110 is used to obtain first information about the human eye part in the first image, and the first information includes at least one of the following: a first value and a second value, the first value represents the state of the human eye part, and the second value represents the image clarity corresponding to the human eye part.

[0191] The above-mentioned processor 110 is also used to, when the first information satisfies the first condition, use the first model to process the first image to obtain the second image, and use the second model to process the first image to obtain the third image, and the image enhancement capability of the first model is higher than the image enhancement capability of the second model; and, perform image fusion processing based on the second image and the third image to obtain the fourth image.

[0192] In a possible implementation, the processor 110 is further configured to determine coordinate information of feature points of the eye region from the first image before acquiring the first information of the eye region in the first image. The processor 110 is specifically configured to calculate the first information based on the coordinate information.

[0193] In a possible implementation, the first information includes a first value. The processor 110 is specifically configured to: calculate at least one first distance and a second distance based on at least four feature points of the human eye, the first distance being the distance between the two feature points in the vertical direction, and the second distance being the distance between the two feature points in the horizontal direction; and, based on the at least one first distance and the second distance, calculate a first value, the first value being the ratio of the length of the human eye in the vertical direction to the length of the human eye in the horizontal direction.

[0194] In a possible implementation, the first information includes the second value. The processor 110 is specifically configured to: establish a bounding rectangle based on the coordinate information, the bounding rectangle including an image area corresponding to a human eye part; and convolve the grayscale value of each pixel point in the bounding rectangle with the filter template to obtain at least one convolution value; and calculate the variance of the at least one convolution value to obtain the second value.

[0195] In a possible implementation, the first information satisfies the first condition including at least one of the following:

[0196] The first value is less than or equal to a first threshold;

[0197] The second value is less than or equal to the second threshold.

[0198] In one possible implementation, the processor 110 is specifically used to: multiply the pixel value of each pixel in the second image by the mark value of the corresponding pixel in the first blank image to obtain the fifth image, and multiply the pixel value of each pixel in the third image by the mark value of the corresponding pixel in the second blank image to obtain the sixth image; and add the pixel value of each pixel in the fifth image to the pixel value of the corresponding pixel in the sixth image to obtain the fourth image; wherein, the mark value of the pixel points in the first area of ​​the first blank image and other areas is different, the mark value of the pixel points in the first area of ​​the second blank image and other areas of the first blank image are the same, and the mark value of the pixel points in the other areas of the second blank image and the first area of ​​the first blank image are the same; the first area is the image area corresponding to the human eye part, and the other areas are the image areas other than the human eye part.

[0199] In one possible implementation, the processor 110 is further used to: create a first blank image and a second blank image before performing image fusion processing based on the second image and the third image to obtain a fourth image; and mark the pixel points in the first area of ​​the first blank image as m, and mark the pixel points in other areas of the first blank image as 1-m; and mark the pixel points in the first area of ​​the second blank image as 1-m, and mark the pixel points in other areas of the second blank image as m; wherein m∈[0,1].

[0200] An embodiment of the present application provides an electronic device. Since the state of a human eye part, that is, whether the human eye part is in a closed state, can be determined by a first value, or the clarity of an image corresponding to the human eye part can be determined by a second value, when the human eye part is in a closed state or the clarity of the image corresponding to the human eye part is low, the electronic device generates a second image with a stronger enhancement effect and a third image with a weaker enhancement effect, and performs image fusion processing based on the second image and the third image, that is, other areas outside the human eye part in the second image are fused with the area of ​​the human eye part in the third image into one image, thereby avoiding the problem of false human eyes or abnormal human eyes caused by strong enhancement effect on human eye parts in a closed state or low-definition human eye parts, thereby improving the image quality processed by the electronic device.

[0201] The electronic device provided in the embodiment of the present application can implement each process implemented in the above method embodiment, and can achieve the same technical effect. To avoid repetition, it is not repeated here. The beneficial effects of various implementations in this embodiment can be specifically referred to the beneficial effects of the corresponding implementations in the above method embodiment. To avoid repetition, it is not repeated here.

[0202] It should be understood that in the embodiment of the present application, the input unit 104 may include a graphics processor (Graphics Processing Unit, GPU) 1041 and a microphone 1042, and the graphics processor 1041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 106 may include a display panel 1061, and the display panel 1061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 107 includes a touch panel 1071 and at least one of other input devices 1072. The touch panel 1071 is also called a touch screen. The touch panel 1071 may include two parts: a touch detection device and a touch controller. Other input devices 1072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0203] The memory 109 can be used to store software programs and various data. The memory 109 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 109 may include a volatile memory or a non-volatile memory, or the memory 109 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 109 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0204] The processor 110 may include one or more processing units; optionally, the processor 110 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 110.

[0205] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned image processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0206] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0207] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0208] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0209] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0210] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0211] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0212] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image processing method, characterized in that: include: Acquire first information of a human eye part in a first image, wherein the first information includes at least one of the following: a first value and a second value, wherein the first value represents a state of the human eye part, and the second value represents image clarity corresponding to the human eye part; When the first information satisfies a first condition, the first image is processed by a first model to obtain a second image output by the first model, and the first image is processed by a second model to obtain a third image output by the second model; An image fusion process is performed based on the second image and the third image to obtain a fourth image.

2. The method according to claim 1, characterized in that Before obtaining the first information of the human eye part in the first image, the method further includes: Determine the coordinate information of the feature points of the human eye from the first image; The step of obtaining first information about a human eye portion in the first image includes: The first information is calculated based on the coordinate information.

3. The method according to claim 2, characterized in that The first information includes the first value; The calculating the first information based on the coordinate information includes: Based on the at least four feature points of the human eye, at least one first distance and a second distance are calculated, wherein the first distance is the distance between the two feature points in the vertical direction, and the second distance is the distance between the two feature points in the horizontal direction; The first value is calculated based on the at least one first distance and the second distance, where the first value is a ratio of the length of the human eye part in the vertical direction to the length of the horizontal direction.

4. The method according to claim 1, characterized in that: The performing image fusion processing based on the second image and the third image to obtain a fourth image includes: Multiplying the pixel value of each pixel in the second image by the label value of the corresponding pixel in the first blank image to obtain a fifth image, and multiplying the pixel value of each pixel in the third image by the label value of the corresponding pixel in the second blank image to obtain a sixth image; Adding a pixel value of each pixel in the fifth image to a pixel value of a corresponding pixel in the sixth image to obtain the fourth image; Among them, the label values ​​of the pixel points in the first area and other areas in the first blank image are different, the label values ​​of the pixel points in the first area in the second blank image and other areas in the first blank image are the same, and the label values ​​of the pixel points in other areas in the second blank image and the first area in the first blank image are the same; the first area is the image area corresponding to the human eye part, and the other areas are the image areas other than the human eye part.

5. The method according to claim 4, characterized in that Before performing image fusion processing based on the second image and the third image to obtain a fourth image, the method further includes: creating the first blank image and the second blank image; Marking the pixel points of the first area in the first blank image as m, and marking the pixel points of the other areas in the first blank image as 1-m; Marking the pixels of the first area in the second blank image as 1-m, and marking the pixels of the other areas in the second blank image as m; Among them, m∈[0,1].

6. An image processing device, characterized in that: include: Acquisition module and processing module; The acquisition module is used to acquire first information of a human eye part in a first image, wherein the first information includes at least one of the following: a first value and a second value, wherein the first value represents a state of the human eye part, and the second value represents image clarity corresponding to the human eye part; The processing module is used to use a first model to process the first image to obtain a second image, and use a second model to process the first image to obtain a third image when the first information obtained by the acquisition module meets a first condition; and perform image fusion processing based on the second image and the third image to obtain a fourth image.

7. The device according to claim 6, characterized in that The device further comprises: a determination module; The determining module is used to determine the coordinate information of the feature points of the human eye part from the first image before acquiring the first information of the human eye part in the first image; The acquisition module is specifically used to calculate the first information based on the coordinate information determined by the determination module.

8. The device according to claim 7, characterized in that The first information includes the first value; The acquisition module is specifically used for: Based on the at least four feature points of the human eye determined by the determination module, calculating at least one first distance and a second distance, wherein the first distance is the distance between the two feature points in the vertical direction, and the second distance is the distance between the two feature points in the horizontal direction; And, based on the at least one first distance and the second distance, the first value is calculated, where the first value is the ratio of the length of the human eye part in the vertical direction and the horizontal direction.

9. The device according to claim 6, characterized in that The processing module is specifically used for: multiplying the pixel value of each pixel in the second image by the label value of the corresponding pixel in the first blank image to obtain a fifth image, and multiplying the pixel value of each pixel in the third image by the label value of the corresponding pixel in the second blank image to obtain a sixth image; and adding the pixel value of each pixel in the fifth image to the pixel value of the corresponding pixel in the sixth image to obtain the fourth image; Among them, the label values ​​of the pixel points in the first area and other areas in the first blank image are different, the label values ​​of the pixel points in the first area in the second blank image and other areas in the first blank image are the same, and the label values ​​of the pixel points in other areas in the second blank image and the first area in the first blank image are the same; the first area is the image area corresponding to the human eye part, and the other areas are the image areas other than the human eye part.

10. The device according to claim 9, characterized in that The processing module is further used for: Before performing image fusion processing based on the second image and the third image to obtain a fourth image, creating the first blank image and the second blank image; and marking the pixel points of the first area in the first blank image as m, and marking the pixel points of the other areas in the first blank image as 1-m; And, marking the pixel points of the first area in the second blank image as 1-m, and marking the pixel points of the other areas in the second blank image as m; Among them, m∈[0,1].

Images