Eye positioning methods, devices, storage media and equipment

Abstract

This invention discloses an eye localization method, apparatus, storage medium, and device, belonging to the field of biometrics. It includes: performing face detection on an image to be localized to obtain a face region image; performing skin color segmentation on the face region image; performing line detection and circle detection on the skin color segmentation result; the line segments obtained from line detection represent the edges of face occlusions, and the circles obtained from circle detection represent the coarse localization result of the eyes; based on the position and size of the face region image, the number and position of the circles obtained from circle detection, and the line segments obtained from line detection, extracting an eye region image from the face region image; and inputting the eye region image into a convolutional neural network to regress and obtain the key points of the eyes. This invention solves the problem of eye localization for faces occluded at a distance, and has good performance in eye position detection under face occlusion conditions such as masks or face shields. It can be applied to multimodal biometrics and long-distance iris recognition applications.

Description

Technical Field

[0001] This invention relates to the field of biometrics, and in particular to an eye positioning method, apparatus, storage medium, and device. Background Technology

[0002] Eye position localization is of great significance for multimodal biometrics (such as face-iris multimodal recognition) and iris recognition. Eye position can be used to control image acquisition by face-iris multimodal devices and adjust the acquisition angle, and it can also be used for long-distance iris recognition tasks.

[0003] Currently, there are two main methods for eye localization, applied in face recognition and iris recognition respectively. In face recognition, the eye localization method regresses key points on the detected face. These key points include the coordinates of major boundary points and corner points on the face, typically including the boundary coordinates of the face and the boundary coordinates of facial features (including the eyes). Facial key points are obtained based on the overall regression results of the face image. If more than 50% of the face image is occluded, it will lead to inaccurate facial key point localization. For example... Figure 1 , 2 The images show situations where people are wearing masks and face shields or other facial coverings. Figure 1 The image should be of a person wearing a mask, with less than half of the face obscured and the outline clearly visible. Figure 2 The image shows a face wearing a mask, with only the area around the eyes unobstructed, making it impossible to discern the facial contours. Figure 1 , 2 The key points located in the two images will have discrepancies, and Figure 2 The deviation in facial landmark localization is even more serious.

[0004] In iris recognition, the eye localization method obtains fine eye coordinates by performing a series of image processing operations such as threshold segmentation and radial transformation on the detected human eye image. This method requires high-resolution eye images, so it cannot solve the problem of human eye localization at long distances. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides an eye positioning method, device, storage medium, and equipment, which has a good effect on eye position detection when the face is obscured by masks or face shields, and can be applied to multimodal biometrics and long-distance iris recognition.

[0006] The technical solution provided by this invention is as follows:

[0007] In a first aspect, the present invention provides an eye positioning method, the method comprising:

[0008] Perform face detection on the image to be located to obtain a face region image;

[0009] Skin color segmentation is performed on the face region image;

[0010] Line detection and circle detection are performed on the skin color segmentation results;

[0011] Wherein, the line segment obtained by the line detection represents the edge of the face occlusion, and the circle obtained by the circle detection represents the coarse positioning result of the eye;

[0012] Based on the position and size of the face region image, the number and position of the circles detected by the circle detection, and the line segments detected by the line detection, an eye region image is extracted from the face region image;

[0013] The eye region image is input into a convolutional neural network, and the key points of the eye are obtained through regression.

[0014] Furthermore, the process of performing line detection and circle detection on the skin color segmentation results includes:

[0015] Hough transform is used to detect straight lines in the skin color segmentation results, and only line segments with an angle to the horizontal direction less than a set angle and a length greater than a set length are retained.

[0016] Hough transform is used to perform circle detection on the skin color segmentation results, and circles that do not meet the requirements are removed based on the position and size of the detected circles.

[0017] Furthermore, the step of removing circles that do not meet the requirements based on the position and size of the detected circles includes:

[0018] Remove circles whose center y-coordinate is greater than ymax or less than ymin;

[0019] Where ymax and ymin are the maximum and minimum values ​​of the ordinate in the line segment obtained by the line detection, respectively;

[0020] Remove circles whose absolute value of the difference between their center ordinate and ymin is greater than a set difference threshold;

[0021] Remove circles with a radius greater than the set radius threshold.

[0022] Furthermore, the step of extracting the eye region image from the face region image based on the position and size of the face region image, the number and position of the circles obtained by the circle detection, and the line segments obtained by the line detection includes:

[0023] If two or more circles are detected on the face region image, the two circles with the largest center ordinates are selected, and the eye region images of the two eyes are obtained based on the two circles with the largest center ordinates.

[0024] If a circle is detected on the face region image, another circle is obtained based on the symmetrical characteristic of the two eyes on the face region image. The eye region images of the two eyes are then obtained by cropping the one circle and the other circle.

[0025] If no circle is detected on the face region image, the positions of the two eyes are calculated based on the nearest straight line to the face region image and the position and size of the face region image, and the eye region images of the two eyes are obtained based on the positions of the two eyes.

[0026] Furthermore, when two or more circles are detected on the face region image, the center coordinates of the two circles with the largest center ordinates are (x1, y1) and (x2, y2), and the cropping range of the eye region image of the two eyes is [x1- :x1+ ,y1- :y1+ ] and [x2- :x2+ ,y2- y2+ ];

[0027] in, The set threshold for the interception scale;

[0028] When a circle is detected on the face region image, the center coordinates of the first circle are (rx, ry), the center coordinates of the second circle are (rx', ry'), and the cropping range of the eye region image for both eyes is [rx- :rx+ ,ry- :ry+ ] and [rx'- :rx'+ , ry'- :ry'+ ];

[0029] Where rx' = 2x + w – rx, ry' = ry, w is the width of the face region image, and x is the x-coordinate of the lower left corner of the face region image;

[0030] When no circle is detected in the face region image, the position coordinates of the two eyes are (x3, y3) and (x4, y4), and the cropping range of the eye region image of the two eyes is [x3- :x3+ ,y3- y3+ ] and [x4- :x4+ ,y4- y4+ ];

[0031] Among them, x3 = x + w / 4, y3 = k(x + w / 4) + b + h / 2, x4 = x + 3*w / 4, y4 = k(x +3 w / 4) + b + h / 2; k and b are the parameters of the line y'=kx'+b that is closest to the face region image, and h is the height of the face region image.

[0032] Furthermore, the step of performing skin color segmentation on the face region image includes:

[0033] For each pixel of the face region image, binary segmentation is performed using the following formula;

[0034]

[0035] in, On the face region image Skin color segmentation results for pixels at a given location. w and h are the width and height of the face region image, respectively;

[0036] for ;

[0037] iRed iGrn and iBlu On the face region image respectively The red, green, and blue channel values ​​of the pixels at that location;

[0038] ;

[0039] ;

[0040] ;

[0041] ;

[0042] The mean filtering and guided filtering are used to denoise the binary segmentation results to obtain the skin color segmentation results.

[0043] Furthermore, the step of inputting the eye region image into a convolutional neural network to regress and obtain the key points of the eye includes:

[0044] The first feature map is obtained by filling several pixels around the eye region image and performing a first convolution operation, a first activation operation, a first absolute value operation and a first pooling operation.

[0045] For the first feature Figure 4 The second feature map is obtained by filling several pixels around the perimeter and performing a second convolution operation, a second activation operation, a second absolute value operation, and a second pooling operation.

[0046] The second feature map is subjected to a third convolution operation, a third activation operation, a third absolute value operation, and a third pooling operation to obtain a third feature map.

[0047] The key points of the eye are obtained through regression of the third feature map.

[0048] In a second aspect, the present invention provides an eye positioning device, the device comprising:

[0049] The face detection module is used to detect faces in the image to be located and obtain face region images;

[0050] A skin color segmentation module is used to segment the skin color of the face region image;

[0051] The line and circle detection module is used to perform line and circle detection on the skin color segmentation results;

[0052] Wherein, the line segment obtained by the line detection represents the edge of the face occlusion, and the circle obtained by the circle detection represents the coarse positioning result of the eye;

[0053] An eye region image acquisition module is used to extract an eye region image from the face region image based on the position and size of the face region image, the number and position of the circles detected by the circle detection, and the line segments detected by the line detection.

[0054] The key point regression module is used to input the eye region image into a convolutional neural network and regress the key points of the eye.

[0055] Furthermore, the line and circle detection module includes:

[0056] The line detection unit is used to perform line detection on the skin color segmentation results using Hough transform, and only retains line segments with an angle less than a set angle to the horizontal direction and a length greater than a set length.

[0057] The circle detection unit is used to perform circle detection on the skin color segmentation results using Hough transform, and remove circles that do not meet the requirements based on the position and size of the detected circles.

[0058] Furthermore, in the circle detection unit, the step of removing circles that do not meet the requirements based on the position and size of the detected circles includes:

[0059] Remove circles whose center y-coordinate is greater than ymax or less than ymin;

[0060] Where ymax and ymin are the maximum and minimum values ​​of the ordinate in the line segment obtained by the line detection, respectively;

[0061] Remove circles whose absolute value of the difference between their center ordinate and ymin is greater than a set difference threshold;

[0062] Remove circles with a radius greater than the set radius threshold.

[0063] Furthermore, the eye region image acquisition module includes:

[0064] The first cropping unit is used to select the two circles with the largest center coordinates if two or more circles are detected on the face region image, and crop the eye region image of the two eyes based on the two circles with the largest center coordinates.

[0065] The second cropping unit is used to obtain another circle based on the symmetrical characteristic of the two eyes on the face region image if a circle is detected on the face region image, and to crop the eye region image of the two eyes based on the one circle and the other circle.

[0066] The third cropping unit is used to calculate the positions of the two eyes based on the nearest straight line to the face region image and the position and size of the face region image if no circle is detected on the face region image, and to crop the eye region image of the two eyes based on the positions of the two eyes.

[0067] Furthermore, when two or more circles are detected on the face region image, the center coordinates of the two circles with the largest center ordinates are (x1, y1) and (x2, y2), and the cropping range of the eye region image of the two eyes is [x1- :x1+ ,y1- :y1+ ] and [x2- :x2+ ,y2- y2+ ];

[0068] in, The set threshold for the interception scale;

[0069] When a circle is detected on the face region image, the center coordinates of the first circle are (rx, ry), the center coordinates of the second circle are (rx', ry'), and the cropping range of the eye region image for both eyes is [rx- :rx+ ,ry- :ry+ ] and [rx'- :rx'+ , ry'- :ry'+ ];

[0070] Where rx' = 2x + w – rx, ry' = ry, w is the width of the face region image, and x is the x-coordinate of the lower left corner of the face region image;

[0071] When no circle is detected in the face region image, the position coordinates of the two eyes are (x3, y3) and (x4, y4), and the cropping range of the eye region image of the two eyes is [x3- :x3+ ,y3- y3+ ] and [x4- :x4+ ,y4- y4+ ];

[0072] Among them, x3 = x + w / 4, y3 = k(x + w / 4) + b + h / 2, x4 = x + 3*w / 4, y4 = k(x +3 w / 4) + b + h / 2; k and b are the parameters of the line y'=kx'+b that is closest to the face region image, and h is the height of the face region image.

[0073] Furthermore, the skin color segmentation module includes:

[0074] The binary segmentation unit is used to perform binary segmentation on each pixel of the face region image using the following formula;

[0075]

[0076] in, On the face region image Skin color segmentation results for pixels at a given location. w and h are the width and height of the face region image, respectively;

[0077] for ;

[0078] iRed iGrn and iBlu On the face region image respectively The red, green, and blue channel values ​​of the pixels at that location;

[0079] ;

[0080] ;

[0081] ;

[0082] ;

[0083] The denoising unit is used to denoise the binary segmentation result using mean filtering and guided filtering to obtain the skin color segmentation result.

[0084] Furthermore, the keypoint regression module is used for:

[0085] The first feature map is obtained by filling several pixels around the eye region image and performing a first convolution operation, a first activation operation, a first absolute value operation and a first pooling operation.

[0086] For the first feature Figure 4 The second feature map is obtained by filling several pixels around the perimeter and performing a second convolution operation, a second activation operation, a second absolute value operation, and a second pooling operation.

[0087] The second feature map is subjected to a third convolution operation, a third activation operation, a third absolute value operation, and a third pooling operation to obtain a third feature map.

[0088] The key points of the eye are obtained through regression of the third feature map.

[0089] Thirdly, the present invention provides a computer-readable storage medium for eye positioning, including a memory for storing processor-executable instructions that, when executed by the processor, implement the steps of the eye positioning method described in the first aspect.

[0090] Fourthly, the present invention provides an eye positioning device, comprising at least one processor and a memory storing computer-executable instructions, wherein the processor executes the instructions to implement the steps of the eye positioning method described in the first aspect.

[0091] The present invention has the following beneficial effects:

[0092] This invention performs skin color segmentation on a face region image and then performs line detection and circle detection on the skin color segmentation results to obtain line segments representing the edges of face occlusions and circles representing the coarse localization results of the eyes. Based on the number and position of the circles obtained from circle detection, the line segments obtained from line detection, and the position and size of the face region image, the positions of the two eyes are estimated. Based on the estimated positions of the two eyes, eye region images are cropped from the face region image. The cropped eye region images are input into a convolutional neural network to regress and obtain the key points of the eyes. This invention solves the problem of eye localization for faces occluded at a distance. It has good results in eye position detection under face occlusion conditions such as masks or face shields, and can be applied to multimodal biometrics and long-distance iris recognition applications. In criminal investigation projects, it can be used in conjunction with law enforcement personnel or apprehended individuals for iris collection authentication or multimodal identity authentication. Attached Figure Description

[0093] Figure 1 This is an illustration of a face wearing a mask.

[0094] Figure 2 A schematic diagram of a face wearing a mask;

[0095] Figure 3 This is a flowchart of the eye positioning method of the present invention;

[0096] Figure 4 This is a schematic diagram of face detection and skin color segmentation when wearing a mask;

[0097] Figure 5 This is a schematic diagram of face detection and skin color segmentation when a face mask is worn.

[0098] Figure 6 This is a schematic diagram of the line segments obtained from line detection;

[0099] Figure 7 This is a schematic diagram of the circle obtained from the circle detection.

[0100] Figure 8 This is an example diagram of the convolutional neural network used in this invention;

[0101] Figure 9 A schematic diagram of the key points of the eye obtained from regression;

[0102] Figure 10This is a schematic diagram of the eye positioning device of the present invention. Detailed Implementation

[0103] To make the technical problems, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. The components of the embodiments of this invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0104] Example 1:

[0105] This invention provides an eye positioning method, such as... Figure 3 As shown, the method includes:

[0106] S100: Perform face detection on the image to be located to obtain a face region image.

[0107] This step performs face detection on the input image to be located, determines the position and size of the face region image, and restricts eye localization to within the face region image, thus narrowing the scope for eye localization.

[0108] This invention does not limit the face detection method; for example, MTCNN, SSD, CenterFace, RetinaFace, etc., can be used. The face detection method can obtain the eye position simultaneously with the face location. This eye position can be obtained from the result of multi-supervised regression of the face detection network or from face localization methods such as SDM. While the eye position is not entirely accurate, it still has some reference value and can be used for reference. Of course, it can also be disregarded without affecting the implementation of this invention.

[0109] The coordinates of the face region image are (x, y, w, h), where (x, y) are the coordinates of the lower left corner of the face region image, and w and h are the width and height of the face region image, respectively.

[0110] S200: Perform skin color segmentation on the face region image.

[0111] Skin color segmentation is used to distinguish between skin regions and non-skin regions in a face image, and to represent the skin regions and non-skin regions with different numerical values ​​to obtain the skin color segmentation result. The skin region is the area of ​​exposed skin on the face, while the non-skin region includes the background, face occlusions (masks, face shields, etc.), etc.

[0112] S300: Performs line detection and circle detection on the skin color segmentation results.

[0113] In this context, the line segment obtained by the line detection represents the edge of the face occlusion, and the circle obtained by the circle detection represents the coarse positioning result of the eye.

[0114] The boundary line between face masks and other face coverings and the face (i.e. the edge of the face covering) is generally approximately a straight line. Therefore, the edge of the face covering can be obtained by line detection. The edge of the eye is approximately a circle. Therefore, the approximate position of the eye can be estimated by circle detection.

[0115] S400: Based on the position and size of the face region image, the number and position of the circles detected by the circle detection, and the line segments detected by the line detection, an eye region image is extracted from the face region image.

[0116] The circles detected in S300 can provide a rough estimate of the eye's position to some extent, but this rough estimate is inaccurate. Furthermore, while humans have two eyes, the number of circles detected is not necessarily two; it might not detect any circles, might only detect one circle, or it might detect two or more circles.

[0117] Therefore, this step comprehensively considers the number and position of circles obtained by circle detection, the line segments obtained by line detection, and the position and size of the face region image. It takes into account factors such as the number of circles, the position of the circles on the face region image, and the relative position of the circles and the lines to estimate the position of the person's two eyes. Based on the estimated position of the two eyes, the eye region image is extracted from the face region image.

[0118] S500: Input the eye region image into a convolutional neural network to regress and obtain the key points of the eye.

[0119] This step uses a convolutional neural network to regress the key points of the eye position. This invention does not limit the structure of the convolutional neural network.

[0120] This invention performs skin color segmentation on a face region image and then performs line detection and circle detection on the skin color segmentation results to obtain line segments representing the edges of face occlusions and circles representing the coarse localization results of the eyes. Based on the number and position of the circles obtained from circle detection, the line segments obtained from line detection, and the position and size of the face region image, the positions of the two eyes are estimated. Based on the estimated positions of the two eyes, eye region images are cropped from the face region image. The cropped eye region images are input into a convolutional neural network to regress and obtain the key points of the eyes. This invention solves the problem of eye localization for faces occluded at a distance. It has good results in eye position detection under face occlusion conditions such as masks or face shields, and can be applied to multimodal biometrics and long-distance iris recognition applications. In criminal investigation projects, it can be used in conjunction with law enforcement personnel or apprehended individuals for iris collection authentication or multimodal identity authentication.

[0121] This invention does not limit the method of skin color segmentation. In one example, S200 includes:

[0122] S210: For each pixel of the face region image, perform binary segmentation using the following formula;

[0123]

[0124] in, On the face region image Skin color segmentation results for pixels at a given location. w and h are the width and height of the face region image, respectively;

[0125] for ;

[0126] iRed iGrn and iBlu On the face region image respectively The red, green, and blue channel values ​​of the pixel at that location. and They are respectively iRed location iGrn and iBlu The maximum and minimum values, that is:

[0127] ;

[0128] .

[0129] On the face region image The difference between the maximum and minimum values ​​of the location, i.e.:

[0130] .

[0131] On the face region image The absolute difference between the green channel value and the blue channel value of the location, i.e.:

[0132] .

[0133] S220: Use mean filtering and guided filtering to denoise the binary segmentation result to obtain the skin color segmentation result.

[0134] The binary segmentation result in S210 contains a lot of noise, which has a significant impact on eye localization. In order to obtain better results, this step uses mean filtering and guided filtering to denoise the binary segmentation result and obtain the skin color segmentation result.

[0135] like Figure 4 , 5 As shown, Figure 4 A and 5A are images of the objects to be located, showing the wearer wearing a mask and face shield, respectively. Figure 4 B and 5B are respectively from Figure 4 Face region images obtained on A and 5A, Figure 4 C and 5C are respectively for Figure 4 The result obtained by binary segmentation of B and 5B Figure 4 D and 5D are respectively for Figure 4 Skin color segmentation results obtained by denoising C and 5C.

[0136] For example, the present invention performs line detection and circle detection on skin color segmentation results using the following method:

[0137] S310: Use Hough transform to perform line detection on skin color segmentation results, and retain only line segments with an angle less than the set angle and a length greater than the set length with respect to the horizontal direction.

[0138] The purpose of this step is to detect the edges of masks or face shields. Specifically, this can be achieved by using a Hough transform to detect straight lines in the skin color segmentation results, retaining only line segments with an angle less than a set angle (e.g., 15 degrees) to the horizontal direction, i.e., retaining only horizontally adjacent line segments and removing vertical line segments. Furthermore, the length of each line segment is calculated, retaining only those with a length greater than a set length (e.g., w / 2).

[0139] The final line segment is as follows Figure 6 As shown, where, Figure 6 A represents the line segment obtained by straight-line detection while wearing a mask. Figure 6B is the line segment obtained by straight line detection while wearing a mask.

[0140] S320: Perform circle detection on the skin color segmentation result using Hough transform, and remove circles that do not meet the requirements based on the position and size of the detected circles.

[0141] To obtain the eye region image later, the position of the eye needs to be roughly located on the skin color segmentation result, and circle detection is performed using Hough transform.

[0142] The circle detection results are as follows Figure 7 As shown, where, Figure 7 A represents the circular detection result when wearing a mask. Figure 7 B represents the circle detection result when the face mask is worn. The circle detection result may contain circles that do not meet the requirements, i.e., noise. Noise is as follows: Figure 7 As shown in B, number 10, noise can affect the judgment of the results and needs to be removed.

[0143] This invention performs relative position analysis based on the position and size of circles to remove circles that do not meet the requirements (noise). Specific implementation methods include:

[0144] 1. Remove circles whose center ordinate is greater than ymax or less than ymin.

[0145] Wherein, ymax and ymin are the maximum and minimum values ​​of the ordinate of the line segment obtained by the line detection, respectively.

[0146] This step calculates the minimum and maximum ordinates ymin and ymax among all detected line segments. If the ordinate y of the center r(x, y) of the detected circle is greater than ymax or less than ymin, the circle is judged as an abnormal detection result and needs to be removed.

[0147] 2. Remove circles whose absolute value of the difference between the center ordinate and ymin is greater than the set difference threshold.

[0148] In this step, |y – ymin| is calculated. If |y – ymin| > the difference threshold, the difference threshold can be, for example, w / 4. That is, if the absolute value of the difference between the ordinate of the circle's center and ymin is greater than 1 / 4 of the face width, then the circle is judged as an abnormal detection result and needs to be removed.

[0149] 3. Remove circles with a radius greater than a set radius threshold, such as 0.1w.

[0150] Other circles that do not meet conditions 1-3 above are judged as qualified circle inspection results and retained.

[0151] As an improvement to an embodiment of the present invention, the aforementioned S400 includes:

[0152] S410: If two or more circles are detected on the face region image, the two circles with the largest center coordinates are selected and considered as the approximate positions of the left and right eyes. Based on the two circles with the largest center coordinates (i.e., the approximate positions of the left and right eyes), the eye region images of the two eyes are obtained.

[0153] For example, the center coordinates of the two circles with the largest ordinates are (x1, y1) and (x2, y2), and the cropping range of the eye region image of the two eyes is [x1- :x1+ ,y1- :y1+ ] and [x2- :x2+ ,y2- y2+ ].

[0154] in, The threshold value for the set cutoff scale can be w / 8.

[0155] S420: If a circle is detected on the face region image, another circle is obtained based on the symmetrical characteristic of the two eyes on the face region image, and the eye region image of the two eyes is obtained by cropping the one circle and the other circle.

[0156] In this step, when a circle is detected on the face region image, this circle is considered to be the approximate location of an eye. The two eyes are approximately symmetrical on the face region image, so the location of the other eye, i.e., the other circle, can be obtained based on the symmetry.

[0157] For example, if the center of a detected circle is (rx, ry), and rx – (x + w / 2) < 0, then the detected circle represents the left eye position. The coordinates of the right eye's center (rx', ry') can be calculated using the following formula:

[0158] rx' = 2x + w – rx;

[0159] ry' = ry;

[0160] This gives us the approximate locations of the left and right eyes. Based on these locations, the cropping range for the eye region images of both eyes is [rx- :rx+ ,ry- :ry+ ] and [rx'- :rx'+ , ry'- :ry'+ ].

[0161] S430: If no circle is detected on the face region image, calculate the positions of the two eyes based on the nearest straight line to the face region image and the position and size of the face region image, and obtain the eye region images of the two eyes based on the positions of the two eyes.

[0162] In this step, if no circle is detected, the following judgment is made based on the result of the line detection:

[0163] Define {L i Let {i = 1, 2, …, n} be the n detected line segments. Calculate the distance d(p, L) from point p(x+w / 2, y+h / 2) to the lines containing all the line segments. i (i=1, 2,…, n), select the line segment with the smallest distance to determine the edge of the mask or face shield, and denote the equation of the line containing it as y'=kx'+b. Then estimate the positions of the two eyes (x3, y3) and (x4, y4), where x3 = x + w / 4, y3 = k(x + w / 4) + b + h / 2, x4 = x + 3*w / 4, y4 = k(x + 3*w / 4). w / 4) +b + h / 2.

[0164] Then, based on the estimated positions of the two eyes, press [x3- :x3+ ,y3- y3+ ] and [x4- :x4+ ,y4- y4+ The image of the eye region of both eyes was obtained by using the method described above.

[0165] After capturing the eye region image, the scale of the eye region image will vary greatly due to the different scales of the human face. Therefore, it is necessary to perform normalization and other operations on the eye region image to scale it to a uniform size. In order to facilitate the localization of details in the eye position, the eye region image is uniformly normalized to a width and height of 40 pixels.

[0166] This invention uses Convolutional Neural Networks (CNNs) to regress key points of the eyes. Since the left and right eyes are essentially symmetrical, the localization of the left and right eyes can be solved through data augmentation. A deep convolutional neural network is used to complete the key point regression task for both eyes. This invention does not limit the structure of the convolutional neural network; one example is shown below. Figure 8 As shown, the specific operation process includes:

[0167] 1. Fill the area around the eye region image with several pixels, and perform a first convolution operation, a first activation operation, a first absolute value operation, and a first pooling operation to obtain a first feature map, i.e. Figure 8 Features shown Figure 4 .

[0168] For example, taking a 3-channel eye region image with a width and height of 40 as input to a convolutional neural network, firstly, the four outermost two pixels of the eye region image are padded with 0s. Then, a 5x5 convolution kernel with a stride of 1 is used for the first convolution operation, resulting in a 16-channel feature map of size 40x40. Next, the Tanh activation function is used for the first activation operation. The Tanh function has the following form:

[0169]

[0170] The activated neurons undergo a first absolute value operation, which serves as the input for the next layer. Next, the neurons after absolute value extraction are subjected to a first pooling operation. This first pooling operation is a non-overlapping max pooling operation, with a pooling kernel size of 2x2 and a stride of 2, resulting in a 16-channel feature set of size 20x20. Figure 4 This is the first feature map.

[0171] 2. Regarding the first feature Figure 4 The second feature map is obtained by padding several pixels around the perimeter and then performing a second convolution operation, a second activation operation, a second absolute value operation, and a second pooling operation. Figure 8 Features shown Figure 8 .

[0172] For example, the first feature of a 16-channel array with a size of 20*20. Figure 4 Each pixel in the circle is padded with 0s, then a second convolution operation is performed using a 3x3 kernel with a stride of 1, resulting in a 48-channel feature map of size 20x20. A second activation operation is then performed using the Tanh activation function, followed by a second absolute value operation on the resulting feature map. Finally, a second pooling operation of 3x3 with a stride of 1 is performed on the resulting feature map, resulting in a 48-channel feature map of size 10x10. Figure 8 This is the second feature map.

[0173] 3. Perform a third convolution operation, a third activation operation, a third absolute value operation, and a third pooling operation on the second feature map to obtain the third feature map, i.e. Figure 8 Feature diagram 12 is shown.

[0174] For example, a third convolution operation is performed on a 48-channel second feature map of size 10*10 using a 3*3 kernel with a stride of 1, resulting in a 64-channel feature map of size 8*8. Then, the Tanh activation function is used for a third activation operation, and the resulting feature map is subjected to a third absolute value operation. Next, a 3*3 pooling operation is performed on the resulting feature map with a stride of 2, resulting in a 64-channel feature map 12 of size 3*3, which is the third feature map.

[0175] 4. Obtain the key points of the eye through regression of the third feature map.

[0176] The aforementioned processes 1-3 constitute the feature map extraction process. A 40*40 pixel, 3-channel color eye region image is processed by a CNN to obtain a 3*3 pixel, 64-channel third feature map. Finally, this third feature map is used for regression analysis on 9 keypoints, such as... Figure 9 As shown, where, Figure 9 A and 9B respectively present the key points of the left and right eyes.

[0177] Example 2:

[0178] This invention provides an eye positioning device, such as... Figure 10 As shown, the device includes:

[0179] Face detection module 1 is used to perform face detection on the image to be located and obtain the face region image.

[0180] Skin color segmentation module 2 is used to segment the skin color of the face region image.

[0181] Line and circle detection module 3 is used to perform line and circle detection on the skin color segmentation results;

[0182] In this context, the line segment obtained by the line detection represents the edge of the face occlusion, and the circle obtained by the circle detection represents the coarse positioning result of the eye.

[0183] The eye region image acquisition module 4 is used to extract an eye region image from the face region image based on the position and size of the face region image, the number and position of the circles detected by the circle detection, and the line segments detected by the line detection.

[0184] Key point regression module 5 is used to input the eye region image into a convolutional neural network to regress the key points of the eye.

[0185] This invention solves the problem of eye location when the face is obscured at a distance. It has a good effect on eye position detection when the face is obscured by masks or face shields. It can be applied to multimodal biometrics and long-distance iris recognition. In criminal investigation and other projects, it can be used to assist on-duty personnel or arrested persons in iris collection and authentication or multimodal identity authentication.

[0186] The aforementioned line and circle detection modules include:

[0187] The line detection unit is used to perform line detection on the skin color segmentation results using Hough transform, and only retains line segments with an angle less than a set angle to the horizontal direction and a length greater than a set length.

[0188] The circle detection unit is used to perform circle detection on the skin color segmentation results using Hough transform, and remove circles that do not meet the requirements based on the position and size of the detected circles.

[0189] Furthermore, in the circle detection unit, the step of removing circles that do not meet the requirements based on the position and size of the detected circles includes:

[0190] Remove circles whose center ordinate is greater than ymax or less than ymin.

[0191] Wherein, ymax and ymin are the maximum and minimum values ​​of the ordinate of the line segment obtained by the line detection, respectively.

[0192] Circles whose absolute value of the difference between their center ordinate and ymin is greater than a set difference threshold are removed.

[0193] Remove circles with a radius greater than the set radius threshold.

[0194] As an improvement to this embodiment of the invention, the eye region image acquisition module includes:

[0195] The first cropping unit is used to select the two circles with the largest center coordinates if two or more circles are detected on the face region image, and crop the eye region image of the two eyes based on the two circles with the largest center coordinates.

[0196] The second cropping unit is used to obtain another circle based on the symmetrical characteristic of the two eyes on the face region image if a circle is detected on the face region image, and to crop the eye region image of the two eyes based on the one circle and the other circle.

[0197] The third cropping unit is used to calculate the positions of the two eyes based on the nearest straight line to the face region image and the position and size of the face region image if no circle is detected on the face region image, and to crop the eye region image of the two eyes based on the positions of the two eyes.

[0198] Furthermore, when two or more circles are detected on the face region image, the center coordinates of the two circles with the largest center ordinates are (x1, y1) and (x2, y2), and the cropping range of the eye region image of the two eyes is [x1- :x1+ ,y1- :y1+ ] and [x2- :x2+ ,y2- y2+ ];

[0199] in, The threshold value for the set cutoff scale.

[0200] When a circle is detected on the face region image, the center coordinates of the first circle are (rx, ry), the center coordinates of the second circle are (rx', ry'), and the cropping range of the eye region image for both eyes is [rx- :rx+ ,ry- :ry+ ] and [rx'- :rx'+ , ry'- :ry'+ ];

[0201] Where rx' = 2x + w – rx, ry' = ry, w is the width of the face region image, and x is the x-coordinate of the lower left corner of the face region image.

[0202] When no circle is detected in the face region image, the position coordinates of the two eyes are (x3, y3) and (x4, y4), and the cropping range of the eye region image of the two eyes is [x3- :x3+ ,y3- y3+ ] and [x4- :x4+ ,y4- y4+ ];

[0203] Among them, x3 = x + w / 4, y3 = k(x + w / 4) + b + h / 2, x4 = x + 3*w / 4, y4 = k(x +3 w / 4) + b + h / 2; k and b are the parameters of the line y'=kx'+b that is closest to the face region image, and h is the height of the face region image.

[0204] As another improvement to this embodiment of the invention, the skin color segmentation module includes:

[0205] The binary segmentation unit is used to perform binary segmentation on each pixel of the face region image using the following formula;

[0206]

[0207] in, On the face region image Skin color segmentation results for pixels at a given location. w and h are the width and height of the face region image, respectively;

[0208] for ;

[0209] iRed iGrn and iBlu On the face region image respectively The red, green, and blue channel values ​​of the pixels at that location;

[0210] ;

[0211] ;

[0212] ;

[0213] ;

[0214] The denoising unit is used to denoise the binary segmentation result using mean filtering and guided filtering to obtain the skin color segmentation result.

[0215] The key point regression module of this invention is used for:

[0216] The first feature map is obtained by filling several pixels around the eye region image and performing a first convolution operation, a first activation operation, a first absolute value operation and a first pooling operation.

[0217] For the first feature Figure 4The second feature map is obtained by filling several pixels around the perimeter and performing a second convolution operation, a second activation operation, a second absolute value operation, and a second pooling operation.

[0218] The second feature map is subjected to a third convolution operation, a third activation operation, a third absolute value operation, and a second pooling operation to obtain a third feature map.

[0219] The key points of the eye are obtained through regression of the third feature map.

[0220] The device provided in this embodiment of the invention has the same implementation principle and technical effects as the aforementioned method embodiment 1. For the sake of brevity, any parts not mentioned in this device embodiment can be referred to the corresponding content in the aforementioned method embodiment 1. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the aforementioned device and unit can all be referred to the corresponding processes in the aforementioned method embodiment 1, and will not be repeated here.

[0221] Example 3:

[0222] The method described in Embodiment 1 of this invention can implement business logic through a computer program and record it on a storage medium. This storage medium can be read and executed by a computer, achieving the effects of the solution described in Embodiment 1 of this specification. Therefore, this invention also provides a computer-readable storage medium for eye positioning, including a memory for storing processor-executable instructions. When executed by a processor, the instructions implement the steps of the eye positioning method of Embodiment 1.

[0223] This invention solves the problem of eye location when the face is obscured at a distance. It has a good effect on eye position detection when the face is obscured by masks or face shields. It can be applied to multimodal biometrics and long-distance iris recognition. In criminal investigation and other projects, it can be used to assist on-duty personnel or arrested persons in iris collection and authentication or multimodal identity authentication.

[0224] The storage medium may include a physical device for storing information, typically digitizing the information and then storing it using electrical, magnetic, or optical methods. The storage medium may include: devices that store information using electrical energy, such as various types of memory, like RAM and ROM; devices that store information using magnetic energy, such as hard disks, floppy disks, magnetic tapes, magnetic core memory, bubble memory, and USB flash drives; and devices that store information using optical methods, such as CDs or DVDs. Of course, there are other readable storage media, such as quantum memories and graphene memories.

[0225] The storage medium described above may also include other implementation methods according to the description of method embodiment 1. The implementation principle and technical effects of this embodiment are the same as those of the aforementioned method embodiment 1. For details, please refer to the description of the relevant method embodiment 1, which will not be repeated here.

[0226] Example 4:

[0227] The present invention also provides a device for eye positioning. This device may be a standalone computer, or it may include an actual operating device that uses one or more of the methods or embodiments described in this specification. The eye positioning device may include at least one processor and a memory storing computer-executable instructions. When the processor executes the instructions, it implements the steps of any one or more of the eye positioning methods described in Embodiment 1.

[0228] This invention solves the problem of eye location when the face is obscured at a distance. It has a good effect on eye position detection when the face is obscured by masks or face shields. It can be applied to multimodal biometrics and long-distance iris recognition. In criminal investigation and other projects, it can be used to assist on-duty personnel or arrested persons in iris collection and authentication or multimodal identity authentication.

[0229] The device described above may include other implementation methods according to the description of method embodiment 1. The implementation principle and technical effects of this embodiment are the same as those of the aforementioned method embodiment 1. For details, please refer to the description of the relevant method embodiment 1, which will not be repeated here.

[0230] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the scope of the technology disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. All should be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An eye positioning method, characterized in that, The method includes: Perform face detection on the image to be located to obtain a face region image; Skin color segmentation is performed on the face region image; Line detection and circle detection are performed on the skin color segmentation results; Wherein, the line segment obtained by the line detection represents the edge of the face occlusion, and the circle obtained by the circle detection represents the coarse positioning result of the eye; Based on the position and size of the face region image, the number and position of the circles detected by the circle detection, and the line segments detected by the line detection, an eye region image is extracted from the face region image; The eye region image is input into a convolutional neural network, and the key points of the eye are obtained through regression. The step of extracting the eye region image from the face region image based on the position and size of the face region image, the number and position of the circles obtained by the circle detection, and the line segments obtained by the line detection includes: If two or more circles are detected on the face region image, the two circles with the largest center ordinates are selected, and the eye region images of the two eyes are obtained based on the two circles with the largest center ordinates. If a circle is detected on the face region image, another circle is obtained based on the symmetrical characteristic of the two eyes on the face region image. The eye region images of the two eyes are then obtained by cropping the one circle and the other circle. If no circle is detected on the face region image, the positions of the two eyes are calculated based on the nearest straight line to the face region image and the position and size of the face region image, and the eye region images of the two eyes are obtained based on the positions of the two eyes. When no circle is detected in the face region image, the position coordinates of the two eyes are (x3, y3) and (x4, y4), and the cropping range of the eye region image of the two eyes is [x3- :x3+ ,y3- y3+ ] and [x4- :x4+ ,y4- y4+ ]; Where, x3 = x + w / 4, y3 = k(x + w / 4) + b + h / 2, x4 = x + 3 w / 4, y4 = k(x + 3) w / 4) + b + h / 2; x is the x-coordinate of the lower left corner of the face region image, k and b are the parameters of the line y'=kx'+b closest to the face region image, h is the height of the face region image, and w is the width of the face region image. The threshold value for the set cutoff scale.

2. The eye localization method according to claim 1, characterized in that, The process of performing line detection and circle detection on the skin color segmentation results includes: Hough transform is used to detect straight lines in the skin color segmentation results, and only line segments with an angle to the horizontal direction less than a set angle and a length greater than a set length are retained. Hough transform is used to perform circle detection on the skin color segmentation results, and circles that do not meet the requirements are removed based on the position and size of the detected circles.

3. The eye positioning method according to claim 2, characterized in that, The step of removing circles that do not meet the requirements based on the position and size of the detected circles includes: Remove circles whose center y-coordinate is greater than ymax or less than ymin; Where ymax and ymin are the maximum and minimum values ​​of the ordinate in the line segment obtained by the line detection, respectively; Remove circles whose absolute value of the difference between their center ordinate and ymin is greater than a set difference threshold; Remove circles with a radius greater than the set radius threshold.

4. The eye localization method according to claim 1, characterized in that, When two or more circles are detected on the face region image, the center coordinates of the two circles with the largest ordinates are (x1, y1) and (x2, y2), and the cropping range of the eye region image for the two eyes is [x1- :x1+ ,y1- :y1+ ] and [x2- :x2+ ,y2- y2+ ]; When a circle is detected on the face region image, the center coordinates of the first circle are (rx, ry), the center coordinates of the second circle are (rx', ry'), and the cropping range of the eye region image for both eyes is [rx- :rx+ ,ry- :ry+ ] and [rx'- :rx'+ , ry'- :ry'+ ]; Where rx' = 2x + w – rx, ry' = ry.

5. The eye localization method according to claim 1, characterized in that, The step of skin color segmentation of the facial region image includes: For each pixel of the face region image, binary segmentation is performed using the following formula; in, On the face region image Skin color segmentation results for pixels at a given location. w and h are the width and height of the face region image, respectively; for ; iRed iGrn and iBlu On the face region image respectively The red, green, and blue channel values ​​of the pixels at that location; ； ； ； ； The mean filtering and guided filtering are used to denoise the binary segmentation results to obtain the skin color segmentation results.

6. The eye localization method according to any one of claims 1-5, characterized in that, The step of inputting the eye region image into a convolutional neural network to regress and obtain the key points of the eye includes: The first feature map is obtained by filling several pixels around the eye region image and performing a first convolution operation, a first activation operation, a first absolute value operation and a first pooling operation. The first feature map is filled with several pixels around it, and then a second convolution operation, a second activation operation, a second absolute value operation, and a second pooling operation are performed to obtain a second feature map. The second feature map is subjected to a third convolution operation, a third activation operation, a third absolute value operation, and a third pooling operation to obtain a third feature map. The key points of the eye are obtained through regression of the third feature map.

7. An eye positioning device, characterized in that, The device includes: The face detection module is used to detect faces in the image to be located and obtain face region images; A skin color segmentation module is used to segment the skin color of the face region image; The line and circle detection module is used to perform line and circle detection on the skin color segmentation results; Wherein, the line segment obtained by the line detection represents the edge of the face occlusion, and the circle obtained by the circle detection represents the coarse positioning result of the eye; An eye region image acquisition module is used to extract an eye region image from the face region image based on the position and size of the face region image, the number and position of the circles detected by the circle detection, and the line segments detected by the line detection. The key point regression module is used to input the eye region image into a convolutional neural network and regress the key points of the eye. The eye region image acquisition module includes: If the first segmentation unit detects two or more circles on the face region image, it selects the two circles with the largest center ordinate and segments the eye region image of the two eyes based on the two circles with the largest center ordinate. The second cropping unit detects a circle on the face region image, and obtains another circle based on the symmetrical characteristic of the two eyes on the face region image. The eye region image of the two eyes is cropped based on the first circle and the second circle. The third cropping unit, if no circle is detected on the face region image, calculates the position of the two eyes based on the nearest straight line to the face region image and the position and size of the face region image, and crops the eye region image of the two eyes based on the position of the two eyes; When no circle is detected in the face region image, the position coordinates of the two eyes are (x3, y3) and (x4, y4), and the cropping range of the eye region image of the two eyes is [x3- :x3+ ,y3- y3+ ] and [x4- :x4+ ,y4- y4+ ]; Where, x3 = x + w / 4, y3 = k(x + w / 4) + b + h / 2, x4 = x + 3 w / 4, y4 = k(x + 3) w / 4) + b + h / 2; x is the x-coordinate of the lower left corner of the face region image, k and b are the parameters of the line y'=kx'+b closest to the face region image, h is the height of the face region image, and w is the width of the face region image. The threshold value for the set cutoff scale.

8. A computer-readable storage medium for eye positioning, characterized in that, It includes a memory for storing processor-executable instructions, which, when executed by the processor, implement the steps of the eye positioning method according to any one of claims 1-6.

9. A device for eye positioning, characterized in that, It includes at least one processor and a memory storing computer-executable instructions, wherein the processor, when executing the instructions, implements the steps of the eye positioning method according to any one of claims 1-6.

Images