Eye occlusion determination method, device, computer-readable storage medium, and device

Abstract

The present invention discloses a method, apparatus, computer-readable storage medium, and device for determining eye occlusion, belonging to the field of biometrics. The method comprises: acquiring a facial image and intercepting an eye region image; detecting a light spot region within the eye region image and calculating the proportion of the light spot region within the eye region image; and inputting eye region images whose proportions do not reach a set threshold into a pre-trained convolutional neural network to determine whether the eyes are occluded. The present invention effectively addresses the problem of misclassification due to light spots during eye occlusion determination, while ensuring classification accuracy and reducing time complexity.

Description

Technical Field

[0001] The present invention relates to the field of biometrics, and in particular to a method, apparatus, computer-readable storage medium, and device for determining eye occlusion. Background Art

[0002] The rise of deep learning has brought tremendous progress to biometric recognition, particularly in areas such as face recognition and image understanding. Biometric features can not only distinguish individuals but also determine their status. Face recognition requires first obtaining a clear, complete facial image in sufficient lighting, followed by data processing. In face recognition tasks, incomplete or occluded facial images can reduce accuracy. Occlusion of facial features, such as the eyes, is particularly detrimental to algorithm performance, making eye occlusion detection of significant research value.

[0003] The currently used method for detecting face occlusion mainly involves first obtaining a face image to be processed, marking the feature points of facial features in the obtained face image to be processed, and importing the image into a pre-trained face occlusion model to obtain occlusion information corresponding to the face occlusion image to be processed. The face occlusion model is used to obtain the occlusion information of the face through the feature points contained in the face occlusion image to be processed, and finally output the occlusion information.

[0004] Using the existing face occlusion detection method for eye occlusion judgment has the following drawbacks:

[0005] First, while methods based on the entire face for eye occlusion detection can utilize more comprehensive texture information, the network input dimension is larger and the algorithm is more time-consuming. Furthermore, the face region has more complex textures than the eye region, and is subject to more interference factors, making it less conducive to optimizing the classification task.

[0006] Secondly, existing face occlusion detection methods are prone to misjudging images with light spots, which can lead to inaccurate detection results for users wearing glasses. Since the light spots generated by the reflection of the lens in the imaging area of ​​the eye interfere with the eyes, even if the area of ​​the light spot is small, it can easily be misjudged as eye occlusion. Summary of the Invention

[0007] To address the technical problems in the prior art of eye occlusion judgment methods, which are time-consuming and susceptible to light spots, the present invention provides an eye occlusion judgment method, apparatus, computer-readable storage medium, and device, which effectively solve the problem of misjudgment caused by light spots during the eye occlusion judgment process, and reduce time complexity while ensuring classification accuracy.

[0008] The present invention provides the following technical solutions:

[0009] In a first aspect, the present invention provides a method for determining eye occlusion, the method comprising:

[0010] Acquire a face image and capture an eye area image;

[0011] Detecting a light spot area on the eye area image and calculating a proportion of the light spot area in the eye area image;

[0012] The eye area image whose proportion does not reach the set proportion threshold is input into the pre-trained convolutional neural network to determine whether the eyes are occluded.

[0013] Furthermore, detecting the light spot area on the eye area image includes:

[0014] Gray-scaling the eye region image to obtain a grayscale image;

[0015] The grayscale value of each pixel on the grayscale image is replaced by the statistical value of the grayscale values ​​of all pixels in a neighborhood of a set size around the pixel to obtain a statistical feature image;

[0016] Divide the statistical feature image into image blocks of a predetermined size, and calculate the proportion of pixels in each image block whose grayscale value is greater than a set grayscale threshold to all pixels in the image block;

[0017] If the ratio exceeds a set ratio threshold, the image block is marked as a light spot, and all light spots are connected to form the light spot area.

[0018] Furthermore, the acquiring of the face image and intercepting of the eye area image includes:

[0019] Acquire a facial image and locate facial key points, wherein the facial key points include eye key points;

[0020] Calculating the tilt angle of the eyes according to the eye key points, and rotating and aligning the face image according to the tilt angle;

[0021] An eye region image is cut out from the aligned face image according to the eye key points and a normalization operation is performed.

[0022] Furthermore, inputting the eye region image whose proportion does not reach the set proportion threshold into a pre-trained convolutional neural network to determine whether the eyes are occluded includes:

[0023] Performing a first convolution operation, a first BN operation, a first ReLU operation, and a first pooling operation on the eye region image whose proportion does not reach the set proportion threshold, to obtain a first feature map;

[0024] Expanding the edge of the first feature map by a number of pixels, and performing a second convolution operation, a second BN operation, a second ReLU operation, and a second pooling operation to obtain a second feature map;

[0025] Expanding the edge of the second feature map by a number of pixels, and performing a third convolution operation, a third BN operation, a third ReLU operation, and a third pooling operation to obtain a third feature map;

[0026] Performing a first fully connected operation, a fourth ReLU operation, a first dropout operation, a second fully connected operation, a fifth ReLU operation, and a second dropout operation on the third feature map to obtain a fourth feature map;

[0027] Performing a third fully connected operation with an output dimension of 2 on the fourth feature map, and performing classification using Softmax to obtain a probability that the eye region image is an eye and a probability that it is not an eye;

[0028] If the probability that the eye region image is an eye exceeds the set occlusion threshold, the eye is judged to be unoccluded; otherwise, the eye is judged to be occluded.

[0029] Furthermore, the convolutional neural network is trained by the following method:

[0030] Constructing a training set, wherein the training set includes a plurality of training samples, wherein the training samples include positive samples and negative samples, wherein the positive samples are unobstructed eye images, and the negative samples are eye images obstructed by non-eye images;

[0031] Performing data expansion on the training set, wherein the data expansion includes translating, rotating, modifying contrast, and / or adding noise to the training samples;

[0032] The convolutional neural network is trained using the data augmented training set.

[0033] Furthermore, the non-eye image includes a non-skin color image and a skin color image outside the eye area;

[0034] The data expansion further includes performing small spot expansion on the training samples, and the small spot expansion includes:

[0035] Randomly obtain training samples from the training set with a certain probability and locate the eye area in the training samples;

[0036] Map the eye area in RGB space to HSV space and reduce the value of brightness space H;

[0037] Small white light spots are randomly added to the eye area, and the proportion of the small light spots in the eye area is less than the proportion threshold.

[0038] In a second aspect, the present invention provides an eye occlusion determination device, the device comprising:

[0039] An image acquisition module is used to acquire a face image and intercept an eye area image;

[0040] a light spot detection module, configured to detect a light spot area on the eye area image and calculate a proportion of the light spot area in the eye area image;

[0041] The occlusion judgment module is used to input the eye area image whose proportion does not reach the set proportion threshold into the pre-trained convolutional neural network to determine whether the eyes are occluded.

[0042] Furthermore, the light spot detection module includes:

[0043] a grayscale unit, configured to grayscale the eye region image to obtain a grayscale image;

[0044] A first statistical unit is used to replace the grayscale value of each pixel on the grayscale image with the statistical value of the grayscale values ​​of all pixels in a neighborhood of a set size around the pixel to obtain a statistical feature image;

[0045] a second statistical unit, configured to divide the statistical feature image into image blocks of a predetermined size, and calculate the proportion of pixels in each image block whose grayscale value is greater than a set grayscale threshold to all pixels in the image block;

[0046] The light spot judgment unit is configured to mark the image block as a light spot if the ratio exceeds a set ratio threshold, and all light spots are connected to form the light spot area.

[0047] Furthermore, the image acquisition module includes:

[0048] An acquisition unit, configured to acquire a face image and locate key points of the face, wherein the key points of the face include eye key points;

[0049] a rotation unit, configured to calculate an eye tilt angle based on the eye key points, and rotate and align the face image based on the tilt angle;

[0050] The clipping and normalization unit is used to clip the eye area image from the aligned face image according to the eye key points and perform normalization operation.

[0051] Furthermore, the occlusion judgment module includes:

[0052] A first operation unit is configured to perform a first convolution operation, a first BN operation, a first ReLU operation, and a first pooling operation on the eye region image whose proportion does not reach a set proportion threshold, to obtain a first feature map;

[0053] A second operation unit is configured to expand the edge of the first feature map by a number of pixels, and perform a second convolution operation, a second BN operation, a second ReLU operation, and a second pooling operation to obtain a second feature map;

[0054] a third operation unit, configured to expand the edge of the second feature map by a number of pixels, and perform a third convolution operation, a third BN operation, a third ReLU operation, and a third pooling operation to obtain a third feature map;

[0055] a fourth operation unit, configured to perform a first fully connected operation, a fourth ReLU operation, a first dropout operation, a second fully connected operation, a fifth ReLU operation, and a second dropout operation on the third feature map to obtain a fourth feature map;

[0056] a classification unit, configured to perform a third fully connected operation with an output dimension of 2 on the fourth feature map, and perform classification using Softmax to obtain a probability that the eye region image is an eye and a probability that the eye region image is not an eye;

[0057] The occlusion judgment unit is used to judge that the eye is not occluded if the probability that the eye area image is an eye exceeds a set occlusion threshold, otherwise judge that the eye is occluded.

[0058] Furthermore, the convolutional neural network is trained by the following modules:

[0059] A training set construction module, configured to construct a training set, wherein the training set includes a plurality of training samples, wherein the training samples include positive samples and negative samples, wherein the positive samples are unobstructed eye images, and the negative samples are eye images obstructed by non-eye images;

[0060] A data expansion module, configured to perform data expansion on the training set, wherein the data expansion includes translating, rotating, modifying contrast, and / or adding noise to the training samples;

[0061] The training module is used to train the convolutional neural network using the data-expanded training set.

[0062] Furthermore, the non-eye image includes a non-skin color image and a skin color image outside the eye area;

[0063] The data expansion further includes performing small spot expansion on the training samples, and the small spot expansion includes:

[0064] A sample acquisition unit, configured to randomly acquire training samples from a training set with a certain probability and locate the eye region in the training samples;

[0065] A brightness adjustment unit, used to map the eye area in the RGB space to the HSV space and reduce the value of the brightness space H;

[0066] The light spot adding unit is used to randomly add small white light spots on the eye area, and the proportion of the small light spots in the eye area is less than the proportion threshold.

[0067] In a third aspect, the present invention provides a computer-readable storage medium for eye occlusion determination, comprising a memory for storing processor-executable instructions, wherein the instructions, when executed by the processor, implement the steps of the eye occlusion determination method described in the first aspect.

[0068] In a fourth aspect, the present invention provides a device for eye occlusion judgment, comprising at least one processor and a memory storing computer-executable instructions, wherein when the processor executes the instructions, the steps of the eye occlusion judgment method described in the first aspect are implemented.

[0069] The present invention has the following beneficial effects:

[0070] This method first uses spot detection to eliminate eye region images with large spot sizes. It then uses a convolutional neural network to classify eye occlusion in images without or with small spot sizes, effectively addressing the problem of misclassification caused by spot sizes during eye occlusion determination. The convolutional neural network uses eye region images as input, which has a smaller dimensionality than the facial image input used in existing techniques. This reduces the time required and increases efficiency, ensuring classification accuracy while reducing time complexity. Furthermore, eye region images are simpler than facial images, with fewer interfering factors, facilitating optimization of the classification task. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 This is a flow chart of the eye occlusion determination method of the present invention;

[0072] Figure 2 This is a flow chart of S200 in the eye occlusion determination method of the present invention;

[0073] Figure 3 This is a flow chart of S100 in the eye occlusion determination method of the present invention;

[0074] Figure 4This is a flow chart of S300 in the eye occlusion determination method of the present invention;

[0075] Figure 5 Schematic diagram of the training process of the convolutional neural network in the present invention;

[0076] Figure 6 This is a schematic diagram of the eye key points in the human face key points;

[0077] Figure 7 Schematic diagram of the process of acquiring and normalizing the eye area image;

[0078] Figure 8 A schematic diagram of all pixels in a 3*3 neighborhood;

[0079] Figure 9 This is an example diagram of the statistical feature image obtained in S220 of the eye occlusion determination method of the present invention;

[0080] Figure 10 Schematic diagram of the structure of the convolutional neural network in the present invention;

[0081] Figure 11 is a schematic diagram of an eye occlusion determination device according to the present invention;

[0082] Figure 12 Schematic diagram of a light spot detection module in the eye occlusion judgment device of the present invention;

[0083] Figure 13 Schematic diagram of an image acquisition module in the eye occlusion determination device of the present invention;

[0084] Figure 14 is a schematic diagram of an occlusion determination module in the eye occlusion determination device of the present invention;

[0085] Figure 15 Schematic diagram of the training module of the convolutional neural network in the present invention. DETAILED DESCRIPTION

[0086] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings and specific embodiments. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. The components of the embodiments of the present invention generally described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

[0087] Example 1:

[0088] The embodiment of the present invention provides a method for determining eye occlusion. Figure 1 As shown, the method includes:

[0089] S100: Acquire a face image and capture an eye area image.

[0090] This step is used to obtain an eye region image. There are many ways to implement this. For example, after acquiring a facial image, face detection and eye key point location are performed, and the eye region image is captured based on the located eye key points. Of course, preprocessing operations such as rotation correction, blur correction, and normalization can also be performed on the facial image or eye region image.

[0091] S200: Detecting a light spot area on the eye area image and calculating a proportion of the light spot area in the eye area image.

[0092] Although the existing face occlusion detection method can also be used for eye occlusion judgment, this method is relatively sensitive to lighting. In particular, when using the face system, users wearing glasses will produce a large light spot under lighting. If the light spot overlaps with the human eye area to a large extent, the occlusion algorithm will make an incorrect judgment.

[0093] In order to solve the above problems, the present invention needs to perform light spot detection on the eye area image and determine the overlapping area of ​​the light spot and the eye area. The overlapping area is specifically represented by the proportion of the light spot area in the eye area image. If the proportion reaches the set proportion threshold (for example, 20%), the light spot is considered to be a large light spot. Large light spots will affect the occlusion judgment, and large light spots will affect the accuracy of recognition such as face recognition or iris recognition. Therefore, images with large light spots are generally unusable. Facial images with large light spots can be directly considered to have lighting problems, judged as unqualified, and refused to log in and use.

[0094] S200: Inputting the eye region image whose proportion does not reach the set proportion threshold into a pre-trained convolutional neural network to determine whether the eyes are occluded.

[0095] If the proportion does not reach the set proportion threshold, it is considered that there is no light spot or the light spot is a small light spot. Only a large light spot will cause the occlusion algorithm to make an incorrect judgment. Therefore, the eye area image with no light spot or a small light spot can be input into the convolutional neural network for eye occlusion judgment.

[0096] Compared to existing technologies, this invention first eliminates eye area images with large light spots through light spot detection. It then uses a convolutional neural network to classify eye area images without or with small light spots as being occluded, effectively addressing the problem of misclassification caused by light spots during eye occlusion determination. The convolutional neural network input is the eye area image. Compared to the facial image input used in existing technologies, the input dimension is smaller, making the convolutional neural network less time-consuming and more efficient. This ensures classification accuracy while reducing time complexity. Furthermore, eye area images are simpler than facial images, with fewer interference factors, which facilitates optimization of the classification task.

[0097] In the present invention, the light spot area can be detected by various methods, which are described below with two specific examples.

[0098] Example 1:

[0099] First, grayscale the eye region image. Convert the three-channel color image to a grayscale image and calculate the grayscale value of each pixel in the grayscale image. Based on the characteristics of eye region images, the grayscale values ​​of pixels in non-light spots should all be less than a grayscale threshold (e.g., 200). Therefore, if there is a pixel in the eye region image with a grayscale value greater than or equal to 200, it can be classified as a light spot area.

[0100] Example 2:

[0101] In the above example 1, the light spot detection is performed based on the statistics of the gray value of each pixel. This method has poor robustness and is prone to false positives. In order to enhance the robustness of light spot detection and reduce false positives, this example uses the following method to detect the light spot area on the eye area image (such as Figure 2 shown):

[0102] S210: grayscale the eye region image to obtain a grayscale image.

[0103] S220: replacing the grayscale value of each pixel on the grayscale image with the statistical value of the grayscale values ​​of all pixels in a neighborhood of a set size around the pixel to obtain a statistical feature image.

[0104] In this step, the size of the neighborhood can be set, for example, a neighborhood of size 3*3. For each pixel, the grayscale value of the pixel is replaced by the statistical value (for example, the average value) of the grayscale values ​​of all pixels in the surrounding 3*3 neighborhood.

[0105] like Figure 8As shown in the following example, the grayscale value at position 0 is replaced by the grayscale values ​​of the surrounding 8 positions. Let s(0) be the grayscale value at position 0 after replacement, and r1, r2, ..., r8 be the grayscale values ​​of positions 1-8 around position 0. Then:

[0106]

[0107] By performing the above operation on each pixel of the eye area image, a statistical feature image can be obtained. The statistical value of each position (x, y) can be recorded as s(x, y), as follows: Figure 9 shown.

[0108] S230: Divide the statistical feature image into image blocks of a predetermined size, and calculate the proportion of pixels in each image block whose grayscale values ​​are greater than a set grayscale threshold to all pixels in the image block.

[0109] In this step, the size of the image block can be set, for example, it can be a 3*3 image block. Statistical voting is performed on the 3*3 image block to calculate the number of pixels with a statistical value exceeding 200.

[0110] S240: If the ratio exceeds a set ratio threshold, the image block is marked as a light spot, and all the light spots are connected to form a light spot area.

[0111] If the statistical values ​​of more than 50% of the pixels exceed 200, the image block is determined to be a light spot; otherwise, the image block is not considered to be a light spot.

[0112] like Figure 9 As shown, the 3*3 statistical values ​​of the (i, j) position are counted. If there are greater than or equal to 4 statistical values ​​exceeding 200, the position is considered to be a light spot, and all the light spots detected by the above method are connected into a light spot area.

[0113] The light spot detection method of this example can relatively quickly detect the position and size of the light spot, calculate the overlapping area between the light spot and the face area, and make a corresponding judgment.

[0114] As an improvement of the embodiment of the present invention, Figure 3 As shown, the aforementioned S100 includes:

[0115] S110: Acquire a face image and locate key points of the face, including key points of the eyes.

[0116] This step does not limit the method of obtaining facial images. For example, facial images can be collected through a facial lens. When collecting, a single image can be collected, or a video can be collected and a single frame or multiple frames of images can be obtained from the video.

[0117] After acquiring a facial image, face detection and keypoint localization are required. For example, an SSD face detector can be used for face detection, followed by a TDCNN method for keypoint localization, obtaining the coordinates of the facial keypoints. Face detectors are not limited to SSD; alternatives include CRAFT and ADABOOST. Similarly, facial keypoint localization is not limited to TDCNN; alternatives include SDM, as long as only the eye keypoints are captured.

[0118] Different facial key point positioning methods obtain different numbers of facial key points, for example, 68, 106, 1000, etc. These key points all need to include eye key points.

[0119] S120: Calculate the tilt angle of the eyes according to the eye key points, and rotate and align the face image according to the tilt angle.

[0120] In this step, the left eye center coordinates (x1, y1) and the right eye center coordinates (x2, y2) are preferably obtained through the eye key points, and the eye tilt angle is calculated based on the left eye center coordinates and the right eye center coordinates. The eye tilt angle is the angle a between the line connecting the left and right eye centers and the horizontal direction. The calculation method of a is as follows:

[0121]

[0122] Rotating the face image by an angle a can align the face image and eliminate the tilt angle of the eyes.

[0123] S130: Cut out the eye region image from the aligned face image according to the eye key points and perform a normalization operation.

[0124] When capturing the eye area image, one of the specific implementation methods is as follows Figure 6 、 7 As shown in the figure: among the located facial key points, find the edge points of the left eye (or right eye), that is, the leftmost, rightmost, topmost and bottommost key points, expand these four key points outward by 5 pixels respectively to obtain 4 new edge key points, obtain the eye area image with these four edge key points as the edge, and scale the image to an image with a width and height of 80 pixels respectively.

[0125] In the present invention, the structure of the convolutional neural network is as follows Figure 10 As shown in the figure, the specific processing process of the convolutional neural network on the eye area image is as follows Figure 4 As shown, such as including:

[0126] S310: Perform a first convolution operation, a first BN operation, a first ReLU operation, and a first pooling operation on the eye area image whose proportion does not reach the set proportion threshold to obtain a first feature map.

[0127] For example, the eye region image (data) is a three-channel image with a width and height of 80 pixels. The first convolution operation (conv1) uses a convolution kernel (kemel size) of 7*7 and a stride of 4. This convolution operation produces a 19*19 feature map with 36 channels. The first batch normalization operation (batch1) and the first Relu1 operation (relu1) are performed on this feature map, resulting in a 19*19 feature map with 36 channels.

[0128] The first pooling operation (pool1) is performed on the above-mentioned 36-channel feature map of size 19*19. The first pooling operation is a Maxpooling operation with a size of 3*3 and a step size of 2, and a 36-channel sampling result of size 9*9 is obtained, which is the first feature map.

[0129] S320: Expand the edge of the first feature map by a number of pixels, and perform a second convolution operation, a second BN operation, a second ReLU operation, and a second pooling operation to obtain a second feature map.

[0130] For example, the edge of the first feature map of 36 channels with a size of 9*9 is expanded by 2 pixels, and a second convolution operation (conv2) is performed with a convolution kernel of size 5*5 and a step size of 1 to obtain a feature map of 96 channels with a size of 9*9. The second BN operation (batch2) and the second ReLU (relu2) operation are performed on the obtained feature map, and the obtained feature map is also a feature map of 96 channels with a size of 9*9.

[0131] The second pooling operation (pool2) is performed on the above-mentioned 96-channel feature map of size 9*9. The second pooling operation is a Maxpooling operation with a size of 3*3 and a step size of 2, and a 96-channel sampling result of size 4*4 is obtained, which is the second feature map.

[0132] S330: Expand the edge of the second feature map by several pixels, and perform a third convolution operation, a third BN operation, a third ReLU operation, and a third pooling operation to obtain a third feature map.

[0133] Exemplarily, the edge of the second feature map of 96 channels with a size of 4*4 is expanded by 1 pixel, and a convolution kernel of size 3*3 is used with a step size of 1 to perform a third convolution operation (conv3) to obtain a feature map of 144 channels with a size of 4*4. The third BN operation (batch3) and the third ReLU operation (relu3) are performed on the obtained feature map, and the obtained feature map is also a feature map of 144 channels with a size of 4*4.

[0134] The third pooling operation (pool5) is performed on the above-mentioned feature map of 144 channels with a size of 4*4. The third pooling operation is a Maxpooling operation with a size of 3*3 and a step size of 2, and a 144-channel sampling result with a size of 2*2 is obtained, which is the third feature map.

[0135] S340: Perform a first fully connected operation, a fourth ReLU operation, a first dropout operation, a second fully connected operation, a fifth ReLU operation, and a second dropout operation on the third feature map to obtain a fourth feature map.

[0136] Exemplarily, a first fully connected operation (fc6) is performed on the third feature map of 144 channels with a size of 2*2 to obtain a feature vector with a dimension of 512, and then a fourth ReLU operation (relu6) and a first dropout operation (drop6) with a ratio of 0.5 are performed on the feature vector.

[0137] The result of the first dropout operation is subjected to a second fully connected operation (fc7) to obtain a feature vector with a dimension of 512. Then, the feature vector is subjected to a fifth ReLU operation (relu7) and a second dropout operation (drop7) with a ratio of 0.5 to obtain the fourth feature map.

[0138] S350: Perform a third fully connected operation (fc8) with an output dimension of 2 on the fourth feature map, and use Softmax to perform classification to obtain the probability that the eye area image is an eye and the probability that it is not an eye.

[0139] The goal of the convolutional neural network classification in the present invention is to determine whether the eye region image is an eye. The output of the network is a two-dimensional vector, where the first dimension represents the probability value of the image not being an eye, and the second dimension represents the probability value of the image being an eye.

[0140] S350: If the probability that the eye region image is an eye exceeds the set occlusion threshold, it is determined that the eye is not occluded; otherwise, it is determined that the eye is occluded.

[0141] This step uses the output value of the second dimension (the probability value of the eye) to equivalently replace the probability that the eye is not occluded, denoted as p. Set the occlusion threshold T. If p>T, it means that the eye is not occluded, otherwise it means that the eye is occluded.

[0142] The network used in the present invention is not only effective but also has fewer parameters. It has a good description effect on the characteristics of whether the eyes are blocked, ensuring the accuracy of classification. At the same time, the depth and width of the network are controlled, the time complexity is reduced, and the classification has a higher accuracy and a faster speed.

[0143] The aforementioned convolutional neural network needs to be pre-trained before use. The training method is as follows: Figure 5 As shown, including:

[0144] S100': constructing a training set, which includes a number of training samples. The training samples include positive samples and negative samples. The positive samples are unobstructed eye images, and the negative samples are eye images obstructed by non-eye images.

[0145] The positive samples of the training set are normal, unoccluded eye images. For negative samples, a variety of non-eye images can be used to occlude the eye images for construction.

[0146] S200': performing data expansion on the training set, where data expansion includes translating, rotating, modifying contrast, and / or adding noise to the training samples.

[0147] In order to obtain better training results, both positive and negative samples need to be expanded to enhance data diversity.

[0148] S300': Use the data-augmented training set to train the convolutional neural network.

[0149] The aforementioned non-eye images include non-skin color images and skin color images outside the eye area. The non-skin color images can be cardboard images, black blocks, etc., and the skin color images outside the eye area can be skin color images of hands, face, etc.

[0150] It should be noted that the light spot detection method of the present invention detects large light spots, which solves the problem of image occlusion misjudgment caused by large light spots. For image occlusion problems with smaller light spot areas, convolutional neural networks are also required for classification and recognition. Therefore, the data augmentation of the present invention also includes small light spot augmentation of the training samples, adding small light spots to the training positive and negative samples. This operation facilitates the network to better classify small light spots.

[0151] The small spot expansion method includes:

[0152] S210': randomly obtain training samples from the training set with a certain probability and locate the eye area in the training samples.

[0153] In this step, a certain probability can be set to 0.5, and training samples are randomly obtained with this probability. Face detection and feature point positioning are performed on these samples (the methods of face detection and feature point positioning are the same as those mentioned above), and the eye area is obtained according to the positioning results.

[0154] S220′: Map the eye area in the RGB space to the HSV space and reduce the value of the brightness space H.

[0155] This step is used to adjust the brightness of the eye area so that the overall brightness of the eye area decreases.

[0156] S230': randomly adding small white light spots to the eye area, where the proportion of the small light spots in the eye area is less than a proportion threshold.

[0157] The size of the small spot should be smaller than that of the large spot, that is, the proportion of the small spot in the eye area should be smaller than the set proportion threshold (20%). For example, the size of the small spot can be 3*3, 9*9, or 27*27.

[0158] Example 2:

[0159] The embodiment of the present invention provides an eye occlusion judgment device, such as Figure 11 As shown, the device includes:

[0160] The image acquisition module 10 is used to acquire a face image and intercept an eye area image.

[0161] The light spot detection module 20 is used to detect the light spot area on the eye area image and calculate the proportion of the light spot area in the eye area image.

[0162] The occlusion judgment module 30 is used to input the eye area image whose proportion does not reach the set proportion threshold into a pre-trained convolutional neural network to determine whether the eyes are occluded.

[0163] Compared to existing technologies, this invention first eliminates eye area images with large light spots through light spot detection. It then uses a convolutional neural network to classify eye area images without or with small light spots as being occluded, effectively addressing the problem of misclassification caused by light spots during eye occlusion determination. The convolutional neural network input is the eye area image. Compared to the facial image input used in existing technologies, the input dimension is smaller, making the convolutional neural network less time-consuming and more efficient. This ensures classification accuracy while reducing time complexity. Furthermore, eye area images are simpler than facial images, with fewer interference factors, which facilitates optimization of the classification task.

[0164] like Figure 12 As shown, the light spot detection module 20 of the present invention includes:

[0165] The grayscale unit 21 is used to grayscale the eye region image to obtain a grayscale image.

[0166] The first statistical unit 22 is used to replace the grayscale value of each pixel on the grayscale image with the statistical value of the grayscale values ​​of all pixels in a neighborhood of a set size around the pixel to obtain a statistical feature image.

[0167] The second statistical unit 23 is used to divide the statistical feature image into image blocks of a predetermined size, and calculate the proportion of pixels in each image block whose grayscale value is greater than a set grayscale threshold to all pixels in the image block.

[0168] The light spot judgment unit 24 is configured to mark the image block as a light spot if the ratio exceeds a set ratio threshold, and all the light spots are connected to form a light spot area.

[0169] As an improvement of the embodiment of the present invention, the aforementioned image acquisition module 10 is as follows Figure 13 As shown, including:

[0170] The acquisition unit 11 is used to acquire a face image and locate key points of the face, including key points of the eyes.

[0171] The rotation unit 12 is used to calculate the tilt angle of the eyes according to the key points of the eyes, and rotate and align the face image according to the tilt angle.

[0172] The clipping and normalization unit 13 is used to clip the eye area image from the aligned face image according to the eye key points and perform normalization operation.

[0173] In the present invention, the structure of the convolutional neural network is as follows Figure 10 As shown, accordingly, the occlusion judgment module 30 is as follows Figure 14 Shown, including:

[0174] The first operation unit 31 is used to perform a first convolution operation, a first BN operation, a first ReLU operation, and a first pooling operation on the eye area image whose proportion does not reach a set proportion threshold, to obtain a first feature map.

[0175] The second operation unit 32 is used to expand the edge of the first feature map by a number of pixels, and perform a second convolution operation, a second BN operation, a second ReLU operation, and a second pooling operation to obtain a second feature map.

[0176] The third operation unit 33 is used to expand the edge of the second feature map by several pixels, and perform a third convolution operation, a third BN operation, a third ReLU operation and a third pooling operation to obtain a third feature map.

[0177] The fourth operation unit 34 is configured to perform a first fully connected operation, a fourth ReLU operation, a first dropout operation, a second fully connected operation, a fifth ReLU operation, and a second dropout operation on the third feature map to obtain a fourth feature map.

[0178] The classification unit 35 is configured to perform a third fully connected operation with an output dimension of 2 on the fourth feature map, and perform classification using Softmax to obtain a probability that the eye region image is an eye and a probability that the eye region image is not an eye.

[0179] The occlusion judgment unit 36 ​​is configured to judge that the eye is not occluded if the probability that the eye region image is an eye exceeds a set occlusion threshold, and otherwise judge that the eye is occluded.

[0180] The aforementioned convolutional neural network needs to be pre-trained before it can be used, such as Figure 15 As shown, it is trained through the following modules:

[0181] The training set construction module 10 ′ is used to construct a training set, which includes a number of training samples. The training samples include positive samples and negative samples. The positive samples are unobstructed eye images, and the negative samples are eye images obstructed by non-eye images.

[0182] The data expansion module 20' is used to perform data expansion on the training set. Data expansion includes translating, rotating, modifying contrast and / or adding noise to the training samples.

[0183] The training module 30' is used to train the convolutional neural network using the data-expanded training set.

[0184] The aforementioned non-eye images include non-skin color images and skin color images outside the eye area.

[0185] The data expansion also includes performing small spot expansion on the training samples, and the small spot expansion includes:

[0186] The sample acquisition unit is used to randomly acquire training samples from the training set with a certain probability and locate the eye area in the training samples.

[0187] The brightness adjustment unit is used to map the eye area in the RGB space to the HSV space and reduce the value of the brightness space H.

[0188] The light spot adding unit is used to randomly add small white light spots on the eye area, and the proportion of the small light spots in the eye area is less than a proportion threshold.

[0189] The device provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned method embodiment 1. For the sake of brevity, any matters not mentioned in the device embodiment can be referred to the corresponding contents in the aforementioned method embodiment 1. Those skilled in the art will clearly understand that for the convenience and brevity of description, the specific working processes of the devices and units described above can all refer to the corresponding processes in the aforementioned method embodiment 1 and will not be repeated here.

[0190] Example 3:

[0191] The method described in Example 1 above, as provided in this specification, can implement the business logic via a computer program and be recorded on a storage medium. This storage medium can be read and executed by a computer to achieve the effects of the solution described in Example 1 of this specification. Therefore, the present invention also provides a computer-readable storage medium for eye occlusion determination, comprising a memory for storing processor-executable instructions. When executed by the processor, these instructions implement the steps of the eye occlusion determination method described in Example 1.

[0192] This method first uses spot detection to eliminate eye region images with large spot sizes. It then uses a convolutional neural network to classify eye occlusion in images without or with small spot sizes, effectively addressing the problem of misclassification caused by spot sizes during eye occlusion determination. The convolutional neural network uses eye region images as input, which has a smaller dimensionality than the facial image input used in existing techniques. This reduces the time required and increases efficiency, ensuring classification accuracy while reducing time complexity. Furthermore, eye region images are simpler than facial images, with fewer interfering factors, facilitating optimization of the classification task.

[0193] The storage medium may include a physical device for storing information, typically digitizing the information and then storing it in a medium utilizing electrical, magnetic, or optical means. Examples of such storage media include: devices that use electrical energy to store information, such as various types of memory, such as RAM and ROM; devices that use magnetic energy to store information, such as hard disks, floppy disks, magnetic tapes, magnetic core memories, bubble memories, and USB flash drives; and devices that use optical means to store information, such as CDs or DVDs. Of course, other types of readable storage media exist, such as quantum memories and graphene memories.

[0194] The above-mentioned device may also include other implementations according to the description of method embodiment 1. The specific implementation methods can refer to the description of the relevant method embodiment 1, and will not be described in detail here.

[0195] Example 4:

[0196] The present invention also provides a device for determining eye occlusion. The device may be a standalone computer or may include an operating device that implements one or more of the methods or apparatuses described in this specification. The device may include at least one processor and a memory storing computer-executable instructions. When the processor executes the instructions, it implements the steps of the eye occlusion determination method described in any one or more of Embodiment 1.

[0197] This method first uses spot detection to eliminate eye region images with large spot sizes. It then uses a convolutional neural network to classify eye occlusion in images without or with small spot sizes, effectively addressing the problem of misclassification caused by spot sizes during eye occlusion determination. The convolutional neural network uses eye region images as input, which has a smaller dimensionality than the facial image input used in existing techniques. This reduces the time required and increases efficiency, ensuring classification accuracy while reducing time complexity. Furthermore, eye region images are simpler than facial images, with fewer interfering factors, facilitating optimization of the classification task.

[0198] The above-mentioned device may also include other implementation methods according to the description of the method or device embodiment. The specific implementation method can refer to the description of the relevant method embodiment 1, and will not be described in detail here.

[0199] It should be noted that the device or system described above in this specification may also include other implementation methods according to the description of the relevant method embodiments. The specific implementation methods can refer to the description of the method embodiments, and will not be described one by one here. The various embodiments in this specification are described in a progressive manner, and the same and similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the hardware + program type and storage medium + program embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0200] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0201] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, an in-vehicle human-computer interaction device, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0202] For the convenience of description, the above devices are described in terms of their functions and are divided into various modules. Of course, when implementing one or more of the present specifications, the functions of each module can be implemented in the same or multiple software and / or hardware, or the module that implements the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0203] Those skilled in the art will also appreciate that, in addition to implementing the controller in pure computer-readable program code, it is entirely possible to implement the same functionality by logically programming the method steps in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, embedded microcontrollers, and the like. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered structures within the hardware component. Alternatively, the devices for implementing various functions can be considered both software modules implementing the method and structures within the hardware component.

[0204] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0205] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0206] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0207] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0208] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the process, method, or apparatus that includes the element.

[0209] Those skilled in the art will appreciate that one or more embodiments of this specification may be provided as methods, systems, or computer program products. Thus, one or more embodiments of this specification may take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0210] One or more embodiments of this specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. One or more embodiments of this specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In distributed computing environments, program modules may be located in local and remote computer storage media, including storage devices.

[0211] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between the various embodiments can be referenced across them. Each embodiment focuses on the differences from the other embodiments. In particular, since the system embodiments are generally similar to the method embodiments, their description is relatively simple. For relevant parts, reference can be made to the description of the method embodiments. Throughout this specification, reference to the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples" indicates that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of this specification. In this specification, the schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, those skilled in the art may combine and integrate the different embodiments or examples, and features of different embodiments or examples, described in this specification, without conflict.

[0212] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, intended to illustrate the technical solutions of the present invention, rather than to limit them. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can, within the technical scope disclosed by the present invention, modify or readily conceive of variations to the technical solutions described in the above-described embodiments, or substitute equivalently for some of the technical features thereof. Such modifications, variations, or substitutions do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. They should all be encompassed within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A method for determining eye occlusion, characterized in that: The method comprises: Acquire a face image and capture an eye area image; Detecting a light spot area on the eye area image and calculating a proportion of the light spot area in the eye area image; Input the eye area image whose proportion does not reach the set proportion threshold into the pre-trained convolutional neural network to determine whether the eyes are occluded; Inputting the eye region image whose proportion does not reach the set proportion threshold into a pre-trained convolutional neural network to determine whether the eyes are occluded includes: Performing a first convolution operation, a first BN operation, a first ReLU operation, and a first pooling operation on the eye region image whose proportion does not reach the set proportion threshold, to obtain a first feature map; Expanding the edge of the first feature map by a number of pixels, and performing a second convolution operation, a second BN operation, a second ReLU operation, and a second pooling operation to obtain a second feature map; Expanding the edge of the second feature map by a number of pixels, and performing a third convolution operation, a third BN operation, a third ReLU operation, and a third pooling operation to obtain a third feature map; Performing a first fully connected operation, a fourth ReLU operation, a first dropout operation, a second fully connected operation, a fifth ReLU operation, and a second dropout operation on the third feature map to obtain a fourth feature map; Performing a third fully connected operation with an output dimension of 2 on the fourth feature map, and performing classification using Softmax to obtain a probability that the eye region image is an eye and a probability that it is not an eye; If the probability that the eye region image is an eye exceeds the set occlusion threshold, the eye is judged to be unoccluded; otherwise, the eye is judged to be occluded.

2. The eye occlusion determination method according to claim 1, wherein: The detecting and obtaining the light spot area on the eye area image includes: Gray-scaling the eye region image to obtain a grayscale image; The grayscale value of each pixel on the grayscale image is replaced by the statistical value of the grayscale values ​​of all pixels in a neighborhood of a set size around the pixel to obtain a statistical feature image; Divide the statistical feature image into image blocks of a predetermined size, and calculate the proportion of pixels in each image block whose grayscale value is greater than a set grayscale threshold to all pixels in the image block; If the ratio exceeds a set ratio threshold, the image block is marked as a light spot, and all light spots are connected to form the light spot area.

3. The eye occlusion determination method according to claim 2, wherein: The step of acquiring a facial image and intercepting an eye region image includes: Acquire a facial image and locate facial key points, wherein the facial key points include eye key points; Calculating the tilt angle of the eyes according to the eye key points, and rotating and aligning the face image according to the tilt angle; An eye region image is cut out from the aligned face image according to the eye key points and a normalization operation is performed.

4. The eye occlusion determination method according to any one of claims 1 to 3, characterized in that: The convolutional neural network is trained by the following method: Constructing a training set, wherein the training set includes a plurality of training samples, wherein the training samples include positive samples and negative samples, wherein the positive samples are unobstructed eye images, and the negative samples are eye images obstructed by non-eye images; Performing data expansion on the training set, wherein the data expansion includes translating, rotating, modifying contrast, and / or adding noise to the training samples; The convolutional neural network is trained using the data augmented training set.

5. The eye occlusion determination method according to claim 4, wherein: The non-eye image includes a non-skin color image and a skin color image outside the eye area; The data expansion further includes performing small spot expansion on the training samples, and the small spot expansion includes: Randomly obtain training samples from the training set with a set probability and locate the eye area in the training samples; Map the eye area in RGB space to HSV space and reduce the value of brightness space H; Small white light spots are randomly added to the eye area, and the proportion of the small light spots in the eye area is less than the proportion threshold.

6. An eye occlusion determination device, characterized in that: The device comprises: An image acquisition module is used to acquire a face image and intercept an eye area image; a light spot detection module, configured to detect a light spot area on the eye area image and calculate a proportion of the light spot area in the eye area image; An occlusion judgment module is used to input the eye area image whose proportion does not reach the set proportion threshold into a pre-trained convolutional neural network to determine whether the eye is occluded; The occlusion judgment module includes: A first operation unit is configured to perform a first convolution operation, a first BN operation, a first ReLU operation, and a first pooling operation on the eye region image whose proportion does not reach a set proportion threshold, to obtain a first feature map; A second operation unit is configured to expand the edge of the first feature map by a number of pixels, and perform a second convolution operation, a second BN operation, a second ReLU operation, and a second pooling operation to obtain a second feature map; a third operation unit, configured to expand the edge of the second feature map by a number of pixels, and perform a third convolution operation, a third BN operation, a third ReLU operation, and a third pooling operation to obtain a third feature map; a fourth operation unit, configured to perform a first fully connected operation, a fourth ReLU operation, a first dropout operation, a second fully connected operation, a fifth ReLU operation, and a second dropout operation on the third feature map to obtain a fourth feature map; a classification unit, configured to perform a third fully connected operation with an output dimension of 2 on the fourth feature map, and perform classification using Softmax to obtain a probability that the eye region image is an eye and a probability that the eye region image is not an eye; The occlusion judgment unit is used to judge that the eye is not occluded if the probability that the eye area image is an eye exceeds a set occlusion threshold, otherwise it is judged that the eye is occluded.

7. A computer-readable storage medium for eye occlusion determination, characterized in that: The invention comprises a memory for storing processor-executable instructions, wherein when the instructions are executed by the processor, the steps of the eye occlusion determination method according to any one of claims 1 to 5 are implemented.

8. A device for eye occlusion judgment, characterized in that: The method comprises at least one processor and a memory storing computer-executable instructions, wherein when the processor executes the instructions, the steps of the eye occlusion determination method according to any one of claims 1 to 5 are implemented.

Images