Eye key point detection method, eye state detection method and related device

Abstract

The application provides an eye key point detection method, an eye state detection method and related devices, including: obtaining a face image of a to-be-detected object; inputting the face image into a pre-trained eye key point detection model, wherein the pre-trained eye key point detection model includes a backbone network, a first key point branch network and a second key point branch network; performing feature extraction on the eye region through the backbone network to obtain eye features; predicting a first key point through the first key point branch network according to the eye features, wherein the first key point is an eye key point of a predetermined part; and predicting a second key point through the second key point branch network according to the eye features and feature information corresponding to the first key point. The application can improve the accuracy of eye key point detection and reduce the false detection rate.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a method for detecting key points in the eye, a method for detecting eye state, and related devices. Background Technology

[0002] Accurately detecting the location of key points in a vehicle's driver fatigue detection system (DMS) is crucial for subsequent assessments of driver fatigue. However, due to the relatively low quality of in-cabin cameras and their susceptibility to external factors such as lighting and eye contact, false alarms and missed detections frequently occur during driver eye detection. Existing eye key point detection algorithms predict all key points in the eye simultaneously, which can lead to false alarms or missed detections for certain points, compromising the accuracy of eye key point detection. Summary of the Invention

[0003] This invention provides a method for detecting key points in the eye, a method for detecting eye state, and a related device to solve the above-mentioned problems.

[0004] This invention provides a method for detecting key points in the eye, comprising:

[0005] Obtain the facial image of the object to be detected;

[0006] The face image is input into a pre-trained eye keypoint detection model, wherein the pre-trained eye keypoint detection model includes a backbone network, a first keypoint branch network, and a second keypoint branch network.

[0007] The eye region is used to extract features through the backbone network to obtain eye features;

[0008] Based on the eye features, a first key point is predicted through the first key point branch network, and the first key point is the eye key point of the predetermined location.

[0009] Based on the eye features and the feature information corresponding to the first key point, the second key point is predicted through the second key point branch network.

[0010] According to the present invention, a method for detecting key points of the eye is provided, wherein the first key point is a key point of the eye at the corner of the eye;

[0011] The first keypoint branch network includes m first convolutional blocks, and the second keypoint branch network includes n second convolutional blocks, where m is less than n;

[0012] Accordingly, based on the eye features, and by predicting the first keypoint through the first keypoint branch network, the first keypoint is obtained, including:

[0013] The eye features are input into the first convolutional block, and the first key point feature information is extracted sequentially through m first convolutional blocks. The first key point is obtained based on the first key point feature information output by the mth first convolutional block.

[0014] Based on the eye features and the feature information corresponding to the first key point, the second key point is predicted through the second key point branch network, including:

[0015] The eye features are input into the second convolutional block, and the second key point feature information is extracted sequentially through the first m-1 second convolutional blocks out of n second convolutional blocks;

[0016] The second keypoint feature information output by the (m-1)th second convolutional block and the first keypoint feature information output by the mth first convolutional block are input into the mth second convolutional block. The second keypoint feature information is extracted sequentially through the remaining n-m+1 second convolutional blocks out of the n second convolutional blocks. The second keypoint is obtained based on the second keypoint feature information output by the nth second convolutional block.

[0017] According to the eye key point detection method provided by the present invention, the pre-trained eye key point detection model is trained in the following manner:

[0018] Obtain an image training set of the eye region and the corresponding eye key point labels;

[0019] The image training set is input into a pre-built eye key point detection model, and the backbone network is used to extract features from the image training set to obtain eye features for training.

[0020] Based on the eye features used for training, an initial first keypoint is predicted through the first keypoint branch network;

[0021] The first loss is obtained by calculating the loss based on the initial first key point and the first key point label in the eye key point label;

[0022] Based on the eye features used for training and the feature information corresponding to the initial first keypoint, the initial second keypoint is predicted through the second keypoint branch network.

[0023] The second loss is obtained by calculating the loss based on the initial second key point and the second key point label in the eye key point label;

[0024] The total loss is obtained based on the first loss and the second loss. The parameters of the pre-built eye keypoint detection model are updated based on the total loss until the pre-built eye keypoint detection model converges, so as to obtain the pre-trained eye keypoint detection model.

[0025] According to the eye key point detection method provided by the present invention, both the first loss and the second loss are calculated by the mean square error loss function.

[0026] The present invention also provides a method for detecting eye state, comprising:

[0027] Obtain a facial image of the object to be detected, and perform eye key point detection on the facial image based on any of the eye key point detection methods described above to obtain eye key points;

[0028] The state of the eyes of the subject to be tested is determined based on the key eye points.

[0029] According to the present invention, an eye state detection method is provided, wherein the eye state is the closed state of the eye and / or the orientation of the eye.

[0030] The present invention also provides an eye key point detection device, comprising:

[0031] The image acquisition module is used to acquire the face image of the object to be detected;

[0032] An image input module is used to input the face image into a pre-trained eye keypoint detection model, wherein the pre-trained eye keypoint detection model includes a backbone network, a first keypoint branch network, and a second keypoint branch network.

[0033] An eye feature extraction module is used to extract features from the eye region through the backbone network to obtain eye features;

[0034] The first key point prediction module is used to predict a first key point based on the eye features and through the first key point branch network. The first key point is an eye key point at a predetermined location.

[0035] The second keypoint prediction module is used to predict the second keypoint based on the eye features and the feature information corresponding to the first keypoint, and through the second keypoint branch network.

[0036] The present invention also provides an eye condition detection device, comprising:

[0037] The image acquisition and detection module is used to acquire the face image of the object to be detected, and to perform eye key point detection on the face image based on the above-mentioned eye key point detection device to obtain eye key points;

[0038] The state determination module is used to determine the state of the eyes of the object to be detected based on the key eye points.

[0039] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements any of the above-described methods for detecting key eye points or any of the above-described methods for detecting eye state.

[0040] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements any of the above-described methods for detecting key eye points or for detecting eye state.

[0041] The present invention provides a method for detecting key eye points, a method for detecting eye state, and related devices. The method for detecting key eye points involves extracting features from the eye region using a backbone network to obtain eye features; predicting a first key point using a first key point branch network based on the eye features; and predicting a second key point using a second key point branch network based on the eye features and the feature information corresponding to the first key point. By utilizing the first key point information, which has significant features and high determinism, to constrain the position of the second key point, the false detection rate is greatly reduced, and the accuracy of eye key point detection is improved. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0043] Figure 1 This is a flowchart illustrating the eye key point detection method provided in an embodiment of the present invention;

[0044] Figure 2 This is a schematic diagram of key eye points provided in an embodiment of the present invention;

[0045] Figure 3 This is a schematic diagram of the network structure of the eye key point detection model provided in an embodiment of the present invention;

[0046] Figure 4 This is a flowchart illustrating the eye state detection method provided in an embodiment of the present invention;

[0047] Figure 5 This is a schematic diagram of the eye key point detection device provided in an embodiment of the present invention;

[0048] Figure 6 This is a schematic diagram of the eye state detection device provided in an embodiment of the present invention;

[0049] Figure 7 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0051] Figure 1 This is a flowchart illustrating the eye key point detection method provided in an embodiment of the present invention; as shown below. Figure 1 As shown, the method for detecting key points in the eye includes:

[0052] S101, Obtain the face image of the object to be detected.

[0053] S102, the face image is input into the pre-trained eye key point detection model.

[0054] The pre-trained eye key point detection model includes a backbone network, a first key point branch network, and a second key point branch network.

[0055] S103, extract features from the eye region through the backbone network to obtain eye features;

[0056] S104, Based on the eye features, a first key point is predicted through the first key point branch network, and the first key point is the eye key point of the predetermined location.

[0057] S105, based on the eye features and the feature information corresponding to the first key point, the second key point is predicted through the second key point branch network.

[0058] Specifically, a diagram of key points around the eye is shown below. Figure 2 As shown, the key points of the eye include a first key point and a second key point. The first key point is a key point of the eye at a predetermined location, such as the key point at the corner of the eye (including the inner canthus near the bridge of the nose and the outer canthus near the temple). Figure 2 Key points numbered 37, 40, 43, and 46), and key points of the pupil (not shown in the figure).

[0059] Since the key point at the corner of the eye is the intersection of two arcs, it is more prominent than the key points on the two arcs at the base of the eyelashes and is considered a key point with high certainty. The key point corresponding to the pupil is relatively independent compared to other key points and is also considered a key point with high certainty. The first key point can also be a key point with high certainty in other locations, and this invention does not limit this.

[0060] The second key point (such as) Figure 2 Key points numbered 38, 39, 42, 41, 44, 45, 48, and 47 are prone to missed or false positives because the geometric information corresponding to their locations is less significant compared to the positions of the corner of the eye and the pupil. It should be noted that... Figure 2 This is merely an illustrative representation of the number and distribution of key points in the eye area; the actual number of key points depends on the specific circumstances, and this invention does not impose any limitations on this.

[0061] Having clarified the definitions of the first and second key points, the detection process of the eye key point detection model will now be introduced.

[0062] Specifically, the backbone network includes multiple convolutional blocks (the basic units of which include convolutional layers, activation functions, and pooling layers; the number and size of the layers are set according to the situation, and this invention does not limit this). In the backbone network, the output of the previous convolutional block is used as the input of the next convolutional block. The eye region in the face image is used as the input of the first convolutional block in the backbone network. Feature extraction is continuously performed through multiple convolutional blocks in the backbone network, and the output of the last convolutional block in the backbone network is used as the eye feature.

[0063] The first keypoint branch network also includes multiple convolutional blocks. The eye features are used as the input to the first convolutional block in the first keypoint branch network. The first keypoint branch network further extracts salient features through multiple convolutional blocks in the first keypoint branch network, and obtains the salient features output by the last convolutional block in the first keypoint branch network, which is the feature information corresponding to the first keypoint. The first keypoint is obtained based on the feature information corresponding to the first keypoint.

[0064] The second keypoint branch network also includes multiple convolutional blocks. Eye features are used as the input to the first convolutional block in the second keypoint branch network, and the feature information corresponding to the first keypoint is used as the input to a certain convolutional block in the second keypoint branch network. Feature extraction is performed through multiple convolutional blocks in the second keypoint branch network, and the second keypoint is determined based on the feature information output by the last convolutional block in the second keypoint branch network. Compared with the first keypoint, the second keypoint here is a keypoint whose features are not very obvious and is easily misdetected or missed.

[0065] In existing technologies, the detection of key eye points outputs both the first and second key points at once. Because this doesn't fully utilize the positional characteristics between key points, it easily leads to false positives and false negatives of key points in insignificant locations. This invention addresses this problem by using the positions of key points with significant and highly deterministic features to constrain the positions of the remaining key points. This overcomes the false positives and false negatives. Therefore, in the process of detecting the second key point, this invention integrates the feature information corresponding to the already detected first key point in addition to the eye features extracted by the backbone network, thereby constraining the position of the second key point and improving detection accuracy.

[0066] The eye keypoint detection method provided in this invention extracts features from the eye region using the backbone network to obtain eye features; based on the eye features, a first keypoint is predicted using a first keypoint branch network; and based on the eye features and the feature information corresponding to the first keypoint, a second keypoint is predicted using a second keypoint branch network. By utilizing the first keypoint information, which has significant features and high determinism, to constrain the position of the second keypoint, the false detection rate is greatly reduced, and the accuracy of eye keypoint detection is improved.

[0067] Furthermore, in this embodiment, eye key points are only divided into first key points and second key points, that is, salient key points and insignificant key points. In other embodiments of the present invention, eye key points can also be divided into first key points, second key points, and third key points, that is, highly salient key points, less salient key points, and insignificant key points. Correspondingly, the eye key point detection model is designed as a backbone network, a first key point branch network, a second key point branch network, and a third key point branch network. The backbone network extracts eye features, the first key point branch network predicts the first key point, the second key point branch network predicts the second key point based on the eye features and the feature information corresponding to the first key point, and the third key point branch network predicts the third key point based on the eye features, the feature information corresponding to the first key point, and the feature information of the second key point. Alternatively, eye key points can be divided into more categories, and the eye key point detection model can be adjusted accordingly to achieve the purpose of constraining the positions of other key points through highly deterministic key point position information. The present invention does not limit this.

[0068] In addition, after detecting all key points around the eyes, facial recognition can be achieved based on these key points, along with key points on other parts of the face, thus improving recognition accuracy. In vehicle driver fatigue detection systems, key points around the eyes can determine whether the driver's eyes are open or closed, and key points at the pupil position can determine the direction of the driver's attention, thereby assessing the driver's driving state.

[0069] Furthermore, the first key point is the key point of the eye at the corner of the eye.

[0070] In this embodiment, through analysis of the eye area, it was found that the geometric information of the corner of the eye is more prominent than that of other locations, so the first key point is determined as the key point of the corner of the eye.

[0071] The first keypoint branch network includes m first convolutional blocks, and the second keypoint branch network includes n second convolutional blocks, where m is less than n, meaning that the first keypoints can be determined first. After the first keypoints are determined, the feature information corresponding to the first keypoints is input into the second keypoint branch network.

[0072] Accordingly, based on the eye features, and by predicting the first keypoint through the first keypoint branch network, the first keypoint is obtained, including:

[0073] The eye features are input into the first convolutional block, and the first key point feature information is extracted sequentially through m first convolutional blocks. The first key point is obtained based on the first key point feature information output by the mth first convolutional block.

[0074] Based on the eye features and the feature information corresponding to the first key point, the second key point is predicted through the second key point branch network, including:

[0075] The eye features are input into the second convolutional block, and the second key point feature information is extracted sequentially through the first m-1 second convolutional blocks out of n second convolutional blocks;

[0076] The second keypoint feature information output by the (m-1)th second convolutional block and the first keypoint feature information output by the mth first convolutional block are input into the mth second convolutional block. The second keypoint feature information is extracted sequentially through the remaining n-m+1 second convolutional blocks out of the n second convolutional blocks. The second keypoint is obtained based on the second keypoint feature information output by the nth second convolutional block.

[0077] In this embodiment, two first convolutional blocks are set in the first keypoint branch network, and three second convolutional blocks are set in the second keypoint branch network, that is, m=2 and n=3. The specific network structure is as follows. Figure 3 As shown.

[0078] exist Figure 3 In this method, the eye portion of the face image is used as the input of the backbone network. After continuous feature extraction through multiple convolutional blocks in the backbone network, the eye features are obtained. The eye features are then used as the input of the first keypoint branch network and the second keypoint branch network.

[0079] In the first keypoint branch network, the eye features are continuously extracted through two first convolutional blocks to obtain the feature information corresponding to the first keypoint (that is, the corner of the eye keypoint), thereby determining the first keypoint, which is to obtain the position information of the corner of the eye keypoint.

[0080] In the second keypoint branch network, the eye features are first extracted through the first second convolutional block. The feature information extracted by the first second convolutional block is used as the input of the second second convolutional block. At the same time, the feature information output by the second first convolutional block is also used as the input of the second second convolutional block. The second second convolutional block calculates the feature information corresponding to the first keypoint and the feature information output by the previous second convolutional block to achieve the purpose of using the positional features of the first keypoint to constrain the position of the remaining keypoints. The feature information corresponding to the second keypoint is extracted through the remaining two second convolutions. The second keypoint is determined based on the feature information, that is, the position of keypoints where other eye features are not significant is obtained.

[0081] The eye keypoint detection model with the above network structure was tested, and the false detection rate was reduced by 4% and the accuracy was improved by 3%.

[0082] It should be noted that in this embodiment, the first keypoint branch network is set to include two first convolutional blocks, and the second keypoint branch network is set to include three second convolutional blocks. In other embodiments of the present invention, the values ​​of n and m can be set with the optimization of model performance as the guide, and the present invention does not limit them.

[0083] The eye key point detection method provided in this embodiment of the invention predicts the second key point based on the output of the backbone network and the output of the first key point branch network. It constrains the position of the remaining key points with key point position feature information with high determinism, thereby reducing the false detection rate and improving the accuracy of eye key point detection.

[0084] Furthermore, the pre-trained eye keypoint detection model is trained in the following manner:

[0085] Obtain an image training set of the eye region and the corresponding eye key point labels.

[0086] The image training set is input into a pre-built eye key point detection model, and the backbone network is used to extract features from the image training set to obtain eye features for training.

[0087] Based on the eye features used for training, the initial first keypoint is predicted through the first keypoint branch network.

[0088] The first loss is obtained by calculating the loss based on the initial first keypoint and the first keypoint label in the eye keypoint label.

[0089] Based on the eye features used for training and the feature information corresponding to the initial first keypoint, the initial second keypoint is predicted through the second keypoint branch network.

[0090] The second loss is obtained by calculating the loss based on the initial second keypoint and the second keypoint label in the eye keypoint label.

[0091] The total loss is obtained based on the first loss and the second loss. The parameters of the pre-built eye keypoint detection model are updated based on the total loss until the pre-built eye keypoint detection model converges, so as to obtain the pre-trained eye keypoint detection model.

[0092] Specifically, we first obtain an image training set related to the eye region and the corresponding eye key point labels, and then construct an eye key point detection model.

[0093] Then, the image training set is input into the constructed eye keypoint detection model for forward propagation. The backbone network extracts training eye features from the image training set, the first keypoint branch network predicts the initial first keypoint based on the training eye features, and the second keypoint branch network predicts the initial second keypoint based on the training eye features and the initial first keypoint.

[0094] Subsequently, the loss value between the initial first keypoint and its corresponding eye keypoint label is calculated, and the result is taken as the first loss (loss1). The loss value between the initial second keypoint and its corresponding eye keypoint label is calculated, and the result is taken as the second loss (loss2). The sum of the first and second losses is taken as the total loss. 总 loss 总 = loss1 + loss2. Based on the total loss... 总 The eye keypoint detection model is continuously updated using the backpropagation algorithm until it converges and stops updating, thus obtaining a trained eye keypoint detection model.

[0095] Both the first loss and the second loss mentioned above are calculated using the mean squared error loss function.

[0096] Furthermore, the training process of the above model can be carried out on the same terminal as the model application, or it can be ported to another terminal after the model training is completed. This invention does not limit this.

[0097] The eye keypoint detection method provided in this embodiment of the invention obtains a total loss based on a first loss corresponding to a first keypoint branch network and a second loss corresponding to a second keypoint branch network, and iteratively updates the model based on the total loss to obtain a finally trained eye keypoint detection model for eye keypoint detection.

[0098] Figure 4 This is a schematic flowchart of the eye state detection method provided in an embodiment of the present invention; as shown below. Figure 4 As shown, the eye state detection method includes:

[0099] S401, Obtain the face image of the object to be detected, and perform eye key point detection on the face image based on any of the above-described eye key point detection methods to obtain eye key points.

[0100] The facial image can be obtained directly by shooting or extracted from video; this invention does not limit the specific method used.

[0101] S402, determine the state of the eyes of the object to be tested based on the key eye points.

[0102] In this embodiment, the application scenario is set in a vehicle driver fatigue detection system. Facial images are acquired from monitoring videos inside the vehicle cabin, and the aforementioned eye key point detection method is used to detect all key points in the eye region of the facial image. Based on the positional information of all key points, the corresponding eye state to be detected is determined. The eye state can be either closed or facing in any direction.

[0103] The eye closure state is determined by calculating the distance between the upper and lower eyelids based on key points at the base of the eyelashes. If this distance is less than a preset distance threshold, the subject's eyes are determined to be closed; if the distance exceeds the preset threshold, the subject's eyes are determined to be open. The subject's fatigue level is determined by observing the eye state in consecutive video frames.

[0104] Determining the direction of the eyes involves identifying the position of the pupil within the eye socket based on key points at the pupil's location and the base of the eyelashes. This position is then used to determine which direction the subject is focusing on, thus assessing their driving status.

[0105] It should be noted that, in addition to being applied to vehicle driver fatigue detection systems, this invention can also be applied to scenarios where there are extremely high requirements for the working condition of employees. The above-mentioned eye condition detection method can accurately assess the fatigue state of employees, thereby avoiding accidents caused by fatigue.

[0106] The eye state detection method provided in this embodiment of the invention accurately detects all key eye points using the aforementioned key eye point detection method, determines the eye state of the object to be detected based on the positional relationship of all key eye points, and then obtains the working state of the object to be detected, so as to prevent accidents caused by fatigue.

[0107] The eye key point detection device provided by the present invention will be described below. The eye key point detection device described below can be referred to in correspondence with the eye key point detection method described above.

[0108] Figure 5 This is a schematic diagram of the eye key point detection device provided in an embodiment of the present invention, as shown below. Figure 5 As shown, the eye key point detection device includes an image acquisition module 501, an image input module 502, an eye feature extraction module 503, a first key point prediction module 504, and a second key point prediction module 505.

[0109] Image acquisition module 501 is used to acquire the face image of the object to be detected.

[0110] The image input module 502 is used to input the face image into a pre-trained eye keypoint detection model, wherein the pre-trained eye keypoint detection model includes a backbone network, a first keypoint branch network, and a second keypoint branch network.

[0111] The eye feature extraction module 503 is used to extract features from the eye region through the backbone network to obtain eye features.

[0112] The first key point prediction module 504 is used to predict a first key point based on the eye features and through the first key point branch network, wherein the first key point is an eye key point of a predetermined location.

[0113] The second keypoint prediction module 505 is used to predict the second keypoint based on the eye features and the feature information corresponding to the first keypoint, and through the second keypoint branch network.

[0114] Specifically, the key points of the eye include a first key point and a second key point. The first key point is a key point of the eye at a predetermined location, such as the key point at the corner of the eye (including the inner canthus near the bridge of the nose and the outer canthus near the temple) and the key point of the pupil. Since the key point at the corner of the eye is the intersection of two arcs, it is more prominent than the key points on the two arcs at the root of the eyelashes and is considered a key point with high certainty. The key point corresponding to the pupil is relatively independent compared to other key points and is also considered a key point with high certainty. The first key point can also be a key point with high certainty in other locations, which is not limited in this invention.

[0115] Having clarified the definitions of the first and second key points, the specific functions of the eye feature extraction module 503, the first key point prediction module 504, and the second key point prediction module 505 are described below.

[0116] Specifically, the backbone network includes multiple convolutional blocks (the basic units of which include convolutional layers, activation functions, and pooling layers; the number and size of the layers are set according to the situation, and this invention does not limit this). In the backbone network, the output of the previous convolutional block is used as the input of the next convolutional block. The eye region in the face image is used as the input of the first convolutional block in the backbone network. Feature extraction is continuously performed through multiple convolutional blocks in the backbone network, and the output of the last convolutional block in the backbone network is used as the eye feature.

[0117] The first keypoint branch network also includes multiple convolutional blocks. The eye features are used as the input to the first convolutional block in the first keypoint branch network. The first keypoint branch network further extracts salient features through multiple convolutional blocks in the first keypoint branch network, and obtains the salient features output by the last convolutional block in the first keypoint branch network, which is the feature information corresponding to the first keypoint. The first keypoint is obtained based on the feature information corresponding to the first keypoint.

[0118] The second keypoint branch network also includes multiple convolutional blocks. Eye features are used as the input to the first convolutional block in the second keypoint branch network, and the feature information corresponding to the first keypoint is used as the input to a certain convolutional block in the second keypoint branch network. Feature extraction is performed through multiple convolutional blocks in the second keypoint branch network, and the second keypoint is determined based on the feature information output by the last convolutional block in the second keypoint branch network. Compared with the first keypoint, the second keypoint here is a keypoint whose features are not very obvious and is easily misdetected or missed.

[0119] In existing technologies, the detection of key eye points outputs both the first and second key points at once. Because this doesn't fully utilize the positional characteristics between key points, it easily leads to false positives and false negatives of key points in insignificant locations. This invention addresses this problem by using the positions of key points with significant and highly deterministic features to constrain the positions of the remaining key points. This overcomes the false positives and false negatives. Therefore, in the process of detecting the second key point, this invention integrates the feature information corresponding to the already detected first key point in addition to the eye features extracted by the backbone network, thereby constraining the position of the second key point and improving detection accuracy.

[0120] The eye keypoint detection device provided in this invention extracts features from the eye region using the backbone network to obtain eye features; based on the eye features, a first keypoint is predicted using a first keypoint branch network; and based on the eye features and the feature information corresponding to the first keypoint, a second keypoint is predicted using a second keypoint branch network. By utilizing the first keypoint information, which has significant features and high determinism, to constrain the position of the second keypoint, the false detection rate is greatly reduced, and the accuracy of eye keypoint detection is improved.

[0121] The eye state detection device provided by the present invention is described below. The eye state detection device described below can be referred to in correspondence with the eye state detection method described above.

[0122] Figure 6 This is a schematic diagram of the eye state detection device provided in an embodiment of the present invention, as shown below. Figure 6 As shown, the eye state detection device includes an image acquisition and detection module 601 and a state determination module 602.

[0123] The image acquisition and detection module 601 is used to acquire a face image of the object to be detected, and to perform eye key point detection on the face image based on the aforementioned eye key point detection device to obtain eye key points.

[0124] The state determination module 602 is used to determine the state of the eyes of the object to be detected based on the key eye points.

[0125] In this embodiment, the application scenario is set in a vehicle driver fatigue detection system. Facial images are acquired from monitoring videos inside the vehicle cabin, and the aforementioned eye key point detection method is used to detect all key points in the eye region of the facial image. Based on the positional information of all key points, the corresponding eye state to be detected is determined. The eye state can be either closed or facing in any direction.

[0126] The eye closure state is determined by calculating the distance between the upper and lower eyelids based on key points at the base of the eyelashes. If this distance is less than a preset distance threshold, the subject's eyes are determined to be closed; if the distance exceeds the preset threshold, the subject's eyes are determined to be open. The subject's fatigue level is determined by observing the eye state in consecutive video frames.

[0127] Determining the direction of the eyes involves identifying the position of the pupil within the eye socket based on key points at the pupil's location and the base of the eyelashes. This position is then used to determine which direction the subject is focusing on, thus assessing their driving status.

[0128] It should be noted that, in addition to being applied to vehicle driver fatigue detection systems, this invention can also be applied to scenarios where there are extremely high requirements for the working condition of employees. The above-mentioned eye condition detection method can accurately assess the fatigue state of employees, thereby avoiding accidents caused by fatigue.

[0129] The eye state detection device provided in this embodiment of the invention accurately detects all key eye points using the aforementioned key eye point detection method, determines the eye state of the object to be detected based on the positional relationship of all key eye points, and then obtains the working state of the object to be detected, so as to prevent accidents caused by fatigue.

[0130] Figure 7 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention, such as... Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other through the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute an eye keypoint detection method. The eye keypoint detection method includes: acquiring a face image of the object to be detected; inputting the face image into a pre-trained eye keypoint detection model, wherein the pre-trained eye keypoint detection model includes a backbone network, a first keypoint branch network, and a second keypoint branch network; extracting features from the eye region through the backbone network to obtain eye features; predicting a first keypoint based on the eye features and through the first keypoint branch network, wherein the first keypoint is an eye keypoint of a predetermined location; and predicting a second keypoint based on the eye features and the feature information corresponding to the first keypoint and through the second keypoint branch network.

[0131] Alternatively, it can be used to perform an eye state detection method, which includes: acquiring a face image of a subject to be detected; performing eye key point detection on the face image based on any of the above-described eye key point detection methods to obtain eye key points; and determining the eye state of the subject to be detected based on the eye key points.

[0132] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0133] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the eye keypoint detection method provided by the above-described methods. The eye keypoint detection method includes: acquiring a facial image of a subject to be detected; inputting the facial image into a pre-trained eye keypoint detection model, wherein the pre-trained eye keypoint detection model includes a backbone network, a first keypoint branch network, and a second keypoint branch network; extracting features from the eye region through the backbone network to obtain eye features; predicting a first keypoint based on the eye features and through the first keypoint branch network, wherein the first keypoint is an eye keypoint at a predetermined location; and predicting a second keypoint based on the eye features and feature information corresponding to the first keypoint through the second keypoint branch network.

[0134] Alternatively, it can be used to perform an eye state detection method, which includes: acquiring a face image of a subject to be detected; performing eye key point detection on the face image based on any of the above-described eye key point detection methods to obtain eye key points; and determining the eye state of the subject to be detected based on the eye key points.

[0135] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0136] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0137] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting key points in the eye, characterized in that, include: Obtain the facial image of the object to be detected; The face image is input into a pre-trained eye keypoint detection model, wherein the pre-trained eye keypoint detection model includes a backbone network, a first keypoint branch network, and a second keypoint branch network; the first keypoint branch network includes m first convolutional blocks, and the second keypoint branch network includes n second convolutional blocks, where m is less than n. The eye region is used to extract features through the backbone network to obtain eye features; Based on the eye features, a first key point is predicted through the first key point branch network, and the first key point is the eye key point of the predetermined location. Based on the eye features and the feature information corresponding to the first key point, and predicting the second key point through the second key point branch network, the process includes: inputting the eye features into the second convolutional block, extracting the second key point feature information sequentially through the first m-1 second convolutional blocks out of n second convolutional blocks; inputting the second key point feature information output by the m-1 second convolutional block and the first key point feature information output by the m first convolutional block into the m second convolutional block, extracting the second key point feature information sequentially through the remaining n-m+1 second convolutional blocks out of n second convolutional blocks, and obtaining the second key point based on the second key point feature information output by the nth second convolutional block; The first key point is a key point that is more significant than the second key point.

2. The method for detecting key eye points according to claim 1, characterized in that, The first key point is the key point of the eye at the corner of the eye; Accordingly, based on the eye features, and by predicting the first keypoint through the first keypoint branch network, the first keypoint is obtained, including: The eye features are input into the first convolutional block, and the first key point feature information is extracted sequentially through m first convolutional blocks. The first key point is obtained based on the first key point feature information output by the mth first convolutional block.

3. The method for detecting key eye points according to claim 1, characterized in that, The pre-trained eye key point detection model was trained in the following way: Obtain an image training set of the eye region and the corresponding eye key point labels; The image training set is input into a pre-built eye key point detection model, and the backbone network is used to extract features from the image training set to obtain eye features for training. Based on the eye features used for training, an initial first keypoint is predicted through a first keypoint branch network. The first loss is obtained by calculating the loss based on the initial first key point and the first key point label in the eye key point label; Based on the eye features used for training and the feature information corresponding to the initial first key point, the initial second key point is predicted through the second key point branch network. The second loss is obtained by calculating the loss based on the initial second keypoint and the second keypoint label in the eye keypoint label; The total loss is obtained based on the first loss and the second loss. The parameters of the pre-built eye keypoint detection model are updated based on the total loss until the pre-built eye keypoint detection model converges, so as to obtain the pre-trained eye keypoint detection model.

4. The method for detecting key eye points according to claim 3, characterized in that, Both the first loss and the second loss are calculated using the mean squared error loss function.

5. A method for detecting eye condition, characterized in that, include: Obtain a facial image of the object to be detected, and perform eye key point detection on the facial image based on the eye key point detection method as described in any one of claims 1-4 to obtain eye key points; The state of the eyes of the subject to be tested is determined based on the key eye points.

6. The eye state detection method according to claim 5, characterized in that, The state of the eyes refers to the closed state of the eyes and / or the direction the eyes are facing.

7. A device for detecting key points of the eye, characterized in that, include: The image acquisition module is used to acquire the face image of the object to be detected; An image input module is used to input the face image into a pre-trained eye keypoint detection model, wherein the pre-trained eye keypoint detection model includes a backbone network, a first keypoint branch network, and a second keypoint branch network; the first keypoint branch network includes m first convolutional blocks, and the second keypoint branch network includes n second convolutional blocks, where m is less than n; An eye feature extraction module is used to extract features from the eye region through the backbone network to obtain eye features; The first key point prediction module is used to predict a first key point based on the eye features and through the first key point branch network. The first key point is an eye key point at a predetermined location. The second keypoint prediction module is used to predict a second keypoint based on the eye features and the feature information corresponding to the first keypoint, and through the second keypoint branch network. The module includes: inputting the eye features into the second convolutional block; sequentially extracting second keypoint feature information through the first m-1 second convolutional blocks out of n second convolutional blocks; inputting the second keypoint feature information output by the m-1 second convolutional block and the first keypoint feature information output by the m-th first convolutional block into the m-th second convolutional block; sequentially extracting second keypoint feature information through the remaining n-m+1 second convolutional blocks out of n second convolutional blocks; and obtaining the second keypoint based on the second keypoint feature information output by the n-th second convolutional block. The first key point is a key point that is more significant than the second key point.

8. An eye condition detection device, characterized in that, include: The image acquisition and detection module is used to acquire a face image of the object to be detected, and to perform eye key point detection on the face image based on the eye key point detection device as described in claim 7, so as to obtain eye key points; The state determination module is used to determine the state of the eyes of the object to be detected based on the key eye points.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the eye key point detection method as described in any one of claims 1 to 4 or the eye state detection method as described in any one of claims 5 to 6.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the eye key point detection method as described in any one of claims 1 to 4 or the eye state detection method as described in any one of claims 5 to 6.

Images