Method and apparatus for detecting pilot fatigue

Abstract

The application relates to a pilot fatigue detection method and device, the method comprising: acquiring images of a pilot in a preset time length collected by a camera; for any image, performing human eye key point detection on the image through a human eye detection network to obtain a first eye feature corresponding to the image; determining whether the image embodies fatigue features according to the first eye feature corresponding to the image, a maximum value and a minimum value of an eye length-width ratio of the pilot; and determining a fatigue detection result of the pilot according to a first number of images embodying fatigue features, a second number of images in the preset time length and a preset fatigue threshold value. The embodiment of the application can effectively improve the efficiency and accuracy of pilot fatigue detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of aviation safety, and in particular to a pilot fatigue detection method and device. BACKGROUND

[0002] Pilot fatigue has become the biggest hidden danger threatening aviation safety. In the field of military flight, with high-intensity tasks such as supersonic penetration, high-g maneuvering and long-time flight, the physical and mental consumption of pilots is intensified, which easily leads to dangerous states such as inattention and slow reaction, and seriously restricts the exertion of combat effectiveness. In the field of civil aviation, trans-meridian long-haul flights and long-range flights also lead to pilot fatigue, thereby affecting flight safety.

[0003] In related technologies, fatigue detection is mainly divided into two categories: subjective detection and objective detection. Subjective detection is performed by subjective scoring through standardized scales (such as Stanford Sleep Scale, Piersol Fatigue Scale, etc.), which has the advantage of simple operation, but its results are easily affected by subjective factors of the measured person, and have limitations such as feedback lag. Objective detection is usually based on physiological signals such as electroencephalogram signals, electromyogram signals and galvanic skin response signals of the measured person. For example, the patent application CN108272463A proposes a pilot fatigue detection method based on electroencephalogram signals, and the patent application CN117407743A proposes a pilot fatigue detection method based on artificial intelligence, which uses heart rate data and pulse signal data to evaluate the fatigue level of pilots. Although objective detection has objectivity and timeliness, it needs to rely on complex detection equipment and has invasiveness (i.e. the need to install equipment on the pilot's body), which has obvious applicability limitations in the harsh environment of the onboard cabin. Therefore, there is an urgent need for a new method for detecting pilot fatigue. SUMMARY

[0004] In view of this, a pilot fatigue detection method and device are proposed.

[0005] In a first aspect, an embodiment of the present application provides a pilot fatigue detection method, which comprises: acquiring images of a pilot within a preset time length collected by a camera; for any image, performing eye key point detection on the image through an eye detection network to obtain a first eye feature corresponding to the image; determining whether the image embodies fatigue features according to the first eye feature corresponding to the image, a maximum value and a minimum value of an eye aspect ratio of the pilot; and determining a fatigue detection result of the pilot according to a first number of images embodying fatigue features, a second number of images within the preset time length and a preset fatigue threshold.

[0006] In some possible implementation manners, the human eye detection network comprises a backbone network, a neck network and a head network, the backbone network is configured to perform feature extraction on an input image to obtain first intermediate features, the neck network is configured to perform feature fusion on the first intermediate features to obtain second intermediate features, and the head network is configured to perform feature detection on the second intermediate features to obtain first eye features, the first eye features comprise a plurality of feature vectors, each feature vector comprises position information of a detection frame, a confidence of the detection frame, position information of 12 eye key points and a category of the detection frame, and each feature vector has a dimension of 30.

[0007] In some possible implementation manners, the determining whether the image represents a fatigue feature according to the first eye features corresponding to the image, the maximum value and the minimum value of the eye length-width ratio of the pilot comprises: determining an eye length-width ratio threshold of the pilot according to the maximum value and the minimum value of the eye length-width ratio of the pilot; determining position information of eye key points of the pilot in the image according to the first eye features corresponding to the image; determining an eye length-width ratio of the pilot in the image according to the position information of the eye key points; and determining whether the image represents a fatigue feature according to the eye length-width ratio of the pilot in the image and the eye length-width ratio threshold of the pilot.

[0008] In some possible implementation manners, the determining the eye length-width ratio of the pilot in the image according to the position information of the eye key points comprises: determining a left eye length-width ratio according to the position information of the key points of the left eye of the pilot in the image; determining a right eye length-width ratio according to the position information of the key points of the right eye of the pilot in the image; and determining the average value of the left eye length-width ratio and the right eye length-width ratio as the eye length-width ratio of the pilot in the image.

[0009] In some possible implementation manners, the determining whether the image represents a fatigue feature according to the eye length-width ratio of the pilot in the image and the eye length-width ratio threshold of the pilot comprises: determining that the image represents a fatigue feature in a case where the eye length-width ratio of the pilot in the image is less than the eye length-width ratio threshold of the pilot.

[0010] In some possible implementation manners, the determining the fatigue detection result of the pilot according to the first quantity of images representing a fatigue feature, the second quantity of images in a preset time length and a preset fatigue threshold comprises: determining a proportion of images representing a fatigue feature in the images in the preset time length according to the first quantity of images representing a fatigue feature and the second quantity of images in the preset time length; and determining that the fatigue detection result of the pilot is in a fatigue state in a case where the proportion is greater than the preset fatigue threshold.

[0011] In some possible implementation manners, the method further includes: acquiring a preset number of calibration images of continuous blinking of the pilot collected by the camera; and determining a maximum value and a minimum value of the eye aspect ratio of the pilot according to the calibration images.

[0012] In some possible implementation manners, the determining the maximum value and the minimum value of the eye aspect ratio of the pilot according to the calibration images includes: for any calibration image, performing human eye key point detection on the calibration image by using the human eye detection network to obtain a second eye feature corresponding to the calibration image; determining the eye aspect ratio of the pilot in the calibration image according to the second eye feature; and determining the maximum value and the minimum value of the eye aspect ratio of the pilot according to the eye aspect ratio of the pilot in each calibration image.

[0013] In some possible implementation manners, the human eye detection network is deployed on an embedded chip, parameters of the human eye detection network are of a floating point type, and the method further includes: quantizing floating point parameters and operations in the backbone network and the neck network into fixed point parameters and operations, and deploying the quantized backbone network and the neck network to the embedded chip; and directly deploying the head network to the embedded chip.

[0014] In a second aspect, an embodiment of the present application provides a pilot fatigue detection device, including a camera and an embedded chip; the camera is configured to collect images of a pilot; the embedded chip stores one or more instructions, and the one or more instructions are executed by the embedded chip to implement any one of the pilot fatigue detection methods in the first aspect.

[0015] The pilot fatigue detection method provided by the embodiment of the present application can perform human eye key point detection on the images of the pilot collected by the camera by using the human eye detection network to obtain first eye features corresponding to the images, then determine whether the images embody fatigue features according to the first eye features and the maximum value and the minimum value of the eye aspect ratio of the pilot, and finally determine a fatigue detection result of the pilot according to a first number of images embodying fatigue features, a second number of images in a preset time period, and a preset fatigue threshold. The embodiment of the present application combines human eye key points and eye aspect ratios to detect fatigue of the pilot, which not only simplifies the device and is non-invasive, but also effectively overcomes the problem of face occlusion of the pilot and is less affected by environmental factors such as light, thereby effectively improving the efficiency and accuracy of pilot fatigue detection.

[0016] These and other aspects of the present application will become more fully understood from the following description of the (multiple) embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate examples of the present application, and together with the description, serve to explain the principles of the present application.

[0018] Figure 1 A schematic diagram showing an application scenario of the pilot fatigue detection method according to an embodiment of the present application.

[0019] Figure 2 A structural schematic diagram of the human eye detection network according to an embodiment of the present application is shown.

[0020] Figure 3 A structural schematic diagram of the backbone start module according to an embodiment of the present application is shown.

[0021] Figure 4 A structural schematic diagram of the cross-stage partial three-convolution bottleneck module according to an embodiment of the present application is shown.

[0022] Figure 5 A structural schematic diagram of the bottleneck module according to an embodiment of the present application is shown.

[0023] Figure 6 A structural schematic diagram of the spatial pyramid pooling module according to an embodiment of the present application is shown.

[0024] Figure 7 A schematic diagram of the output vector of the human eye detection network according to an embodiment of the present application is shown.

[0025] Figure 8 A schematic diagram of the data label according to an embodiment of the present application is shown.

[0026] Figure 9 A flowchart of the pilot fatigue detection method according to an embodiment of the present application is shown.

[0027] Figure 10 A position schematic diagram of the eye key point according to an embodiment of the present application is shown.

[0028] Figure 11 A schematic diagram of the processing process of the pilot fatigue detection method according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0029] Various exemplary embodiments, features and aspects of the present application will be described in detail below with reference to the accompanying drawings. The same reference numbers in the drawings represent functionally the same or similar elements. Although various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

[0030] The term "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

[0031] In addition, for the purpose of better illustrating the present application, numerous specific details are set forth in the following detailed description. One skilled in the art will appreciate that the present application can be practiced without the specific details, and that the present application is capable of many implementations. In some instances, well-known methods, structures, elements, and circuits have not been described in detail in order to avoid obscuring the present application.

[0032] Since the fatigue detection method based on facial features has the characteristics of non-intrusiveness, and only needs to be realized through a simple image acquisition device, the fatigue detection method based on facial features shows better applicability in complex environments. Among many facial features, the eye movement feature has the most intuitive and close relationship with the degree of fatigue. In the airborne environment, especially in the military flight scene, the facial area of the pilot except the eye is often blocked by the oxygen mask and other devices, which leads to the fact that the related features cannot be obtained. Therefore, the eye movement feature becomes the most suitable facial feature for detecting the fatigue state of the pilot.

[0033] At present, the fatigue detection method based on eye movement features is mainly applied in the field of fatigue detection of automobile drivers. For example, the fatigue detection method based on face detection and eye state recognition proposed in the invention patent application CN106250801A mainly locates the eye area by performing binaryzation processing on the image. However, this method has high requirements for the image, and is easily affected by environmental factors such as light, especially in the airborne environment where the light condition may change extremely, its effectiveness is significantly reduced.

[0034] In view of this, the present application proposes a pilot fatigue detection method, comprising: acquiring images of a pilot collected by a camera within a preset time length; for any image, detecting eye key points of the image through an eye detection network to obtain a first eye feature corresponding to the image; determining whether the image embodies fatigue features according to the first eye feature corresponding to the image, a maximum value and a minimum value of an eye aspect ratio of the pilot; and determining a fatigue detection result of the pilot according to a first number of images embodying fatigue features, a second number of images within the preset time length, and a preset fatigue threshold.

[0035] The pilot fatigue detection method of the embodiment of the present application can detect the image of the pilot collected by the camera through the human eye detection network to obtain the first eye feature corresponding to the image, and then determine whether the image reflects the fatigue feature according to the first eye feature and the maximum and minimum values of the eye aspect ratio of the pilot. Then, the fatigue detection result of the pilot is determined according to the first number of the images reflecting the fatigue feature, the second number of the images in the preset time length, and the preset fatigue threshold. The embodiment of the present application combines the human eye key point and the eye aspect ratio to detect the fatigue of the pilot. The equipment is simple, non-invasive, can effectively overcome the problem of face shielding of the pilot, is not easily affected by environmental factors such as light, and can effectively improve the efficiency and accuracy of the pilot fatigue detection.

[0036] The pilot fatigue detection method of the embodiment of the present application can be applied to the scene of detecting the fatigue of the pilot in the fields of civil aviation, general aviation, military aviation, etc. Through the pilot fatigue detection method of the embodiment of the present application, the fatigue state of the pilot can be detected in real time, so that effective measures can be taken in time to avoid risks in advance when fatigue driving occurs.

[0037] The pilot fatigue detection method of the embodiment of the present application can be applied to an embedded chip (also known as a system on chip, SoC for short) to meet the embedded application of the airborne environment.

[0038] Figure 1 A schematic diagram showing the application scene of the pilot fatigue detection method according to an embodiment of the present application is shown. As shown in Figure 1 The pilot fatigue detection method of the embodiment of the present application is applied to an embedded chip 110, such as a domestic embedded chip RK3588, etc. The specific type of the embedded chip 110 is not limited in the present application.

[0039] The camera 120 is connected with the embedded chip 110 and is used to collect the image of the pilot. The image includes the face of the pilot. The camera 120 can be an infrared camera, a normal camera, a high-definition camera, etc. The specific type of the camera is not limited in the present application.

[0040] The embedded chip 110 is used to obtain the image of the pilot collected by the camera 120 in a preset time length. For any image, the human eye key point detection is performed on the image through the human eye detection network to obtain the first eye feature corresponding to the image, and whether the image reflects the fatigue feature is determined according to the first eye feature corresponding to the image, the maximum and minimum values of the eye aspect ratio of the pilot. The fatigue detection result of the pilot is determined according to the first number of the images reflecting the fatigue feature, the second number of the images in the preset time length, and the preset fatigue threshold.

[0041] In some possible implementation manners, the human eye detection network comprises a stem network, a neck network, and a head network. The stem network is configured to perform feature extraction on an input image to obtain first intermediate features; the neck network is configured to perform feature fusion on the first intermediate features to obtain second intermediate features; and the head network is configured to perform feature detection on the second intermediate features to obtain first eye features.

[0042] Figure 2 A structure diagram of a human eye detection network according to an embodiment of the present application is shown. As shown in the figure, Figure 2 The human eye detection network comprises a stem network 220, a neck network 230, and a head network 240. After an image 210 of a pilot captured by a camera is input into the human eye detection network, the image 210 is sequentially processed by the stem network 220, the neck network 230, and the head network 240, and first eye features 250 corresponding to the image 210 are obtained.

[0043] The stem network 220 comprises a stem start module (Stem) 2201, a cross-stage partial three convolution bottleneck module (C3) 2202, a convolution module (Conv) 2203, a cross-stage partial three convolution bottleneck module (C3) 2204, a convolution module (Conv) 2205, a cross-stage partial three convolution bottleneck module (C3) 2206, a convolution module (Conv) 2207, a spatial pyramid pooling module (SSP) 2208, and a cross-stage partial three convolution bottleneck module (C3) 2209.

[0044] The image 210 of the pilot is input into the stem network 220 and sequentially processed by the above modules, and first intermediate features output by the stem network 220 are obtained. The first intermediate features comprise features output by the cross-stage partial three convolution bottleneck module 2204, the cross-stage partial three convolution bottleneck module 2206, and the cross-stage partial three convolution bottleneck module 2209.

[0045] The neck network 230 comprises a convolution module (Conv) 2301, an up-sampling module (Upsample) 2302, a concatenation processing (Concat) 2303, a cross-stage partial three convolution bottleneck module (C3) 2304, a convolution module (Conv) 2305, an up-sampling module (Upsample) 2306, a concatenation processing (Concat) 2307, a cross-stage partial three convolution bottleneck module (C3) 2308, a convolution module (Conv) 2309, a concatenation processing (Concat) 2310, a cross-stage partial three convolution bottleneck module (C3) 2311, a convolution module (Conv) 2312, a concatenation processing (Concat) 2313, and a cross-stage partial three convolution bottleneck module (C3) 2314.

[0046] After the first intermediate feature is input into the neck network 230, the feature output from the cross-stage partial triple convolutional bottleneck module 2209 is processed by the convolution module 2301 and the upsampling module 2302, and then concatenated with the feature output from the cross-stage partial triple convolutional bottleneck module 2206 2303; the feature output from the concatenation process 2303 is processed by the cross-stage partial triple convolutional bottleneck module 2304, the convolution module 2305 and the upsampling module 2306, and then concatenated with the feature output from the cross-stage partial triple convolutional bottleneck module 2204 2307; concatenation process 230... The features output by 7 are processed by the cross-stage partial three-convolution bottleneck module 2308 and convolution module 2309, and then concatenated with the features output by convolution module 2305 for 2310. The features output by concatenation 2310 are processed by the cross-stage partial three-convolution bottleneck module 2311 and convolution module 2312, and then concatenated with the features output by convolution module 2301 for 2313. The features output by concatenation 2313 are processed by the cross-stage partial three-convolution bottleneck module 2314, and then used as part of the second intermediate features output by the neck network 230.

[0047] The first intermediate feature is input into the neck network 230, and after the above processing, the second intermediate feature output by the neck network 230 is obtained. The second intermediate feature includes the features output by three modules: cross-stage partial 3-convolutional bottleneck module 2308, cross-stage partial 3-convolutional bottleneck module 2311, and cross-stage partial 3-convolutional bottleneck module 2314.

[0048] The head network 240 includes convolutional modules (Conv) 2401, 2402, 2403, vector transformation 2404, and splicing processing 2405.

[0049] The second intermediate feature is input into the head network 240. The features output by the cross-stage part of the three-convolution bottleneck module 2308 are processed by the convolution module 2401 and vector transformation 2404 to obtain the third intermediate feature. The features output by the cross-stage part of the three-convolution bottleneck module 2311 are processed by the convolution module 2402 and vector transformation 2404 to obtain the fourth intermediate feature. The features output by the cross-stage part of the three-convolution bottleneck module 2314 are processed by the convolution module 2403 and vector transformation 2404 to obtain the fifth intermediate feature. The third, fourth, and fifth intermediate features are concatenated by 2405 to obtain the first eye feature 250, which is the output of the human eye detection network.

[0050] Figure 3 A schematic diagram of the structure of a trunk start module according to an embodiment of this application is shown. Figure 3As shown, the backbone starting module includes convolutional modules (Conv) 301, 302, and 303, concatenation processing (Concat) 304, convolutional module (Conv) 305, and max pooling (Maxpool) 306. The concatenation processing 304 is used to concatenate the output of convolutional module 303 with the output of max pooling 306.

[0051] Figure 4 A schematic diagram of the structure of a cross-stage partial three-convolution bottleneck module according to an embodiment of this application is shown. Figure 4 As shown, the cross-stage partial triple convolutional bottleneck module includes a convolutional module (Conv) 401, n bottleneck modules (Bottleneck) 402, a concatenation process (Concat) 403, a convolutional module 404 (Conv), and a convolutional module (Conv) 405. The concatenation process 403 is used to concatenate the input of the cross-stage partial triple convolutional bottleneck module with the outputs of the n bottleneck modules 402.

[0052] Figure 5 A schematic diagram of the bottleneck module according to an embodiment of this application is shown. Figure 5 As shown, the bottleneck module includes a convolutional module (Conv) 501, a convolutional module (Conv) 502, and an addition process (Add) 503. The addition process 503 is used to add the input of the bottleneck module to the output of the convolutional module 503.

[0053] Figure 6 A schematic diagram of the structure of a spatial pyramid pooling module according to an embodiment of this application is shown. Figure 6 As shown, the spatial pyramid pooling module includes a convolutional module (Conv) 601, max pooling (Maxpool) 602, max pooling (Maxpool) 603, max pooling (Maxpool) 604, concatenation processing (Concat) 605, and a convolutional module (Conv) 606. The concatenation processing 605 is used to concatenate the outputs of convolutional module 601, max pooling 602, max pooling 603, and max pooling 604.

[0054] The above Figures 2 to 6 In the process, each convolutional module (Conv) performs the same processing, including two-dimensional convolution (Conv2d), batch normalization (BN), and SILU (Sigmoid-Weighted LinearUnit) activation.

[0055] In some possible implementation manners, the human eye detection network according to the embodiments of the present application can be implemented based on an improved YOLOv5 face detection model. The YOLOv5 face detection model (i.e., YOLOv5-Face model) is based on a YOLOv5 (You Only Look Once version 5) architecture and is mainly used for face recognition of images.

[0056] The YOLOv5-Face model simplifies the human face key points into 5 points. In order to meet the requirement of human eye key point detection, the output vector of the YOLOv5-Face model is improved in the present application. The dimension of the output vector of the YOLOv5-Face model is expanded from 16 to 30, that is, the dimension of the output vector of the improved YOLOv5-face model is 30. Further, the first eye feature output by the human eye detection network according to the embodiments of the present application includes a plurality of feature vectors, and the dimension of each feature vector is 30.

[0057] Each feature vector includes the position of the detection frame, the confidence of the detection frame, the position information of 12 eye key points, and the category of the detection frame (the category of the detection frame in the embodiments of the present application is human face). The position information of the detection frame occupies 4 dimensions; the confidence of the detection frame occupies 1 dimension; the position information of each eye key point occupies 2 dimensions, and the position information of 12 eye key points occupies 24 dimensions; and the category of the detection frame occupies 1 dimension.

[0058] Figure 7 A schematic diagram of the output vector of the human eye detection network according to an embodiment of the present application is shown. As shown in Figure 7 The dimension of the output vector of the YOLOv5 model is 6. Among them, the position information of the detection frame (bbox) occupies 4 dimensions, the confidence of the detection frame (conf) occupies 1 dimension, and the category of the detection frame (cls) occupies 1 dimension.

[0059] The dimension of the output vector of the YOLOv5-Face model (i.e., the standard YOLOv5-Face model) is 16. Among them, the position information of the detection frame (bbox) occupies 4 dimensions, the confidence of the detection frame (conf) occupies 1 dimension, the position information of 5 face key points (landmarks) occupies 10 dimensions in total, and the category of the detection frame (cls) occupies 1 dimension.

[0060] The output vector (i.e., the feature vector in the first eye feature) of the human eye detection network (i.e., the improved YOLOv5-Face model) has a dimension of 30. Among them, the position information (bbox) of the detection box accounts for 4 dimensions, the confidence (conf) of the detection box accounts for 1 dimension, the position information of each eye key point accounts for 2 dimensions, the position information (landmarks) of 12 eye key points accounts for 24 dimensions in total, and the class (cls) of the detection box accounts for 1 dimension.

[0061] In some possible implementations, before the human eye detection network is deployed to the embedded chip, the human eye detection network can be trained using a preset training set. In an example, the human eye detection network can be trained using a public dataset LaPa (Landmark guided face Parsing Dataset, a dataset for face key point detection released by Jingdong Artificial Intelligence, which labels the position information of 106 face key points) and WFLW (Wider Facial Landmarks in the Wild, a face dataset released by Sogou, which labels the position information of 98 face key points). Since the application scenario of the present application only requires the position information of 12 eye key points, the present application simplifies the data format of the label in the above two training sets, and the simplified label data format includes the bounding box coordinates, the eye key point coordinates and the class label (and the face).

[0062] Figure 8 A schematic diagram of a data label according to an embodiment of the present application is shown. As Figure 4 shown, when training the face detection network of the present application, the data label used includes the bounding box coordinates, the 6 key point coordinates of each eye, and the class label (Class: 0).

[0063] In the embodiments of the present application, the human eye detection network is implemented based on the improved YOLOv5-Face model, thereby being able to realize full-scene, efficient and accurate eye key point position detection, and thereby improving the accuracy of pilot fatigue detection.

[0064] In some possible implementations, after the face detection network is trained, it can be deployed on an embedded chip. In the following, taking a domestic low-power and high-performance embedded chip RK3588 as an example, the deployment of the face detection network will be exemplarily described.

[0065] The embedded chip RK3588 can realize the whole-process technical chain from model conversion, evaluation to board deployment, thanks to its integrated NPU (Neural-network Processing Unit) and RKNN-Toolkit2 environment (hereinafter referred to as Toolkit2) on the PC side. The Toolkit2 is a matching development tool of RK3588 on the PC side, which is used to convert the model file of different frameworks (such as PyTorch, ONNX and TensorFlow, etc.) trained on the PC side into RKNN format supporting NPU operation.

[0066] In the model conversion process, the Toolkit2 provides a model quantization interface, which can convert the floating-point parameters and operations in the deep learning model into fixed-point representation, for example, convert FLOAT32 into INT8. Such quantization can reduce memory occupation and speed up the model inference process, but may cause a certain accuracy loss.

[0067] Since the parameters of the human eye detection network are floating-point type, in order to balance the relationship between model performance and accuracy, the embodiment of the present application adopts a mixed quantization method when deploying the human eye detection model. Specifically, the floating-point parameters and operations in the backbone network and neck network of the human eye detection model are quantized into fixed-point parameters and operations, for example, the parameters and operations of FLOAT16 are quantized into the parameters and operations of INT8, that is, the process of converting floating-point parameters and operations into fixed-point parameters and operations is called quantization; and then the quantized backbone network and neck network are deployed on the embedded chip RK3588. As for the head network, it is not quantized, the accuracy of FLOAT16 is retained, and it is directly deployed on the embedded chip RK3588. In this way, the human eye detection model can be mixed quantized, so as to not only meet the needs of airborne embedded applications, but also effectively balance the real-time performance and accuracy of human eye key point detection.

[0068] In some possible implementation manners, the quantization of floating-point numbers (such as FLOAT16) into signed fixed-point numbers (such as INT8) can be realized through the following formula (1):

[0069] (1)

[0070] In formula (1), is a floating-point number; is a quantized fixed-point number; is a rounding operation; is a quantization scale factor; is a quantization zero point; is a quantization bit width, such as 8 in the INT8 data type; For the truncation operation, the following formula (2) can be used to represent:

[0071] (2)

[0072] In formula (2), represents ; represents ; represents .

[0073] The process of converting a signed fixed-point number (for example, INT8) to a floating-point number (for example, FLOAT16) is called a dequantization process, which can be represented by the following formula (3):

[0074] (3)

[0075] In formula (3), represents the calculation result; represents taking as the result after dequantization.

[0076] The quantization parameter in the above formula (1) and (3) can be determined by the following formula (4), and the quantization parameter can be determined by the following formula (5):

[0077] (4)

[0078] (5)

[0079] wherein, represents the quantization range, that is, the minimum value and the maximum value in the output vector of the human eye detection network; represents the truncation range, that is, the range of the quantized fixed-point number, for example, the minimum value and the maximum value of INT8.

[0080] Since the value range of the position information (for example, coordinates) of the detection frame and the key point in the output vector of the human eye detection network is much larger than the value range of the confidence, if the quantization range is directly determined according to the position information of the detection frame and the key point, and then the quantization parameter is calculated, the model quantization will cause a serious loss of confidence accuracy, thereby greatly affecting the detection result. In order to overcome this problem, the embodiment of the present application first normalizes the position information of the detection frame and the key point in the output vector (for example, divides the actual coordinates of both by the size of the image), and then determines the quantization range.

[0081] ​Since the human eye detection network is subjected to hybrid quantization, after obtaining the output vector of the human eye detection network (i.e., the feature vector in the first eye feature), the position information in the output vector can be subjected to dequantization in the manner described in equation (3) so as to correspond to the input image.

[0082] Figure 9 A flowchart of a pilot fatigue detection method according to an embodiment of the present application is shown. As shown in the flowchart, the method comprises: Figure 9

[0083] In step S910, an image of a pilot captured by a camera within a preset time period is obtained.

[0084] The preset time period can be set by a person skilled in the art according to actual requirements. For example, the preset time period is 1 minute, 2 minutes, etc. The present application does not limit the specific value of the preset time period.

[0085] The image of the pilot captured by the camera includes the face of the pilot, so as to be subjected to human eye key point detection in step S920. The image of the pilot captured by the camera within the preset time period is a plurality of images.

[0086] In step S920, for any image, a human eye key point detection is performed on the image by a human eye detection network to obtain a first eye feature corresponding to the image.

[0087] For any image of the pilot obtained, the image can be input into the human eye detection network. A human eye key point detection is performed on the image by the human eye detection network to obtain a first eye feature corresponding to the image. The first eye feature includes a plurality of feature vectors.

[0088] In step S930, whether the image embodies a fatigue feature is determined according to the first eye feature corresponding to the image, the maximum and minimum values of the eye aspect ratio of the pilot.

[0089] The maximum and minimum values of the eye aspect ratio of the pilot can be preset values or values after individualized calibration. A person skilled in the art can determine the setting mode of the maximum and minimum values of the eye aspect ratio according to actual conditions, and the present application does not limit this.

[0090] The maximum and minimum values of the eye aspect ratio of the pilot can be used to determine an eye aspect ratio threshold of the pilot, and the eye aspect ratio of the pilot in the image can be determined according to the first eye feature corresponding to the image. Then, the eye aspect ratio of the pilot is compared with the eye aspect ratio threshold, and whether the image embodies a fatigue feature is determined.

[0091] ​In some possible implementation manners, step S930 can include: determining an eye aspect ratio threshold of the pilot according to the maximum value and the minimum value of the eye aspect ratio of the pilot; determining position information of an eye key point of the pilot in the image according to the first eye feature corresponding to the image; determining the eye aspect ratio of the pilot in the image according to the position information of the eye key point; and determining whether the image embodies the fatigue feature according to the eye aspect ratio of the pilot in the image and the eye aspect ratio threshold of the pilot.

[0092] The eye aspect ratio threshold of the pilot can be determined according to the maximum value and the minimum value of the eye aspect ratio of the pilot. First, a difference between the maximum value and the minimum value of the eye aspect ratio of the pilot can be determined, and a product of the difference and a preset coefficient can be determined. Then, a sum of the product and the minimum value of the eye aspect ratio of the pilot is determined as the eye aspect ratio threshold of the pilot. The preset coefficient can be set as 0.2.

[0093] The eye aspect ratio threshold of the pilot can be calculated by the following formula (6), that is, :

[0094] (6)

[0095] In the formula (6), represents the maximum value of the eye aspect ratio of the pilot; represents the minimum value of the eye aspect ratio of the pilot; represents the preset coefficient.

[0096] The non-maximum suppression algorithm can be used to select a target feature vector from the plurality of feature vectors of the first eye feature corresponding to the image, and the position information of the eye key point in the target feature vector is determined as the position information of the eye key point of the pilot in the image. Then, the eye aspect ratio of the pilot in the image can be calculated according to the position information of the eye key point. The eye aspect ratio EAR can be used to determine whether the human eye is in an open state or a closed state. When the human eye is in the open state, the EAR fluctuates up and down around a certain value; when the human eye is in the closed state, the EAR rapidly decreases. When the EAR is lower than a certain threshold, it can be determined that the eye is in the closed state.

[0097] When calculating the eye aspect ratio of the pilot in the image, the position information of the key point of the left eye of the pilot in the image can be used to determine the left eye aspect ratio, and the position information of the key point of the right eye of the pilot in the image can be used to determine the right eye aspect ratio. Then, the average value of the left eye aspect ratio and the right eye aspect ratio is determined as the eye aspect ratio of the pilot in the image.

[0098] In some possible implementations, the eye aspect ratio of the left eye or the right eye of the pilot in the image can be calculated by the following formula (7) :

[0099] (7)

[0100] In the formula (7), represents the position of the key point of the inner corner of the eye (which can be represented by coordinates); represents the position of the key point of the upper eyelid close to the inner corner of the eye (which can be represented by coordinates); represents the position of the key point of the upper eyelid close to the outer corner of the eye (which can be represented by coordinates); represents the position of the key point of the outer corner of the eye (which can be represented by coordinates); represents the position of the key point of the lower eyelid close to the outer corner of the eye (which can be represented by coordinates); represents the position of the key point of the lower eyelid close to the inner corner of the eye (which can be represented by coordinates); represents the L2 norm.

[0101] Figure 10 FIG. 1 shows a schematic diagram of the positions of eye key points according to an embodiment of the present application. As shown in FIG. 1, the positions of the six eye key points are respectively: Figure 10 represents the position of the key point of the inner corner of the eye, represents the position of the key point of the upper eyelid close to the inner corner of the eye, represents the position of the key point of the upper eyelid close to the outer corner of the eye, represents the position of the key point of the outer corner of the eye, represents the position of the key point of the lower eyelid close to the outer corner of the eye, represents the position of the key point of the lower eyelid close to the inner corner of the eye. After the eye aspect ratio of the pilot in the image and the eye aspect ratio threshold of the pilot are determined, whether the image embodies the fatigue feature can be determined according to the eye aspect ratio of the pilot in the image and the eye aspect ratio threshold of the pilot. The eye aspect ratio of the pilot in the image can be compared with the eye aspect ratio threshold of the pilot. In the case where the eye aspect ratio of the pilot in the image is less than the eye aspect ratio threshold of the pilot, it can be determined that the image embodies the fatigue feature. In the case where the eye aspect ratio of the pilot in the image is greater than or equal to the eye aspect ratio threshold of the pilot, it can be determined that the image does not embody the fatigue feature.

[0102] In step S940, the fatigue detection result of the pilot is determined according to the first number of images embodying the fatigue feature, the second number of images in the preset time length, and the preset fatigue threshold.

[0103]

[0104] ​By the steps S920 and S930, it can be determined whether each image within the preset time length embodies the fatigue feature, and then the first quantity of images embodying the fatigue feature is determined through statistics. Then, the fatigue detection result of the pilot can be determined according to the first quantity, the second quantity of images within the preset time length, and the preset fatigue threshold. The fatigue threshold is, for example, 0.5. It should be noted that the specific value of the fatigue threshold can be set by a person skilled in the art according to the actual situation, and the present application does not limit this. The fatigue detection result is in a non-fatigue state or in a fatigue state.

[0105] In some possible implementation manners, the step S940 can include: determining a proportion of the images embodying the fatigue feature in the images within the preset time length according to the first quantity of images embodying the fatigue feature and the second quantity of images within the preset time length; and determining that the fatigue detection result of the pilot is in a fatigue state in a case where the proportion is greater than a preset fatigue threshold.

[0106] The ratio of the first quantity to the second quantity can be determined as the proportion of the images embodying the fatigue feature in the images within the preset time length, and then the proportion is compared with the preset fatigue threshold. In a case where the proportion is less than or equal to the preset fatigue threshold, it can be determined that the fatigue detection result of the pilot is in a non-fatigue state. In a case where the proportion is greater than the preset fatigue threshold, it can be determined that the fatigue detection result of the pilot is in a fatigue state.

[0107] The pilot fatigue detection method of the embodiments of the present application can perform eye key point detection on the images of the pilot collected by the camera through the human eye detection network to obtain the first eye feature corresponding to the images, and then determine whether the images embody the fatigue feature according to the first eye feature and the maximum value and the minimum value of the eye aspect ratio of the pilot. Then, the fatigue detection result of the pilot is determined according to the first quantity of images embodying the fatigue feature, the second quantity of images within the preset time length, and the preset fatigue threshold. The embodiments of the present application combine the eye key point and the eye aspect ratio to detect the fatigue of the pilot. The equipment is simple, it is non-invasive, it can effectively overcome the problem of face occlusion of the pilot, it is not easily affected by environmental factors such as light, and thus the efficiency and accuracy of the pilot fatigue detection can be effectively improved.

[0108] Since there are differences in eye features of different individuals (i.e., different pilots), it is difficult to uniformly quantify the EAR threshold for representing the eye closure state. In view of this, the pilot fatigue detection method of the embodiments of the present application also considers individual calibration. Therefore, the pilot fatigue detection method of the embodiments of the present application further includes: acquiring a preset quantity of calibration images of the pilot continuously blinking eyes collected by the camera; and determining the maximum value and the minimum value of the eye aspect ratio of the pilot according to the calibration images.

[0109] In the individual calibration process, the camera collects images of the pilot while the pilot blinks frequently. A preset number of calibration images of the pilot blinking can be obtained, where the preset number can be, for example, 50, 100, etc., and the application does not limit the specific value of the preset number.

[0110] The eye aspect ratio of the pilot can then be determined according to the calibration images. For any calibration image, the human eye detection network can be used to detect the key points of the human eye in the calibration image to obtain the second eye feature corresponding to the calibration image. According to the second eye feature, the eye aspect ratio of the pilot in the calibration image can be determined. Specifically, the position information of the eye key points of the pilot in the calibration image can be determined according to the second eye feature corresponding to the calibration image. Then, the left eye aspect ratio is determined according to the position information of the key points of the left eye of the pilot in the calibration image. The right eye aspect ratio is determined according to the position information of the key points of the right eye of the pilot in the calibration image. The average of the left eye aspect ratio and the right eye aspect ratio is determined as the eye aspect ratio of the pilot in the calibration image.

[0111] After determining the eye aspect ratio of the pilot in each calibration image by the above method, the maximum and minimum values of the eye aspect ratio of the pilot can be determined according to the eye aspect ratio of the pilot in each calibration image. For example, the minimum value of the eye aspect ratio of the pilot in all calibration images is determined as the minimum value of the eye aspect ratio of the pilot. The maximum value of the eye aspect ratio of the pilot in all calibration images is determined as the maximum value of the eye aspect ratio of the pilot.

[0112] In this way, the differences in eye features of different pilots can be fully considered in the pilot fatigue detection method of the application, so as to reduce the determination error caused by individual differences, and further improve the adaptability and accuracy of pilot fatigue detection.

[0113] Figure 11 A schematic diagram showing the processing process of the pilot fatigue detection method according to an embodiment of the application is shown. As shown in Figure 11 The processing process of the pilot fatigue detection method of the application includes:

[0114] Step S1101, determine whether to perform individual calibration; for example, the pilot can be identified by face recognition. After identifying the pilot, it can be checked whether the maximum and minimum values of the eye aspect ratio of the pilot are stored in the chip. If they are stored, individual calibration is not needed. If they are not stored, individual calibration is needed.

[0115] If individual calibration is needed, the following steps are performed:

[0116] Step S1102, a preset number of calibration images of continuous blinking of the pilot collected by the camera are obtained;

[0117] Step S1103, for any calibration image, the human eye key point detection is performed on the calibration image by the human eye detection network to obtain the second eye feature corresponding to the calibration image;

[0118] Step S1104, the position information of the eye key point of the pilot in the calibration image is determined according to the second eye feature;

[0119] Step S1105, the length-width ratio of the left eye of the pilot is determined according to the position information of the key point of the left eye in the calibration image;

[0120] Step S1106, the length-width ratio of the right eye of the pilot is determined according to the position information of the key point of the right eye in the calibration image;

[0121] Step S1107, the average value of the length-width ratio of the left eye and the length-width ratio of the right eye is determined as the length-width ratio of the eyes of the pilot in the calibration image;

[0122] After the above steps S1103 to S1107 are performed on each calibration image, step S1108 is performed to determine the minimum value of the length-width ratio of the eyes of the pilot in all calibration images as the minimum value of the length-width ratio of the eyes of the pilot, and determine the maximum value of the length-width ratio of the eyes of the pilot in all calibration images as the maximum value of the length-width ratio of the eyes of the pilot; step S1109, the maximum value and the minimum value of the length-width ratio of the eyes of the pilot are stored.

[0123] If individual calibration is not required, or after step S1109, the following steps are performed:

[0124] Step S1110, the maximum value and the minimum value of the length-width ratio of the eyes of the pilot stored are obtained;

[0125] Step S1111, the images of the pilot within a preset time length collected by the camera are obtained;

[0126] Step S1112, for any image, the human eye key point detection is performed on the image by the human eye detection network to obtain the first eye feature corresponding to the image;

[0127] Step S1113, the length-width ratio threshold of the eyes of the pilot is determined according to the maximum value and the minimum value of the length-width ratio of the eyes of the pilot;

[0128] Step S1114, the position information of the eye key point of the pilot in the image is determined according to the first eye feature corresponding to the image;

[0129] Step S1115, the length-width ratio of the eyes of the pilot in the image is determined according to the position information of the eye key point;

[0130] Step S1116, determining whether the image embodies the fatigue feature according to the length-width ratio of the pilot's eyes in the image and the length-width ratio threshold of the pilot's eyes;

[0131] Step S1117, determining a proportion of the images embodying the fatigue feature in the images in the preset time length according to the first number of the images embodying the fatigue feature and the second number of the images in the preset time length;

[0132] Step S1118, judging whether the proportion is greater than a preset fatigue threshold;

[0133] If the proportion is greater than the preset fatigue threshold, step S1119 is performed to determine that the pilot is in a fatigue state; otherwise, step S1120 is performed to determine that the pilot is in a non-fatigue state.

[0134] Embodiments of the present application also provide a pilot fatigue detection device, comprising a camera and an embedded chip; the camera is used to collect images of a pilot; the embedded chip stores one or more instructions, and the one or more instructions are executed by the embedded chip to implement the pilot fatigue detection method of any of the above embodiments.

[0135] The flowcharts and block diagrams in the drawings show the possible implementation architecture, function and operation of the apparatus, system, method and computer program product according to the embodiments of the present application. In this regard, each block in the flowchart or block diagram can represent a module, program segment or part of instructions containing one or more executable instructions for implementing the specified logic function. In some alternative implementations, the functions noted in the blocks can also occur in different order from that noted in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and sometimes they can be executed in reverse order, depending on the functions involved.

[0136] It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by hardware (such as a circuit or an ASIC (Application Specific Integrated Circuit)) that performs the corresponding function or action, or can be implemented by a combination of hardware and software, such as firmware, etc.

[0137] Although the application has been described in connection with various embodiments, it will be understood that the application is capable of further modifications. These and other changes, along with the apparent alternatives and equivalents, fall within the scope of the claimed application. The description herein is intended to be illustrative only and is presented to enable any person skilled in the art to make and use the application. Numerous modifications and adaptations will be apparent to those skilled in the art without departing from the scope of the described application. The scope of the described application is not to be limited by the specific illustrative embodiments contained herein but only by the scope of the appended claims, which follow this disclosure.

[0138] Various embodiments of the application have been described in connection with the embodiments described above. The description is intended to be illustrative only and not limiting of the application. Many modifications and variations of the described embodiments are possible in light of the above teachings. The terms used in the following claims should not be construed to limit the application to the specific embodiments disclosed in the specification and drawings. It is, therefore, to be understood that within the scope of the appended claims the application can be practiced otherwise than is specifically described in the specification and drawings.

Claims

1. A pilot fatigue detection method characterized by, The method comprises: acquiring images of the pilot collected by the camera within a preset time length; for any image, detecting the image through a human eye detection network to obtain first eye features corresponding to the image; acquiring a preset number of calibration images of the pilot continuously blinking collected by the camera; determining a maximum value and a minimum value of the eye aspect ratio of the pilot according to the calibration images; determining an eye aspect ratio threshold of the pilot according to the maximum value and the minimum value of the eye aspect ratio of the pilot, the threshold being determined according to the following standard: determining the difference between the maximum value and the minimum value of the eye aspect ratio of the pilot, and determining the product of the difference and a preset coefficient; then determining the sum of the product and the minimum value of the eye aspect ratio of the pilot as the eye aspect ratio threshold of the pilot; determining the position information of the eye key points of the pilot in the image according to the first eye features corresponding to the image; determining the eye aspect ratio of the pilot in the image according to the position information of the eye key points; determining whether the image embodies fatigue features according to the eye aspect ratio of the pilot in the image and the eye aspect ratio threshold of the pilot; determining the fatigue detection result of the pilot according to the first number of images embodying fatigue features, the second number of images within the preset time length, and a preset fatigue threshold; The human eye detection network comprises a backbone network, a neck network and a head network, the backbone network is used for feature extraction of an input image to obtain first intermediate features; the neck network is used for feature fusion of the first intermediate features to obtain second intermediate features; and the head network is used for feature detection of the second intermediate features to obtain first eye features; The first eye features comprise a plurality of feature vectors, each feature vector comprises position information of a detection frame, confidence of the detection frame, position information of 12 eye key points and a category of the detection frame, and the dimension of each feature vector is 30; The human eye detection network is deployed on an embedded chip, the parameters of the human eye detection network are floating-point type, and the method further comprises: quantizing floating-point parameters and operations in the backbone network and the neck network into fixed-point parameters and operations, and deploying the quantized backbone network and neck network to the embedded chip, wherein the quantization first normalizes the position information of the detection frame and the key points in the output vector, and then determines the quantization range; the head network is directly deployed to the embedded chip.

2. The method of claim 1, wherein, The determination of the eye aspect ratio of the pilot in the image according to the position information of the eye key points comprises: determining a left eye aspect ratio according to the position information of the key points of the left eye of the pilot in the image; determining a right eye aspect ratio according to the position information of the key points of the right eye of the pilot in the image; determining the average value of the left eye aspect ratio and the right eye aspect ratio as the eye aspect ratio of the pilot in the image.

3. The method of claim 1, wherein, The determination of whether the image embodies fatigue features according to the eye aspect ratio of the pilot in the image and the eye aspect ratio threshold of the pilot comprises: In a case where the eye length-width ratio of the pilot in the image is less than the eye length-width ratio threshold of the pilot, it is determined that the image embodies a fatigue feature.

4. The method of claim 1, wherein, The fatigue detection result of the pilot is determined according to the first quantity of images embodying fatigue features, the second quantity of images in the preset time length, and a preset fatigue threshold. The proportion of images embodying fatigue features in the images in the preset time length is determined according to the first quantity of images embodying fatigue features and the second quantity of images in the preset time length. In a case where the proportion is greater than the preset fatigue threshold, it is determined that the fatigue detection result of the pilot is in a fatigue state.

5. The method of claim 1, wherein, The maximum value and the minimum value of the eye length-width ratio of the pilot are determined according to the calibration images, including: For any calibration image, the second eye feature corresponding to the calibration image is obtained by performing eye key point detection on the calibration image by using the human eye detection network; and the eye length-width ratio of the pilot in the calibration image is determined according to the second eye feature. The maximum value and the minimum value of the eye length-width ratio of the pilot are determined according to the eye length-width ratio of the pilot in each calibration image.

6. A pilot fatigue detection device characterized by comprising: The device comprises a camera and an embedded chip; the camera is used to collect images of the pilot; and the embedded chip stores one or more instructions, which are executed by the embedded chip to implement the pilot fatigue detection method in any one of claims 1-5.

Images