Driver fatigue monitoring method and device, computer equipment and medium

Abstract

The embodiment of the invention provides a driver fatigue monitoring method, which relates to a driver fatigue technology, and comprises the following steps: aiming at each frame of an upper body image of a driver, judging an eye state and a mouth state of the frame of image; extracting high-level features of a plurality of RO I areas of each frame of image, and fusing the high-level features to obtain combined features; performing feature feedback prediction to obtain an initial fatigue state; determining an initial eye state and an initial mouth state; and carrying out weighted integration on the states to obtain a final fatigue estimation result. According to the technical scheme, firstly, a plurality of RO I regions are introduced to obtain combined features with higher characterization capability, in addition, an existing sequential network structure is improved, and a more effective fatigue probability prediction value is obtained through feature feedback; and the fatigue prediction value after weighted synthesis is obtained by combining the fatigue prediction value of the key point detection and the fatigue prediction value of the sequential network, so that the fatigue prediction result is more accurate, and the problems of false report and missing report in the current fatigue detection technology can be effectively solved.

Description

Technical Field

[0001] The present invention relates to the technical field of fatigue driving, and in particular to a driver fatigue monitoring method, device, computer equipment and medium. Background Art

[0002] With the popularization of cars and the increasing congestion of road traffic, traffic accidents caused by driver fatigue driving occur frequently, becoming a major hidden danger to road traffic safety.

[0003] Most existing driver fatigue detection technologies are based on the detection of facial key points, obtaining the key point information of the eyes and mouth, and then calculating their opening and closing degrees to determine whether the driver is in a fatigued state. This technology based on shallow facial features often misdetects some situations where the eyes have a small opening and closing degree at the image imaging level, such as misidentifying the "looking down" state as "fatigue", resulting in false fatigue alarms. In addition, there are also some technologies that use convolutional networks to extract high-level features and then use temporal networks to input multi-frame feature information to determine fatigue. However, since high-level feature extraction is based on the entire face, it does not capture fatigue features as well as the eye area. Even if an attention mechanism is added to the convolutional network, it often has poor generalization in new scenes, resulting in fatigue omissions.

[0004] Patent CN202210679109.7 proposes a time-series-based end-to-end fatigue driving detection method, which uses a convolutional network with an attention mechanism to extract high-level features of each frame image, and then inputs the high-level features of continuous frames into a long short-term memory neural network to obtain fatigue prediction results. In order to make fatigue detection based on time series prediction have a certain degree of generalization in some complex scenes, although this method expands the training set by simulating changes in illumination and camera position, after training the model based on the original training set, the model is trained again using the expanded training set, and the generalization ability is actually limited. On the one hand, it is because the expanded data set cannot simulate various complex environments, and on the other hand, it is because the driver's facial information is rich, and the fatigue forms of different drivers are also different. It is difficult to capture abstract fatigue features based on high-level feature extraction based only on facial features, and it is difficult to solve even with the addition of an attention mechanism. This will lead to the problem of missed detection and false detection of driver fatigue based on the current method.

[0005] Therefore, developing a driver fatigue detection system based on time series and multi-feature fusion to achieve comprehensive and accurate detection of the driver's fatigue state has important practical significance and application value for improving driver safety. Summary of the invention

[0006] In view of this, an embodiment of the present invention provides a driver fatigue monitoring method based on a novel timing network to solve the problems of false alarm and missed alarm in the existing fatigue detection technology.

[0007] In order to achieve the above object, the present invention provides a driver fatigue monitoring method, comprising:

[0008] For each frame of the driver's upper body image, the facial key points of the driver in the frame image are obtained through a facial key point detection network, the facial ROI area, the eye ROI area and the mouth ROI area of ​​the frame image are determined based on the facial key points, and the eye state and the mouth state of the frame image are judged based on the facial key points;

[0009] The face ROI region, the eye ROI region and the mouth ROI region of the frame image are input into the feature extraction model to extract the high-level features of each region, and the high-level features of each region are fused to obtain the combined features of the frame image;

[0010] The combined features of multiple consecutive frames of images are input into the feedback temporal network to perform feature feedback prediction and obtain the initial fatigue state;

[0011] Determine an initial eye state and an initial mouth state according to the eye states and the mouth states of the continuous multiple frames of images;

[0012] The initial eye state, the initial mouth state and the initial fatigue state are weighted and integrated to obtain a final fatigue estimation result.

[0013] The driver fatigue monitoring method provided by the present invention also has the following characteristics: determining the facial ROI area, the eye ROI area and the mouth ROI area of ​​the frame image based on the facial key points comprises:

[0014] The facial ROI area of ​​the frame image is determined based on the minimum circumscribed rectangle formed by all facial contour key points among the facial key points, the eye ROI area is determined based on the minimum circumscribed rectangle formed by all eye key points among the facial key points, and the mouth ROI area is determined based on the minimum circumscribed rectangle formed by all mouth key points among the facial key points.

[0015] The driver fatigue monitoring method provided by the present invention also has the following characteristics: determining the face ROI area of ​​the frame image based on the minimum bounding rectangle formed by all facial contour key points among the face key points, determining the eye ROI area based on the minimum bounding rectangle formed by all eye key points among the face key points, and determining the mouth ROI area based on the minimum bounding rectangle formed by all mouth key points among the face key points, including:

[0016] The facial ROI area, eye ROI area and mouth ROI area are determined by the following formulas:

[0017] Region(eye)=Rect(eye)+D1

[0018] Region(mouth)=Rect(mouth)+D2

[0019] Region(face)=Rect(face)+D3

[0020] D1={(-0.2d1,-0.2d1),(0.2d1,0.2d1)}

[0021] D2={(-0.2d2,-0.2d2),(0.2d2,0.2d2)}

[0022] D3={(-0.2d3,-0.2d4),(0.2d3,0.2d4)}

[0023] Among them, Region(eye) is the eye ROI area, Region(mouth) is the mouth ROI area, and Region(face) is the face ROI area. i (i=1,2,3) respectively represents the expansion of the regions of Rect(eye), Rect(mouth), and Rect(face), so that the three ROI regions can include the eyes, mouth, face, and nearby pixels. d1 represents the eye width, that is, the width of the minimum circumscribed rectangle of the eye Rect(eye); d2 represents the mouth width, that is, the width of the minimum circumscribed rectangle of the mouth Rect(mouth); d3 represents the face width, that is, the width of the minimum circumscribed rectangle of the face Rect(face); d4 represents the face height, that is, the height of the minimum circumscribed rectangle of the face Rect(face).

[0024] The driver fatigue monitoring method provided by the present invention also has the following characteristics: the face ROI area, the eye ROI area and the mouth ROI area of ​​the frame image are input into a feature extraction model, high-level features of the respective areas are extracted, and the high-level features of the respective areas are fused to obtain the combined features of the frame image, including:

[0025] Inputting the facial ROI region into the first multi-layer convolution layer in the feature extraction model to extract the first high-level feature; inputting the eye ROI region into the second multi-layer convolution layer in the feature extraction model to extract the second high-level feature; inputting the mouth ROI region into the third multi-layer convolution layer in the feature extraction model to extract the third high-level feature, wherein the weight parameters are shared among the first multi-layer convolution layer, the second multi-layer convolution layer and the third multi-layer convolution layer;

[0026] The first high-level feature, the second high-level feature and the third high-level feature are combined to obtain a combined feature of the frame image.

[0027] The driver fatigue monitoring method provided by the present invention also has the following characteristics: inputting the combined features of multiple consecutive frames of images into a feedback timing network, performing feature feedback, and obtaining an initial fatigue state, including:

[0028] The combined features of images of multiple consecutive frames form a time series sequence of preset length. In each time series window, the feedback time series network outputs the output time series features, fatigue prediction probability value and non-fatigue probability prediction value in the current time series window, and the output time series features in the current time series window and the time series sequence to be input in the next time series window are simultaneously input into the feedback time series network in the next time series window until the inference is iterated a preset number of times, and the initial fatigue state is output, and the initial fatigue state includes the initial fatigue prediction probability value and the initial non-fatigue probability prediction value. The preset number of times is the same as the number of images of multiple consecutive frames.

[0029] The driver fatigue monitoring method provided by the present invention also has the following characteristics: the initial eye state, the initial mouth state and the initial fatigue state are weighted and integrated to obtain a final fatigue estimation result, including:

[0030] Performing a weighted combination of the initial fatigue prediction probability value in the initial fatigue state and the eye-closing state value and the eye-squinting state value in the initial eye state to obtain a final state value of eye-closing fatigue;

[0031] Performing a weighted combination of the initial fatigue prediction probability value in the initial fatigue state and the yawning state value in the initial mouth state to obtain a yawning fatigue final state value;

[0032] The fatigue estimation result is obtained according to the magnitude relationship between the final state value of eye-closing fatigue and the first threshold, or the magnitude relationship between the final state value of yawning fatigue and the second threshold.

[0033] The driver fatigue monitoring method provided by the present invention also has the following characteristics: the fatigue estimation result is obtained according to the magnitude relationship between the final state value of eye-closing fatigue and the first threshold value, or the magnitude relationship between the final state value of yawning fatigue and the second threshold value, including:

[0034]

[0035] Among them, ε1 is the first threshold for judging eye fatigue, ε2 is the second threshold for judging yawning fatigue, and p x Indicates the final state value of eye-closing fatigue, py Indicates the final state value of yawning fatigue, z=1 indicates that the fatigue estimation result is fatigue, and z=0 indicates that the fatigue estimation result is non-fatigue.

[0036] A second object of the present invention is to provide a driver fatigue monitoring device, comprising:

[0037] An image acquisition module, used for acquiring the upper body image of the driver in each frame;

[0038] The feature screening module is provided with a facial key point detection system, which is used to determine the facial ROI area, eye ROI area and mouth ROI area of ​​the frame image through the key points of the driver of the frame image obtained by the image acquisition module, judge the eye state and mouth state of the frame image, and input the facial ROI area, eye ROI area and mouth ROI area of ​​the frame image, extract the high-level features of each area, and fuse the high-level features of each area to obtain the combined features of the frame image;

[0039] The feedback timing module is used to perform feature feedback on the combined features of multiple consecutive frames to obtain the initial fatigue state;

[0040] The fatigue judgment module is used to determine the initial eye state and the initial mouth state according to the eye state and the mouth state of multiple consecutive frames of images, and to weight and integrate the initial eye state, the initial mouth state and the initial fatigue state to obtain the final fatigue estimation result.

[0041] The third object of the present invention is to provide a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the aforementioned driver fatigue monitoring methods when executing the computer program.

[0042] A fourth object of the present invention is to provide a computer-readable storage medium, wherein the computer-readable storage medium stores a driver fatigue monitoring method for executing any one of the aforementioned methods.

[0043] Compared with the prior art, the beneficial effects that can be achieved by at least one of the above-mentioned technical solutions adopted in the embodiments of this specification include at least the following: the technical solution provided by the present invention, first, introduces multiple ROI areas to obtain combined features with stronger characterization capabilities, and secondly, improves the existing timing network structure to obtain a more effective fatigue probability prediction value through feature feedback, and finally introduces multi-classification, adds a "squinting" category, and combines the fatigue prediction value of key point detection with the fatigue prediction value of the timing network to obtain a weighted comprehensive fatigue prediction value, so that the fatigue prediction result is more accurate and can effectively deal with the false alarm and missed alarm problems in the current fatigue detection technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 is a flow chart of a driver fatigue monitoring method provided by an embodiment of the present invention;

[0046] Figure 2 is a structural diagram of fatigue calculation in an embodiment of the present invention;

[0047] Figure 3 is a structural diagram of the overall network structure in an embodiment of the present invention;

[0048] Figure 4 is a point information map of key points on the driver's face in an embodiment of the present invention;

[0049] Figure 5 is a structural block diagram of a computer device provided by an embodiment of the present invention;

[0050] Figure 6 It is a structural block diagram of a driver fatigue monitoring device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0052] The following describes the implementation methods of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, in the absence of conflict, the following embodiments and the features in the embodiments can be combined with each other. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work belong to the scope of protection of the present application.

[0053] In an embodiment of the present invention, a driver fatigue monitoring method is provided. Figure 1 As shown, including:

[0054] For each frame of the driver's upper body image, the facial key points of the driver in the frame image are obtained through a facial key point detection network, the facial ROI area, the eye ROI area and the mouth ROI area of ​​the frame image are determined based on the facial key points, and the eye state and the mouth state of the frame image are judged based on the facial key points;

[0055] The face ROI region, the eye ROI region and the mouth ROI region of the frame image are input into the feature extraction model to extract the high-level features of each region, and the high-level features of each region are fused to obtain the combined features of the frame image;

[0056] The combined features of multiple consecutive frames of images are input into the feedback temporal network to perform feature feedback prediction and obtain the initial fatigue state;

[0057] Determine an initial eye state and an initial mouth state according to the eye states and the mouth states of the continuous multiple frames of images;

[0058] The initial eye state, the initial mouth state and the initial fatigue state are weighted and integrated to obtain a final fatigue estimation result.

[0059] In the above embodiment, the image of the driver's upper body is obtained by the vehicle-mounted infrared camera, and the key points of the driver's face are obtained through the face key point detection network. Based on the facial contour key points, the facial ROI area is determined; based on the eye key points, on the one hand, the eye ROI area is cut out, and on the other hand, the open, squinting and closed eyes states are judged by the eye opening degree; based on the mouth key points, on the one hand, the mouth ROI area is cut out, and on the other hand, the yawning and non-yawning states are judged by the mouth opening degree. The introduction of ROI-based combined features, combined with key point detection based on facial surface information and feedback timing networks based on high-level combined features, can obtain more stable and accurate fatigue detection results, which can be applied to the fatigue states of different drivers and more driving scenarios.

[0060] In some embodiments, determining the facial ROI region, the eye ROI region, and the mouth ROI region of the frame image based on the facial key points includes:

[0061] The facial ROI area of ​​the frame image is determined based on the minimum circumscribed rectangle formed by all facial contour key points among the facial key points, the eye ROI area is determined based on the minimum circumscribed rectangle formed by all eye key points among the facial key points, and the mouth ROI area is determined based on the minimum circumscribed rectangle formed by all mouth key points among the facial key points.

[0062] In some embodiments, the driver's upper body image is obtained by an infrared camera. In-vehicle fatigue monitoring cameras generally use infrared cameras, which are applicable to daytime and nighttime scenes. The driver's upper body grayscale image can be obtained by the infrared camera, and the driver's facial key point information can be obtained by using the facial key point detection model. In some embodiments, the driver's facial key point information takes 68 key points as an example, and the key point position information obtained is as follows: Figure 4 As shown: Through the key point information, the minimum bounding rectangles of the face, eyes and mouth can be obtained respectively, namely:

[0063] ①Rect(eye left )={(min(kpx 36 , …, kpx 41 ), min(kpy 36 , …, kpy 41 )),(max(kpx 36 , …, kpx 41 ), max(kpy 36 , …, kpy 41 ))}

[0064] ②Rect(eye right )={(min(kpx 42 , …, kpx 47 ), min(kpy 42 , …, kpy 47 )),(max(kpx 42 , …, kpx 47 ), max(kpy 42 , …, kpy 47 ))}

[0065] ③Rect(eye)={(min(kpx 36 , …, kpx 47 ), min(kpy 36 , …, kpy 47 )),(max(kpx 36 , …, kpx 47 ), max(kpy 36 , …, kpy 47 ))}

[0066] ④Rect(mouth)={(min(kpx 48 , …, kpx 67 ), min(kpy 48 , …, kpy 67 )),

[0067] (max(kpx48 , …, kpx 67 ), max(kpy 48 , …, kpy 67 ))}

[0068] ⑤Rect(face)={(min(kpx0,...,kpx 67 ), min(kpy0, …, kpy 67 )),(max(kpx0,…,kpx 67 ), max(kpy0, …, kpy 67 ))}

[0069] Among them, kpx i (i=0, ..., 67) represents the horizontal coordinate of the key point, kpy i (i=0, ..., 67) represents the ordinate of the key point. Rect(·) represents the minimum circumscribed rectangle, which is expressed in the form of {upper left vertex coordinate, lower right vertex coordinate}.

[0070] According to the above minimum bounding rectangle, three ROI areas can be obtained:

[0071] ①Region(eye)=Rect(eye)+D1

[0072] ②Region(mouth)=Rect(mouth)+D2

[0073] ③Region(face)=Rect(face)+D3

[0074] Among them, Region(eye) is the eye ROI area, Region(mouth) is the mouth ROI area, and Region(face) is the face ROI area. i (i=1, 2, 3) means expanding the regions of Rect(eye), Rect(mouth), and Rect(face) respectively, so that the three ROI regions can include the eyes, mouth, face, and nearby pixels.

[0075] In some embodiments, on the one hand, because there may be errors in the detection of key points, appropriate expansion can eliminate the impact of key point errors, so that the offset of key points will also be within the ROI area; on the other hand, in addition to the fatigue characteristics being reflected from the eyes and mouth, the texture information formed by the surrounding pixels can also reflect the fatigue characteristics. Therefore, the facial ROI area, the eye ROI area, and the mouth ROI area need to be expanded. The determination of the facial ROI area, the eye ROI area, and the mouth ROI area includes:

[0076] The face, eyes and mouth are expanded. The expansion methods of the three areas are described by the following formulas:

[0077] D1={(-0.2d1,-0.2d1),(0.2d1,0.2d1)}

[0078] D2={(-0.2d2,-0.2d2),(0.2d2,0.2d2)}

[0079] D3={(-0.2d3,-0.2d4),(0.2d3,0.2d4)}

[0080] Among them, d1 represents the eye width, that is, the width of the minimum circumscribed rectangle of the eye Rect(eye); d2 represents the mouth width, that is, the width of the minimum circumscribed rectangle of the mouth Rect(mouth); d3 represents the face width, that is, the width of the minimum circumscribed rectangle of the face Rect(face); d4 represents the face height, that is, the height of the minimum circumscribed rectangle of the face Rect(face).

[0081] In this embodiment, when the face is expanded horizontally and vertically, the expansion is performed based on the width and height of the face respectively, while when the eyes and mouth are expanded horizontally and vertically, the expansion is performed based on the corresponding width, and the height is not included in the calculation. The reason for this difference is that the eyes and mouth are closed, which will cause the height of their minimum circumscribed rectangle to be close to 0. In order to maintain normal expansion without being affected by closure, the width is still used as the measurement for the vertical expansion of the eyes and mouth.

[0082] In some embodiments, based on the detected key point information, the expression of the distance between the opening and closing degree of the eyes and the mouth can be obtained:

[0083]

[0084] Among them, i = 37, 38, 43, 44, j = 41, 40, 47, 46 (note that the sequence number j needs to correspond to the key point of sequence number i), kpy i Indicates the vertical coordinate value of the key point with sequence number i.

[0085]

[0086] Among them, i=61,62,63, j=67,66,65 (note that the sequence number j needs to correspond to the key point of sequence number i).

[0087] Due to the differences in the sitting postures of different drivers and the installation positions of the infrared cameras, the facial sizes of different drivers in the infrared images will be different. In order to eliminate the influence of the dimension, the above distance needs to be divided by the corresponding width, that is:

[0088]

[0089] The above are the expressions for the eye opening and mouth opening, respectively. Where d1 and d2 represent the eye width and mouth width, respectively, which have been defined before. It refers to the degree of eye opening and mouth closing.

[0090] Define eye thresholds ε1, ε2 and mouth threshold η, where ε1 describes the boundary between a single frame of closed eyes and a single frame of squinting, ε2 describes the boundary between a single frame of squinting and a single frame of open eyes, and η describes the boundary between a single frame of yawning and a single frame of non-yawning. Their values ​​are set according to the empirical values ​​in the experiment.

[0091] The definition of single-frame eye state judgment is as follows:

[0092]

[0093] The state judgment of the mouth of a single frame is defined as follows:

[0094]

[0095] The length of the continuous frame sequence is T, and the eye opening and closing degrees and mouth opening and closing degrees in the T frames are counted. The driver's eye and facial states are judged by the number of consecutive T frames.

[0096] For the eye state judgment of consecutive T frames, the probability is defined as follows:

[0097]

[0098] For the mouth state judgment of consecutive T frames, the probability is defined as follows:

[0099]

[0100] Where T i (i=1, 2, 3) respectively represent the number of frames with single-frame closed eyes, single-frame squinting eyes, and single-frame open eyes in the continuous T frames, T i (i=4, 5) respectively represent the number of frames with single yawn and single non-yawn in the continuous T frames.

[0101] In some embodiments, a method of key point detection based on facial surface information can be used to obtain preliminary fatigue-related state values, such as closing eyes, squinting, and yawning. In order to further improve the detection effect of fatigue detection, high-level semantic features are extracted here, and combined features are extracted based on the facial ROI area, eye ROI area, and mouth ROI area, so that high-level semantic features have better representation capabilities for fatigue. For the facial ROI area, eye ROI area, and mouth ROI area, high-level features are extracted according to a multi-layer convolutional network, and the extracted high-level features are fused to obtain combined features, including:

[0102] Inputting the facial ROI region into the first multi-layer convolution layer in the feature extraction model to extract the first high-level feature; inputting the eye ROI region into the second multi-layer convolution layer in the feature extraction model to extract the second high-level feature; inputting the mouth ROI region into the third multi-layer convolution layer in the feature extraction model to extract the third high-level feature, wherein the weight parameters are shared among the first multi-layer convolution layer, the second multi-layer convolution layer and the third multi-layer convolution layer;

[0103] The first high-level feature, the second high-level feature and the third high-level feature are combined to obtain a combined feature of the frame image.

[0104] The overall network structure is used for fatigue data training and network prediction, and the network structure includes a feature extraction network and a feedback timing network structure. The overall network structure is as follows Figure 3 shown.

[0105] In some embodiments, the combined features of multiple consecutive frames of images are input into a feedback temporal network to perform feature feedback to obtain an initial fatigue state, including:

[0106] The combined features of images of multiple consecutive frames form a time series sequence of preset length. In each time series window, the feedback time series network outputs the output time series features, fatigue prediction probability value and non-fatigue probability prediction value in the current time series window, and the output time series features in the current time series window and the time series sequence to be input in the next time series window are simultaneously input into the feedback time series network in the next time series window until the inference is iterated a preset number of times, and the initial fatigue state is output, and the initial fatigue state includes the initial fatigue prediction probability value and the initial non-fatigue probability prediction value. The preset number of times is the same as the number of images of multiple consecutive frames.

[0107] In some embodiments, the training of fatigue data includes:

[0108] The training set consists of N driver video clips of the same length, including fatigue driving video clip samples, and the length of each video clip is n (n>T);

[0109] These video clips are converted into image sequences of length T, and the sample size of the converted training set is N×(n-T+1). The label corresponding to each image sequence includes video ID, image sequence ID and fatigue status.

[0110] Specifically, the videos used for training are all about 1 minute long, from the driver monitoring platform, and the video frame rate is 15FPS, that is, the length n of each video is 900 frames. Assuming the time sequence length T is 50, after the video is cut into picture sequences, a single video can be cut into 851 picture sequences of length 50. For the label representation of each sequence sample, such as the label of the first sequence is {000001,001,0}, where the first value 000001 represents the sequence number of the video, the second value 001 represents the sequence number of the picture sequence corresponding to the video after it is cut into picture sequences, and the third value 0 indicates that there is no fatigue in the picture sequence of length 50.

[0111] Set the training batch length to batch = 128. During training, perform shuffle operations according to the video sequence number, randomly select batch videos, and slide the window in sequence according to the image sequence corresponding to the batch videos until the entire video length is traversed. For the first image sequence S1 of length T of each video, the "previous temporal feature" (denoted as F0) used as input in its network structure is initialized to a full 0 matrix of the same dimension as the high-level combined feature, and the temporal output feature of the network structure is stored as the current temporal feature F1; for the subsequent image sequences of each video, the "previous temporal features" are F1, F2, ..., F 850 wait.

[0112] The training function used in the above model training is the cross entropy loss function, which is expressed as follows:

[0113]

[0114] Among them, y i represents the fatigue state label value of sample i, the positive class is 1 and the negative class is 0; p i It represents the probability that sample i is predicted to be a positive class.

[0115] In some embodiments, in the feature extraction network, the parameters of multiple convolutional layers of different image inputs are shared, which can greatly reduce the GPU usage during training and also achieve lightweight subsequent deployment.

[0116] In some embodiments, after completing the model training, for the model inference stage, the above network structure needs to be exported into two sub-models, one is a feature extraction sub-model, and the other is a time series classification sub-model. Since the weights are shared between the multi-layer convolutional networks in the high-level feature extraction module, when deriving the feature extraction sub-model, only one feature extraction network branch in the above network structure needs to be exported as a sub-model.

[0117] After the driver's infrared image is detected through key points to determine the facial ROI area, eye ROI area and mouth ROI area, it is input into the feature extraction sub-model to obtain high-level features feature1, feature2 and feature3 respectively, and then they are combined to obtain a combined feature with stronger representation ability (denoted as f i , i=1,2,…,T). The above feature combination method adopts the feature concatenation method, but is not limited to other commonly used feature combination methods, including feature superposition (add) and weighted superposition (weight add).

[0118] In some embodiments, the initial eye state, the initial mouth state, and the initial fatigue state are weighted and integrated to obtain a final fatigue estimation result, including:

[0119] Performing a weighted combination of the initial fatigue prediction probability value in the initial fatigue state and the eye-closing state value and the eye-squinting state value in the initial eye state to obtain a final state value of eye-closing fatigue;

[0120] Performing a weighted combination of the initial fatigue prediction probability value in the initial fatigue state and the yawning state value in the initial mouth state to obtain a yawning fatigue final state value;

[0121] The fatigue estimation result is obtained according to the magnitude relationship between the final state value of eye-closing fatigue and the first threshold, or the magnitude relationship between the final state value of yawning fatigue and the second threshold.

[0122] In some embodiments, the design structure of the feedback timing network is shown in the structural diagram of the overall network structure, that is, Figure 3 As shown, in the inference phase, it is derived from the network structure as a time series classification sub-model.

[0123] According to the sequence length T, after high-level feature extraction for each frame, a set of combined features f can be obtained i (i=1,2,…,T). Set the initial state of the “previous time series feature” in the time series network sub-model to a matrix of all zeros, and combine it with the combined feature f i Input them into the time series network together to obtain the output time series feature F1 and the fatigue prediction probability value and non-fatigue probability prediction value In the next timing window, F1 and f i (i=2,3,…,T+1) is used as the input of the model to obtain the output time series feature F2 and the fatigue prediction probability value and non-fatigue probability prediction value By performing reasoning iterations in sequence, we can obtain that at the tth reasoning, the input of the feedback timing network is F t-1 With f i (i=t,t+1,…,T+t-1), and its model inference output is the time series feature F t And fatigue prediction probability value and non-fatigue probability prediction value

[0124] In some embodiments, the fatigue estimation result is obtained according to the magnitude relationship between the final state value of eye-closing fatigue and the first threshold, or the magnitude relationship between the final state value of yawning fatigue and the second threshold. The structure diagram is as follows: Figure 2 As shown, including:

[0125] The fatigue-related state value of the continuous T frames obtained by the key point detection based on the facial surface information, that is, the yawning state value p m , closed eyes state value p a , squinting state value p b , and the fatigue state value p of the continuous T frames obtained by the feedback timing network based on the high-level features of multiple ROI regions of the face u , and weighted combination of them is expressed as follows:

[0126] p x =w 11 p a +w 12 p b +w 13 p u

[0127] p y =w 21 p m +w 22 p u

[0128] Among them, p x Indicates the final state value of eye-closing fatigue, p y Indicates the final state value of yawning fatigue, w ij (i=1,2,j=1,2,3) are weight coefficients set based on empirical values ​​in the experiment.

[0129] The final fatigue judgment result is:

[0130]

[0131] Among them, ε1 is the first threshold for judging eye fatigue, ε2 is the second threshold for judging yawning fatigue, and p x Indicates the final state value of eye-closing fatigue, p y Indicates the final state value of yawning fatigue, w ij (i=1,2,j=1,2,3) are weight coefficients set based on empirical values ​​in the experiment.

[0132] z=1 indicates that the prediction result is fatigue, and z=0 indicates that the prediction result is non-fatigue.

[0133] The driver fatigue monitoring method provided in the above embodiment is based on the combined feature feedback time series network and the fatigue detection technology of multi-strategy fusion, which can detect the driver's fatigue characteristics more effectively and accurately, making fatigue detection more stable and better ensuring the driver's driving safety. The technology has good generalization, strong versatility and wide applicability. And the technical solution solves the following problems:

[0134] 1. Construct ROI region combination features to make high-level semantics more representative and better learn fatigue features.

[0135] 2. The key point detection outputs multiple categories and adds a "squinting" category, so that all possible fatigue values ​​can participate in subsequent weighted calculations to reduce fatigue missed detection situations.

[0136] 3. Design a feedback timing network to improve the existing timing structure and make fatigue prediction of consecutive frames more stable.

[0137] 4. Construct a weighted combination to integrate the fatigue values ​​of multiple strategies to make fatigue prediction more accurate.

[0138] In some embodiments, a computer device is provided, such as Figure 5 As shown, it includes a memory 201, a processor 202, and a computer program stored in the memory 201 and executable on the processor 202. When the processor 202 executes the computer program, the driver fatigue monitoring method of any of the above embodiments is implemented.

[0139] Specifically, the computer device may be a computer terminal, a server or a similar computing device.

[0140] In some embodiments, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a method for monitoring driver fatigue as described in any of the above embodiments.

[0141] Specifically, computer-readable storage media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer-readable storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable storage media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0142] Based on the same inventive concept, a driver fatigue monitoring device is also provided in an embodiment of the present invention, as described in the following embodiments. Since the principle of solving the problem by a driver fatigue monitoring device is similar to that of a driver fatigue monitoring method, the implementation of a driver fatigue monitoring device can refer to the implementation of the driver fatigue monitoring method, and the repeated parts will not be repeated. As used below, the term "unit" or "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0143] A driver fatigue monitoring device, the structural block diagram is as follows Figure 6 As shown, it includes: an image acquisition module, a feature screening module, a feedback timing module and a fatigue judgment module.

[0144] An image acquisition module 301 is used to acquire the upper body image of the driver in each frame;

[0145] The feature screening module 302 is provided with a facial key point detection system, which is used to determine the facial ROI area, eye ROI area and mouth ROI area of ​​the frame image through the key points of the driver of the frame image obtained by the image acquisition module, judge the eye state and mouth state of the frame image, and input the facial ROI area, eye ROI area and mouth ROI area of ​​the frame image, extract the high-level features of each area, and fuse the high-level features of each area to obtain the combined features of the frame image;

[0146] A feedback timing module 303 is used to perform feature feedback on the combined features of multiple consecutive frames to obtain an initial fatigue state;

[0147] The fatigue judgment module 304 is used to determine the initial eye state and the initial mouth state according to the eye state and the initial mouth state of multiple consecutive frames of images, and weightedly integrate the initial eye state, the initial mouth state and the initial fatigue state to obtain a final fatigue estimation result.

[0148] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present invention are not limited to any specific combination of hardware and software.

[0149] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A driver fatigue monitoring method, characterized in that: include: For each frame of the driver's upper body image, the facial key points of the driver in the frame image are obtained through a facial key point detection network, the facial ROI area, the eye ROI area and the mouth ROI area of ​​the frame image are determined based on the facial key points, and the eye state and the mouth state of the frame image are judged based on the facial key points; The face ROI region, the eye ROI region and the mouth ROI region of the frame image are input into the feature extraction model to extract the high-level features of each region, and the high-level features of each region are fused to obtain the combined features of the frame image; The combined features of multiple consecutive frames of images are input into the feedback temporal network to perform feature feedback prediction and obtain the initial fatigue state; Determine an initial eye state and an initial mouth state according to the eye states and the mouth states of the continuous multiple frames of images; The initial eye state, the initial mouth state and the initial fatigue state are weighted and integrated to obtain a final fatigue estimation result.

2. The driver fatigue monitoring method according to claim 1, characterized in that: The determining of the face ROI region, the eye ROI region and the mouth ROI region of the frame image based on the face key points includes: The facial ROI area of ​​the frame image is determined based on the minimum circumscribed rectangle formed by all facial contour key points among the facial key points, the eye ROI area is determined based on the minimum circumscribed rectangle formed by all eye key points among the facial key points, and the mouth ROI area is determined based on the minimum circumscribed rectangle formed by all mouth key points among the facial key points.

3. The driver fatigue monitoring method according to claim 1, characterized in that: The method comprises: determining a facial ROI region of the frame image based on a minimum bounding rectangle formed by all facial contour key points among the facial key points, determining an eye ROI region based on a minimum bounding rectangle formed by all eye key points among the facial key points, and determining a mouth ROI region based on a minimum bounding rectangle formed by all mouth key points among the facial key points, including: The facial ROI area, eye ROI area and mouth ROI area are determined by the following formulas: Region(eye)=Rect(eye)+1 Region(mouth)=Rect(mouth)+2 Region(face)=Rect(face)+3 D1={(-0.21,-0.2d1),(0.21,0.2d1)} D2={(-0.22,-0.22),(0.22,0.2d2)} D3={(-0.23,-0.24),(0.23,0.2d4)} Among them, Region(eye) is the eye ROI area, Region(mouth) is the mouth ROI area, and Region(face) is the face ROI area. i (i=1,2,3) respectively represent the expansion of the regions of Rect(eye), Rect(mouth), and Rect(face), so that the three ROI regions can include the eyes, mouth, face, and nearby pixels. d1 represents the eye width, that is, the width of the minimum circumscribed rectangle of the eye Rect(eye); d2 represents the mouth width, that is, the width of the minimum circumscribed rectangle of the mouth Rect(mouth); d3 represents the face width, that is, the width of the minimum circumscribed rectangle of the face Rect(face); d4 represents the face height, that is, the height of the minimum circumscribed rectangle of the face Rect(face).

4. The driver fatigue monitoring method according to claim 1, characterized in that: The face ROI region, the eye ROI region and the mouth ROI region of the frame image are input into the feature extraction model, high-level features of the respective regions are extracted, and the high-level features of the respective regions are fused to obtain the combined features of the frame image, including: Inputting the facial ROI region into the first multi-layer convolution layer in the feature extraction model to extract the first high-level feature; inputting the eye ROI region into the second multi-layer convolution layer in the feature extraction model to extract the second high-level feature; inputting the mouth ROI region into the third multi-layer convolution layer in the feature extraction model to extract the third high-level feature, wherein the weight parameters are shared among the first multi-layer convolution layer, the second multi-layer convolution layer and the third multi-layer convolution layer; The first high-level feature, the second high-level feature and the third high-level feature are combined to obtain a combined feature of the frame image.

5. The driver fatigue monitoring method according to claim 1, characterized in that: The combined features of the continuous multi-frame images are input into the feedback temporal network to perform feature feedback to obtain the initial fatigue state, including: The combined features of images of multiple consecutive frames form a time series sequence of preset length. In each time series window, the feedback time series network outputs the output time series features, fatigue prediction probability value and non-fatigue probability prediction value in the current time series window, and the output time series features in the current time series window and the time series sequence to be input in the next time series window are simultaneously input into the feedback time series network in the next time series window until the inference is iterated a preset number of times, and the initial fatigue state is output, and the initial fatigue state includes the initial fatigue prediction probability value and the initial non-fatigue probability prediction value. The preset number of times is the same as the number of images of multiple consecutive frames.

6. The driver fatigue monitoring method according to claim 1, characterized in that: The initial eye state, the initial mouth state and the initial fatigue state are weighted and integrated to obtain a final fatigue estimation result, including: Performing a weighted combination of the initial fatigue prediction probability value in the initial fatigue state and the eye-closing state value and the eye-squinting state value in the initial eye state to obtain a final state value of eye-closing fatigue; Performing a weighted combination of the initial fatigue prediction probability value in the initial fatigue state and the yawning state value in the initial mouth state to obtain a yawning fatigue final state value; The fatigue estimation result is obtained according to the magnitude relationship between the final state value of eye-closing fatigue and the first threshold, or the magnitude relationship between the final state value of yawning fatigue and the second threshold.

7. The driver fatigue monitoring method according to claim 1, characterized in that: The fatigue estimation result is obtained according to the magnitude relationship between the final state value of eye-closing fatigue and the first threshold, or the magnitude relationship between the final state value of yawning fatigue and the second threshold, including: Among them, ε1 is the first threshold for judging eye fatigue, ε2 is the second threshold for judging yawning fatigue, and p x Indicates the final state value of eye-closing fatigue, p y Indicates the final state value of yawning fatigue, z=1 indicates that the fatigue estimation result is fatigue, and z=0 indicates that the fatigue estimation result is non-fatigue.

8. A driver fatigue monitoring device, characterized in that: include: An image acquisition module, used for acquiring the upper body image of the driver in each frame; The feature screening module is provided with a facial key point detection system, which is used to determine the facial ROI area, eye ROI area and mouth ROI area of ​​the frame image through the key points of the driver of the frame image obtained by the image acquisition module, judge the eye state and mouth state of the frame image, and input the facial ROI area, eye ROI area and mouth ROI area of ​​the frame image, extract the high-level features of each area, and fuse the high-level features of each area to obtain the combined features of the frame image; The feedback timing module is used to perform feature feedback on the combined features of multiple consecutive frames to obtain the initial fatigue state; The fatigue judgment module is used to determine the initial eye state and the initial mouth state according to the eye state and the mouth state of multiple consecutive frames of images, and to weight and integrate the initial eye state, the initial mouth state and the initial fatigue state to obtain the final fatigue estimation result.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the driver fatigue monitoring method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a method for executing the driver fatigue monitoring method according to any one of claims 1 to 7.

Images