A method for detecting driver fatigue in logistics transportation

Abstract

The invention relates to a method for detecting fatigue driving of a logistics transportation driver. During driving, a driver image is captured in real time. When eye and mouth features are identified at the same time, the number of times the driver's eyes are closed and the number of times the driver's mouth is opened within a T period are identified and counted. When the number of times the driver closes his eyes reaches a preset number of times or the number of times the driver opens his mouth reaches a preset number of times within a period, it is considered that the driver is in a fatigue state. When only eye features can be identified but not mouth features, an eye region image is intercepted, and a facial expression image is automatically matched for the eye region image according to the eye expression. Then, an area in the driver's real-time facial image that may cause a mask shape change, i.e., a detection area, is determined according to a facial shape change area of ​​the matched facial expression image. Then, whether the driver yawns is determined according to a change in the shape of mask stripes in the detection area. The driver is considered to be fatigued.

Description

Technical Field

[0001] The present invention relates to the technical field of facial image recognition and processing, and in particular to a method for detecting fatigue driving of logistics transportation drivers. Background Art

[0002] Fatigue driving refers to dangerous driving behavior caused by mental and physical fatigue when the driver drives continuously for a long time or lacks sufficient rest. Fatigue driving can cause the driver to lose concentration, react slowly, and have reduced judgment, which in turn affects driving skills and traffic safety. According to statistics, fatigue driving is one of the main causes of traffic accidents and cannot be ignored.

[0003] In the logistics and transportation industry, traffic accidents occur every year due to fatigue driving, causing serious casualties and economic losses.

[0004] In order to prevent fatigue driving, logistics and transportation companies generally install video monitoring equipment in the cab, capture the driver's facial image during driving, and use face recognition technology to identify the driver's eye, mouth movements and facial expressions to determine whether the driver is fatigued. When fatigue driving is detected, the monitoring equipment will issue a sound prompt or the rear management will call to remind the driver to stop and rest. This method can effectively prevent fatigue driving.

[0005] However, the video surveillance equipment also has an obvious defect. When the driver wears a mask, it is impossible to identify whether the driver opens his mouth, which will affect the accuracy of fatigue judgment. This defect needs to be improved. Summary of the invention

[0006] Based on the above statements, the present invention provides a method for detecting fatigue driving of logistics transportation drivers, which can detect whether the driver is driving fatigued when the driver wears a mask.

[0007] The technical solution of the present invention to solve the above technical problems is as follows:

[0008] A method for detecting driver fatigue in logistics transportation, which captures driver images in real time during driving, identifies and tracks the facial images in the driver images, and identifies the features of eyes and mouth in the facial images;

[0009] When the features of eyes and mouth are recognized at the same time, the number of times the driver's eyes are opened and closed and the number of times the driver's mouth is opened in the T period are recognized and counted. When the number of times the driver closes his eyes reaches the preset number of times or the number of times he opens his mouth reaches the preset number of times in a period, the driver is considered to be in a fatigue state, and a reminder message is sent to the rear management personnel at this time;

[0010] When only the eye features can be recognized but the mouth features cannot be recognized, a rectangular image area S1 within a certain range of the driver's eye area is captured, wherein the upper part of the mask enters the S1 area; the eye expression in the S1 area image is recognized and the corresponding facial expression image is matched according to the recognition result; various facial expressions are compared with the standard facial expressions in advance, and the change areas of various facial expressions are calibrated accordingly; after obtaining the currently matched facial expression image, the corresponding change area is automatically selected; then a detection frame is defined in the driver's real-time facial image according to the center point coordinates and coverage of the selected change area; the mask image in the detection frame is captured to obtain a mask block, and then the mask block is processed to identify the stripe features in the block, and the stripe shape is judged by comparing the stripe features. When it is judged that the stripe shape has changed, it is considered that the driver is yawning, and an early warning message is sent to the rear management personnel.

[0011] As a preferred solution: when processing the mask block, the mask block is first gray-scaled to reflect the stripes of the mask block, and then the stripe segments in the mask block are extracted, and one of the extracted stripe segments is selected as the target stripe segment; three points A, B, and C are evenly selected on the target stripe segment and the coordinate position of each point is obtained, and then the slope k1 between points A and B and the slope k2 between points B and C are calculated, that is, the stripe features of the mask in the current frame driver's facial image are obtained; the slopes k1 of the target stripe segments in the previous and next two frames of images are compared, and the slopes k2 of the target stripe segments in the previous and next two frames of images are compared. When the rate of change of the slope k1 or k2 exceeds its preset reference value, it is considered that the shape of the target stripe has changed.

[0012] As a preferred solution: the S1 area image is also gray-processed to reflect the outline of the upper edge of the mask; then the contour line of the upper edge of the mask is extracted, the highest point on the contour line is defined as the position point X0 of the mask and the coordinates of point X0 are obtained; similarly, the coordinates of the inner end points P1 and P2 of the driver's eyes can be calculated, and then the coordinates of the center point P0 between P1 and P2 are calculated, and P0 is defined as the reference point; then the calculation unit calculates the vertical distance L between the position point X0 and the reference point P0; the comparison unit compares the L values ​​in the previous and next two frames of the S1 area image. When the rate of change of the L value exceeds its preset reference value, it is considered that the height position of the mask has changed significantly. At this time, it is judged that the driver is yawning, and an early warning message is sent to the rear management personnel.

[0013] As a preferred solution: a recognition model is pre-built to identify the open and closed state of the eyes and whether the mouth is open. After the recognition model is built, a large number of eye image samples are provided, and these samples are divided into open and closed categories according to the degree of eye opening. The classified eye image samples are input into the recognition model, and the recognition model is trained. The trained recognition model can accurately identify the open and closed state of the eyes; a large number of mouth image samples are provided, and these samples are divided into open and closed categories according to the degree of mouth opening. The classified mouth images are input into the recognition model to complete the model training. The trained recognition model can accurately identify the open and closed state of the mouth.

[0014] As a preferred solution, the specific implementation process of matching facial expressions is: pre-building an expression matching model, the expression matching model establishes a matching library of eye area expressions and facial expressions based on a large amount of facial image data, when the S1 area image is input into the expression matching model, the expression matching model recognizes the eye expression in the S1 area and automatically selects the facial expression image with the highest matching pair based on the recognition result.

[0015] As a preferred solution, the implementation process of delineating a detection frame in the driver's real-time facial image is as follows: various facial expressions are compared in advance with the facial expressions in a standard state, i.e., the standard facial expressions, and the areas where the facial shapes of the various facial expressions change significantly, i.e., the change areas, are calibrated accordingly. The calibration content includes the center point coordinates and the coverage range of the change area. After obtaining the matching facial expression image, the corresponding change area can be automatically selected; then, a detection frame is delineated in the driver's real-time facial image based on the center point coordinates and the coverage range of the selected change area. The center point coordinates and the coverage range of the detection frame are the center point coordinates and the coverage range of the change area.

[0016] Compared with the prior art, the technical solution of the present application has the following beneficial technical effects: the present application divides drivers into two situations: wearing masks and not wearing masks. When the driver does not wear a mask, it is directly judged whether the driver is driving fatigued by identifying and counting the frequency of closing eyes and opening mouth. When the driver's face is blocked by wearing a mask, the eye area image is intercepted, and the facial expression image is automatically matched to the eye area image according to the eye expression, and then the area in the driver's real-time facial image that may cause the mask shape to change, that is, the detection area, is determined according to the facial shape change area of ​​the matched facial expression image, and then the driver is judged whether he is yawning according to the change in the shape of the mask stripes in the detection area; in addition, the present application also judges whether the driver is yawning by detecting the change in the position of the mask. Through this application, it is possible to detect whether the driver is driving fatigued when the driver is wearing a mask. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a block diagram of the detection system in this embodiment;

[0018] Figure 2 is a schematic diagram of a driver image in this embodiment;

[0019] Figure 3 is a schematic diagram of an image of the driver's eye area in this embodiment;

[0020] Figure 4 Schematic diagram of a mask detection frame in this embodiment;

[0021] Figure 5 Schematic diagram of the mask block in this embodiment.

[0022] 1. Eyes; 2. Mask; 3. Stripes. DETAILED DESCRIPTION

[0023] Reference Figure 1 A method for detecting fatigue driving of logistics transportation drivers is based on a fatigue driving detection system. The fatigue driving detection system in this embodiment includes a camera module, an image acquisition module, a storage module, a face recognition module, a first image capture module, an expression matching model, a positioning matching module, a second image capture module, an image processing module, a judgment and analysis module, and an abnormal alarm module.

[0024] The camera module is installed in the cab and located in front of the driver's face. During driving, the camera module captures the driver's image in real time; the image acquisition module is used to collect and transmit image data obtained by the camera; and the storage module is used to store image data.

[0025] The face recognition module identifies and tracks the facial image in the driver image, and the face recognition module identifies the eye and mouth features in the facial image. When the driver is not wearing a mask, the driver's eyes and mouth can be identified.

[0026] When the face recognition module recognizes the features of eyes and mouth at the same time, the judgment and analysis module identifies and counts the number of times the driver's eyes are closed and the number of times his mouth is opened within the T period.

[0027] The judgment and analysis module in this embodiment includes a pre-built recognition model. After the recognition model is built, a large number of eye image samples need to be provided in advance. These samples are divided into two categories: open and closed according to the degree of eye opening. The classified eye image samples are input into the recognition model, and the recognition model is trained. The trained recognition model can accurately identify the open and closed state of the eyes.

[0028] Similarly, a large number of mouth image samples are required, and these samples are divided into open and closed categories according to the degree of mouth opening. The classified mouth images are input into the recognition model to complete the model training. The trained recognition model can accurately identify the opening and closing state of the mouth.

[0029] The duration of the cycle T in this embodiment is set to 5 minutes. When the face recognition model recognizes the driver's eye and mouth features at the same time, the first image capture module captures images of the eye area and mouth area from the current facial image at regular intervals, and inputs the captured eye images and mouth images into the recognition model. The recognition model recognizes and counts the number of times the eyes and mouth are opened and closed within a cycle T.

[0030] When the driver closes his eyes the number of times he closes his eyes reaches the preset number of times or opens his mouth the number of times he opens his mouth within a cycle, the judgment and analysis module deems that the driver is in a fatigued state, and the abnormal alarm module sends a reminder message to the rear management personnel.

[0031] When the driver wears a mask, the face recognition module can only recognize the eye features but not the mouth features. At this time, the first image capture module captures a rectangular image area S1 (such as Figure 2 As shown in the dashed box, when the mask is worn normally, the upper part of the mask enters the S1 area). The captured S1 area image is input into the expression matching model, and the expression matching model establishes a matching library of eye area expressions and facial expressions (the facial expressions referred to in this embodiment refer to the expressions of the facial area below the eye area) based on a large amount of facial image data. When the S1 area image is input into the expression matching model, the expression matching model recognizes the eye expression in the S1 area and automatically selects the facial expression image with the highest matching pair based on the recognition result; then the expression matching model transmits the selected facial expression image to the image positioning module, and the image positioning module compares various facial expressions with the facial expressions in the standard state, that is, the standard facial expressions, in advance, and calibrates the area where the facial shape under various facial expressions changes significantly, that is, the change area. The calibration content includes the coordinates of the center point of the change area (taking the center point P0 of the inner end points P1 and P2 of the two eyes in the eye area image as the reference point, as shown in FIG. Figure 3 After receiving the current facial expression image, the image positioning module automatically selects the corresponding change area.

[0032] The image positioning module then defines a detection frame (such as Figure 4 As shown in the S2 area, Figure 4 Each dotted box in the figure represents a detection box). The center point coordinates and coverage range of the detection box are the center point coordinates and coverage range of the change area. The detection box is located within the coverage range of the mask.

[0033] The second image capture module captures the mask image in the detection frame to obtain a mask block (such as Figure 5). The image processing module then processes the mask block to identify the stripe features in the block, and determines whether the stripe shape has changed by comparing the stripe features. When it is determined that the stripe has changed, it is considered that the driver is yawning, and the abnormal alarm module sends a warning message to the rear management personnel.

[0034] The image processing module in this embodiment includes a preprocessing unit, a feature extraction unit, a calculation unit and a comparison unit.

[0035] The preprocessing unit is used to perform grayscale processing on the mask block to reflect the stripes of the mask block; the feature extraction unit is used to extract the stripe segments in the mask block; one of the extracted stripe segments is selected as the target stripe segment; the calculation unit is used to evenly select three points A, B, and C on the target stripe segment (that is, the target stripe segment is evenly divided into four segments in the horizontal coordinate direction, and three points can be obtained, such as Figure 5 As shown). The calculation unit can then obtain the coordinate position of each point according to the pixel position corresponding to each point of the center point coordinate of the block. Then the calculation unit calculates the slope between points A and B and the slopes k1 and k2 between points B and C, so as to obtain the stripe features of the mask in the current frame driver's facial image (the stripe features referred to in this embodiment are the slopes k1 and k2 of the target stripe segments).

[0036] The comparison unit compares the slope k1 of the target stripe segment in the two frames of images, and compares the slope k2 of the target stripe segment in the two frames of images. When the change rate of the slope k1 or k2 exceeds its preset reference value, it is considered that the shape of the target stripe has changed significantly, and it is judged that the driver is yawning, and the abnormal alarm module sends a warning message to the rear management personnel.

[0037] In this embodiment, whether the driver is yawning is also determined by detecting changes in the overall position of the mask.

[0038] Specifically: when the driver wears a mask, the face recognition module can only recognize the eye features but not the mouth features. At this time, the first image capture module captures a rectangular image area S1 within a certain range of the driver's eye area. The captured S1 area image is also transmitted to the image processing module. The preprocessing unit performs grayscale processing on the S1 area image so that the contour of the upper edge of the mask can be clearly reflected; then the feature extraction unit extracts the contour line of the upper edge of the mask, and defines the highest point on the contour line as the position point of the mask, as shown in the X0 point in the figure. The calculation unit calculates the coordinates of point X0 based on the coordinates of the center point of the original image of the driver's face and the relative pixel position between point X0 and the center point of the original image. Similarly, the coordinates of the inner end points P1 and P2 of the driver's eyes can be calculated, and then the coordinates of the center point P0 between P1 and P2 are calculated, and P0 is defined as the reference point; then the calculation unit calculates the vertical distance L between the position point X0 and the reference point P0; the comparison unit compares the L values ​​in the two frames of S1 area images before and after. When the change rate of the L value exceeds its preset reference value, it is considered that the height position of the mask has changed significantly. At this time, it is judged that the driver is yawning, and the abnormal alarm module sends a warning message to the rear management personnel.

[0039] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for detecting fatigue driving of logistics transportation drivers, characterized by: The driver's image is captured in real time during driving, the facial image in the driver's image is recognized and tracked, and the eye and mouth features in the facial image are recognized; When the eye and mouth features are recognized at the same time, the number of times the driver's eyes are closed and the number of times the mouth is opened in the T period are recognized and counted. When the number of times the driver closes his eyes reaches the preset number of times or the number of times he opens his mouth reaches the preset number of times in a period, the driver is considered to be in a fatigue state, and a reminder message is sent to the rear management personnel at this time; When only the eye features can be identified but the mouth features cannot be identified, a rectangular image area S1 within a certain range of the driver's eye area is intercepted, wherein the upper part of the mask enters the S1 area; the eye expression in the S1 area image is identified and the corresponding facial expression image is matched according to the identification result; various facial expressions are compared with the standard facial expressions in advance, and the change areas of various facial expressions are calibrated accordingly; after obtaining the currently matched facial expression image, the corresponding change area is automatically selected; then a detection frame is defined in the driver's real-time facial image according to the center point coordinates and coverage of the selected change area; the mask image in the detection frame is intercepted to obtain a mask block, and then the mask block is processed to identify the stripe features in the block, and the stripe shape is judged by comparing the stripe features. When it is judged that the stripe shape has changed, it is considered that the driver is yawning, and an early warning message is sent to the rear management personnel at this time; When processing the mask block, the mask block is firstly gray-scaled to reflect the stripes of the mask block, and then the stripe line segments in the mask block are extracted, and one of the stripe line segments extracted is selected as the target stripe line segment; Uniformly select three points A, B, and C on the target stripe line segment and obtain the coordinates of each point. Then calculate the slope k1 between points A and B and the slope k2 between points B and C, and obtain the stripe features of the mask in the driver's facial image of the current frame. Compare the slopes k1 of the target stripe segments in the two frames of images, and compare the slopes k2 of the target stripe segments in the two frames of images. When the change rate of the slope k1 or k2 exceeds a preset reference value, it is considered that the shape of the target stripe has changed. The S1 area image is also gray-scaled to reflect the outline of the upper edge of the mask; then the contour line of the upper edge of the mask is extracted, the highest point on the contour line is defined as the position point X0 of the mask, and the coordinates of the X0 point are obtained; similarly, the coordinates of the inner end points P1 and P2 of the driver's eyes can be calculated, and then the coordinates of the center point P0 between P1 and P2 are calculated, and P0 is defined as the reference point; then the calculation unit calculates the vertical distance L between the position point X0 and the reference point P0; the comparison unit compares the L values ​​in the previous and next two frames of the S1 area image. When the rate of change of the L value exceeds its preset reference value, it is considered that the height position of the mask has changed significantly. At this time, it is judged that the driver is yawning, and an early warning message is sent to the rear management personnel.

2. The method for detecting fatigue driving of logistics transportation drivers according to claim 1 is characterized by: A recognition model is pre-built to identify the open and closed state of the eyes and whether the mouth is open. After the recognition model is built, a large number of eye image samples are provided. These samples are divided into open and closed categories according to the degree of eye opening. The classified eye image samples are input into the recognition model to complete the training of the recognition model. The trained recognition model can accurately identify the open and closed state of the eyes; a large number of mouth image samples are provided. These samples are divided into open and closed categories according to the degree of mouth opening. The classified mouth images are input into the recognition model to complete the training of the model. The trained recognition model can accurately identify the open and closed state of the mouth.

3. The method for detecting fatigue driving of logistics transportation drivers according to claim 1 is characterized in that: The specific implementation process of matching facial expressions is as follows: pre-build an expression matching model, and the expression matching model establishes a matching library of eye area expressions and facial expressions based on a large amount of facial image data. When the S1 area image is input into the expression matching model, the expression matching model recognizes the eye expression in the S1 area and automatically selects the facial expression image with the highest matching pair based on the recognition result.

4. The method for detecting fatigue driving of logistics transportation drivers according to claim 3 is characterized in that: The implementation process of delineating the detection frame in the real-time facial image of the driver is as follows: various facial expressions are compared with the facial expressions in the standard state, i.e., the standard facial expressions, in advance, and the areas where the facial shapes of the various facial expressions change significantly, i.e., the change areas, are calibrated accordingly. The calibration content includes the center point coordinates and the coverage range of the change area. After obtaining the matching facial expression image, the corresponding change area can be automatically selected; then, the detection frame is delineated in the real-time facial image of the driver according to the center point coordinates and the coverage range of the selected change area. The center point coordinates and the coverage range of the detection frame are the center point coordinates and the coverage range of the change area.

Images