Driving safety supervision method based on image detection

Abstract

The invention discloses a driving safety supervision method based on image detection. The method comprises the steps of driver identity recognition, driver and vehicle information matching and mood monitoring and fatigue monitoring in the driving process. According to the invention, the driver and the vehicle are in one-to-one correspondence, the accident risk caused by unfamiliarity with the vehicle is reduced, the psychological state and fatigue state of the driver are monitored in real time, an alarm is given, the driver is prompted to adjust the psychological state or rest in time, and thus the traffic safety accidents are effectively reduced.

Description

Technical Field

[0001] The present invention relates to the field of automated software testing, and in particular to a driving safety supervision method based on image detection. Background Art

[0002] Driving safety is of vital importance, among which the driver's mastery of the vehicle, the driver's psychological state and the management of fatigue driving are particularly important. First of all, the driver's familiarity with the vehicle is the basis for ensuring safe driving. Drivers who are not familiar with the vehicle cannot fully utilize the vehicle's performance or cope with complex driving environments, thereby increasing the possibility of accidents.

[0003] The driver's mental state is also an important factor in driving safety. Extreme mental states often lead to driving safety accidents. By detecting the driver's facial expressions, their mental state can be predicted, so that measures can be taken to ensure driving safety.

[0004] In addition, fatigue driving is another key factor in driving safety. Even if the driver knows the vehicle well, fatigue driving is still a hidden danger that cannot be ignored. Fatigue is a normal physiological reaction of the human body. When working for a long time or doing a single job for too long, the body will become tired. In the field of transportation, fatigue driving is one of the main causes of traffic accidents. Traffic safety accidents caused by fatigue occur frequently every year, posing a great hidden danger to the safety of life and property of the country and the people. Summary of the invention

[0005] The purpose of the present invention is to provide a driving safety supervision method based on image detection, which matches drivers with vehicles one by one, reduces the risk of accidents caused by unfamiliarity with vehicles, and monitors the driver's psychological state and fatigue state in real time, issues alarms, and prompts the driver to adjust his mentality and rest in time, thereby effectively reducing the occurrence of driving safety accidents.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A driving safety supervision method based on image detection comprises the following steps:

[0008] Step 1: Perform facial recognition and identity recognition on the driver, compare it with the information in the background database, determine the driver's identity information, and compare it with the information of the vehicle being driven. After confirming that it is the driver of the vehicle, allow the vehicle to be started;

[0009] Step 2: Detect and identify the license plate of the driver's vehicle at the vehicle entrance and exit, compare the detected vehicle information with the vehicle information in the system database, retrieve the driver information corresponding to the vehicle, and allow the vehicle to pass after comparing and confirming the driver information with the vehicle information;

[0010] Step 3: Recognize the driver's facial expression during driving, and remind the driver to drive safely if an abnormal mood is detected;

[0011] Step 4: During driving, the driver is tested for eye closure, mouth opening and head movement to comprehensively determine whether the driver is drowsy while driving the vehicle, and the driver is reminded through an alarm system.

[0012] Furthermore, the face recognition method described in step 1 adopts a feature matching-based method, and the face recognition algorithm is implemented as follows: using the EigenFaces algorithm, using the function in OpenCV

[0013] The EigenFaces recognizer instance model generated by cv2.face.EigenFaceRecognizer_create() is then trained with cv2.face_FaceRecognizer_train(), and finally face recognition is completed through cv2.face_FaceRecognizer_predict().

[0014] Furthermore, the identity recognition described in step one is divided into four steps: driver face collection, preprocessing, feature extraction, matching and recognition. First, the sample is collected using a cascade classifier based on Haar features, and then the image is preprocessed for feature extraction. Subsequently, the extracted features are compared with the information in the background database to achieve the identity recognition of the driver.

[0015] Furthermore, the license plate detection and recognition process in step 2 includes the following steps:

[0016] (1) Image input

[0017] Image input is connected to a computer or image acquisition card through a camera device for real-time acquisition to obtain image information of the original image;

[0018] (2) Pretreatment

[0019] The image to be identified is grayed and image noise is removed, and then the image is binarized, closed, eroded and expanded respectively;

[0020] (3) License plate positioning

[0021] First, a large-scale search is performed on the collected image to find candidate areas that match the characteristics of the license plate. Then, these candidate areas are analyzed in depth to determine the area that is most likely to be the license plate, and the area with the highest probability is cut out from the image;

[0022] (4) Character segmentation

[0023] After locating and extracting the license plate area, the vertical projection method is used to segment the license plate characters;

[0024] (5) Character Recognition

[0025] The template matching algorithm is used for character recognition. First, the segmented characters are binarized and the character size is adjusted to ensure that it is consistent with the template size in the database. Then matching recognition is performed and the result with the highest matching similarity is selected as the recognition result.

[0026] (6) Output the recognition result of the license plate

[0027] By calling the PIL library, the final recognition result is displayed on the collected original image, and the person and vehicle information is compared at the same time.

[0028] Furthermore, the process of driver facial expression recognition described in step three is: using Python language Pandas library and Kears framework to process data, using facial expression dataset fer2013 to build and train the model to achieve the purpose of facial expression detection, identifying and judging seven basic expressions: anger, disgust, fear, happy, sad, surprised and normal, and when the driver's expression is other than normal, the driver is reminded to drive safely.

[0029] Furthermore, the step 4 of determining whether the driver is drowsy while driving the vehicle mainly includes the following steps:

[0030] (1) Perform face recognition;

[0031] (2) Use the Dlib library file to locate key points of the collected facial information;

[0032] (3) Based on the location of key points on the face, the length-to-width ratio of the eyes and mouth is calculated to determine whether the eyes are closed or yawning; based on the head posture detection method, it is determined whether the head is nodding;

[0033] Define the human eye aspect ratio EAR, then Where P 37 is the left eye end point, P 38 is the upper left position of the left eye, P 39 is the upper right position of the left eye, P 40 is the left eye corner position point, P 41 is the lower right side of the left eye, P 42 is the point on the lower left side of the left eye; set the threshold of the EAR value, record the EAR values ​​of both eyes, and then take the average value as the final judgment value. When the judgment value of EAR is less than the set threshold, it is judged as closed eyes;

[0034] Define the mouth length-to-width ratio MAR, then Where P 49 is the left corner of the mouth, P 51 is the left side of the upper lip, P 53 is the right side point of the upper lip, P 55 is the right corner of the mouth, P 57 is the right side point of the lower lip, P 59 is the point on the left side of the lower lip. Similarly, the threshold of the MAR value is set. When the MAR judgment value is greater than the set threshold, it is judged as yawning.

[0035] In 3D space, there are three directions for the head to rotate: Yaw, Pitch, and Roll. Yaw is the yaw angle, which rotates around the Y axis; Pitch is the pitch angle, which rotates around the X axis; Roll is the roll angle, which rotates around the Z axis; when a nod occurs, Pitch changes the most, followed by Yaw, and Roll changes the least, and the corresponding X-axis parameter increases in the positive direction, which is judged as bowing the head; when the X-axis parameter returns from an integer to 0, the system determines that a nod has been performed;

[0036] (4) Determine whether the driver is drowsy based on the length of time the eyes are closed, the frequency of yawning, and the angle of the head nodding, and issue an alarm to remind the driver

[0037] ① For eye closure, the EAR threshold is pre-set to 0.2. When the EAR value is less than the set threshold, the frame counter starts counting. When the eye closure time is greater than 3 frames, it is judged as long-term eye closure and the number of eye closures is increased by one; when the number of eye closures is greater than 5 times, it means that the driver is drowsy and an alarm is issued;

[0038] ② For yawning, the MAR threshold is preset to 0.9. When the MAR value is greater than the preset threshold, it is judged as opening the mouth. When the frame counter is greater than 3 frames, the number of yawns increases by one. When the number of yawns is greater than 5 times, it means that the driver is sleepy and an alarm is issued;

[0039] ③ For nodding, the corresponding X-axis parameter increases in the positive direction when nodding, which is judged as lowering the head. When the X-axis parameter returns from an integer to 0, the system determines that a nod has been performed, and the number of nods increases by one; when the number of nods is greater than 5 times, it means that the driver is drowsy and an alarm is issued.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] The present invention introduces information and intelligent technology into the vehicle management system, provides a scientific management solution for matching driver and vehicle information, thereby reducing the risk of accidents caused by the driver's unfamiliarity with the vehicle. By using face detection and recognition technology, the driver's basic information can be accurately identified to prevent safety hazards caused by private vehicle replacement. In addition, the system can also predict the driver's psychological state by analyzing his facial expressions and help adjust the driver's mood in time. More importantly, the system monitors the driver's facial state, such as closing his eyes, yawning, etc., comprehensively judges his fatigue condition, and reminds the driver to rest through the alarm system, thereby effectively reducing the occurrence of traffic accidents.

[0042] Furthermore, the present invention adopts a feature matching-based method and an EigenFaces algorithm for face recognition, which improves the accuracy and efficiency of identity verification through precise feature matching. The driver's identity can be accurately verified by collecting, preprocessing, extracting features and matching the driver's face for identity recognition.

[0043] Furthermore, the license plate detection and recognition process of the present invention achieves efficient and accurate license plate recognition through systematic steps from image input to final result output. First, a clear license plate image is acquired through real-time image acquisition, and then a series of preprocessing steps are performed to remove noise and optimize image quality. Next, the system locates the license plate and accurately finds the license plate area, then recognizes the license plate characters through character segmentation and template matching algorithms, and finally visualizes the recognition results and compares them with database information. This process greatly improves the accuracy and real-time performance of license plate recognition and enhances the capabilities of vehicle management and safety monitoring.

[0044] Furthermore, the present invention uses Python's Pandas library and Keras framework to build and train the expression recognition model for the fer2013 dataset, which can monitor the driver's emotional state in real time and recognize seven basic expressions. When the driver's expression is detected to be an emotion other than normal, the system will issue a safe driving reminder, thereby enhancing the driver's safety awareness, reducing the risk of accidents caused by emotional instability, and improving the overall driving experience.

[0045] Furthermore, the present invention uses face recognition technology and key point positioning to calculate the aspect ratio of human eyes and mouth, so as to determine whether the driver has closed eyes or yawned, and to determine whether the driver nods by means of head posture detection method. By real-time monitoring and analysis of the driver's facial features, setting thresholds and counting mechanisms, the system can accurately identify the driver's fatigue state, and issue an alarm in time when signs of fatigue are detected (such as closing eyes for a long time, yawning many times or nodding frequently), helping to reduce the risk of accidents caused by driver fatigue and improving driving safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 The original image of the vehicle collected;

[0047] Figure 2 This is the image binarization effect diagram for the collected original vehicle image;

[0048] Figure 3 This is the effect diagram of the closing operation on the collected original vehicle image;

[0049] Figure 4 This is the effect diagram of corrosion and expansion operation on the collected original vehicle picture;

[0050] Figure 5 To capture the license plate position rendering;

[0051] Figure 6 This is the effect diagram of license plate character segmentation;

[0052] Figure 7 This is the license plate recognition effect diagram;

[0053] Figure 8 This is a schematic diagram of key points on a human face;

[0054] Fig. 9 Schematic diagram of key points of the right eye;

[0055] Fig.10 This is a diagram of the key points of the mouth. DETAILED DESCRIPTION

[0056] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is described in detail below in conjunction with specific embodiments.

[0057] The present invention provides a driving safety supervision method based on image detection, which is implemented using Python language and based on the OpenCV library, and includes the following steps:

[0058] Step 1: Perform facial recognition and identification on the driver, compare it with the information in the background database, determine the driver's identity information, and compare it with the information of the vehicle being driven. After confirming that it is the driver of the vehicle, allow the vehicle to be started.

[0059] Face recognition uses a camera to collect information about the driver's facial features. The collected information can be image information or video information. Then the collected information is scanned row by row, column by column, and region by region to detect whether the image or video contains general features of a human face, that is, a nose next to one eye and another eye, and a mouth and eyebrows around it. If these facial features have a certain position and area range, then this area can be determined to be a human face area and marked.

[0060] There are many methods for face recognition, such as knowledge-based, feature-based, template matching, appearance-based, etc. The present invention mainly adopts a method based on feature matching, which locates the face by analyzing the structure of the face and then is used to distinguish between the face and non-face area.

[0061] OpenCV provides a variety of trained classifiers that have certain recognition capabilities. Classifiers with different recognition functions are cascaded to form a cascade classifier. The face recognition classifier faceCascade is used to locate the position of the face, and the eye classifier eyeCascade is used to locate the position of the eye, thereby achieving more accurate face positioning based on face recognition.

[0062] Face recognition is implemented through the EigenFaces algorithm:

[0063] EigenFaces is also called eigenface. It uses principal component analysis to process high-dimensional face data into low-dimensional data and then analyze and process the data to obtain recognition results. Using the EigenFaces algorithm, the EigenFaces recognizer instance model is generated by the function cv2.face.EigenFaceRecognizer_create() in OpenCV, and then trained by cv2.face_FaceRecognizer_train(), and finally face recognition is completed by cv2.face_FaceRecognizer_predict().

[0064] In actual applications, in order to ensure the safety of vehicles, cameras are installed in the vehicle management area to identify the driver's face. In the driving safety supervision system, the face recognition interface can obtain the driver's face image by opening the camera for direct recognition and uploading photos to perform identity recognition, facial expression recognition, etc.

[0065] Identity recognition is divided into four steps: driver face collection, preprocessing, feature extraction, matching and recognition. First, the sample is collected using a cascade classifier based on Haar features, and then the image is preprocessed for feature extraction. Since the original images acquired by the system are often affected by various conditions and random interference, the images are usually not directly usable. Therefore, it is necessary to perform preprocessing steps such as grayscale correction and noise filtering on the images in the early stages of processing. These processes help improve image quality and make the subsequent feature extraction process more accurate. After the preprocessing is completed, the EigenFace classic face recognition algorithm provided in OpenCV is used to extract features from the face. Subsequently, the extracted features are compared with the information in the background database to achieve the identity recognition of the driver.

[0066] When the driver's identity is successfully identified, the system will display the driver's name and ID number, and automatically unlock the vehicle driven by the driver, while displaying the corresponding license plate number. After identification, only the vehicle assigned when the driver's information was registered can be unlocked to ensure that the vehicle and the driver correspond one to one. After unlocking, the driver can operate the designated vehicle. If the driver's information is not registered, it is impossible to identify his identity or unlock the vehicle, thus avoiding the problem of mixing people and vehicles, achieving strict binding between people and vehicles, and ensuring standardized and orderly use of vehicles.

[0067] Step 2: Perform license plate detection and recognition on the driver's vehicle at the vehicle entrance and exit, compare the detected vehicle information with the vehicle information in the system database, retrieve the driver information corresponding to the vehicle, and allow the vehicle to pass after the driver information is compared and confirmed.

[0068] License plate detection and recognition is the use of computer intelligent detection and recognition of vehicle license plates, thereby realizing the recognition of vehicle information. The license plate detection and recognition process includes the following steps:

[0069] (1) Image input

[0070] Image input generally uses a camera connected to a computer or an image acquisition card to collect images in real time to achieve image information collection. The original vehicle image collected is as follows: Figure 1 shown.

[0071] (2) Pretreatment

[0072] Graying the image to be identified and removing image noise can better highlight the license plate features and extract the specific area where the vehicle license plate is located. The image is then subjected to binarization, closing, erosion and dilation processing. Binarization converts the image to contain only two colors, usually black and white, to simplify the image content and highlight the main features, such as edges and shapes. Closing removes small noise and fills holes in objects by filling small black areas and breaks connecting white areas in the image. The role of erosion and dilation processing is to optimize the morphology of the foreground area and improve the quality of the image by denoising, filling holes and enhancing the structure. The processed picture is as follows Figure 2 , 3 , as shown in Figure 4.

[0073] (3) License plate positioning

[0074] In natural environments, vehicle license plate images are affected by light intensity and background factors. Accurately locating the license plate area is a key step in the license plate detection and recognition process. First, a large-scale search is performed on the acquired image to find candidate areas that meet the characteristics of the license plate. Then, these candidate areas are deeply analyzed to determine the area that is most likely to be the license plate, and the area with the highest probability is cut out from the image. The license plate position image is cut out. Figure 5 shown.

[0075] (4) Character segmentation

[0076] After locating and extracting the license plate area, the characters are segmented one by one. In complex environments, vertical projection is usually used for license plate character segmentation, which usually achieves better results. Figure 6 shown.

[0077] (5) Character Recognition

[0078] The template matching algorithm is used for character recognition. First, the segmented characters are binarized and the character size is adjusted to ensure that it is consistent with the template size in the database. Then matching recognition is performed and the result with the highest matching similarity is selected as the recognition result. The image to be recognized is directly input into the network, and its features are automatically extracted until the recognition output result is completed.

[0079] (6) Output the recognition result of the license plate

[0080] By calling the PIL library, the final recognition result is displayed on the collected original image, such as Figure 7 As shown, the person and vehicle information are compared at the same time.

[0081] In actual applications, to ensure safety, cameras are installed at the entrance and exit of vehicles to obtain license plate images. In the driving safety supervision system, the license plate image to be detected and identified can be directly obtained through the camera or by uploading pictures, and compared with the license plate data stored in the database to identify the vehicle. After the vehicle is successfully identified, the license plate number of the successfully identified vehicle and the name, certificate number and photo of the driver driving the vehicle are displayed, and the vehicle is automatically released. If the identification fails, the vehicle will be prohibited from passing.

[0082] Step 3: Recognize the driver’s facial expression during driving, and remind the driver to drive safely if abnormal mood is detected

[0083] The data is processed using Python language Pandas library and Kears framework. The facial expression dataset fer2013 is used to build and train the model to detect facial expressions. Seven basic expressions are identified: anger, disgust, fear, happy, sad, surprised, and normal. The driver's mental state is judged by his facial expression. If the driver's expression is other than normal (neutral), the driver is reminded to drive safely, providing necessary protection for driving safety.

[0084] Step 4: Perform eye closure detection, mouth opening detection and head movement detection on the driver during driving to comprehensively determine whether the driver is drowsy while driving the vehicle, and remind the driver through the alarm system

[0085] The image or video is analyzed based on computer vision technology to determine whether the driver is drowsy while driving the vehicle, mainly including the following steps:

[0086] (1) Perform face recognition

[0087] (2) Use the Dlib library file to locate key points of the collected facial information

[0088] The most obvious feature of a face is the five features, which are the five key points for face recognition. However, the uncertainty of face recognition detection based on only five key points is too large, so the number of facial feature points can be expanded to 68, such as Figure 8 As shown in Figure 2, there are 51 internal key points and 17 contour key points.

[0089] (3) Based on the key points of the face, the aspect ratio of the eyes and mouth is calculated to determine whether the eyes are closed or yawning, and whether the head is nodded based on the head posture detection method.

[0090] When people are tired, their eyes will close automatically and involuntarily. Among the 68 feature points of the human face, each eye has 6 key points, 2 feature points above and below the eye socket, and 2 feature points at the corner of the eye. Fig. 9 As shown. Define the human eye aspect ratio EAR, then Where P 37 is the left eye end point, P 38 is the upper left position of the left eye, P 39 is the upper right position of the left eye, P 40 is the left eye corner position point, P 41 is the lower right side of the left eye, P 42 It is the lower left point of the left eye.

[0091] When the eyes are closed, 38 and P 42 , P 39 and P 41 The distance will obviously become smaller, and P 37 and P 40 When the eyes are open, the EAR value is basically stable and fluctuates within a range, while when the eyes are closed, the EAR value is almost kept at 0. Since the eye sizes of different people are different, in order to eliminate the interference of accidental situations, the EAR values ​​of both eyes are generally recorded, and the average value is taken as the final judgment value. Set the threshold value of the EAR value. When the judgment value of the EAR is less than the set threshold, it is judged as closed eyes; when the eyes are closed for too long, it is considered as dozing off.

[0092] The detection of yawning is similar to the detection of blinking. The mouth feature points are as follows: Fig.10 As shown. Define the mouth length-to-width ratio MAR, then Where P 49 is the left corner of the mouth, P 51 is the left side of the upper lip, P 53 is the right side point of the upper lip, P 55 is the right corner of the mouth, P 57 is the right side point of the lower lip, P 59 It is the left side point of the lower lip.

[0093] When the mouth is closed, the key points P of the left and right corners of the mouth are 49 and P 55 The distance between the lips and the protruding points P above and below 51 and P 59 , P 53 and P 57 The distance between the key point P and the yawning point remains basically unchanged. 51 and P 59 , P 53 and P 57 The distance will become larger 49 and P 55 The distance will be slightly smaller. By setting the MAR value, the sensitivity of the detection can be adjusted to achieve the best detection effect. Similarly, the threshold value of the MAR value is set. When the MAR judgment value is greater than the set threshold, it is judged as yawning.

[0094] According to the head posture detection method, determine whether to nod. This method involves the conversion relationship from 3D reference points to 2D reference points. It is a typical PNP problem. Its purpose is to solve the pose from the camera coordinate system to the world coordinate system. The solvePNP function in the OpenCV library can be used for detection and measurement. First, perform 2D facial key point detection, then use a trained 3D face model to match the 2D model, solve the conversion relationship between the two, and then obtain the rotation matrix, and further solve the Euler angle.

[0095] In 3D space, there are three directions for the rotation of the head: Yaw, Pitch, and Roll. Yaw is the yaw angle, which rotates around the Y axis, that is, the left and right direction of the head. Pitch is the pitch angle, which rotates around the X axis, that is, the up and down direction of the head. Roll is the roll angle, which rotates around the Z axis, and is the direction of the head to the sides. When a nodding gesture occurs, the change in Pitch is the largest, followed by Yaw, and the change in Roll is the smallest. The angles of Pitch and Yaw are calculated comprehensively. When nodding, the corresponding X-axis parameter increases in the positive direction, and it is judged as bowing the head. When the X-axis parameter returns from an integer to 0, the system determines that a nodding action has been performed.

[0096] (4) Determine whether the driver is drowsy based on the length of time the eyes are closed, the frequency of yawning, and the angle of the head nodding, and issue an alarm to remind the driver

[0097] ① For eye closure, the EAR threshold is pre-set to 0.2. When the EAR value is less than the set threshold, the frame counter starts counting. When the eye closure time is greater than 3 frames, it is judged as long-term eye closure and the number of eye closures is increased by 1. When the number of eye closures is greater than 5 times, it means that the driver is drowsy and an alarm is issued.

[0098] ② For yawning, the MAR threshold is preset to 0.9. When the MAR value is greater than the preset threshold, it is judged as opening the mouth. When the frame counter is greater than 3 frames, the number of yawns increases by 1. When the number of yawns is greater than 5 times, it means that the driver is sleepy and an alarm is issued.

[0099] ③ For nodding, the corresponding X-axis parameter increases in the positive direction when nodding, which is judged as lowering the head. When the X-axis parameter returns from an integer to 0, the system determines that a nod has been made, and the number of nods increases by 1. When the number of nods is greater than 5 times, it means that the driver is drowsy and an alarm is issued.

[0100] By detecting the driver's sleepiness, the system can comprehensively determine whether he is sleepy while driving, and alert the driver through an alarm to prevent traffic accidents.

Claims

1. A driving safety supervision method based on image detection, characterized in that: The following steps are involved: Step 1: Perform facial recognition and identity recognition on the driver, compare it with the information in the background database, determine the driver's identity information, and compare it with the information of the vehicle being driven. After confirming that it is the driver of the vehicle, allow the vehicle to be started; Step 2: Detect and identify the license plate of the driver's vehicle at the vehicle entrance and exit, compare the detected vehicle information with the vehicle information in the system database, retrieve the driver information corresponding to the vehicle, and allow the vehicle to pass after comparing and confirming the driver information with the vehicle information; Step 3: Recognize the driver's facial expression during driving, and remind the driver to drive safely if an abnormal mood is detected; Step 4: During driving, the driver is tested for eye closure, mouth opening and head movement to determine whether the driver is drowsy while driving the vehicle, and the driver is reminded through an alarm system.

2. The driving safety supervision method based on image detection according to claim 1 is characterized in that: The face recognition method described in step 1 adopts a feature matching-based method. The face recognition algorithm is implemented as follows: using the EigenFaces algorithm, the EigenFaces recognizer instance model generated by the function cv2.face.EigenFaceRecognizer_create() in OpenCV, and then using cv2.face_FaceRecognizer_train() to complete the training, and finally completing the face recognition through cv2.face_FaceRecognizer_predict().

3. The driving safety supervision method based on image detection according to claim 1 is characterized in that: The identity recognition described in step 1 is divided into four steps: driver face collection, preprocessing, feature extraction, matching and recognition. First, the sample is collected using a cascade classifier based on Haar features, and then the image is preprocessed for feature extraction. Subsequently, the extracted features are compared with the information in the background database to achieve the identity recognition of the driver.

4. The driving safety supervision method based on image detection according to claim 1 is characterized in that: The license plate detection and recognition process described in step 2 includes the following steps: (1) Image input Image input is connected to a computer or image acquisition card through a camera device for real-time acquisition to obtain image information of the original image; (2) Pretreatment The image to be identified is grayed and image noise is removed, and then the image is binarized, closed, eroded and expanded respectively; (3) License plate positioning First, a large-scale search is performed on the collected image to find candidate areas that match the characteristics of the license plate. Then, these candidate areas are analyzed in depth to determine the area that is most likely to be the license plate, and the area with the highest probability is cut out from the image; (4) Character segmentation After locating and extracting the license plate area, the vertical projection method is used to segment the license plate characters; (5) Character Recognition The template matching algorithm is used for character recognition. First, the segmented characters are binarized and the character size is adjusted to ensure that it is consistent with the template size in the database. Then matching recognition is performed and the result with the highest matching similarity is selected as the recognition result. (6) Output the recognition result of the license plate By calling the PIL library, the final recognition result is displayed on the collected original image, and the person and vehicle information is compared at the same time.

5. The driving safety supervision method based on image detection according to claim 1 is characterized in that: The process of driver expression recognition described in step three is: using Python language Pandas library and Kears framework to process data, using facial expression dataset fer2013 to build and train the model to achieve the purpose of facial expression detection, identifying and judging seven basic expressions: anger, disgust, fear, happy, sad, surprised and normal. When the driver's expression is other than normal, the driver is reminded to drive safely.

6. The driving safety supervision method based on image detection according to claim 1 is characterized in that: Step 4, determining whether the driver is drowsy while driving the vehicle, mainly includes the following steps: (1) Perform face recognition; (2) Use the Dlib library file to locate key points of the collected facial information; (3) Based on the location of key points on the face, the length-to-width ratio of the eyes and mouth is calculated to determine whether the eyes are closed or yawning; based on the head posture detection method, it is determined whether the head is nodding; Define the human eye aspect ratio EAR, then Where P 37 is the left eye end point, P 38 is the upper left position of the left eye, P 39 is the upper right position of the left eye, P 40 is the left eye corner position point, P 41 is the lower right side of the left eye, P 42 is the point on the lower left side of the left eye; set the threshold of the EAR value, record the EAR values ​​of both eyes, and then take the average value as the final judgment value. When the judgment value of EAR is less than the set threshold, it is judged as closed eyes; Define the mouth length-to-width ratio MAR, then Where P 49 is the left corner of the mouth, P 51 is the left side of the upper lip, P 53 is the right side point of the upper lip, P 55 is the right corner of the mouth, P 57 is the right side point of the lower lip, P 59 is the point on the left side of the lower lip. Similarly, the threshold of the MAR value is set. When the MAR judgment value is greater than the set threshold, it is judged as yawning. In 3D space, there are three directions for the head to rotate: Yaw, Pitch, and Roll. Yaw is the yaw angle, which rotates around the Y axis; Pitch is the pitch angle, which rotates around the X axis; Roll is the roll angle, which rotates around the Z axis; when a nod occurs, Pitch changes the most, followed by Yaw, and Roll changes the least, and the corresponding X-axis parameter increases in the positive direction, which is judged as bowing the head; when the X-axis parameter returns from an integer to 0, the system determines that a nod has been performed; (4) Determine whether the driver is drowsy based on the length of time the eyes are closed, the frequency of yawning, and the angle of the head nodding, and issue an alarm to remind the driver ① For eye closure, the EAR threshold is pre-set to 0.

2. When the EAR value is less than the set threshold, the frame counter starts counting. When the eye closure time is greater than 3 frames, it is judged as long-term eye closure and the number of eye closures is increased by one; when the number of eye closures is greater than 5 times, it means that the driver is drowsy and an alarm is issued; ② For yawning, the MAR threshold is preset to 0.

9. When the MAR value is greater than the preset threshold, it is judged as opening the mouth. When the frame counter is greater than 3 frames, the number of yawns increases by one. When the number of yawns is greater than 5 times, it means that the driver is sleepy and an alarm is issued; ③ For nodding, when nodding, the corresponding X-axis parameter increases in the positive direction, which is judged as bowing. When the X-axis parameter returns from an integer to 0, the system determines that a nodding action has been performed, and the number of nodding times increases by one; When the number of nods is greater than 5 times, it means the driver is drowsy and an alarm will be issued.

Images