Driver behavior analysis method and electronic equipment

Abstract

The invention discloses a driver behavior analysis method and electronic equipment, and the method comprises the following steps: S1, obtaining a picture frame for monitoring driver behaviors, the picture frame containing a steering wheel; s2, training the target detection model until the loss function of the target detection model converges, and obtaining a trained target detection model; s3, picture frames shot by the camera are transmitted to the trained target detection model in real time, whether a driver has a prohibition behavior or not is judged, and if the prohibition behavior exists, the corresponding picture frame serves as an event occurrence frame; and S4, judging whether the percentage a of the number of the event generation frames accounting for all the picture frames in the preset number of continuous picture frames is greater than a preset threshold b, if so, performing a prohibition behavior alarm, and otherwise, continuing monitoring. According to the method, various behaviors of the driver can be detected and judged, and the behaviors of the driver are judged according to the proportion of the number of the event occurrence frames to the number of the picture frame queues, so that the behavior judgment is more accurate, and the possibility of misjudgment is reduced.

Description

Technical Field

[0001] The present invention relates to the field of image recognition, and in particular to a driver behavior analysis method and electronic equipment. Background Art

[0002] With the rapid economic growth and rapid urbanization in my country, urban traffic congestion is becoming increasingly serious. At present, buses play an important role in alleviating the travel pressure of residents. At the same time, the behavior of bus drivers has also received great attention from researchers. Many traffic congestion and traffic accidents are related to the driver's driving behavior. Real-time monitoring and evaluation of the driver's driving behavior and driving status, and timely reminder of the driver's dangerous driving behavior can avoid traffic accidents.

[0003] For this purpose, people have designed many monitoring methods. For example, an invention patent discloses a driver status information monitoring method including: real-time acquisition of monitoring video image data of the driver; the monitoring video image data includes first time information; face image detection is performed on the monitoring video image data to obtain the driver's facial image and the detection result information of the driver's facial image; according to the detection result information and the first time information, it is determined whether the driver's on-the-job status is normal or abnormal; if the on-the-job status is abnormal, a driver off-the-job alarm information is generated; if the on-the-job status is normal, the driver's facial image is analyzed to generate the driver's working status information; the driver's working status information includes: blinking behavior information and / or eye closing behavior information; according to the blinking behavior information and / or eye closing behavior information, a driver fatigue index is generated; when the fatigue index exceeds a preset fatigue index threshold, a driver fatigue alarm information is generated.

[0004] Invention patent CN202110737698.5 discloses a method, device, equipment and medium for identifying conflict behaviors of people in a vehicle cabin. The method comprises: obtaining multiple video frame images in a video stream in the vehicle cabin; performing feature extraction on the multiple video frame images respectively to obtain multiple image features corresponding to the multiple video frame images, wherein the image features represent the action features of the multiple people in the vehicle cabin; obtaining the action feature sequences of the multiple people in the vehicle cabin corresponding to the multiple video frame images based on the multiple image features; identifying the conflict behaviors between the multiple people in the vehicle cabin based on the action feature sequences, and obtaining the identification results of the conflict behaviors between the multiple people in the vehicle cabin.

[0005] However, the above method can only monitor one situation of the driver. In addition, when the driver is driving a vehicle, there are three states: two-handed driving, one-handed driving and hands-free driving. Only one of them can exist at the same time, while the existing model will detect two or even three at the same time, resulting in false alarms. Moreover, when the driver turns the steering wheel, it is possible that he holds the steering wheel with one hand temporarily, but it is not one-handed driving. The existing method easily judges it as one-handed driving, resulting in false alarms, so it needs to be improved. Summary of the invention

[0006] In order to solve the above technical problems, the present invention proposes a driver behavior analysis method and electronic device. The purpose of the present invention is achieved through the following technical solutions: A driver behavior analysis method comprises the following steps: S1, obtaining a picture frame for monitoring driver behavior, wherein the picture frame contains a steering wheel; S2, training the target detection model until the loss function of the target detection model converges, and obtaining a trained target detection model; The loss function of the target detection model As shown below:

[0007] in ,y k is the category label, , Predict categorical values ​​for the model, is the sigmoid function; j represents the jth category, k represents the kth category; S3, transmitting the image frame to the trained target detection model in real time to determine whether the driver has any prohibited behavior. If so, the corresponding image frame is used as the event occurrence frame; S4. Determine whether the percentage a of event occurrence frames in a preset number of consecutive picture frames is greater than a preset threshold b of all picture frames. If yes, issue a prohibition behavior alarm. Otherwise, continue monitoring.

[0008] As a further improvement, the prohibited behaviors include leaving one's post, one-handed driving, hands-free driving, physical conflict and playing with mobile phones.

[0009] Further improvement, b≥0.8.

[0010] As a further improvement, the specific steps of step S4 are as follows: put the image frames into a queue of preset length in the order of the video stream. When the queue is full, continuously discard the result of the last frame and put in the result of the latest frame; when the queue is full, determine whether the first frame and the last frame are frames for the same type of event, and if so, obtain the percentage a of the number of frames for such event occurrence to the total number of image frames in the queue; if a>preset threshold b, issue a prohibited behavior alarm, otherwise continue monitoring.

[0011] For further improvement, the queue length = the preset prohibited behavior duration alarm time in seconds * the number of frames taken per second.

[0012] Further improvements, ;in, They represent one-handed driving, two-handed driving, hands-free driving, mobile phone and hand respectively.

[0013] As a further improvement, in step S3, if a mobile phone appears in the picture frame, it is judged that the driver is playing with the mobile phone; if no hand is detected in the picture frame, it is judged that the driver is away from his post; if there is only one hand in the steering wheel area in the picture frame, it is judged that the driver is driving with one hand; if there is a hand in the picture frame but no hand in the steering wheel area, it is judged that the driver is driving hands-free; if the number of hands in the picture frame is greater than 2, it is judged that the driver is in a physical conflict.

[0014] A further improvement is that the target detection model is a yolov5 target detection model.

[0015] As a further improvement, the camera is installed on the top of the left wall of the cab, at a position centered on the central axis of the steering wheel, and the camera mirror is tilted downward to be able to fully capture the driver and the steering wheel.

[0016] An electronic device comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of any of the above methods.

[0017] The beneficial effects of the present invention are: The present invention can detect and judge various behaviors of the driver, and use the ratio of the number of event occurrence frames to the number of picture frame queues to judge the driver's behavior, so as to make the judgment of the driver's behavior more accurate and reduce the possibility of misjudgment. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The present invention is further described with reference to the accompanying drawings, but the contents in the accompanying drawings do not constitute any limitation to the present invention.

[0019] Figure 1 It is a schematic diagram of the process of the present invention; Figure 2 This is a schematic diagram of the camera installation location; Figure 3 This is a schematic diagram of the camera screen; Figure 4 This is a schematic diagram of driving with a mobile phone in one hand; Figure 5 Schematic diagram of the alarm logic for leaving the post, playing with mobile phones and physical conflicts; Figure 6A schematic diagram of the combined warning logic for one-handed driving and hands-free driving; Figure 7 A schematic diagram of grabbing the steering wheel during a physical conflict; Figure 8 A diagram showing the assault on a driver during a physical altercation; Fig. 9 This is a diagram of taking both hands off the steering wheel. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solutions and advantages of the invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and examples.

[0021] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0022] In the description of the present application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, which are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first" and "second" are used only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0023] In this application, the word "exemplary" is used to mean "serving as an example, illustration, or description". Any embodiment described in this application as "exemplary" is not necessarily to be construed as being preferred or advantageous over other embodiments. The following description is given to enable any technician in the field to implement and use the present application. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present application can be implemented without using these specific details. In other instances, well-known structures and processes will not be elaborated in detail to avoid obscuring the description of the present application with unnecessary details. Therefore, the present application is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present application.

[0024] The present invention discloses a driver behavior analysis method and electronic equipment. The working method comprises the following steps: Step 1: Install the camera according to the camera calibration method.

[0025] The camera is installed on the top of the left wall of the cab. The best position is centered on the central axis of the steering wheel. If this is not possible, it can be installed within 10 cm to the left and right of the center line of the steering wheel. The closer to the central axis, the better. The installation height should be about 1 meter from the bottom of the left window. The camera mirror is tilted downward to fully capture the driver and the steering wheel. Figure 2 As displayed.

[0026] Among them, the camera calibration method can adopt common calibration methods such as Tsai two-step method, Zhang calibration method or camera calibration method based on active vision: Tsai two-step method: first linearly obtain the camera parameters, and then consider the distortion factor. It is faster, but only considers radial distortion.

[0027] Zhang’s calibration method: uses a calibration plate composed of two-dimensional grids for calibration, which is simple to operate and has high accuracy.

[0028] Camera calibration method based on active vision: The camera is controlled to perform specific movements through an active system. A precise control platform is required. After the camera is installed, it is necessary to confirm that the camera image meets the algorithm analysis requirements.

[0029] The camera angle should be adjusted to ensure that the window frame and the screen frame are as parallel as possible. If they are not parallel, the window part cannot exceed one grid line. The steering wheel is completely visible and the size is not less than 16 grids (the longest diameter of the steering wheel is not less than five grids). It must be ensured that the driver can appear completely in the screen. Figure 3 The grid lines are the grid lines that come with the camera, such as Figure 3 As shown, in one picture frame, there are 24 grids horizontally and 14 grids vertically.

[0030] Step 2: Get the image detection results. Access the real-time video, get the video frames, pre-process the video frames and input them into a target detection model (yolov5) to get the detection frame and detection category. The detection model detects five categories [0, 1, 2, 3, 4], corresponding to [singhand, twohand, nohand, phone, hand]. Among them, the steering wheel will produce one of the detection results of single-handed driving, two-handed driving or no-handed driving. Figure 4 As shown, the detection results and categories of one-handed driving, mobile phone and hand are displayed.

[0031] YOLO, an acronym for “You only look once,” is an open-source software tool that has the highly efficient ability to detect objects in a particular image in real-time. The YOLO algorithm uses a convolutional neural network (CNN) model to detect objects in an image. The algorithm only needs to do a single forward propagation through a given neural network to detect all objects in an image. This gives the YOLO algorithm an advantage over other algorithms in terms of speed, making it one of the most famous detection algorithms to date. The YOLO algorithm not only provides high detection speed and performance through its single forward propagation capability, but it also detects them very accurately and precisely.

[0032] The present invention utilizes the YOLOv5 model structure and modifies the loss function to utilize the spatial information and prior information of the category, as follows: The original loss function of YOLOv5 is: + ),in The classification loss is the cross entropy function. Modifications have been made. Details are as follows: Original

[0033] new

[0034] in is the category label, Predict categorical values ​​for the model, is the sigmoid function.

[0035] This loss function adds a loss constraint on the steering wheel category based on the original loss, because only one of the three states of the steering wheel, two-handed driving, one-handed driving, and hands-free driving, can exist at the same time, while the original model will detect two or even three at the same time, resulting in false positives. The new loss will constrain the prediction probability of the first three categories, namely the steering wheel-related categories, so that only one category has a large prediction value, while the other two categories are as small as possible.

[0036] Step 3: Single-frame image behavior judgment: Analyze the results in step 2 to determine whether the person is off duty, whether the person drops the hands, whether there is a physical conflict, or whether the person is playing with the phone: (1) The detection and classification of the steering wheel can directly determine whether the steering wheel is out of hand or whether the driver is driving with one hand.

[0037] (2) Whether the user is playing with the mobile phone is determined based on whether the mobile phone is detected in the test results.

[0038] (3) The number of hands in the detection result is used to determine whether the person has left his post or engaged in a physical conflict. If the number of hands detected is 0, the person has left his post; if the number of hands detected is greater than 2, the person has engaged in a physical conflict.

[0039] The judgment process is as follows.

[0040] 1) Initialization result: singhand (one-hand driving) = 0; nohand (hands-free driving) = 0; phone (playing with mobile phone) = 0; conflict (physical conflict) = 0; hand (hand) = 0; leave (leaving the post) = 0 2) Detection model to obtain detection results: dets = model (image) (dets represents the detection result, model represents the detection model) 3) Behavior judgment based on single-frame images based on the results of calendar detection: for det in dets: if det.class==0: if driving with one hand singhand=1 means only one hand is detected in the steering wheel area if det.class==1: if it is two-handed driving twohand=1 means two hands are detected in the steering wheel area.

[0041] if det.class==2: if hands-free driving nohand=1 means there is no hand in the steering wheel area, but a hand is detected in the image frame; ifdet.class==3: means if a mobile phone is detected phone=1 means playing with mobile phone if det.class==4: hand=hand+1 (Hand is of int type, records the number of detected hands, and adds 1 to the number of hands when a hand is detected) if hand==0: leave=1 means the person is off duty if hand>2: conflict=1 means it is a physical conflict in Step 4: Obtain an alarm result by combining the behavior results of multiple consecutive frames through alarm logic; For leaving the post, playing with mobile phones and physical conflicts, the alarm logic is designed as follows: set the maximum length of the queue to the alarm time threshold (queue length = continuous alarm time in seconds * number of frames per second; continuous alarm time in seconds is the number of seconds the set violation lasts; number of frames per second is the number of image frames extracted from the video stream per second, such as extracting 5 frames per second for judgment and detection. For example, playing with mobile phones or taking both hands off the steering wheel while driving is more dangerous. Set the alarm to last for 2 seconds. If 5 frames are taken per second, the queue length is 2*5=10). Put the judgment result of each frame in step 3 into the queue. When the queue is full, continuously discard the result of the oldest frame and put in the result of the latest frame. When the queue length reaches the alarm time threshold, first determine whether the first frame and the last frame are judged as event occurrence frames (for leaving the post, the event occurrence frame is the frame with the detection result of leaving the post. leave=1), and then determine whether the percentage of event occurrence frames in the queue is greater than the set percentage. If it is greater, a corresponding behavior alarm is generated. Figure 5 The detailed alarm steps are as follows: Initialize the queue to be empty, set the alarm time threshold and set the maximum queue length to the alarm time threshold length, set the alarm percentage threshold, and set the alarm time interval.

[0042] Get a new frame of detection results.

[0043] Store the results in a queue. If the queue exceeds the queue length, discard the oldest frame's results.

[0044] Calculate the number of frames in which events occurred in the queue and divide it by the queue length to get the percentage.

[0045] If the ratio is greater than the percentage threshold and the current frame is an event occurrence frame, it is further determined whether the difference between the current frame time and the alarm timestamp is greater than the alarm time interval. If it is greater than the alarm time interval, an event alarm is issued and the alarm timestamp is updated.

[0046] Repeat steps 2)-5).

[0047] For hands-off steering, the alarm logic is as follows: set one-handed driving and hands-off driving queues, the maximum length of the queues are the one-handed warning time threshold length and the hands-off warning time threshold length respectively, obtain the judgment result of each frame of the steering wheel in step 3, if the current frame result is one-handed driving, the one-handed driving queue result is stored in 1, and the hands-off driving queue result is stored in 0. If the current frame result is hands-off driving, the hands-off driving queue is stored in 1, and it is determined whether the hands-off driving event is an alarm. If the hands-off driving does not alarm, the one-handed driving queue is stored in 1. If the hands-off driving alarm is, the one-handed driving queue is cleared. The process is as follows Figure 6 The detailed alarm steps are as follows: Initialize the queue to be empty, set the one-handed driving warning time threshold and the hands-off driving event threshold, and set the one-handed driving time threshold to be greater than the hands-off driving time threshold. Set the maximum length of the one-handed driving queue to the one-handed driving time threshold length, set the maximum length of the hands-off driving queue to the hands-off driving time threshold length, set the one-handed driving and hands-off driving warning percentage thresholds and warning time intervals respectively.

[0048] Get a new frame of detection results.

[0049] Determine whether the current frame is leaving the post. If it is leaving the post, that is, leave==1, then set the values ​​of one-handed driving and hands-free driving to 0, that is, singhand=0, nohand=0. Then determine whether the current frame is two-handed driving. If it is two-handed driving, that is, twohand==1, then set the values ​​of one-handed driving and hands-free driving to 0, that is, singhand=0, nohand=0. Then determine whether the current frame is a hand-off. If it is a hand-off, that is, nohand==1, then store the value of nohand in the hand-off queue.

[0050] Determine whether to generate a hands-off warning, that is, calculate the percentage of events in the hands-off queue. If the ratio is greater than the hands-off driving warning percentage threshold, and the difference between the current frame time and the hands-off warning timestamp is greater than the hands-off warning time interval, a hands-off driving event alarm is issued, and the hands-off driving warning timestamp is updated, and the one-handed driving queue is cleared at the same time. Otherwise, no hands-off time alarm is issued, and the one-handed driving result is set to 1, that is, singhand=1. The value of singhand is stored in the one-handed driving queue, and the percentage of events in the one-handed driving queue is calculated. If the ratio is greater than the one-handed driving warning percentage threshold, and the difference between the current frame time and the one-handed warning timestamp is greater than the one-handed warning time interval, a one-handed driving event alarm is issued and the one-handed driving warning timestamp is updated.

[0051] Repeat steps 2)-7) Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A driver behavior analysis method, characterized in that: The steps include: S1, obtaining a picture frame for monitoring driver behavior, wherein the picture frame contains a steering wheel; S2, training the target detection model until the loss function of the target detection model converges, thereby obtaining a trained target detection model; The loss function of the target detection model As shown below: ; in ,y k is the category label, , Predict categorical values ​​for the model, is the sigmoid function; j represents the jth category, k represents the kth category; S3, transmitting the image frame to the trained target detection model in real time to determine whether the driver has any prohibited behavior. If so, the corresponding image frame is used as the event occurrence frame; S4. Determine whether the percentage a of event occurrence frames in a preset number of consecutive picture frames is greater than a preset threshold b of all picture frames. If yes, issue a prohibition behavior alarm. Otherwise, continue monitoring.

2. The driver behavior analysis method according to claim 1, characterized in that: The prohibited behaviors include leaving one's post, one-handed driving, hands-free driving, physical conflict and playing with mobile phones.

3. The driver behavior analysis method according to claim 2, characterized in that: b≥0.8。 4. The driver behavior analysis method according to claim 1, characterized in that: The specific steps of step S4 are as follows: put the image frames into a queue of preset length in the order of the video stream. When the queue is full, continuously discard the result of the last frame and put in the result of the latest frame; when the queue is full, determine whether the first frame and the last frame are frames for the same type of event. If so, obtain the percentage a of the number of frames for such event occurrence to the total number of image frames in the queue; if a>preset threshold b, issue a prohibited behavior alarm, otherwise continue monitoring.

5. The driver behavior analysis method according to claim 4, characterized in that: The length of the queue = the preset prohibited behavior duration alarm time in seconds * the number of frames taken per second.

6. The driver behavior analysis method according to claim 2, characterized in that: ;in, They represent one-handed driving, two-handed driving, hands-free driving, mobile phone and hand respectively.

7. The driver behavior analysis method according to claim 2, characterized in that: In step S3, if a mobile phone appears in the picture frame, it is judged that the driver is playing with the mobile phone; if no hand is detected in the picture frame, it is judged that the driver is away from his post; if there is only one hand in the steering wheel area in the picture frame, it is judged that the driver is driving with one hand; if there is a hand in the picture frame but no hand in the steering wheel area, it is judged that the driver is driving hands-free; if the number of hands in the picture frame is greater than 2, it is judged that the driver is in a physical conflict.

8. The driver behavior analysis method according to claim 1, characterized in that: The target detection model is the yolov5 target detection model.

9. The driver behavior analysis method according to claim 1, characterized in that: The camera is installed on the top of the left wall of the cab, at a position centered on the central axis of the steering wheel. The camera mirror is tilted downward and can fully capture the driver and the steering wheel.

10. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 9.

Images