Driving behavior and state detection method and device based on deep learning

Abstract

The application discloses a driving behavior and state detection method and device based on deep learning, the detection method comprising: configuring a vehicle-mounted collection device to collect driving images of a driver in an actual driving state in real time; constructing a convolutional neural network model, the convolutional neural network model being configured to receive the driving images, perform convolution processing on the driving images, and output driving behavior categories on the driving images, driving behavior position information, driving state categories, and driving state position information having a relative position correlation with the driving behavior position; and configuring a decision module, the decision module being configured to receive output results of the convolutional neural network model, analyze the output results, and determine whether the driver is driving while tired. The application has high detection precision and efficiency, reduces the space, computing power and time consumption of the system, and greatly improves the real-time performance of the system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automobile driving, in particular to a driving behavior and state detection method and device based on deep learning. BACKGROUND

[0002] For the detection of the driving behavior and state of the driver, the mainstream method at present is to use a deep learning method for detection, specifically: using two neural network algorithms to respectively detect the driving behavior and driving state of the driver.

[0003] The above method can detect the behavior and state of the driver, but it increases the hardware memory space and computing power overhead, indirectly increasing the cost.

[0004] In particular, for the detection of the state of the driver, the common method at present is to first use a face key point neural network algorithm to detect the key points of the face of the driver, and then use the calculated key point information to perform logical calculation, and finally judge the state of the driver at this time, such as the opening and closing of the eyes, the opening and closing of the mouth, etc. Such logical calculation also occupies a certain computing power and time, and has a certain impact on the real-time performance of the entire system.

[0005] Therefore, the present application is proposed. SUMMARY

[0006] In order to overcome the above defects, the present application provides a driving behavior and state detection method and device based on deep learning, which not only has high detection accuracy and high detection efficiency, but also greatly reduces the space, computing power and time overhead of the system, and greatly improves the real-time performance of the system.

[0007] The technical scheme adopted by the present application to solve its technical problems is: a driving behavior and state detection method based on deep learning, comprising:

[0008] A vehicle-mounted collection device is configured to collect driving images of the driver in the actual driving state in real time;

[0009] A convolutional neural network model is constructed, which is configured to receive the driving images, and after convolution processing of the driving images, outputs driving behavior categories, driving behavior position information, driving state categories, and driving state position information having a relative position correlation with the driving behavior position based on the driving images;

[0010] A decision module is configured to receive the output results of the convolutional neural network model, and analyze the output results to determine whether the driver is driving while tired.

[0011] As a further improvement of the present application, a training data set and a verification data set are established for the convolutional neural network model to learn and train.

[0012] As a further improvement of the present application, the driver's to-be-trained images in the actual driving state are collected, the obtained to-be-trained images are preprocessed, and after the preprocessed to-be-trained images are classified and labeled, the training data set and the verification data set for the convolutional neural network model to learn and train are obtained.

[0013] As a further improvement of the present application, the content of the labeling of the preprocessed to-be-trained images includes driving behavior information and driving state information, wherein,

[0014] The driving behavior information includes a driving behavior position frame and a specific category of driving behavior, and the category of driving behavior includes normal driving of the driver, the driver is smoking, the driver is drinking water, and the driver is making or receiving a call.

[0015] The driving state information includes a driving state position frame and a specific category of driving state, and the category of driving state includes a left eye open state, a left eye closed state, a right eye open state, a right eye closed state, a mouth open state, and a mouth closed state.

[0016] As a further improvement of the present application, the vehicle-mounted collection device uses a vehicle-mounted camera; and the convolutional neural network model uses a VGGNet convolutional network model as a feature extraction subnetwork.

[0017] As a further improvement of the present application, based on the learning and training of the convolutional neural network model, the convolutional neural network model obtains the specific category of driving behavior by performing feature extraction on the driving image.

[0018] And the convolutional neural network model obtains the driving behavior position information, the driving state position information having a relative position correlation with the driving behavior position, and the driving state category by detecting and calculating the driving image.

[0019] As a further improvement of the present application, the specific operation method of the convolutional neural network model for detecting and calculating the driving image includes:

[0020] The driving behavior position box on the driving image is predicted by using a prior box detection method, to obtain a position offset A of a center point position of the driving behavior position box relative to a center point position of a prior box, a stretching amount A of a box width of the driving behavior position box relative to a box width of the prior box, and a stretching amount B of a box height of the driving behavior position box relative to a box height of the prior box; the obtained position offset A, stretching amount A and stretching amount B cooperatively reflect a neural network position regression value of the driving behavior position box.

[0021] According to the center point position coordinates, the box width and the box height of the driving behavior position box on the driving image in the image coordinate system, and the relative position relationship between the driving behavior position box and the driving state position box on the driving image, a position offset B of a center point position of the driving state position box relative to a calibration point position of the driving behavior position box, normalized values of the position offset B relative to the box width and the box height of the driving behavior position box respectively, a stretching amount a of a box width of the driving state position box relative to a box width of the driving behavior position box, and a stretching amount b of a box height of the driving state position box relative to a box height of the driving behavior position box are calculated, and the obtained normalized values, stretching amount a and stretching amount b cooperatively reflect a neural network position regression value of the driving state position box.

[0022] As a further improvement of the present application, in the image coordinate system of the driving image, the center point position coordinates of the driving behavior position box are (X0, Y0), the calibration point position coordinates are (X1, Y1), the box width is W, and the box height is H.

[0023] The driving state position box includes a left eye position box, a right eye position box and a mouth position box, wherein the center point position coordinates of the left eye position box are (X l-eye ,Y l-eye ), the box width is W l-eye , and the box height is H l-eye ; the center point position coordinates of the right eye position box are (X r-eye ,Y r-eye ), the box width is W r-eye , and the box height is H r-eye ; the center point position coordinates of the mouth position box are (X mouth ,Y mouth ), the box width is W mouth , and the box height is H mouth .

[0024] The position offset B of the center point position of the left eye position box relative to the calibration point position of the driving behavior position box is recorded as a left eye position offset B, and the left eye position offset B satisfies the following relationship:

[0025] △Xl-eye = X l-eye - X1, ΔY l-eye = Y l-eye - Y1;

[0026] and the normalized value of the left eye position offset B relative to the frame width and frame height of the driving behavior position frame respectively satisfies the following relationship:

[0027]

[0028] Record the stretching and shrinking amount a of the frame width of the left eye position frame relative to the frame width of the driving behavior position frame as the left eye frame width stretching and shrinking amount a, and the left eye frame width stretching and shrinking amount a satisfies the following relationship:

[0029] Record the stretching and shrinking amount b of the frame height of the left eye position frame relative to the frame height of the driving behavior position frame as the left eye frame height stretching and shrinking amount b, and the left eye frame height stretching and shrinking amount b satisfies the following relationship:

[0030] That is: (X l-true , Y l-true , w l-eye , h l-eye ) reflects the neural network position regression value of the left eye position frame;

[0031] Record the position offset B of the center point position of the right eye position frame relative to the calibration point position of the driving behavior position frame as the right eye position offset B, and the right eye position offset B satisfies the following relationship:

[0032] ΔX r-eye = X r-eye - X1, ΔY r-eye = Y r-eye - Y1;

[0033] and the normalized value of the right eye position offset B relative to the frame width and frame height of the driving behavior position frame respectively satisfies the following relationship:

[0034]

[0035] Record the stretching and shrinking amount a of the frame width of the right eye position frame relative to the frame width of the driving behavior position frame as the right eye frame width stretching and shrinking amount a, and the right eye frame width stretching and shrinking amount a satisfies the following relationship:

[0036] Record the stretching and shrinking amount b of the frame height of the right eye position frame relative to the frame height of the driving behavior position frame as the right eye frame height stretching and shrinking amount b, and the right eye frame height stretching and shrinking amount b satisfies the following relationship:

[0037] That is: (X r-true ,Y r-true ,w r-eye ,h r-eye ) reflects the neural network position regression value of the right eye position frame;

[0038] The position offset B of the center point position of the mouth position frame relative to the calibration point position of the driving behavior position frame is recorded as the mouth position offset B, and the mouth position offset B satisfies the following relationship:

[0039] △X mouth =X mouth -X1,△Y mouth =Y mouth -Y1;

[0040] And the normalized value of the mouth position offset B relative to the frame width and frame height of the driving behavior position frame respectively satisfies the following relationship:

[0041]

[0042] The stretching amount a of the frame width of the mouth position frame relative to the frame width of the driving behavior position frame is recorded as the mouth frame width stretching amount a, and the mouth frame width stretching amount a satisfies the following relationship:

[0043] The stretching amount b of the frame height of the mouth position frame relative to the frame height of the driving behavior position frame is recorded as the mouth frame height stretching amount b, and the mouth frame height stretching amount b satisfies the following relationship:

[0044] That is: (X m-true ,Y m-true ,w mouth ,h mouth ) reflects the neural network position regression value of the mouth position frame.

[0045] As a further improvement of the application, based on the neural network position regression value of the driving state position frame and the specific category of the driving state, the occurrence probability of any specific category of the driving state can be calculated;

[0046] The decision module analyzes and determines whether the driver is fatigue driving according to the neural network position regression value of the driving behavior position frame, the specific category of the driving behavior, the neural network position regression value of the driving state position frame, and the occurrence probability of the specific category of the driving state.

[0047] The application also provides a driving behavior and state detection device based on deep learning, comprising:

[0048] A vehicle-mounted collection device is configured to collect driving images of a driver in an actual driving state in real time.

[0049] A convolutional neural network model is configured to receive the driving images and output driving behavior categories, driving behavior position information, driving state categories and driving state position information having a relative position correlation with the driving behavior position based on the driving images after convolutional processing of the driving images.

[0050] A decision module is configured to receive output results of the convolutional neural network model and analyze the output results to determine whether the driver is driving while tired.

[0051] The present application has the advantages that the present application first collects driving images of a driver in an actual driving state in real time through a vehicle-mounted collection device, then constructs a convolutional neural network model with a special processing algorithm to perform convolutional processing on the driving images to output driving behavior categories, driving behavior position information, driving state categories and driving state position information having a relative position correlation with the driving behavior position based on the driving images, and finally uses a decision module to analyze output results of the convolutional neural network model to determine whether the driver is driving while tired. On the one hand, the detection method provided by the present application can simultaneously detect driving behavior and driving state of a driver, i.e., end-to-end detection is realized, which improves detection efficiency and greatly reduces space, computing power and time consumption of the system, greatly improving real-time performance of the system. On the other hand, when detecting and processing driving state position information, the present application does not independently regress, but correlates it with driving behavior position information and uses a special relative position method for calculation and processing. This can greatly improve detection rate and accuracy of small targets and always correlate behavior and state of the same person, avoiding confusion between behavior and state when there are multiple detection results and the problem of difficult correct matching. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 A flowchart of the driving behavior and state detection method based on deep learning according to the present application;

[0053] Figure 2 A specific working flowchart of the convolutional neural network model according to the present application;

[0054] Figure 3 A schematic diagram of a relative position relationship between the prior box and the driving behavior position box in an image coordinate system;

[0055] Figure 4 A schematic diagram of a relative position relationship between the driving behavior position box and the driving state position box in an image coordinate system of the driving image;

[0056] Figure 5 FIG. 1 shows a block diagram of a driving behavior and state detection device based on deep learning according to the present application.

[0057] The following description is made in conjunction with the accompanying drawings:

[0058] 1, vehicle-mounted collection device; 2, convolutional neural network model; 3, decision module. DETAILED DESCRIPTION

[0059] The preferred embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0060] Example 1

[0061] Please refer to FIG. 1, which is a flowchart of a driving behavior and state detection method based on deep learning according to the present application. Figure 1

[0062] The driving behavior and state detection method based on deep learning according to the present application includes the following working steps:

[0063] S1: configuring a vehicle-mounted collection device 1 for real-time collection of driving images of a driver in an actual driving state.

[0064] In this example, the vehicle-mounted collection device 1 uses a vehicle-mounted camera, but other collection devices can also be used, which are not limited in this example.

[0065] S2: constructing a convolutional neural network model 2, which is configured to receive the driving images and output driving behavior categories, driving behavior location information, driving state categories, and driving state location information having a relative position correlation with the driving behavior location after convolutional processing of the driving images.

[0066] In this example, the convolutional neural network model 2 uses a VGGNet convolutional network model as a feature extraction subnetwork, and further preferably uses a VGG16 convolutional neural network model. In particular, unlike the existing use of two neural network algorithms to detect driving behavior and driving state of a driver, the convolutional neural network model 2 in this example has only one operation channel, which utilizes the relative position correlation between the driving behavior location and the driving state location to simultaneously detect the driving behavior and the driving state of the driver, i.e., to achieve end-to-end detection, which greatly reduces the space, computing power and time overhead of the system, and greatly improves the real-time performance of the system.

[0067] S3: configuring a decision module 3, which is configured to receive the output results of the convolutional neural network model 2 and analyze the output results to determine whether the driver is driving while tired.​

[0068] In this example, the decision module 3 adopts a logical judgment module, and the specific structure form is not limited as long as it has a logical judgment function.

[0069] In addition, in this embodiment, a training data set and a verification data set are also established for the convolutional neural network model 2 to train and learn.

[0070] The specific establishment method of the training data set and the verification data set is: collecting the to-be-trained images of the driver in the actual driving state, pre-processing the obtained to-be-trained images, and classifying and labeling the pre-processed to-be-trained images to obtain the training data set and the verification data set for the convolutional neural network model 2 to train and learn. Note: ① The above pre-processing of the image is mainly processing the image format, such as cropping the image, etc.; ② The above “labeling” operation is preferably manually labeled, which can make the training accuracy higher; ③ When training the convolutional neural network model, the collected images are generally divided into two groups according to a certain proportion (common proportions are 9:1 or 8:2 or 1:1, etc., which are not limited in this example, and are determined according to actual needs). One group of images is used to train the convolutional neural network model, and the other group of images is used to verify the accuracy of the convolutional neural network model. This is a conventional technical means familiar to those skilled in the art, and therefore will not be described in detail here. In summary, by establishing the training data set and the verification data set, and using the forward reasoning and error back propagation algorithm to iteratively train and optimize the global parameters of the convolutional neural network, the loss function value output by the convolutional neural network is finally reduced and converged, and the detection accuracy is improved.

[0071] Further, the content of the labeled pre-processed to-be-trained image in this example mainly includes driving behavior information and driving state information, wherein,

[0072] The driving behavior information includes a driving behavior position box and a specific category of driving behavior, and the category of driving behavior includes normal driving of the driver, the driver is smoking, the driver is drinking water, and the driver is making or receiving a call;

[0073] The driving state information includes a driving state position box and a specific category of driving state, and the category of driving state includes a left eye open state, a left eye closed state, a right eye open state, a right eye closed state, a mouth open state, and a mouth closed state.

[0074] Further, when the convolutional neural network model 2 is trained, the loss function of the convolutional neural network model 2 can be specifically divided into two categories: classification loss and regression loss, wherein,

[0075] ① Classification loss, including: driving behavior position box confidence loss; driving behavior classification loss; left eye, right eye, and mouth existence classification loss; left eye, right eye, and mouth open or close classification loss. Specifically:

[0076] I) The expression of driving behavior position box confidence loss is:

[0077]

[0078] In the above formula, indicates whether the i-th row and j-th column of the output layer feature map is a driving behavior position box.

[0079] II) The expression of driving behavior classification loss is:

[0080]

[0081] As above, in the above formula, indicates whether the i-th row and j-th column of the output layer feature map is a driving behavior position box.

[0082] III) The expression of left eye, right eye, and mouth existence classification loss is:

[0083]

[0084] In the above formula, indicates whether the i-th row and j-th column of the output layer feature map exists left eye, right eye, or mouth.

[0085] IV) The expression of left eye, right eye, and mouth open or close classification loss is:

[0086]

[0087] In the above formula, open and close represent the opening and closing of the mouth / eyes.

[0088] ② Regression loss, including: driving behavior position box regression loss; left eye, right eye, and mouth position box regression loss. Specifically:

[0089] I) The expression of driving behavior position box regression loss is:

[0090]

[0091] In the above formula, (x i ,y i ,w i ,h i ) represents the true value of the driving behavior position box, represents the neural network position regression value of the driving behavior position box. ​​​

[0092] II) The expression of the regression loss of the left eye, right eye, and mouth position box is:

[0093]

[0094] In the above formula, (x i ,y i ,w i ,h i ) represents the position true value of the driving state (driver's eyes / mouth), represents the neural network position regression value of the driving state position box.

[0095] All the above losses are calculated only when the driving behavior position box actually exists. In particular, for the left eye, right eye, and mouth position regression loss, the condition that the respective target exists must be met. If the target does not exist, the calculation of the corresponding regression loss is not performed. Therefore, the loss function expression of the convolutional neural network model 2 is:

[0096] loss(dms)=loss(fs)+loss(fc)+loss(oc)+loss(oi)+loss(f_coord)+loss(o_coord).

[0097] The above design method of the loss function helps the network pay more attention to the detection when the target exists, thereby improving the detection accuracy.

[0098] In summary, the present application first collects the driving images of the driver in the actual driving state in real time through the vehicle-mounted collection device 1. Then, the convolutional neural network model 2 with a special processing algorithm is constructed to perform convolution processing on the driving images, so as to output the driving behavior category, driving behavior position information, driving state category, and driving state position information having a relative position correlation with the driving behavior position based on the driving images. Finally, the output result of the convolutional neural network model 2 is analyzed by using the decision module 3 to determine whether the driver is fatigue driving. On the one hand, the detection method provided by the present application can simultaneously realize the detection of the driving behavior and driving state of the driver, i.e., realizes end-to-end detection, which improves the detection efficiency and greatly reduces the space, computing power, and time overhead of the system, thereby greatly improving the real-time performance of the system. On the other hand, when detecting and processing the driving state position information, the present application does not independently regress, but correlates it with the driving behavior position information and uses a special relative position method for calculation and processing. In this way, the detection rate and accuracy of small targets can be greatly improved, and the behavior and state of the same person can be always correlated, thereby avoiding the confusion of behavior and state when multiple detection results exist and the problem of difficult correct matching.

[0099] Example 2:

[0100] Please refer to the attached Figure 2 Fig. 1 shows a specific workflow diagram of the convolutional neural network model according to the present application.

[0101] Based on the training learning of the convolutional neural network model 2, the convolutional neural network model 2 obtains the specific category of the driving behavior by feature extraction on the driving image; and the convolutional neural network model 2 obtains the driving behavior position information, the driving state position information having a relative position correlation with the driving behavior position, and the driving state category by detection and calculation on the driving image.

[0102] Further, the specific operation method of the convolutional neural network model 2 for detection and calculation on the driving image includes:

[0103] S21: using a prior box detection method to predict the driving behavior position box on the driving image to obtain the position offset A of the center point position of the driving behavior position box relative to the center point position of the prior box, the stretching amount A of the frame width of the driving behavior position box relative to the frame width of the prior box, and the stretching amount B of the frame height of the driving behavior position box relative to the frame height of the prior box; the obtained position offset A, stretching amount A and stretching amount B cooperatively reflect the neural network position regression value of the driving behavior position box.

[0104] Wherein, the prior box is obtained by using the kmeans clustering algorithm on the above training data set, and the prior box detection method belongs to the conventional technical means in the convolutional neural network model, so it is not described in detail here.

[0105] Please refer to the attached Figure 3 Fig. 2 shows that in the image coordinate system, the center point position coordinates of the prior box are (X * ,Y * ), the frame width is W * , and the frame height is H * ; the center point position coordinates of the driving behavior position box are (X0,Y0), the frame width is W, and the frame height is H. Then,

[0106] The position offset A of the center point position of the driving behavior position box relative to the center point position of the prior box is: ΔX=X * -X0, and ΔY=Y * -Y0;

[0107] The stretching amount A of the frame width of the driving behavior position box relative to the frame width of the prior box is The stretching amount B of the frame height of the driving behavior position box relative to the frame height of the prior box is Wherein, w and h are both normalized values.

[0108] The (△X, △Y, w, h) above reflects the neural network position regression value of the driving behavior position box.

[0109] S22: According to the center point position coordinates of the driving behavior position box on the driving image in the image coordinate system, the box width and the box height, and the relative position relationship between the driving behavior position box and the driving state position box on the driving image, the position offset B of the center point position of the driving state position box relative to the calibration point position of the driving behavior position box, the normalized values of the position offset B relative to the box width and the box height of the driving behavior position box, the stretching amount a of the box width of the driving state position box relative to the box width of the driving behavior position box, and the stretching amount b of the box height of the driving state position box relative to the box height of the driving behavior position box are calculated, and the normalized values of the position offset B, the stretching amount a and the stretching amount b cooperatively reflect the neural network position regression value of the driving state position box.

[0110] See the accompanying Figure 4 illustrated, in the image coordinate system of the driving image, the center point position coordinates of the driving behavior position box are (X0, Y0), the calibration point position coordinates are (X1, Y1), the box width is W, and the box height is H; and the calibration point position coordinates (X1, Y1) and the center point position coordinates (X0, Y0) satisfy the following relationship:

[0111]

[0112] The driving state position box includes a left eye position box, a right eye position box and a mouth position box, wherein the center point position coordinates of the left eye position box are (X l-eye ,Y l-eye ), the box width is W l-eye , and the box height is H l-eye ; the center point position coordinates of the right eye position box are (X r-eye ,Y r-eye ), the box width is W r-eye , and the box height is H r-eye ; the center point position coordinates of the mouth position box are (X mouth ,Y mouth ), the box width is W mouth , and the box height is H mouth ;

[0113] The position offset B of the center point position of the left eye position box relative to the calibration point position of the driving behavior position box is recorded as the left eye position offset B, and the left eye position offset B satisfies the following relationship:

[0114] △Xl-eye = X l-eye - X1, ΔY l-eye = Y l-eye - Y1;

[0115] and the normalized value of the left eye position offset B relative to the frame width and frame height of the driving behavior position frame respectively satisfies the following relationship:

[0116]

[0117] The frame width of the left eye position frame relative to the frame width of the driving behavior position frame is recorded as the left eye frame width scaling amount a, and the left eye frame width scaling amount a satisfies the following relationship: In the formula, w l-eye is a normalized value;

[0118] The frame height of the left eye position frame relative to the frame height of the driving behavior position frame is recorded as the left eye frame height scaling amount b, and the left eye frame height scaling amount b satisfies the following relationship: In the formula, h l-eye is a normalized value;

[0119] That is: (X l-true , Y l-true , w l-eye , h l-eye ) reflects the neural network position regression value of the left eye position frame;

[0120] The center point position of the right eye position frame relative to the calibration point position of the driving behavior position frame is recorded as the right eye position offset B, and the right eye position offset B satisfies the following relationship:

[0121] ΔX r-eye = X r-eye - X1, ΔY r-eye = Y r-eye - Y1;

[0122] and the normalized value of the right eye position offset B relative to the frame width and frame height of the driving behavior position frame respectively satisfies the following relationship:

[0123]

[0124] The frame width of the right eye position frame relative to the frame width of the driving behavior position frame is recorded as the right eye frame width scaling amount a, and the right eye frame width scaling amount a satisfies the following relationship: In the formula, w r-eye is a normalized value;

[0125] A scaling amount b of the height of the right eye position frame relative to the height of the driving behavior position frame is recorded as a right eye height scaling amount b, and the right eye height scaling amount b satisfies the following relationship: In the formula, h r-eye is a normalized value.

[0126] That is, (X r-true , Y r-true , w r-eye , h r-eye ) reflects the neural network position regression value of the right eye position frame.

[0127] A position offset B of the center point of the mouth position frame relative to the calibration point of the driving behavior position frame is recorded as a mouth position offset B, and the mouth position offset B satisfies the following relationship:

[0128] △X mouth = X mouth - X1, and △Y mouth = Y mouth - Y1.

[0129] And the normalized value of the mouth position offset B relative to the width and height of the driving behavior position frame satisfies the following relationship:

[0130]

[0131] A scaling amount a of the width of the mouth position frame relative to the width of the driving behavior position frame is recorded as a mouth width scaling amount a, and the mouth width scaling amount a satisfies the following relationship: In the formula, w mouth is a normalized value.

[0132] A scaling amount b of the height of the mouth position frame relative to the height of the driving behavior position frame is recorded as a mouth height scaling amount b, and the mouth height scaling amount b satisfies the following relationship: In the formula, h mouth is a normalized value.

[0133] That is, (X m-true , Y m-true , w mouth , h mouth ) reflects the neural network position regression value of the mouth position frame.

[0134] In addition, based on the neural network position regression value of the driving state position frame and the specific category of the driving state, the occurrence probability of any specific category of the driving state can be calculated. For example, by detecting (X r-true , Y r-true , wr-eye h r-eye )、(X l-true ,Y l-true ,w l-eye ,h l-eye ), the open-eye and closed-eye probabilities of the left and right eyes of the driver in this period of time can be obtained. At this time, the decision module 3 analyzes and determines whether the driver is fatigue driving according to the neural network position regression value of the driving behavior position frame, the specific category of the driving behavior, the neural network position regression value of the driving state position frame, and the occurrence probability of the specific category of the driving state.

[0135] In summary, the convolutional neural network model 2 does not independently regress when detecting and processing the driving state position information, but associates it with the driving behavior position information and uses a specific relative position method for calculation and processing.

[0136] The above detection method of the convolutional neural network model 2 has the following advantages:

[0137] ① Generally, the target size of the driving behavior position frame is larger, which is conducive to detection, while the target size of the driving state position frame (left eye position frame, right eye position frame and mouth position frame) is very small, which is very unfavorable for detection. Associating the two is equivalent to requiring the convolutional neural network model 2 to find the positions of the eyes and mouth in the easily detected driving behavior position frame, rather than in the full image, which to some extent solves the difficulty of detecting small targets such as eyes and mouth and improves the detection rate of small targets.

[0138] ② By finding the positions of the eyes and mouth in the driving behavior position frame, it can be ensured that the detected eye and mouth positions are contained in the driving behavior position frame, which also conforms to the actual fact, avoids the bizarre error that the detection result of the eyes and mouth is located outside the driving behavior position frame, and to some extent improves the position detection accuracy.

[0139] ③ Compared with the method of separately detecting the eyes and mouth, such detection can always associate the driving behavior and driving state of the same person, realize end-to-end synchronous detection, and avoid the problem that the behavior and state of the same person cannot be correctly matched when there are multiple detection results in the image.

[0140] Embodiment 3:

[0141] Please refer to the accompanying Figure 5 , which is a block structure schematic diagram of the driving behavior and state detection device based on deep learning.

[0142] The driving behavior and state detection device based on deep learning comprises a vehicle-mounted collection device 1, a convolutional neural network model 2 and a decision module 3, wherein

[0143] The vehicle-mounted collection device 1 is configured to collect driving images of a driver in an actual driving state in real time.

[0144] The convolutional neural network model 2 is configured to receive the driving images, and output driving behavior categories, driving behavior position information, driving state categories and driving state position information having a relative position correlation with the driving behavior position on the driving images after convolutional processing of the driving images.

[0145] The decision module 3 is configured to receive the output result of the convolutional neural network model 2 and analyze the output result to determine whether the driver is driving while tired.

[0146] Further preferably, the vehicle-mounted collection device 1 adopts a vehicle-mounted camera.

[0147] The convolutional neural network model 2 adopts a VGGNet convolutional network model as a feature extraction subnetwork, and further preferably adopts a VGG16 convolutional neural network model.

[0148] The decision module 3 adopts a logic judgment module, and the specific structure form is not limited as long as it has a logic judgment function.

[0149] The driving behavior and state detection device provided in the embodiment can execute the driving behavior and state detection method based on deep learning provided in any embodiment of the application, and has the beneficial effects corresponding to the execution method.

[0150] In summary, the driving behavior and state detection method based on deep learning has high detection accuracy and high detection efficiency, and also reduces the space, computing power and time consumption of the system, and greatly improves the real-time performance of the system.

[0151] In the above description, many specific details are set forth in order to provide a thorough understanding of the present application. However, the above description is only a preferred embodiment of the present application, and the present application can be implemented in many other ways different from the description. Therefore, the present application is not limited by the specific implementation disclosed above. Meanwhile, any person skilled in the art can make many possible changes and modifications to the technical solutions disclosed above, or modify them into equivalent embodiments, without departing from the scope of the technical solutions of the present application. Any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present application, without departing from the technical solutions of the present application, all still belong to the protection scope of the technical solutions of the present application.

Claims

1. A method for detecting driving behavior and state based on deep learning, characterized in that: include: Configure an on-board acquisition device (1), which uses an on-board camera to collect driving images of the driver in actual driving conditions in real time; Convolutional neural network model (2) is constructed. The convolutional neural network model (2) is configured to receive the driving image, perform convolution processing on the driving image, and output the driving behavior category, driving behavior location information, driving state category, and driving state location information that has a relative positional relationship with the driving behavior location on the driving image. The convolutional neural network model (2) uses the VGGNet convolutional network model as the feature extraction sub-network, and the specific operation method of the convolutional neural network model (2) for detecting and calculating the driving image includes: The driving behavior location box on the driving image is predicted using a prior box detection method. The resulting values ​​are: position offset A of the center point of the driving behavior location box relative to the center point of a pre-set prior box; expansion A of the width of the driving behavior location box relative to the width of the prior box; and expansion B of the height of the driving behavior location box relative to the height of the prior box. The obtained position offset A, expansion A, and expansion B together reflect the neural network position regression value of the driving behavior location box. Based on the center point coordinates, width, and height of the driving behavior position box in the image coordinate system, and the relative positional relationship between the driving behavior position box and the driving state position box, the following calculations are made: the position offset B of the center point of the driving state position box relative to the calibration point of the driving behavior position box; the normalized values ​​of the position offset B relative to the width and height of the driving behavior position box; the scaling a of the width of the driving state position box relative to the width of the driving behavior position box; and the scaling b of the height of the driving state position box relative to the height of the driving behavior position box. The obtained normalized values, scaling a, and scaling b together reflect the neural network position regression value of the driving state position box. The configuration decision module (3) is used to receive the output of the convolutional neural network model (2) and analyze the output to determine whether the driver is driving while fatigued.

2. The deep learning-based driving behavior and state detection method according to claim 1, characterized in that: Establish training and validation datasets for training the convolutional neural network model (2).

3. The deep learning-based driving behavior and state detection method according to claim 2, characterized in that: The training dataset and validation dataset are obtained by collecting images of the driver in actual driving conditions, preprocessing the obtained training images, classifying and labeling the preprocessed training images, and then training the convolutional neural network model (2) for training.

4. The deep learning-based driving behavior and state detection method according to claim 3, characterized in that: The annotations on the preprocessed training images include driving behavior information and driving state information, among which... The driving behavior information includes a driving behavior location box and a specific category of driving behavior, and the categories of driving behavior include normal driving, driving while smoking, driving while drinking water, and driving while making or receiving phone calls. The driving status information includes a driving status location box and a specific category of driving status, and the categories of driving status include left eye open, left eye closed, right eye open, right eye closed, mouth open, and mouth closed.

5. The deep learning-based driving behavior and state detection method according to claim 4, characterized in that: Based on the training and learning of the convolutional neural network model (2), the convolutional neural network model (2) extracts features from the driving image to obtain the specific category of the driving behavior; Furthermore, the convolutional neural network model (2) obtains the driving behavior location information, the driving state location information with a relative positional relationship to the driving behavior location, and the driving state category by detecting and calculating the driving image.

6. The deep learning-based driving behavior and state detection method according to claim 5, characterized in that: In the image coordinate system of the driving image, the center point coordinates of the driving behavior location box are: The coordinates of the calibration point are The frame width is The frame height is ; The driving state position frame includes a left eye position frame, a right eye position frame, and a mouth position frame, wherein the center point coordinates of the left eye position frame are: The frame width is The frame height is The coordinates of the center point of the right eye position frame are: The frame width is The frame height is The coordinates of the center point of the mouth position frame are: The frame width is The frame height is ; The positional offset B of the center point of the left eye position frame relative to the calibration point of the driving behavior position frame is denoted as the left eye position offset B, and the left eye position offset B satisfies the following relationship: ; Furthermore, the normalized values ​​of the left eye position offset B relative to the width and height of the driving behavior position box satisfy the following relationships: ； The amount of scaling 'a' between the width of the left eye position frame and the width of the driving behavior position frame is denoted as the left eye frame width scaling 'a'. The left eye frame width scaling 'a' satisfies the following relationship: ; The amount of scaling (b) of the left eye position frame height relative to the driving behavior position frame height is denoted as the left eye frame height scaling (b). The left eye frame height scaling (b) satisfies the following relationship: ; Right now: This reflects the neural network position regression value of the left eye location box; The positional offset B of the center point of the right eye position frame relative to the calibration point of the driving behavior position frame is denoted as the right eye position offset B, and the right eye position offset B satisfies the following relationship: ； Furthermore, the normalized values ​​of the right eye position offset B relative to the width and height of the driving behavior position box satisfy the following relationships: ； The amount of scaling 'a' between the width of the right eye position frame and the width of the driving behavior position frame is denoted as the right eye frame width scaling 'a'. The right eye frame width scaling 'a' satisfies the following relationship: ; The amount of scaling up or down of the right eye position frame relative to the height of the driving behavior position frame, denoted as b, is the right eye frame height scaling up or down amount. The right eye frame height scaling up or down amount b satisfies the following relationship: ; Right now: This reflects the neural network position regression value of the right eye location box; The offset B of the center point of the mouth position box relative to the calibration point of the driving behavior position box is denoted as the mouth position offset B, and the mouth position offset B satisfies the following relationship: ; Furthermore, the normalized values ​​of the mouth position offset B relative to the width and height of the driving behavior position box satisfy the following relationships: ； The amount of scaling 'a' between the width of the mouth position frame and the width of the driving behavior position frame is denoted as the mouth frame width scaling 'a'. The mouth frame width scaling 'a' satisfies the following relationship: ; The amount of scaling (b) of the height of the mouth position frame relative to the height of the driving behavior position frame is denoted as the mouth frame height scaling (b). The mouth frame height scaling (b) satisfies the following relationship: ; Right now: This reflects the neural network position regression value of the mouth position box.

7. The deep learning-based driving behavior and state detection method according to claim 6, characterized in that: Based on the neural network position regression value of the driving state position box and the specific category of the driving state, the probability of occurrence of any specific category of the driving state can be calculated. The decision module (3) analyzes and determines whether the driver is driving while fatigued based on the neural network position regression value of the driving behavior position box, the specific category of the driving behavior, the neural network position regression value of the driving state position box, and the probability of occurrence of the specific category of the driving state.

8. A driving behavior and state detection device based on deep learning, characterized in that: include: Vehicle-mounted acquisition device (1), the vehicle-mounted acquisition device (1) uses a vehicle-mounted camera to collect driving images of the driver in actual driving conditions in real time; The convolutional neural network model (2) is configured to receive the driving image, perform convolution processing on the driving image, and output driving behavior category, driving behavior location information, driving state category, and driving state location information that has a relative positional relationship with the driving behavior location on the driving image. In addition, the convolutional neural network model (2) uses the VGGNet convolutional network model as the feature extraction sub-network, and the specific operation method of the convolutional neural network model (2) for detecting and calculating the driving image includes: The driving behavior location box on the driving image is predicted using a prior box detection method. The resulting values ​​are: position offset A of the center point of the driving behavior location box relative to the center point of a pre-set prior box; expansion A of the width of the driving behavior location box relative to the width of the prior box; and expansion B of the height of the driving behavior location box relative to the height of the prior box. The obtained position offset A, expansion A, and expansion B together reflect the neural network position regression value of the driving behavior location box. Based on the center point coordinates, width, and height of the driving behavior position box in the image coordinate system, and the relative positional relationship between the driving behavior position box and the driving state position box, the following calculations are made: the position offset B of the center point of the driving state position box relative to the calibration point of the driving behavior position box; the normalized values ​​of the position offset B relative to the width and height of the driving behavior position box; the scaling a of the width of the driving state position box relative to the width of the driving behavior position box; and the scaling b of the height of the driving state position box relative to the height of the driving behavior position box. The obtained normalized values, scaling a, and scaling b together reflect the neural network position regression value of the driving state position box. The decision module (3) is configured to receive the output of the convolutional neural network model (2) and analyze the output to determine whether the driver is driving while fatigued.

Images