Driver attention monitoring method and device with embedded driving style

Abstract

The invention relates to the technical field of automobile safety, in particular to a driver attention monitoring method and device with an embedded driving style. The method comprises the following steps of: firstly, acquiring sight angle data of a driver at the current moment, vehicle driving state data, driver operation data at the previous moment and current actual driver operation data; and then, inputting the sight angle data of the driver at the current moment, the vehicle driving state data and the driver operation data at the previous moment into the trained recurrent neural network model together, so as to obtain the currently predicted driver operation data. And determining the driving style of the current driver based on the current predicted driver operation data and the actual operation data. If the driving style of the current driver is significantly different from the driving style in the historical record, it is indicated that the driver may be inattention. At the moment, corresponding prompt information needs to be generated to remind a driver to concentrate attention, and driving safety is ensured.

Description

Technical Field

[0001] The present invention relates to the field of automobile safety technology, and in particular to a driver attention monitoring method and device with embedded driving style. Background Art

[0002] With the rapid development of my country's economy and the improvement of people's living standards, cars have gradually become an indispensable means of transportation for people's daily travel. In recent years, although China's road traffic construction has achieved rapid development, some traffic travelers have weak safety awareness and legal awareness, resulting in frequent road traffic accidents. In a small sense, traffic accidents are an infringement of others' personal health rights and property ownership, affecting the happiness of individuals or families; in a big sense, traffic accidents weaken the competitiveness of the entire country and have a huge impact on the country's economic construction, social stability and long-term peace.

[0003] In recent years, with the continuous improvement of vehicle intelligence, a large number of sensors on the vehicle can obtain the status signals of the driver and the vehicle body in real time; at the same time, the rapid rise of artificial intelligence algorithms such as deep learning has provided a technical basis for analyzing the above signal data; finally, the continuous improvement of the computing power of on-board chips has provided a platform foundation for the operation of these complex algorithms. Based on the above factors, research on driver attention monitoring has gradually emerged in recent years and has become a research hotspot in the field of traffic safety.

[0004] There are already many technologies related to driver monitoring. One method is to determine whether the driver is in a distracted driving state by analyzing the driver's eyes, mouth, head posture, steering wheel, acceleration and deceleration and other characteristics; another mainstream method is to use on-board sensors to collect information such as the vehicle's running trajectory, acceleration and deceleration, and compare it with the parameters under normal driving conditions to analyze whether the driver is in a distracted driving state.

[0005] However, the actual effect of the above driver attention monitoring technology is not ideal. The reason is that the above methods do not take into account the impact of driving style on driver behavior characteristics and vehicle behavior characteristics. In fact, every driver has his own driving style, and the driver behavior characteristics and vehicle behavior characteristics under different driving styles are different. It is difficult to use a unified logical criterion to determine whether the driver is in a state of distraction.

[0006] Based on this, the present invention proposes a driver attention monitoring method and device with embedded driving style to solve the above technical problems. Summary of the invention

[0007] The present invention describes a driver attention monitoring method and device with built-in driving style, which can accurately monitor the driver's attention and thus ensure driving safety.

[0008] According to a first aspect, the present invention provides a method for monitoring driver attention embedded in driving style, the method comprising:

[0009] Obtain the current driver's sight angle, the current vehicle driving state data, the driver's operation data at the previous moment, and the current actual driver operation data;

[0010] Inputting the current driver's sight angle, the current vehicle driving state data and the previous driver's operation data into the trained recurrent neural network model to obtain the currently predicted driver's operation data;

[0011] determining a current driver's driving style based on the current predicted driver operation data and the current actual driver operation data;

[0012] Prompt information is determined based on the current driver's driving style and historical drivers' driving styles.

[0013] According to a second aspect, the present invention provides a driver attention monitoring device with built-in driving style, comprising:

[0014] An acquisition unit is configured to acquire the current driver's sight angle, the current vehicle driving state data, the driver's operation data at the previous moment, and the current actual driver operation data;

[0015] A first data processing unit is configured to input the current driver's sight angle, the current vehicle driving state data and the driver's operation data at the previous moment into a trained recurrent neural network model to obtain currently predicted driver operation data;

[0016] a second data processing unit configured to determine a driving style of a current driver based on the current predicted driver operation data and the current actual driver operation data;

[0017] The third data processing unit is configured to determine prompt information based on the current driver's driving style and the historical driver's driving style.

[0018] In a third aspect, an embodiment of this specification further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in any embodiment of this specification is implemented.

[0019] In a fourth aspect, an embodiment of the present specification further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method described in any embodiment of the present specification.

[0020] According to the driver attention monitoring method and device with embedded driving style provided by the present invention, first, the driver's line of sight angle data, vehicle driving status data, driver operation data at the previous moment and current actual driver operation data are collected. Subsequently, the driver's line of sight angle data, vehicle driving status data at the current moment and driver operation data at the previous moment are input into the trained recurrent neural network model together to obtain the current predicted driver operation data. Based on the current predicted driver operation data and actual operation data, the current driver's driving style is determined. If there is a significant difference between the current driver's driving style and the driving style in the historical records, it indicates that the driver may be inattentive. At this time, corresponding prompt information needs to be generated to remind the driver to concentrate and ensure driving safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 A schematic flow chart of a method for monitoring driver attention embedded in driving style according to an embodiment is shown;

[0023] Figure 2 A schematic block diagram of a driver attention monitoring device embedded with driving style according to one embodiment is shown;

[0024] Figure 3 A schematic diagram showing the Euler angles of the driver's eyeballs and Euler angles of the driver's face according to one embodiment. DETAILED DESCRIPTION

[0025] The solution provided by the present invention is described below in conjunction with the accompanying drawings.

[0026] Figure 1 FIG. 1 is a flow chart showing a method for monitoring driver attention embedded in driving style according to an embodiment. It is understood that the method can be executed by any device, equipment, platform, or device cluster with computing and processing capabilities. Figure 1 As shown, the method includes:

[0027] Step 100, obtaining the current driver's sight angle, the current vehicle driving state data, the driver's operation data at the previous moment, and the current actual driver operation data;

[0028] Step 102: inputting the current driver's sight angle, the current vehicle driving state data and the driver's operation data at the previous moment into the trained recurrent neural network model to obtain the currently predicted driver operation data;

[0029] Step 104: determining the current driver's driving style based on the currently predicted driver operation data and the current actual driver operation data;

[0030] Step 106: Determine prompt information based on the current driver's driving style and historical drivers' driving styles.

[0031] In this embodiment, first, the driver's line of sight angle data, vehicle driving state data, driver operation data at the previous moment and actual driver operation data at the moment are collected. Subsequently, the driver's line of sight angle data, vehicle driving state data and driver operation data at the previous moment are input into the trained recurrent neural network model together to obtain the currently predicted driver operation data. Based on the currently predicted driver operation data and actual operation data, the current driver's driving style is determined. If there is a significant difference between the current driver's driving style and the driving style in the historical records, it indicates that the driver may be inattentive. At this time, a corresponding prompt message needs to be generated to remind the driver to concentrate and ensure driving safety.

[0032] In one embodiment of the present invention, the vehicle driving state data includes the vehicle speed, acceleration, yaw rate and sideslip angle of the center of mass of the vehicle, and the driver operation data includes the driver's steering wheel angle, accelerator pedal depth and brake pedal depth.

[0033] In this embodiment, the vehicle speed, acceleration, yaw rate, center of mass sideslip angle, driver's steering wheel angle, accelerator pedal depth and brake pedal depth can be collected through the vehicle bus. In addition, the vehicle bus is a network for data communication between various electronic control units (ECUs) in the vehicle. It can efficiently and accurately transmit the data collected by various sensors, such as the driver's sight angle sensor, the vehicle status monitoring sensor, etc., to the module responsible for processing, so as to input these data into the recurrent neural network model. At the same time, after determining the driver's driving style, if it is found that there is a lack of concentration, the generated prompt information may also be quickly transmitted to the display device or voice prompt system in the car with the help of the vehicle bus to remind the driver in time. Different types of vehicle buses, such as controller area network (CAN) bus, local interconnect network (LIN) bus, FlexRay bus, etc., have their own characteristics in data transmission rate, reliability, etc., and are suitable for different data transmission scenarios. For example, the CAN bus is often used to transmit vehicle driving status data with high reliability requirements due to its high reliability and good error detection mechanism; while the LIN bus is suitable for some sensor data transmission with low data transmission rate requirements, such as the status data of some simple switches in the car, due to its low cost and low speed.

[0034] In one embodiment of the present invention, the current driver's sight angle is determined by the following steps:

[0035] Acquire image data of the driver's upper body at the current moment;

[0036] Determine the Euler angles of the driver's face, the eyeball Euler angles of the left eye, and the eyeball Euler angles of the right eye based on the image data of the upper body;

[0037] Based on the Euler angles of the driver's face, the eyeball Euler angles of the left eye, and the eyeball Euler angles of the right eye, the driver's sight angle at the current moment is determined.

[0038] In this embodiment, the vehicle's built-in camera can be used to collect image data of the driver's upper body at the current moment, and the image data of the upper body is used to extract eye feature information and facial feature information, thereby determining the driver's face Euler angles, left eye Euler angles, and right eye Euler angles, and then based on the driver's face Euler angles, left eye Euler angles, and right eye Euler angles, the driver's line of sight angle at the current moment is jointly determined to improve the accuracy of detecting the driver's line of sight angle at the current moment.

[0039] In one embodiment of the present invention, based on the Euler angles of the driver's face, the Euler angles of the left eye, and the Euler angles of the right eye, determining the sight angle of the driver at the current moment includes:

[0040] Determine a first sight angle according to the yaw angle in the Euler angle of the face and the sight angle in the Euler angle of the eyeball of the left eye;

[0041] Determine a second sight angle according to the yaw angle in the Euler angle of the face and the sight angle in the Euler angle of the eyeball of the right eye;

[0042] Based on the first sight line angle and the second sight line angle, the sight line angle of the driver at the current moment is determined.

[0043] like Figure 3 As shown, in this embodiment, the driver's left eyeball Euler angle (x e y e z e ), the Euler angles of the driver's face (xh yh zh), where the yaw angle x h is the turning angle of the driver to observe the surroundings, x e is the sight angle when the driver observes the surrounding environment. h and x e The coordinate axes in the image are parallel, and the distance deviation between the two is not large. Therefore, the present invention adds the two together as the driver's sight angle in the driver's face coordinate system, that is, α=x e +x h α is the first sight angle. Similarly, the second sight angle is calculated in the same way as the first sight angle. Since the sight angles calculated for the left and right eyes are different, the present invention adds the first sight angle and the second sight angle and divides them by two to determine the sight angle of the driver at the current moment, thereby improving the accuracy of calculating the sight angle to ensure the recognition accuracy of the subsequent model.

[0044] In one embodiment of the present invention, determining the current driver's driving style based on the current predicted driver operation data and the current actual driver operation data includes:

[0045] determining a change amount of the driver operation data according to currently predicted driver operation data and currently actual driver operation data;

[0046] Determine the current driving style of the driver based on the change in the driver's operation data;

[0047] The current driver's driving style includes a first driving style, a second driving style, a third driving style and a fourth driving style.

[0048] In this embodiment, the change amount of the driver operation data is determined based on the current predicted driver operation data minus the current actual driver operation data. The greater the absolute value of the change amount of the driver operation data, the more intense the driving style is. The current driver's driving style can be determined based on the change amount of the driver operation data. The current driver's driving style includes a first driving style, a second driving style, a third driving style, and a fourth driving style. The driving intensity of the first driving style, the second driving style, the third driving style, and the fourth driving style increases in sequence.

[0049] In one embodiment of the present invention, based on the current driver's driving style and the historical driver's driving style, the prompt information is determined, including:

[0050] Determining the difference in driving style according to the current driver's driving style and the historical driver's driving style;

[0051] Determine prompt information based on driving style differences;

[0052] The prompt information includes first prompt information, second prompt information, third prompt information and fourth prompt information.

[0053] In this embodiment, the driving style of the current driver is compared with the driving style of the historical driver to determine the difference of the driving style. Different quantitative standards can be set to measure this difference, for example, the difference range of a certain driving style characteristic parameter is used as the basis for judgment, or the difference between the first driving style and the third driving style is two. Based on the difference of driving style, the corresponding prompt information is determined. The prompt information includes the first prompt information, the second prompt information, the third prompt information and the fourth prompt information. Specifically: if the driving style difference is in a lower range, it means that the current driving style is less different from the historical style. At this time, it is determined as the first prompt information, such as "the driving state is stable, continue to maintain", to confirm the current driving state of the driver. When the difference rises to a medium-low range, the second prompt information is given, such as "the driving style has changed slightly, please pay attention to driving habits", to remind the driver to pay attention to the changes in his own driving behavior. If the difference is in a medium-high range, it means that the driving style difference is more obvious, and it is determined as the third prompt information, such as "the driving style has changed a lot, please pay attention to concentrate", to warn the driver that there may be a situation of inattention. Once the difference reaches a higher range, it means that there is a significant difference in driving style. At this time, the fourth prompt information is given, such as "Abnormal driving style, be sure to adjust the driving state immediately", emphasizing the seriousness of the situation and urging the driver to make quick adjustments. At the same time, after determining the driver's driving style, if it is found that there is a lack of concentration, the generated prompt information can also be quickly transmitted to the display device or voice prompt system in the car through the vehicle bus to remind the driver in time.

[0054] In one embodiment of the present invention, the recurrent neural network model is determined by the following steps:

[0055] Obtain historical driver operation data and historical vehicle driving status data;

[0056] The initial recurrent neural network model is trained based on the historical driver operation data and the historical vehicle driving state data to obtain a trained recurrent neural network model;

[0057] Among them, the initial recurrent neural network model is obtained by migrating data from the existing model.

[0058] In this embodiment, the determination process of the recurrent neural network model is as follows: first, historical driver operation data and historical vehicle driving state data are obtained. The initial recurrent neural network model is obtained by data migration of the existing model. The principle of data migration learning is to use the model parameters pre-trained on other related tasks or data sets as the initialization parameters of the new model. This is because the model pre-trained on large-scale data has learned some common feature representations, which can accelerate the convergence speed of the model and improve performance for new similar tasks. Then, based on the acquired historical driver operation data and historical vehicle driving state data, the initial model is trained. The core principle of the recurrent neural network (RNN) is that its hidden layer can not only receive the current input information, but also retain the hidden state information of the previous moment, so as to have memory ability for the sequence data. During the training process, the gradient is calculated and the model parameters are updated by the back propagation time algorithm (BPTT), so that the model can learn the complex relationship between the input sequence (historical driver operation data and historical vehicle driving state data) and the output (predicted driver operation data), and finally obtain the trained recurrent neural network model.

[0059] According to another embodiment, the present invention provides a driver attention monitoring device with built-in driving style. Figure 2 FIG. 1 is a schematic block diagram of a driver attention monitoring device embedded with driving style according to an embodiment. It is understood that the device can be implemented by any device, equipment, platform and device cluster with computing and processing capabilities. Figure 2 As shown, the device includes: an acquisition unit 200, a first data processing unit 202, a second data processing unit 204, and a third data processing unit 206. The main functions of each component unit are as follows:

[0060] The acquisition unit 200 is configured to acquire the current driver's sight angle, the current vehicle driving state data, the driver's operation data at the previous moment, and the current actual driver operation data;

[0061] The first data processing unit 202 is configured to input the current driver's sight angle, the current vehicle driving state data and the driver's operation data at the previous moment into the trained recurrent neural network model to obtain the currently predicted driver's operation data;

[0062] A second data processing unit 204 is configured to determine a driving style of a current driver based on the current predicted driver operation data and the current actual driver operation data;

[0063] The third data processing unit 206 is configured to determine prompt information based on the current driver's driving style and the historical driver's driving style.

[0064] In one embodiment of the present invention, the vehicle driving state data includes the vehicle speed, acceleration, yaw rate and sideslip angle of the center of mass of the vehicle, and the driver operation data includes the driver's steering wheel angle, accelerator pedal depth and brake pedal depth.

[0065] In one embodiment of the present invention, the current driver's sight angle is determined by the following steps:

[0066] Acquire image data of the driver's upper body at the current moment;

[0067] Based on the image data of the upper body, determining the Euler angles of the driver's face, the eyeball Euler angles of the left eye, and the eyeball Euler angles of the right eye;

[0068] Based on the Euler angles of the driver's face, the eyeball Euler angles of the left eye, and the eyeball Euler angles of the right eye, the sight angle of the driver at the current moment is determined.

[0069] In one embodiment of the present invention, determining the sight angle of the driver at the current moment based on the Euler angles of the driver's face, the eyeball Euler angles of the left eye, and the eyeball Euler angles of the right eye includes:

[0070] Determine a first sight angle according to the yaw angle in the Euler angles of the face and the sight angle in the eyeball Euler angles of the left eye;

[0071] Determine a second sight angle according to the yaw angle in the Euler angles of the face and the sight angle in the eyeball Euler angles of the right eye;

[0072] Based on the first sight line angle and the second sight line angle, the sight line angle of the driver at the current moment is determined.

[0073] In one embodiment of the present invention, the second data processing unit 204 is configured to perform the following steps:

[0074] determining a change amount of the driver operation data according to the currently predicted driver operation data and the currently actual driver operation data;

[0075] determining the driving style of the current driver based on the amount of change in the driver operation data;

[0076] The driving style of the current driver includes a first driving style, a second driving style, a third driving style and a fourth driving style.

[0077] In one embodiment of the present invention, the third data processing unit 206 is configured to perform the following steps:

[0078] Determining a difference in driving style according to the current driver's driving style and the historical driver's driving style;

[0079] Determining prompt information based on the difference in driving styles;

[0080] The prompt information includes first prompt information, second prompt information, third prompt information and fourth prompt information.

[0081] In one embodiment of the present invention, the recurrent neural network model is determined by the following steps:

[0082] Obtain historical driver operation data and historical vehicle driving status data;

[0083] Training an initial recurrent neural network model based on the historical driver operation data and the historical vehicle driving state data to obtain a trained recurrent neural network model;

[0084] Among them, the initial recurrent neural network model is obtained by migrating data from the existing model.

[0085] According to another embodiment, there is also provided a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute a combination of Figure 1 The method described.

[0086] According to another embodiment of the present invention, there is also provided an electronic device, comprising a memory and a processor, wherein the memory stores an executable code, and when the processor executes the executable code, the Figure 1 The method described.

[0087] The various embodiments of the present invention are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0088] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the present invention can be implemented by hardware, software, firmware or any combination thereof. When implemented by software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0089] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solution of the present invention should be included in the scope of protection of the present invention.

Claims

1. A driver attention monitoring method with built-in driving style, characterized in that: The method comprises: Obtain the current driver's sight angle, the current vehicle driving state data, the driver's operation data at the previous moment, and the current actual driver operation data; Inputting the current driver's sight angle, the current vehicle driving state data and the previous driver's operation data into the trained recurrent neural network model to obtain the currently predicted driver's operation data; determining a current driver's driving style based on the current predicted driver operation data and the current actual driver operation data; Prompt information is determined based on the current driver's driving style and historical drivers' driving styles.

2. The method according to claim 1, characterized in that The vehicle driving state data includes the vehicle speed, acceleration, yaw rate and sideslip angle of the center of mass of the vehicle, and the driver operation data includes the driver's steering wheel angle, accelerator pedal depth and brake pedal depth.

3. The method according to claim 2, characterized in that The current driver's sight angle is determined by the following steps: Acquire image data of the driver's upper body at the current moment; Based on the image data of the upper body, determining the Euler angles of the driver's face, the eyeball Euler angles of the left eye, and the eyeball Euler angles of the right eye; Based on the Euler angles of the driver's face, the eyeball Euler angles of the left eye, and the eyeball Euler angles of the right eye, the sight angle of the driver at the current moment is determined.

4. The method according to claim 3, characterized in that The determining the sight angle of the driver at the current moment based on the Euler angles of the driver's face, the eyeball Euler angles of the left eye, and the eyeball Euler angles of the right eye includes: Determine a first sight angle according to the yaw angle in the Euler angles of the face and the sight angle in the eyeball Euler angles of the left eye; Determine a second sight angle according to the yaw angle in the Euler angles of the face and the sight angle in the eyeball Euler angles of the right eye; Based on the first sight line angle and the second sight line angle, the sight line angle of the driver at the current moment is determined.

5. The method according to claim 4, characterized in that The determining the driving style of the current driver based on the current predicted driver operation data and the current actual driver operation data includes: determining a change amount of the driver operation data according to the currently predicted driver operation data and the currently actual driver operation data; determining the driving style of the current driver based on the amount of change in the driver operation data; The driving style of the current driver includes a first driving style, a second driving style, a third driving style and a fourth driving style.

6. The method according to claim 5, characterized in that The determining of prompt information based on the current driver's driving style and the historical driver's driving style includes: Determining a difference in driving style according to the current driver's driving style and the historical driver's driving style; Determining prompt information based on the difference in driving styles; The prompt information includes first prompt information, second prompt information, third prompt information and fourth prompt information.

7. The method according to claim 6, characterized in that The recurrent neural network model is determined by the following steps: Obtain historical driver operation data and historical vehicle driving status data; Training an initial recurrent neural network model based on the historical driver operation data and the historical vehicle driving state data to obtain a trained recurrent neural network model; Among them, the initial recurrent neural network model is obtained by migrating data from the existing model.

8. A driver attention monitoring device with built-in driving style, characterized in that: include: An acquisition unit is configured to acquire the current driver's sight angle, the current vehicle driving state data, the driver's operation data at the previous moment, and the current actual driver operation data; A first data processing unit is configured to input the current driver's sight angle, the current vehicle driving state data and the driver's operation data at the previous moment into a trained recurrent neural network model to obtain currently predicted driver operation data; a second data processing unit configured to determine a driving style of a current driver based on the current predicted driver operation data and the current actual driver operation data; The third data processing unit is configured to determine prompt information based on the current driver's driving style and the historical driver's driving style.

9. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed in a computer, the computer is caused to execute the method according to any one of claims 1 to 7.

Images