Driver behavior analysis safety early warning system and method based on deep learning

Abstract

The invention discloses a driver behavior analysis safety early warning system and method based on deep learning, and relates to the technical field of driver behavior analysis. The method comprises the steps of collecting driving data of a driver, and preprocessing to obtain original data; after feature extraction is carried out by using a deep learning algorithm, model training is respectively carried out; analyzing the physiological and psychological states of the driver and obtaining a physiological value and a psychological value; the driving skill level of the driver is obtained, and an operation danger value of the driver is obtained by combining operation feature analysis; acquiring reflection conditions of crowds around a driver, and analyzing by combining environment characteristics to obtain an environment danger value; performing single judgment on the data trained by the model, and giving an alarm if the driver behavior is dangerous; and if the driver behavior is not dangerous, carrying out integration judgment on all data passing model training, and if the driver behavior is dangerous, giving an alarm. According to the invention, the accuracy of driver behavior analysis safety early warning based on deep learning is improved.

Description

Technical Field

[0001] The present application relates to the technical field of driver behavior analysis, and in particular to a driver behavior analysis safety warning system and method based on deep learning. Background Art

[0002] With the rapid development of the transportation industry and the sharp increase in the number of vehicles, road traffic safety issues have become increasingly prominent. As a core component of the transportation system, the driving behavior of drivers has a vital impact on road traffic safety. In recent years, traffic accidents caused by improper behaviors such as driver fatigue driving, distracted driving, and illegal operations have occurred frequently, causing huge casualties and property losses to the society. Since there are no traffic lights, standardized road driving and other equipment rules at the construction site, the operating behavior of construction equipment drivers is more seriously related to the safety of construction workers. Analyzing the behavior of construction equipment drivers and issuing timely safety warnings in dangerous situations can effectively improve the safety of the construction site.

[0003] Currently, driving safety warnings are implemented by identifying whether the driver's operating behavior complies with traffic rules. However, since there are no traffic equipment at the construction site and the driving rules of construction equipment are different, the existing driving behavior safety warnings are difficult to accurately apply to construction site equipment and the coverage is not comprehensive, resulting in inaccurate safety warning results. Summary of the invention

[0004] The purpose of the present invention is to provide a driver behavior analysis safety warning system and method based on deep learning to solve the problems raised in the above background technology.

[0005] In the first aspect, the driver behavior analysis safety warning method based on deep learning provided by the present application adopts the following technical solutions:

[0006] The driver's driving data is collected through monitoring videos and sensors, and the driving data is pre-processed to obtain raw data;

[0007] The original data is input into the deep learning algorithm to extract features, obtain physiological features, operational features and environmental features, and the extracted features are trained on physiological models, operational models and environmental models respectively;

[0008] The physiological model training analyzes the driver's physiological state and psychological state according to the physiological characteristics and obtains physiological values ​​and psychological values;

[0009] The operation model training obtains the driver's driving skill level, analyzes the driver's operation behavior risk level in combination with the operation characteristics, and obtains the operation risk value;

[0010] The environmental model training obtains the feedback of people around the driver and obtains the environmental risk value by combining the environmental characteristics analysis;

[0011] The data trained by the model will be used to make a single judgment to determine whether the driver's behavior is dangerous. If the driver's behavior is dangerous, an alarm will be issued in time;

[0012] If the driver's behavior is not dangerous, all the data trained by the model will be integrated and calculated to obtain a comprehensive danger value to determine whether the driver's behavior is dangerous. If the driver's behavior is dangerous, the dangerous situation will be extracted and an alarm will be issued.

[0013] Preferably, the physiological model training, the step of analyzing the driver's physiological state and psychological state according to the physiological characteristics and obtaining the physiological value and the psychological value, is specifically:

[0014] The physiological characteristics include eye characteristics, sweat characteristics, pressure characteristics and human body characteristics;

[0015] The driver's health status is obtained based on human characteristics and recorded as a health value, and a health value threshold is set. If the health value is less than the health value threshold, an alarm is directly issued;

[0016] If the health value is not less than the health value threshold, the driver's physiological state is analyzed according to the eye characteristics and the pressure characteristics, and the physiological value is obtained;

[0017] The driver's psychological state is analyzed according to the sweat characteristics and pressure characteristics, and the psychological value is obtained.

[0018] Preferably, if the health value is not less than the health value threshold, the step of analyzing the driver's physiological state according to the eye characteristics and the pressure characteristics and obtaining the physiological value is specifically as follows:

[0019] Extracting feature information of eye features, wherein the feature information includes eyeball information and periorbital information;

[0020] Extract the color and texture of the periorbital area based on the periorbital information, obtain the standard eye features of the driver, and obtain the degree of the driver's dark circles AH and the degree of the driver's eye bags AD by comparison;

[0021] The red blood streak area, red blood streak density and pupil area in the eyeball are extracted according to the eyeball information, and the red blood streak area increase value AS, red blood streak density increase value AM and pupil area decrease value AT are calculated according to the standard eye characteristics;

[0022] By fatigue value correlation function The fatigue value is calculated, where , is the scale factor and is greater than 0;

[0023] The driver's distraction value BF is obtained by analyzing the pressure characteristics and eye characteristics, and the physiological correlation function The physiological value CS is calculated, where , is the scaling factor and is greater than 0.

[0024] Preferably, the step of obtaining the driver's distraction value BF according to the pressure characteristics and eye characteristics analysis is specifically as follows:

[0025] The motion trajectory of the driver's pupil position is extracted based on the eye features, and the continuous residence time of the pupil at different positions is counted;

[0026] Set a time threshold to filter the number of positions whose continuous stay time is less than the time threshold. If the number of positions is not 0, extract the real-time pressure of the steering wheel based on the pressure feature.

[0027] Acquire historical driving data of the driver, and extract the average pressure of the steering wheel in the historical driving data;

[0028] Calculate the pressure difference between the average pressure and the real-time pressure, and find the corresponding distraction value BF according to the preset pressure difference and distraction value correlation curve;

[0029] If the number of positions is 0, the driver's visual range is counted to obtain the standard range that the driver needs to observe during operation;

[0030] Determine whether the sight range is within the standard range. If the sight range is within the standard range, the distraction value BF=0;

[0031] If the sight range is not within the standard range, the range difference between the sight range and the standard range is calculated, and the corresponding distraction value BF is found according to the preset range difference and distraction value correlation curve.

[0032] Preferably, the step of analyzing the driver's psychological state according to the sweat characteristics and the pressure characteristics and obtaining the psychological value is specifically:

[0033] Extract the content of sweat substances associated with stress in the sweat characteristics and record it as stress substance content DY;

[0034] Extract skin conductivity DD from sweat characteristics;

[0035] Extract the driver's facial expression changes according to the images of the surveillance video to obtain the facial change value DM;

[0036] According to the psychological association function The psychological value DX is calculated, where , , is the scaling factor and is greater than 0.

[0037] Preferably, the step of training the environmental model, obtaining the feedback of people around the driver, and obtaining the environmental risk value by combining environmental feature analysis is specifically as follows:

[0038] Obtaining the response of the crowd around the driving device, the response including the crowd volume, crowd movement trajectory and crowd voice;

[0039] Obtain the average volume of the crowd in front of the driver operating the construction equipment, and calculate the volume difference EY between the crowd volume and the average volume;

[0040] Obtain the average moving speed of the crowd at the construction site, and extract the trajectory distance EG where the crowd moving speed is greater than the average moving speed based on the crowd moving trajectory;

[0041] Set the error keywords of the speech of the surrounding crowd when the driver makes an operation error, use named entity recognition to record the speech containing the error keywords as error speech, and calculate the ratio of error speech to crowd speech EW;

[0042] According to the reflection correlation function Get the EA reflecting the risk value, where , , is the scale factor and is greater than 0;

[0043] According to the environmental characteristics, the driving risk value FA is obtained, and the environmental risk association function The environmental hazard value GA is calculated, where , is the scaling factor and is greater than 0.

[0044] Preferably, the step of obtaining the driving risk value FA according to the environmental characteristics analysis is specifically:

[0045] According to the environmental characteristics, the weather severity FT, driving terrain flatness FP and operation difficulty FN are extracted;

[0046] Obtaining the standard terrain flatness FB of the construction equipment operated by the driver;

[0047] Combined with the driver's driving skill level FS, according to the driving risk association function The driving risk value FA is calculated, where , , , is the scaling factor and is greater than 0.

[0048] Preferably, the step of making a single judgment on the data trained by the model to judge whether the driver's behavior is dangerous, and promptly giving an alarm if the driver's behavior is dangerous, specifically comprises:

[0049] Set a physiological value threshold. When the physiological value CS reaches the physiological value threshold, the driver's behavior is judged to be dangerous, a physiological alarm is issued, and physiological related data is transmitted to the terminal;

[0050] Set the psychological value threshold. When the psychological value DX reaches the psychological value threshold, the driver's behavior is judged to be dangerous, a psychological alarm is issued, and psychological related data is transmitted to the terminal;

[0051] Set an operation risk value threshold. When the operation risk value reaches the operation risk value threshold, the driver's behavior is judged to be dangerous, an operation alarm is issued, and operation-related data is transmitted to the terminal;

[0052] Set an environmental hazard value threshold. When the environmental hazard value GA reaches the environmental hazard value threshold, the driver's behavior is judged to be dangerous, an environmental alarm is issued, and environmental related data is transmitted to the terminal.

[0053] Preferably, if the driver's behavior is not dangerous, all data trained by the model are integrated and calculated to obtain a comprehensive danger value to determine whether the driver's behavior is dangerous; if the driver's behavior is dangerous, the dangerous situation is extracted to give a prompt alarm, specifically:

[0054] If the driver's behavior is judged not to be dangerous, then according to the comprehensive correlation function The comprehensive risk value ZA is calculated, where , , , is the scale factor and is greater than 0;

[0055] Set a comprehensive risk value threshold. If the comprehensive risk value ZA reaches the comprehensive risk value threshold, the driver's behavior is judged to be dangerous.

[0056] Calculate the differences between the physiological value, psychological value, operational risk value, environmental risk value and the corresponding thresholds respectively, compare and obtain the value with the smallest difference and record it as the alarm value;

[0057] Find the alarm type corresponding to the alarm value, issue an alarm, and transmit the associated data corresponding to the alarm type to the terminal.

[0058] In the second aspect, the driver behavior analysis safety warning system based on deep learning provided by this application adopts the following technical solutions:

[0059] The data collection module collects the driver's driving data through monitoring videos and sensors, and pre-processes the driving data to obtain raw data;

[0060] The feature extraction module inputs the original data into the deep learning algorithm to extract features, obtain physiological features, operational features and environmental features, and conducts physiological model training, operational model training and environmental model training on the extracted features respectively;

[0061] A physiological training module, wherein the physiological model training analyzes the physiological state and psychological state of the driver according to the physiological characteristics and obtains physiological values ​​and psychological values;

[0062] The operation training module, the operation model training, obtains the driver's driving skill level, analyzes the driver's operation behavior risk level in combination with the operation characteristics, and obtains the operation risk value;

[0063] Environmental training module: the environmental model training obtains the feedback of people around the driver and obtains the environmental risk value by combining environmental characteristics analysis;

[0064] The single judgment module uses the data trained by the model to make a single judgment to determine whether the driver's behavior is dangerous. If the driver's behavior is dangerous, an alarm will be issued in time;

[0065] The comprehensive judgment module integrates and calculates all the data trained by the model to obtain a comprehensive danger value to determine whether the driver's behavior is dangerous. If the driver's behavior is dangerous, the dangerous situation is extracted and an alarm is issued.

[0066] In summary, the present application includes at least one of the following beneficial technical effects:

[0067] 1. Collect data through monitoring videos and sensors, extract data features, and conduct physiological model training, operational model training, and environmental model training respectively to obtain physiological values, psychological values, operational risk values, and environmental risk values, so as to determine whether the driver has dangerous behavior and whether a safety warning is needed. Analyzing driver behavior from multiple angles is conducive to obtaining more accurate results and improving the accuracy of driver behavior analysis safety warnings based on deep learning.

[0068] 2. The driver's stress can be reflected through the content of sweat components, and the skin conductivity can be obtained through the sweat components, thereby reflecting the driver's emotional excitement and comprehensively obtaining the driver's psychological condition. The psychological condition can be analyzed based on sweat to judge the driver's driving risk. Sweat collection reduces interference to the driver and improves the convenience of safety warning based on deep learning driver behavior analysis.

[0069] 3. According to the volume, movement trajectory and voice output content of the external crowd operated by the driver, it reflects whether the driver has made an operation error, reduces the driving risk situations missed due to the normal operation of the driver, can better meet the complexity of the construction site, and improves the comprehensiveness of the driver behavior analysis safety warning based on deep learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 It is a schematic diagram of the specific steps of an embodiment of the driver behavior analysis safety warning method based on deep learning of the present invention.

[0071] Figure 2 It is a module connection diagram of an embodiment of a driver behavior analysis safety warning system based on deep learning of the present invention. DETAILED DESCRIPTION

[0072] Below is a combination of the embodiments and Figure 1-Figure 2 The present invention is further described in detail, but the embodiments of the present invention are not limited thereto.

[0073] The present invention discloses a driver behavior analysis safety warning method based on deep learning, which specifically includes the following steps:

[0074] Step S1, collecting the driver's driving data through monitoring video and sensors, and pre-processing the driving data to obtain raw data.

[0075] The driver's facial expressions and other conditions can be monitored through surveillance video, and sensors can collect various data of the driver, including operation data, etc. For example, a wearable sweat sensor collects sweat samples of the driver.

[0076] Step S2, input the original data into the deep learning algorithm to perform feature extraction to obtain physiological features, operational features and environmental features, and perform physiological model training, operational model training and environmental model training on the extracted features respectively.

[0077] Deep learning is a branch of machine learning that uses deep neural networks to learn data representation and feature extraction algorithms. Deep learning learns high-level abstract features in the data by stacking multiple processing layers (usually including convolutional layers, pooling layers, fully connected layers, etc.) to perform multi-layer nonlinear transformations on the input data. These features can be used to solve complex machine learning tasks such as image recognition, speech recognition, natural language processing, recommendation systems, etc. The core of deep learning is that it can automatically learn effective feature representations from large amounts of data without the need for manual feature engineering. This gives deep learning a significant advantage in processing large-scale and complex data. With the continuous improvement of computing power and the continuous improvement of algorithms, the application prospects of deep learning are becoming more and more broad.

[0078] Step S3, physiological model training, analyzing the driver's physiological state and psychological state according to the physiological characteristics and obtaining physiological values ​​and psychological values.

[0079] Step S4, operation model training, obtains the driver's driving skill level, analyzes the driver's operation behavior risk level in combination with the operation characteristics, and obtains the operation risk value.

[0080] The standard operation is used as a reference and compared with the actual operation. At the same time, the driver's operating behavior is judged in combination with the driving skill level. For example, the driver should brake at 45 seconds, but the driver brakes suddenly at 48 seconds, which is considered an operating error. The occurrence of sudden braking does not necessarily mean that the driver has made an operating error. It may be an emergency measure made due to special circumstances. Therefore, the higher the driver's driving skill level, the lower the probability of operating errors reflected by sudden braking. The operating risk value is obtained based on the driver's actual operation and the driver's driving skill level.

[0081] Step S5, environmental model training, obtaining feedback from people around the driver, and analyzing the environmental characteristics to obtain the environmental risk value.

[0082] Step S6, making a single judgment on the data trained by the model to determine whether the driver's behavior is dangerous, and if the driver's behavior is dangerous, an alarm is issued in time.

[0083] Step S7: If the driver's behavior is not dangerous, all the data trained by the model are integrated and calculated to obtain a comprehensive danger value to determine whether the driver's behavior is dangerous. If the driver's behavior is dangerous, the dangerous situation is extracted to issue an alarm.

[0084] In actual use, the behavior of construction equipment drivers will seriously affect the safety of the construction site. Analyzing the driver's behavior can issue safety warnings in time and reduce the accident rate. The driver's physiological condition, psychological condition, operating conditions and driving environment will affect the driver's actual driving behavior, thereby changing the safety risk of driving. Comprehensive model training for a variety of situations can obtain more accurate analysis results and improve the safety of the construction site. At the same time, the driver's driving skill level also has a significant impact on driving safety. For example, under normal circumstances, the driver should operate the excavator pole to descend to the ground, and the descent height is 2 meters. However, if the driver operates the excavator to descend 2.5 meters, the operation is not standard and the operation risk value is high.

[0085] Physiological model training, the steps of analyzing the driver's physiological state and psychological state according to physiological characteristics and obtaining physiological values ​​and psychological values ​​are as follows:

[0086] Step S31, the physiological characteristics include eye characteristics, sweat characteristics, pressure characteristics and body characteristics.

[0087] Step S32, the health status of the driver is obtained according to the human body characteristics and recorded as a health value, and a health value threshold is set. If the health value is less than the health value threshold, an alarm is directly issued.

[0088] Step S33: If the health value is not less than the health value threshold, the driver's physiological state is analyzed according to the eye characteristics and pressure characteristics, and the physiological value is obtained.

[0089] Step S34, analyzing the driver's psychological state according to the sweat characteristics and pressure characteristics, and obtaining a psychological value.

[0090] In actual use, whether the driver is physically or mentally uncomfortable, it will increase the risk of driving, thus increasing the accident rate. Therefore, analyzing the driver's physiological and psychological conditions based on physiological characteristics is conducive to more accurate safety warnings. Based on human characteristics, including heart rate, breathing rate and other characteristics, it can be judged whether the driver is in a healthy state. If the driver is sick, the driver cannot operate the construction equipment accurately, and the alarm should be called in time to stop the driver's operation. When the driver is in a healthy state, the driver's driving will be affected by physiological conditions such as fatigue, and psychological conditions such as stress and emotional fluctuations will also affect driving. Therefore, it is necessary to pay attention to the driver's physiological and psychological state when in a healthy state.

[0091] If the health value is not less than the health value threshold, the physiological state of the driver is analyzed according to the eye characteristics and the pressure characteristics, and the steps of obtaining the physiological value are specifically as follows:

[0092] Step S331, extracting feature information of eye features, the feature information including eyeball information and periorbital information.

[0093] Step S332, extracting the color and texture of the periorbital area according to the periorbital information, obtaining the standard eye features of the driver, and obtaining the driver's dark circle degree AH and the driver's eye bag degree AD by comparison.

[0094] Step S333, extracting the red blood streak area, red blood streak density and pupil area in the eyeball according to the eyeball information, and calculating the red blood streak area increase value AS, red blood streak density increase value AM and pupil area decrease value AT according to standard eye features.

[0095] Step S334, through the fatigue value correlation function The fatigue value is calculated, where , is the scaling factor and is greater than 0.

[0096] Step S335: Analyze the driver's distraction value BF based on the pressure characteristics and eye characteristics, and calculate the driver's distraction value BF based on the physiological correlation function. The physiological value CS is calculated, where , is the scaling factor and is greater than 0.

[0097] In actual use, fatigue not only causes pupil constriction, but also causes the driver's eyeballs to become congested, that is, red bloodshot appears in the eyeballs. The area and density of the red bloodshot can reflect the driver's fatigue level. At the same time, when the driver is fatigued and sleep deprived for a long time, it is easy to cause bags under the eyes and dark circles. The driver's fatigue level can be reflected through the eyeballs and the area around the eyes. At the same time, even if the driver is not tired, he may cause an accident due to distraction. Therefore, the driver's physiological value is obtained based on the fatigue value and distraction value. The higher the physiological value, the greater the risk of the driver driving construction equipment.

[0098] The steps of obtaining the driver's distraction value BF according to the pressure characteristics and eye characteristics are as follows:

[0099] Step S3351, extracting the motion trajectory of the driver's pupil position according to the eye features, and counting the continuous stay time of the pupil at different positions.

[0100] Step S3352, setting a time threshold, screening the number of positions whose continuous stay time is less than the time threshold, if the number of positions is not 0, extracting the real-time pressure of the steering wheel according to the pressure feature.

[0101] Step S3353, obtaining the driver's historical driving data, and extracting the average pressure of the steering wheel in the historical driving data.

[0102] Step S3354, calculate the pressure difference between the average pressure and the real-time pressure, and find out the corresponding distraction value BF according to the preset pressure difference and distraction value correlation curve.

[0103] Step S3355, if the number of positions is 0, the driver's field of vision is counted to obtain the standard range that the driver needs to observe for operation.

[0104] Step S3356, determine whether the sight range is within the standard range. If the sight range is within the standard range, determine that the distraction value BF=0.

[0105] Step S3357, if the sight range is not within the standard range, the range difference between the sight range and the standard range is calculated, and the corresponding distraction value BF is found according to the preset range difference and distraction value correlation curve.

[0106] In actual use, when drivers are distracted or inattentive, their pupils will rotate or drift slightly, which is a normal physiological phenomenon. Pupil drift or rotation is unconscious, so the duration is usually short. When the driver is conscious, he or she will usually stay for a slightly longer time to obtain information. If the pupil drifts and other phenomena are found based on time screening, the driver's distraction and the degree of distraction can be analyzed based on the pressure distribution of the steering wheel. When the driver is distracted, he or she cannot accurately perceive and respond to changes in the vehicle and road, which causes the driver to react slowly or over-adjust when the steering wheel needs to be adjusted, thereby increasing the steering wheel turning angle and strength. If the pupil is normal, it does not mean that the driver is not distracted. The driver's distraction can be determined based on the driver's field of vision. For example, the driver's pupil is normal, but according to the standard operating procedure, the driver needs to observe the situation on the left and right sides of the construction equipment before operating, but the driver's field of vision only involves the front, so the driver is also judged to be distracted. If the driver's pupil is abnormal, the degree of distraction is obtained based on the steering wheel pressure.

[0107] Step S341, analyzing the driver's psychological state according to the sweat characteristics and pressure characteristics, and obtaining the psychological value, is specifically:

[0108] Step S342: extract the sweat substance content associated with pressure in the sweat characteristics and record it as the pressure substance content DY.

[0109] Step S343, extracting skin conductivity DD from the sweat characteristics.

[0110] Step S344, extracting the driver's facial expression changes based on the image of the monitoring video to obtain a facial change value DM.

[0111] Step S345, according to the psychological association function The psychological value DX is calculated, where , , is the scaling factor and is greater than 0.

[0112] In actual use, cortisol is a substance called stress hormone. When the human body is under stress, the concentration of cortisol in the blood will increase. Cortisol with the same concentration in the blood can also be detected in sweat, thus reflecting the stress level of the human body. In addition, there are some other biochemical markers in sweat that are related to stress levels and can reflect the stress of the driver. Skin conductivity is a commonly used psychophysiological indicator used to reflect the activity level of the sympathetic nervous system. When the driver is emotionally excited, the sympathetic nerve activity is enhanced, the sweat gland secretion increases, and the skin conductivity increases. Therefore, the change in skin conductivity can be used as an effective indicator to judge the change of the driver's emotional excitement. The salt concentration in sweat is the main reason for changing the conductivity of the skin, so the skin conductivity can be extracted through sweat. When the driver is under great pressure and has great emotional fluctuations, it will lead to the driver's operation accuracy, thereby changing the driving risk. Therefore, through the analysis of the driver's psychological state, it is conducive to better achieve safety warning and reduce the accident rate.

[0113] The steps of environmental model training, obtaining feedback from people around the driver, and analyzing environmental characteristics to obtain environmental hazard values ​​are as follows:

[0114] Step S51, obtaining the response of the crowd around the driving device, including the crowd volume, crowd movement trajectory and crowd voice.

[0115] Step S52, obtaining the average volume of the crowd before the driver operates the construction equipment, and calculating the volume difference EY between the crowd volume and the average volume.

[0116] Step S53, obtaining the average moving speed of the crowd at the construction site, and extracting the trajectory distance EG where the crowd moving speed is greater than the average moving speed according to the crowd moving trajectory.

[0117] Step S54, setting error keywords of the speech of the surrounding crowd when the driver makes an operation error, using named entity recognition to record the speech containing the error keywords in the crowd speech as error speech, and counting the ratio EW of the error speech to the crowd speech.

[0118] Step S55, according to the reflection correlation function Get the EA reflecting the risk value, where , , is the scaling factor and is greater than 0.

[0119] Step S56: Analyze the environmental characteristics to obtain the driving risk value FA, and then calculate the driving risk value FA according to the environmental risk correlation function. The environmental hazard value GA is calculated, where , is the scaling factor and is greater than 0.

[0120] In actual use, even if the driver is tired or emotional, safety accidents will not occur due to different driving skills. Including the driver's sudden braking and sharp turns, it does not mean that the driver made an operational error. Similarly, even if the driver's physiology and psychology are within the normal range, it does not mean that the driver has no mistakes. When the construction equipment driver drives, he is usually at the construction site, and a large number of other workers will gather at the construction site. The driver's driving situation can be reflected through the feedback of other workers. For example, the driver operates an excavator. In theory, the excavator rod can rise 2 meters after digging. The driver's physiological and psychological values ​​are within the normal range, and the excavator rod is also operated to rise two meters. However, there are a lot of wood piled up at the actual construction site, and it is necessary to cross the wood, so it needs to rise three meters. Therefore, the driver made an operational error, and in the case of an operational error, the surrounding people will stop and discuss, and the volume will increase. At the same time, the crowd will disperse to avoid the falling wood, or gather because of sorting wood, so the movement trajectory of the crowd will also increase. Based on the voice output of the crowd, it is possible to determine whether the driver has made a mistake and judge the driver's driving condition from the external environment, reducing the situation where misjudgment is caused by normal indicators and improving the accuracy of driving behavior analysis and warning.

[0121] The steps of obtaining the driving risk value FA according to the environmental characteristics analysis are as follows:

[0122] Step S561, extracting weather severity FT, driving terrain flatness FP and operation difficulty FN according to environmental features.

[0123] The difficulty of operation is determined by the terrain and material stacking conditions of the construction site. The more terrain obstacles there are and the more materials are stacked, the greater the difficulty of operation.

[0124] Step S562, obtaining the standard terrain flatness FB of the construction equipment operated by the driver.

[0125] Step S563: Combine the driver's driving skill level FS and calculate the driving risk correlation function The driving risk value FA is calculated, where , , , is the scaling factor and is greater than 0.

[0126] In actual use, in addition to the driver's driving operation and status affecting driving safety, the driving environment also affects driving safety. Especially at construction sites, uneven roads and too many materials will increase the difficulty of operation. Bad weather will affect the driver's operation and increase the risk of driving. For example, rainy weather will affect the driver's vision, and the flatness of the terrain will increase the bumpiness of the construction equipment. The greater the difficulty of operation and the worse the driver's technical level, the greater the driving risk.

[0127] The steps of making a single judgment on the data trained by the model to determine whether the driver's behavior is dangerous and promptly giving an alarm if the driver's behavior is dangerous are as follows:

[0128] Step S61, setting a physiological value threshold. When the physiological value CS reaches the physiological value threshold, the driver's behavior is judged to be dangerous, a physiological alarm is issued, and physiological related data is transmitted to the terminal.

[0129] Step S62, setting a psychological value threshold. When the psychological value DX reaches the psychological value threshold, the driver's behavior is judged to be dangerous, a psychological alarm is issued, and psychological related data is transmitted to the terminal.

[0130] Step S63, setting an operation risk value threshold. When the operation risk value reaches the operation risk value threshold, the driver's behavior is judged to be dangerous, an operation alarm is issued, and operation-related data is transmitted to the terminal.

[0131] Step S64, setting an environmental hazard value threshold. When the environmental hazard value GA reaches the environmental hazard value threshold, the driver's behavior is judged to be dangerous, an environmental alarm is issued, and environmental related data is transmitted to the terminal.

[0132] In actual use, when any indicator of the driver exceeds the threshold, no matter how good the other indicators are, it still poses a huge driving risk, so timely warning is needed. For example, if the driver's current physiological value exceeds the threshold, it means that the driver is no longer able to operate the driving accurately, and an alarm is needed in time, and adjustments are made according to the alarm situation. If it is a physiological alarm, the driver should be replaced in time. If it is an environmental alarm, the operation of the construction equipment should be suspended, or adjustments should be made to the construction site.

[0133] If the driver's behavior is not dangerous, all the data trained by the model will be integrated and calculated to obtain a comprehensive danger value to determine whether the driver's behavior is dangerous. If the driver's behavior is dangerous, the steps of extracting dangerous situations and giving warnings are as follows:

[0134] Step S71: If the driver's behavior is not dangerous, The comprehensive risk value ZA is calculated, where , , , is the scaling factor and is greater than 0.

[0135] Step S72, setting a comprehensive risk value threshold, if the comprehensive risk value ZA reaches the comprehensive risk value threshold, the driver's behavior is judged to be dangerous.

[0136] Step S73, respectively calculating the differences between the physiological value, psychological value, operational risk value, environmental risk value and the corresponding thresholds, comparing and obtaining the value with the smallest difference and recording it as the alarm value.

[0137] Step S74, searching for the alarm type corresponding to the alarm value to generate an alarm, and transmitting the associated data corresponding to the alarm type to the terminal.

[0138] In actual use, if the driver's individual indicators do not exceed the threshold, it does not mean that driving is very safe. When individual indicators do not exceed the threshold, all indicators need to be integrated for judgment. For example, if the driver is tired, the driver's skills are not good, and the driving environment is relatively bad, the risk of the driver having an accident is still very high. When the comprehensive risk value reaches the threshold, the most serious indicator needs to be selected and uploaded to the terminal when the alarm is sounded. Suppose it is drizzling, the driver is a little tired, and the operation is normal. At this time, the difference between the physiological value and the threshold is the smallest, indicating that the main factor affecting the driving risk is the physiological condition. Then, the physiological condition is selected and uploaded to the terminal, and a solution for changing the driver can be obtained.

[0139] The driver behavior analysis safety warning system based on deep learning, by applying the driver behavior analysis safety warning method based on deep learning as described above, includes:

[0140] The data collection module collects the driver's driving data through monitoring videos and sensors, and pre-processes the driving data to obtain raw data.

[0141] The feature extraction module inputs the original data into the deep learning algorithm to extract features, obtain physiological features, operational features and environmental features, and performs physiological model training, operational model training and environmental model training on the extracted features respectively.

[0142] Physiological training module, physiological model training, analyzes the driver's physiological and psychological states according to physiological characteristics and obtains physiological and psychological values.

[0143] The operation training module, operation model training, obtains the driver's driving skill level, analyzes the risk level of the driver's operation behavior in combination with the operation characteristics, and obtains the operation risk value.

[0144] Environmental training module, environmental model training, obtains feedback from people around the driver, and obtains environmental hazard value based on environmental characteristics analysis.

[0145] The single judgment module will make a single judgment based on the data trained by the model to determine whether the driver's behavior is dangerous. If the driver's behavior is dangerous, an alarm will be issued in time.

[0146] The comprehensive judgment module integrates and calculates all the data trained by the model to obtain a comprehensive danger value to determine whether the driver's behavior is dangerous. If the driver's behavior is dangerous, the dangerous situation is extracted and an alarm is issued.

[0147] In actual application, driver behavior analysis and safety warning are realized through different modules, which reduces the subjectivity of manual analysis and improves the accuracy of safety warning. At the same time, it reduces the waste of human resources and improves the convenience of safety warning.

[0148] The implementation principle of this system is as follows: the data acquisition module collects the driver's driving data through monitoring videos and sensors, and pre-processes the driving data to obtain the original data. It includes operations such as image denoising, transformation, and normalization to improve the quality and consistency of the data and facilitate subsequent processing and analysis. The feature extraction module inputs the original data into the deep learning algorithm, extracts features, obtains physiological features, operational features, and environmental features, and performs physiological model training, operational model training, and environmental model training on the extracted features. The physiological training module analyzes the driver's fatigue and distraction based on physiological features and obtains physiological values, and analyzes the driver's stress and emotional fluctuations through the driver's sweat and obtains psychological values. The operation training module obtains the driver's driving skill level, analyzes the degree of danger of the driver's operation behavior in combination with the operation characteristics, and obtains the operation danger value. The environmental training module obtains the feedback of the people around the driver, and obtains the environmental danger value based on the volume, movement trajectory, and output voice content of the surrounding people and the analysis of environmental features. The single judgment module makes a single judgment on the data trained by the model to determine whether the driver's behavior is dangerous. If the driver's behavior is dangerous, an alarm is issued in time. The comprehensive judgment module integrates and calculates all the data trained by the model to obtain a comprehensive danger value to determine whether the driver's behavior is dangerous. If the driver's behavior is dangerous, the dangerous situation is extracted and an alarm is issued.

[0149] The above are all preferred embodiments of the present application, and the protection scope of the present application is not limited thereto. Therefore, any equivalent changes made according to the structure, shape, and principle of the present application should be included in the protection scope of the present application.

Claims

1. A driver behavior analysis safety warning method based on deep learning, characterized in that: The following steps are involved: The driver's driving data is collected through monitoring videos and sensors, and the driving data is pre-processed to obtain raw data; The original data is input into the deep learning algorithm to extract features, obtain physiological features, operational features and environmental features, and the extracted features are trained on physiological models, operational models and environmental models respectively; The physiological model training analyzes the driver's physiological state and psychological state according to the physiological characteristics and obtains physiological values ​​and psychological values; The operation model training obtains the driver's driving skill level, analyzes the driver's operation behavior risk level in combination with the operation characteristics, and obtains the operation risk value; The environmental model training obtains the feedback of people around the driver and obtains the environmental risk value by combining the environmental characteristics analysis; The data trained by the model will be used to make a single judgment to determine whether the driver's behavior is dangerous. If the driver's behavior is dangerous, an alarm will be issued in time; If the driver's behavior is not dangerous, all the data trained by the model will be integrated and calculated to obtain a comprehensive danger value to determine whether the driver's behavior is dangerous. If the driver's behavior is dangerous, the dangerous situation will be extracted and an alarm will be issued.

2. The driver behavior analysis safety warning method based on deep learning according to claim 1 is characterized in that: The physiological model training, the step of analyzing the physiological state and psychological state of the driver according to the physiological characteristics and obtaining the physiological value and the psychological value, is specifically as follows: The physiological characteristics include eye characteristics, sweat characteristics, pressure characteristics and human body characteristics; The driver's health status is obtained based on human characteristics and recorded as a health value, and a health value threshold is set. If the health value is less than the health value threshold, an alarm is directly issued; If the health value is not less than the health value threshold, the driver's physiological state is analyzed according to the eye characteristics and the pressure characteristics, and the physiological value is obtained; The driver's psychological state is analyzed according to the sweat characteristics and pressure characteristics, and the psychological value is obtained.

3. The driver behavior analysis safety warning method based on deep learning according to claim 2 is characterized in that: If the health value is not less than the health value threshold, the physiological state of the driver is analyzed according to the eye characteristics and the pressure characteristics, and the steps of obtaining the physiological value are specifically as follows: Extracting feature information of eye features, wherein the feature information includes eyeball information and periorbital information; Extract the color and texture of the periorbital area based on the periorbital information, obtain the standard eye features of the driver, and obtain the degree of the driver's dark circles AH and the degree of the driver's eye bags AD by comparison; The red blood streak area, red blood streak density and pupil area in the eyeball are extracted according to the eyeball information, and the red blood streak area increase value AS, red blood streak density increase value AM and pupil area decrease value AT are calculated according to the standard eye characteristics; By fatigue value correlation function The fatigue value is calculated, where , is the scale factor and is greater than 0; The driver's distraction value BF is obtained by analyzing the pressure characteristics and eye characteristics, and the physiological correlation function The physiological value CS is calculated, where , is the scaling factor and is greater than 0.

4. The driver behavior analysis safety warning method based on deep learning according to claim 3 is characterized in that: The step of obtaining the driver's distraction value BF according to the pressure characteristics and eye characteristics analysis is specifically as follows: The motion trajectory of the driver's pupil position is extracted based on the eye features, and the continuous residence time of the pupil at different positions is counted; Set a time threshold to filter the number of positions whose continuous stay time is less than the time threshold. If the number of positions is not 0, extract the real-time pressure of the steering wheel based on the pressure feature. Acquire historical driving data of the driver, and extract the average pressure of the steering wheel in the historical driving data; Calculate the pressure difference between the average pressure and the real-time pressure, and find the corresponding distraction value BF according to the preset pressure difference and distraction value correlation curve; If the number of positions is 0, the driver's visual range is counted to obtain the standard range that the driver needs to observe during operation; Determine whether the sight range is within the standard range. If the sight range is within the standard range, the distraction value BF=0; If the sight range is not within the standard range, the range difference between the sight range and the standard range is calculated, and the corresponding distraction value BF is found according to the preset range difference and distraction value correlation curve.

5. The driver behavior analysis safety warning method based on deep learning according to claim 4 is characterized in that: The step of analyzing the driver's psychological state according to the sweat characteristics and the pressure characteristics and obtaining the psychological value is specifically as follows: Extract the content of sweat substances associated with stress in the sweat characteristics and record it as stress substance content DY; Extract skin conductivity DD from sweat characteristics; Extract the driver's facial expression changes according to the images of the surveillance video to obtain the facial change value DM; According to the psychological association function The psychological value DX is calculated, where , , is the scaling factor and is greater than 0.

6. The driver behavior analysis safety warning method based on deep learning according to claim 5 is characterized in that: The steps of training the environment model, obtaining the feedback of people around the driver, and obtaining the environmental risk value by combining environmental feature analysis are specifically as follows: Obtaining the response of the crowd around the driving device, the response including the crowd volume, crowd movement trajectory and crowd voice; Obtain the average volume of the crowd in front of the driver operating the construction equipment, and calculate the volume difference EY between the crowd volume and the average volume; Obtain the average moving speed of the crowd at the construction site, and extract the trajectory distance EG where the crowd moving speed is greater than the average moving speed based on the crowd moving trajectory; Set the error keywords of the speech of the surrounding crowd when the driver makes an operation error, use named entity recognition to record the speech containing the error keywords as error speech, and calculate the ratio of error speech to crowd speech EW; According to the reflection correlation function Get the EA reflecting the risk value, where , , is the scale factor and is greater than 0; According to the environmental characteristics, the driving risk value FA is obtained, and the environmental risk association function The environmental hazard value GA is calculated, where , is the scaling factor and is greater than 0.

7. The driver behavior analysis safety warning method based on deep learning according to claim 6 is characterized in that: The step of obtaining the driving risk value FA according to the environmental characteristics analysis is specifically as follows: According to the environmental characteristics, the weather severity FT, driving terrain flatness FP and operation difficulty FN are extracted; Obtaining the standard terrain flatness FB of the construction equipment operated by the driver; Combined with the driver's driving skill level FS, according to the driving risk association function The driving risk value FA is calculated, where , , , is the scaling factor and is greater than 0.

8. The driver behavior analysis safety warning method based on deep learning according to claim 7 is characterized in that: The steps of making a single judgment on the data trained by the model to judge whether the driver's behavior is dangerous and promptly giving an alarm if the driver's behavior is dangerous are specifically as follows: Set a physiological value threshold. When the physiological value CS reaches the physiological value threshold, the driver's behavior is judged to be dangerous, a physiological alarm is issued, and physiological related data is transmitted to the terminal; Set the psychological value threshold. When the psychological value DX reaches the psychological value threshold, the driver's behavior is judged to be dangerous, a psychological alarm is issued, and psychological related data is transmitted to the terminal; Set an operation risk value threshold. When the operation risk value reaches the operation risk value threshold, the driver's behavior is judged to be dangerous, an operation alarm is issued, and operation-related data is transmitted to the terminal; Set an environmental hazard value threshold. When the environmental hazard value GA reaches the environmental hazard value threshold, the driver's behavior is judged to be dangerous, an environmental alarm is issued, and environmental related data is transmitted to the terminal.

9. The driver behavior analysis safety warning method based on deep learning according to claim 8 is characterized in that: If the driver's behavior is not dangerous, all the data trained by the model are integrated and calculated to obtain a comprehensive danger value to determine whether the driver's behavior is dangerous. If the driver's behavior is dangerous, the dangerous situation is extracted to give a prompt alarm, which is specifically as follows: If the driver's behavior is judged not to be dangerous, then according to the comprehensive correlation function The comprehensive risk value ZA is calculated, where , , , is the scale factor and is greater than 0; Set a comprehensive risk value threshold. If the comprehensive risk value ZA reaches the comprehensive risk value threshold, the driver's behavior is judged to be dangerous. Calculate the differences between the physiological value, psychological value, operational risk value, environmental risk value and the corresponding thresholds respectively, compare and obtain the value with the smallest difference and record it as the alarm value; Find the alarm type corresponding to the alarm value, issue an alarm, and transmit the associated data corresponding to the alarm type to the terminal.

10. A driver behavior analysis safety warning system based on deep learning, characterized in that: By applying the driver behavior analysis safety warning method based on deep learning as described in any one of claims 1 to 9, the method comprises: The data collection module collects the driver's driving data through monitoring videos and sensors, and pre-processes the driving data to obtain raw data; The feature extraction module inputs the original data into the deep learning algorithm to extract features, obtain physiological features, operational features and environmental features, and conducts physiological model training, operational model training and environmental model training on the extracted features respectively; A physiological training module, wherein the physiological model training analyzes the physiological state and psychological state of the driver according to the physiological characteristics and obtains physiological values ​​and psychological values; The operation training module, the operation model training, obtains the driver's driving skill level, analyzes the driver's operation behavior risk level in combination with the operation characteristics, and obtains the operation risk value; Environmental training module: the environmental model training obtains the feedback of people around the driver and obtains the environmental risk value by combining environmental characteristics analysis; The single judgment module uses the data trained by the model to make a single judgment to determine whether the driver's behavior is dangerous. If the driver's behavior is dangerous, an alarm will be issued in time; The comprehensive judgment module integrates and calculates all the data trained by the model to obtain a comprehensive danger value to determine whether the driver's behavior is dangerous. If the driver's behavior is dangerous, the dangerous situation is extracted and an alarm is issued.

Images