Driver behavior analysis safety warning system and method based on deep learning

Abstract

This application discloses a deep learning-based driver behavior analysis safety warning system and method, relating to the technical field of driver behavior analysis. The method includes: collecting the driver's driving data and preprocessing it to obtain raw data; using a deep learning algorithm to extract features and then performing model training; analyzing the driver's physiological and psychological states to obtain physiological and psychological values; obtaining the driver's driving skill level and combining it with operational characteristics to obtain the driver's operational risk value; obtaining feedback from people around the driver and combining it with environmental characteristics to obtain an environmental risk value; making a single judgment based on the data trained by the model, and issuing an alarm if the driver's behavior is dangerous; if the driver's behavior is not dangerous, then integrating all the data trained by the model to make a judgment, and issuing an alarm if the driver's behavior is dangerous. This application improves the accuracy of deep learning-based driver behavior analysis safety warnings.

Description

Technical Field

[0001] The present application relates to the technical field of driver behavior analysis, and in particular to a driver behavior analysis safety warning system and method based on deep learning. Background Art

[0002] With the rapid development of the transportation industry and the dramatic increase in vehicle ownership, road traffic safety issues are becoming increasingly prominent. As a core component of the transportation system, drivers' driving behavior has a crucial impact on road traffic safety. In recent years, traffic accidents caused by improper driver behavior, such as fatigue, distracted driving, and illegal operations, have occurred frequently, resulting in significant casualties and property losses. Because construction sites lack traffic lights, regulated road conditions, and other equipment regulations, the operating behavior of construction equipment drivers poses a significant threat to the safety of construction workers. Analyzing the behavior of construction equipment drivers and providing timely safety warnings in times of danger can effectively improve construction site safety.

[0003] Currently, driving safety warnings are implemented by identifying whether the driver's operating behavior complies with traffic regulations. However, since there are no traffic equipment on construction sites and the driving rules of construction equipment are different, existing driving behavior safety warnings are difficult to accurately apply to construction site equipment, and the coverage is incomplete, resulting in inaccurate safety warning results. Summary of the Invention

[0004] The purpose of the present invention is to provide a driver behavior analysis safety warning system and method based on deep learning to solve the problems raised in the above background technology.

[0005] In the first aspect, the driver behavior analysis safety warning method provided by this application based on deep learning adopts the following technical solutions:

[0006] The driver's driving data is collected through monitoring videos and sensors, and the driving data is pre-processed to obtain raw data;

[0007] The raw data is input into the deep learning algorithm for feature extraction to obtain physiological features, operational features, and environmental features. The extracted features are then used for physiological model training, operational model training, and environmental model training respectively.

[0008] The physiological model training analyzes the driver's physiological state and psychological state according to physiological characteristics and obtains physiological values and psychological values;

[0009] The operation model training obtains the driver's driving skill level, analyzes the driver's operation behavior risk level in combination with the operation characteristics, and obtains the operation risk value;

[0010] The environmental model training obtains the feedback of people around the driver and obtains the environmental risk value by combining the environmental characteristics analysis;

[0011] The data trained by the model will be used to make a single judgment to determine whether the driver's behavior is dangerous. If the driver's behavior is dangerous, an alarm will be issued in time;

[0012] If the driver's behavior is not dangerous, all the data trained by the model will be integrated and calculated to obtain a comprehensive risk value to determine whether the driver's behavior is dangerous. If the driver's behavior is dangerous, the dangerous situation will be extracted and an alarm will be issued.

[0013] Preferably, the physiological model training, the step of analyzing the driver's physiological state and psychological state according to physiological characteristics and obtaining physiological values and psychological values, is specifically:

[0014] The physiological characteristics include eye characteristics, sweat characteristics, pressure characteristics and human body characteristics;

[0015] The driver's health status is obtained based on human body characteristics and recorded as a health value. A health value threshold is set. If the health value is less than the health value threshold, an alarm is directly triggered.

[0016] If the health value is not less than the health value threshold, the driver's physiological state is analyzed based on the eye characteristics and pressure characteristics, and a physiological value is obtained;

[0017] The driver's psychological state is analyzed based on sweat characteristics and pressure characteristics, and a psychological value is obtained.

[0018] Preferably, if the health value is not less than the health value threshold, the step of analyzing the driver's physiological state according to the eye characteristics and the pressure characteristics and obtaining the physiological value is specifically as follows:

[0019] Extracting feature information of eye features, wherein the feature information includes eyeball information and periorbital information;

[0020] Extract the color and texture of the eye area based on the peri-eye information to obtain the driver's standard eye features. By comparing them, the driver's dark circle degree AH and the driver's eye bag degree AD are obtained.

[0021] The red blood streak area, red blood streak density and pupil area in the eyeball are extracted based on the eyeball information, and the red blood streak area increase value AS, red blood streak density increase value AM and pupil area decrease value AT are calculated based on the standard eye characteristics;

[0022] Through fatigue value correlation function The fatigue value is calculated, where 、 is the scale factor and is greater than 0;

[0023] The driver's distraction value BF is obtained by analyzing the pressure characteristics and eye characteristics, and the physiological correlation function The physiological value CS is calculated, where 、 is the scaling factor and is greater than 0.

[0024] Preferably, the step of obtaining the driver's distraction value BF according to the pressure characteristics and eye characteristics analysis is specifically as follows:

[0025] The motion trajectory of the driver's pupil position is extracted based on the eye features, and the continuous residence time of the pupil in different positions is counted;

[0026] Set a time threshold to filter the number of locations where the continuous dwell time is less than the time threshold. If the number of locations is not 0, extract the real-time steering wheel pressure based on the pressure feature.

[0027] Obtain the driver's historical driving data and extract the average steering wheel pressure from the historical driving data;

[0028] Calculate the pressure difference between the average pressure and the real-time pressure, and find the corresponding distraction value BF according to the preset pressure difference and distraction value correlation curve;

[0029] If the number of positions is 0, the driver's visual range is counted to obtain the standard range that the driver needs to observe during operation;

[0030] Determine whether the sight range is within the standard range. If the sight range is within the standard range, the distraction value BF=0;

[0031] If the sight range is not within the standard range, the range difference between the sight range and the standard range is calculated, and the corresponding distraction value BF is found according to the preset range difference and distraction value correlation curve.

[0032] Preferably, the step of analyzing the driver's psychological state according to the sweat characteristics and the pressure characteristics and obtaining the psychological value is specifically as follows:

[0033] Extract the sweat substance content associated with stress from the sweat characteristics and record it as stress substance content DY;

[0034] Extract skin conductivity DD from sweat characteristics;

[0035] Extract the driver's facial expression changes based on the surveillance video image and obtain the facial change value DM;

[0036] According to the psychological correlation function Calculate the psychological value DX, where 、 、 is the scaling factor and is greater than 0.

[0037] Preferably, the steps of training the environmental model, obtaining feedback from people around the driver, and obtaining the environmental risk value by combining environmental feature analysis are specifically as follows:

[0038] Obtaining the response of the crowd around the driving device, including the crowd volume, crowd movement trajectory, and crowd voice;

[0039] Obtain the average volume of the crowd in front of the driver operating the construction equipment, and calculate the volume difference EY between the crowd volume and the average volume;

[0040] Obtain the average moving speed of the crowd at the construction site, and extract the trajectory distance EG where the crowd's moving speed is greater than the average moving speed based on the crowd's movement trajectory;

[0041] Set error keywords for the speech of the surrounding crowd when the driver makes an operation error. Use named entity recognition to record the speech containing the error keyword as error speech. Calculate the ratio of error speech to crowd speech (EW).

[0042] According to the reflection correlation function Get the EA reflecting the risk value, where 、 、 is the scale factor and is greater than 0;

[0043] According to the environmental characteristics analysis, the driving risk value FA is obtained, and the environmental risk correlation function is used The environmental hazard value GA is calculated, where 、 is the scaling factor and is greater than 0.

[0044] Preferably, the step of obtaining the driving risk value FA based on environmental characteristics analysis is specifically as follows:

[0045] According to the environmental characteristics, the weather severity FT, driving terrain flatness FP and operation difficulty FN are obtained;

[0046] Obtaining the standard terrain flatness FB of the construction equipment operated by the driver;

[0047] Combined with the driver's driving skill level FS, according to the driving risk association function The driving risk value FA is calculated, where 、 、 、 is the scaling factor and is greater than 0.

[0048] Preferably, the step of making a single judgment based on the data trained by the model to determine whether the driver's behavior is dangerous and promptly issuing an alarm if the driver's behavior is dangerous is specifically as follows:

[0049] Set a physiological value threshold. When the physiological value CS reaches the physiological value threshold, the driver's behavior is judged to be dangerous, a physiological alarm is issued, and physiological related data is transmitted to the terminal;

[0050] Set a psychological value threshold. When the psychological value DX reaches the psychological value threshold, the driver's behavior is judged to be dangerous, a psychological alarm is issued, and psychological related data is transmitted to the terminal;

[0051] Set an operation risk value threshold. When the operation risk value reaches the operation risk value threshold, the driver's behavior is judged to be dangerous, an operation alarm is issued, and operation-related data is transmitted to the terminal;

[0052] Set an environmental hazard value threshold. When the environmental hazard value GA reaches the environmental hazard value threshold, the driver's behavior is judged to be dangerous, an environmental alarm is issued, and environmental related data is transmitted to the terminal.

[0053] Preferably, if the driver's behavior is not dangerous, all data trained by the model are integrated and calculated to obtain a comprehensive risk value to determine whether the driver's behavior is dangerous. If the driver's behavior is dangerous, the dangerous situation is extracted and an alarm is issued, specifically:

[0054] If the driver's behavior is judged to be not dangerous, then according to the comprehensive correlation function The comprehensive risk value ZA is calculated, where 、 、 、 is the scale factor and is greater than 0;

[0055] Set a comprehensive risk value threshold. If the comprehensive risk value ZA reaches the comprehensive risk value threshold, the driver's behavior is judged to be dangerous.

[0056] Calculate the difference between the physiological value, psychological value, operational risk value, environmental risk value and the corresponding threshold value respectively, compare and obtain the value with the smallest difference and record it as the alarm value;

[0057] Find the alarm type corresponding to the alarm value, issue an alarm, and transmit the associated data corresponding to the alarm type to the terminal.

[0058] Secondly, the driver behavior analysis safety warning system based on deep learning provided by this application adopts the following technical solutions:

[0059] The data collection module collects the driver's driving data through monitoring videos and sensors, and pre-processes the driving data to obtain raw data;

[0060] The feature extraction module inputs the raw data into the deep learning algorithm to extract features, obtain physiological features, operational features, and environmental features, and then performs physiological model training, operational model training, and environmental model training on the extracted features respectively;

[0061] Physiological training module, the physiological model training analyzes the driver's physiological state and psychological state according to physiological characteristics and obtains physiological values and psychological values;

[0062] The operation training module, which trains the operation model, obtains the driver's driving skill level, analyzes the risk level of the driver's operation behavior in combination with the operation characteristics, and obtains the operation risk value;

[0063] Environmental training module, the environmental model training, obtains the feedback of people around the driver, and combines the environmental characteristics analysis to obtain the environmental risk value;

[0064] The single judgment module uses the data trained by the model to make a single judgment to determine whether the driver's behavior is dangerous. If the driver's behavior is dangerous, an alarm will be issued in time;

[0065] The comprehensive judgment module integrates and calculates all the data trained by the model to obtain a comprehensive danger value to determine whether the driver's behavior is dangerous. If the driver's behavior is dangerous, the dangerous situation will be extracted and an alarm will be issued.

[0066] In summary, this application includes at least one of the following beneficial technical effects:

[0067] 1. By collecting data from surveillance videos and sensors, the system extracts data features and then trains physiological, operational, and environmental models. This system generates physiological, psychological, operational, and environmental risk values, thereby determining whether the driver is engaging in dangerous behavior and whether a safety warning is necessary. Analyzing driver behavior from multiple perspectives yields more accurate results and enhances the accuracy of deep learning-based driver behavior analysis and safety warnings.

[0068] 2. The driver's stress is reflected through the content of sweat components, and skin conductivity is derived from sweat components, which reflects the driver's emotional agitation and provides a comprehensive understanding of the driver's psychological state. By analyzing the driver's psychological state based on sweat, the driver's driving risk is determined. Sweat collection reduces driver interference and improves the convenience of deep learning-based driver behavior analysis and safety warnings.

[0069] 3. Based on the volume, movement trajectory and voice output content of the external crowd during the driver's operation, it reflects whether the driver has made an error in operation, reducing driving risks that are missed due to normal driver operation. It can better meet the complexity of the construction site and improve the comprehensiveness of the driver behavior analysis safety warning based on deep learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 It is a schematic diagram of the specific steps of an embodiment of the driver behavior analysis safety warning method based on deep learning of the present invention.

[0071] Figure 2 This is a module connection diagram of an embodiment of a driver behavior analysis safety warning system based on deep learning of the present invention. DETAILED DESCRIPTION

[0072] Below is a combination of the embodiments and Figure 1-Figure 2 The present invention will be described in further detail, but the embodiments of the present invention are not limited thereto.

[0073] The present invention discloses a driver behavior analysis safety warning method based on deep learning, which specifically includes the following steps:

[0074] Step S1: collecting the driver's driving data through monitoring video and sensors, and pre-processing the driving data to obtain raw data.

[0075] The driver's facial expressions and other conditions can be monitored through surveillance video, and sensors can collect various driver data, including operational data, etc. For example, a wearable sweat sensor can collect sweat samples from the driver.

[0076] In step S2, the original data is input into the deep learning algorithm to perform feature extraction to obtain physiological features, operational features, and environmental features. The extracted features are then trained on physiological models, operational models, and environmental models respectively.

[0077] Deep learning is a branch of machine learning that uses deep neural networks to learn data representations and extract features. Deep learning stacks multiple processing layers (typically including convolutional layers, pooling layers, and fully connected layers) to perform multiple layers of nonlinear transformations on input data, thereby learning high-level abstract features from the data. These features can be used to solve complex machine learning tasks such as image recognition, speech recognition, natural language processing, and recommender systems. The core of deep learning lies in its ability to automatically learn effective feature representations from large amounts of data without the need for manual feature engineering. This gives deep learning a significant advantage in processing large-scale and complex data. With the continuous improvement of computing power and algorithmic advancements, the application prospects of deep learning are becoming increasingly broad.

[0078] Step S3: physiological model training, analyzing the driver's physiological state and psychological state according to the physiological characteristics and obtaining physiological values and psychological values.

[0079] Step S4: Operation model training, obtaining the driver's driving skill level, analyzing the driver's operation behavior risk level in combination with the operation characteristics, and obtaining the operation risk value.

[0080] Standard operation is used as a reference for comparison with actual operation, and the driver's driving skill level is also considered to determine the risk level of the driver's operating behavior. For example, if the driver should brake at 45 seconds, but suddenly brakes at 48 seconds, it is considered an operational error. However, sudden braking does not necessarily mean that the driver has made an operational error; it may be an emergency measure taken due to a special situation. Therefore, the higher the driver's driving skill level, the lower the probability of an operational error reflected by sudden braking. The operational risk value is determined based on the driver's actual operation and driving skill level.

[0081] Step S5: Environmental model training, obtaining feedback from people around the driver, and analyzing the environmental characteristics to obtain an environmental risk value.

[0082] In step S6, a single judgment is made on the data trained by the model to determine whether the driver's behavior is dangerous. If the driver's behavior is dangerous, an alarm is issued in time.

[0083] In step S7, if the driver's behavior is not dangerous, all the data trained by the model are integrated and calculated to obtain a comprehensive risk value to determine whether the driver's behavior is dangerous. If the driver's behavior is dangerous, the dangerous situation is extracted and an alarm is issued.

[0084] In practice, the behavior of construction equipment drivers can seriously impact site safety. Analyzing driver behavior can provide timely safety warnings and reduce accident rates. The driver's physical and psychological condition, operating conditions, and driving environment all influence their actual driving behavior, thereby altering driving safety risks. Model training based on a comprehensive range of scenarios can yield more accurate analysis results, improving construction site safety. Furthermore, the driver's driving skill significantly impacts driving safety. For example, under normal circumstances, the driver should lower the excavator boom to the ground at a height of 2 meters. However, if the driver lowers the excavator 2.5 meters, this is considered non-standard operation and carries a high risk.

[0085] Physiological model training involves analyzing the driver's physiological and psychological states based on their physiological characteristics and obtaining physiological and psychological values. Specifically, the steps are as follows:

[0086] Step S31 : Physiological characteristics include eye characteristics, sweat characteristics, pressure characteristics and body characteristics.

[0087] Step S32: The driver's health status is obtained based on the human body characteristics and recorded as a health value. A health value threshold is set. If the health value is less than the health value threshold, an alarm is directly issued.

[0088] Step S33: If the health value is not less than the health value threshold, the driver's physiological state is analyzed according to the eye characteristics and pressure characteristics, and the physiological value is obtained.

[0089] Step S34: Analyze the driver's psychological state according to the sweat characteristics and pressure characteristics, and obtain a psychological value.

[0090] In practice, whether a driver experiences physical or psychological discomfort, it increases driving risk and, consequently, the accident rate. Therefore, analyzing the driver's physical and psychological condition based on physiological characteristics facilitates more accurate safety warnings. Human characteristics, including heart rate and respiratory rate, can be used to determine a driver's health. If a driver is ill, they will be unable to operate construction equipment accurately, and an alarm should be activated promptly to halt the driver's operation. However, even when a driver is healthy, their driving can be affected by physical conditions such as fatigue, as well as psychological conditions such as stress and mood swings. Therefore, attention should be paid to the driver's physical and psychological state during a healthy state.

[0091] If the health value is not less than the health value threshold, the driver's physiological state is analyzed based on the eye characteristics and the pressure characteristics, and the steps of obtaining the physiological value are specifically as follows:

[0092] Step S331: extracting feature information of eye features, where the feature information includes eyeball information and periorbital information.

[0093] Step S332 , extracting the color and texture of the peri-eye area based on the peri-eye information, obtaining the standard eye features of the driver, and obtaining the driver's dark circle degree AH and the driver's eye bag degree AD by comparison.

[0094] Step S333: extract the red blood streak area, red blood streak density and pupil area in the eyeball according to the eyeball information, and calculate the red blood streak area increase value AS, red blood streak density increase value AM and pupil area decrease value AT according to the standard eye characteristics.

[0095] Step S334, through the fatigue value correlation function The fatigue value is calculated, where 、 is the scaling factor and is greater than 0.

[0096] Step S335: Analyze the pressure characteristics and eye characteristics to obtain the driver's distraction value BF, and calculate the driver's distraction value BF according to the physiological correlation function. The physiological value CS is calculated, where 、 is the scaling factor and is greater than 0.

[0097] In practice, fatigue not only causes pupil constriction but also causes congestion of the driver's eyeballs, which is the appearance of red bloodshot. The area and density of the red bloodshot can reflect the driver's fatigue level. Furthermore, chronic fatigue and sleep deprivation can easily lead to bags under the eyes and dark circles. The eyeballs and the area around the eyes can reflect the driver's fatigue level. Even when a driver is not fatigued, distraction can still lead to accidents. Therefore, a driver's physiological value is calculated based on fatigue and distraction values. The higher the physiological value, the greater the risk to the driver when operating construction equipment.

[0098] The steps for obtaining the driver's distraction value BF based on the pressure characteristics and eye characteristics are as follows:

[0099] Step S3351: extract the movement trajectory of the driver's pupil position based on the eye features, and count the continuous stay time of the pupil at different positions.

[0100] Step S3352: Set a time threshold, filter the number of positions where the continuous stay time is less than the time threshold, and if the number of positions is not 0, extract the real-time pressure of the steering wheel based on the pressure feature.

[0101] Step S3353: Obtain the driver's historical driving data and extract the average steering wheel pressure in the historical driving data.

[0102] Step S3354: Calculate the pressure difference between the average pressure and the real-time pressure, and find the corresponding distraction value BF according to the preset pressure difference and distraction value correlation curve.

[0103] Step S3355: If the number of positions is 0, the driver's visual range is counted to obtain the standard range that the driver needs to observe for operation.

[0104] Step S3356, determining whether the sight range is within the standard range. If the sight range is within the standard range, determining that the distraction value BF=0.

[0105] Step S3357: If the sight range is not within the standard range, the range difference between the sight range and the standard range is calculated, and the corresponding distraction value BF is found according to the preset range difference and distraction value correlation curve.

[0106] In practice, when drivers are distracted or inattentive, their pupils may slightly rotate or drift, which is a normal physiological phenomenon. Because pupil drift or rotation occurs unconsciously, it typically lasts for a short time. When drivers are conscious, they typically remain in the field of view for slightly longer periods of time to absorb information. If pupil drift is detected based on time, the driver's distraction and degree of distraction can be analyzed based on the pressure distribution on the steering wheel. When a driver is distracted, they cannot accurately perceive and respond to changes in the vehicle and road, resulting in slow or excessive steering adjustments, which in turn increases the steering angle and force. Even if the driver's pupils are normal, this does not necessarily mean the driver is distracted. Therefore, driver distraction can be determined based on the driver's field of vision. For example, if the driver's pupils are normal, but standard operating procedures require observation of the construction equipment to the left and right before operating it, the driver's field of vision is limited to the front, thus also indicating distraction. If the driver's pupils are abnormal, the degree of distraction can be determined based on the steering wheel pressure.

[0107] Step S341, analyzing the driver's psychological state based on the sweat characteristics and pressure characteristics and obtaining a psychological value, specifically comprises:

[0108] Step S342 : extracting the sweat substance content associated with pressure from the sweat characteristics and recording it as the pressure substance content DY.

[0109] Step S343: extracting skin conductivity DD from the sweat characteristics.

[0110] Step S344: extract the driver's facial expression changes based on the monitoring video image to obtain a facial change value DM.

[0111] Step S345: According to the psychological association function Calculate the psychological value DX, where 、 、 is the scaling factor and is greater than 0.

[0112] In practice, cortisol is a substance known as a stress hormone. When the body is under stress, its concentration in the blood increases. Cortisol, at the same concentration as in the blood, can also be detected in sweat, reflecting stress levels. In addition, several other biochemical markers in sweat are associated with stress levels and can reflect driver stress. Skin conductivity is a commonly used psychophysiological indicator that reflects the activity of the sympathetic nervous system. When a driver is emotionally agitated, sympathetic nerve activity increases, sweat gland secretion increases, and skin conductivity rises. Therefore, changes in skin conductivity can be used as an effective indicator of driver emotional agitation. Salt concentration in sweat is the primary factor that alters the skin's electrical conductivity, so skin conductivity can be extracted from sweat. High levels of stress and emotional fluctuations can affect driver operating accuracy, thereby altering driving risk. Therefore, analyzing the driver's psychological state can help improve safety warnings and reduce accident rates.

[0113] The steps for environmental model training, obtaining feedback from people around the driver, and analyzing environmental characteristics to obtain environmental risk values are as follows:

[0114] Step S51: Acquire the response of the crowd around the driving device, including the crowd volume, crowd movement trajectory and crowd voice.

[0115] Step S52 : obtaining the average volume of the crowd before the driver operates the construction equipment, and calculating the volume difference EY between the crowd volume and the average volume.

[0116] Step S53 , obtaining the average moving speed of the crowd at the construction site, and extracting the trajectory distance EG where the crowd moving speed is greater than the average moving speed according to the crowd moving trajectory.

[0117] Step S54: set error keywords for the speech of the surrounding crowd when the driver makes an operation error, recognize the speech containing the error keywords in the crowd speech through named entity recognition and record it as error speech, and calculate the ratio EW of error speech to crowd speech.

[0118] Step S55, according to the reflection correlation function Get the EA reflecting the risk value, where 、 、 is the scaling factor and is greater than 0.

[0119] Step S56: Analyze the environmental characteristics to obtain the driving risk value FA, and then calculate the driving risk value FA according to the environmental risk correlation function. The environmental hazard value GA is calculated, where 、 is the scaling factor and is greater than 0.

[0120] In practice, even if a driver is fatigued or emotionally unstable, different drivers' driving skills don't necessarily mean an accident will occur. A sudden braking or sharp turn doesn't necessarily indicate driver error. Similarly, even if a driver's physical and mental health are within normal ranges, it doesn't mean they haven't made mistakes. Construction equipment drivers often operate at construction sites, where numerous other workers gather. Their feedback can reveal the driver's driving performance. For example, a driver operating an excavator should theoretically raise the boom two meters after digging. The driver's physical and mental health are within normal ranges, and they operate the boom two meters. However, in reality, the construction site is stacked with a large amount of lumber, and the driver needs to cross it, requiring a three-meter raise. Therefore, the driver made an error. When a driver makes an error, the crowd around them will intervene, discuss, and raise their voices. Furthermore, the crowd may disperse to avoid falling lumber or gather to sort the lumber, increasing their movement patterns. Based on the voice output of the crowd, it is possible to determine whether the driver has made a mistake and judge the driver's driving situation from the external environment, reducing the situation where misjudgment occurs due to normal indicators and improving the accuracy of driving behavior analysis and warning.

[0121] The steps for obtaining the driving risk value FA based on environmental characteristics analysis are as follows:

[0122] Step S561 : extracting the weather severity FT, the driving terrain flatness FP, and the operation difficulty FN according to the environmental characteristics.

[0123] The difficulty of operation is determined by the terrain and material stacking conditions at the construction site. The more terrain obstacles there are and the more materials are stacked, the greater the difficulty of operation.

[0124] Step S562: Obtain the standard terrain flatness FB of the construction equipment operated by the driver.

[0125] Step S563: Combine the driver's driving skill level FS and calculate the driving risk correlation function The driving risk value FA is calculated, where 、 、 、 is the scaling factor and is greater than 0.

[0126] In practice, not only does the driver's driving operation and state affect driving safety, but the driving environment also impacts safety. Construction sites, in particular, can be challenging due to uneven roads and excessive material accumulation. Inclement weather can further impede driver operation and increase driving risks. For example, rain can impair visibility, while uneven terrain can increase the jolting of construction equipment. Greater operational difficulty and poorer driver skills increase driving risks.

[0127] The data trained by the model is used to make a single judgment to determine whether the driver's behavior is dangerous. If the driver's behavior is dangerous, an alarm is issued in a timely manner. The specific steps are as follows:

[0128] Step S61, setting a physiological value threshold. When the physiological value CS reaches the physiological value threshold, the driver's behavior is judged to be dangerous, a physiological alarm is issued, and physiological related data is transmitted to the terminal.

[0129] Step S62, setting a psychological value threshold. When the psychological value DX reaches the psychological value threshold, the driver's behavior is judged to be dangerous, a psychological alarm is issued, and psychological related data is transmitted to the terminal.

[0130] Step S63: Setting an operation risk value threshold. When the operation risk value reaches the operation risk value threshold, the driver's behavior is judged to be dangerous, an operation alarm is issued, and operation-related data is transmitted to the terminal.

[0131] Step S64, setting an environmental risk value threshold. When the environmental risk value GA reaches the environmental risk value threshold, the driver's behavior is judged to be dangerous, an environmental alarm is issued, and environmental related data is transmitted to the terminal.

[0132] In practice, if any driver metric exceeds a threshold, regardless of how excellent other metrics may be, it still poses a significant driving risk, necessitating timely warnings. For example, if a driver's current physiological value exceeds a threshold, it indicates that the driver is no longer able to operate accurately, necessitating a prompt alarm and appropriate adjustments. If it's a physiological alarm, the driver should be replaced immediately; if it's an environmental alarm, construction equipment operations should be suspended or adjustments should be made to the construction site.

[0133] If the driver's behavior is not dangerous, all the data trained by the model will be integrated and calculated to obtain a comprehensive risk value to determine whether the driver's behavior is dangerous. If the driver's behavior is dangerous, the dangerous situation will be extracted and an alarm will be issued. The specific steps are as follows:

[0134] Step S71: If the driver's behavior is not dangerous, then The comprehensive risk value ZA is calculated, where 、 、 、 is the scaling factor and is greater than 0.

[0135] Step S72: Set a comprehensive risk value threshold. If the comprehensive risk value ZA reaches the comprehensive risk value threshold, the driver's behavior is judged to be dangerous.

[0136] Step S73, respectively calculating the differences between the physiological value, psychological value, operational risk value, environmental risk value and the corresponding thresholds, comparing and obtaining the value with the smallest difference and recording it as the alarm value.

[0137] Step S74: Look up the alarm type corresponding to the alarm value, issue an alarm, and transmit the associated data corresponding to the alarm type to the terminal.

[0138] In practice, even if a driver's individual indicators do not exceed the threshold, it does not necessarily mean that driving is completely safe. When individual indicators do not exceed the threshold, all indicators must be integrated for judgment. For example, if the driver is tired, has poor driving skills, and is in a harsh driving environment, the risk of an accident remains high. When the combined risk value reaches the threshold, the most serious indicator is selected and uploaded to the terminal for alarm. For example, if it is lightly raining and the driver is somewhat tired but operating normally, the difference between the physiological value and the threshold is minimal, indicating that physiological condition is the primary factor affecting driving risk. Selecting physiological condition and uploading it to the terminal can lead to a driver replacement solution.

[0139] The deep learning-based driver behavior analysis safety warning system, by applying the above-mentioned deep learning-based driver behavior analysis safety warning method, includes:

[0140] The data acquisition module collects the driver's driving data through monitoring videos and sensors, and pre-processes the driving data to obtain raw data.

[0141] The feature extraction module inputs the original data into the deep learning algorithm to perform feature extraction to obtain physiological features, operational features and environmental features. The extracted features are then trained on physiological models, operational models and environmental models respectively.

[0142] Physiological training module, physiological model training, analyzes the driver's physiological and psychological states according to physiological characteristics and obtains physiological and psychological values.

[0143] The operation training module, operation model training, obtains the driver's driving skill level, analyzes the driver's operating behavior risk level in combination with the operation characteristics, and obtains the operation risk value.

[0144] Environmental training module, environmental model training, obtains feedback from people around the driver, and combines environmental characteristics analysis to obtain environmental hazard values.

[0145] The single judgment module will make a single judgment based on the data trained by the model to determine whether the driver's behavior is dangerous. If the driver's behavior is dangerous, an alarm will be issued in time.

[0146] The comprehensive judgment module integrates and calculates all the data trained by the model to obtain a comprehensive danger value to determine whether the driver's behavior is dangerous. If the driver's behavior is dangerous, the dangerous situation will be extracted and an alarm will be issued.

[0147] In actual application, driver behavior analysis and safety warning are realized through different modules, which reduces the subjectivity of manual analysis, improves the accuracy of safety warning, reduces the waste of human resources, and improves the convenience of safety warning.

[0148] The system's implementation principle is as follows: The data acquisition module collects driving data from surveillance video and sensors and preprocesses it to generate raw data. This includes image denoising, transformation, and normalization to improve data quality and consistency, facilitating subsequent processing and analysis. The feature extraction module feeds the raw data into a deep learning algorithm for feature extraction, generating physiological, operational, and environmental features. These extracted features are then used to train physiological, operational, and environmental models. The physiological training module analyzes the driver's fatigue and distraction based on physiological features to generate physiological values, and analyzes stress and emotional fluctuations through sweat to generate psychological values. The operational training module determines the driver's driving skill level and, based on operational features, analyzes the riskiness of their operational behavior to generate an operational risk value. The environmental training module collects feedback from people around the driver and, based on the volume, movement trajectory, and output speech content of the surrounding people, combines this information with environmental features to generate an environmental risk value. The single judgment module uses the model-trained data to perform a single judgment, determining whether the driver's behavior is dangerous and issuing a timely alarm if it is. The comprehensive judgment module integrates and calculates all the data trained by the model to obtain a comprehensive danger value to determine whether the driver's behavior is dangerous. If the driver's behavior is dangerous, the dangerous situation will be extracted and an alarm will be issued.

[0149] The above are all preferred embodiments of the present application, and are not intended to limit the scope of protection of the present application. Therefore, any equivalent changes made based on the structure, shape, and principle of the present application should be included in the scope of protection of the present application.

Claims

1. A driver behavior analysis safety warning method based on deep learning, characterized by: The following steps are involved: The driver's driving data is collected through monitoring videos and sensors, and the driving data is pre-processed to obtain raw data; The raw data is input into the deep learning algorithm for feature extraction to obtain physiological features, operational features, and environmental features. The extracted features are then used for physiological model training, operational model training, and environmental model training respectively. The physiological model training analyzes the driver's physiological state and psychological state according to physiological characteristics and obtains physiological values and psychological values; The operation model training obtains the driver's driving skill level, analyzes the driver's operation behavior risk level in combination with the operation characteristics, and obtains the operation risk value; The environmental model training obtains the feedback of people around the driver and obtains the environmental risk value by combining the environmental characteristics analysis; The data trained by the model will be used to make a single judgment to determine whether the driver's behavior is dangerous. If the driver's behavior is dangerous, an alarm will be issued in time; If the driver's behavior is not dangerous, all the data trained by the model will be integrated and calculated to obtain a comprehensive risk value to determine whether the driver's behavior is dangerous. If the driver's behavior is dangerous, the dangerous situation will be extracted and an alarm will be issued; The physiological model training, which analyzes the driver's physiological state and psychological state according to physiological characteristics and obtains physiological values and psychological values, is specifically as follows: The physiological characteristics include eye characteristics, sweat characteristics, pressure characteristics and human body characteristics; The driver's health status is obtained based on human body characteristics and recorded as a health value. A health value threshold is set. If the health value is less than the health value threshold, an alarm is directly triggered. If the health value is not less than the health value threshold, the driver's physiological state is analyzed based on the eye characteristics and pressure characteristics, and a physiological value is obtained; Analyze the driver's psychological state based on sweat characteristics and stress characteristics, and obtain the psychological value; If the health value is not less than the health value threshold, the driver's physiological state is analyzed based on the eye characteristics and the pressure characteristics, and the steps of obtaining the physiological value are specifically as follows: Extracting feature information of eye features, wherein the feature information includes eyeball information and periorbital information; Extract the color and texture of the eye area based on the peri-eye information to obtain the driver's standard eye features. By comparing them, the driver's dark circle degree AH and the driver's eye bag degree AD are obtained. The red blood streak area, red blood streak density and pupil area in the eyeball are extracted based on the eyeball information, and the red blood streak area increase value AS, red blood streak density increase value AM and pupil area decrease value AT are calculated based on the standard eye characteristics; Through fatigue value correlation function The fatigue value is calculated, where 、 is the scale factor and is greater than 0; The driver's distraction value BF is obtained by analyzing the pressure characteristics and eye characteristics, and the physiological correlation function The physiological value CS is calculated, where 、 is the scale factor and is greater than 0; The steps of training the environmental model, obtaining feedback from people around the driver, and analyzing the environmental characteristics to obtain the environmental risk value are specifically as follows: Obtaining the response of the crowd around the driving device, including the crowd volume, crowd movement trajectory, and crowd voice; Obtain the average volume of the crowd in front of the driver operating the construction equipment, and calculate the volume difference EY between the crowd volume and the average volume; Obtain the average moving speed of the crowd at the construction site, and extract the trajectory distance EG where the crowd's moving speed is greater than the average moving speed based on the crowd's movement trajectory; Set error keywords for the speech of the surrounding crowd when the driver makes an operation error. Use named entity recognition to record the speech containing the error keyword as error speech. Calculate the ratio of error speech to crowd speech (EW). According to the reflection correlation function Get the EA reflecting the risk value, where 、 、 is the scale factor and is greater than 0; According to the environmental characteristics analysis, the driving risk value FA is obtained, and the environmental risk correlation function is used The environmental hazard value GA is calculated, where 、 is the scaling factor and is greater than 0.

2. The driver behavior analysis safety warning method based on deep learning according to claim 1 is characterized in that: The step of obtaining the driver's distraction value BF based on the pressure characteristics and eye characteristics analysis is specifically as follows: The motion trajectory of the driver's pupil position is extracted based on the eye features, and the continuous residence time of the pupil in different positions is counted; Set a time threshold to filter the number of locations where the continuous dwell time is less than the time threshold. If the number of locations is not 0, extract the real-time steering wheel pressure based on the pressure feature. Obtain the driver's historical driving data and extract the average steering wheel pressure from the historical driving data; Calculate the pressure difference between the average pressure and the real-time pressure, and find the corresponding distraction value BF according to the preset pressure difference and distraction value correlation curve; If the number of positions is 0, the driver's visual range is counted to obtain the standard range that the driver needs to observe during operation; Determine whether the sight range is within the standard range. If the sight range is within the standard range, the distraction value BF=0; If the sight range is not within the standard range, the range difference between the sight range and the standard range is calculated, and the corresponding distraction value BF is found according to the preset range difference and distraction value correlation curve.

3. The driver behavior analysis safety warning method based on deep learning according to claim 2 is characterized in that: The step of analyzing the driver's psychological state according to the sweat characteristics and the pressure characteristics and obtaining the psychological value is specifically as follows: Extract the sweat substance content associated with stress from the sweat characteristics and record it as stress substance content DY; Extract skin conductivity DD from sweat characteristics; Extract the driver's facial expression changes based on the surveillance video image and obtain the facial change value DM; According to the psychological correlation function Calculate the psychological value DX, where 、 、 is the scaling factor and is greater than 0.

4. The driver behavior analysis safety warning method based on deep learning according to claim 3 is characterized in that: The step of obtaining the driving risk value FA based on the environmental characteristics analysis is specifically as follows: According to the environmental characteristics, the weather severity FT, driving terrain flatness FP and operation difficulty FN are obtained; Obtaining the standard terrain flatness FB of the construction equipment operated by the driver; Combined with the driver's driving skill level FS, according to the driving risk association function The driving risk value FA is calculated, where 、 、 、 is the scaling factor and is greater than 0.

5. The driver behavior analysis safety warning method based on deep learning according to claim 4 is characterized in that: The steps of making a single judgment based on the data trained by the model to determine whether the driver's behavior is dangerous and promptly issuing an alarm if the driver's behavior is dangerous are specifically as follows: Set a physiological value threshold. When the physiological value CS reaches the physiological value threshold, the driver's behavior is judged to be dangerous, a physiological alarm is issued, and physiological related data is transmitted to the terminal; Set a psychological value threshold. When the psychological value DX reaches the psychological value threshold, the driver's behavior is judged to be dangerous, a psychological alarm is issued, and psychological related data is transmitted to the terminal; Set an operation risk value threshold. When the operation risk value reaches the operation risk value threshold, the driver's behavior is judged to be dangerous, an operation alarm is issued, and operation-related data is transmitted to the terminal; Set an environmental hazard value threshold. When the environmental hazard value GA reaches the environmental hazard value threshold, the driver's behavior is judged to be dangerous, an environmental alarm is issued, and environmental related data is transmitted to the terminal.

6. The driver behavior analysis safety warning method based on deep learning according to claim 5 is characterized in that: If the driver's behavior is not dangerous, all the data trained by the model are integrated and calculated to obtain a comprehensive risk value to determine whether the driver's behavior is dangerous. If the driver's behavior is dangerous, the dangerous situation is extracted and an alarm is issued. Specifically, the steps are: If the driver's behavior is judged to be not dangerous, then according to the comprehensive correlation function The comprehensive risk value ZA is calculated, where 、 、 、 is the scale factor and is greater than 0; Set a comprehensive risk value threshold. If the comprehensive risk value ZA reaches the comprehensive risk value threshold, the driver's behavior is judged to be dangerous. Calculate the difference between the physiological value, psychological value, operational risk value, environmental risk value and the corresponding threshold value respectively, compare and obtain the value with the smallest difference and record it as the alarm value; Find the alarm type corresponding to the alarm value, issue an alarm, and transmit the associated data corresponding to the alarm type to the terminal.

7. Driver behavior analysis safety warning system based on deep learning, characterized by: The method for driver behavior analysis and safety warning based on deep learning according to any one of claims 1 to 6 comprises: The data collection module collects the driver's driving data through monitoring videos and sensors, and pre-processes the driving data to obtain raw data; The feature extraction module inputs the raw data into the deep learning algorithm to extract features, obtain physiological features, operational features, and environmental features, and then performs physiological model training, operational model training, and environmental model training on the extracted features respectively; Physiological training module, the physiological model training analyzes the driver's physiological state and psychological state according to physiological characteristics and obtains physiological values and psychological values; The operation training module, which trains the operation model, obtains the driver's driving skill level, analyzes the risk level of the driver's operation behavior in combination with the operation characteristics, and obtains the operation risk value; Environmental training module, the environmental model training, obtains the feedback of people around the driver, and combines the environmental characteristics analysis to obtain the environmental risk value; The single judgment module uses the data trained by the model to make a single judgment to determine whether the driver's behavior is dangerous. If the driver's behavior is dangerous, an alarm will be issued in time; The comprehensive judgment module integrates and calculates all the data trained by the model to obtain a comprehensive danger value to determine whether the driver's behavior is dangerous. If the driver's behavior is dangerous, the dangerous situation will be extracted and an alarm will be issued.

Images