A safe driving monitoring system for engineering vehicles based on the fusion of facial and behavioral recognition

Abstract

The present invention relates to the technical field of monitoring systems, and specifically to a safe driving monitoring system for engineering vehicles based on the fusion of facial and behavioral recognition. First, the present invention proposes a dual-branch perception model for facial perception, which captures the driver's overall facial features and local detail features respectively through global and local attention mechanisms, thereby improving the recognition accuracy of fatigue driving and distracted driving. Secondly, the present invention proposes a multimodal recognition word vector analysis method for behavioral perception, which uses different models to recognize the driver's torso and surrounding environment images in real time, and compares the behavioral word vector with the object word vector to determine whether the driver has any irregular operation. Finally, the present invention integrates and learns a variety of data through the Q-Learning algorithm, and generates accurate assisted driving instructions through reinforcement learning, thereby improving the driver's user experience.

Description

Technical Field

[0001] The present invention relates to the technical field of monitoring systems, and in particular to an engineering vehicle safe driving monitoring system based on the fusion of facial and behavior recognition. Background Art

[0002] Construction vehicles play a vital role in transportation and urban development. These vehicles often operate for extended periods on construction sites and roads, carrying out heavy transport and construction tasks. Due to their large size and heavy loads, traffic accidents can be more severe, potentially resulting in casualties and property damage. Therefore, safe driving inspections for construction vehicles are crucial to ensure drivers remain alert and minimize the likelihood of accidents.

[0003] Currently, safe driving detection for construction vehicles relies primarily on image processing, machine learning, and sensor technologies. These technologies monitor driver behavior and vehicle status in real time, enabling timely identification of potentially dangerous behaviors such as fatigue and distracted driving. These advanced technologies can improve the safety of construction vehicles, reduce accident risks, enhance driver safety awareness, and enhance the working environment.

[0004] However, current safety testing for engineering vehicles still faces some challenges. First, motion recognition is not sufficiently precise. For example, a driver's hair-twisting gesture could be misidentified as a phone call, or a lip-wiping gesture could be identified as smoking. This can lead to frequent, unnecessary alarm triggering. Second, the system can frequently sound alarms even during breaks, distracting drivers and potentially reducing their willingness to use the system, thus impacting their motivation to drive safely.

[0005] To this end, a safe driving monitoring system for engineering vehicles based on the fusion of face and behavior recognition is proposed. Summary of the Invention

[0006] The purpose of the present invention is to provide an engineering vehicle safe driving monitoring system based on the fusion of facial and behavioral recognition. First, the present invention proposes a dual-branch perception model for facial perception, which captures the overall and local features of the driver's face through global and local attention mechanisms, respectively, thereby improving the recognition accuracy of fatigue and distracted driving. Secondly, the present invention proposes a multimodal recognition word vector analysis method, which generates behavioral word vectors and object word vectors through real-time image recognition of the driver's torso and the surrounding environment, and compares them to determine whether there are irregular operations. Finally, the present invention integrates and learns multiple data through the Q-Learning algorithm, optimizes the driving decision-making process, reduces the false alarm rate, and improves driving safety and user experience.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] A safety driving monitoring system for engineering vehicles based on the fusion of facial and behavioral recognition, including:

[0009] The facial perception module is used to obtain real-time images of the driver's face and build a two-branch perception model. The gaze monitoring branch analyzes the overall facial features through a global attention mechanism to obtain global features; the fatigue monitoring branch focuses on the facial features through a local attention mechanism to obtain local features. The global features and local features are fused using a self-attention weighted algorithm to obtain the facial perception result.

[0010] A behavior perception module is used to obtain real-time images of the driver's torso and the environment, obtain the driver's current action through an action recognition model, obtain objects around the driver through an object detection model, and perform correlation analysis between the action and the surrounding objects based on a multimodal recognition word vector analysis method to obtain behavior perception results;

[0011] A face and behavior fusion module, configured to fuse the face perception result and the behavior perception result to obtain driving status data;

[0012] A traffic light recognition module is used to obtain a real-time image of traffic lights on an external road, and classify the traffic light status of the real-time image of the traffic light to obtain traffic signal data;

[0013] The start-stop sensing module is used to obtain the current vehicle driving status and vehicle start-stop data;

[0014] An algorithm parsing module is used to connect the facial and behavior fusion module, the traffic light recognition module, and the start-stop perception module, and generate assisted driving instructions based on the Q-Learning algorithm in combination with the driving status data, the traffic signal data, and the vehicle start-stop data.

[0015] Furthermore, the facial perception results include normal driving, distracted driving and fatigue driving, and the behavior perception results include standard driving and irregular driving, and the irregular driving includes at least one of smoking, drinking water or making phone calls.

[0016] Furthermore, the facial perception module includes: a sight line monitoring unit for monitoring whether the driver's sight deviates from the road ahead; a fatigue monitoring unit for monitoring the driver's fatigue state; obtaining the facial perception result includes:

[0017] Using a first camera to obtain a real-time facial image of the driver;

[0018] identifying, based on the gaze monitoring unit, whether the real-time facial image shows a face turning state, and if so, recording the face turning angle and turning time; and determining distracted driving if the face turning angle is greater than a turning angle threshold and the turning time is greater than a turning time threshold;

[0019] Based on the fatigue monitoring unit, it is identified whether the real-time facial image is in a closed-eye state or a yawning state. If it is in a closed-eye state, the closed-eye time and the closed-eye frequency are recorded. If the closed-eye time is greater than the closed-eye time threshold or the closed-eye frequency is greater than the closed-eye frequency threshold, it is determined to be fatigue driving; if it is in a yawning state, the yawning frequency is recorded. If the yawning frequency is greater than the yawning frequency threshold, it is determined to be fatigue driving.

[0020] Furthermore, the dual-branch perception model includes:

[0021] The input layer is used to receive the real-time facial image of the driver acquired by the first camera;

[0022] The gaze monitoring branch is used to analyze the overall facial features through a global attention mechanism to obtain global features, which include facial contour features and facial orientation features;

[0023] The fatigue monitoring branch is used to focus on facial features through a local attention mechanism to obtain local features, including eye features and mouth features;

[0024] A branch fusion layer, configured to fuse the global features and the local features using a self-attention weighted algorithm to generate comprehensive facial features;

[0025] A classification layer, for classifying the comprehensive facial features and outputting facial perception results;

[0026] The input layer is a 3×3 convolutional layer;

[0027] The gaze monitoring branch includes a downsampling layer and four non-local residual blocks connected in sequence, and the non-local residual block includes three residual blocks connected in sequence, a maximum pooling layer and a non-local attention module;

[0028] The fatigue monitoring branch includes five sequentially connected local direct convolution blocks, each of which includes three sequentially connected 3×3 convolution layers, a maximum pooling layer, and a local attention module;

[0029] The branch fusion layer fuses the global features and the local features through a self-attention weighted algorithm. The formula of the self-attention weighted algorithm is:

[0030] F fused =F global +α·V local ;

[0031] Among them, F fused Indicates comprehensive facial features, F global represents the global feature, α represents the attention weight, V local represents the local eigenvalue;

[0032] The classification layer classifies the fused features through the softmax function.

[0033] Furthermore, the behavior perception module includes a behavior recognition unit, a target detection unit, and a correction unit; obtaining the behavior perception result includes:

[0034] A second camera is used to obtain a real-time image of the driver's torso, and a third camera is used to obtain a real-time image of the driver's surrounding environment;

[0035] The behavior recognition unit processes the real-time torso image, identifies the driver's behavior by extracting key points of the driver, and converts the recognition result into a behavior word vector;

[0036] The target detection unit analyzes the real-time image of the environment, identifies the objects in the image, and converts the recognition results into object word vectors;

[0037] Expand the five related word vectors of the behavior word vector through the ontology library and compare them with the object word vector to obtain a comparison result;

[0038] The correction unit determines whether the comparison result is correct. If the comparison result is correct, the behavior recognition is confirmed as an irregular operation; if the comparison result is incorrect, the behavior recognition result is corrected.

[0039] Furthermore, the face and behavior fusion module adopts a weighted voting mechanism for fusion, and the calculation formula of the weighted voting mechanism is:

[0040] C combined =W face ×C face +W behavior ×C behavior ;

[0041] Among them, C combined represents the perception fusion score, W face represents the facial perception weight, C face represents the face perception confidence, W behavior represents the behavior perception weight, C behavior Indicates the confidence of behavior perception.

[0042] Furthermore, the signal light status includes: red light status, reminding the driver to stop; yellow light status, reminding the driver to slow down or prepare to stop; green light status, reminding the driver to pass safely; no signal light status detected, reminding the driver to drive carefully and make judgments based on the actual road conditions.

[0043] Furthermore, the vehicle start-stop data is acquired by an acceleration sensor, and the vehicle start-stop data includes a start-stop state, a steering state, an acceleration state and a current speed.

[0044] Furthermore, the algorithm parsing module includes: a data processing unit for preprocessing data transmitted by the face and behavior fusion module, the traffic light recognition module, and the start-stop perception module to obtain preprocessed data; a decision algorithm unit for performing reinforcement learning on the preprocessed data using the Q-Learning algorithm according to preset driving safety rules and road traffic regulations, and generating assisted driving instructions; generating the assisted driving instructions includes:

[0045] Determine the state space, action space, and reward function, define the combination of driving behavior, traffic signals, and vehicle status, and set driving instructions and reward mechanisms based on the combination;

[0046] Build a driving simulation environment that provides reward information based on real-time feedback of the current state and execution actions;

[0047] Initialize the Q table and select actions through the ε-greedy strategy, observe the environment feedback to update the Q value, and store the Q value in the Q table;

[0048] Monitor the status during actual driving, use the Q table to query the best action and generate corresponding driving instructions;

[0049] Regularly evaluate model performance, monitor safety and efficiency through testing, and adjust parameters to optimize learning results.

[0050] Furthermore, the driving monitoring system further includes:

[0051] A voice broadcast module, connected to the algorithm analysis module, for providing voice prompts or warnings according to the assisted driving instructions generated by the algorithm analysis module;

[0052] A communication module is used to communicate with a remote server or external device, upload the monitored driving status data, the traffic signal data, and the vehicle start and stop data, and receive control instructions from the remote server. The communication module supports wireless communication protocols, including at least one or more of 4G, 5G, and Wi-Fi wireless communication technologies, and is used to achieve two-way data transmission with the remote server.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] This paper proposes a dual-branch perception model. This model uses a global attention mechanism to analyze the driver's entire face to capture global features and promptly identify distracted driving. Simultaneously, a local attention mechanism focuses on capturing local features, such as eye state and mouth movements, to further identify fatigue. By combining these two mechanisms with a weighted self-attention algorithm, the model achieves a more granular facial perception, improving the accuracy of fatigue and distracted driving detection, thereby providing reliable technical support for real-time driving safety monitoring.

[0055] 2. This paper proposes a multimodal recognition word vector analysis method. This method uses different models to identify real-time images of the driver's torso and the environment, converting the recognition results into behavior word vectors and object word vectors. The method then uses an ontology library to expand the related word vectors of the behavior word vectors and compares them with the object word vectors to determine whether the driver has engaged in irregular operations. Compared to existing technologies, this method can correlate behavior recognition results with target detection results through word vector comparison, thereby reducing false positive and false alarm rates and improving the accuracy and reliability of the monitoring system.

[0056] 3. This invention uses a Q-Learning algorithm to perform reinforcement learning on driving status data, traffic signal data, and vehicle start-stop data to reduce the frequency of false alarms. By continuously iterating and updating the Q value, the system can identify and learn the driver's behavior patterns in different situations, thereby optimizing alarm triggering conditions and reducing unnecessary alarms caused by misjudgments. This intelligent decision-making process not only improves the driver experience but also enhances system reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A schematic diagram of the structure of a safe driving monitoring system for engineering vehicles based on the fusion of facial and behavioral recognition provided by an embodiment of the present invention;

[0058] Figure 2 A schematic diagram of the dual-branch perception model structure provided by an embodiment of the present invention;

[0059] Figure 3 A schematic diagram of a non-local residual block structure provided by an embodiment of the present invention;

[0060] Figure 4 A schematic diagram of the structure of a local direct convolution block provided by an embodiment of the present invention;

[0061] Figure 5 A schematic diagram of a basic residual block structure provided by an embodiment of the present invention;

[0062] Figure 6This is a flowchart of an execution of an engineering vehicle safe driving monitoring system based on the fusion of facial and behavior recognition provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0063] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0064] Example 1

[0065] A safety driving monitoring system for engineering vehicles based on the fusion of facial and behavioral recognition, such as Figure 1 As shown, including:

[0066] The facial perception module is used to obtain real-time images of the driver's face and build a two-branch perception model. The gaze monitoring branch analyzes the overall facial features through a global attention mechanism to obtain global features; the fatigue monitoring branch focuses on the facial features through a local attention mechanism to obtain local features. The global features and local features are fused using a self-attention weighted algorithm to obtain the facial perception result.

[0067] A behavior perception module is used to obtain real-time images of the driver's torso and the environment, obtain the driver's current action through an action recognition model, obtain objects around the driver through an object detection model, and perform correlation analysis between the action and the surrounding objects based on a multimodal recognition word vector analysis method to obtain behavior perception results;

[0068] A face and behavior fusion module, configured to fuse the face perception result and the behavior perception result to obtain driving status data;

[0069] A traffic light recognition module is used to obtain a real-time image of traffic lights on an external road, and classify the traffic light status of the real-time image of the traffic light to obtain traffic signal data;

[0070] The start-stop sensing module is used to obtain the current vehicle driving status and vehicle start-stop data;

[0071] An algorithm parsing module is used to connect the facial and behavior fusion module, the traffic light recognition module, and the start-stop perception module, and generate assisted driving instructions based on the Q-Learning algorithm in combination with the driving status data, the traffic signal data, and the vehicle start-stop data.

[0072] Furthermore, the facial perception results include normal driving, distracted driving and fatigue driving, and the behavior perception results include standard driving and irregular driving, and the irregular driving includes at least one of smoking, drinking water or making phone calls.

[0073] Facial perception results include normal driving, distracted driving, and fatigued driving, effectively distinguishing between driver attention and fatigue. Behavioral perception results determine whether driving is standard, with irregular driving behaviors including smoking, drinking, and making phone calls. By combining these perception results, the system can more accurately identify and warn of potential driving risks, thereby improving driving safety and reducing false alarms.

[0074] Furthermore, the facial perception module includes: a sight line monitoring unit for monitoring whether the driver's sight deviates from the road ahead; a fatigue monitoring unit for monitoring the driver's fatigue state; obtaining the facial perception result includes:

[0075] Using a first camera to obtain a real-time facial image of the driver;

[0076] identifying, based on the gaze monitoring unit, whether the real-time facial image shows a face turning state, and if so, recording the face turning angle and turning time; and determining distracted driving if the face turning angle is greater than a turning angle threshold and the turning time is greater than a turning time threshold;

[0077] Based on the fatigue monitoring unit, it is identified whether the real-time facial image is in a closed-eye state or a yawning state. If it is in a closed-eye state, the closed-eye time and the closed-eye frequency are recorded. If the closed-eye time is greater than the closed-eye time threshold or the closed-eye frequency is greater than the closed-eye frequency threshold, it is determined to be fatigue driving; if it is in a yawning state, the yawning frequency is recorded. If the yawning frequency is greater than the yawning frequency threshold, it is determined to be fatigue driving.

[0078] In a feasible implementation scheme, facial turning includes looking up, looking down, turning to the left and turning to the right, where the turning angle thresholds for looking up, looking down, turning to the left and turning to the right are [30°, 30°, 30°, 45°] respectively, the eye closure time threshold is 3 seconds, the eye closure frequency threshold is 15 times / minute, and the yawning frequency threshold is 3 times / minute.

[0079] The coordinated work of the Gaze Monitoring Unit and the Fatigue Monitoring Unit enables precise assessment of driver distraction and fatigue. Using real-time facial images of the driver, the Facial Perception Module identifies key behavioral characteristics such as turning, eye closure, and yawning, and makes judgments based on predefined angle, time, and frequency thresholds. This reduces the frequency of alerts caused by checking the left and right rearview mirrors or personal eye or mouth habits, thereby enhancing the user-friendly nature of the safe driving monitoring system.

[0080] Furthermore, the dual-branch perception model is as follows Figure 2 Shown, including:

[0081] The input layer is used to receive the real-time facial image of the driver acquired by the first camera;

[0082] The gaze monitoring branch is used to analyze the overall facial features through a global attention mechanism to obtain global features, which include facial contour features and facial orientation features;

[0083] The fatigue monitoring branch is used to focus on facial features through a local attention mechanism to obtain local features, including eye features and mouth features;

[0084] A branch fusion layer, configured to fuse the global features and the local features using a self-attention weighted algorithm to generate comprehensive facial features;

[0085] A classification layer, for classifying the comprehensive facial features and outputting facial perception results;

[0086] The input layer is a 3×3 convolutional layer;

[0087] The line of sight monitoring branch includes a downsampling layer and four non-local residual blocks connected in sequence, and the non-local residual blocks are as follows: Figure 3 As shown, it includes three residual blocks, a maximum pooling layer and a non-local attention module connected in sequence;

[0088] The fatigue monitoring branch includes five local direct convolution blocks connected in sequence, and the local direct convolution blocks are as follows: Figure 4 As shown, it includes three 3×3 convolutional layers, a maximum pooling layer and a local attention module connected in sequence;

[0089] The branch fusion layer fuses the global features and the local features through a self-attention weighted algorithm. The formula of the self-attention weighted algorithm is:

[0090] F fused =F global +α·V local ;

[0091] Among them, F fused Indicates comprehensive facial features, F global represents the global feature, α represents the attention weight, V local represents the local eigenvalue;

[0092] The classification layer classifies the fused features through the softmax function.

[0093] Specifically, the number of channels in the input layer is 64, the number of channels of a downsampling layer and four non-local residual blocks connected in sequence to the gaze monitoring branch are [64, 64, 128, 256, 512] respectively, and the number of channels of the five local direct convolution blocks connected in sequence to the fatigue monitoring branch are [64, 128, 256, 512, 512] respectively; the residual block is the basic residual block of ResNet-34, and its structure is as follows: Figure 5 As shown; the non-local attention module is a non-local self-attention module, and the local attention module is a SE attention module; the self-attention weighted algorithm formula is: F fused =F global +α·V local , Q global =W q ·F global , K local =W k ·F local , V local =W v ·F local , where F fused Indicates comprehensive facial features, F global represents the global feature, F local represents local features, α represents attention weight, V local represents the local eigenvalue, Q global represents the global feature query mapping matrix, K local represents the local feature key mapping matrix, softmax() represents the activation function, d k represents the scaling factor, W q represents the query mapping matrix, W k represents the key mapping matrix, W v Represents the value mapping matrix.

[0094] The dual-branch perception model achieves accurate recognition of the driver's facial features through its dual-branch structure. The gaze monitoring branch uses a global attention mechanism to capture overall facial features, such as facial contours and head orientation, ensuring that the system focuses on the driver's overall dynamics. The fatigue monitoring branch utilizes a local attention mechanism to focus on capturing local facial features, particularly key details of the eyes and mouth, such as fatigue signals like eye closure and yawning. Through the collaboration of these two branches, the system simultaneously extracts features at both the global and local levels, and after fusion, achieves more accurate driving status judgment, thereby improving the detection accuracy of fatigue and distracted driving.

[0095] Furthermore, the behavior perception module includes a behavior recognition unit, a target detection unit, and a correction unit; obtaining the behavior perception result includes:

[0096] A second camera is used to obtain a real-time image of the driver's torso, and a third camera is used to obtain a real-time image of the driver's surrounding environment;

[0097] The behavior recognition unit processes the real-time torso image, identifies the driver's behavior by extracting key points of the driver, and converts the recognition result into a behavior word vector;

[0098] The target detection unit analyzes the real-time image of the environment, identifies the objects in the image, and converts the recognition results into object word vectors;

[0099] Expand the five related word vectors of the behavior word vector through the ontology library and compare them with the object word vector to obtain a comparison result;

[0100] The correction unit determines whether the comparison result is correct. If the comparison result is correct, the behavior recognition is confirmed as an irregular operation; if the comparison result is incorrect, the behavior recognition result is corrected.

[0101] Specifically, the behavior recognition unit is implemented by an action recognition model, which can be an I3D, C3D or SlowFast model, and the target detection unit is implemented by a target detection model, preferably a YOLO v5 model. The conversion of behavior recognition results into behavior word vectors and the conversion of target detection results into object word vectors are both implemented using Word2Vec. The ontology library expands the behavior word vectors using WordNet. In one feasible implementation, the action "making a phone call" is converted into a word vector V0 by the Word2Vec model. The five related word vectors expanded by the ontology library are [V1, V2, V3, V4, V5], corresponding to "telephone," "mobile phone," "pager," "fax," and "mail," respectively. The word vector corresponding to the object "mobile phone" is V2. At this time, the object word vector is among the related word vectors, the comparison result is correct, and it is determined that the current driver is smoking. In another feasible implementation, the action "smoking" is converted into a word vector W0 by the Word2Vec model. The five related word vectors expanded by the ontology library are [W1, W2, W3, W4, W5], corresponding to "cigarette," "cigarette," "ash," "tar smoke," and "smoke," respectively. The target detection unit does not detect the hand holding an object, and the corresponding result is NONE. The object word vector is not in the related word vectors, the comparison result is incorrect, and the behavior recognition result is corrected.

[0102] The multimodal recognition word vector analysis method improves the accuracy of identifying irregular driving operations by combining driver behavior recognition and environmental perception. A second camera captures real-time images of the driver's torso, extracts key points to identify driving actions, and generates a behavior word vector. Simultaneously, a third camera captures the driver's surroundings, identifies relevant objects, and generates an object word vector. After expanding the behavior word vector using an ontology library, it is compared with the object word vector to determine the correlation between the action and the object. This ensures more accurate identification of irregular driving operations, reduces false positives, and improves system reliability.

[0103] Furthermore, the perception fusion unit adopts a weighted voting mechanism for fusion, and the weighted voting calculation formula is:

[0104] C combined =W face ×C face +W behavior ×C behavior ;

[0105] Among them, C combined represents the perception fusion score, W face represents the facial perception weight, C face represents the face perception confidence, W behavior represents the behavior perception weight, C behavior Indicates the confidence of behavior perception.

[0106] The recognition results of the facial and behavioral perception units are integrated through a weighted voting mechanism, and the driver's status is comprehensively judged based on the weighted voting calculation, ensuring that the contribution ratio of each perception module can be flexibly adjusted. This not only improves the accuracy of the recognition results, but also reduces the impact of misjudgments by a single module, thereby enhancing the overall stability and intelligence of the system.

[0107] Furthermore, the signal light status includes: red light status, reminding the driver to stop; yellow light status, reminding the driver to slow down or prepare to stop; green light status, reminding the driver to pass safely; no signal light status detected, reminding the driver to drive carefully and make judgments based on the actual road conditions.

[0108] By monitoring traffic light status in real time, the system provides precise alerts for different states, such as red, yellow, and green, helping drivers make quick and correct decisions in complex traffic environments. If no traffic light is detected, the system reminds the driver to proceed with caution and makes independent judgments based on the actual road conditions, effectively enhancing driving safety and flexibility.

[0109] Furthermore, the vehicle start-stop data is acquired by an acceleration sensor, and the vehicle start-stop data includes a start-stop state, a steering state, an acceleration state and a current speed.

[0110] Specifically, the start-stop state includes starting and stopping, the turning state includes left turning, right turning and straight driving, and the acceleration state includes acceleration, deceleration and constant speed.

[0111] Accelerometers capture data such as the vehicle's start / stop status, steering state, acceleration status, and current speed, accurately monitoring the vehicle's dynamic behavior. This start / stop data helps optimize driving command generation in various driving scenarios, improving driving safety and response efficiency.

[0112] Furthermore, the algorithm parsing module includes: a data processing unit for preprocessing data transmitted by the face and behavior fusion module, the traffic light recognition module, and the start-stop perception module to obtain preprocessed data; a decision algorithm unit for performing reinforcement learning on the preprocessed data using the Q-Learning algorithm according to preset driving safety rules and road traffic regulations, and generating assisted driving instructions; generating the assisted driving instructions includes:

[0113] Determine the state space, action space, and reward function, define the combination of driving behavior, traffic signals, and vehicle status, and set driving instructions and reward mechanisms based on the combination;

[0114] Build a driving simulation environment that provides reward information based on real-time feedback of the current state and execution actions;

[0115] Initialize the Q table and select actions through the ε-greedy strategy, observe the environment feedback to update the Q value, and store the Q value in the Q table;

[0116] Monitor the status during actual driving, use the Q table to query the best action and generate corresponding driving instructions;

[0117] Regularly evaluate model performance, monitor safety and efficiency through testing, and adjust parameters to optimize learning results.

[0118] Specifically, the state space includes driving state data (smoking, fatigue or distraction, etc.), traffic signal data (red light, green light, yellow light or no signal light) and vehicle start and stop data (start, turn, accelerate or decelerate, etc.). The action space is the current action of the vehicle, including acceleration, deceleration, braking, stopping or turning. The reward function is set according to driving safety rules and road traffic regulations, including positive rewards (safely passing the green light, accelerating smoothly or driving attentively) and negative rewards (sudden braking, running a red light or fatigue driving); a driving simulation environment is constructed for training the Q-Learning model. The environment should be able to provide real-time feedback on the current state, executed actions and rewards obtained. The current state includes driving state data, traffic signal data and vehicle start and stop data; a Q table is created, with rows representing states and columns representing actions. All Q values ​​are initialized to 0. At each time step t, according to the current state S t Select Action A t , when executing action A t When observing the next state S of the environmental feedback t+1 and the reward R t And update the Q value, where the Q value update algorithm is:

[0119]

[0120] Among them, Q(S t ,A t ) indicates that in state S t Select Action A t The expected cumulative reward that can be obtained after α represents the learning rate, ranging from 0≤α≤1, R t Current status S t Execute action A t The immediate reward obtained after γ represents the discount factor, ranging from 0≤γ≤1. Indicates the maximum expected reward after taking the optimal action in the next state; in the actual driving process, real-time monitoring of driving behavior, traffic signals and vehicle status forms the current state S t , query the Q table according to the current state, select the action with the highest Q value as the driving instruction, convert the selected action into actual driving instructions, and feed it back to the vehicle control system; by testing in different scenarios, evaluate the decision-making effect of the model, monitor indicators such as safety and driving efficiency, and adjust parameters such as reward function, learning rate, discount factor, etc. according to the evaluation results to optimize the learning effect.

[0121] The algorithm parsing module preprocesses data transmitted by the facial and behavioral fusion module, the traffic light recognition module, and the start-stop perception module through the data processing unit, ensuring efficient integration and accuracy of input data. Combined with the Q-Learning algorithm in the decision-making algorithm module, the system performs reinforcement learning based on driving safety rules and road traffic regulations, intelligently optimizing driving decisions. This solution not only improves the accuracy of assisted driving but also continuously adapts to complex road conditions and driving behaviors through self-learning, thereby enhancing driving safety and the driver's operating experience.

[0122] The engineering vehicle safety driving monitoring system of this invention integrates a facial and behavioral fusion module, a signal light recognition and monitoring module, a start-stop perception module, and an algorithm analysis module to achieve comprehensive, real-time monitoring of the driver's status, traffic signals, and vehicle driving conditions. It also utilizes a Q-Learning algorithm for reinforcement learning to generate precise assisted driving instructions. This multi-layered data collection and analysis improves driving safety and real-time responsiveness, enhancing the driver's operational experience and trust in the system.

[0123] Furthermore, the driving monitoring system further includes:

[0124] A voice broadcast module, connected to the algorithm analysis module, for providing voice prompts or warnings according to the assisted driving instructions generated by the algorithm analysis module;

[0125] A communication module is used to communicate with a remote server or external device, upload the monitored driving status data, the traffic signal data, and the vehicle start and stop data, and receive control instructions from the remote server. The communication module supports wireless communication protocols, including at least one or more of 4G, 5G, and Wi-Fi wireless communication technologies, and is used to achieve two-way data transmission with the remote server.

[0126] The system uses the voice broadcast module to convert assisted driving instructions into instant voice prompts or warnings, helping drivers to grasp safety information in a timely manner; through the communication module, it ensures efficient two-way data transmission with remote servers or external devices, not only uploading monitoring data such as driving status, traffic signals, vehicle start and stop, but also receiving remote commands, providing convenient and reliable communication support for remote monitoring and control.

[0127] The embodiment provides a safety driving monitoring system for engineering vehicles based on the fusion of facial and behavioral recognition, the flow chart of which is shown in FIG6 , including:

[0128] A real-time facial image of the driver is acquired and a two-branch perception model is constructed. The gaze monitoring branch uses a global attention mechanism to analyze the overall facial features to obtain global features. The fatigue monitoring branch uses a local attention mechanism to focus on the facial features to obtain local features. The global features and local features are fused using a self-attention weighted algorithm to obtain the facial perception result.

[0129] Acquire real-time images of the driver's torso and the surrounding environment, obtain the driver's current action through an action recognition model, obtain objects around the driver through an object detection model, and perform correlation analysis between the action and the surrounding objects based on a multimodal recognition word vector analysis method to obtain behavior perception results;

[0130] fusing the facial perception result and the behavioral perception result to obtain driving status data;

[0131] Acquire a real-time image of a traffic light on an external road, and classify the traffic light status of the real-time image of the traffic light to obtain traffic signal data;

[0132] Get the current vehicle driving status and vehicle start and stop data;

[0133] generating an assisted driving instruction based on a Q-Learning algorithm in combination with the driving state data, the traffic signal data, and the vehicle start and stop data;

[0134] Providing voice prompts or warnings based on the assisted driving instructions generated by the parsing host;

[0135] The monitored driving status data, traffic signal data and vehicle start and stop data are uploaded, and control instructions from a remote server are received.

[0136] Example 2

[0137] Before adopting the engineering vehicle safe driving monitoring system proposed in the present invention, a certain company had been using monitoring system A based on Dlib facial key point detection and YOLO v5 target detection. Monitoring system A has the problem of low recognition rate for fatigue driving and distracted driving, and monitoring system A will identify some driving behaviors as irregular driving, resulting in false alarms in the system. In addition, monitoring system A only alerts the driver's driving behavior and cannot give specific driving instructions based on specific scenarios. In order to solve these problems, the company is currently using the engineering vehicle safe driving monitoring system B proposed in the present invention based on the fusion of facial and behavior recognition to improve the accuracy of monitoring and the driver experience.

[0138] Table 1. Comparison of recognition rates of fatigue driving and distracted driving before and after the application of monitoring system B

[0139] index Monitoring System A Monitoring System B Fatigue driving recognition rate 93.5％ 98.2％ Distracted driving recognition rate 92.2％ 97.6％

[0140] As shown in Table 1, monitoring system A's recognition rates for fatigue driving and distracted driving are lower than those of the currently used monitoring system B, with a 4.7% difference in fatigue driving recognition and a 5.4% difference in distracted driving recognition. This is primarily due to the fact that monitoring system B's facial perception unit employs a dual-branch perception model, extracting features simultaneously from both the global and local levels. The gaze monitoring branch, based on a global attention mechanism, focuses on the overall head contour and movement, which facilitates the model's recognition of distracted driving. The fatigue monitoring branch, based on a local attention mechanism, focuses on the mouth and eyes, further improving the accuracy of fatigue driving recognition. The features acquired from these two branches are fused using a self-attention weighted algorithm, resulting in more accurate facial perception results.

[0141] Table 2. Comparison of the number of alarms before and after the application of a safety driving monitoring system for engineering vehicles based on the fusion of facial and behavioral recognition

[0142]

[0143] Furthermore, as shown in Table 2, monitoring system B significantly outperforms monitoring system A in recognizing irregular driving behaviors. The recognition rates for three specific irregular driving behaviors—smoking, drinking, and talking on the phone—were 98.3%, 97.7%, and 96.8%, respectively, exceeding those of monitoring system A by 6.8%, 7.1%, and 4.6%. Furthermore, although both monitoring systems A and B had zero recognition rates for wearing glasses, twirling hair, and wiping lips, monitoring system B's object detection module failed to detect objects associated with these actions, thus triggering no alarms and reducing the frequency of false alarms. This result is primarily due to monitoring system B's simultaneous recognition and detection of torso movements and the surrounding environment within its behavior perception unit. Furthermore, it associates torso movements with objects in the environment using word embeddings, thereby providing more accurate driving behavior monitoring.

[0144]

[0145] As shown in Table 3, Monitoring System B generates corresponding assisted driving instructions based on driving status data, traffic signal data, and vehicle start-stop data. It first determines whether the driver is in an abnormal driving state (such as fatigue or distraction) based on the driving status. It then uses reinforcement learning results to further determine whether the driver is in an abnormal driving state. Based on this determination, it issues corresponding assisted driving instructions and alerts. Compared to Monitoring System A, Monitoring System B can intelligently determine whether abnormal driving is permitted under the corresponding traffic signal and vehicle start-stop conditions based on reinforcement learning results. This reduces the frequency of false alarms and improves the driver's operating experience.

[0146] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A construction vehicle safety driving monitoring system based on the fusion of facial and behavioral recognition, characterized by: include: The facial perception module is used to obtain real-time images of the driver's face and construct a two-branch perception model. The gaze monitoring branch uses a global attention mechanism to analyze overall facial features to obtain global features, including facial contour features and facial orientation features. The fatigue monitoring branch uses a local attention mechanism to focus on facial features to obtain local features, including eye features and mouth features. The global features and local features are fused using a self-attention weighted algorithm to obtain facial perception results. The gaze monitoring branch includes a downsampling layer and four non-local residual blocks connected in sequence, and the non-local residual block includes three residual blocks connected in sequence, a maximum pooling layer and a non-local attention module; The fatigue monitoring branch includes five sequentially connected local direct convolution blocks, each of which includes three sequentially connected 3×3 convolution layers, a maximum pooling layer, and a local attention module; The self-attention weighted algorithm formula is: F fused =F global +α·V local ; Among them, F fused Indicates comprehensive facial features, F global represents the global feature, α represents the attention weight, V local represents the local eigenvalue; The non-local attention module is a non-local self-attention module, and the local attention module is a SE attention module; A behavior perception module is used to obtain real-time images of the driver's torso and the environment, obtain the driver's current action through an action recognition model, obtain objects around the driver through an object detection model, and perform correlation analysis between the action and the surrounding objects based on a multimodal recognition word vector analysis method to obtain behavior perception results; The face and behavior fusion module is used to fuse the face perception results and the behavior perception results to obtain driving status data; the face and behavior fusion module adopts a weighted voting mechanism for fusion, and the calculation formula of the weighted voting mechanism is: C combined =W face ×C face +W behavior ×C behavior ; Among them, C combined represents the perception fusion score, W face represents the facial perception weight, C face represents the face perception confidence, W behavior represents the behavior perception weight, C behavior Indicates the confidence of behavior perception; A traffic light recognition module is used to obtain a real-time image of traffic lights on an external road, and classify the traffic light status of the real-time image of the traffic light to obtain traffic signal data; The start-stop sensing module is used to obtain the current vehicle driving status and vehicle start-stop data; An algorithm parsing module is used to connect the facial and behavior fusion module, the traffic light recognition module, and the start-stop perception module, and generate assisted driving instructions based on the Q-Learning algorithm in combination with the driving status data, the traffic signal data, and the vehicle start-stop data.

2. The engineering vehicle safe driving monitoring system based on the fusion of facial and behavioral recognition according to claim 1 is characterized in that: The facial perception results include normal driving, distracted driving and fatigue driving, and the behavior perception results include standard driving and irregular driving, and the irregular driving includes at least one of smoking, drinking water or making phone calls.

3. The engineering vehicle safe driving monitoring system based on the fusion of facial and behavioral recognition according to claim 1 is characterized in that: The facial perception module includes: a sight line monitoring unit for monitoring whether the driver's sight is deviated from the road ahead; a fatigue monitoring unit for monitoring the driver's fatigue state; obtaining the facial perception result includes: Using a first camera to obtain a real-time facial image of the driver; identifying, based on the gaze monitoring unit, whether the real-time facial image shows a face turning state, and if so, recording the face turning angle and turning time; and determining distracted driving if the face turning angle is greater than a turning angle threshold and the turning time is greater than a turning time threshold; Based on the fatigue monitoring unit, it is identified whether the real-time facial image is in a closed-eye state or a yawning state. If it is in a closed-eye state, the closed-eye time and the closed-eye frequency are recorded. If the closed-eye time is greater than the closed-eye time threshold or the closed-eye frequency is greater than the closed-eye frequency threshold, it is determined to be fatigue driving; if it is in a yawning state, the yawning frequency is recorded. If the yawning frequency is greater than the yawning frequency threshold, it is determined to be fatigue driving.

4. The engineering vehicle safe driving monitoring system based on the fusion of facial and behavioral recognition according to claim 1 is characterized in that: The behavior perception module includes a behavior recognition unit, a target detection unit and a correction unit; Obtaining the behavior perception result includes: A second camera is used to obtain a real-time image of the driver's torso, and a third camera is used to obtain a real-time image of the driver's surrounding environment; The behavior recognition unit processes the real-time torso image, identifies the driver's behavior by extracting key points of the driver, and converts the recognition result into a behavior word vector; The target detection unit analyzes the real-time image of the environment, identifies the objects in the image, and converts the recognition results into object word vectors; Expand the five related word vectors of the behavior word vector through the ontology library and compare them with the object word vector to obtain a comparison result; The correction unit determines whether the comparison result is correct. If the comparison result is correct, the behavior recognition is confirmed as an irregular operation; if the comparison result is incorrect, the behavior recognition result is corrected.

5. The engineering vehicle safe driving monitoring system based on the fusion of facial and behavioral recognition according to claim 1 is characterized in that: The signal light status includes: red light status, reminding the driver to stop; yellow light status, reminding the driver to slow down or prepare to stop; green light status, reminding the driver to pass safely; no signal light status, reminding the driver to drive carefully and make judgments based on the actual road conditions.

6. The engineering vehicle safe driving monitoring system based on the fusion of facial and behavioral recognition according to claim 1 is characterized in that: The vehicle start-stop data is acquired by an acceleration sensor, and the vehicle start-stop data includes a start-stop state, a steering state, an acceleration state, and a current speed.

7. The engineering vehicle safe driving monitoring system based on the fusion of facial and behavioral recognition according to claim 1 is characterized in that: The algorithm parsing module includes: a data processing unit for preprocessing data transmitted by the face and behavior fusion module, the traffic light recognition module, and the start-stop perception module to obtain preprocessed data; a decision algorithm unit for performing reinforcement learning on the preprocessed data using the Q-Learning algorithm according to preset driving safety rules and road traffic regulations, and generating assisted driving instructions; generating the assisted driving instructions includes: Determine the state space, action space, and reward function, define the combination of driving behavior, traffic signals, and vehicle status, and set driving instructions and reward mechanisms based on the combination; Build a driving simulation environment that provides reward information based on real-time feedback of the current state and execution actions; Initialize the Q table and select actions through the ε-greedy strategy, observe the environment feedback to update the Q value, and store the Q value in the Q table; Monitor the status during actual driving, use the Q table to query the best action and generate corresponding driving instructions; Regularly evaluate model performance, monitor safety and efficiency through testing, and adjust parameters to optimize learning results.

8. The engineering vehicle safe driving monitoring system based on the fusion of facial and behavioral recognition according to claim 1 is characterized in that: The driving monitoring system further includes: A voice broadcast module, connected to the algorithm analysis module, for providing voice prompts or warnings according to the assisted driving instructions generated by the algorithm analysis module; A communication module is used to communicate with a remote server or external device, upload the monitored driving status data, the traffic signal data, and the vehicle start and stop data, and receive control instructions from the remote server. The communication module supports wireless communication protocols, including at least one or more of 4G, 5G, and Wi-Fi wireless communication technologies, and is used to achieve two-way data transmission with the remote server.

Images