Engineering truck safe driving monitoring system based on face and behavior recognition fusion

Abstract

The invention relates to the technical field of monitoring systems, in particular to an engineering vehicle safe driving monitoring system based on face and behavior recognition fusion. Firstly, a double-branch perception model is provided for face perception, and overall face features and local detail features of a driver are respectively captured through global and local attention mechanisms, so that the recognition accuracy of fatigue driving and distraction driving is improved. Secondly, the invention provides a multi-modal recognition word vector analysis method used for behavior perception, different models are used for recognizing images of the trunk and the surrounding environment of the driver in real time, and behavior word vectors are compared with object word vectors to judge whether the driver has non-standard operation or not. And finally, integration and learning are performed on various data through a Q-Learning algorithm, and an accurate auxiliary driving instruction is generated through reinforcement learning, so that the use experience of the driver is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of monitoring systems, and in particular to a safe driving monitoring system for engineering vehicles based on the fusion of face and behavior recognition. Background Art

[0002] Construction vehicles play a vital role in transportation and urban construction. These vehicles usually travel for long periods of time on construction sites and roads, carrying out tasks such as heavy-load transportation and construction. Due to their large size and heavy load, the consequences of traffic accidents are often more serious, which may lead to casualties and property losses. Therefore, it is particularly important to detect the safe driving of construction vehicles to ensure that drivers remain highly alert during driving and reduce the probability of accidents.

[0003] At present, the safe driving detection of engineering vehicles is mainly based on image processing, machine learning and sensor technology. These technologies can timely identify potential dangerous behaviors such as fatigue driving and distracted driving by monitoring the driver's behavior and vehicle status in real time. The use of these advanced technologies can improve the safety of engineering vehicles and reduce the risk of accidents. At the same time, it also helps to improve the safety awareness of drivers and improve the working environment.

[0004] However, there are still some problems with the current safety driving detection of engineering vehicles. First, the level of precision of action recognition is not enough. For example, the driver's hair-twisting action is misjudged as making a phone call, or the action of wiping lips is recognized as smoking, which will cause unnecessary alarms to be triggered frequently. Secondly, the system will also frequently issue alarms during the driver's rest period, which not only distracts the driver's attention, but also may reduce their willingness to use the system, thereby affecting their enthusiasm for safe driving.

[0005] Therefore, a safe driving monitoring system for engineering vehicles based on the fusion of face and behavior recognition is proposed. Summary of the invention

[0006] The purpose of the present invention is to provide a safe driving monitoring system for engineering vehicles based on the fusion of face and behavior recognition. First, the present invention proposes a dual-branch perception model for face perception, which captures the overall and local features of the driver's face through global and local attention mechanisms, respectively, so as to improve the recognition accuracy of fatigue and distracted driving. Secondly, the present invention proposes a multimodal recognition word vector analysis method, which generates behavior word vectors and object word vectors through real-time image recognition of the driver's torso and the surrounding environment, and compares them to determine whether there is any irregular operation. Finally, the present invention integrates and learns multiple data through the Q-Learning algorithm, optimizes the driving decision-making process, reduces the false alarm rate, and improves driving safety and user experience.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] A safe driving monitoring system for engineering vehicles based on the fusion of facial and behavioral recognition, comprising:

[0009] The facial perception module is used to obtain the real-time image of the driver's face and build a dual-branch perception model, in which the sight monitoring branch analyzes the overall facial features through the global attention mechanism to obtain global features; the fatigue monitoring branch focuses on the facial features through the local attention mechanism to obtain local features; the global features and the local features are fused through the self-attention weighted algorithm to obtain the facial perception result;

[0010] A behavior perception module is used to obtain a real-time image of the driver's torso and the environment, obtain the driver's current execution action through an action recognition model, obtain the driver's surrounding objects through a target detection model, and perform a correlation analysis on the execution action and the surrounding objects based on a multimodal recognition word vector analysis method to obtain a behavior perception result;

[0011] A face and behavior fusion module, used for fusing the face perception result and the behavior perception result to obtain driving status data;

[0012] A traffic light recognition module is used to obtain a real-time image of traffic lights on external roads, and classify the states of the traffic lights in the real-time image to obtain traffic signal data;

[0013] The start-stop sensing module is used to obtain the current vehicle driving status and vehicle start-stop data;

[0014] The algorithm analysis module is used to connect the facial and behavior fusion module, the traffic light recognition module and the start-stop perception module, and generate an assisted driving instruction based on the Q-Learning algorithm in combination with the driving status data, the traffic signal data and the vehicle start-stop data.

[0015] Furthermore, the facial perception results include normal driving, distracted driving and fatigue driving, and the behavior perception results include standard driving and irregular driving, and the irregular driving includes at least one of smoking, drinking water or making phone calls.

[0016] Furthermore, the facial perception module includes: a sight line monitoring unit for monitoring whether the driver's sight line deviates from the road ahead; a fatigue monitoring unit for monitoring the driver's fatigue state; and obtaining the facial perception result includes:

[0017] Using the first camera to obtain a real-time facial image of the driver;

[0018] Based on the sight monitoring unit, identifying whether the real-time facial image is in a face turning state, and if so, recording the face turning angle and turning time; if the face turning angle is greater than a turning angle threshold, and the turning time is greater than a turning time threshold, determining that it is distracted driving;

[0019] Based on the fatigue monitoring unit, it is identified whether the real-time facial image is in the eyes-closed state or the yawning state. If it is in the eyes-closed state, the eyes-closed time and the eyes-closed frequency are recorded. If the eyes-closed time is greater than the eyes-closed time threshold or the eyes-closed frequency is greater than the eyes-closed frequency threshold, it is determined to be fatigue driving; if it is in the yawning state, the yawning frequency is recorded. If the yawning frequency is greater than the yawning frequency threshold, it is determined to be fatigue driving.

[0020] Furthermore, the dual-branch perception model includes:

[0021] An input layer, used to receive a real-time image of the driver's face acquired by the first camera;

[0022] The gaze monitoring branch is used to analyze the overall facial features through a global attention mechanism to obtain global features, where the global features include facial contour features and facial turning features;

[0023] The fatigue monitoring branch is used to focus on the facial features through the local attention mechanism to obtain local features, including eye features and mouth features;

[0024] A branch fusion layer, used for fusing the global features and the local features through a self-attention weighted algorithm to generate a comprehensive facial feature;

[0025] A classification layer, used to classify the comprehensive facial features and output facial perception results;

[0026] The input layer is a 3×3 convolutional layer;

[0027] The line of sight monitoring branch includes a downsampling layer and four non-local residual blocks connected in sequence, and the non-local residual block includes three residual blocks connected in sequence, a maximum pooling layer and a non-local attention module;

[0028] The fatigue monitoring branch includes five sequentially connected local direct convolution blocks, and the local direct convolution block includes three sequentially connected 3×3 convolution layers, a maximum pooling layer and a local attention module;

[0029] The branch fusion layer fuses the global features and the local features through a self-attention weighted algorithm, and the formula of the self-attention weighted algorithm is:

[0030] F fused =F global +α·V local ;

[0031] Among them, F fused Indicates comprehensive facial features, F global represents the global feature, α represents the attention weight, V local represents the local eigenvalue;

[0032] The classification layer classifies the fused features through a softmax function.

[0033] Furthermore, the behavior perception module includes a behavior recognition unit, a target detection unit and a correction unit; obtaining the behavior perception result includes:

[0034] Using the second camera to obtain a real-time image of the driver's torso, and using the third camera to obtain a real-time image of the environment around the driver;

[0035] The behavior recognition unit processes the real-time image of the torso, identifies the driver's behavior by extracting the key points of the driver, and converts the recognition result into a behavior word vector;

[0036] The target detection unit analyzes the real-time image of the environment, identifies the objects in the image, and converts the recognition results into object word vectors;

[0037] Expanding 5 related word vectors of the behavior word vector through the ontology library, and comparing them with the object word vector to obtain a comparison result;

[0038] The correction unit determines whether the comparison result is correct. If the comparison result is correct, the behavior recognition is confirmed as an irregular operation; if the comparison result is incorrect, the behavior recognition result is corrected.

[0039] Furthermore, the face and behavior fusion module adopts a weighted voting mechanism for fusion, and the calculation formula of the weighted voting mechanism is:

[0040] C combined =W face ×C face +W behavior ×C behavior ;

[0041] Among them, C combined represents the perceptual fusion score, W face represents the face perception weight, C face represents the face perception confidence, W behavior represents the behavior perception weight, C behavior Indicates the confidence of behavior perception.

[0042] Furthermore, the signal light status includes: a red light status, reminding the driver to stop; a yellow light status, reminding the driver to slow down or prepare to stop; a green light status, reminding the driver to pass safely; and a signal light not detected status, reminding the driver to drive carefully and make judgments based on actual road conditions.

[0043] Furthermore, the vehicle start-stop data is acquired by an acceleration sensor, and the vehicle start-stop data includes a start-stop state, a steering state, an acceleration state and a current speed.

[0044] Furthermore, the algorithm analysis module includes: a data processing unit, which is used to pre-process the data transmitted by the face and behavior fusion module, the signal light recognition module and the start-stop perception module to obtain pre-processed data; a decision algorithm unit, which is used to perform reinforcement learning on the pre-processed data using the Q-Learning algorithm according to preset driving safety rules and road traffic regulations, and generate assisted driving instructions; generating the assisted driving instructions includes:

[0045] Determine the state space, action space and reward function, define the combination of driving behavior, traffic signal and vehicle state, and set driving instructions and reward mechanism according to the combination;

[0046] Build a driving simulation environment that provides reward information based on real-time feedback of the current state and execution actions;

[0047] Initialize the Q table and select actions through the ε-greedy strategy, observe the environmental feedback to update the Q value, and store the Q value in the Q table;

[0048] Monitor the status during actual driving, use the Q table to query the best action and generate corresponding driving instructions;

[0049] Regularly evaluate model performance, monitor safety and efficiency through testing, and adjust parameters to optimize learning results.

[0050] Furthermore, the driving monitoring system further comprises:

[0051] A voice broadcast module, used to connect to the algorithm analysis module and provide voice prompts or warnings according to the auxiliary driving instructions generated by the algorithm analysis module;

[0052] A communication module is used to communicate with a remote server or an external device, upload the monitored driving status data, the traffic signal data and the vehicle start and stop data, and receive control instructions from the remote server. The communication module supports wireless communication protocols, including at least one or more of 4G, 5G, and Wi-Fi wireless communication technologies, and is used to realize two-way data transmission with a remote server.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] 1. This paper proposes a dual-branch perception model that analyzes the driver's face as a whole through a global attention mechanism to capture global features and identify distracted driving in a timely manner; at the same time, the local attention mechanism focuses on capturing local features, such as eye status and mouth movements, to deeply identify fatigue status. By combining these two mechanisms with a self-attention weighted algorithm, the model can perceive the face in a more fine-grained manner, improve the recognition accuracy of fatigue driving and distracted driving, and thus provide reliable technical support for real-time driving safety monitoring.

[0055] 2. The present invention proposes a multimodal recognition word vector analysis method, which uses different models to recognize the real-time image of the driver's torso and the real-time image of the environment respectively, and converts the recognition results into behavior word vectors and object word vectors, and then uses the ontology library to expand the related word vectors of the behavior word vectors, and compares them with the object word vectors to determine whether the driver has irregular operations. Compared with the prior art, this method can associate the behavior recognition results with the target detection results through the comparison between word vectors, thereby reducing the misjudgment rate and false alarm rate, and improving the accuracy and reliability of the monitoring system.

[0056] 3. The present invention uses the Q-Learning algorithm to perform reinforcement learning on driving status data, traffic signal data, and vehicle start-stop data to reduce the frequency of false alarms. By continuously iterating and updating the Q value, the system can identify and learn the driver's behavior patterns in different situations, thereby optimizing the alarm triggering conditions and reducing unnecessary alarms caused by misjudgments. This intelligent decision-making process not only improves the driver's experience, but also enhances the reliability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A schematic diagram of the structure of a safe driving monitoring system for engineering vehicles based on the fusion of face and behavior recognition provided by an embodiment of the present invention;

[0058] Figure 2 A schematic diagram of the dual-branch perception model structure provided by an embodiment of the present invention;

[0059] Figure 3 A schematic diagram of a non-local residual block structure provided by an embodiment of the present invention;

[0060] Figure 4 A schematic diagram of the structure of a local direct convolution block provided in an embodiment of the present invention;

[0061] Figure 5 A schematic diagram of a basic residual block structure provided by an embodiment of the present invention;

[0062] Figure 6An execution flow chart of an engineering vehicle safe driving monitoring system based on the fusion of face and behavior recognition provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0063] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] Embodiment 1

[0065] A safe driving monitoring system for engineering vehicles based on the fusion of facial and behavior recognition, such as Figure 1 As shown, including:

[0066] The facial perception module is used to obtain the real-time image of the driver's face and build a dual-branch perception model, in which the sight monitoring branch analyzes the overall facial features through the global attention mechanism to obtain global features; the fatigue monitoring branch focuses on the facial features through the local attention mechanism to obtain local features; the global features and the local features are fused through the self-attention weighted algorithm to obtain the facial perception result;

[0067] A behavior perception module is used to obtain a real-time image of the driver's torso and the environment, obtain the driver's current execution action through an action recognition model, obtain the driver's surrounding objects through a target detection model, and perform a correlation analysis on the execution action and the surrounding objects based on a multimodal recognition word vector analysis method to obtain a behavior perception result;

[0068] A face and behavior fusion module, used for fusing the face perception result and the behavior perception result to obtain driving status data;

[0069] A traffic light recognition module is used to obtain a real-time image of traffic lights on external roads, and classify the states of the traffic lights in the real-time image to obtain traffic signal data;

[0070] The start-stop sensing module is used to obtain the current vehicle driving status and vehicle start-stop data;

[0071] The algorithm analysis module is used to connect the facial and behavior fusion module, the traffic light recognition module and the start-stop perception module, and generate an assisted driving instruction based on the Q-Learning algorithm in combination with the driving status data, the traffic signal data and the vehicle start-stop data.

[0072] Furthermore, the facial perception results include normal driving, distracted driving and fatigue driving, and the behavior perception results include standard driving and irregular driving, and the irregular driving includes at least one of smoking, drinking water or making phone calls.

[0073] The facial perception results include normal driving, distracted driving and fatigue driving, which can effectively distinguish the driver's attention and fatigue; the behavior perception results determine whether the driving is standard, among which irregular driving behaviors at least include smoking, drinking water and making phone calls. By combining these perception results, the system can more accurately identify and warn potential driving risks, thereby improving driving safety and reducing the occurrence of false alarms.

[0074] Furthermore, the facial perception module includes: a sight line monitoring unit for monitoring whether the driver's sight line deviates from the road ahead; a fatigue monitoring unit for monitoring the driver's fatigue state; and obtaining the facial perception result includes:

[0075] Using the first camera to obtain a real-time facial image of the driver;

[0076] Based on the sight monitoring unit, identifying whether the real-time facial image is in a face turning state, and if so, recording the face turning angle and turning time; if the face turning angle is greater than a turning angle threshold, and the turning time is greater than a turning time threshold, determining that it is distracted driving;

[0077] Based on the fatigue monitoring unit, it is identified whether the real-time facial image is in the eyes-closed state or the yawning state. If it is in the eyes-closed state, the eyes-closed time and the eyes-closed frequency are recorded. If the eyes-closed time is greater than the eyes-closed time threshold or the eyes-closed frequency is greater than the eyes-closed frequency threshold, it is determined to be fatigue driving; if it is in the yawning state, the yawning frequency is recorded. If the yawning frequency is greater than the yawning frequency threshold, it is determined to be fatigue driving.

[0078] In a feasible implementation scheme, facial turning includes looking up, looking down, turning to the left and turning to the right, wherein the turning angle thresholds for looking up, looking down, turning to the left and turning to the right are [30°, 30°, 30°, 45°] respectively, the eye closure time threshold is 3 seconds, the eye closure frequency threshold is 15 times / minute, and the yawning frequency threshold is 3 times / minute.

[0079] Through the coordinated work of the sight monitoring unit and the fatigue monitoring unit, the driver's distraction and fatigue status can be accurately judged. Through the real-time facial image of the driver, the facial perception module can identify key behavioral characteristics such as facial turning, closing eyes and yawning, and make judgments based on the set angle, time and frequency thresholds, reducing the frequency of alarms caused by checking the left and right rearview mirrors or personal eye or mouth habits, and improving the humanization of the safe driving monitoring system.

[0080] Furthermore, the dual-branch perception model is as follows Figure 2 As shown, including:

[0081] An input layer, used to receive a real-time image of the driver's face acquired by the first camera;

[0082] The gaze monitoring branch is used to analyze the overall facial features through a global attention mechanism to obtain global features, where the global features include facial contour features and facial turning features;

[0083] The fatigue monitoring branch is used to focus on the facial features through the local attention mechanism to obtain local features, including eye features and mouth features;

[0084] A branch fusion layer, used for fusing the global features and the local features through a self-attention weighted algorithm to generate a comprehensive facial feature;

[0085] A classification layer, used to classify the comprehensive facial features and output facial perception results;

[0086] The input layer is a 3×3 convolutional layer;

[0087] The line of sight monitoring branch includes a downsampling layer and four non-local residual blocks connected in sequence, wherein the non-local residual blocks are as follows: Figure 3 As shown, it includes three residual blocks connected in sequence, a maximum pooling layer and a non-local attention module;

[0088] The fatigue monitoring branch includes five local direct convolution blocks connected in sequence, and the local direct convolution blocks are as follows: Figure 4 As shown, it includes three 3×3 convolutional layers, a maximum pooling layer and a local attention module connected in sequence;

[0089] The branch fusion layer fuses the global features and the local features through a self-attention weighted algorithm, and the formula of the self-attention weighted algorithm is:

[0090] F fused =F global +α·V local ;

[0091] Among them, F fused Indicates comprehensive facial features, F global represents the global feature, α represents the attention weight, V local represents the local eigenvalue;

[0092] The classification layer classifies the fused features through a softmax function.

[0093] Specifically, the number of channels in the input layer is 64, the number of channels of a downsampling layer and four non-local residual blocks connected in sequence to the sight monitoring branch are [64, 64, 128, 256, 512] respectively, and the number of channels of five local direct convolution blocks connected in sequence to the fatigue monitoring branch are [64, 128, 256, 512, 512] respectively; the residual block is the basic residual block of ResNet-34, and its structure is as follows Figure 5 As shown; the non-local attention module is a non-local self-attention module, and the local attention module is a SE attention module; the self-attention weighted algorithm formula is: F fused =F global +α·V local , Q global =W q ·F global , K local =W k ·F local , V local =W v ·F local , where F fused Indicates comprehensive facial features, F global represents the global feature, F local represents the local feature, α represents the attention weight, V local represents the local eigenvalue, Q global represents the global feature query mapping matrix, K local represents the local feature key mapping matrix, softmax() represents the activation function, d k represents the scaling factor, W q represents the query mapping matrix, W k represents the key mapping matrix, W v Represents the value mapping matrix.

[0094] The dual-branch perception model achieves accurate recognition of the driver's facial features through a dual-branch structure. The sight monitoring branch captures the overall features of the face, such as facial contours and head turning, through a global attention mechanism to ensure that the system can focus on the overall dynamics of the driver. The fatigue monitoring branch uses a local attention mechanism to focus on capturing local facial features, especially key details of the eyes and mouth, such as fatigue signals such as closing eyes and yawning. Through the cooperation of these two branches, the system simultaneously extracts features at the global and local levels, and achieves more accurate driving status judgment after fusion, thereby improving the detection accuracy of fatigue driving and distracted driving.

[0095] Furthermore, the behavior perception module includes a behavior recognition unit, a target detection unit and a correction unit; obtaining the behavior perception result includes:

[0096] Using the second camera to obtain a real-time image of the driver's torso, and using the third camera to obtain a real-time image of the environment around the driver;

[0097] The behavior recognition unit processes the real-time image of the torso, identifies the driver's behavior by extracting the key points of the driver, and converts the recognition result into a behavior word vector;

[0098] The target detection unit analyzes the real-time image of the environment, identifies the objects in the image, and converts the recognition results into object word vectors;

[0099] Expanding 5 related word vectors of the behavior word vector through the ontology library, and comparing them with the object word vector to obtain a comparison result;

[0100] The correction unit determines whether the comparison result is correct. If the comparison result is correct, the behavior recognition is confirmed as an irregular operation; if the comparison result is incorrect, the behavior recognition result is corrected.

[0101] Specifically, the behavior recognition unit is implemented by an action recognition model, which may be an I3D, C3D or SlowFast model, and the target detection unit is implemented by a target detection model, preferably a YOLO v5 model. The conversion of behavior recognition results into behavior word vectors and the conversion of target detection results into object word vectors are both implemented through Word2Vec, and the ontology library expansion behavior word vectors are implemented through WordNet. In a feasible implementation scheme, the action "making a phone call" is converted into a word vector V0 by the Word2Vec model, and the five related word vectors expanded by the ontology library are [V1, V2, V3, V4, V5], corresponding to "telephone", "mobile phone", "pager", "fax" and "mail" respectively. The word vector corresponding to the object "mobile phone" is V2. At this time, the object word vector is in the related word vector, and the comparison result is correct, and it is determined that the current driver is smoking; in another feasible implementation scheme, the action "smoking" is converted into a word vector W0 by the Word2Vec model, and the five related word vectors expanded by the ontology library are [W1, W2, W3, W4, W5], corresponding to "cigarette", "cigar", "ash", "tar smoke" and "smoke" respectively. The target detection unit did not detect the object held by the hand, and the corresponding result is NONE. The object word vector is not in the related word vector, and the comparison result is incorrect, so the behavior recognition result is corrected.

[0102] The multimodal recognition word vector analysis method improves the recognition accuracy of irregular operations by combining the driver's behavior recognition and environmental perception. It obtains the real-time image of the driver's torso through the second camera, extracts key points to recognize driving actions, and generates a behavior word vector. At the same time, the third camera captures the driver's surroundings, identifies related objects and generates an object word vector. After the behavior word vector is expanded through the ontology library, it is compared with the object word vector to determine the correlation between the action and the object, ensuring more accurate recognition of irregular operations, thereby reducing false alarms and improving the reliability of the system.

[0103] Furthermore, the perception fusion unit adopts a weighted voting mechanism for fusion, and the weighted voting calculation formula is:

[0104] C combined =W face ×C face +W behavior ×C behavior ;

[0105] Among them, C combined represents the perceptual fusion score, W face represents the face perception weight, C face represents the face perception confidence, W behavior represents the behavior perception weight, C behavior Indicates the confidence of behavior perception.

[0106] The recognition results of the facial perception unit and the behavior perception unit are integrated through a weighted voting mechanism, and the driver's status is comprehensively judged based on the weighted voting calculation, ensuring that the contribution ratio of each perception module can be flexibly adjusted. This not only improves the accuracy of the recognition results, but also reduces the impact of misjudgment of a single module, thereby improving the overall stability and intelligence of the system.

[0107] Furthermore, the signal light status includes: a red light status, reminding the driver to stop; a yellow light status, reminding the driver to slow down or prepare to stop; a green light status, reminding the driver to pass safely; and a signal light not detected status, reminding the driver to drive carefully and make judgments based on actual road conditions.

[0108] By monitoring the status of traffic lights in real time, it provides accurate reminders for different states such as red, yellow, and green lights, helping drivers to quickly make correct decisions in complex traffic environments. When no traffic lights are detected, the system reminds the driver to drive carefully and makes independent judgments based on the actual road conditions, effectively enhancing driving safety and flexibility.

[0109] Furthermore, the vehicle start-stop data is acquired by an acceleration sensor, and the vehicle start-stop data includes a start-stop state, a steering state, an acceleration state and a current speed.

[0110] Specifically, the start-stop state includes starting and stopping, the turning state includes left turning, right turning and straight driving, and the acceleration state includes acceleration, deceleration and constant speed.

[0111] The acceleration sensor acquires data such as the vehicle's start / stop status, steering status, acceleration status, and current speed to accurately monitor the vehicle's dynamic behavior. These start / stop data help optimize the generation of driving instructions in a variety of driving scenarios, improving driving safety and response efficiency.

[0112] Furthermore, the algorithm analysis module includes: a data processing unit, which is used to pre-process the data transmitted by the face and behavior fusion module, the signal light recognition module and the start-stop perception module to obtain pre-processed data; a decision algorithm unit, which is used to perform reinforcement learning on the pre-processed data using the Q-Learning algorithm according to preset driving safety rules and road traffic regulations, and generate assisted driving instructions; generating the assisted driving instructions includes:

[0113] Determine the state space, action space and reward function, define the combination of driving behavior, traffic signal and vehicle state, and set driving instructions and reward mechanism according to the combination;

[0114] Build a driving simulation environment that provides reward information based on real-time feedback of the current state and execution actions;

[0115] Initialize the Q table and select actions through the ε-greedy strategy, observe the environmental feedback to update the Q value, and store the Q value in the Q table;

[0116] Monitor the status during actual driving, use the Q table to query the best action and generate corresponding driving instructions;

[0117] Regularly evaluate model performance, monitor safety and efficiency through testing, and adjust parameters to optimize learning results.

[0118] Specifically, the state space includes driving state data (smoking, fatigue or distraction, etc.), traffic signal data (red light, green light, yellow light or no signal light) and vehicle start and stop data (start, turn, accelerate or decelerate, etc.), the action space is the current action of the vehicle, including acceleration, deceleration, braking, stopping or turning, etc. The reward function is set according to driving safety rules and road traffic regulations, including positive rewards (safely passing the green light, smoothly accelerating or focusing on driving) and negative rewards (sudden braking, running a red light or fatigue driving); build a driving simulation environment for training the Q-Learning model, which should be able to provide real-time feedback on the current state, executed actions and rewards obtained. The current state includes driving state data, traffic signal data and vehicle start and stop data; create a Q table, with rows representing states and columns representing actions, initialize all Q values ​​to 0, and at each time step t, according to the current state S t Select Action A t , when executing action A t , observe the next state S of the environmental feedback t+1 and the reward R t And update the Q value, where the Q value update algorithm is:

[0119]

[0120] Among them, Q(S t ,A t ) indicates that in state S t Select Action A t The expected cumulative reward that can be obtained after α represents the learning rate, ranging from 0≤α≤1, R t Current Status t Execute action A t The immediate reward obtained after γ represents the discount factor, ranging from 0≤γ≤1. Indicates the maximum expected reward after taking the best action in the next state; in the actual driving process, real-time monitoring of driving behavior, traffic signals and vehicle status forms the current state S t , query the Q table according to the current state, select the action with the highest Q value as the driving instruction, convert the selected action into actual driving instructions, and feed back to the vehicle control system; by testing in different scenarios, evaluate the decision-making effect of the model, monitor indicators such as safety and driving efficiency, and adjust parameters such as reward function, learning rate, discount factor, etc. according to the evaluation results to optimize the learning effect.

[0121] The algorithm analysis module pre-processes the data transmitted by the face and behavior fusion module, the signal light recognition module, and the start-stop perception module through the data processing unit to ensure the efficient integration and accuracy of the input data. Combined with the Q-Learning algorithm of the decision algorithm module, the system can perform reinforcement learning based on driving safety rules and road traffic regulations to intelligently optimize driving decisions. This solution not only improves the accuracy of assisted driving, but also continuously adapts to complex road environments and driving behaviors through self-learning, thereby improving driving safety and the driver's operating experience.

[0122] The engineering vehicle safe driving monitoring system of the present invention integrates the facial and behavior fusion module, the signal light recognition monitoring module, the start-stop perception module and the algorithm analysis module to realize comprehensive real-time monitoring of the driver's status, traffic signals and vehicle driving conditions, and uses the Q-Learning algorithm for reinforcement learning to generate accurate auxiliary driving instructions. This multi-level data collection and analysis improves driving safety and real-time response capabilities, and enhances the driver's operating experience and trust in the system.

[0123] Furthermore, the driving monitoring system further comprises:

[0124] A voice broadcast module, used to connect to the algorithm analysis module and provide voice prompts or warnings according to the auxiliary driving instructions generated by the algorithm analysis module;

[0125] A communication module is used to communicate with a remote server or an external device, upload the monitored driving status data, the traffic signal data and the vehicle start and stop data, and receive control instructions from the remote server. The communication module supports wireless communication protocols, including at least one or more of 4G, 5G, and Wi-Fi wireless communication technologies, and is used to realize two-way data transmission with a remote server.

[0126] The system converts assisted driving instructions into instant voice prompts or warnings through the voice broadcast module, helping the driver to grasp safety information in a timely manner; through the communication module, it ensures efficient two-way data transmission with remote servers or external devices, not only uploading monitoring data such as driving status, traffic signals, vehicle start and stop, but also receiving remote commands, providing convenient and reliable communication support for remote monitoring and control.

[0127] A safety driving monitoring system for engineering vehicles based on the fusion of face and behavior recognition provided in this embodiment is shown in a flowchart as shown in Figure 6, including:

[0128] Acquire the real-time image of the driver's face and build a dual-branch perception model, in which the sight monitoring branch analyzes the overall facial features through the global attention mechanism to obtain global features; the fatigue monitoring branch focuses on the facial features through the local attention mechanism to obtain local features; the global features and the local features are fused through the self-attention weighted algorithm to obtain the facial perception result;

[0129] Acquire a real-time image of the driver's torso and a real-time image of the environment, acquire the driver's current action through an action recognition model, acquire the driver's surrounding objects through a target detection model, and perform a correlation analysis on the action and the surrounding objects based on a multimodal recognition word vector analysis method to obtain a behavior perception result;

[0130] fusing the facial perception result and the behavior perception result to obtain driving status data;

[0131] Acquire a real-time image of a traffic light on an external road, and classify the states of the traffic lights in the real-time image of the traffic light to obtain traffic signal data;

[0132] Get the current vehicle driving status and vehicle start and stop data;

[0133] Generate an assisted driving instruction based on a Q-Learning algorithm in combination with the driving state data, the traffic signal data and the vehicle start and stop data;

[0134] Providing voice prompts or warnings according to the assisted driving instructions generated by the parsing host;

[0135] The monitored driving status data, traffic signal data and vehicle start and stop data are uploaded, and control instructions from a remote server are received.

[0136] Embodiment 2

[0137] Before adopting the engineering vehicle safe driving monitoring system proposed in the present invention, a certain company had been using monitoring system A based on Dlib face key point detection and YOLO v5 target detection. Monitoring system A has the problem of low recognition rate for fatigue driving and distracted driving, and monitoring system A will identify some driving behaviors as irregular driving, causing the system to falsely report. In addition, monitoring system A only alerts the driver's driving behavior and cannot give specific driving instructions based on specific scenarios. In order to solve these problems, the company is currently adopting an engineering vehicle safe driving monitoring system B based on the fusion of facial and behavioral recognition proposed in the present invention to improve the accuracy of monitoring and the driver experience.

[0138] Table 1. Comparison of recognition rates of fatigue driving and distracted driving before and after the application of monitoring system B

[0139] index Monitoring System A Monitoring system B Fatigue driving recognition rate 93.5％ 98.2％ Distracted driving recognition rate 92.2％ 97.6％

[0140] As shown in Table 1, the recognition rate of fatigue driving and distracted driving by monitoring system A is lower than that of monitoring system B currently used. The difference in fatigue driving recognition rate is 4.7%, and the difference in distracted driving recognition rate is 5.4%. The main reason is that the facial perception unit of monitoring system B adopts a dual-branch perception model, which extracts features from the global and local levels simultaneously in the form of dual branches. The line of sight monitoring branch focuses on the overall contour and movement of the head based on the global attention mechanism, which is conducive to the model's recognition of distracted driving. The fatigue monitoring branch focuses on the mouth and eyes based on the local attention mechanism, thereby improving the recognition accuracy of fatigue driving. The features obtained by the two branches are fused through the self-attention weighted algorithm to achieve more accurate facial perception results.

[0141] Table 2. Comparison of the number of alarms before and after the application of a safe driving monitoring system for engineering vehicles based on the fusion of facial and behavioral recognition

[0142]

[0143] In addition, as can be seen from Table 2, the recognition rate of monitoring system B for irregular driving behaviors is also significantly better than that of monitoring system A. The recognition rates of three specific irregular driving behaviors, smoking, drinking water and making phone calls, are 98.3%, 97.7% and 96.8%, respectively, which are 6.8%, 7.1% and 4.6% higher than those of monitoring system A. At the same time, although the recognition rates of wearing glasses, lifting hair and wiping lips by monitoring system A and monitoring system B are both 0, the target detection module of monitoring system B does not detect objects related to smoking, drinking water and making phone calls, so no alarm is triggered, reducing the frequency of false alarms. The main reason for the above results is that monitoring system B synchronously recognizes and detects the torso action and the surrounding environment in the behavior perception unit, and associates the torso action with the objects in the environment according to the word vector, thereby providing more accurate driving behavior monitoring.

[0144]

[0145] As shown in Table 3, monitoring system B can generate corresponding assisted driving instructions based on driving status data, traffic signal data, and vehicle start-stop data. First, it determines whether it is in an abnormal driving state (fatigue driving, distraction, etc.) through the driving status, and then combines traffic signal data and vehicle start-stop data to make further judgments based on reinforcement learning results, and then gives corresponding assisted driving instructions and alarms based on the judgment results. Compared with monitoring system A, monitoring system B can intelligently determine whether abnormal driving is permitted under the corresponding traffic signals and vehicle start-stop according to reinforcement learning results, thereby reducing the frequency of false alarms and improving the driver's operating experience.

[0146] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A safe driving monitoring system for engineering vehicles based on the fusion of facial and behavior recognition, characterized in that: include: The facial perception module is used to obtain the real-time image of the driver's face and build a dual-branch perception model, in which the gaze monitoring branch analyzes the overall facial features through the global attention mechanism to obtain global features; The fatigue monitoring branch focuses on the facial features through a local attention mechanism to obtain local features; the global features and the local features are fused through a self-attention weighted algorithm to obtain a facial perception result; A behavior perception module is used to obtain a real-time image of the driver's torso and the environment, obtain the driver's current execution action through an action recognition model, obtain the driver's surrounding objects through a target detection model, and perform a correlation analysis on the execution action and the surrounding objects based on a multimodal recognition word vector analysis method to obtain a behavior perception result; A face and behavior fusion module, used for fusing the face perception result and the behavior perception result to obtain driving status data; A traffic light recognition module is used to obtain a real-time image of traffic lights on external roads, and classify the states of the traffic lights in the real-time image to obtain traffic signal data; The start-stop sensing module is used to obtain the current vehicle driving status and vehicle start-stop data; The algorithm analysis module is used to connect the facial and behavior fusion module, the traffic light recognition module and the start-stop perception module, and generate an assisted driving instruction based on the Q-Learning algorithm in combination with the driving status data, the traffic signal data and the vehicle start-stop data.

2. According to claim 1, a construction vehicle safe driving monitoring system based on face and behavior recognition fusion is characterized in that: The facial perception results include normal driving, distracted driving and fatigue driving, and the behavior perception results include standard driving and irregular driving, and the irregular driving includes at least one of smoking, drinking water or making phone calls.

3. According to claim 1, a construction vehicle safe driving monitoring system based on face and behavior recognition fusion is characterized in that: The facial perception module includes: a sight line monitoring unit for monitoring whether the driver's sight line deviates from the road ahead; a fatigue monitoring unit for monitoring the driver's fatigue state; and obtaining the facial perception result includes: Using the first camera to obtain a real-time facial image of the driver; Based on the sight monitoring unit, identifying whether the real-time facial image is in a face turning state, and if so, recording the face turning angle and turning time; if the face turning angle is greater than a turning angle threshold, and the turning time is greater than a turning time threshold, determining that it is distracted driving; Based on the fatigue monitoring unit, it is identified whether the real-time facial image is in a closed-eye state or a yawning state. If it is in a closed-eye state, the closed-eye time and the closed-eye frequency are recorded. If the closed-eye time is greater than the closed-eye time threshold or the closed-eye frequency is greater than the closed-eye frequency threshold, it is determined to be fatigue driving; if it is in a yawning state, the yawning frequency is recorded. If the yawning frequency is greater than the yawning frequency threshold, it is determined to be fatigue driving.

4. The engineering vehicle safe driving monitoring system based on face and behavior recognition fusion according to claim 1 is characterized in that: The dual-branch perception model includes: An input layer, used to receive a real-time image of the driver's face acquired by the first camera; The gaze monitoring branch is used to analyze the overall facial features through a global attention mechanism to obtain global features, where the global features include facial contour features and facial turning features; The fatigue monitoring branch is used to focus on the facial features through the local attention mechanism to obtain local features, including eye features and mouth features; A branch fusion layer, used for fusing the global features and the local features through a self-attention weighted algorithm to generate a comprehensive facial feature; A classification layer, used to classify the comprehensive facial features and output facial perception results; The input layer is a 3×3 convolutional layer; The line of sight monitoring branch includes a downsampling layer and four non-local residual blocks connected in sequence, and the non-local residual block includes three residual blocks connected in sequence, a maximum pooling layer and a non-local attention module; The fatigue monitoring branch includes five sequentially connected local direct convolution blocks, and the local direct convolution block includes three sequentially connected 3×3 convolution layers, a maximum pooling layer and a local attention module; The branch fusion layer fuses the global features and the local features through a self-attention weighted algorithm, and the formula of the self-attention weighted algorithm is: F fused =F global +α·V local ; Among them, F fused Indicates comprehensive facial features, F global represents the global feature, α represents the attention weight, V local represents the local eigenvalue; The classification layer classifies the fused features through a softmax function.

5. The engineering vehicle safe driving monitoring system based on face and behavior recognition fusion according to claim 1 is characterized in that: The behavior perception module includes a behavior recognition unit, a target detection unit and a correction unit; Obtaining the behavior perception result includes: Using the second camera to obtain a real-time image of the driver's torso, and using the third camera to obtain a real-time image of the environment around the driver; The behavior recognition unit processes the real-time image of the torso, identifies the driver's behavior by extracting the key points of the driver, and converts the recognition result into a behavior word vector; The target detection unit analyzes the real-time image of the environment, identifies the objects in the image, and converts the recognition results into object word vectors; Expanding 5 related word vectors of the behavior word vector through the ontology library, and comparing them with the object word vector to obtain a comparison result; The correction unit determines whether the comparison result is correct. If the comparison result is correct, the behavior recognition is confirmed as an irregular operation; if the comparison result is incorrect, the behavior recognition result is corrected.

6. The engineering vehicle safe driving monitoring system based on face and behavior recognition fusion according to claim 1 is characterized in that: The face and behavior fusion module adopts a weighted voting mechanism for fusion, and the calculation formula of the weighted voting mechanism is: C combined =W face ×C face +W behavior ×C behavior ; Among them, C combined represents the perceptual fusion score, W face represents the face perception weight, C face represents the face perception confidence, W behavior represents the behavior perception weight, C behavior Indicates the confidence of behavior perception.

7. The engineering vehicle safe driving monitoring system based on face and behavior recognition fusion according to claim 1 is characterized in that: The signal light status includes: a red light status, reminding the driver to stop; a yellow light status, reminding the driver to slow down or prepare to stop; a green light status, reminding the driver to pass safely; and a signal light not detected status, reminding the driver to drive carefully and make judgments based on the actual road conditions.

8. The engineering vehicle safe driving monitoring system based on face and behavior recognition fusion according to claim 1 is characterized in that: The vehicle start-stop data is acquired by an acceleration sensor, and the vehicle start-stop data includes a start-stop state, a steering state, an acceleration state, and a current speed.

9. The engineering vehicle safe driving monitoring system based on face and behavior recognition fusion according to claim 1 is characterized in that: The algorithm analysis module includes: a data processing unit, which is used to pre-process the data transmitted by the face and behavior fusion module, the signal light recognition module and the start-stop perception module to obtain pre-processed data; a decision algorithm unit, which is used to perform reinforcement learning on the pre-processed data using the Q-Learning algorithm according to preset driving safety rules and road traffic regulations, and generate assisted driving instructions; generating the assisted driving instructions includes: Determine the state space, action space and reward function, define the combination of driving behavior, traffic signal and vehicle state, and set driving instructions and reward mechanism according to the combination; Build a driving simulation environment that provides reward information based on real-time feedback of the current state and execution actions; Initialize the Q table and select actions through the ε-greedy strategy, observe the environmental feedback to update the Q value, and store the Q value in the Q table; Monitor the status during actual driving, use the Q table to query the best action and generate corresponding driving instructions; Regularly evaluate model performance, monitor safety and efficiency through testing, and adjust parameters to optimize learning results.

10. The engineering vehicle safe driving monitoring system based on face and behavior recognition fusion according to claim 1 is characterized in that: The driving monitoring system further comprises: A voice broadcast module, used to connect to the algorithm analysis module and provide voice prompts or warnings according to the auxiliary driving instructions generated by the algorithm analysis module; A communication module is used to communicate with a remote server or an external device, upload the monitored driving status data, the traffic signal data and the vehicle start and stop data, and receive control instructions from the remote server. The communication module supports wireless communication protocols, including at least one or more of 4G, 5G, and Wi-Fi wireless communication technologies, and is used to realize two-way data transmission with a remote server.

Images