Driver risk behavior prediction system based on MoE model

Through the MoE model-based driver risk behavior prediction system, combined with full-color cameras and edge computing hosts, real-time monitoring and analysis of driver behavior, the problem of inability to comprehensively evaluate drivers' operating habits and psychological state in the existing technology is solved, and the engineering vehicle is realized The high accuracy and real-time risk warning significantly improve the safety of industrial operations.

CN118781583BActive Publication Date: 2024-11-22FUJIAN SPECIAL EQUIP TESTING RES INST
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411244406.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-06
Publication Date
2024-11-22
Estimated Expiration
2044-09-06

AI Technical Summary

Technical Problem

The existing monitoring system cannot comprehensively evaluate the driver's operating habits and psychological state, lacks real-time and accuracy, and it is difficult to timely warning and prevent the occurrence of engineering vehicle accidents.

Method used

Using a driver risk behavior prediction system based on the MoE model, combining full-color cameras, edge computing hosts and AI inference chips, it monitors and analyzes the driver's facial expressions, line of sight direction and body posture in real time, and through identity recognition, feature extraction, behavior serialization and Risk prediction module, generates risk assessment results and provides early warnings.

Benefits of technology

It has achieved comprehensive monitoring and risk prediction of driver behavior, improved the accuracy and real-time nature of risk prediction, and prompt warning, significantly reduced the incidence of safety accidents and improved the safety of industrial operations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118781583B_ABST
    Figure CN118781583B_ABST
Patent Text Reader

Abstract

The present invention provides a driver risk behavior prediction system based on the MoE model, the prediction system is installed on an engineering vehicle, and the prediction system includes: a full-color camera, an edge computing host, and an AI reasoning chip mounted on the edge computing host; the AI ​​reasoning chip realizes real-time reasoning of the MoE model, and the edge computing host is provided with: an identity recognition module, a feature extraction module, a behavior sequence module, a risk prediction module, and an early warning and decision-making module; the system realizes the identification of the driver's identity and behavior monitoring through a full-color camera installed in the forklift cab, combined with the host of the AI ​​reasoning chip. The system can analyze the driver's facial features and posture movements in real time, and predict his risk behaviors such as fatigue level, distraction, and abnormal emotions, and report to the early warning center in time to take corresponding measures to ensure operational safety.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and machine vision technology, in particular to a driver risk behavior prediction system based on an MoE model, which is used for real-time monitoring and analysis of driver behavior, predicting potential risky behavior, and improving the safety of industrial operations. Background Art

[0002] With the rapid development of industrial automation, engineering vehicles such as forklifts play an increasingly important role in modern logistics and manufacturing. The use of these vehicles not only improves work efficiency, but also greatly promotes the improvement of productivity. However, with it comes the safety issues related to the operation of engineering vehicles.

[0003] Currently, monitoring systems on the market mainly rely on basic sensor technology and manual observation to monitor the operating status of engineering vehicles. These methods have obvious limitations:

[0004] 1) Sensor limitations: Existing sensors can usually only monitor certain basic parameters of the vehicle, such as speed and direction, but cannot fully assess the driver's operating habits and psychological state, resulting in the inability to detect potential risk factors in a timely manner;

[0005] 2) Insufficient manual observation: Manual observation relies on the experience and attention of the monitoring personnel, which is easily affected by subjective judgment and prone to fatigue and omissions during long-term monitoring;

[0006] 3) Real-time and accuracy issues: Existing systems often lack real-time performance and cannot respond quickly to emergencies. They also have deficiencies in the accuracy of risk assessment, making it difficult to achieve effective early warning of risks under complex working conditions.

[0007] 4) Insufficient data processing capabilities: Existing monitoring systems have limited capabilities in data processing and analysis and are unable to fully utilize large amounts of operational data to predict and prevent potential safety risks.

[0008] In view of the above problems, there is an urgent need for a more advanced and intelligent monitoring system to achieve real-time and accurate risk assessment and early warning of engineering vehicle operations. Such a system should be able to comprehensively use a variety of sensor data, combined with advanced data analysis technology, to monitor the driver's physiological and psychological state, as well as the vehicle's operating conditions in real time, so as to effectively prevent and reduce the occurrence of engineering vehicle accidents. Summary of the invention

[0009] To overcome the above problems, the purpose of the present invention is to provide a driver risk behavior prediction system based on the MoE model to achieve comprehensive monitoring and risk prediction of driver behavior.

[0010] The present invention is implemented by the following scheme: a driver risk behavior prediction system based on the MoE model, the prediction system is installed on an engineering vehicle, and the prediction system includes: a full-color camera, an edge computing host, and an AI reasoning chip mounted on the edge computing host; the AI ​​reasoning chip realizes real-time reasoning of the MoE model, and the edge computing host is provided with: an identity recognition module, a feature extraction module, a behavior sequence module, a risk prediction module, and an early warning and decision-making module;

[0011] The full-color camera captures the behavioral feature data of the driver's facial expression, line of sight and body posture, and sends the behavioral feature data to the edge computing host. The edge computing host identifies the driver's identity and monitors the behavior. The identity recognition module is used to verify the driver's identity; the feature extraction module, after verifying the driver's identity, extracts the feature data in real time and transmits the extracted feature data to the behavior sequence module; the behavior sequence module preprocesses and serializes the feature data to form serialized data suitable for MoE model analysis; the risk prediction module receives the serialized data, performs in-depth analysis, and generates risk assessment results; the early warning and decision-making module executes early warning according to the risk assessment results. The MoE model is a deep learning model composed of multiple experts and networks. Each expert and network analyzes different aspects of the driver's behavior. The MoE model improves the recognition and prediction capabilities of complex behavior characteristics through integrated learning. The analysis in the experts and networks uses the MTCNN model to extract the face position information and face feature information in the image from the input image information; MMPose is used to extract the position information of limb feature points; the Mahalanobis distance formula is used to calculate the distance information based on the extracted face features, and the information is compared with the database of the edge computing host to complete the driver's identity authentication.

[0012] Furthermore, the edge computing host has a 4G communication module, a dual-mode satellite positioning module, a 9-axis attitude sensor and a standard communication protocol bus; the edge computing host is provided with a mainboard chip, the 4G communication module, the dual-mode satellite positioning module, and the 9-axis attitude sensor are all connected to the mainboard chip through a serial bus, and the standard communication protocol bus is led out by the I2C pin of the mainboard chip; the 4G communication module: realizes remote transmission of feature data to ensure the real-time and accuracy of information; the dual-mode satellite positioning module: provides precise location information of the vehicle, supports GPS and Beidou satellite systems, and ensures that vehicle positioning can be achieved worldwide; the 9-axis attitude sensor is used to detect the vehicle's speed, acceleration and inclination attitude information; the 9-axis attitude sensor includes a three-axis accelerometer, a three-axis gyroscope and a three-axis magnetometer; the standard communication protocol bus: supports multiple communication protocols to achieve seamless docking with various sensors and external devices.

[0013] Furthermore, the AI ​​inference chip: mounted on the edge computing host, has efficient data processing capabilities, is designed to process input multimodal data, can respond quickly, and realize real-time reasoning of the MoE model, ensuring that the prediction system can timely identify and predict the driver's risky behavior.

[0014] Furthermore, the identity recognition module uses facial recognition technology to verify the driver's identity in real time, ensuring that only authorized drivers can operate engineering vehicles; by comparing the driver's facial feature data stored in the database of the edge computing host, fast and accurate recognition can be achieved; wherein, the identity recognition module extracts 478 feature points on the face, including 468 facial information and 10 pupil coordinate information, and then uses the Mahalanobis distance formula to calculate the feature point distance, thereby realizing the driver's identity authentication.

[0015] Furthermore, the feature extraction module performs structured analysis on the video picture information, and extracts facial feature information of the video picture information and limb feature information of the head, shoulders, and arms through the feature analysis layer of the MoE model.

[0016] Furthermore, the behavior sequence module collects limb feature points, including the head, shoulders, and arms, and integrates facial feature points to form a behavior time series in a set time dimension. The serialized data includes not only spatial position information but also time series information, providing a basis for in-depth analysis of the risk prediction module.

[0017] Furthermore, the risk prediction module conducts in-depth analysis on the behavior serialization data to identify potential risk factors of fatigue, distraction and emotional state. The MoE model learns the behavior patterns of drivers in different states to accurately predict the driver's behavior risk. The risk prediction module comprehensively predicts the driving risk level based on the detected fatigue level and emotional state data, combined with driving data, vehicle speed and acceleration, and timely warns of high-risk behaviors.

[0018] The fatigue level:

[0019] Normal: The driver is awake and alert and can respond normally to road conditions;

[0020] Mild fatigue: The driver begins to lose concentration and has a prolonged reaction time;

[0021] Moderate fatigue: The driver's attention and judgment are further reduced, and the driver may close his eyes or take a nap for a short time;

[0022] Severe fatigue: The driver's alertness is greatly reduced, and his reaction is slow. There is a risk of dozing off or momentary loss of consciousness;

[0023] Extreme fatigue: The driver may completely lose control of the vehicle, risking a serious accident;

[0024] Described driving risk level:

[0025] Normal: The driver's status is clear, and the driving speed and posture are within normal range;

[0026] Low risk: The driver slightly exceeded the speed limit, failed to use the seat belt as required, or had a minor setback in driving;

[0027] Medium risk: The driver exceeds the speed limit by 10% to 20%, does not drive in the specified area, and has high acceleration and angular velocity values;

[0028] High risk: speeding 20%~50%, and the acceleration and angular velocity values ​​change dramatically;

[0029] Extremely high risk: exceeding the speed limit by more than 50%, with extremely high or drastic changes in acceleration and angle values.

[0030] Furthermore, the warning and decision-making module is responsible for timely generating warning information based on the analysis results of the risk prediction module, and reporting the information to relevant personnel or the prediction system through the 4G communication module of the prediction system; at the same time, the decision support includes: suggesting that the driver take a rest, adjust the work plan or take preventive measures to reduce safety risks.

[0031] The beneficial effects of the present invention are as follows: 1. Comprehensive monitoring and analysis: The system realizes comprehensive monitoring of driver behavior through the coordinated work of the identity recognition module, feature extraction module and behavior sequence module. This not only includes real-time verification of the driver's identity, but also covers the monitoring of the driver's continuous posture movements of key parts such as the head, shoulders, arms, and the capture of facial expressions and features. This comprehensive monitoring provides rich data support for behavior analysis.

[0032] 2. Improve the accuracy of risk prediction: Using the deep learning capabilities of the MoE model, the system can accurately analyze the driver's behavioral characteristics, thereby significantly improving the accuracy of risk prediction. By integrating multiple experts and networks, the MoE model can learn and identify the driver's behavior patterns from different angles and effectively predict potential risks such as fatigue, distraction, and mood swings.

[0033] 3. Enhanced real-time risk prediction: The system uses an end-to-end model analysis method, combined with the high-speed processing capabilities of the AI ​​inference chip, to achieve real-time monitoring of driver behavior and immediate risk assessment. This real-time performance ensures that the system can respond quickly, identify abnormal behavior in a timely manner, and take corresponding measures.

[0034] 4. Timely warning and decision support: The warning and decision module can generate warning information in time according to the analysis results of the risk prediction module, and report the information to relevant personnel or systems through the system's 4G communication module. This timely warning mechanism can effectively avoid potential safety accidents and ensure the safety of personnel and equipment.

[0035] 5. Reduce safety accidents: Through the close cooperation of the above modules, the system can timely detect and predict the driver's abnormal behavior, such as smoking, making phone calls, etc., which may lead to serious safety accidents during driving. The system's early warning and decision support functions help to take preventive measures, thereby significantly reducing the incidence of safety accidents.

[0036] 6. Improve the safety of industrial operations: The system of the present invention significantly improves the safety of engineering vehicle operations through real-time monitoring, accurate prediction and timely warning. This has important practical application value for industrial environments that require high security, such as logistics warehouses, manufacturing plants and construction sites.

[0037] 7. Promote technological progress and innovation: The implementation of this invention not only improves the level of existing security monitoring technology, but also promotes technological progress and innovative applications in fields such as artificial intelligence, machine vision and edge computing. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a schematic diagram of the hardware framework of the prediction system of the present invention.

[0039] Figure 2 It is a schematic diagram of the installation position of the full-color camera in an embodiment of the present invention.

[0040] Figure 3 It is a schematic diagram of extracting driver facial features according to an embodiment of the present invention.

[0041] Figure 4 It is a schematic diagram of limb feature extraction according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The present invention will be further described below in conjunction with the accompanying drawings.

[0043] See also Figure 1 and Figure 2As shown, the driver risk behavior prediction system based on the MoE model of the present invention is installed on an engineering vehicle, and the prediction system includes: a full-color camera, an edge computing host, and an AI reasoning chip mounted on the edge computing host; the AI ​​reasoning chip realizes real-time reasoning of the MoE model, and the edge computing host is provided with: an identity recognition module, a feature extraction module, a behavior sequence module, a risk prediction module, and an early warning and decision-making module; the edge computing host communicates with the engineering vehicle via the communication bus CAN or the communication bus MODBUS.

[0044] The full-color camera captures the behavioral feature data of the driver's facial expressions, line of sight and body posture, and sends the behavioral feature data to the edge computing host. The edge computing host identifies the driver's identity and monitors the behavior. The identity recognition module is used to verify the driver's identity; the feature extraction module, after verifying the driver's identity, extracts the feature data in real time, and transmits the extracted feature data to the behavior sequence module; the behavior sequence module preprocesses and serializes the feature data to form serialized data suitable for MoE model analysis; the risk prediction module receives the serialized data, performs in-depth analysis, and generates risk assessment results; the early warning and decision-making module performs early warning and decision support functions according to the risk assessment results to ensure timely response to potential risks and achieve comprehensive monitoring of driver behavior.

[0045] Triple-proof full-color camera: installed in front of the forklift cab, waterproof, dustproof, and shockproof to ensure safety and stability in open environments. Mainly responsible for capturing the driver's facial features and body movements. High-resolution imaging technology is used to capture key behavioral features such as the driver's facial expressions, sight direction, and body posture.

[0046] AI inference chip: The host has a built-in domestic AI inference chip for real-time processing of image and behavior data. The dedicated AI chip is used to accelerate the point multiplication calculation of the convolution layer, and the model supports quantized model reasoning with int4 and int8 precision to accelerate AI calculations.

[0047] Behavior monitoring: Through periodic identity authentication, monitor whether there are any risky behaviors of changing drivers. At the same time, collect limb feature points, including head, shoulders, arms and other limbs. Fusion of facial feature points, in the 30s time dimension, form a behavior time series, input the behavior sequence into the end-to-end model for behavior analysis. Realize the monitoring of driver fatigue, distraction, emotional state, and additional smoking, making phone calls and other behaviors.

[0048] The edge computing host has a 4G communication module, a dual-mode satellite positioning module, a 9-axis attitude sensor and a standard communication protocol bus; the edge computing host is provided with a mainboard chip, the 4G communication module, the dual-mode satellite positioning module, and the 9-axis attitude sensor are all connected to the mainboard chip through a serial bus, and the standard communication protocol bus is led out by the I2C pin of the mainboard chip; the 4G communication module: realizes the remote transmission of feature data to ensure the real-time and accuracy of information; the dual-mode satellite positioning module: provides the vehicle's precise location information, supports GPS and Beidou satellite systems, and ensures that vehicle positioning can be achieved worldwide; the 9-axis attitude sensor is used to detect the vehicle's speed, acceleration and inclination attitude information; the 9-axis attitude sensor includes a three-axis accelerometer, a three-axis gyroscope and a three-axis magnetometer; the standard communication protocol bus: supports multiple communication protocols to achieve seamless docking with various sensors and external devices.

[0049] The AI ​​inference chip: carried on the edge computing host, has efficient data processing capabilities, is designed to process input multimodal data, can respond quickly, and realize real-time reasoning of the MoE model, ensuring that the prediction system can timely identify and predict the driver's risky behavior.

[0050] The MoE model is a deep learning model composed of multiple experts and networks. Each expert and network analyzes different aspects of the driver's behavior. The MoE model improves the recognition and prediction capabilities of complex behavioral features through integrated learning. The analysis in the experts and networks uses the MTCNN model to extract face position information and face feature information in the image from the input image information; uses MMPose to extract limb feature point position information; uses the Mahalanobis distance formula to calculate the distance information based on the extracted face features and compares it with the database of the edge computing host to complete the driver's identity authentication.

[0051] The prediction system extracts feature data and performs preprocessing and serialization on the feature data in the following specific implementations:

[0052] (1) Extract visual data: Use the MTCNN model to extract facial features. , where I is the input image, is the extracted facial feature set; MMPose is used to extract limb features , Where is the set of human features;

[0053] (2) Vehicle speed information: The vehicle speed is directly used as a feature and normalized: ;

[0054] is through The function is normalized to between 0 and 1 to prevent gradient explosion in subsequent training; that is The vehicle speed normalized by the sigmoid function;

[0055] (3) 9-axis attitude data: It contains the acceleration, angular velocity and magnetic field angle information of the x, y and z axes: , where S is the sensor data;

[0056] It is a feature function extracted from the data, that is, a data processing set, which includes the conversion of the original data of the sensor, and finally obtains the accurate posture feature data of the acceleration and posture Euler angle. ;

[0057] (4) Feature fusion of feature data: Use weighted fusion at the current moment: ,

[0058] It refers to the aggregation matrix of the four types of feature matrices at the current moment, where They are the fusion weight values ​​of the face feature matrix, the limb feature matrix, the vehicle speed feature matrix and the posture feature matrix;

[0059] At the time series level, the feature data of the behavior serialization within 30 seconds is integrated as follows: ; The feature data representing the fused behavior serialization;

[0060] (5) Behavior analysis: Input the fused feature data into the MoE model for inference and prediction output: ,in is the predicted output of the model; is the inference function of the MoE model;

[0061] (6) Risk assessment: Based on the output of the MoE model, assess the risk level:

[0062] ; It is an end-to-end evaluation prediction model, inputting the above Analyze the results and output the risk level;

[0063] (7) Decision making: Make corresponding decisions based on the risk assessment results and specific behavioral results:

[0064] ; D means The behavioral decision results output by the end-to-end execution decision model.

[0065] The identification module uses facial recognition technology to verify the driver's identity in real time, ensuring that only authorized drivers can operate engineering vehicles; by comparing the driver's facial feature data stored in the database of the edge computing host, fast and accurate identification is achieved; Figure 3 As shown, the identity recognition module extracts 478 feature points on the face, including 468 facial information and 10 pupil coordinate information, and then uses the Mahalanobis distance formula to calculate the feature point distance, thereby realizing the driver's identity authentication.

[0066] The feature extraction module performs structured analysis on the video picture information, and extracts facial feature information of the video picture information and limb feature information of the head, shoulders, and arms through the feature analysis layer of the MoE model.

[0067] like Figure 4 As shown, the behavior sequence module collects limb feature points, including the head, shoulders, and arms, and integrates facial feature points to form a behavior time series in a set time dimension. The serialized data includes not only spatial position information but also time series information, providing a basis for in-depth analysis of the risk prediction module.

[0068] The risk prediction module conducts in-depth analysis of the behavior serialization data to identify potential risk factors of fatigue, distraction and emotional state. The MoE model learns the driver's behavior patterns in different states to accurately predict the driver's behavior risk. The risk prediction module comprehensively predicts the driving risk level based on the detected fatigue level and emotional state data, combined with driving data, vehicle speed and acceleration, and timely warns of high-risk behaviors.

[0069] The fatigue level:

[0070] Normal: The driver is awake and alert and can respond normally to road conditions;

[0071] Mild fatigue: The driver begins to lose concentration and has a prolonged reaction time;

[0072] Moderate fatigue: The driver's attention and judgment are further reduced, and the driver may close his eyes or take a nap for a short time;

[0073] Severe fatigue: The driver's alertness is greatly reduced, and his reaction is slow. There is a risk of dozing off or momentary loss of consciousness;

[0074] Extreme fatigue: The driver may completely lose control of the vehicle, risking a serious accident;

[0075] Described driving risk level:

[0076] Normal: The driver's status is clear, and the driving speed and posture are within normal range;

[0077] Low risk: The driver slightly exceeded the speed limit, failed to use the seat belt as required, or had a minor setback in driving;

[0078] Medium risk: The driver exceeds the speed limit by 10% to 20%, does not drive in the specified area, and has high acceleration and angular velocity values;

[0079] High risk: speeding 20%~50%, drastic changes in acceleration and angular velocity

[0080] Extremely high risk: exceeding the speed limit by more than 50%, with extremely high or drastic changes in acceleration and angle values.

[0081] The warning and decision-making module is responsible for timely generating warning information based on the analysis results of the risk prediction module, and reporting the information to relevant personnel or the prediction system through the 4G communication module of the prediction system; at the same time, the decision support includes: suggesting that the driver take a rest, adjust the work plan or take preventive measures to reduce safety risks.

[0082] This system uses facial recognition and behavior monitoring technology to achieve real-time detection and early warning of driver risk behaviors. The following are specific examples of the effects of the system implementation:

[0083] Fatigue driving warning:

[0084] The system automatically determines the fatigue level by analyzing the driver's facial features, such as the frequency and duration of eye closure. The behavioral model is an end-to-end model, with non-human-defined logical judgments (such as how long the eyes are closed to determine the fatigue level, etc.), and is self-learned based on the input training data. The labeled data is manually classified from the video images and various vehicle body data, and has a certain degree of perceptual and subjective components.

[0085] According to the driver's fatigue state and the current vehicle speed and acceleration, the system intelligently outputs warning signals of different levels.

[0086] Warning signs:

[0087] Fatigue level:

[0088] Normal: The driver is awake and alert and is able to respond normally to road conditions.

[0089] Mild fatigue: The driver begins to show symptoms such as lack of concentration and prolonged reaction time.

[0090] Moderate fatigue: The driver's attention and judgment are further reduced, and the driver may close his eyes or take a nap for a short time.

[0091] Severe fatigue: The driver's alertness is greatly reduced, his reaction is slow, and there is a high risk of falling asleep or losing consciousness momentarily.

[0092] Extreme fatigue: The driver may completely lose control of the vehicle, risking a serious accident.

[0093] Driving risk level:

[0094] Normal: The driver is in a clear state, and the driving speed and posture are within normal range.

[0095] Low risk: The driver is slightly speeding, fails to use the seat belt as required, or has minor driving jerks, etc.

[0096] Medium risk: The driver exceeds the speed limit by 10% to 20%, does not drive in the specified area, and the acceleration and angular velocity values ​​are large.

[0097] High risk: overspeeding by 20%~50%, and drastic changes in acceleration and angular velocity.

[0098] Extremely high risk: exceeding the speed limit by more than 50%, with extremely high or drastic changes in acceleration and angle values.

[0099] Misoperation warning and intervention:

[0100] When the system detects that the driver is making abnormal body movements such as lowering his head or bending over, and the vehicle speed increases abnormally, the system will output an accidental accelerator pedal warning.

[0101] After confirming the warning, the system can automatically intervene and perform braking operations to prevent potential accidents.

[0102] Abnormal emotions and dangerous driving warning:

[0103] The system uses facial expression analysis technology to identify the driver's emotional state, such as frustration or anger.

[0104] If it detects an abnormal increase in vehicle speed and unstable acceleration axis, the system will determine possible intentional dangerous driving behavior and immediately issue a warning and emergency braking command.

[0105] Fainting state recognition and intervention:

[0106] Through comprehensive analysis of facial features and body movements, the system can determine whether the driver is in a fainting state.

[0107] As mentioned above, it is related to the manually labeled training samples. The end-to-end model is trained by manually comprehensively judging the driver's driving fatigue level and driving risk level and then classifying the samples, which has a certain black box effect.

[0108] Once such a situation is detected, the system will automatically issue a warning and intervene in braking to ensure safety.

[0109] Attitude instability warning:

[0110] The system monitors the vehicle's posture through a gyroscope. If instability is detected, it will automatically issue a warning and reduce the vehicle speed to deal with potential risks.

[0111] Driver standing warning:

[0112] If the system detects that the driver is standing, it will issue a standing warning to remind the driver or manager to pay attention to safety.

[0113] Driver change warning:

[0114] The system can identify driver changes and issue timely warnings to ensure that only authorized personnel operate the vehicle.

[0115] Chemical operation area behavior standard warning:

[0116] In special operation areas such as chemical industry, the system will monitor whether the driver has any illegal behaviors such as smoking or making phone calls, and issue an early warning.

[0117] The above warning logic is partly based on human settings, and partly automatically implemented through a specially trained end-to-end model. The end-to-end model uses input multimodal data for deep learning and automatic reasoning and judgment.

[0118] In summary, the prediction system of the present invention, through its innovative design and efficient operation, provides users with a comprehensive, accurate, and real-time driver behavior monitoring and risk prediction solution, effectively improving the safety and reliability of industrial operations.

[0119] The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made according to the scope of the patent application of the present invention should fall within the scope of the present invention.

Claims

1. A driver risk behavior prediction system based on the MoE model, the prediction system is installed on an engineering vehicle, and is characterized by: The prediction system includes: a full-color camera, an edge computing host, and an AI reasoning chip mounted on the edge computing host; the AI ​​reasoning chip realizes real-time reasoning of the MoE model, and the edge computing host is provided with: an identity recognition module, a feature extraction module, a behavior sequence module, a risk prediction module, and an early warning and decision-making module; The full-color camera captures the behavioral feature data of the driver's facial expression, line of sight and body posture, and sends the behavioral feature data to the edge computing host. The edge computing host identifies the driver's identity and monitors the driver's behavior. The identity recognition module is used to verify the driver's identity; the feature extraction module extracts the feature data in real time after verifying the driver's identity, and transmits the extracted feature data to the behavior sequence module; the behavior sequence module preprocesses and serializes the feature data to form serialized data suitable for MoE model analysis; the risk prediction module receives the serialized data, performs in-depth analysis, and generates risk assessment results; the early warning and decision-making module performs early warning and decision support functions based on the risk assessment results; the MoE model is composed of multiple experts and networks, each of which analyzes different aspects of the driver's behavior. The analysis in the experts and networks is to use the model MTCNN to extract the face position information and face feature information in the image from the input image information; use MMPose to extract the position information of limb feature points; use the Mahalanobis distance formula to calculate the distance information based on the extracted face features, and compare the distance information with the database of the edge computing host to complete the driver's identity authentication; The prediction system extracts feature data and performs preprocessing and serialization on the feature data in the following specific implementations: S1. Extract visual data: Use the MTCNN model to extract facial features, F faceFeature =MTCNN(I), where I is the input image, F faceFeature is the extracted face feature set; MMPose is used to extract limb features, F bodyFeature =MMPose(I), where F bodyFeature is the set of human features; S2. Vehicle speed information: The vehicle speed is directly used as a feature and the vehicle speed is normalized: F speed =f(speed); F speed is through The function is normalized to between 0 and 1 to prevent gradient explosion in subsequent training; S3, 9-axis attitude data: It contains the acceleration, angular velocity and magnetic field angle information of the x-axis, y-axis and z-axis: F sensor = extractFeatures(S), where S is the sensor data; extractFeatures() is a sensor data feature extraction function, i.e., a data processing set, which includes the conversion of the sensor raw data, and finally obtains the accurate attitude feature data F of the acceleration and attitude Euler angle that meet the accuracy requirements sensor ; S4. Perform feature fusion on the feature data to obtain the feature value after weighted fusion processing at the current moment: F combined_(t) =[w1·F faceFeature :w2·F bodyFeature :w3·F speed :w4·F sensor ], F combined_(t) It refers to the fusion matrix of the four types of feature matrices at time t, where w1, w2, w3, and w4 are the fusion weight values ​​of the face feature matrix, the limb feature matrix, the vehicle speed feature matrix, and the posture feature matrix respectively; At the time series level, the feature data of the behavior serialization within 30 seconds is integrated as follows: F combined =[F combined_(t-30) :F combined_(t-29) :…:F combined_(t-1) :F combined_(t) ]; F combined The feature data representing the fused behavior serialization; S5. Behavior analysis: Input the fused feature data into the MoE model for prediction output: in is the predicted output of the model; S6. Risk Assessment: Based on the output of the MoE model, assess the risk level: risk_assessment() is an end-to-end assessment prediction model that outputs the model's predictions Take input and output risk level; S7. Decision making: Make corresponding decisions based on risk assessment results and specific behavior results: D refers to the behavioral decision result output by the decison_maker end-to-end execution decision model.

2. The driver risk behavior prediction system based on the MoE model according to claim 1 is characterized by: The edge computing host has a 4G communication module, a dual-mode satellite positioning module, a 9-axis attitude sensor and a standard communication protocol bus; the edge computing host is provided with a mainboard chip, the 4G communication module, the dual-mode satellite positioning module, and the 9-axis attitude sensor are all connected to the mainboard chip through a serial bus, and the standard communication protocol bus is led out by the I2C pin of the mainboard chip; the 4G communication module: realizes the remote transmission of feature data; the dual-mode satellite positioning module: provides the vehicle's precise location information, supports GPS and Beidou satellite systems, and ensures that vehicle positioning can be achieved worldwide; the 9-axis attitude sensor is used to detect the vehicle's speed, acceleration and inclination attitude information; the 9-axis attitude sensor includes a three-axis accelerometer, a three-axis gyroscope and a three-axis magnetometer; the standard communication protocol bus: supports multiple communication protocols to achieve seamless docking with various sensors and external devices.

3. The driver risk behavior prediction system based on the MoE model according to claim 1 is characterized by: The AI ​​inference chip: carried on the edge computing host, the AI ​​inference chip is designed to process input multimodal data, so that the prediction system can timely identify and predict the driver's risky behavior.

4. The driver risk behavior prediction system based on the MoE model according to claim 1 is characterized by: The identity recognition module uses facial recognition technology to verify the driver's identity in real time, and compares the feature data with the driver's facial feature data stored in the database of the edge computing host to achieve fast and accurate recognition; wherein, the identity recognition module extracts 478 feature points on the face, including 468 facial information and 10 pupil coordinate information, and then uses the Mahalanobis distance formula to calculate the feature point distance, thereby realizing the driver's identity authentication.

5. The driver risk behavior prediction system based on the MoE model according to claim 1 is characterized by: The feature extraction module performs structured analysis on the video picture information, and extracts facial feature information of the video picture information and limb feature information of the head, shoulders and arms through the feature analysis layer of the MoE model.

6. The driver risk behavior prediction system based on the MoE model according to claim 1 is characterized by: The behavior sequence module forms a behavior time series by fusing the collected limb feature points including the head, shoulders and arms with the facial feature points, and in a set time dimension. The serialized feature data includes not only spatial position information but also time series information.

7. The driver risk behavior prediction system based on the MoE model according to claim 1 is characterized by: The risk prediction module conducts in-depth analysis on the behavior serialization data to identify potential risk factors of fatigue, distraction and emotional state. The MoE model learns the behavior patterns of drivers in different states to accurately predict the driver's behavior risk. The risk prediction module comprehensively predicts the driving risk level based on the detected fatigue level and emotional state data, combined with driving data, vehicle speed, and acceleration, and provides timely warnings for high-risk behaviors. The fatigue level: Normal: The driver is awake and alert and can respond normally to road conditions; Mild fatigue: The driver begins to lose concentration or has a prolonged reaction time; Moderate fatigue: The driver's attention and judgment are further reduced, and the driver may close his eyes or take a nap for a short time; Severe fatigue: The driver's alertness is greatly reduced, and his reaction is slow. There is a risk of dozing off or momentary loss of consciousness; Extreme fatigue: The driver may completely lose control of the vehicle, risking a serious accident; Described driving risk level: Normal: The driver's status is clear, and the driving speed and posture are within normal range; Low risk: The driver is slightly speeding, not using the seat belt as required, or driving slightly jerk; Medium risk: The driver exceeds the speed limit by 10% to 20%, does not drive in the specified area, and has large acceleration and angular velocity values; High risk: overspeeding by 20% to 50%, with drastic changes in acceleration and angular velocity values; Extremely high risk: exceeding the speed limit by more than 50%, with extremely high acceleration and angular velocity values ​​or extremely drastic changes.

8. The driver risk behavior prediction system based on the MoE model according to claim 1 is characterized by: The warning and decision-making module generates warning information based on the analysis results of the risk prediction module, and reports the information to relevant personnel or the prediction system through the 4G communication module of the prediction system; at the same time, the decision support includes: suggesting that the driver take a rest, adjust the work plan or take preventive measures to reduce safety risks.