Dynamic monitoring system for safe driving behavior and safe production of mobile machinery on-site based on AI big model

Abstract

The present invention relates to the field of dynamic monitoring technology, and more specifically, to a dynamic monitoring system for safe driving behavior and safe production of mobile machinery on-site based on an AI large model. It includes a multimodal data acquisition unit, which collects real-time video data, motion data, control instruction data, and obstacle data around the vehicle; a multimodal data fusion unit is used to achieve time domain alignment of video data and motion data through a dynamic time offset compensation model and an interpolation algorithm, and then establish a causal association model between control instruction data and motion data based on a structural equation model, and reversely correct the time domain alignment parameters; a safety monitoring unit detects abnormal driving behavior in real time, and triggers an early warning mechanism when an abnormality is detected. The system can accurately adjust the alignment parameters of the time domain alignment module, while improving the sensitivity and accuracy of abnormal driving behavior detection, and can also improve the adaptability of the safety monitoring system to dynamic environments.

Description

Technical Field

[0001] The present invention relates to the field of dynamic monitoring technology, and in particular to a dynamic monitoring system for safe driving behavior and safe production of mobile machinery on-site based on an AI large model. Background Art

[0002] With the continuous development of industrial automation, especially in the increasingly widespread application environment of mobile machinery on site (such as forklifts, handling robots, etc.), real-time monitoring of the safe driving behavior and dynamic production status of these machines has become particularly important. In a complex working environment, mechanical equipment needs to respond to various sudden operating situations in real time, such as obstacle avoidance, sharp turns or sudden acceleration. Therefore, how to achieve high-precision and safety assurance for these mechanical equipment, especially accurate driving behavior monitoring in dynamic environments, has become an important research direction for improving production efficiency and ensuring safety.

[0003] However, existing monitoring systems often face the problem of insufficient synchronization accuracy of multimodal sensor data. The timing misalignment of data generated by multiple sensors (such as cameras, lidar, IMU, ultrasonic sensors, etc.) will affect the accuracy of monitoring results, especially in dynamic environments. Traditional methods often ignore timing errors when processing these multimodal data, resulting in the inability to accurately synchronize data from different sensors, thus affecting the reliability of subsequent analysis and decision-making.

[0004] In addition, traditional models are designed with the assumption that the environment is static, ignoring the existence and changes of dynamic obstacles in the environment. In actual driving, the distance, shape and changes of obstacles will directly affect the driver's reaction and the vehicle's movement behavior. Therefore, a dynamic monitoring system for safe driving behavior and safe production of mobile machinery in the field based on the AI ​​large model is designed. Summary of the invention

[0005] The purpose of the present invention is to provide a dynamic monitoring system for safe driving behavior and safe production of mobile machinery on-site based on an AI large model, so as to solve the problem that the traditional methods proposed in the above background technology often ignore the timing error when processing these multimodal data, resulting in the inability to accurately synchronize the data of different sensors and ignoring the existence and changes of dynamic obstacles in the environment.

[0006] To achieve the above objectives, the present invention aims to provide a dynamic monitoring system for safe driving behavior and safe production of mobile machinery on-site based on an AI large model, comprising: A multimodal data acquisition unit, which is used to collect real-time video data through a vehicle-mounted camera, collect real-time motion data of mechanical equipment through an IMU sensor, obtain real-time control instruction data of the vehicle through a CAN bus system, and collect obstacle data around the vehicle through a laser radar and ultrasonic sensor; A multimodal data fusion unit, the multimodal data fusion unit is used to achieve time domain alignment of video data and motion data through a dynamic time offset compensation model and an interpolation algorithm, introduce a nonlinear attenuation factor into the dynamic time offset compensation model for optimization, and then establish a causal association model between control instruction data and motion data based on a structural equation model, and at the same time, reversely correct the time domain alignment parameters by using the deviation between the result predicted by the causal association model and the actual data; Among them, the multimodal data fusion unit includes a time domain alignment module and a causal association modeling module; A safety monitoring unit detects abnormal driving behavior in real time based on the time-domain aligned video data and motion data and a causal model, and triggers an early warning mechanism when an abnormality is detected.

[0007] As a further improvement of the technical solution, in the multimodal data fusion unit: The time domain alignment module is used to achieve time domain alignment of video data and motion data through a dynamic time offset compensation model and an interpolation algorithm, and to generate a video data sequence and a motion data sequence after time domain alignment. A nonlinear attenuation factor is introduced into the dynamic time offset compensation model for optimization, and the influence of obstacle data is introduced into the process of establishing a causal association model between control instruction data and motion data for optimization. The causal association modeling module is used to take the aligned motion data sequence and the aligned video data sequence as input, and establish a causal association model between the control instruction data and the motion data based on the structural equation model. In the process of establishing the causal association model between the control instruction data and the motion data, the influence of obstacle data is introduced for optimization. At the same time, the error term of the causal association model is used to reversely correct the time domain alignment parameters.

[0008] As a further improvement of the technical solution, the specific steps of the temporal alignment module to achieve temporal alignment of video data and motion data are as follows: S21, establishing a dynamic time offset compensation model, and dynamically correcting the timestamp by combining the drift amount and the event trigger signal; S22, defining the matching degree between the video data event and the motion data event, adjusting the alignment window size in real time in combination with the corrected timestamp and the event matching degree, and generating an adaptive window size; S23. Based on the adaptive window size, the motion data is interpolated by cubic splines with kinematic constraints, and the video data is interpolated by video data with optical flow constraints, and finally a video data sequence and a motion data sequence after time domain alignment are generated.

[0009] As a further improvement of the technical solution, the specific steps of S21 are as follows: Dynamic time offset compensation model: ; in, For sensor In time The amount of clock drift; For sensor In time The amount of clock drift; is the inertia factor; is the actual observed time difference; is the theoretical time difference; is the number of samples for the average drift; Timestamp correction: ; in, For the corrected sensor timestamp; For the original sensor timestamp; Compensate for the event weight; For the The trigger flag of the class event; is the number of event types; .

[0010] As a further improvement of the technical solution, the clock drift of the sensor in the dynamic time offset compensation model changes greatly after long-term operation, and the drift of the sensor is affected by external temperature and power supply fluctuations. Therefore, a nonlinear attenuation factor is introduced into the dynamic time offset compensation model for optimization. The optimized dynamic time offset compensation model is specifically as follows: ; in, For optimized sensors In time The amount of clock drift; is the drift decay rate; for the current moment; For sensor The measurement noise variance of is the hyperbolic tangent function; ; in, For optimized and corrected sensors The timestamp of the

[0011] As a further improvement of the technical solution, the specific steps of S22 are as follows: ; in, is the matching degree between video data and motion data; is video data; For sports data; is the total number of frames; ; For the Frame video data; For the Feature intensity of ROI area in frame video data; For the Frame motion data; For the Edge strength of frame motion data; For video data The Frobenius norm of ; For sports data The Frobenius norm of ; is the video event time window; is the motion event time window; is the intersection-and-union ratio of the video event time window and the motion event time window; ; in, For The time window size; is the base window size; The time point at which the event is triggered; is the standard deviation of the time window.

[0012] As a further improvement of the technical solution, the specific steps of S23 are as follows: ; in, is the aligned motion data sequence; To align the window size; is the timestamp of the previous known data point; is the coefficient one; is the coefficient of two; is coefficient three; is coefficient four; ; in, is the aligned video data sequence; For the Frame to target time The optical flow field; For the Frame video data; The time window size involved in the optical flow field calculation; .

[0013] As a further improvement of the technical solution, the causal association modeling module uses the aligned multimodal data sequence as input and establishes a causal association model between the control instruction data and the motion data based on the structural equation model, as follows: The aligned motion data And the aligned video data As input to a causal relationship model between control command data and motion data; ; in, For time Motion data at that time; is the baseline bias of the model; For control instructions Sports data Impact factor; For in time Control instructions at all times; For control instructions time delay; is the total number of control instructions; ; For video data Sports data Impact factor; For video data time delay; is the total number of video data; ; For sports data Sports data Impact factor; For sports data time delay; is the total number of motion data; ; is the error term; In a complex operating environment, the position and state of mobile vehicles, pedestrians and other dynamic objects change over time, making it impossible to accurately predict the behavior changes of mobile machinery in a dynamic environment. Therefore, the influence of obstacle data is introduced in the process of establishing a causal relationship model between control instruction data and motion data for optimization. After optimization, the specific results are as follows: ; in, For Time Barrier The influence of location and size; For obstacles The size factor of For time When the vehicle and the obstacle The relative distance is the distance attenuation factor; is the relative angle between the obstacle and the vehicle; is the angle sensitivity factor; is the relative speed between the vehicle and the obstacle; is the relative speed sensitivity factor; ; in, For the optimized time Motion data at that time; For obstacles The factors affecting position and size on motion data; is the total number of obstacles; .

[0014] As a further improvement of the technical solution, the causal association modeling module uses the error term of the causal association model to reversely correct the time domain alignment parameter, as follows: ; in, is the corrected alignment error; is the current alignment error; is the learning rate; is the derivative of the alignment error with respect to the time domain alignment parameter; is a constant; is the total time step; ; The corrected alignment error is introduced into the timestamp, and then the adaptive window size is corrected, and finally the corrected video data sequence and motion video sequence are generated.

[0015] As a further improvement of this technical solution, the specific process of the safety monitoring unit is as follows: S31. Using a dataset from normal driving behavior, a machine learning algorithm is used to generate a relationship between the driver's behavior and the vehicle motion data under normal driving conditions, thereby constructing a normal driving behavior model; the normal driving behavior model is specifically: ; in, For normal driving Movement data at time; Operate history sequence for the driver; for Time: Real-time status of vehicle; for Temporal environment awareness data; is a parameterized machine learning mapping function; for Temporal white noise process; S32. Calculate the driving behavior deviation at each moment according to the output of the established normal driving behavior model and the causal association model. If the deviation exceeds a preset threshold, the driving behavior is determined to be abnormal. The driving behavior deviation is specifically: ; in, for Driving behavior deviation over time; S33, after determining that the driving behavior is abnormal, using an action recognition algorithm to extract the driver's action features from the aligned video data sequence, using a time series analysis method to extract the vehicle's behavior features from the aligned motion data sequence, using a multimodal neural network to perform feature fusion on the driver's action features and the vehicle's behavior features, and then using a classification model to classify the fused features to generate a driving behavior category; S34. After the driving behavior category is generated, the early warning mechanism is triggered immediately.

[0016] Compared with the prior art, the present invention has the following beneficial effects: 1. In the on-site mobile machinery safe driving behavior and safe production dynamic monitoring system based on the AI ​​large model, the causal relationship model between control instructions and motion data is used to correct the alignment error. It can accurately adjust the alignment parameters of the time domain alignment module to avoid data synchronization errors caused by timing misalignment, ensure better alignment of data from different sensors, and provide a more accurate data basis for subsequent decision-making.

[0017] 2. In the on-site mobile machinery safe driving behavior and safe production dynamic monitoring system based on the AI ​​large model, the influence of obstacles is introduced in the process of establishing the causal correlation model between control command data and motion data. This can more realistically reflect the vehicle's behavioral response in a complex environment, thereby more accurately describing the actual impact of control commands on motion data, improving the sensitivity and accuracy of abnormal driving behavior detection, and can also enhance the adaptability of the safety monitoring system to dynamic environments, providing a more reliable basis for real-time decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is the overall flow chart of the present invention; The meaning of each number in the figure is: 1. Multimodal data acquisition unit; 2. Multimodal data fusion unit; 21. Time domain alignment module; 22. Causal association modeling module; 3. Safety monitoring unit. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] See also Figure 1 As shown, a dynamic monitoring system for safe driving behavior and safe production of mobile machinery on-site based on an AI large model is provided, including a multimodal data acquisition unit 1, a multimodal data fusion unit 2 and a safety monitoring unit 3; The multimodal data acquisition unit 1 is used to collect real-time video data through the vehicle camera, collect real-time motion data of mechanical equipment through the IMU sensor, obtain real-time control command data of the vehicle through the CAN bus system (such as steering wheel angle, accelerator / brake pedal status), and collect obstacle data (distance and position information) around the vehicle through the laser radar and ultrasonic sensor; wherein the video data includes driver behavior, the surrounding environment of the mechanical equipment and the interface video stream; The multimodal data fusion unit 2 is used to achieve time domain alignment of video data and motion data through a dynamic time offset compensation model and an interpolation algorithm, introduce a nonlinear attenuation factor into the dynamic time offset compensation model for optimization, and then establish a causal association model between control instruction data and motion data based on a structural equation model. At the same time, the deviation between the result predicted by the causal association model and the actual data is used to reversely correct the time domain alignment parameters; Among them, the multimodal data fusion unit 2 includes a time domain alignment module 21 and a causal association modeling module 22; the time domain alignment module 21 is used to achieve time domain alignment of video data and motion data through a dynamic time offset compensation model and an interpolation algorithm, and generate a video data sequence and a motion data sequence after time domain alignment, introduce a nonlinear attenuation factor into the dynamic time offset compensation model for optimization, and introduce the influence of obstacle data into the process of establishing a causal association model between control instruction data and motion data for optimization; The specific steps of the time domain alignment module 21 to achieve the time domain alignment of video data and motion data are as follows: S21, establishing a dynamic time offset compensation model, and dynamically correcting the timestamp by combining the drift amount and the event trigger signal; S22, defining the matching degree between the video data event (such as steering wheel rotation) and the motion data event (such as angular velocity mutation), adjusting the alignment window size in real time based on the corrected timestamp and event matching degree, and generating an adaptive window size; S23, based on the adaptive window size, interpolating the motion data by using cubic splines with kinematic constraints, and interpolating the video data by using video data with optical flow constraints, and finally generating a video data sequence and a motion data sequence after time domain alignment; The specific steps of S21 are as follows: Dynamic time offset compensation model: ; in, For sensor In time The amount of clock drift; For sensor In time The amount of clock drift; is the inertia factor, between 0 and 1, which determines the degree of memory of the drift. Indicates that the current drift has a greater impact on the correction, and a lower This means that historical drift has a greater impact on the correction; is the actual observed time difference; is the theoretical time difference; is the number of samples for the average drift; Timestamp correction: ; in, For the corrected sensor timestamp; For the original sensor timestamp; Compensation weights for events (learned from historical data, such as ), indicating the degree of influence of each event type on timestamp correction; For the Trigger flags for class events (such as the issuance of a steering instruction); is the number of event types; ; This formula is used to correct the sensor's timestamp to eliminate the time deviation caused by clock drift and event triggering. It is the drift calculated by the dynamic time offset compensation model to correct the inherent bias of the hardware clock. It is an event-triggered compensation item, which can dynamically adjust the timestamp according to the occurrence of key events (such as the issuance of steering instructions), further improve the alignment accuracy, ensure the consistency of multimodal data on the timeline, and provide a basis for subsequent interpolation and causal modeling.

[0021] The clock drift of the sensor in the dynamic time offset compensation model in the time domain alignment module 21 changes greatly after long-term operation. At this time, the traditional linear compensation model cannot accurately capture the dynamic change of the drift, because the traditional linear compensation model assumes that the drift change is stable. The drift of the sensor is affected by external temperature and power supply fluctuations and exhibits nonlinear characteristics. Therefore, a nonlinear attenuation factor is introduced into the dynamic time offset compensation model for optimization. The introduction of the nonlinear attenuation factor can help eliminate the large error caused by sensor drift over a long period of time, make the drift compensation more stable, and avoid unnecessary impact of long-term drift on the system. The optimization is as follows: ; in, For optimized sensors In time The amount of clock drift; The drift attenuation rate controls the degree of drift attenuation. Generally, the larger the value, the faster the drift correction decreases. This attenuation factor gradually weakens the influence of past drift on the current moment, thereby preventing historical drift from having too great an impact on the current timestamp correction. for the current moment; For sensor The measurement noise variance is used to indicate the noise level of the sensor. The larger the noise variance, the lower the measurement accuracy of the sensor; is a hyperbolic tangent function, which is used to suppress the weight of high-noise sensors, so that the result is suppressed when the input value is large (even if the weight of the sensor with excessive noise is small); Because the clock drift of the sensor usually does not grow infinitely, the drift will tend to stabilize or decay over time. By introducing the attenuation factor To suppress the accumulation of drift, it can prevent past drift from having too much impact on the current moment; The noise variance reflects the accuracy of the sensor. When the sensor noise is large, its contribution to time alignment should be reduced by using the hyperbolic tangent function. Smoothing out the noise can prevent sensors with excessive noise from having a disproportionate impact on the entire time domain alignment process; In summary, by introducing the nonlinear attenuation factor ,The model can adaptively reduce the impact of historical drift, so as to more accurately reflect the current clock deviation. As a noise suppression term, it can effectively reduce the weight of high-noise sensors and avoid the interference of abnormal data on drift estimation. The ultimate goal of this formula is to achieve high-precision synchronization of multi-sensor timestamps and ensure the reliability of subsequent data alignment; ; in, For optimized and corrected sensors timestamp; The specific steps of S22 are as follows: ; in, is the matching degree between video data and motion data; is video data; For sports data; is the total number of frames; ; For the Frame video data; For the Feature intensity of ROI area in frame video data; For the Frame motion data; For the Edge strength of frame motion data; For video data The Frobenius norm of ; For sports data The Frobenius norm of ; is the video event time window; is the motion event time window; is the intersection-and-union ratio of the video event time window and the motion event time window; Define video event time windows and motion event time window ; is the start time of the video event, The end time of the video event; is the start time of the motion event, The end time of the sports event; Compute the intersection and union of time windows: ; in is the intersection length; is the length of the union; Express Perform logarithmic transformation to enhance the sensitivity of small value intervals. hour, ;when hour, , used to enhance the sensitivity of small value intervals and avoid Smaller matching results in a smaller error in the matching calculation.

[0022] The first term is used to quantify the spatial matching degree between video events and motion events; the second term is used to quantify the temporal overlap between video events and motion events. By combining spatial and temporal information, the video data is calculated. and sports data The matching degree is used to guide the subsequent adaptive alignment window adjustment; ; in, For The time window size; is the benchmark window size (e.g. 10ms), which is set according to the data sampling frequency and system requirements; The time point when the event is triggered (such as the time when the steering wheel rotation command is issued); is the standard deviation of the time window, which controls the smoothness of the window; This item is used to dynamically adjust the window size. , indicating that the video time and motion events are highly matched, and the window size , narrow the window to improve the local alignment accuracy; when , indicating that the video time and motion events do not match, the window size , enlarge the window to avoid missed alignment; This item is used to trigger the event at the time point A higher weight is given to the vicinity, and the weight gradually decays when it is far away from the event triggering time point. , you can control the smoothness of the window and its sensitivity to events; The specific steps of S23 are as follows: ; in, is the aligned motion data sequence, ; To align the window size; is the timestamp of the previous known data point. Assume there is a data sequence , where each It's in time The data points collected. is the previous time point before the current interpolation point; is the coefficient one; is the coefficient of two; is coefficient three; is coefficient four; cubic spline interpolation uses four coefficients to determine the shape of each segment of the curve. , , , Determined by the constraints of the known data points and their neighbors, they are usually determined by solving a system of equations to ensure that the interpolation function is smooth and continuous between the data points; ; in, is the aligned video data sequence, ; For the Frame to target time The optical flow field; For the Frame video data; The time window size involved in the optical flow field calculation; ; The causal association modeling module 22 is used to take the aligned motion data sequence and the aligned video data sequence as input, establish a causal association model between the control instruction data and the motion data based on the structural equation model, introduce the influence of the obstacle data in the process of establishing the causal association model between the control instruction data and the motion data for optimization, and at the same time, use the error term of the causal association model to reversely correct the time domain alignment parameter; The causal association modeling module 22 takes the aligned multimodal data sequence as input and establishes a causal association model between the control instruction data and the motion data based on the structural equation model, as follows: The aligned motion data And the aligned video data As input to a causal relationship model between control command data and motion data; ; in, For time The motion data at that time indicates the real-time motion status of the vehicle (such as acceleration, steering angle, speed, etc.); is the baseline bias of the model; For control instructions Sports data The influence factor indicates the influence of different control commands (such as throttle, brake, steering, etc.) on the motion data; For in time Control instructions at all times (such as throttle, brake, steering angle, etc.); For control instructions The time delay indicates the lag effect of control instructions on motion data; is the total number of control instructions; ; For video data Sports data The impact factor represents the impact of video data (such as driver operation, environmental changes, etc.) on vehicle behavior; For video data The time delay indicates that the video data has a lagging effect on the motion data; is the total number of video data; ; For sports data Sports data The influence factor represents the influence of the previous moment's motion data on the current motion data; For sports data time delay; is the total number of motion data; ; is the error term, which represents the part of the motion data that is not explained by the control command, video data, or obstacle position and size; The causal association model has the problem of interference between the vehicle's motion data and control instructions in a complex dynamic environment. Because in a complex operating environment, the position and state of mobile vehicles, pedestrians and other dynamic objects change over time. The traditional causal association model does not consider the real-time information of dynamic obstacles, resulting in the inability to accurately predict the behavior changes of mobile machinery in a dynamic environment. Therefore, in the process of establishing the causal association model between control instruction data and motion data, the influence of obstacle data is introduced for optimization. After optimization, the specific results are as follows: ; in, For Time Barrier The influence of location and size; For obstacles The size factor represents the size or volume of the obstacle (e.g., the radius or length and width of the obstacle). The size of the obstacle affects its effect on the vehicle's motion; For time When the vehicle and the obstacle The relative distance of the obstacle. Generally, the closer the obstacle is, the greater the impact on the vehicle. Here, the inverse square of the distance is used to simulate distance attenuation; The distance decay factor controls the rate of distance decay. It can be adjusted according to specific driving scenarios to optimize the response of the model; is the relative angle between the obstacle and the vehicle. Generally, obstacles have a greater impact on vehicle motion when they are in front or closer to the front. Therefore, an angle function is used to weight the impact; is the angle sensitivity factor, which controls the sensitivity of the angle to the impact. For smaller angles (i.e. when the obstacle is facing the vehicle), the impact will be more significant; is the relative speed between the vehicle and the obstacle. The greater the relative speed between the vehicle and the obstacle, the more violent the vehicle's reaction to the obstacle may be. Here, the square of the relative speed is considered to model the impact; is the relative speed sensitivity factor, which controls the sensitivity of speed to the impact. If the relative speed between the vehicle and the obstacle is high, the impact will be enhanced; ; in, For the optimized time Motion data at that time; For obstacles The influence factors of position and size on motion data indicate the degree of influence of obstacles, which can usually be modeled by the distance, relative speed and size between the obstacle and the vehicle; is the total number of obstacles; ; In the process of establishing a causal relationship model between control command data and motion data, introducing the influence of obstacles can significantly improve the accuracy and robustness of the model. By incorporating environmental factors (such as obstacles) into the model, the behavioral response of the vehicle in a complex environment can be more realistically reflected, thereby more accurately describing the actual impact of the control command on the motion data. For example, when an obstacle approaches or the relative speed is large, the model can automatically adjust the prediction of the control command to reduce the error caused by obstacle interference. In addition, the relative angle and size of the obstacle enable the model to adapt to different driving scenarios more flexibly, improving the sensitivity and accuracy of abnormal driving behavior detection. This causal modeling method that comprehensively considers multiple factors can improve the adaptability of the safety monitoring system to dynamic environments and provide a more reliable basis for real-time decision-making.

[0023] The causal association modeling module 22 uses the error term of the causal association model to reversely correct the time domain alignment parameter, as follows: ; in, is the corrected alignment error, which indicates the updated alignment time difference after correcting the causal association model. This correction value is based on system feedback (i.e. error Updated with respect to the time domain alignment parameters; is the current alignment error, which indicates the data deviation corrected in time. It reflects the initial alignment between the control instruction data and the motion data; is the learning rate, which indicates the degree of adjustment to the current error at each correction. It is usually a small positive number that controls the rate of correction. will result in fast adjustments but may be unstable; smaller This will result in slower convergence, but more stable; is the derivative of the alignment error with respect to the time domain alignment parameter. By calculating the gradient, we can understand how to adjust To reduce the error; is a constant to avoid the denominator being zero and to ensure the stability of the calculation; is the total time step; ; The corrected alignment error is introduced into the timestamp, and then the adaptive window size is corrected, and finally the corrected video data sequence and motion video sequence are generated; This formula uses the error term of the causal model Reverse correction time alignment parameters ,By normalizing the gradient direction, the model can quickly converge to the optimal alignment parameters,,ensure high-precision synchronization of multimodal data, and achieve closed-loop optimization,of time domain alignment and causal modeling; The core purpose of time domain alignment is to ensure the temporal consistency of different modal data (such as video data and motion data). Without accurate time domain alignment, subsequent analysis (such as abnormal behavior detection, control command prediction, etc.) will be greatly affected. Using the causal relationship model between control commands and motion data to correct alignment errors can accurately adjust the alignment parameters of the time domain alignment module, avoid data synchronization errors caused by timing misalignment, and ensure better alignment of data from different sensors, thereby providing a more accurate data basis for subsequent decision-making.

[0024] The safety monitoring unit 3 detects abnormal driving behavior in real time based on the time-domain aligned video data and motion data and the causal model, and triggers the early warning mechanism when an abnormality is detected; The specific process of safety monitoring unit 3 is as follows: S31. Using a dataset from normal driving behavior, a convolutional neural network is used to generate the relationship between the driver's behavior and the vehicle motion data under normal driving conditions, thereby constructing a normal driving behavior model; by learning normal driving behavior, the system can establish a baseline model, and subsequent driving behaviors will be compared with this baseline. The model provides a standard for subsequent deviation calculations to ensure that the system can effectively distinguish between normal and abnormal driving behaviors; the normal driving behavior model is specifically: ; in, For normal driving Movement data at time; Operate history sequence for the driver; for Time: Real-time status of vehicle; for Temporal environment awareness data; is a parameterized machine learning mapping function; for Temporal white noise process; S32. Calculate the driving behavior deviation at each moment according to the established normal driving behavior model and causal association model. If the deviation exceeds a preset threshold, the driving behavior is determined to be abnormal. The system calculates the deviation between the current driving behavior and the expected behavior by comparing the difference between the driving behavior characteristics collected in real time and the normal behavior pattern. The driving behavior deviation is specifically: ; in, for Driving behavior deviation over time; S33. After determining that the driving behavior is abnormal, extract the driver's action features (such as steering wheel rotation, driver's facial expression changes, etc.) from the aligned video data sequence using a long short-term memory network (using the long short-term memory network as an action recognition algorithm), extract the vehicle's behavior features from the aligned motion data sequence using a time series analysis method, fuse the driver's action features and vehicle behavior features using a multimodal neural network, and then classify the fused features using a support vector machine classification model to generate driving behavior categories; the driving behavior categories include fatigue driving, distracted driving, and dangerous operation; S34. After the driving behavior category is generated, the warning mechanism is triggered immediately. When the driver shows signs of fatigue driving, inattention or other dangerous behaviors, timely reminders or intervention measures can be taken to reduce the occurrence of accidents, including issuing warnings to the driver (such as sound or visual prompts) or automatically adjusting vehicle behavior (such as automatic braking or deceleration).

[0025] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments, and the above embodiments and descriptions are only preferred examples of the present invention, and are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements all fall within the scope of the present invention to be protected.

Claims

1. A dynamic monitoring system for safe driving behavior and safe production of mobile machinery on-site based on AI large model, characterized by: include: A multimodal data acquisition unit (1), the multimodal data acquisition unit (1) being used to acquire real-time video data through a vehicle-mounted camera, to acquire real-time motion data of mechanical equipment through an IMU sensor, to acquire real-time control instruction data of the vehicle through a CAN bus system, and to acquire obstacle data around the vehicle through a laser radar and an ultrasonic sensor; A multimodal data fusion unit (2), the multimodal data fusion unit (2) is used to achieve time domain alignment of video data and motion data through a dynamic time offset compensation model and an interpolation algorithm, introduce a nonlinear attenuation factor into the dynamic time offset compensation model for optimization, and then establish a causal association model between control instruction data and motion data based on a structural equation model, and at the same time, reversely correct the time domain alignment parameter by using the deviation between the result predicted by the causal association model and the actual data; Wherein, the multimodal data fusion unit (2) includes a time domain alignment module (21) and a causal association modeling module (22); A safety monitoring unit (3) detects abnormal driving behavior in real time based on the video data and motion data after time domain alignment and a causal model, and triggers an early warning mechanism when an abnormality is detected.

2. The AI ​​large model-based on-site mobile machinery safe driving behavior and safe production dynamic monitoring system according to claim 1 is characterized in that: In the multimodal data fusion unit (2): The time domain alignment module (21) is used to achieve time domain alignment of video data and motion data through a dynamic time offset compensation model and an interpolation algorithm, and generate a video data sequence and a motion data sequence after time domain alignment, introduce a nonlinear attenuation factor into the dynamic time offset compensation model for optimization, and introduce the influence of obstacle data into the process of establishing a causal association model between control instruction data and motion data for optimization; The causal association modeling module (22) is used to take the aligned motion data sequence and the aligned video data sequence as input, establish a causal association model between the control instruction data and the motion data based on the structural equation model, introduce the influence of obstacle data in the process of establishing the causal association model between the control instruction data and the motion data for optimization, and at the same time, use the error term of the causal association model to reversely correct the time domain alignment parameters.

3. The AI ​​large model-based on-site mobile machinery safe driving behavior and safe production dynamic monitoring system according to claim 2 is characterized by: The specific steps of the time domain alignment module (21) to achieve the time domain alignment of video data and motion data are as follows: S21, establishing a dynamic time offset compensation model, and dynamically correcting the timestamp by combining the drift amount and the event trigger signal; S22, defining the matching degree between the video data event and the motion data event, adjusting the alignment window size in real time in combination with the corrected timestamp and the event matching degree, and generating an adaptive window size; S23. Based on the adaptive window size, the motion data is interpolated by cubic splines with kinematic constraints, and the video data is interpolated by video data with optical flow constraints, and finally a video data sequence and a motion data sequence after time domain alignment are generated.

4. The AI ​​large model-based on-site mobile machinery safe driving behavior and safe production dynamic monitoring system according to claim 3 is characterized in that: The specific steps of S21 are as follows: Dynamic time offset compensation model: ； in, For sensor In time The amount of clock drift; For sensor In time The amount of clock drift; is the inertia factor; is the actual observed time difference; is the theoretical time difference; is the number of samples for the average drift; Timestamp correction: ； in, For the corrected sensor timestamp; For the original sensor timestamp; Compensate for the event weight; For the The trigger flag of the class event; is the number of event types; .

5. The AI ​​large model-based on-site mobile machinery safe driving behavior and safe production dynamic monitoring system according to claim 4 is characterized by: The clock drift of the sensor changes greatly after the dynamic time offset compensation model runs for a long time, and the drift of the sensor is affected by external temperature and power supply fluctuations. Therefore, a nonlinear attenuation factor is introduced into the dynamic time offset compensation model for optimization. The optimized dynamic time offset compensation model is specifically as follows: ； in, For optimized sensors In time The amount of clock drift; is the drift decay rate; for the current moment; For sensor The measurement noise variance of is the hyperbolic tangent function; ； in, For optimized and corrected sensors The timestamp of the 6. The AI ​​large model-based on-site mobile machinery safe driving behavior and safe production dynamic monitoring system according to claim 5 is characterized in that: The specific steps of S22 are as follows: ； in, is the matching degree between video data and motion data; is video data; For sports data; is the total number of frames; ; For the Frame video data; For the Feature intensity of ROI area in frame video data; For the Frame motion data; For the Edge strength of frame motion data; For video data The Frobenius norm of ; For sports data The Frobenius norm of ; is the video event time window; is the motion event time window; is the intersection-and-union ratio of the video event time window and the motion event time window; ； in, For The time window size; is the base window size; The time point at which the event is triggered; is the standard deviation of the time window.

7. The AI ​​large model-based on-site mobile machinery safe driving behavior and safe production dynamic monitoring system according to claim 6 is characterized in that: The specific steps of S23 are as follows: ； in, is the aligned motion data sequence; To align the window size; is the timestamp of the previous known data point; is the coefficient one; is the coefficient of two; is coefficient three; is coefficient four; ； in, is the aligned video data sequence; For the Frame to target time The optical flow field; For the Frame video data; The time window size involved in the optical flow field calculation; .

8. The AI ​​large model-based on-site mobile machinery safe driving behavior and safe production dynamic monitoring system according to claim 7 is characterized by: The causal association modeling module (22) uses the aligned multimodal data sequence as input and establishes a causal association model between the control instruction data and the motion data based on the structural equation model, as follows: The aligned motion data And the aligned video data As input to a causal relationship model between control command data and motion data; ； in, For time Motion data at that time; is the baseline bias of the model; For control instructions Sports data Impact factor; For in time Control instructions at all times; For control instructions time delay; is the total number of control instructions; ; For video data Sports data Impact factor; For video data time delay; is the total number of video data; ; For sports data Sports data Impact factor; For sports data time delay; is the total number of motion data; ; is the error term; In a complex operating environment, the position and state of mobile vehicles, pedestrians and other dynamic objects change over time, making it impossible to accurately predict the behavior changes of mobile machinery in a dynamic environment. Therefore, the influence of obstacle data is introduced in the process of establishing a causal relationship model between control instruction data and motion data for optimization. After optimization, the specific results are as follows: ； in, For Time Barrier The influence of location and size; For obstacles The size factor of For time When the vehicle and the obstacle The relative distance is the distance attenuation factor; is the relative angle between the obstacle and the vehicle; is the angle sensitivity factor; is the relative speed between the vehicle and the obstacle; is the relative speed sensitivity factor; ； in, For the optimized time Motion data at that time; For obstacles The factors affecting position and size on motion data; is the total number of obstacles; .

9. The AI ​​large model-based on-site mobile machinery safe driving behavior and safe production dynamic monitoring system according to claim 8 is characterized by: The causal association modeling module (22) uses the error term of the causal association model to reversely correct the time domain alignment parameter, as follows: ； in, is the corrected alignment error; is the current alignment error; is the learning rate; is the derivative of the alignment error with respect to the time domain alignment parameter; is a constant; is the total time step; ; The corrected alignment error is introduced into the timestamp, and then the adaptive window size is corrected, and finally the corrected video data sequence and motion video sequence are generated.

10. The AI ​​large model-based on-site mobile machinery safe driving behavior and safe production dynamic monitoring system according to claim 9 is characterized in that: The specific process of the safety monitoring unit (3) is as follows: S31. Using a dataset from normal driving behavior, a machine learning algorithm is used to generate a relationship between the driver's behavior and the vehicle motion data under normal driving conditions, thereby constructing a normal driving behavior model; the normal driving behavior model is specifically: ； in, For normal driving Movement data at time; Operate history sequence for the driver; for Time: Real-time status of vehicle; for Temporal environment awareness data; is a parameterized machine learning mapping function; for Temporal white noise process; S32. Calculate the driving behavior deviation at each moment according to the output of the established normal driving behavior model and the causal association model. If the deviation exceeds a preset threshold, the driving behavior is determined to be abnormal. The driving behavior deviation is specifically: ； in, for Driving behavior deviation over time; S33, after determining that the driving behavior is abnormal, using an action recognition algorithm to extract the driver's action features from the aligned video data sequence, using a time series analysis method to extract the vehicle's behavior features from the aligned motion data sequence, using a multimodal neural network to perform feature fusion on the driver's action features and the vehicle's behavior features, and then using a classification model to classify the fused features to generate a driving behavior category; S34. After the driving behavior category is generated, the early warning mechanism is triggered immediately.

Images