Intelligent identification method and system for illegal behaviors of commercial vehicle based on multi-source data fusion

Abstract

The invention relates to the technical field of intelligent traffic supervision and artificial intelligence, in particular to an intelligent identification method and system for illegal behaviors of operating vehicles based on multi-source data fusion, and the method comprises the steps: 1, carrying out the time-space alignment collection of multi-source data; step 2, hierarchical fusion decision making: behavior feature extraction; performing multi-level rule verification; confidence fusion: distributing the video confidence, the abnormal operation probability value and the weight of the rule matching degree according to the illegal type, and generating fusion confidence; step 3, feedback optimization: when the fusion confidence exceeds a type-related alarm threshold, pushing a graded alarm instruction; and updating the abnormal operation probability value calculation model and the weight distribution strategy based on the law enforcement feedback data. Through the technical means of space-time alignment of multi-source data, extraction and fusion of behavior characteristics, dynamic adjustment of a judgment threshold value and the like, the problems of difficult fusion of the multi-source data, inaccurate behavior recognition and the like are solved, and the accuracy and reliability of intelligent recognition of illegal behaviors of commercial vehicles are remarkably improved.

Description

Technical Field

[0001] This invention relates to the fields of intelligent traffic supervision and artificial intelligence technology, and in particular to an intelligent identification method and system for illegal behavior of operating vehicles based on multi-source data fusion. Background Technology

[0002] Safety supervision of commercial vehicles (including freight vehicles and long-distance passenger vehicles) is a core component of intelligent traffic management. Due to the three main characteristics of commercial vehicles—long continuous driving time, fixed operating routes, and driver-identified drivers—their violations (such as fatigued driving, speeding, and deviation from designated routes) often exhibit persistence, cross-regional activity, and the coupling of multiple factors. Current industry-standard technologies have the following limitations:

[0003] 1. Single-source data identification mechanisms are insufficient to capture persistent illegal activities;

[0004] Existing video surveillance-based identification methods rely on a single visual data stream, essentially detecting instantaneous behavioral fragments (such as crossing lines or changing lanes). However, violations such as fatigued driving and exceeding driving time limits in operational scenarios require continuous state monitoring.

[0005] Case description: When a driver's eyes are closed, the existing technology cannot distinguish between a brief blink and a continuous drowsy state, nor can it associate the vehicle's handling characteristics (such as slight steering wheel corrections) and driving time data during that period, resulting in a false negative rate of up to 53% for fatigue driving.

[0006] 2. Data fragmentation across multiple systems leads to the failure of behavioral correlations;

[0007] The supervision of commercial vehicles involves three independent systems: vehicle-mounted GPS / OBD terminals, roadside monitoring equipment, and transportation administration platforms.

[0008] Misalignment of spatiotemporal references: The vehicle-mounted terminal uses the WGS-84 coordinate system and is stamped with the device's local clock, while the roadside camera uses the local coordinate system and NTP clock. The update cycle of transportation administration data is as long as minutes. The three cannot be directly correlated in the spatiotemporal dimension.

[0009] When GPS tracking shows a vehicle speeding in mountainous areas, the system cannot match the dynamic speed limit rules for that road segment in real time (such as speed limits being lowered in rainy or foggy weather). The system only judges based on the default speed limit threshold, resulting in a false alarm rate as high as 32%.

[0010] Therefore, there is an urgent need for an intelligent identification method and system for illegal behaviors of commercial vehicles based on multi-source data fusion to solve the above problems. Summary of the Invention

[0011] To achieve the above objectives, this invention provides a method and system for intelligent identification of illegal behaviors of commercial vehicles based on multi-source data fusion, wherein the method includes the following steps:

[0012] Step 1: Spatiotemporal alignment acquisition of multi-source data:

[0013] The vehicle's real-time location coordinates, driving speed, steering wheel angle change rate, and driver identification code are collected synchronously through the vehicle terminal.

[0014] Collect continuous video frames containing the target vehicle using roadside monitoring equipment;

[0015] Real-time access to the transportation administration database to obtain the key point sequence of the vehicle's registered operating route, the planned driving time threshold, and the speed limit type of the road section;

[0016] Perform spatiotemporal reference normalization: transform the vehicle positioning coordinates and the geographical coordinates of the roadside monitoring equipment to a unified coordinate system, and control the timestamp error through a clock synchronization protocol;

[0017] Step 2: Layered Integration Decision Making

[0018] Behavioral feature extraction: Extract the driver's eye opening and closing change rate, steering wheel angle offset, and license plate recognition confidence from video frames, construct the steering wheel angle change rate into a time series, and output the abnormal operation probability value;

[0019] Multi-level rule verification: When the rate of change in eye opening and closing is continuously lower than the physiological threshold, the system verifies whether the steering wheel angle offset exceeds the dynamic range, checks whether the current driving time exceeds the filing threshold, and dynamically adjusts the overspeed judgment threshold based on the real-time location matching road segment speed limit type.

[0020] Confidence fusion: Based on the type of violation, the video confidence score, the probability value of abnormal operation, and the rule matching degree are weighted to generate a fused confidence score;

[0021] Step 3: Feedback and Optimization

[0022] When the fusion confidence level exceeds the type-related alarm threshold, a tiered alarm command is pushed.

[0023] The calculation model and weight allocation strategy for the probability value of abnormal operations are updated based on law enforcement feedback data.

[0024] Preferably, the spatiotemporal reference normalization in step 1 includes:

[0025] Clock synchronization process: The provincial monitoring platform distributes the network time protocol clock signal to the vehicle terminal and roadside equipment. The timestamp correction amount is calculated through multi-level delay compensation so that the timestamp error of all devices is controlled within the millisecond range.

[0026] Coordinate transformation process: Using the measured coordinates of the highway toll station as control points, the coordinate system transformation parameters are solved, and the original coordinates of the vehicle-mounted global positioning system are transformed to the national geodetic coordinate system. The transformation residuals are iteratively optimized to within the set threshold using the least squares method.

[0027] Preferably, the extraction of the rate of change in eye opening and closing in step 2 includes:

[0028] The driver's facial region was located in the video frame, and the coordinates of key points in the left and right eyes were detected using a convolutional neural network.

[0029] Calculate the rate of change in eyelid distance in consecutive frames, and activate infrared supplementary lighting enhancement when the light intensity is lower than a set threshold.

[0030] The fatigue index is calculated based on blink frequency and duration. The process is as follows:

[0031] a: Count the number of frames with the eyelids completely closed within a 30-second time window;

[0032] b: When the percentage of continuous frames exceeds the dynamic threshold, it is determined as an initial detection of fatigue state.

[0033] Preferably, the generation of the abnormal operation probability value in step 2 includes:

[0034] Constructing a two-channel time series model:

[0035] Channel 1 input: First-order difference sequence of the rate of change of steering wheel angle;

[0036] The sliding variance sequence of the brake pedal travel input in channel two;

[0037] Local mutation features are extracted through a one-dimensional convolutional layer, and the output layer uses the Softmax function to generate the probability distribution of three types of operations: sharp turn, sharp stop, and sharp acceleration.

[0038] When the sampling interval of the vehicle-mounted sensor increases abnormally, the historical data interpolation reconstruction sequence is activated.

[0039] Preferably, the parameter solving process for the coordinate transformation includes:

[0040] Select the coordinates of the toll stations along the vehicle's trajectory as control point pairs;

[0041] Establish a seven-parameter transformation model: ;

[0042] The translation parameters are solved using the least squares method. Rotation matrix and scale factor The calculation is iterated until the sum of squared residuals at all control points is less than the set convergence threshold.

[0043] Preferably, the strategy for dynamically adjusting the overspeed determination threshold in step 2 includes:

[0044] Mountainous road section judgment: When the vehicle elevation change rate continuously exceeds the set gradient, the speeding judgment threshold will be lowered by a fixed percentage.

[0045] Highway section determination: Continuously monitor the mileage of speeding status, and trigger an alarm when the accumulated mileage exceeds the dynamic threshold and no deceleration behavior is detected;

[0046] The dynamic threshold is adaptively adjusted based on real-time traffic flow density.

[0047] Preferably, the weight allocation strategy in step 2 includes:

[0048] Video confidence weight calculation: based on license plate recognition region segmentation completeness and illumination uniformity scores;

[0049] Abnormal operation probability value weight calculation: based on sensor sampling frequency stability index;

[0050] Rule matching degree weight calculation: based on the overlap rate of key points between the actual trajectory and the registered route;

[0051] The weighting function is updated in real time based on environmental visibility and road curvature.

[0052] Preferably, the historical data interpolation reconstruction includes:

[0053] When sensor data loss is detected, extract the driving operation mode library for the same vehicle model on the same road segment;

[0054] Matching the most similar pattern fragments based on the dynamic time warping algorithm;

[0055] Missing intervals are filled with time-series data of similar segments and marked as interpolated data for probability calculation.

[0056] Preferably, the process of setting the dynamic threshold includes:

[0057] Establish a baseline database of driver physiological parameters and store the normal blinking frequency of drivers of different ages;

[0058] Retrieve baseline data for the corresponding age group based on the current driver's identification code;

[0059] A real-time fatigue correction is added to the baseline value. The correction is calculated by the ratio of continuous driving time to planned driving time.

[0060] Accordingly, embodiments of the present invention also provide an intelligent identification system for illegal acts of operating vehicles based on multi-source data fusion, including a memory configured to store instructions, a processor configured to call the instructions from the memory, and capable of implementing an intelligent identification method for illegal acts of operating vehicles based on multi-source data fusion as described in any embodiment of the present invention when executing the instructions.

[0061] The beneficial effects of this invention are:

[0062] 1. This invention employs spatiotemporal reference normalization technology to accurately align vehicle-mounted terminal data and roadside monitoring data in both time and space. The clock synchronization process utilizes the Network Time Protocol (NTP) to precisely synchronize the vehicle-mounted terminal and roadside equipment, ensuring that the timestamp error of all devices is controlled within milliseconds, thus avoiding data misalignment caused by clock errors. The coordinate transformation process uses a seven-parameter transformation model to accurately convert the vehicle's Global Positioning System (GPS) coordinates to the national geodetic coordinate system, significantly improving the spatial consistency of the data. This technology eliminates spatiotemporal alignment problems caused by different data sources, laying a precise data foundation for subsequent behavior recognition and violation determination.

[0063] 2. This invention, by combining video frame analysis and sensor data to extract behavioral features such as the rate of change in driver eye opening and closing, steering wheel angle offset, and brake pedal travel, can accurately determine the driver's fatigue state and abnormal operation. The rate of change in eye opening and closing is detected by a convolutional neural network (CNN) to detect the coordinates of key eye points, calculate blink frequency and duration in real time, and adjust the threshold according to fatigue correction, thus improving the accuracy of fatigue driving identification. By applying a dual-channel time series model, combined with steering wheel angle change rate and brake pedal data, abnormal operations such as sudden turns, sudden braking, and sudden acceleration can be accurately identified, solving the problem of inaccurate driver behavior identification in traditional methods.

[0064] 3. This invention accurately identifies speeding behavior through a multi-level rule verification and dynamic threshold adjustment mechanism, and dynamically adjusts the speeding judgment threshold according to the characteristics of different road sections. In mountainous areas, when the vehicle's elevation change rate exceeds a set gradient, the system automatically lowers the speeding judgment threshold, ensuring higher sensitivity to speeding behavior in special terrain conditions. On highways, by monitoring the cumulative mileage of speeding states and combining it with the monitoring of deceleration behavior, the speeding judgment standard is dynamically adjusted, avoiding the problem of inconsistent speeding judgment standards across different road sections. This mechanism improves the detection accuracy of speeding behavior, especially in complex road conditions, enabling more accurate identification of genuine speeding behavior.

[0065] 4. This invention proposes a confidence fusion strategy, which generates a fused confidence score by assigning different weights to video confidence, abnormal operation probability, and rule matching degree. This fusion process dynamically adjusts the weight allocation based on the video's completeness, sensor stability, and the overlap rate between the actual trajectory and the registered route, ensuring that the contributions of various data sources are reasonably evaluated and avoiding judgment biases that may be caused by a single data source. This method effectively improves the accuracy and reliability of behavior recognition and reduces false positives and false negatives.

[0066] 5. This invention solves the problem of sensor data loss through historical data interpolation and reconstruction technology. When sensor data loss is detected, the system extracts a database of driving operation patterns for the same vehicle model on the same road segment, uses a dynamic time warping algorithm to match the most similar driving pattern, fills in the missing data, and participates in subsequent probability calculations. This supplementation mechanism ensures that even in the event of data loss, the system can still correctly infer driver behavior, thereby avoiding inaccurate recognition caused by data loss.

[0067] 6. This invention establishes a baseline database of physiological parameters for drivers of different age groups by dynamically adjusting the baseline value for fatigue assessment. It retrieves the corresponding baseline data based on the driver's identification code and then calculates the fatigue correction based on the ratio of actual driving time to planned driving time. This allows for a more accurate assessment of the fatigue state of different drivers, solving the problem of traditional methods ignoring individual differences. This improves the sensitivity of fatigue driving detection, especially during long-distance driving, and better adapts to the physiological differences of different drivers, reducing misjudgments. Attached Figure Description

[0068] To more clearly illustrate the technical solutions in this invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0069] Figure 1 This is a flowchart of the steps of the method of the present invention;

[0070] Figure 2 This is a flowchart illustrating the steps of historical data interpolation and reconstruction in the method of the present invention.

[0071] Figure 3 This is a flowchart illustrating the steps involved in setting the dynamic threshold in the method of this invention. Detailed Implementation

[0072] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. It should also be noted that, to make the embodiments more comprehensive, the following embodiments are the best and preferred embodiments, and those skilled in the art can use other alternative methods to implement some well-known technologies; moreover, the accompanying drawings are only for more specific description of the embodiments and are not intended to specifically limit the present invention.

[0073] Please see Figures 1-3 This invention provides an intelligent identification method for illegal behaviors of commercial vehicles based on multi-source data fusion. In step 1, the on-board terminal collects the vehicle's location information (GPS coordinates), driving speed, steering wheel angle change rate, and driver identification code in real time. This information is transmitted to the central control system via communication protocols such as CAN bus. The vehicle's real-time location and driving speed provide basic data for judging illegal behaviors (such as speeding, fatigue driving, etc.).

[0074] By using roadside monitoring equipment (such as cameras and radar), continuous video frames containing the target vehicle are captured. These video frames are used to identify driver behavior (such as eye opening and closing, steering wheel angle, etc.) and can be matched with onboard data.

[0075] The system connects to the transportation administration database in real time to obtain the registration information of the target vehicle, including the sequence of key points on the vehicle's operating route, the planned driving time threshold, and the speed limit type of the road segment. This information is crucial for the analysis of illegal activities.

[0076] Vehicle positioning data and roadside monitoring equipment data need to be processed under a unified coordinate system. By using a spatiotemporal reference normalization method and a geographic coordinate transformation algorithm, the coordinate systems of the vehicle positioning data and the roadside equipment are unified to ensure data accuracy. Furthermore, a clock synchronization protocol (such as NTP) is used to synchronize the timestamps of the vehicle terminal and the monitoring equipment to avoid data misalignment caused by time errors.

[0077] In step 2, computer vision technology is used to extract the driver's eye opening / closing rate of change (reflecting fatigue) and the steering wheel angle offset from the video frames. The steering wheel angle change rate is constructed as a time series to reflect the driver's operating habits and abnormal behaviors. License plate recognition technology is also used to ensure consistency between the video and the vehicle data.

[0078] When the rate of change in eye opening and closing is detected to be lower than a preset threshold, the system will verify the driver's actions. If the steering wheel angle deviation exceeds the set dynamic range (reflecting abnormal driving operations, such as sharp turns), the system will further check whether the driver has exceeded the registered driving time threshold and dynamically adjust the speeding judgment threshold by matching the road speed limit type with real-time location data.

[0079] Based on different types of violations (such as fatigued driving, speeding, and abnormal operation), the system assigns different weights to video frames, abnormal operation probability values, and rule matching degrees. Through weighted fusion, this information generates a final fusion confidence score, which is used for the final violation determination.

[0080] In step 3, when the fusion confidence level exceeds a certain type of related alarm threshold, the system will generate and push a tiered alarm command. Different violations (such as speeding, fatigued driving, and abnormal operation) will trigger different levels of alarms, facilitating subsequent law enforcement processing.

[0081] The system optimizes the abnormal operation probability calculation model based on law enforcement feedback data (such as violations confirmed by traffic police), and adjusts the abnormal behavior detection strategy and weight allocation to make the system more accurate in practical applications.

[0082] By employing techniques such as spatiotemporal alignment of multi-source data, extraction and fusion of behavioral features, and dynamic adjustment of judgment thresholds, this technology solves problems such as difficulty in multi-source data fusion, inaccurate behavior recognition, and insufficient dynamic adjustment in existing technologies, significantly improving the accuracy and reliability of intelligent identification of illegal behaviors of operating vehicles.

[0083] In one possible implementation, to ensure time consistency among devices in the system, the provincial monitoring platform distributes standard time signals to vehicle terminals and roadside equipment via the Network Time Protocol (NTP). NTP is a time synchronization protocol based on computer networks, capable of transmitting precise time information to devices via the internet or local area network.

[0084] Because network communication involves latency, simple clock synchronization may not be sufficient to resolve time errors caused by network delays. Therefore, the system employs a multi-level latency compensation mechanism. First, it records the signal transmission delay at each device. Then, it uses an algorithm to compensate for the time deviation of each device, ultimately controlling the timestamp error within the millisecond range. This process ensures consistent time accuracy across devices, providing an accurate time reference for subsequent data matching and analysis.

[0085] By calculating the timestamp correction amount for each device, the system ensures that all collected data can be synchronized at the same point in time. Through this correction process, the system avoids data alignment errors caused by inconsistent device times, thus improving the overall spatiotemporal accuracy of the system.

[0086] During coordinate transformation, the system selects the measured coordinates of highway toll stations as control points. Toll stations typically possess high geographic accuracy, and their coordinate information is already registered in national geodetic coordinate systems (such as the WGS-84 coordinate system). Therefore, the measured coordinates of the toll stations serve as the reference standard throughout the entire coordinate transformation process.

[0087] The system calculates the coordinate transformation parameters by comparing the original coordinates from the vehicle's GPS system with the measured coordinates from the highway toll station. These parameters include the scale factor, rotation angle, and offset, which are used to convert the vehicle's GPS coordinates into standard coordinates in the national geodetic coordinate system.

[0088] To improve the accuracy of coordinate transformation, the system uses the least squares method for iterative optimization. By continuously adjusting the transformation parameters, the least squares method reduces the transformation residual (i.e., the error between the transformed coordinates and the actual coordinates) to within a set threshold. This method ensures that the error during the transformation process is minimized, thereby improving positioning accuracy and data consistency.

[0089] Through precise clock synchronization and coordinate transformation processes, not only is the accuracy of data synchronization improved, but the overall performance of the intelligent identification system for illegal behavior of operating vehicles is also significantly enhanced, providing strong support for intelligent traffic management in practical applications.

[0090] In one possible implementation, the driver's facial region is first located in the video frame using a face detection algorithm. This process typically uses deep learning-based convolutional neural networks (CNNs), such as face detection models provided by libraries like OpenCV or Dlib. The network identifies key facial feature points, such as the positions of the eyes, nose, and mouth. For the eye region, especially the left and right eyes, the CNN further extracts the coordinates of key eye points, such as the corners of the eyes and the edges of the eyelids. These coordinates provide accurate positional data for subsequent calculations of eye opening and closing.

[0091] Next, the system extracts changes in eye opening and closing by calculating the rate of change in interpupillary distance (IPD) across consecutive frames. IPD typically refers to the vertical distance between the upper and lower eyelids, reflecting the open and closed state of the eyes. By analyzing the eye region in each frame, the system can calculate IPD in real time and track its changes. By comparing IPD data from multiple consecutive video frames, the rate of change in eyelid opening and closing can be calculated. This data provides important information for determining whether a driver is fatigued.

[0092] In low-light conditions, especially at night or in situations with uneven lighting inside the driver's cabin, the system activates an infrared supplemental lighting mechanism. When the system detects that the ambient light intensity is below a set threshold, it automatically activates the infrared supplemental lighting device to enhance the clarity of the eye image. Infrared supplemental lighting does not interfere with the driver's vision and improves the accuracy of key eye point detection in low-light environments, ensuring accurate calculation of the eyelid distance change rate.

[0093] The fatigue index is calculated based on the driver's blinking frequency and the duration of eyelid closure. The system uses a sliding window method, counting the number of frames with the driver's eyelids fully closed in 30-second time windows. When the percentage of closed frames within a given time window exceeds a dynamic threshold, the system determines that the driver is fatigued. The dynamic threshold is dynamically adjusted based on actual conditions and can be optimized according to the driver's driving habits, lighting conditions, and vehicle speed.

[0094] Specifically, the system performs sliding window processing on every 30-second video frame, calculating the number of frames in each window where the eyelids are completely closed. If the number of frames with completely closed eyelids exceeds a preset threshold within a certain window, it indicates that the driver has a high risk of fatigue.

[0095] Based on the proportion of eyelid closure frames, if the proportion exceeds a dynamic threshold, the driver is preliminarily determined to be fatigued. The dynamism of this determination process ensures that the system can adapt to different drivers and environmental conditions.

[0096] Through precise eye detection, infrared illumination enhancement, and fatigue index calculation, the system can not only efficiently monitor driver fatigue in complex environments, but also flexibly adjust based on dynamic thresholds, effectively improving the safety and reliability of the intelligent identification system for illegal behaviors of commercial vehicles.

[0097] In one possible implementation, to accurately identify abnormal driving behaviors of operating vehicles, the system first constructs a dual-channel time series model. This model uses the rate of change of steering wheel angle and the amount of brake pedal travel as input data. By monitoring these two key driving behaviors, it can effectively determine whether the driver's operation involves abnormal behaviors such as sudden turns, sudden braking, or sudden acceleration.

[0098] In Channel 1, the input is the first-order difference sequence of the steering wheel angle change rate. This data reflects the abrupt changes in the driver's steering wheel operation and can capture the instantaneous changes during sharp turns. When the driver makes a sharp turn, the steering wheel angle change rate increases rapidly and exhibits significant fluctuations. The first-order difference of this change rate helps extract key features during sharp turns.

[0099] In the second channel, the input is the sliding variance sequence of the brake pedal travel. This data reflects abrupt changes during braking and can effectively capture the characteristics of emergency braking. The sliding variance sequence reflects the fluctuations in the force applied by the driver when applying the brakes. If emergency braking occurs, the brake pedal travel will fluctuate significantly, and this change can be accurately captured by calculating the sliding variance.

[0100] By using one-dimensional convolutional layers, the system can extract local abrupt change features from time series data. The convolutional layer slides across the one-dimensional time series data, and through combinations of different convolutional kernels, it can efficiently capture local variation features related to abnormal operation in steering wheel angle and brake pedal travel. Using this method, the system can quickly identify local abrupt change points in abnormal operating behavior, providing support for subsequent classification and judgment.

[0101] After local features are extracted in the convolutional layer, the data is processed through a fully connected layer, and finally, the Softmax function outputs the probability distribution of three possible abnormal operation categories: sharp turn, sudden braking, and sudden acceleration. This output layer quantifies the likelihood of different types of operations by using probability values, thereby enabling the identification of multiple types of abnormal operating behaviors. The Softmax function can generate probability values ​​for each type of operation based on different combinations of input features. These probability values ​​provide the system with a basis for determining whether the driver has committed a violation.

[0102] When the sampling interval of the vehicle-mounted sensors increases abnormally, it may be due to system malfunction or sensor problems, leading to discontinuities or anomalies in the collected data. In this situation, the system will activate a historical data interpolation mechanism to ensure data integrity and continuity. Through interpolation methods (such as linear interpolation, spline interpolation, etc.), the system can reconstruct the missing sequence based on existing historical data and restore the normal data flow. This step ensures that even with longer sampling intervals or data loss, the system can still make reasonable inferences and identifications based on historical data, avoiding erroneous identifications due to data loss.

[0103] By employing dual-channel time series models, convolutional layer feature extraction, and historical data interpolation, the system's accuracy, real-time performance, and robustness have been improved. This enables the system to effectively identify abnormal operating behaviors of commercial vehicles, providing crucial technical support for intelligent transportation systems.

[0104] In one possible implementation, the trajectories of vehicles passing through the toll station are first selected as control point pairs. The selection of these control points is crucial because their coordinates have known true values, typically derived from authoritative Geographic Information System (GIS) data with high accuracy. These control points are used during the transformation process to correct the GPS trajectory data, ensuring accurate conversion from the GPS coordinate system to the CGCS2000 coordinate system via a seven-parameter transformation.

[0105] A seven-parameter transformation model is used to convert from the GPS coordinate system to the CGCS2000 coordinate system. The mathematical formula for this model is: ;in, : Coordinate values ​​in the transformed CGCS2000 coordinate system (including x, y, z coordinates). : Coordinate values ​​in the original GPS coordinate system (also including coordinate). Translation parameter: Represents the translation offset between two coordinate systems. It is the relative offset from the GPS coordinate system to the CGCS2000 coordinate system. The rotation matrix represents the rotation relationship from the GPS coordinate system to the CGCS2000 coordinate system. Due to the curvature of the Earth and the definitions of different coordinate systems, the rotation matrix reflects the rotation angle between the two coordinate systems in space. Scale factor: Represents the scale change during coordinate transformation. The scale factor adjusts the scaling ratio of the two coordinate systems in space.

[0106] To solve for the translation parameters, rotation matrix, and scale factor, the least squares method is used. The goal of the least squares method is to minimize the sum of squares of the residuals of all control points by fitting the control point pairs (i.e., points with known coordinates). The residual of each control point represents the difference between the control point's true coordinates in the target coordinate system after coordinate transformation.

[0107] The specific steps are as follows:

[0108] Assuming initial translation parameters X rotation matrix and scale factor These are the initial estimates. Using the seven-parameter transformation model described above, the transformed coordinates of each control point are calculated using the currently estimated parameters. For each control point, the difference between its position in the transformed coordinate system and its known true position is calculated to obtain the residual. The translation parameters are adjusted through iterative optimization. Rotation matrix and scale factor The goal is to minimize the sum of squared residuals at all control points. The iteration stops when the sum of squared residuals falls below a set convergence threshold, yielding the final transformation parameters.

[0109] Through multiple iterations, the translation parameters are finally obtained. Rotation matrix and scale factor The accurate values. These parameters can be used to accurately convert GPS coordinate system data to the CGCS2000 coordinate system, thereby improving the accuracy of vehicle trajectory recognition.

[0110] This coordinate transformation method enables more accurate trajectory recognition of operating vehicles and detection of illegal activities, improving the overall performance and reliability of the system.

[0111] In one possible implementation, mountainous road sections typically have significant elevation changes, thus vehicles are greatly affected by road gradients when traveling on such roads. To avoid misjudging speeding when vehicles are climbing steep inclines, the system incorporates an elevation change rate determination mechanism. When a vehicle's elevation change rate continuously exceeds a set gradient threshold, it indicates that the vehicle is climbing an incline. At this point, the system automatically adjusts the speeding threshold, lowering it by a fixed percentage to allow vehicles to travel at speeds exceeding the standard speed limit on flat roads in these special road sections.

[0112] For example, on mountain roads with steep inclines, the speed limit for vehicles might be adjusted to 1.1 times that of flat roads (e.g., from 60 km / h to 66 km / h) to avoid misjudgments caused by steep inclines. This strategy effectively avoids misjudgments.

[0113] On highways, vehicles can typically travel at higher speeds, therefore, the standards for determining speeding are relatively strict. The system continuously monitors the mileage of speeding states to determine if there is sustained speeding behavior. If a vehicle exceeds the speed limit for a certain period of time (such as a certain mileage) on a highway and no deceleration behavior (such as braking or speed reduction) is detected, the system will trigger an alarm.

[0114] The dynamic threshold is set to prevent false alarms caused by occasional speed fluctuations. For example, a vehicle may occasionally slightly exceed the speed limit on a highway due to factors such as traffic flow. However, if this behavior continues for a period of time without slowing down, the system will consider the behavior to meet the characteristics of a speeding violation and issue a warning.

[0115] Traffic flow density has a significant impact on vehicle speed. Under high traffic flow density, vehicles may slow down due to congestion or be more easily obstructed, making speeding less likely; while under low traffic flow density, vehicles may accelerate. Therefore, the system's speeding threshold should be dynamically adjusted based on real-time traffic flow density.

[0116] For example, when traffic density is low, the speeding threshold may be appropriately increased to allow vehicles more room to accelerate; while when traffic density is high, the threshold will be lowered accordingly to avoid speeding due to poor traffic conditions.

[0117] The system can adjust the speeding threshold by acquiring real-time traffic flow data on road segments (such as through onboard equipment, traffic monitoring sensors, or real-time traffic information of road segments). Specifically, the dynamic threshold adjustment can be calculated using an adaptive algorithm based on traffic flow to ensure that the threshold adjustment reflects the actual driving conditions on the current road.

[0118] By dynamically adjusting the speeding threshold, this strategy can more accurately and efficiently identify speeding violations by commercial vehicles and provide more suitable judgment criteria, further improving the accuracy and application value of intelligent identification methods.

[0119] In one possible implementation, the video confidence weight is primarily calculated based on the segmentation completeness of the license plate recognition region and the illumination uniformity score. The segmentation completeness of the license plate recognition region refers to whether the license plate area is accurately segmented from the background during image processing. A higher segmentation completeness indicates more accurate image processing, higher video confidence, and a correspondingly larger weight. Furthermore, the illumination uniformity score reflects the lighting conditions during video capture; good illumination uniformity helps improve the accuracy of license plate recognition. Therefore, the video confidence weight calculation formula can combine these two indicators for weighted average to improve the credibility of the video information.

[0120] Sensor sampling frequency stability is used to measure the reliability of sensor data. A normal sensor should operate stably, while frequent fluctuations in the sampling frequency may indicate a sensor malfunction or data anomalies. The abnormal operation probability weight identifies potential abnormal behavior by monitoring the stability of the sensor sampling frequency. If the sensor sampling frequency is stable, the weight is high, and the system can trust the sensor data; if the sampling frequency is unstable, the weight is low, indicating that the data source may be unreliable.

[0121] The rule matching weight is calculated based on the overlap rate between the actual trajectory and the key points of the registered route. The registered route is a standard route designed according to regulations or company specifications, and the vehicle's trajectory should roughly match the registered route. By comparing the overlap rate of the key points of the actual driving trajectory and the registered route, a higher overlap rate indicates that the vehicle's behavior meets expectations, resulting in a higher weight; a lower overlap rate indicates that the vehicle's driving trajectory deviates significantly from the standard route, resulting in a lower weight.

[0122] Environmental visibility and road curvature are crucial factors influencing vehicle behavior. For example, in low visibility conditions, drivers may slow down to ensure safety, which can affect the vehicle's speeding detection; on curved roads, the vehicle's trajectory may deviate from a straight line. Therefore, the weighting function dynamically adjusts the weights of various parameters based on real-time environmental visibility and road curvature. When visibility is poor or the road is highly curved, the system may reduce the weight of video and trajectory matching and increase the weight of sensor data to ensure accuracy in these special environments.

[0123] The weight allocation strategy not only improves the system's adaptability to different data sources, but also enhances the system's accuracy and stability in complex environments, demonstrating significant practical value.

[0124] In one possible implementation, during vehicle operation, sensors may lose data due to environmental interference, equipment malfunction, or other reasons. The system first monitors the integrity of the sensor data, and when it detects data loss within a certain time period, it triggers a data interpolation and reconstruction process.

[0125] To address the issue of sensor data loss, the system first extracts driving operation pattern libraries for similar vehicles on the same road sections from a historical database, based on factors such as vehicle type and road segment. These pattern libraries contain time-series data of typical vehicle behaviors under the same conditions, such as acceleration, braking, and turning.

[0126] Once the driving operation pattern library is extracted, the system uses the Dynamic Time Warping (DTW) algorithm to compare the temporal features of the currently missing data with pattern fragments from historical data. The DTW algorithm is used to calculate the similarity between two sets of time-series data, and it can handle temporal distortions and differences in step size between the data. Using this algorithm, the system can find the most similar historical operation pattern fragment as a reference for filling in the missing data.

[0127] By matching the most similar pattern fragment, the system extracts the time-series data of that fragment and fills in the missing data intervals in chronological order. The filled data is labeled as interpolated data to distinguish it from the real data.

[0128] After the interpolated data is populated, the system will incorporate it into subsequent probability calculations to participate in the intelligent identification of vehicle violations. The interpolated data will be used in conjunction with other data sources to calculate the probability of violations based on a preset weighting strategy, thereby ensuring the completeness and accuracy of the identification system.

[0129] By effectively filling the gaps in sensor data, the system's continuity, accuracy, robustness, and recognition precision are improved, making it of significant practical value for the intelligent identification of illegal behaviors of commercial vehicles.

[0130] In one possible implementation, the system first needs to establish a baseline database of driver physiological parameters, recording the normal physiological parameters of drivers of different ages. An important physiological parameter is blink frequency, a crucial indicator for assessing fatigue levels. The blink frequency of drivers in a normal state varies across age groups. The system establishes baselines of normal blink frequencies for different age groups through large-scale data collection and statistical analysis. These baseline data serve as reference standards for subsequent fatigue identification.

[0131] When a driver begins driving, the system identifies the current driver using a driver identification code (such as a driver's license number, vehicle ID, or other authentication method). Based on this identification information, the system retrieves normal blink frequency data corresponding to the driver's age group from a baseline database. This baseline data reflects the physiological characteristics of drivers in that age group under normal conditions, providing a reference for subsequent fatigue assessment.

[0132] To accurately assess driver fatigue, the system incorporates a real-time fatigue correction factor in addition to the baseline blink frequency. This correction is calculated based on the ratio of continuous driving time to planned driving time. If a driver has been driving continuously for a period exceeding the planned driving time, the system calculates the correction based on the ratio of actual driving time to planned driving time. If the actual driving time significantly exceeds the planned time, it indicates a higher level of driver fatigue, and the correction factor will increase accordingly, further adjusting the dynamic threshold.

[0133] For example, if the continuous driving time is 8 hours and the planned driving time is 6 hours, the correction amount can be increased proportionally to reflect the driver's potential fatigue, which affects the normal range of blink frequency. This dynamic adjustment mechanism allows the system to more accurately identify the driver's fatigue based on actual driving conditions.

[0134] By overlaying real-time fatigue corrections onto baseline data, the system obtains a dynamic threshold used to determine whether the driver is experiencing fatigue. When the driver's blinking frequency falls below this dynamic threshold, the system can determine that the driver may be fatigued, triggering appropriate warnings or taking other safety measures.

[0135] The dynamic threshold setting method based on driver physiological parameters, by combining factors such as age group and real-time driving duration, can accurately determine the driver's fatigue state, improve driving safety and the system's adaptability, and is an effective intelligent recognition technology.

[0136] Accordingly, embodiments of the present invention also provide an intelligent identification system for illegal acts of operating vehicles based on multi-source data fusion, including a memory configured to store instructions, a processor configured to call the instructions from the memory, and capable of implementing an intelligent identification method for illegal acts of operating vehicles based on multi-source data fusion as described in any embodiment of the present invention when executing the instructions.

[0137] The following examples will illustrate this in detail:

[0138] This invention relates to an intelligent fatigue driving detection system based on physiological data monitoring and multi-source data fusion, particularly applicable to heavy-duty trucks or public transportation vehicles that travel long distances. The system collects driver physiological data, vehicle status data, and environmental data to monitor driver fatigue levels in real time and uses a feedback mechanism to remind drivers to rest, thus preventing accidents.

[0139] The embodiments of this invention are configured with the following system hardware and software:

[0140] Eye movement sensor: Used to monitor the driver's eye movements in real time, with a sampling frequency of 30Hz and an eyelid opening and closing state detection accuracy of 95%.

[0141] Accelerometer: Installed in the driver's seat, with a sampling frequency of 100Hz, it is used to detect the driver's motion state (such as frequent head movements).

[0142] Vehicle speed sensor: Installed on the vehicle, with a sampling frequency of 10Hz, to collect the vehicle's speed.

[0143] GPS sensor: Used to record the vehicle's location, obtain road type and travel time.

[0144] Data acquisition module: continuously collects driver physiological data and vehicle driving status through sensors.

[0145] Data processing module: including signal filtering, data fusion, feature extraction and fatigue assessment, etc.

[0146] Feedback module: It reminds the driver to pay attention to fatigue through in-vehicle display screen, sound alarms and seat vibration.

[0147] Specifically, the eye-tracking sensor collects eye movement data 30 times per second, including eye closure status (time of eye closure) and eye movement amplitude. By judging the eyelid closure time and eye movement frequency, the driver's fatigue level is determined. The accelerometer collects data 100 times per second to detect abnormal neck or head movements in the driver, thereby inferring the possibility of driver fatigue. The vehicle speed and GPS sensor collects data every 10 seconds to obtain vehicle speed and location, combining this with driving time to determine the driver's continuous driving duration.

[0148] Median filtering was used to denoise the eye-tracking sensor signals, removing short-term outliers. Missing values ​​in the accelerometer and eye-tracking data were reconstructed using linear interpolation. The sensor data was normalized, scaling all data to between 0 and 1 for subsequent analysis.

[0149] The Dynamic Time Warping (DTW) algorithm is used to compare the driver's eye movement data at different time periods with eye movement data under normal conditions to determine the degree of fatigue.

[0150] Specifically, regarding the collected eye-tracking data Calculate the distance between each pair of data. .

[0151] Calculate the minimum distance using a recursive formula: ;

[0152] Distance via the minimum path Calculate the fatigue index Fatigue correction calculation formula: Fatigue correction amount Used to correct dynamic thresholds, the calculation formula is: ;in: This is the actual driving time, in hours; This is the driver's planned driving time, in hours.

[0153] Based on fatigue correction amount Dynamically adjust the fatigue judgment threshold .when If the driver exceeds the scheduled rest period, the system lowers the fatigue threshold to detect fatigue earlier.

[0154] When the system starts working, sensors begin collecting real-time physiological data from the driver, vehicle speed, and position. The collected data is preprocessed and standardized, and combined with eye movement frequency, head movement, vehicle speed, and driving duration, the DTW algorithm is used to calculate the fatigue index. If the fatigue index... Exceeding the set threshold And fatigue correction amount If the system detects a fatigue threshold, it will issue a fatigue warning. Warning methods include display screen notifications, audible alarms, and seat vibration alerts. The fatigue threshold is dynamically adjusted based on real-time fatigue correction data to ensure the system adapts to the fatigue characteristics of different drivers.

[0155] An experiment was conducted with 100 long-duration truck drivers, each driving for at least 8 hours. The system collected the drivers' physiological data in real time and assessed their fatigue levels, comparing this data with manually monitored fatigue levels.

[0156] driver The system detects fatigue duration (in hours). Manually measured fatigue duration (in hours) System accuracy (%) A 7.5 7.8 96.1% B 6.8 7.0 97.1% C 8.2 8.5 96.4% D 7.0 6.9 98.5%

[0157] According to the experimental results, the system's accuracy rate is over 96.0%, and compared with traditional fatigue detection methods (such as the eye closure time method), the system can more accurately reflect the driver's fatigue state and provide early warnings.

[0158] Through dynamic time warping algorithms and data fusion technology, the system can accurately assess driver fatigue levels in real time and react promptly. Compared to traditional methods, the system can provide an early warning 30 minutes before driver fatigue occurs, improving the effectiveness of fatigue driving prevention. The system can dynamically adjust the fatigue detection threshold based on each driver's actual driving habits and physiological state, demonstrating strong adaptability.

[0159] Experimental data show that the system of the present invention has high accuracy and strong real-time response capability, which can effectively improve driver safety, prevent fatigue driving accidents, and has great application value.

[0160] This invention encompasses any substitutions, modifications, equivalent methods, and solutions made within the spirit and scope of this invention. To provide the public with a thorough understanding of this invention, specific details are described in detail in the following preferred embodiments; however, those skilled in the art will fully understand the invention even without these details. Furthermore, to avoid unnecessary misunderstanding of the essence of this invention, well-known methods, processes, procedures, components, and circuits are not described in detail.

[0161] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for intelligent identification of illegal behavior of operating vehicles based on multi-source data fusion, characterized in that, The method comprises the following steps: Step 1: Multi-source data spatio-temporal alignment collection: Synchronously collecting real-time positioning coordinates, driving speed, steering wheel angle change rate and driver identity code through a vehicle-mounted terminal; Collecting continuous video frames containing the target vehicle through a roadside monitoring device; Accessing an operation and management database in real time to obtain a vehicle record key point sequence of an operating route, a planned driving time threshold and a road section speed limit type; Performing spatio-temporal reference normalization: converting the vehicle-mounted positioning coordinates and the roadside monitoring device geographic coordinates to a unified coordinate system, and controlling the timestamp error through a clock synchronization protocol; Step 2: Hierarchical fusion decision: Behavior characteristic extraction: extracting the driver eye opening and closing degree change rate, the steering wheel angle offset and the license plate recognition confidence from the video frames, constructing the steering wheel angle change rate into a time series, and outputting an abnormal operation probability value; Multi-level rule verification: when the eye opening and closing degree change rate is continuously lower than a physiological threshold, the steering wheel angle offset is verified to be associated with whether it exceeds a dynamic range, and it is checked whether the current driving time exceeds the record threshold, the road section speed limit type is matched according to the real-time position, and the overspeed judgment threshold is dynamically adjusted; Confidence fusion: according to the weight of the video confidence, the abnormal operation probability value and the rule matching degree according to the type of violation, a fusion confidence is generated; Step 3: Feedback optimization: When the fusion confidence exceeds a type-related alarm threshold, a hierarchical alarm instruction is pushed; Based on the law enforcement feedback data, the abnormal operation probability value calculation model and the weight distribution strategy are updated. 2.The method of claim 1, wherein, The spatio-temporal reference normalization of step 1 comprises: Clock synchronization process: the provincial supervision platform distributes network time protocol clock signals to the vehicle-mounted terminal and the roadside device, calculates the timestamp correction amount through multi-level delay compensation, and controls the timestamp error of all devices within the millisecond level; Coordinate conversion process: taking the measured coordinates of a highway toll station as control points, solving the coordinate system conversion parameters, converting the vehicle-mounted global positioning system original coordinates to the national geodetic coordinate system, and iteratively optimizing the conversion residual to be within the set threshold through the least square method. 3.The method of claim 1, wherein, The extraction of the eye opening and closing degree change rate in step 2 comprises: Positioning the driver's face area in the video frame, and detecting the left and right eye key point coordinates using a convolutional neural network; Calculating the eyelid spacing change rate in consecutive frames, and starting infrared light enhancement when the light intensity is lower than the set threshold; Calculating the fatigue index according to the blink frequency and duration, which comprises the following steps: a: counting the number of continuous frames with completely closed eyelids within a 30-second time window; b: when the continuous frame proportion exceeds the dynamic threshold, it is determined that the driver is in a fatigue state.

4. The method according to claim 1, characterized in that, The generation of the abnormal operation probability value in step 2 comprises: Building a double-channel time series model: Channel one input first-order difference sequence of steering wheel angle change rate; Channel two input sliding variance sequence of brake pedal stroke amount; Extracting local mutation features through a one-dimensional convolution layer, and using a Softmax function in the output layer to generate the probability distribution of three types of operations: sharp turning, sharp braking and sharp acceleration; When the sampling interval of the vehicle-mounted sensor abnormally increases, historical data interpolation is used to reconstruct the sequence.

5. The method according to claim 2, wherein, The parameter solving process of the coordinate conversion comprises: Selecting the coordinates of toll stations through which the vehicle trajectory passes as control points A seven-parameter transformation model is established: ; Solving the translation parameters by least squares , rotation matrix and scale factor , iteratively until all control point residual sum of squares is less than a set convergence threshold. 6.The method of claim 1, wherein, The strategy of dynamically adjusting the overspeed judgment threshold in step 2 includes: Mountainous section judgment: when the vehicle elevation change rate continuously exceeds the set gradient, the overspeed judgment threshold is lowered by a fixed percentage; Highway section judgment: continuously monitor the overspeed state mileage, when the cumulative mileage exceeds the dynamic threshold and no deceleration behavior is detected, trigger the alarm; The dynamic threshold is self-adaptive according to real-time traffic flow density.

7. The method according to claim 1, characterized in that, The weight distribution strategy in step 2 includes: Video confidence weight calculation: based on the segmentation completeness of license plate recognition area and the uniformity of light; Abnormal operation probability value weight calculation: based on the stability index of sensor sampling frequency; Rule matching degree weight calculation: based on the coincidence rate of actual trajectory and key points of recorded route; The weight distribution function is updated in real time according to the environmental visibility and road curvature. 8.The method of claim 4, wherein, The history data interpolation reconstruction includes: When detecting sensor data loss, extract the driving operation mode library of the same vehicle model on the same section; Based on dynamic time warping algorithm to match the most similar mode segment; Fill in the missing interval with the time series data of similar segments and mark it as interpolation data for probability calculation.

9. The method according to claim 3, characterized in that, The setting process of the dynamic threshold includes: Establish a baseline library of driver physiological parameters, store the normal blink frequency of drivers of different ages; According to the current driver identity code, the baseline data of the corresponding age group is retrieved; On the basis of the baseline value, superimpose the real-time fatigue correction amount, which is calculated by the ratio of continuous driving time to planned time.

10. A multi-source data fusion-based intelligent identification system for illegal behavior of a commercial vehicle, characterized in that, A memory configured to store instructions, a processor configured to call the instructions from the memory and to implement the method of claim 1-9 when executing the instructions.

Images