Traffic illegal behavior management and control method, electronic equipment, storage medium and program product

Abstract

The invention provides a traffic illegal behavior management and control method, electronic equipment, a storage medium and a program product, and relates to the field of traffic safety. The method comprises the following steps: acquiring multi-modal data acquired by vehicle-mounted multi-source equipment, wherein the multi-modal data comprises image data, laser point cloud data, millimeter wave point cloud data and positioning data; performing time alignment processing on the multi-modal data to obtain multi-modal time alignment data at the target moment; performing data fusion on the time alignment data by adopting a Kalman filtering algorithm to obtain an operation state of the target vehicle associated with the multi-modal data in a three-dimensional space; and according to the time alignment data and the operation state, identifying whether the target vehicle has traffic violation behaviors. According to the invention, traffic illegal behaviors can be accurately identified.

Description

Technical Field

[0001] This application relates to the field of traffic safety, and in particular to a method for controlling traffic violations, electronic equipment, storage media, and program products. Background Technology

[0002] Traffic safety is closely related to people's lives and property and has received widespread attention. Among the main factors threatening traffic safety, traffic violations are a key factor, and a good traffic safety situation depends on strict control over traffic violations.

[0003] Current traffic violation control primarily relies on manual patrols and fixed cameras. Specifically, fixed cameras collect images of the corresponding areas at their deployment locations, and control traffic violations by identifying whether the images contain such images. However, this method suffers from insufficient accuracy in identifying traffic violations. Summary of the Invention

[0004] This application provides a method, electronic device, storage medium, and program product for controlling traffic violations, which can accurately identify traffic violations and effectively control them.

[0005] In a first aspect, embodiments of this application provide a method for controlling traffic violations, including:

[0006] Acquire multimodal data collected by vehicle-mounted multi-source devices. The multimodal data includes: image data, laser point cloud data, millimeter-wave point cloud data, and positioning data.

[0007] Perform time alignment processing on the multimodal data to obtain time-aligned multimodal data at the target time.

[0008] The Kalman filter algorithm is used to fuse time-aligned data to obtain the running state of the target vehicle in three-dimensional space through multimodal data association.

[0009] Based on time-aligned data and operational status, identify whether the target vehicle has committed any traffic violations.

[0010] In one possible implementation, time alignment processing is performed on the multimodal data to obtain time-aligned multimodal data at a target time, including:

[0011] Determine the target time based on the acquisition timestamps corresponding to the multimodal data;

[0012] Among the modal data contained in the multimodal data, the first data whose corresponding acquisition timestamp is closest to the target time is determined, and based on the first data, the time alignment data of the corresponding modality at the target time is determined;

[0013] Based on the time alignment data of each modality at the target time, the time alignment data of the multimodal at the target time is obtained.

[0014] In one possible implementation, determining the time alignment data of the corresponding mode at the target time based on the first data includes:

[0015] If the time difference between the collection timestamp corresponding to the first data and the target time is less than the time threshold, then the first data is determined to be the time-aligned data of the corresponding modality at the target time.

[0016] If the time difference between the collection timestamp corresponding to the first data and the target time is greater than the time threshold, then in the data of the corresponding modality, the second data collected at the first time and the third data collected at the second time are determined. The first time is the collection time that is before the target time and closest to the target time, and the second time is the collection time that is after the target time and closest to the target time. Based on the second data and the third data, the time-aligned data of the corresponding modality at the target time is obtained by linear interpolation.

[0017] In one possible implementation, a Kalman filter algorithm is used to fuse time-aligned data to obtain the target vehicle's operational state in three-dimensional space through multimodal data association, including:

[0018] Based on time-aligned data, obtain the operational status observations of the target vehicle;

[0019] Use the following Kalman filter state prediction formula to obtain the predicted operating state of the target vehicle:

[0020]

[0021] in, X is the predicted operating state value of the target vehicle. t-1 Let F be the operating state of the target vehicle at the previous time step, B be the state transition matrix, and u be the control input matrix. t-1 For control input;

[0022] Based on the predicted operating state of the target vehicle and the Kalman filter state update formula, the operating state of the target vehicle in three-dimensional space is determined:

[0023]

[0024] Among them, X t This refers to the operational state of the target vehicle in three-dimensional space. K is the predicted value of the operating state of the target vehicle. t For Kalman gain, Z tH represents the observed operating status of the target vehicle, and H is the observation matrix.

[0025] In one possible implementation, identifying whether a target vehicle has committed a traffic violation based on time-aligned data and operational status includes:

[0026] Time-aligned data and operational status are input into the traffic violation recognition model to identify traffic violations and obtain the confidence level of the target vehicle for different traffic violations.

[0027] The training samples for the traffic violation recognition model include: multimodal data samples corresponding to the original traffic scene and enhanced samples obtained by data augmentation of the multimodal data samples. The enhanced samples include images after adding noise to the original traffic scene images and / or images after motion blurring of the original traffic scene images.

[0028] In one possible implementation, the traffic violation identification model includes a feature extraction module and a traffic violation identification module; wherein:

[0029] The feature extraction module is used to extract the first feature of the image data in the time-aligned data, which includes the edge, texture, and contour of the target vehicle; extract the second feature of the laser point cloud data in the time-aligned data, which includes the three-dimensional shape, spatial distribution of the point cloud, and distance of the target vehicle; extract the third feature of the millimeter-wave point cloud data and positioning data in the time-aligned data, which includes the speed, acceleration, relative distance, and heading angle of the target vehicle; and stitch and normalize the first, second, and third features to obtain the running state features corresponding to the time-aligned data.

[0030] The traffic violation identification module is used to identify traffic violations based on operational status characteristics and operational status, and to obtain the confidence level of different traffic violations corresponding to the target vehicle.

[0031] In one possible implementation, it also includes:

[0032] After obtaining the operating status of the target vehicle, the operating status is transmitted to the traffic violation recognition model through the ROS2 robot operating system message system.

[0033] In one possible implementation, it also includes:

[0034] When a target vehicle commits a traffic violation, the category of the traffic violation and the data related to the target vehicle contained in the multimodal data are packaged to generate an evidence chain;

[0035] The evidence chain was encrypted and uploaded to the traffic management platform.

[0036] In one possible implementation, it also includes:

[0037] Before encrypting the evidence chain and uploading it to the traffic management platform, the multimodal data contained in the evidence chain is checked for consistency to ensure the integrity of the evidence chain.

[0038] Secondly, embodiments of this application provide a traffic violation control device, comprising:

[0039] The acquisition module is used to acquire multimodal data collected by the vehicle-mounted multi-source device. The multimodal data includes: image data, laser point cloud data, millimeter-wave point cloud data, and positioning data.

[0040] The processing module is used to perform time alignment processing on multimodal data to obtain time-aligned multimodal data at the target time.

[0041] The fusion module is used to perform data fusion on time-aligned data using the Kalman filter algorithm to obtain the running status of the target vehicle in three-dimensional space associated with multimodal data.

[0042] The identification module is used to identify whether a target vehicle has committed traffic violations based on time-aligned data and operating status.

[0043] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0044] The memory stores instructions that the computer executes;

[0045] The processor executes computer execution instructions stored in memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0046] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0047] Fifthly, embodiments of this application provide a computer program product, including a computer program, which, when executed, implements the first aspect and / or various possible implementations of the first aspect.

[0048] The traffic violation control method, electronic device, storage medium, and program product provided in this application acquire multimodal data collected by in-vehicle multi-source devices. This multimodal data includes image data, laser point cloud data, millimeter-wave point cloud data, and positioning data. The multimodal data undergoes time alignment processing to obtain time-aligned multimodal data at the target time, ensuring temporal consistency before data processing. A Kalman filter algorithm is used to fuse the time-aligned data, obtaining the target vehicle's operational status in three-dimensional space associated with the multimodal data. Based on the time-aligned data and operational status, the method identifies whether the target vehicle has committed a traffic violation. The traffic violation control method provided in this application fully utilizes in-vehicle multi-source devices to fuse and analyze multimodal data, achieving accurate identification of traffic violations by target vehicles. Attached Figure Description

[0049] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0050] Figure 1 A flowchart illustrating the traffic violation control method provided in this application embodiment. Figure 1 ;

[0051] Figure 2 A flowchart illustrating the traffic violation control method provided in this application embodiment. Figure 2 ;

[0052] Figure 3 A schematic diagram illustrating the training and deployment verification process of the traffic violation recognition model provided in this application embodiment;

[0053] Figure 4 A schematic diagram of the traffic violation control device provided in this application;

[0054] Figure 5 A schematic diagram of the structure of the electronic device provided in this application.

[0055] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0056] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0057] Traffic safety has always been a widely concerned issue. Current technologies for controlling traffic violations rely on manual inspections or the installation of fixed cameras. Manual inspections require manual identification and collection of evidence, which is inefficient and prone to misjudgments. Using fixed cameras to capture images of specific areas and employing image recognition technology to identify traffic violations suffers from insufficient accuracy. Therefore, a more efficient and accurate traffic violation control solution is urgently needed.

[0058] To address the aforementioned technical problems, this application provides a traffic violation control method that fully utilizes vehicle-mounted multi-source devices for traffic violation control. This method acquires multimodal data collected by the vehicle-mounted multi-source devices, including image data, laser point cloud data, millimeter-wave point cloud data, and positioning data. The multimodal data undergoes time alignment processing to obtain time-aligned multimodal data at the target time, ensuring time consistency of the data before processing. A Kalman filter algorithm is used to fuse the time-aligned data, obtaining the target vehicle's operational status in three-dimensional space associated with the multimodal data, thus improving the accuracy of vehicle operational status calculation. Finally, based on the time-aligned data and operational status, the method accurately identifies the target vehicle's traffic violations.

[0059] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0060] Figure 1 A flowchart illustrating the traffic violation control method provided in this application embodiment. Figure 1 In some implementations, the traffic violation control method provided in this application can be executed on a processor with data processing capabilities, such as a vehicle control unit or a smart cockpit controller. Figure 1 As shown, the method includes:

[0061] S101. Acquire multimodal data collected by the vehicle-mounted multi-source device. The multimodal data includes: image data, laser point cloud data, millimeter-wave point cloud data, and positioning data.

[0062] Among them, vehicle-mounted multi-source equipment includes vehicle-mounted cameras, millimeter-wave radar, lidar, positioning modules, etc. Using vehicle-mounted multi-source equipment as distributed monitoring nodes can break through the geographical limitations of fixed cameras and make the coverage of traffic violation control wider.

[0063] Multimodal data includes: image data, laser point cloud data, millimeter-wave point cloud data, and positioning data. For example, image data is acquired by an onboard camera, laser point cloud data is acquired by a lidar, millimeter-wave point cloud data is acquired by a millimeter-wave radar, and positioning data is acquired by a positioning module.

[0064] S102. Perform time alignment processing on the multimodal data to obtain time-aligned multimodal data at the target time.

[0065] In actual operation of in-vehicle multi-source devices, the data acquisition cycles of each device may differ. For example, an in-vehicle camera may acquire an image every 20ms, while a millimeter-wave radar may acquire millimeter-wave point cloud data every 30ms. Time alignment processing of multimodal data can ensure the consistency of multimodal data in time, avoiding recognition errors caused by time misalignment of multimodal data. Using time-aligned data can improve the accuracy of traffic violation recognition.

[0066] S103. Using the Kalman filter algorithm, time-aligned data are fused to obtain the running status of the target vehicle in three-dimensional space through multimodal data association.

[0067] Kalman filtering is a state estimation algorithm. In the actual process of acquiring multimodal data from multiple sources, the measurement errors inherent in the multi-source devices or environmental factors may prevent the obtained multimodal data from accurately representing the target vehicle's operating state. Therefore, it is necessary to use the Kalman filtering algorithm based on time-aligned data to estimate the accurate operating state of the target vehicle in three-dimensional space. Furthermore, the target vehicle's operating state in three-dimensional space can be derived from the various sensor data contained within the multimodal data. For example, the target vehicle's speed information can be obtained from laser point cloud data, millimeter-wave point cloud data, and positioning data; similarly, the target vehicle's position information can be obtained from image data, laser point cloud data, millimeter-wave point cloud data, and positioning data. Fusing multimodal data yields a more accurate vehicle operating state than data from a single sensor.

[0068] S104. Based on time-aligned data and operating status, identify whether the target vehicle has committed any traffic violations.

[0069] The vehicle operating status includes data representing the vehicle's speed, acceleration, and other parameters. In addition to the target vehicle's operating status, the time-aligned data also includes other features related to the target vehicle, such as its edges, textures, contours, 3D shape, spatial distribution, relative distance, and heading angle.

[0070] Therefore, in this embodiment, multimodal data is first fused to achieve multi-source observation fusion of the same physical quantity to obtain accurate vehicle operating status. Then, when identifying traffic violations, time-aligned data and operating status are used together to identify the vehicle's traffic violations. This can obtain richer and higher-dimensional vehicle operating features, thereby making the identification of traffic violations more accurate.

[0071] The traffic violation control method provided in this application expands the scope of traffic violation control by utilizing multi-modal data collected by vehicle-mounted multi-source devices. It performs time alignment processing on the multimodal data to obtain time-aligned data of the multimodal data at the target time, ensuring time consistency of the data before processing and avoiding identification errors due to time misalignment of the multimodal data. A Kalman filter algorithm is used to fuse the time-aligned data, obtaining accurate operational status of the target vehicle in three-dimensional space associated with the multimodal data. The time-aligned data and operational status provide rich and accurate operational characteristics of the target vehicle, enabling accurate identification of traffic violations by the target vehicle based on these data.

[0072] Figure 2 A flowchart illustrating the traffic violation control method provided in this application embodiment. Figure 2 .like Figure 2 As shown, in one possible implementation, time alignment processing is performed on the multimodal data to obtain time-aligned multimodal data at a target time, including:

[0073] S201. Determine the target time based on the acquisition timestamps corresponding to the multimodal data.

[0074] Each in-vehicle multi-source device, such as a camera, LiDAR, millimeter-wave radar, and positioning module, matches a high-precision timestamp to each frame of data during data acquisition. The timestamps are derived from a unified system clock, such as GPS timing or the vehicle's master clock. However, because the data acquisition cycles and frequencies of in-vehicle multi-source devices differ, it is impossible to obtain modal data at the same moment. Through time alignment processing, multimodal data with the same or similar times can be combined to form time-aligned data.

[0075] The specific time alignment process is as follows: Based on the collection timestamps corresponding to the currently collected multimodal data, a target time is determined. Using the target time as a reference, data with timestamps that are the same as or similar to the target time are found from the multimodal data to form time-aligned data.

[0076] S202. Among the modal data included in the multimodal data, determine the first data whose corresponding acquisition timestamp is closest to the target time, and determine the time alignment data of the corresponding modality at the target time based on the first data.

[0077] For example, the acquisition time of image data in multimodal data is determined as the target time, and laser point cloud data, millimeter-wave point cloud data and positioning data corresponding to timestamps that are the same as or close to the target time are searched.

[0078] S203. Based on the time alignment data of each modality at the target time, obtain the time alignment data of the multimodal at the target time.

[0079] In some implementations, after acquiring image data, laser point cloud data, millimeter-wave point cloud data, and positioning data corresponding to timestamps that are the same as or close to the target time, it is necessary to determine whether each data satisfies the time alignment condition. For example, the time difference between the timestamp of each data and the target time can be determined. If the time alignment condition is not met, multimodal data close to the target time can be estimated by linear interpolation.

[0080] The traffic violation control method provided in this application considers the acquisition time error of each data in the multimodal data before identifying traffic violations based on multimodal data. It performs time alignment processing on the multimodal data to ensure the consistency of the multimodal data in time. For the judgment of traffic violations, ensuring the consistency of the time of the multimodal data is conducive to improving the judgment accuracy and avoiding missed judgments and misjudgments.

[0081] In one possible implementation, determining the time alignment data of the corresponding mode at the target time based on the first data includes:

[0082] If the time difference between the acquisition timestamp corresponding to the first data and the target time is less than a time threshold, then the first data is determined to be the time-aligned data of the corresponding modality at the target time. If the time difference between the acquisition timestamp corresponding to the first data and the target time is greater than a time threshold, then the data of the corresponding modality is determined to include the second data acquired at the first time and the third data acquired at the second time. The first time is the acquisition time before the target time and closest to the target time, and the second time is the acquisition time after the target time and closest to the target time. Based on the second data and the third data, the time-aligned data of the corresponding modality at the target time is obtained by linear interpolation.

[0083] Exemplarily, the acquisition time of the image data in the multimodal data is determined as the target time, and the multimodal data corresponding to the same or similar timestamps as the target time is searched for as the first data. The first data includes the image data, the laser point cloud data, the millimeter-wave point cloud data, and the positioning data corresponding to the timestamp closest to the target moment. The acquisition timestamp of each modality data represents the data acquisition time. Since the acquisition periods and frequencies of each multimodal device are different, the acquisition timestamps of each data are different. It is judged whether the difference between the acquisition timestamp of each modality data in the first data and the target moment is less than the time threshold. If it is less, it can be considered that the data is the time-aligned data at the target moment. Exemplarily, the time threshold can be 20 ms.

[0084] If the time difference between the acquisition timestamp corresponding to the first data and the target moment is greater than the time threshold, the time-aligned data can be obtained by means of linear interpolation. Exemplarily, the target moment is t, and the acquisition timestamps of the laser point cloud data for two consecutive acquisitions satisfy t1 < t < t2, and the time differences between t1 and t2 and t are both greater than the time threshold. Then, the laser point cloud data in the time-aligned data is determined by the following linear interpolation formula:

[0085]

[0086] where I(t) is the laser point cloud data in the time-aligned data, I(t1) is the laser point cloud data at the moment t1, I(t2) is the laser point cloud data at the moment t2, and · represents the dot product.

[0087] The difference processing of the image data, the millimeter-wave point cloud data, and the positioning data is the same as the method in the above example.

[0088] The traffic violation control method provided by the embodiments of the present application determines whether the multimodal data is time-aligned according to whether the time difference between the acquisition timestamp of the multimodal data and the target moment is less than the time threshold. If the time threshold condition cannot be satisfied, the time-aligned data is obtained by means of linear interpolation, which fully ensures the consistency of the data and guarantees the accuracy of traffic violation recognition.

[0089] In a possible implementation manner, the Kalman filter algorithm is used to perform data fusion on the time-aligned data to obtain the running state of the target vehicle associated with the multimodal data in the three-dimensional space, including:

[0090] According to the time-aligned data, the observation value of the running state of the target vehicle is obtained;

[0091] The following Kalman filter state prediction formula is used to obtain the predicted value of the running state of the target vehicle:

[0092]

[0093] in, X is the predicted operating state value of the target vehicle. t-1 Let F be the operating state of the target vehicle at the previous time step, B be the state transition matrix, and u be the control input matrix. t-1 For control input;

[0094] Based on the predicted operating state of the target vehicle and the Kalman filter state update formula, the operating state of the target vehicle in three-dimensional space is determined:

[0095]

[0096] Among them, X t This refers to the operational state of the target vehicle in three-dimensional space. K is the predicted value of the operating state of the target vehicle. t For Kalman gain, Z t H represents the observed operating status of the target vehicle, and H is the observation matrix.

[0097] In Kalman filtering, state estimation is divided into two steps: predicting the state at the current time based on the state estimate at the previous time step; and correcting the predicted value by combining the current observation value to obtain a more accurate estimate for the current time step.

[0098] Specifically, the time-aligned data is first used to assemble the operational status observations of the target vehicle, as shown in the example below:

[0099]

[0100] Among them, Z t x represents the observed operating status of the target vehicle. cam (t) represents the detected location of the target vehicle in the image data, x lidar (t) represents the position of the target vehicle detected in the laser point cloud data, v radar (t) represents the speed of the target vehicle detected in the millimeter-wave point cloud data, l gps (t) represents the location information of the target vehicle in the location data.

[0101] Then, using the target vehicle's operating state from the previous moment, the estimated operating state of the current vehicle is determined. The target vehicle's operating state from the previous moment is a value pre-stored in the storage space, as shown in the example below:

[0102]

[0103] in, X is the predicted operating state value of the target vehicle. t-1Let F be the operating state of the target vehicle at the previous moment, and let F be the state transition matrix, describing the change of state over time, such as a uniform speed or uniform acceleration model, which can be derived from vehicle dynamics. Let B be the control input matrix, reflecting the influence of external control on the state. t-1 For control inputs, such as throttle and brake, the actual inputs can be obtained from the vehicle bus.

[0104] Determine the Kalman gain using the following formula:

[0105] K t =P t|t-1 H T HP t|t-1 H T +R) -1

[0106] Among them, K t The Kalman gain, P, determines the weights of the predicted and observed values ​​in the state update. t|t-1 The covariance matrix is ​​used to predict the uncertainty of the current state prediction. H is the observation matrix, which maps the state space to the observation space. R is the observation noise covariance matrix, reflecting the sensor measurement error (obtained through calibration experiments). T This represents the transpose of matrix H, which means swapping the rows and columns of the matrix.

[0107] The purpose of this step is to determine whether to trust predictions or observations more. K t The larger the value, the more trust is placed in the observational data; K t The smaller the value, the more confident the model's predictions are. P t|t-1 Let K represent the uncertainty (covariance) in predicting the current state, where H is the observation matrix and R is the covariance of the observation noise (i.e., sensor measurement error). Understandably, if the sensor is very accurate (R is small), K... t The system will be larger and more dependent on observations; if the model predictions are very accurate (P... t|t-1 (Very small), K t The system will be smaller and will rely more on its own predictions.

[0108] In determining the operating status observation value Z of the target vehicle t Predicted operating status of the target vehicle and Kalman gain K t Then, the target vehicle's operating state in three-dimensional space is determined according to the following state update formula:

[0109]

[0110] Among them, X t This refers to the operational state of the target vehicle in three-dimensional space. K is the predicted value of the operating state of the target vehicle. tFor Kalman gain, Z t H represents the observed operating status of the target vehicle, and H is the observation matrix.

[0111] This step involves weighted fusion of the predicted and observed values ​​to obtain the optimal state estimate for the current moment. Z t These are data actually observed. It is the value that maps the predicted state to the observation space. It is the difference between observation and prediction, also known as the residual. K t This determines how much influence the residual has on the final result. Understandably, if the difference between the observation and the prediction is large, and K... t If K is large, the system will significantly correct the prediction results; if K t The smaller the value, the smaller the correction range.

[0112] In the above formulas, F, B, and H are designed based on the vehicle dynamics model and the installation positions of each sensor on the vehicle body, while R is obtained by collecting data from each sensor multiple times in both static and dynamic states and statistically analyzing the error variance.

[0113] In one implementation, the aforementioned state estimation process can be used to estimate the state of the vehicle where the multimodal device is located, as well as the state of vehicles other than the vehicle where the multimodal device is located. State modeling and prediction are performed separately for the vehicle itself and other detected vehicles. For the vehicle where the multimodal device is located, data collected by its own multimodal device is fused to estimate its precise position, velocity, and acceleration. For vehicles other than the vehicle where the multimodal device is located, target information observed by sensors such as cameras and radar is combined, and methods such as Kalman filtering are used to estimate their motion state in three-dimensional space. This enables synchronous, high-precision tracking and behavior modeling of multiple vehicles in a traffic scene, providing fundamental data support for subsequent illegal behavior identification and evidence chain generation.

[0114] The traffic violation control method provided in this application estimates the precise operating status of the target vehicle through Kalman filtering, providing basic data support for subsequent traffic violation identification and evidence chain generation, which helps to improve the accuracy of traffic violation identification.

[0115] In one possible implementation, identifying whether a target vehicle has committed a traffic violation based on time-aligned data and operational status includes:

[0116] Time-aligned data and operational status are input into the traffic violation recognition model to identify traffic violations and obtain the confidence level of the target vehicle for different traffic violations.

[0117] The training samples for the traffic violation recognition model include: multimodal data samples corresponding to the original traffic scene and enhanced samples obtained by data augmentation of the multimodal data samples. The enhanced samples include images after adding noise to the original traffic scene images and / or images after motion blurring of the original traffic scene images.

[0118] The operational status determined using the Kalman filter algorithm accurately reflects the target vehicle's position, speed, and acceleration, while time-aligned data contains more feature information about the target vehicle, such as its edges, texture, contours, 3D shape, spatial distribution, and relative distance. Inputting both time-aligned data and operational status into the traffic violation recognition model provides accurate data while retaining rich details, enabling the model to accurately identify traffic violations.

[0119] In one implementation, the traffic violation recognition model is a deep learning model. When training the traffic violation recognition model, augmented samples are used. The process of generating augmented samples follows the formula:

[0120]

[0121] Among them, I ′ cam(t) is the augmented sample, I cam (t) represents the original sample, and Kblur is the motion blur convolution kernel, simulating the blur effect caused by camera shake or rapid object movement during vehicle movement. It can be adjusted according to the actual scene to more realistically reproduce different types of motion blur. Gaussian noise is used to simulate image quality degradation caused by camera sensor noise or adverse weather conditions (such as rain or fog). * indicates a convolution operation.

[0122] By using the above-mentioned augmented samples to train the model, the model will see more types of interference samples during training, thereby learning to ignore non-ideal factors such as noise and blur, and improving its adaptability to real complex environments. This method belongs to adversarial training and can improve the generalization ability and robustness of deep learning models.

[0123] In some implementations, the model training and deployment process is as follows: Figure 3As shown, the data preparation stage obtains the original training samples. In the model training stage, a CSPDarknet53+Transformer neural network architecture is constructed. The model is trained adversarially using augmented samples, and the model is lightweighted using quantization compression tools such as TensorRT inference engines. After completing the model training stage, in the deployment and validation stage, the model deployment is accelerated using accelerated deployment tools, and the model is cross-validated using multi-source data to ensure the accuracy of the model's recognition results.

[0124] The traffic violation control method provided in this application inputs time-aligned data and operational status into a traffic violation recognition model. This ensures the accuracy of the input data while providing the model with rich vehicle operational characteristic information, thus improving the accuracy of the traffic violation recognition model. Furthermore, training the traffic violation recognition model with augmented samples enhances its generalization ability and robustness.

[0125] In one possible implementation, the traffic violation identification model includes a feature extraction module and a traffic violation identification module; wherein:

[0126] The feature extraction module is used to extract the first feature of the image data in the time-aligned data, which includes the edge, texture, and contour of the target vehicle; extract the second feature of the laser point cloud data in the time-aligned data, which includes the three-dimensional shape, spatial distribution of the point cloud, and distance of the target vehicle; extract the third feature of the millimeter-wave point cloud data and positioning data in the time-aligned data, which includes the speed, acceleration, relative distance, and heading angle of the target vehicle; and stitch and normalize the first, second, and third features to obtain the running state features corresponding to the time-aligned data.

[0127] The traffic violation identification module is used to identify traffic violations based on operational status characteristics and operational status, and to obtain the confidence level of different traffic violations corresponding to the target vehicle.

[0128] Specifically, the feature extraction module uses a real-time target detection algorithm to extract the first feature from the time-aligned image data. This first feature includes the target vehicle's edges, texture, and contour. Using a point cloud processing model, such as PointNet or PointNet++, it extracts the second feature from the laser point cloud data in the time-aligned data. This second feature includes the target vehicle's 3D shape, point cloud spatial distribution, and distance. The millimeter-wave point cloud data and positioning data in the time-aligned data are normalized to eliminate the influence of dimensions, yielding the third feature, which includes the target vehicle's speed, acceleration, relative distance, and heading angle. If temporal modeling is used, a sequence of features from consecutive time points can be input to capture dynamic changes in behavior.

[0129] By concatenating the above feature vectors into a unified high-dimensional feature vector and normalizing it, the running state features corresponding to the time-aligned data can be obtained. This can be further processed through fully connected layers or normalization layers to improve the feature representation capability.

[0130] The traffic violation recognition module performs a linear transformation on the operating state features and operating state based on the trained weight matrix and bias vector. The result of the linear transformation is a vector of length n, where n is the number of traffic violation categories. Each element in the vector corresponds to a violation category, such as speeding or crossing the line. A normalization function, such as the Softmax function, is used to convert the values ​​of each element in the vector into probabilities, ensuring that the sum of all probabilities equals 1. The probability values ​​of each category are compared with a preset probability threshold. If the probability value exceeds the threshold, it indicates that the target vehicle has committed a traffic violation of the category corresponding to that probability value.

[0131] In one implementation, the category with the highest probability in the vector is found, and the category of traffic violation and its corresponding probability are output as the recognition result.

[0132] For example, suppose there are 3 types of violations: speeding, crossing the line, and illegal parking. The model output is [0.1, 0.7, 0.2], with a maximum value of 0.7, corresponding to the second type (crossing the line). The output of the traffic violation identification model is: identified as "crossing the line" with a confidence level of 0.7.

[0133] The traffic violation control method provided in this application uses a traffic violation behavior recognition model to identify the traffic violations of a target vehicle. First, it extracts features from each modal data in the time-aligned data to obtain rich operational status features. Based on the operational status features and operational status, it obtains the categories of possible traffic violations of the vehicle and their corresponding confidence levels, which can accurately determine the traffic violations of the target vehicle.

[0134] In one possible implementation, it also includes:

[0135] After obtaining the operating status of the target vehicle, the operating status is transmitted to the traffic violation recognition model through the ROS2 robot operating system message system.

[0136] Specifically, a custom message type (e.g., VehicleState.msg) is defined in the ROS2 messaging system, including fields such as location, speed, and acceleration, ensuring standardized data structures and facilitating multi-module collaboration. The Kalman filter module acts as a Publisher node, encapsulating the target vehicle's operational status at each moment into a message, which is then published through a specified Topic (e.g., vehicle / state). Downstream modules (e.g., the traffic violation recognition model) act as Subscriber nodes, subscribing to this Topic to receive and process vehicle status data in real time, achieving decoupling and efficient communication between modules. Appropriate Quality of Service (QoS) strategies are configured based on the system's real-time and reliability requirements, such as small historical cache, reliable transmission, and minimal latency, ensuring message delivery stability. All messages have a unified timestamp, facilitating multi-source data alignment and event tracing for subsequent modules, improving overall system consistency.

[0137] ROS2's distributed architecture supports parallel deployment of multiple nodes, making it easy to add new functional modules or expand the system scale in the future.

[0138] The traffic violation control method provided in this application uses the ROS2 messaging system to transmit the running status to the traffic violation identification model, realizing efficient, real-time, and decoupled communication between modules, supporting distributed deployment and system expansion, and ensuring data consistency and traceability.

[0139] In one possible implementation, it also includes:

[0140] When a target vehicle commits a traffic violation, the category of the traffic violation and the data related to the target vehicle contained in the multimodal data are packaged to generate an evidence chain;

[0141] The evidence chain was encrypted and uploaded to the traffic management platform.

[0142] When a traffic violation is detected in a target vehicle, the timestamp t of the violation is first determined, and the original evidence data F(t) at that moment is retrieved and extracted, ensuring that all data was collected at the same time. F(t) typically includes: image data I cam (t), laser point cloud data P lidar (t), millimeter-wave point cloud data D radar (t), GPS positioning data L gps (t), IMU speed V imu (t).

[0143] The identification result C(t) is combined with the original evidence data F(t) to generate a structured evidence chain E(t) = {C(t), I} cam (t),Plidar (t),D radar (t),L gps (t),V imu C(t)}, where C(t) is the output of the traffic violation identification model.

[0144] The evidence chain is packaged and encrypted according to the encryption method required by the traffic management department. The encrypted evidence chain is then uploaded to the traffic management platform API in real time via a 5G C-V2X communication module. The communication process uses the TLS / SSL protocol to ensure secure transmission.

[0145] The traffic violation control method provided in this application realizes the automated and structured association between the violation identification results and the original evidence, ensuring that each report has a complete and traceable chain of evidence, and guaranteeing the legal compliance and automated processing capabilities of traffic violation control.

[0146] In one possible implementation, it also includes:

[0147] Before encrypting the evidence chain and uploading it to the traffic management platform, the multimodal data contained in the evidence chain is checked for consistency to ensure the integrity of the evidence chain.

[0148] For example, verifying the GPS speed V in the chain of evidence. gps (t) and radar velocity S radar The difference between (t):

[0149] |V gps (t)-S radar (t)|<∈

[0150] Wherein, ∈ is the allowable error threshold, for example, 2km / h, obtained through system calibration.

[0151] The traffic violation control method provided in this application performs consistency verification on the multimodal data in the evidence chain before uploading the evidence chain, ensuring the integrity and usability of the evidence chain and improving the efficiency and effectiveness of traffic violation control.

[0152] The following is an example of applying the traffic violation control method provided in the embodiments of this application.

[0153] 1. Data Collection

[0154] The vehicle-mounted forward-facing camera continuously captures video streams of the road ahead. cam (t). Millimeter-wave radar detects vehicles traveling in the same direction ahead in real time and obtains their relative speed v. rel (t) and distance d rel (t). The vehicle's onboard positioning module (GPS / IMU) records the vehicle's speed v. ego(t) and location information L ego (t).

[0155] 2. Data synchronization

[0156] The synchronization controller aligns the aforementioned multi-source data using a unified timestamp t, forming a synchronization data packet S(t) = {I cam (t),v rel (t),d rel (t),v ego (t),L ego (t)}.

[0157] 3. Data fusion

[0158] Using algorithms such as Kalman filtering, the relative velocity v of the millimeter-wave radar is... rel (t) and the vehicle's speed v ego (t) Fusion, calculate the absolute speed of the target vehicle:

[0159] v target (t)=v ego (t)+v rel (t)

[0160] By combining camera images and radar target tracking, the uniqueness and trajectory of the target vehicle can be confirmed.

[0161] 4. Identification of illegal activities

[0162] The system will determine the absolute speed v of the target vehicle. target (t) and the current road speed limit v limit Compare:

[0163] If v target (t)>v limit If the tolerance is +Δv, then it is determined to be speeding.

[0164] By combining image recognition, the system can automatically locate the license plate of the target vehicle, ensuring the uniqueness of the violator.

[0165] 5. Evidence generation

[0166] Automatically capture video clips, speed curves, and location data of the target vehicle during its speeding period to generate an evidence package.

[0167] The evidence package includes: the time of the violation, an image of the target vehicle, its license plate, speed data, GPS coordinates, and timestamps.

[0168] 6. Reporting Process

[0169] The evidence package is encrypted and uploaded to the traffic management platform API in real time via a 5G communication module.

[0170] The reporting process is fully automated and requires no human intervention.

[0171] The above example illustrates the process of controlling speeding by vehicles traveling in the same direction during driving. By applying the traffic violation control method provided in this application, the entire process of monitoring and reporting can be automated.

[0172] The following is another example of applying the traffic violation control method provided in the embodiments of this application.

[0173] 1. Data Collection

[0174] The vehicle-mounted forward and side cameras continuously collect video streams from both sides of the road. cam (t).

[0175] LiDAR scans both sides of the road in real time to acquire spatial point cloud P of stationary targets. lidar (t).

[0176] The positioning module (GPS / IMU) records the vehicle's current location L. ego (t) and travel speed v ego (t).

[0177] 2. Data synchronization

[0178] The synchronization controller aligns the aforementioned multi-source data using a unified timestamp t, forming a synchronization data packet S(t) = {I cam (t),P lidar (t),L ego (t),v ego (t)}.

[0179] 3. Data fusion

[0180] Using lidar point clouds and camera images, stationary vehicle targets on both sides of the road are detected.

[0181] By tracking multiple frames, it is determined that the target vehicle's position changes very little within a continuous time period (Δx < ∈, such as ∈ = 0.5m), and is therefore judged to be stationary.

[0182] Based on the vehicle's speed and location, estimate the relative position and speed of the target vehicle relative to the vehicle to confirm the illegally parked vehicle.

[0183] Mathematical formula:

[0184] If v target (t)<∈and|L target (t)-L ego If (t)|<∈, it is determined as a parking violation.

[0185] in,

[0186] ∈ represents the preset static threshold.

[0187] Δx represents the displacement of the target vehicle in two consecutive frames.

[0188] L target (t) represents the position of the target vehicle.

[0189] L ego (t) represents the position of this vehicle.

[0190] v target (t) represents the speed of the target vehicle.

[0191] |L target (t)-L ego (t) represents the positional deviation of the target vehicle relative to the current vehicle.

[0192] v target (t)<∈ means that the speed of the target vehicle is less than the preset stationary threshold.

[0193] |L target (t)-L ego (t)|<∈ means that the positional deviation of the target vehicle relative to the current vehicle is less than the preset stationary threshold.

[0194] Based on the vehicle's location data, determine the road segment where the target vehicle is located.

[0195] 4. Identification of illegal activities

[0196] The system compares whether the target vehicle's location is a no-parking zone (e.g., identified through electronic maps or road signs).

[0197] If the target vehicle's side marker lights and daytime running lights are not illuminated in a no-parking zone, it is deemed to be illegally parked.

[0198] If the target vehicle is in a permitted parking area but violates parking regulations (such as parking in the wrong direction), it will be deemed as illegally parked.

[0199] By combining image recognition, the system can automatically locate the license plate of the target vehicle, ensuring the uniqueness of the violator.

[0200] 5. Evidence generation

[0201] It automatically captures video clips, point cloud sequences, and location data of the target vehicle during the period of illegal parking, and generates an evidence package.

[0202] The evidence package includes: the time of the violation, images of the target vehicle, license plate, stationary trajectory, GPS coordinates, timestamps, etc.

[0203] 6. Reporting Process

[0204] The evidence package is encrypted and uploaded to the traffic management platform API in real time via a 5G communication module. The reporting process is fully automated and requires no manual intervention.

[0205] The above example illustrates the entire process of controlling traffic violations by illegally parked vehicles. By applying the traffic violation control method provided in this application, the entire process of monitoring and reporting can be automated.

[0206] Figure 4 The schematic diagram of the traffic violation control device provided in this application is as follows: Figure 4 As shown, the traffic violation control device 40 provided in this embodiment includes:

[0207] The acquisition module 401 is used to acquire multimodal data collected by the vehicle-mounted multi-source device. The multimodal data includes: image data, laser point cloud data, millimeter-wave point cloud data, and positioning data.

[0208] Processing module 402 is used to perform time alignment processing on multimodal data to obtain time-aligned multimodal data at the target time.

[0209] The fusion module 403 is used to perform data fusion on time-aligned data using the Kalman filter algorithm to obtain the running status of the target vehicle in three-dimensional space associated with multimodal data.

[0210] The identification module 404 is used to identify whether a target vehicle has committed traffic violations based on time-aligned data and operating status.

[0211] In one possible implementation, the processing module 402 is specifically used for:

[0212] Determine the target time based on the acquisition timestamps corresponding to the multimodal data;

[0213] Among the modal data contained in the multimodal data, the first data whose corresponding acquisition timestamp is closest to the target time is determined, and based on the first data, the time alignment data of the corresponding modality at the target time is determined;

[0214] Based on the time alignment data of each modality at the target time, the time alignment data of the multimodal at the target time is obtained.

[0215] In one possible implementation, the processing module 402 is further configured to:

[0216] If the time difference between the collection timestamp corresponding to the first data and the target time is less than the time threshold, then the first data is determined to be the time-aligned data of the corresponding modality at the target time.

[0217] If the time difference between the collection timestamp corresponding to the first data and the target time is greater than the time threshold, then in the data of the corresponding modality, the second data collected at the first time and the third data collected at the second time are determined. The first time is the collection time that is before the target time and closest to the target time, and the second time is the collection time that is after the target time and closest to the target time. Based on the second data and the third data, the time-aligned data of the corresponding modality at the target time is obtained by linear interpolation.

[0218] In one possible implementation, the fusion module 403 is specifically used for:

[0219] Based on time-aligned data, obtain the operational status observations of the target vehicle;

[0220] Use the following Kalman filter state prediction formula to obtain the predicted operating state of the target vehicle:

[0221]

[0222] in, X is the predicted operating state value of the target vehicle. t-1 Let F be the operating state of the target vehicle at the previous time step, B be the state transition matrix, and u be the control input matrix. t-1 For control input;

[0223] Based on the predicted operating state of the target vehicle and the Kalman filter state update formula, the operating state of the target vehicle in three-dimensional space is determined:

[0224]

[0225] Among them, X t This refers to the operational state of the target vehicle in three-dimensional space. K is the predicted value of the operating state of the target vehicle. t For Kalman gain, Z t H represents the observed operating status of the target vehicle, and H is the observation matrix.

[0226] In one possible implementation, the identification module 404 is specifically used for:

[0227] Time-aligned data and operational status are input into the traffic violation recognition model to identify traffic violations and obtain the confidence level of the target vehicle for different traffic violations.

[0228] The training samples for the traffic violation recognition model include: multimodal data samples corresponding to the original traffic scene and enhanced samples obtained by data augmentation of the multimodal data samples. The enhanced samples include images after adding noise to the original traffic scene images and / or images after motion blurring of the original traffic scene images.

[0229] In one possible implementation, the identification module 404 includes a feature extraction module and a traffic violation identification module; wherein:

[0230] The feature extraction module is used to extract the first feature of the image data in the time-aligned data, which includes the edge, texture, and contour of the target vehicle; extract the second feature of the laser point cloud data in the time-aligned data, which includes the three-dimensional shape, spatial distribution of the point cloud, and distance of the target vehicle; extract the third feature of the millimeter-wave point cloud data and positioning data in the time-aligned data, which includes the speed, acceleration, relative distance, and heading angle of the target vehicle; and stitch and normalize the first, second, and third features to obtain the running state features corresponding to the time-aligned data.

[0231] The traffic violation identification module is used to identify traffic violations based on operational status characteristics and operational status, and to obtain the confidence level of different traffic violations corresponding to the target vehicle.

[0232] In one possible implementation, the processing module 402 is further configured to:

[0233] After obtaining the operating status of the target vehicle, the operating status is transmitted to the traffic violation recognition model through the ROS2 robot operating system message system.

[0234] In one possible implementation, the processing module 402 is further configured to:

[0235] When a target vehicle commits a traffic violation, the category of the traffic violation and the data related to the target vehicle contained in the multimodal data are packaged to generate an evidence chain;

[0236] The evidence chain was encrypted and uploaded to the traffic management platform.

[0237] In one possible implementation, the processing module 402 is further configured to:

[0238] Before encrypting the evidence chain and uploading it to the traffic management platform, the multimodal data contained in the evidence chain is checked for consistency to ensure the integrity of the evidence chain.

[0239] The traffic violation control device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0240] Figure 5 A schematic diagram of the structure of the electronic device provided in this application. Figure 5 As shown, the electronic device 50 provided in this embodiment includes at least one processor 501 and a memory 502. Optionally, the device 50 further includes a communication interface 503. The processor 501, memory 502, and communication interface 503 are connected via a communication bus 504.

[0241] In a specific implementation, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to perform the above-described method.

[0242] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0243] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0244] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0245] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0246] This application also provides a computer program product, including a computer program that, when executed, implements the above-described method.

[0247] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed, implement the above-described method.

[0248] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0249] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an application-specific integrated circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0250] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0251] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0252] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0253] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0254] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0255] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A method for controlling traffic violations, characterized in that, include: Acquire multimodal data collected by vehicle-mounted multi-source devices, including: image data, laser point cloud data, millimeter-wave point cloud data, and positioning data; The multimodal data is time-aligned to obtain time-aligned multimodal data at the target time. The Kalman filter algorithm is used to fuse the time-aligned data to obtain the running state of the target vehicle in three-dimensional space associated with the multimodal data; Based on the time-aligned data and the operating status, it is determined whether the target vehicle has committed any traffic violations.

2. The method for controlling traffic violations according to claim 1, characterized in that, The step of performing time alignment processing on the multimodal data to obtain time-aligned multimodal data at the target time includes: The target time is determined based on the acquisition timestamps corresponding to the multimodal data; Among the modal data included in the multimodal data, the first data whose corresponding acquisition timestamp is closest to the target time is determined, and the time alignment data of the corresponding modality at the target time is determined based on the first data. Based on the time alignment data of each modality at the target time, the time alignment data of the multimodal at the target time is obtained.

3. The method for controlling traffic violations according to claim 2, characterized in that, The step of determining the time alignment data of the corresponding mode at the target time based on the first data includes: If the time difference between the collection timestamp corresponding to the first data and the target time is less than the time threshold, then the first data is determined to be the time-aligned data of the corresponding modality at the target time; If the time difference between the collection timestamp corresponding to the first data and the target time is greater than a time threshold, then in the data of the corresponding modality, the second data collected at the first time and the third data collected at the second time are determined, wherein the first time is the collection time before the target time and closest to the target time, and the second time is the collection time after the target time and closest to the target time; based on the second data and the third data, the time-aligned data of the corresponding modality at the target time is obtained by linear interpolation.

4. The method for controlling traffic violations according to any one of claims 1 to 3, characterized in that, The step of identifying whether the target vehicle has committed a traffic violation based on the time-aligned data and the operating status includes: The time-aligned data and the running status are input into the traffic violation recognition model to identify traffic violations and obtain the confidence level of the target vehicle for different traffic violations. The training samples of the traffic violation recognition model include: multimodal data samples corresponding to the original traffic scene and enhanced samples obtained by data augmentation of the multimodal data samples. The enhanced samples include images after adding noise to the original traffic scene images and / or images after motion blurring of the original traffic scene images.

5. The method for controlling traffic violations according to claim 4, characterized in that, The traffic violation identification model includes a feature extraction module and a traffic violation identification module; wherein: The feature extraction module is used to extract a first feature from the image data in the time-aligned data, the first feature including the edge, texture, and contour of the target vehicle; extract a second feature from the laser point cloud data in the time-aligned data, the second feature including the three-dimensional shape, spatial distribution of the point cloud, and distance of the target vehicle; extract a third feature from the millimeter-wave point cloud data and positioning data in the time-aligned data, the third feature including the speed, acceleration, relative distance, and heading angle of the target vehicle; and stitch and normalize the first feature, the second feature, and the third feature to obtain the running state features corresponding to the time-aligned data. The traffic violation identification module is used to identify traffic violations based on operating status characteristics and the operating status, and to obtain the confidence level of the target vehicle for different traffic violations.

6. The method for controlling traffic violations according to claim 4, characterized in that, Also includes: After obtaining the operating status of the target vehicle, the operating status is transmitted to the traffic violation recognition model through the ROS2 robot operating system message system.

7. The method for controlling traffic violations according to any one of claims 1 to 3, characterized in that, Also includes: When the target vehicle commits a traffic violation, the category of the traffic violation and the data related to the target vehicle contained in the multimodal data are packaged to generate an evidence chain; The evidence chain is encrypted and uploaded to the traffic management platform.

8. The method for controlling traffic violations according to claim 7, characterized in that, Also includes: Before encrypting the evidence chain and uploading it to the traffic management platform, the multimodal data contained in the evidence chain is subjected to consistency verification to ensure the integrity of the evidence chain.

9. A traffic violation control device, characterized in that, include: The acquisition module is used to acquire multimodal data collected by the vehicle-mounted multi-source device. The multimodal data includes: image data, laser point cloud data, millimeter-wave point cloud data, and positioning data. The processing module is used to perform time alignment processing on the multimodal data to obtain time-aligned multimodal data at the target time. The fusion module is used to perform data fusion on the time-aligned data using the Kalman filter algorithm to obtain the running state of the target vehicle in three-dimensional space associated with the multimodal data; The identification module is used to identify whether the target vehicle has committed any traffic violations based on the time-aligned data and the operating status.

10. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-8.

Images