A detection method applied to a traffic monitoring system and related equipment

Abstract

The application provides a detection method applied to a traffic monitoring system and related equipment, and is used for solving the problem that the existing multi-sensor fusion technology cannot realize high-precision time synchronization of multiple different types or same types of sensor data, thereby causing the ITS to be difficult to accurately detect abnormal traffic events based on the sensor data. The method comprises the following steps: acquiring spatial data respectively collected by different sensors in a same time period based on collection time stamps of the spatial data collected by the different sensors; the collection time stamp is a time stamp corrected based on a computer time synchronization protocol; extracting features of target objects from the spatial data respectively collected by the different sensors in the same time period; fusing the features of the target objects to obtain fused features; and detecting whether an abnormal traffic event exists based on the fused features.

Description

Technical Field

[0001] This invention relates to the field of intelligent traffic monitoring technology, and in particular to a detection method and related equipment applied to traffic monitoring systems. Background Technology

[0002] Intelligent Transport Systems (ITS) are comprehensive systems that integrate information technology, communication technology, and sensing technology to intelligently perceive, schedule, and manage all elements of transportation (people, vehicles, roads, and the environment). Their core objective is to improve traffic efficiency, safety, and service levels. Key technologies of ITS include: real-time collection of traffic data such as vehicle location, road conditions, and pedestrian status through cameras, radar, and GPS; and analysis of massive amounts of traffic data to support intelligent scheduling, route planning, and risk warning decisions.

[0003] With the development of smart cities, extremely high demands are placed on the accuracy of environmental perception in ITS (Information Technology Systems). To meet this requirement, multi-sensor fusion technology has become the mainstream solution in ITS. The core idea of ​​multi-sensor fusion technology is to process and complement data from multiple sensors of different or similar types (such as image data, point cloud data, etc.) through algorithms, ultimately outputting more accurate and reliable comprehensive information. The core is to overcome the perception limitations of a single sensor.

[0004] Currently, the main challenge of multi-sensor fusion technology lies in achieving high-precision time synchronization of data from multiple sensors of different or similar types. This issue makes it difficult for ITS systems using this technology to accurately detect abnormal traffic events based on sensor data.

[0005] For example, some technologies propose using software to timestamp sensor data or synchronizing sensor data from different sensors based on a simple Network Time Protocol (NTP). However, the synchronization accuracy is often on the order of tens or even hundreds of milliseconds. For high-speed moving traffic targets, even millisecond-level synchronization errors can lead to spatial displacements of tens of centimeters, severely reducing the fusion effect during subsequent sensor data fusion. For instance, it can cause targets in images to fail to match precisely in space with targets in point clouds, resulting in "ghosting" or mismatches. Summary of the Invention

[0006] This application provides a detection method for traffic monitoring systems to address the problem that existing multi-sensor fusion technologies struggle to achieve high-precision time synchronization of data from multiple different or similar sensors, making it difficult for ITS to accurately detect abnormal traffic events based on sensor data.

[0007] This application also provides a detection device, equipment, computer-readable storage medium, and computer program product for use in traffic monitoring systems.

[0008] The embodiments of this application adopt the following technical solutions: A detection method for a traffic monitoring system includes: acquiring spatial data collected by different sensors within the same time period based on the acquisition timestamps of the spatial data collected by different sensors; the acquisition timestamps are timestamps corrected based on a computer time synchronization protocol; extracting features of target objects from the spatial data collected by different sensors within the same time period; fusing the extracted features of target objects to obtain fused features; and detecting the existence of abnormal traffic events based on the fused features.

[0009] A detection device for a traffic monitoring system includes: a spatial data acquisition unit, used to acquire spatial data collected by different sensors within the same time period based on the acquisition timestamps of the spatial data collected by different sensors; the acquisition timestamps are timestamps corrected based on a computer time synchronization protocol; a feature extraction unit, used to extract features of target objects from the spatial data collected by different sensors within the same time period; a fusion and detection unit, used to fuse the extracted features of the target objects to obtain fused features; and based on the fused features, to detect whether there are abnormal traffic events.

[0010] A computing device includes: a memory and a processor, wherein the memory is used to store a computer program; and the processor is coupled to the memory and used to execute the computer program stored in the memory to perform the methods described above.

[0011] A computer-readable storage medium storing a computer program that, when executed by a computer, enables the implementation of the above-described method.

[0012] A computer program product that stores instructions that, when executed by a computer, cause the computer to perform the methods described above.

[0013] The above-described technical solutions adopted in the embodiments of this application can achieve the following beneficial effects: By ensuring data consistency through time synchronization, and then achieving multi-source information complementarity through feature extraction and fusion, a closed loop of "time synchronization-feature extraction-fusion detection" is formed, which fundamentally solves the problem of inaccurate detection caused by insufficient time synchronization accuracy in existing technologies. Attached Figure Description

[0014] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A flowchart illustrating the specific implementation of a detection method applied to a traffic monitoring system, as provided in this application embodiment; Figure 2 This is a schematic diagram of the specific structure of a detection device applied to a traffic monitoring system according to an embodiment of this application; Figure 3 A schematic diagram of a system architecture for implementing the method provided in this application embodiment is provided; Figure 4 A schematic diagram of the specific structure of the computing device provided in the embodiments of this application. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0016] As will be known to those skilled in the art, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0017] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0018] Embodiment 1 of this application provides a detection method for traffic monitoring systems to solve the problem that existing multi-sensor fusion technologies have difficulty in achieving high-precision time synchronization of data from multiple different or the same types of sensors, which makes it difficult for ITS to accurately detect abnormal traffic events based on sensor data.

[0019] The subject executing this method can be any computing device capable of implementing the method, such as a server, mobile phone, personal computer, smart wearable device, smart robot, etc.

[0020] Different steps of this method can be implemented by the same execution entity or by different execution entities. This application does not limit which execution entity is used to implement the method.

[0021] Furthermore, the embodiments of this application do not limit the execution order of different steps. When using the method provided in the embodiments of this application, the execution order of different steps can be adjusted according to actual needs.

[0022] For ease of description, a traffic monitoring system is used as the execution subject of this method as an example to provide a detailed description of the method provided in this application embodiment.

[0023] like Figure 1 The diagram shown is a flowchart illustrating a specific implementation of a detection method for a traffic monitoring system provided in this application, comprising the following steps: Step 11: Based on the acquisition timestamps of the spatial data collected by different sensors, obtain the spatial data collected by different sensors within the same time period; In this embodiment, the different sensors can be the same type of sensor, such as cameras; or they can be different types of sensors, such as cameras, lidar, and millimeter-wave radar.

[0024] Since the method provided in this application ultimately aims to detect whether there are abnormal traffic events, in some optional implementations, the different sensors can be, for example, fixed sensing devices such as poles, gantries, traffic signal poles, and road monitoring poles set on both sides of the road, or sensing devices set on mobile carriers such as motor vehicles and drones traveling on the road.

[0025] Specifically, the location and deployment method of the aforementioned sensors can be flexibly configured according to actual traffic incident detection needs, for example: For fixed-deployment sensors, they can be deployed evenly or densely along key traffic sections such as main urban roads, highways, and intersections according to preset spatial spacing and coverage range, so as to achieve all-round, no-dead-angle spatial data collection of the target traffic area and ensure complete coverage of various road scenarios where abnormal traffic events may occur. For mobile sensors, they can be mounted on motor vehicles such as buses, taxis, and private cars, or on drones with aerial patrol capabilities. Following the movement trajectory of the vehicle or drone, they can dynamically and in real time collect spatial data along the road and in specific areas, effectively making up for the shortcomings of fixed sensors in terms of coverage and blind spots, and improving the comprehensiveness and timeliness of abnormal traffic event detection.

[0026] In this embodiment of the application, spatial data refers to various types of sensor data that are acquired by the sensor to perceive and collect the physical space of the traffic monitoring scene. These data can represent one or more of the following characteristics: the spatial position, morphological features, motion state, and scene environment features of the target object in the scene. Spatial data is the digital representation data of the traffic physical space by the sensor.

[0027] The data types of spatial data vary depending on the sensing type of the sensor. For example, they include video image data collected by cameras, three-dimensional point cloud data collected by lidar, and target motion parameter data collected by millimeter-wave radar (such as one or more of the following: relative distance data between the target object and the millimeter-wave radar, radial velocity data of the target object, horizontal azimuth angle data of the target object, and relative radial acceleration data of the target object).

[0028] In one alternative implementation, after the sensor collects spatial data, it can encapsulate the spatial data and the timestamp of the spatial data collection to obtain a data packet containing the collected spatial data and the timestamp, and then send it to the traffic monitoring system.

[0029] In another alternative implementation, the sensor may send the spatial data and the timestamp of the spatial data acquisition directly to the traffic monitoring system without encapsulating them.

[0030] In some specific examples, cameras can push video streams containing spatial data and acquisition timestamps to the traffic monitoring system in real time via Real-Time Streaming Protocol (RTSP) or dedicated interfaces; LiDAR can send spatial data and acquisition timestamps to the traffic monitoring system via Ethernet or serial interfaces; and millimeter-wave radar can send spatial data and acquisition timestamps to the traffic monitoring system via Controller Area Network (CAN) bus or wireless modules.

[0031] After receiving spatial data and the timestamps of the spatial data collected by different sensors, the traffic monitoring system can determine which spatial data were collected within the same time period based on the timestamps.

[0032] For example, if the time synchronization window is set to 100 milliseconds, and the video image data collected by the camera at 10:00:00.000, the 3D point cloud data collected by the LiDAR at 10:00:00.050, and the target motion parameter data collected by the millimeter-wave radar at 10:00:00.080 are all collected within the same time period from 10:00:00.000 to 10:00:00.100, the traffic monitoring system will determine that the three sets of spatial data were collected within the same time period.

[0033] For example, with a time matching interval of 50 milliseconds, if the video image data collected by the camera at 10:00:00.030, the 3D point cloud data collected by the LiDAR at 10:00:00.050, and the target motion parameter data collected by the millimeter-wave radar at 10:00:00.070 are all within this time synchronization interval, the system can extract these three sets of data as spatial data collected in the same time period.

[0034] It is important to note that in this embodiment, the timestamp collected is a timestamp corrected based on a computer time synchronization protocol.

[0035] In one optional implementation, each sensor can perform local time calibration based on the Network Time Protocol (NTP) before recording a timestamp when acquiring spatial data, ensuring that the spatial data acquired by each sensor is time-aligned. This allows for accurate detection of any abnormal traffic events occurring within a specific time period / moment by acquiring and analyzing the spatial data collected by different sensors during that time period / moment.

[0036] NTP is an application layer protocol based on the TCP / IP protocol suite. Its core function is to realize time synchronization of various nodes in a distributed system through computer networks, so that the local clocks of all networked devices in the network maintain high-precision time consistency. This protocol can achieve time synchronization accuracy at the millisecond level or even the microsecond level, which can meet the core requirement of unified timestamps in multi-sensor data acquisition scenarios.

[0037] NTP employs a layered time server architecture, using Coordinated Universal Time (UTC, also known as World Standard Time) as a unified time reference. It utilizes clock synchronization algorithms to calculate and correct deviations in the local clocks of devices. Simultaneously, it uses mechanisms such as network latency compensation and clock drift correction to offset time deviations caused by network transmission latency and device hardware clock errors, ensuring the stability of time synchronization between different devices.

[0038] Sensors such as cameras, LiDAR, and millimeter-wave radar equipped with network communication modules all have built-in local real-time clock modules. The time correction process of these sensors based on NTP can adopt a combination of active time synchronization and periodic synchronization. The specific implementation process of this mode is as follows: 1. After the sensor completes network access, it will initiate a network connection request to the NTP time server in the network according to the preset configuration information (NTP server IP address / domain name) to establish a communication link between the sensor and the NTP time server. The NTP time server can be a dedicated NTP server in the local area network or a general NTP server on the public network.

[0039] 2. The sensor sends a time synchronization request message containing its local clock's timestamp to the NTP time server. Upon receiving the message, the NTP time server returns a response message carrying its standard UTC timestamp. This response message includes key time information such as the requested reception time and the response transmission time. Based on the time information from the transmitted and received messages, and combined with the NTP clock synchronization algorithm, the sensor calculates the time deviation between its local clock and the NTP time server's standard clock, while also compensating for round-trip delays during network transmission.

[0040] 3. Based on the calculated time deviation, the sensor adjusts the current time of its built-in local real-time clock module to calibrate the local clock to the standard UTC time consistent with the NTP time server, completing the first active time synchronization. The calibrated local clock will serve as the time reference for generating the acquisition timestamp when the sensor acquires spatial data.

[0041] 4. Due to the inherent drift of the sensor's local hardware clock, time deviation will occur again after long-term operation. Therefore, the sensor will repeat the above steps 2-3 according to a preset period (such as 1 minute / 5 minutes / 30 minutes, which can be configured according to actual accuracy requirements) to perform periodic time synchronization with the NTP time server, continuously correct the drift error of the local clock, and ensure that the sensor's local clock always maintains high precision consistency with the standard UTC time throughout the entire working period.

[0042] In another alternative implementation, each sensor can also perform local time calibration based on a Global Positioning System (GPS) reference. The core of this GPS-based local time calibration is to use a standardized time signal broadcast by satellite as a benchmark. The sensor's built-in GPS receiver module captures this time signal and decodes it to obtain the precise GPS standard time. This corrects the time deviation of the sensor's built-in local real-time clock module, achieving high-precision local time synchronization.

[0043] The GPS system's time reference is highly consistent with UTC (with a deviation of no more than 1 microsecond). Therefore, if the sensor's local time is corrected based on GPS, the corrected sensor local time can also form a unified reference with the time corrected based on NTP, meeting the timestamp consistency requirements when multiple sensors collect spatial data.

[0044] Correcting the local time of the sensors is the premise and foundation for the effective fusion of multi-sensor spatial data and the accurate detection and reliable judgment of abnormal traffic events in this solution. Its core value lies in unifying the acquisition benchmark of all sensors from the time dimension, eliminating the timestamp error caused by the local clock deviation of different sensors, ensuring that the "spatial data within the same time period" filtered based on the acquisition timestamp has real time consistency, thereby supporting the subsequent fusion analysis of multi-source heterogeneous spatial data and event detection.

[0045] In this embodiment, the spatial data collected by cameras, LiDAR, and millimeter-wave radar with unified time-corrected acquisition timestamps are filtered and matched using a data synchronization algorithm with the acquisition timestamp as the core criterion. For example, multi-source spatial data whose acquisition time difference falls within a preset 50-millisecond time window can be determined as valid matching data. That is, this part of the data is considered to be a perception acquisition of the same spatiotemporal state in the traffic monitoring scene, which can be used for subsequent multi-source data fusion and abnormal traffic event detection. Multi-source spatial data whose time difference exceeds the preset time window can be determined as invalid matching data and filtered out, and will not participate in subsequent processing.

[0046] In a specific example, if the video image data acquired by the camera at 10:00:00.000, the 3D point cloud data acquired by the LiDAR at 10:00:00.020, and the target motion parameter data acquired by the millimeter-wave radar at 10:00:00.040 have time differences of less than 50 milliseconds, they are accurately matched and enter the subsequent fusion stage as multi-source spatial data in the same spatiotemporal state. However, if the acquisition timestamp of the millimeter-wave radar is 10:00:00.060, the time difference between it and the camera data is 60 milliseconds, which exceeds the 50-millisecond time window. In this case, the millimeter-wave radar data cannot be accurately matched with the other two and will be filtered out.

[0047] Step 12: Extract the features of the target object from the spatial data collected by different sensors within the same time period obtained by performing Step 11; The target objects referred to here are various traffic participants and key objects that may affect the traffic operation status within the traffic monitoring scenario. Specifically, they include dynamic traffic participants such as motor vehicles, non-motor vehicles, and pedestrians, as well as static road facilities such as road guardrails, traffic signs, road markings, streetlights, and roadside structures.

[0048] The aforementioned target objects are the core analysis objects for abnormal traffic event detection. Extracting their features is the basis for subsequent identification of traffic scene status and determination of whether abnormal traffic events exist.

[0049] In an optional implementation, when the spatial data acquired by the traffic monitoring system from different sensors within the same time period are spatial data of different data types, specifically including image data, point cloud data, and echo signal data, the specific implementation of step 12 may include: Extracting semantic features about the target object from image data; Extracting geometric morphological features and spatial clustering features of target objects from point cloud data; Extract motion state features of the target object from echo signal data.

[0050] The following details how the above features are extracted.

[0051] 1. Extract semantic features of the target object from image data. In one alternative implementation, a convolutional neural network (CNN) or a neural network model based on the Transformer architecture can be used to extract semantic features from the video image data captured by the camera. The output result is the semantic features of the target object, which may specifically include: the category information, bounding box coordinate information, and pose information of the target object.

[0052] The aforementioned neural network model can be fine-tuned and optimized based on a pre-trained model (such as the YOLO model, Mask R-CNN model, or DETR model) and combined with the actual data of the traffic monitoring scenario in this application, so as to improve the accuracy and adaptability of feature extraction.

[0053] In a specific example, a traffic monitoring scene image dataset containing vehicles, pedestrians, non-motorized vehicles, and traffic facilities can be selected. The output layer parameters of the pre-trained YOLO model can be adjusted, the bottom feature extraction network can be frozen, and the mini-batch gradient descent method can be used for iterative training to adapt the model to the features of the target objects in the scene and improve the accuracy of target category recognition and bounding box localization.

[0054] 2. Extracting geometric morphological features and spatial clustering features of target objects from point cloud data. In one alternative implementation, a point cloud segmentation algorithm (such as a clustering algorithm based on Euclidean distance, a Random Sample Consensus (RANSAC) algorithm, or a PointNet series neural network model) can be used to extract features from the 3D point cloud data acquired by the LiDAR. The output results are the geometric morphological features and spatial clustering features of the target object, which may specifically include: the outline size information, surface area information, volume information, spatial location coordinate information, and point cloud cluster information of the target object.

[0055] The above-mentioned point cloud feature extraction algorithm can be optimized by combining the point cloud data characteristics of the traffic monitoring scenario in this application (such as the point cloud distribution of the road environment and the point cloud density of the target object) to improve the robustness and scene adaptability of feature extraction.

[0056] In a specific example, a 3D point cloud dataset from a highway or urban main road scenario can be selected. The distance threshold and minimum number of points in the Euclidean distance clustering algorithm can be adjusted to remove background point cloud interference such as road surface and guardrail, accurately cluster the point cloud clusters corresponding to vehicles and pedestrians, and then extract the geometric features such as length, width, and height of each target object as well as the spatial coordinate clustering features.

[0057] 3. Extract the motion state features of the target object from the echo signal data. In one optional implementation, signal filtering, peak detection, and Doppler effect calculation algorithms can be used to extract features from the echo signal data acquired by the millimeter-wave radar. The output results are the motion state features of the target object, which may specifically include: relative distance information, radial velocity information, relative radial acceleration information, and horizontal azimuth information of the target object.

[0058] In a specific example, a dataset of millimeter-wave radar echo signals in scenarios such as peak traffic hours and low illumination can be selected. The Kalman filter algorithm is used to denoise the original echo signal, optimize the signal peak detection threshold, accurately calculate the Doppler frequency shift of the incident signal, and then extract the motion state features such as the real-time relative distance and radial velocity of each target object, effectively suppressing the errors caused by environmental noise and multi-target interference.

[0059] In one alternative implementation, when a traffic monitoring system acquires spatial data of the same type collected by different sensors within the same time period, these sensors (sensors of the same type) generally exhibit a collaborative relationship of "multi-view complementarity, data redundancy verification, and full-domain coverage enhancement." Each sensor independently completes data acquisition but functions in tandem, jointly providing more comprehensive and accurate basic data for feature extraction of target objects, supporting subsequent feature fusion and abnormal traffic event detection.

[0060] In one optional implementation, step 12 can extract features of the same target object from the spatial data collected by different sensors within the same time period. Here, the target object is the same physical entity in the traffic monitoring scenario (such as a car, a pedestrian crossing the road, or a non-motorized vehicle).

[0061] In a specific example, a traffic monitoring system can extract semantic features (category "small car", bounding box coordinates, vehicle posture) of a small car from the spatial data of video images collected by a camera; extract the geometric shape and spatial clustering features (vehicle length, width, height, point cloud cluster information) of the small car from the spatial data of three-dimensional point cloud collected by LiDAR; and extract the motion state features (relative distance, radial velocity, horizontal azimuth angle) of the small car from the spatial data of echo signals collected by millimeter-wave radar.

[0062] In another alternative implementation, step 12 can extract features of different target objects (e.g., multiple different physical entities in a traffic monitoring scenario) from the spatial data collected by different sensors within the same time period.

[0063] In a specific example, a traffic monitoring system can extract semantic features (category "small car", bounding box coordinates, vehicle posture), semantic features (category "electric bicycle", bounding box coordinates, vehicle posture), and semantic features (category "pedestrian", bounding box coordinates, posture information) of a small car, an electric bicycle, and a pedestrian from the spatial data of video images collected by cameras; it can extract geometric shape and spatial clustering features (vehicle length, width, height, point cloud cluster information) of the small car, the electric bicycle, and the pedestrian from the 3D point cloud spatial data collected by LiDAR; and it can extract motion state features (relative distance, radial velocity, horizontal azimuth) of the small car, the electric bicycle, and the pedestrian from the echo signal spatial data collected by millimeter-wave radar.

[0064] In one alternative implementation, the spatial data can be preprocessed before features are extracted from it. Specifically, to ensure the accuracy of subsequent multi-source spatial data fusion, coordinate unification and viewpoint alignment—i.e., spatial alignment—can be performed on the data from each sensor.

[0065] In one optional implementation, the external parameters of the camera and the LiDAR can be calibrated in advance using a standard calibration board to obtain the rotation matrix and translation vector between them, providing a parameter basis for the spatial alignment of subsequent image data and point cloud data; then, the Iterative Closest Point (ICP) algorithm is used to spatially align the 3D point cloud data acquired by the LiDAR with the camera image viewpoint, so that the spatial coordinate system of the point cloud data and the image data is consistent, ensuring that the position information of the target object is matched consistently; For the echo signal data collected by millimeter-wave radar, coordinate transformation is performed based on its known installation position and angle parameters. The target coordinates output by the millimeter-wave radar are transformed into a unified coordinate system consistent with those of the camera and lidar, ensuring the coordinate consistency of the output data of the three types of sensors and laying the foundation for subsequent multi-source feature fusion.

[0066] In a specific example, the following steps can be used to unify the coordinates and align the viewpoints of the data from each sensor: Step a: Set a unified coordinate system as the camera coordinate system (as the alignment reference for multiple sensors, and all subsequent data from both types of sensors will be mapped to this coordinate system). Using a 12×9 grid standard calibration board (grid size 20mm) as a reference, perform extrinsic parameter calibration on the camera and LiDAR: Specifically, place the calibration board at the center of the common field of view of both types of sensors, collect multiple sets of calibration board images and corresponding point cloud data, and calculate the spatial attitude and position correlation parameters of the two sensors through the calibration algorithm—rotation matrix R (3×3 matrix, example: [[1,0.002,0.001],[0.002,1,0.003],[0.001,0.003,1]]) and translation vector T (example: T=[0.5, 0.2, ...). [0.3]m); Subsequently, this matrix and vector are called to transform the original point cloud coordinates P_lidar (3D coordinates, example: P_lidar=[2.3,1.1,0.8]m) of the lidar. The transformation formula is P_unified = R×P_lidar + T, and the point cloud coordinates P_unified in the unified coordinate system are calculated (example: P_unified=[2.3×1+1.1×0.002+0.8×0.001+0.5, 2.3×0.002+1.1×1+0.8×0.003+0.2, 2.3×0.001+1.1×0.003+0.8×1+0.3]≈[2.803,1.304,1.103]m).

[0067] Step b: Based on the calibration parameters and converted LiDAR point cloud data from step a, select the same target object (e.g., a small car) in the traffic monitoring scenario: Simultaneously acquire the camera image of the vehicle (output pixel coordinates, already in a unified coordinate system, example: [320, 240] pixels, corresponding to spatial coordinates [2.8, 1.3, 1.1] m) and the converted vehicle point cloud data (e.g., the average of multiple sets of point cloud coordinates [2.803, 1.304, 1.103] m); Input the two types of coordinates into the Iterative Closest Point (ICP) algorithm for precise viewpoint alignment: Using the vehicle spatial coordinates corresponding to the image as a reference, iteratively find the closest corresponding point pair in the point cloud, optimize the rotation and translation fine-tuning parameters in each iteration, stop after 5-8 iterations, and finally make the mapping deviation between the vehicle point cloud coordinates and the corresponding spatial coordinates of the image ≤ 2cm, realizing the precise spatial matching of LiDAR and camera data (providing consistent basic data for subsequent multi-feature fusion).

[0068] Step c: For millimeter-wave radar data, first collect its installation parameters (relative to the origin of the unified coordinate system, installation position: x=1.2m, y=0.0m, z=2.5m; horizontal installation angle: 0°, i.e., the radar detection front is parallel to the X-axis of the unified coordinate system); based on these parameters, construct a coordinate transformation matrix M (3×3 rotation matrix + 3×1 translation vector, integrating installation attitude and position information); for the original target coordinates P_radar calculated from the millimeter-wave radar echo signal (example: P_radar=[3.0,0.5,0.2]m, in the radar local coordinate system), perform rotation and translation operations through matrix M (first rotate to correct the installation angle deviation, then translate to correct the installation position deviation), and transform it to the unified coordinate system set in step a. After transformation, the coordinates of the vehicle target are shown in the example: [2.798,1.299,1.102]m, with a deviation of ≤3cm from the target coordinates in the unified coordinate system, thus achieving coordinate unification between the millimeter-wave radar and the data from the first two types of sensors.

[0069] Step 13: Fuse the features of the target objects extracted in Step 12 to obtain fused features; In an optional implementation, when the extracted target object features include semantic features, geometric features, spatial clustering features, and motion state features, step 13 may specifically include: concatenating the semantic features, geometric features, spatial clustering features, and motion state features to obtain concatenated features; inputting the concatenated features into a deep fusion network based on a multimodal attention mechanism, which then generates a cross-modal attention feature map based on the concatenated features. This cross-modal attention feature map is the fusion feature.

[0070] Step 14: Based on the fused features obtained by performing Step 13, detect whether there are abnormal traffic events.

[0071] In an optional implementation, if the fused features include the cross-modal attention feature map described above, then the specific implementation of step 14 may include: Performed by a deep fusion network with a multimodal attention mechanism: The cross-modal attention feature map is sliced ​​to obtain feature slices of different data types; Feature slices of different data types are input into pre-set single-mode decision modules adapted to the corresponding data types to obtain single-mode detection results generated by each single-mode decision module; wherein, the single-mode detection result represents the decision result of the corresponding single-mode decision module on whether there is an abnormal traffic event in the target object; All single-mode detection results are fused to obtain the final fusion decision result used to characterize whether the target object has an abnormal traffic event.

[0072] In this embodiment, each single-module decision-making module is adapted to the selection of corresponding data type characteristics, as follows: A single-mode decision module that adapts to semantic feature slices of image data: for example, a lightweight convolutional neural network (such as MobileNet, ShuffleNet) can be used, focusing on fast reasoning and classification of semantic features; A single-mode decision module that adapts to the geometric shape and spatial clustering feature slices of point cloud data: for example, a lightweight version of PointNet / PointNet++ can be used to adapt to feature learning of unstructured point cloud data; A single-mode decision module that adapts to motion state feature slices of echo signal data: for example, a fully connected neural network (FCN) or a gated recurrent unit (GRU) can be used to focus on pattern recognition of temporal motion parameters.

[0073] In this embodiment of the application, the single-mode decision module can be trained in the following manner: First, the dataset is constructed: labeled data (including normal / abnormal event samples, such as speeding, normal driving, etc.) are collected according to data type in traffic monitoring scenarios to form a single-modal exclusive training set; Then, the single-mode decision model is pre-trained and fine-tuned: the model is pre-trained based on a public traffic dataset, and then the network parameters are fine-tuned using a pre-constructed single-mode-specific training set to optimize the feature extraction capability adapted to this scenario. During training, the binary classification cross-entropy loss function (determining abnormal / normal) is used as the optimization objective, and iterative training is performed until the accuracy on the validation set is stable at ≥95%, thus completing the training of the single-mode decision model.

[0074] The trained single-model decision-making system can be deployed to traffic monitoring systems.

[0075] In a specific example, taking the single-mode decision module for adapting semantic feature slices of image data as an example, it can collect traffic monitoring scene image data (including normal / abnormal event samples, with a sample size of ≥100,000 frames), label the target semantic features (category, bounding box, pose) and abnormal labels (such as speeding, running red lights, etc.); then, based on the pre-trained YOLO / DETR model, freeze 80% of the bottom layer feature extraction network, and only fine-tune the parameters of the top layer and output layer; use mini-batch (batch size = 32) gradient descent method, with binary classification cross-entropy as the loss function, and iterate for ≥50 rounds. During training, the validation set and training set are divided in a 1:4 ratio. Training is stopped when the accuracy of the validation set is stable at ≥95% and the loss value is ≤0.08, finally obtaining the image data decision module adapted to semantic feature slices.

[0076] The following explanation, using the example of a traffic monitoring system detecting whether a small car is speeding, and incorporating the vehicle's multimodal characteristics, details how steps 13 and 14 above are implemented: In this specific example, assume that the monitoring system has extracted four types of features of the target object (a small car) from multi-sensor spatial data within the same time period: 1. Semantic features (from camera video image spatial data): including vehicle category, bounding box position, vehicle orientation, etc. 2. Geometric features (from LiDAR point cloud spatial data): including vehicle length, width, height, outline, wheelbase, etc. 3. Spatial clustering features (from LiDAR point cloud spatial data): including vehicle point cloud cluster density, spatial distance from surrounding obstacles, etc. 4. Motion state characteristics (from spatial data of millimeter-wave radar echo signals): including relative distance between vehicles, radial velocity, acceleration, etc.

[0077] In this specific example, after extracting the above four types of features, the traffic monitoring system further performs the following operations: Step 1: The traffic monitoring system performs dimensional concatenation and splicing of the four types of features in a fixed order of "semantic features → geometric morphological features → spatial clustering features → motion state features"; For example, assuming that the semantic features are N-dimensional vectors, the geometric morphology features are M-dimensional vectors, the spatial clustering features are P-dimensional vectors, and the motion state features are Q-dimensional vectors, the concatenated features are a one-dimensional feature vector containing (N+M+P+Q) dimensional feature elements. Step II: The traffic monitoring system inputs the spliced ​​features into the pre-trained "multimodal attention mechanism deep fusion network". The core of this network focuses on key features through attention weight allocation and finally outputs a visualized cross-modal attention feature map. Specifically, in step II, the network first maps the concatenated features (such as a one-dimensional feature vector with (N+M+P+Q) dimensional feature elements) to a high-dimensional feature vector (such as expanding the dimension through convolutional layers / fully connected layers). Based on the multimodal attention mechanism, the attention weights of four types of features are calculated (e.g., motion state features are more critical to "speed detection" and are assigned a weight of 0.4; semantic features are assigned 0.3; geometric morphology / spatial clustering features are each assigned 0.15). The network performs weighted fusion of features from different modalities based on weights, strengthening key features and weakening secondary features; The fused high-dimensional features are converted into two-dimensional feature maps (e.g., 8×8 / 16×16 size), which are cross-modal attention feature maps. The higher the pixel value in the feature map, the more important the corresponding feature is to "speed detection" (e.g., the pixel value of the speed-related feature is significantly higher than that of other regions).

[0078] Step 3: The traffic monitoring system slices the feature map, thereby splitting the cross-modal attention feature map into 3 feature slices according to data type: semantic feature slice, geometric shape + spatial clustering feature slice, and motion state feature slice; It should be noted that the slicing operation of cross-modal attention feature maps by traffic monitoring systems is essentially to divide exclusive regions from the complete feature map according to the preset mapping rules of "feature source-data type".

[0079] Cross-modal attention feature maps are structured two-dimensional / three-dimensional feature matrices (e.g., an 8×8×32 three-dimensional feature map: 8 rows × 8 columns × 32 feature channels). Assume the predefined mapping rule for "channel / region → data type" in the traffic monitoring system is as follows: Of the 32 feature channels, channels 1-16 correspond to the semantic features of the image data; Of the 32 feature channels, channels 17-26 correspond to the geometric shape and spatial clustering features of the point cloud data. Of the 32 feature channels, channels 27-32 correspond to the motion state features of the radar data.

[0080] In step III, the traffic monitoring system can read the preset mapping rule and then extract all rows / columns of data from channels 1-16 of the complete feature map to form a "semantic feature slice" (size 8×8×16); extract all rows / columns of data from channels 17-26 to form a "geometric shape + spatial clustering feature slice" (size 8×8×10); and extract all rows / columns of data from channels 27-32 to form a "motion state feature slice" (size 8×8×6). These three extracted feature slices are stored separately for use by the corresponding single-mode decision modules in subsequent steps.

[0081] Step IV: The traffic monitoring system inputs the three feature slices into the single-mode decision modules that correspond one-to-one with the data types of the three feature slices, and obtains the single-mode detection results output by each single-mode decision module based on the feature slices; In the traffic monitoring system, three single-mode decision modules can be preset, each corresponding to a different data type (adapting to "video image data / point cloud data / echo signal data" respectively). Each module only processes the feature region of the corresponding data type in the cross-modal attention feature map—that is, it processes the feature slice of the corresponding data type, and then outputs an independent single-mode detection result, i.e., the single-mode detection result.

[0082] In this embodiment, the three single-mode decision modules, each corresponding to a data type, are: an image data decision module, a point cloud data decision module, and a radar data decision module. In step IV, the traffic monitoring system inputs semantic feature slices into the image data decision module, geometric shape + spatial clustering feature slices into the point cloud data decision module, and motion state feature slices into the radar data decision module.

[0083] For each single-mode decision module, a fully connected layer and a classifier are used to perform a binary classification judgment on the corresponding slice to determine whether it is speeding (output 0 = not speeding / 1 = speeding, or a probability value). For example: Image data decision module: Based on the displacement frame difference of the vehicle in the semantic feature slice, output "not speeding (0)"; Point cloud data decision module: Based on the spatial displacement of vehicles in geometric / cluster feature slices, output "not speeding (0)"; Radar data decision module: Based on the radial velocity of the vehicle in the motion feature slice, output "overspeed (1)".

[0084] Step V: The traffic monitoring system uses a "weighted voting method" to integrate the detection results of the three single-mode systems and generate the final integrated decision result.

[0085] In this embodiment, it is assumed that the weights of the single-mode modules are preset as follows: the radar data module is more reliable for velocity detection, with a weight of 0.6; the point cloud data module has a weight of 0.2; and the image data module has a weight of 0.2. Then, the final decision value can be calculated as follows: Final decision value = (Image module result × 0.2) + (Point cloud module result × 0.2) + (Radar module result × 0.6) Assuming the pre-set decision threshold is 0.5, then if the calculated final decision value is ≥0.5, it is determined to be "speeding"; otherwise, it is "not speeding".

[0086] For example, following the previous example, the final decision value can be calculated as (0×0.2)+(0×0.2)+(1×0.6)=0.6≥0.5. Therefore, the traffic monitoring system can generate the final fusion decision result: "The small car is speeding abnormally".

[0087] The method provided in this application ensures data consistency through time synchronization, and then achieves multi-source information complementarity through feature extraction and fusion, forming a closed loop of "time synchronization-feature extraction-fusion detection", which fundamentally solves the problem of inaccurate detection caused by insufficient time synchronization accuracy in the prior art.

[0088] In this embodiment of the application, the fusion of single-mode detection results to obtain the final fusion decision result is mainly aimed at the determination of abnormal traffic events of "single frame, instantaneous" (such as "whether the vehicle is speeding in the current frame"). For the determination of abnormal traffic events of "continuous, time sequence", "multi-target tracking and real-time abnormal behavior detection" can be further adopted.

[0089] In an optional implementation, the method provided in this application can further perform temporal correlation and behavioral analysis on multi-frame continuous target information to overcome the limitations of single-frame determination. Specifically, the method provided in this application may further include the following steps: (1) First, the traffic monitoring system performs multi-target tracking. The traffic monitoring system performs data association on the continuous target information output from the feature fusion process, and uses the Kalman filter algorithm combined with the Hungarian algorithm to generate the target trajectory.

[0090] Among them, the Kalman filter algorithm is used to predict the real-time state of the target object (such as spatial coordinates and motion speed) and correct measurement errors; the Hungarian algorithm is used to solve the optimal matching problem between multiple targets and multiple measurements, ensuring the unique association of the target object in continuous frame data.

[0091] In one alternative implementation, the multi-target tracking process includes the following steps: Step A: After extracting the features of the target object (such as semantic features, geometric features, spatial clustering features, and motion state features, which can also be collectively referred to as multimodal features), the traffic monitoring system extracts the static attributes and dynamic behavior parameters of the target object from the multimodal features; Among them, the static attributes of the target object can be, for example, the category of the target object, the three-dimensional spatial coordinates under a unified coordinate system, the target morphological parameters, etc., which can be extracted from "semantic features and geometric morphological features"; the dynamic behavior parameters can be, for example, relative distance, radial velocity, acceleration, vehicle posture, etc., which can be extracted from "motion state features".

[0092] Step B: The traffic monitoring system assigns a unique target ID to each target object, and encapsulates the static attributes and dynamic behavior parameters of each target object with the collection timestamp corrected based on the computer time synchronization protocol and the target ID to form a single frame target information unit; The single-frame target information unit corresponding to a certain target object includes: the target ID, static attributes, dynamic behavior parameters and corresponding acquisition timestamp of the target object.

[0093] Step C: During the continuous acquisition period of the sensor, the traffic monitoring system accumulates the target information units of each frame formed during the continuous acquisition period in the order of timestamps, thereby obtaining a set of multiple frames of continuous target information units of the target object (i.e., continuous target information). In this set, each frame of target information unit contains the target object's target ID, static attributes, dynamic behavior parameters, and collection timestamp. Furthermore, the information of the same target object in different frames is uniquely associated through the target ID, ensuring the continuity and traceability of time-series data.

[0094] Step D: The traffic monitoring system uses the Kalman filter algorithm combined with the Hungarian algorithm to perform data association processing on continuous target information, and obtains the complete time-series trajectory data of each target object from entering the monitoring area to leaving, as well as the status information of each target object updated in real time in each frame; The complete time-series trajectory data of the target object from entering the monitoring area to leaving is formed by concatenating multiple frames of target spatial coordinates sorted by the corrected acquisition timestamps.

[0095] The target object's state information includes its precise three-dimensional spatial coordinates in a unified coordinate system, corrected radial velocity and acceleration, attitude, and relative distance to surrounding targets. After algorithm error correction, the spatial position deviation is ≤2cm and the velocity deviation is ≤0.1m / s, ensuring the continuity and accuracy of the time series data and providing highly reliable time series support for subsequent abnormal traffic event determination.

[0096] In step D, the Kalman filter algorithm can predict the theoretical state (such as estimated three-dimensional spatial coordinates) of the target object in the current frame based on the dynamic behavior parameters (such as radial velocity and acceleration) of the target object in the previous frame, and correct the prediction error by combining the actual extracted dynamic behavior parameters in the current frame, and output the optimal state estimate of the target object - that is, output the predicted state information of the target object.

[0097] The Hungarian algorithm, on the other hand, solves the problem of optimal matching of multiple target objects and multiple information units across multiple frames based on indicators such as spatial coordinate deviation and static attribute similarity corresponding to the target ID, thus avoiding identity confusion, trajectory breakage or repeated tracking during target tracking.

[0098] In this embodiment, the trajectory data of the target object provides the "spatiotemporal path" of the target, while the state information of the target object (such as position, direction, speed, attitude, relative distance, etc.) supplements the "behavioral details" of each node on the path. Only by combining the two can the transformation from "path judgment" to "behavioral judgment" be realized, avoiding misjudgment caused by relying solely on trajectory data (such as the same "stationary trajectory", only by combining "stationary state + no-stopping zone coordinates" can illegal parking be judged, rather than temporary parking).

[0099] Through the above processing, the traffic monitoring system generates a complete motion trajectory for each target object from entering the monitoring area to leaving. This trajectory consists of multiple frames of three-dimensional spatial coordinates sorted by timestamp under a unified coordinate system. At the same time, the status information of the target object is updated in real time, forming a time-series target tracking record, which provides continuous and complete time-series data support for subsequent abnormal traffic event judgment.

[0100] (2) Then, the traffic monitoring system determines abnormal traffic events. Traffic monitoring systems determine abnormal traffic events through two methods: invoking rule-based models and invoking machine learning. The underlying principles of both are as follows: Rule-based model determination: The traffic monitoring system establishes a rule-based determination model based on traffic regulations and scenario requirements, and presets abnormal traffic event determination conditions (such as red light zone boundary coordinates, road driving direction thresholds, illegal parking time thresholds, etc.); combined with target trajectory data generated by multi-target tracking, it determines whether the target object has triggered preset rules (such as vehicles entering red light zones, driving in the opposite direction to the road regulations, or staying in no-stopping zones for more than the threshold), and then determines whether there are abnormal behaviors such as running red lights, driving against traffic, and illegal parking.

[0101] Machine learning judgment: The traffic monitoring system normalizes the temporal features of the target trajectory (such as instantaneous speed, acceleration, deflection angle of the trajectory curve, trajectory continuity, etc.), and inputs the processed trajectory features into a pre-trained classifier (such as support vector machine (SVM), deep neural network, etc.). Through the classifier's pattern recognition and classification of the trajectory features, it determines whether the target object has abnormal behavior (such as sudden acceleration and deceleration, irregular lane changes, pedestrians suddenly crossing the motor vehicle lane, etc.).

[0102] In this embodiment, a dual judgment mechanism of rule model and machine learning is adopted, which not only covers anomalies with clear rules such as running red lights, driving in the wrong direction, and speeding, but also identifies complex behavioral anomalies such as sudden acceleration / deceleration, irregular lane changes, and pedestrians crossing the motor vehicle lane, filling the scenario gap of single judgment method.

[0103] In one optional implementation, when the traffic monitoring system detects an abnormal traffic event through any of the above-mentioned determination methods, the edge computing unit encapsulates the abnormal event information (including target ID, abnormality type, occurrence timestamp, on-site spatial coordinates, and associated trajectory segments); and uses a low-latency communication method (such as User Datagram Protocol (UDP), WebSocket protocol, etc.) to report the encapsulated warning information to the traffic control center in real time to ensure that the abnormal event can be handled in a timely manner.

[0104] The edge computing unit can be a component of the traffic monitoring system, deployed at edge nodes near the sensors (such as roadside edge boxes or local intelligent monitoring equipment), and directly connected to the external traffic control center to achieve "local fast processing + low-latency reporting".

[0105] The following is a specific example illustrating how a traffic monitoring system can achieve multi-target tracking and abnormal traffic event detection: In this specific example, taking a city intersection monitoring scenario as an example, assume that the monitoring scenario contains 3 target objects (target 1: a small car, target 2: an electric bicycle, and target 3: a pedestrian). The traffic monitoring system has generated 10 consecutive frames (frame numbers 0-9, each frame is collected at an interval of 50ms, corresponding to timestamps 10:00:00.000-10:00:00.450, all corrected by computer time synchronization protocol) of single-frame target information units, as shown in the table below:

[0106] Furthermore, suppose the parameters for the multi-target tracking algorithm are set as follows: Kalman filter parameters: The state equation has a dimension of 5 (x, y, z, v, a), and the observation equation has a dimension of 5; the process noise covariance matrix Q = diag([1e-4, 1e-4, 1e-4, 1e-3, 1e-3]), and the observation noise covariance matrix R = diag([1e-3, 1e-3, 1e-3, 1e-2, 1e-2]); where, the meaning of each parameter in the state equation (x, y, z, v, a) is as follows: x: the X-axis coordinate of the target object in the unified coordinate system (unit: m), representing the spatial position of the target object in the horizontal direction (such as the direction of road extension); y ... horizontal direction The Y-axis coordinate (unit: m) in the coordinate system represents the spatial position of the target in the horizontal and vertical directions (such as the road width direction); z: the Z-axis coordinate (unit: m) of the target object in the unified coordinate system represents the height position of the target (such as the vertical coordinate corresponding to the height of a vehicle or the height of a pedestrian); v: the radial velocity of the target object (unit: m / s) represents the real-time moving speed of the target along its own direction of motion, extracted from the "motion state characteristics" mentioned above; a: the relative radial acceleration of the target object (unit: m / s²) represents the rate of change of the target's velocity (positive for acceleration and negative for deceleration), also derived from the "motion state characteristics".

[0107] Hungarian algorithm parameters: The matching cost matrix uses "spatial coordinate Euclidean distance + velocity deviation weight" as the index, with coordinate distance weight of 0.7, velocity deviation weight of 0.3, and matching threshold set to 0.5 (when the cost value > 0.5, it is determined to be a different target).

[0108] Based on the above assumptions, take target tracking from frame 0 to frame 1 as an example: First, the traffic monitoring system uses Kalman filtering. Based on the state of target 1 (i.e., target with ID 1) in frame 0 (x=20.5, y=3.2, z=1.1, v=15.3, a=0.8), the theoretical state of frame 1 is predicted to be (20.5+15.3×0.05, 3.2+0×0.05, 1.1+0×0.05, 15.3+0.8×0.05, 0.8)≈(21.265, 3.2, 1.1, 15.34, 0.8). Then, observation correction is performed: based on the actual observed state of target 1 in frame 1 (21.2,3.3,1.1,15.5,0.7), the optimal state estimate is calculated through the Kalman filter update equation to obtain the corrected state (21.22,3.28,1.1,15.48,0.76). Specifically, for observation correction, Kalman filtering achieves error correction through three stages: "prediction → observation → update." The core principle is to assign weights based on the reliability of the predicted and observed values. Taking "Frame 1, Target 1" as an example, the implementation process of the three stages is as follows: 1. Prediction stage: Based on the prediction state of target 1 in frame 1 (21.265,3.2,1.1,15.34,0.8), the process noise covariance matrix Q (characterizing prediction uncertainty) is preset, and the confidence level of the prediction value is determined.

[0109] 2. Observation phase: The actual observation status of frame 1 is (21.2, 3.3, 1.1, 15.5, 0.7). The confidence level of the observed value is determined by the observation noise covariance matrix R (characterizing the observation uncertainty).

[0110] 3. Update Phase: Calculate the Kalman gain K (balancing the weights of prediction and observation; the smaller the Q / R ratio, the higher the corresponding data weight), and substitute it into the update equation: "Optimal State Estimation = Predicted State + K × (Observed State - Predicted State)" for weighted calculation. x-dimension: 21.265 + 0.091 × (21.2 - 21.265) ≈ 21.22 y-dimension: 3.2 + 0.091 × (3.3 - 3.2) ≈ 3.28 z-dimension: 1.1 + 0.091 × (1.1 - 1.1) = 1.1 Velocity (v) dimension: 15.34 + 0.083 × (15.5 - 15.34) ≈ 15.48 Acceleration (a) dimension: 0.8 + 0.083 × (0.7 - 0.8) ≈ 0.76 Thus, the corrected state (21.22, 3.28, 1.1, 15.48, 0.76) is finally obtained, realizing the dynamic correction of prediction error.

[0111] In addition, the traffic monitoring system uses the Hungarian algorithm to calculate the matching cost value of the target object (target) between two consecutive frames, and obtains a matching cost matrix composed of the matching cost values; further, based on the matching cost matrix, it outputs the optimal matching result of multiple target objects in two consecutive frames.

[0112] Taking frames 0 and 1 as an example, the matching cost of each target in frames 0 and 1 can be calculated using the Hungarian algorithm, yielding the following results: (0.5 is the preset threshold for matching the same target) Therefore, it is determined that target X in frame 0 and target Y in frame 1 are the same target.

[0113] In this calculation formula: 0.7 is the weighting coefficient for spatial location differences; 0.3 is the weighting coefficient for the difference in motion speed; 21.2 and 21.265 are the X-axis coordinates of two targets in frames 0 and 1, respectively, that are to be determined to be the same target; 3.3 and 3.2 are the Y-coordinates of the two targets in frame 0 and frame 1, respectively; 1.1 and 1.1 are the Z-axis coordinates of the two targets in frame 0 and frame 1, respectively; This indicates that the Euclidean distance of the three-dimensional spatial position is calculated based on the above coordinate values; 15.5 and 15.34 are the radial velocity values ​​of these two targets in frames 0 and 1, respectively.

[0114] The physical meaning of the "preset matching threshold for the same target" is to characterize the maximum degree of feature difference allowed for "being judged as the same target". If the matching cost value is less than this value, it means that the changes in the spatial position and motion state of the target between multiple frames have not exceeded the "reasonable fluctuation range of the same target", that is, target 1 in frame 0 and frame 1 are the same target.

[0115] In the formula for calculating the matching cost of target 1: 0.7×0.119≈0.083 is the spatial position difference item. This item is based on the Euclidean distance calculated by three-dimensional coordinates, reflecting the spatial offset of the target in a unified coordinate system. The offset here is extremely small (about 0.119m), which is far below the upper limit of sensor error (2cm), indicating that the spatial position continuity is good; (0.3×0.16≈0.048) is the movement speed difference item. This item reflects the fluctuation of the target speed. The speed deviation here is only 0.16m / s, which does not exceed the reasonable fluctuation range of normal movement speed (such as the instantaneous changes of vehicle acceleration and deceleration).

[0116] Based on the Hungarian algorithm, the traffic monitoring system can use the weighted sum of the Euclidean distance (weighted 0.7) between the spatial positions of each target in the previous frame and the absolute difference of the radial velocity (weighted 0.3) between the targets in the next frame and the targets in the previous frame as columns, as the matching cost.

[0117] In this matching cost matrix, the physical meaning of the matrix elements is: the comprehensive feature difference quantification value (matching cost value) between a target object in the previous frame and a target object in the next frame, which is used to characterize the similarity between two targets in two frames that are the same physical entity (the smaller the element value, the higher the similarity; the larger the element value, the lower the similarity).

[0118] Suppose that in a traffic monitoring scenario, three target objects were detected in the previous frame (frame t), and they are labeled as follows: Target A (small car, ID: 001) Target B (Electric bicycle, ID: 002) Target C (Pedestrian, ID: 003) In the next frame (frame t+1), four target objects were detected and marked as follows: Target 1 (small car) Target 2 (Electric bicycle) Target 3 (pedestrians) Target 4 (Large trucks, newly added) The Hungarian algorithm calculates the matching cost (Euclidean distance in spatial position × 0.7 + absolute difference in radial velocity × 0.3) between each target in frame t and each target in frame t+1, constructs a matching cost matrix, and solves for the optimal combination. The optimal matching result is as follows: 1. Frame t - Target A (ID: 001) Frame t+1 - Target 1 (cost value ≈ 0.131 < 0.5, determined to be the same small car). 2. Frame t - Target B (ID: 002) Frame t+1 - Target 2 (cost value ≈ 0.156 < 0.5, determined to be the same electric bicycle); 3. Frame t - Target C (ID: 003) Frame t+1 - Target 3 (cost value ≈ 0.098 < 0.5, determined to be the same pedestrian); 4. Frame t has no corresponding target. Frame t+1 - Target 4 (no matching relationship that meets the threshold, determined to be a new target, assigned a new ID: 004).

[0119] Specifically, if only two targets (target 1 and target 2) are detected in frame t+1, then the following additional output will be provided: 5. Frame t - Target C (ID: 003) Frame t+1 has no corresponding target (no matching relationship that meets the threshold, it is determined that target C has left the monitoring area, and its ID tracking is terminated).

[0120] In a specific example, by executing the Kalmar filtering algorithm, the trajectory data consists of the corrected 3D coordinates of each target object from entering the monitoring area to the current frame (frame 9), arranged in ascending order according to the timestamps corrected based on the computer time synchronization protocol, thus completely recording the spatiotemporal movement path of the target, as follows: Target 1 (small car) trajectory data: The trajectory sequence is [(20.5,3.2,1.1),(21.22,3.28,1.1),(21.95,3.35,1.1),(22.68,3.42,1.1),(23.41,3.49,1.1),(24.85,3.61,1.1),(25.97,3.72,1.1),(26.83,3.81,1.1),(27.36,3.85,1.1),(27.78,3.89,1.1)], the trajectory length (straight-line distance from frame 0 to frame 9) is approximately 7.3m, and the average speed is approximately 15.7m / s (56.5km / h). Target 2 (Electric Bicycle) trajectory data: The trajectory sequence is [(18.2,5.1,1.0),(18.62,5.19,1.0),(19.05,5.27,1.0),(19.58,5.35,1.0),(20.11,5.43,1.0),(20.76,5.51,1.0),(21.32,5.59,1.0),(21.88,5.67,1.0),(22.34,5.73,1.0),(22.79,5.78,1.0)], trajectory length ≈ 4.6m, average speed ≈ 9.2m / s (33.1km / h); Target 3 (pedestrian) trajectory data: The trajectory sequence is [(15.6,2.8,1.6),(15.81,2.89,1.6),(16.05,2.98,1.6),(16.32,3.07,1.6),(16.68,3.18,1.6),(17.05,3.29,1.6),(17.42,3.39,1.6),(17.78,3.49,1.6),(18.03,3.59,1.6),(18.29,3.69,1.6)], trajectory length ≈ 2.7m, average speed ≈ 1.35m / s.

[0121] By combining the error correction function of the Kalman filter algorithm, the state information of each target object can be updated in real time, covering dynamic motion parameters and spatial correlation information. Taking frame 9 as an example, the dynamic state of each target in frame 9 is as follows: Target 1 (small car): radial velocity v = 16.2 m / s, radial acceleration a = 0.5 m / s², vehicle attitude is forward driving, relative distance to target 2 ≈ 5.2 m, relative distance to the right edge of the road ≈ 1.8 m; Target 2 (electric bicycle): radial velocity v = 9.5 m / s, radial acceleration a = 0.1 m / s², the bicycle is traveling in a forward direction, the relative distance to Target 1 is ≈ 5.2 m, and the relative distance to Target 3 is ≈ 4.7 m; Target 3 (pedestrian): Radial velocity v = 1.5 m / s, radial acceleration a = 0.0 m / s², posture is upright walking, relative distance to target 2 ≈ 4.7 m, relative distance to the boundary of the pedestrian crossing ≈ 0.3 m.

[0122] The Hungarian algorithm is used to construct an inter-frame target matching cost matrix by calculating the matching cost of "Euclidean distance of spatial location × 0.7 + absolute difference of radial velocity × 0.3". The optimal matching combination with a cost value less than a preset threshold (0.5) is then selected. The output results are as follows (taking frames 8 and 9 as examples): Frame 8 - Target 1 (ID: 001) Frame 9 - Target 1 (cost value ≈ 0.128 < 0.5), determined to be the same small sedan; Frame 8 - Target 2 (ID: 002) Frame 9 - Target 2 (cost value ≈ 0.143 < 0.5), determined to be the same electric bicycle; Frame 8 - Target 3 (ID: 003) Frame 9 - Target 3 (cost value ≈ 0.089 < 0.5), determined to be the same pedestrian; No new or missing targets were added, and the target IDs remained continuous and stable.

[0123] Furthermore, based on the inter-frame matching results of the Hungarian algorithm, the traffic monitoring system concatenates the discrete single-frame fusion features of each target in ascending order of the corrected timestamps to form complete trajectory data.

[0124] For example, the complete trajectory data of target 001 (a small car) consists of the concatenated corrected three-dimensional coordinates from frame 0 to frame 9, with the sequence: [(18.3,4.2,1.1),(19.05,4.28,1.1),(19.82,4.35,1.1),(20.59,4.42,1.1),(21.37,4.49,1.1),(22.15,4.56,1.1),(22.93,4.63,1.1),(23.71,4.70,1.1),(24.49,4.77,1.1)]. This complete trajectory data fully records the spatiotemporal path of the small car from entering the monitored area to its continuous driving.

[0125] For example, the complete trajectory data of target 002 (electric bicycle) is composed of the corrected three-dimensional coordinates concatenated from frame 0 to frame 9, with the sequence: [(15.1,2.7,1.0),(15.54,2.78,1.0),(15.98,2.86,1.0),(16.42,2.94,1.0),(16.86,3.02,1.0),(17.30,3.10,1.0),(17.74,3.18,1.0),(18.18,3.26,1.0),(18.62,3.34,1.0)].

[0126] While serializing the trajectory, the target's state information can be updated in real time by combining the error correction results of the fused features of each frame using Kalman filtering.

[0127] For example, the relevant corrections performed for frame 5 are as follows: Target 001: Speed ​​15.3m / s (corrected), acceleration 0.4m / s² (corrected), relative distance to target 002 ≈ 4.8m, vehicle attitude is forward travel; Target 002: Speed ​​9.1 m / s (corrected), acceleration 0.1 m / s² (corrected), relative distance to target 001 ≈ 4.8 m, relative distance to target 003 ≈ 3.6 m, vehicle attitude is forward travel; Target 003: speed 1.3m / s (corrected), acceleration 0.0m / s² (corrected), relative distance to target 002 ≈ 3.6m, posture is upright walking.

[0128] Based on the aforementioned trajectory data, real-time status information, and optimal matching results, the traffic monitoring system utilizes a dual-determination mechanism combining a rule-based model and a machine learning model to detect the presence of abnormal traffic events, as detailed below: (a) Rule model determination (based on traffic regulations and scenario-preset rules) Preset judgment conditions: The speed limit for small cars on this monitored road section is 50km / h, and the speed limit for electric bicycles is 30km / h; the boundary coordinates of the no-parking zone are (25.0-26.0, 3.0-4.0, 1.0-1.2); the safe distance threshold is ≥3m; the boundary coordinates of the pedestrian crossing area are (17.0-18.5, 3.0-4.0, 1.5-1.7).

[0129] Judgment process: Speed ​​violation determination: Target 1's average speed is 56.5 km / h, and its real-time speed in frame 9 is 16.2 m / s (58.3 km / h), both exceeding the 50 km / h speed limit, triggering the "speeding" rule; Target 2's average speed is 33.1 km / h, and its real-time speed is 9.5 m / s (34.2 km / h), exceeding the 30 km / h speed limit, triggering the "speeding" rule; Target 3's speed is 1.35-1.5 m / s, which is within the normal pedestrian crossing speed range, and no rule was triggered.

[0130] Area violation determination: In the trajectory sequence of target 1, the coordinates of frame 5 (24.85, 3.61, 1.1) and frame 6 (25.97, 3.72, 1.1) are as follows. The coordinates of frame 6 fall into the no-stopping area (25.0-26.0, 3.0-4.0, 1.0-1.2), and the dwell time (the time window corresponding to 1 frame) does not exceed the illegal parking time threshold (3 frames), so the "illegal parking" rule is not triggered. The trajectory of target 3 is located within the pedestrian crossing area throughout, and the "pedestrian illegally entering the motor vehicle lane" rule is not triggered.

[0131] Safe distance determination: The relative distance between target 1 and target 2 is approximately 5.2m and ≥ 3m, and the relative distance between target 2 and target 3 is approximately 4.7m and ≥ 3m. Both meet the safe following distance requirements and the "following too closely" rule is not triggered.

[0132] (II) Machine learning model determination (based on trajectory time series features and state information) Feature processing: The trajectory temporal features (trajectory deflection angle, velocity change rate) of each target are jointly normalized with the inter-frame state information (acceleration, attitude change) to form a feature set, which is then input into the pre-trained deep neural network classifier.

[0133] Judgment process: Target 1: The trajectory deflection angle is ≤3°, the rate of change of velocity is ≤0.8m / s², the acceleration is stable at 0.5m / s², the classifier judges it as "normal speeding", and there are no other abnormal behaviors; Target 2: Trajectory deflection angle ≤ 2°, velocity change rate ≤ 0.3m / s², acceleration stable at 0.1m / s², classifier judges it as "normal speeding", no other abnormal behavior; Target 3: The trajectory has good continuity, uniform speed, and zero acceleration. The classifier classifies it as "normal pedestrian passage" with no abnormal behavior.

[0134] Finally, based on the combined outputs of the rule model and the machine learning model, the traffic monitoring system determines that: Target 1 (a small car) has an abnormal traffic event of "speeding", Target 2 (an electric bicycle) has an abnormal traffic event of "speeding", and Target 3 (a pedestrian) has no abnormal traffic event; it generates warning information including the type of anomaly, the timestamp of occurrence (corresponding time period from frame 0 to frame 9), the on-site coordinates (target 1 coordinates 27.78, 3.89, 1.1 in frame 9; target 2 coordinates 22.79, 5.78, 1.0 in frame 9), and the associated trajectory segment, and reports it to the traffic control center.

[0135] The pre-trained deep neural network classifier can be trained in the following way: Collect multimodal labeled data (including semantic, geometric, motion state, and other features) of normal / abnormal events in traffic monitoring scenarios, and divide them into training and validation sets according to proportions; A classifier is built based on pre-trained models in the public transportation field (such as CNN, Transformer variants), and the bottom feature extraction layer is frozen. Using the cross-entropy loss function as the optimization objective, the top-level network parameters are dynamically fine-tuned through iterative training using mini-batch gradient descent. When the accuracy of the validation set is stable at ≥95%, the loss value continues to converge, and there is no obvious overfitting, training is stopped, and a deep neural network classifier adapted to this scenario is obtained.

[0136] In this embodiment, Kalman filtering combined with the Hungarian algorithm is used to achieve efficient association of multi-frame continuous target information, ensuring that the same target has a unique identity and continuous trajectory within the monitoring area, and avoiding identity confusion, trajectory breakage or repeated tracking.

[0137] Furthermore, based on core parameters such as 3D position, radial velocity, and acceleration in a unified coordinate system, sensor observation errors are dynamically corrected, outputting accurate real-time target status and historical trajectory data. This provides continuous and complete temporal support for anomaly detection, overcoming the limitations of fragmented single-frame data. Thus, it can simultaneously and stably track and monitor multiple targets, including motor vehicles, non-motorized vehicles, and pedestrians, adapting to scenarios with high target density such as peak traffic hours and complex intersections, meeting the needs of comprehensive, blind-spot-free traffic monitoring.

[0138] Please refer to the attached instruction manual. Figure 2 This is a schematic diagram of a system architecture that can be used to implement the above solution. This diagram illustrates the hierarchical relationship and data flow logic of the functional units in this solution, as described below: The architecture, from top to bottom, includes: 1. Data Acquisition Unit: Includes three types of sensors: camera, lidar, and millimeter-wave radar, used to collect spatial data of traffic scenarios; the output of each sensor is connected to the multimodal sensing acquisition unit, which aggregates the raw data from the three types of sensors. 2. Data Synchronization and Calibration Module: Its input end is connected to the output end of the multimodal sensing acquisition unit. It is used to select multi-sensor spatial data within the same time period based on the acquisition timestamps after autonomous calibration of each sensor for the multi-sensor data acquired by the acquisition unit. This enables temporal matching of multi-source data and performs spatial calibration to ensure the temporal uniformity and spatial consistency of multi-source data. 3. Edge computing and data processing unit: Its input end is connected to the output end of the data synchronization calibration module, and it is used to extract target object features (including spatial position, motion parameters, morphological attributes, etc.) from the synchronized calibration data, and output the target feature data corresponding to each sensor; 4. Backend data fusion and recognition module: Its input end is connected to the output end of the edge computing and data processing unit. It is used to fuse the extracted multi-source target features to obtain fused features (including target state information) containing information such as the target's unique identifier, three-dimensional spatial coordinates in a unified coordinate system, radial velocity, acceleration, vehicle posture, and relative distance to surrounding targets. 5. Multi-target tracking (MOT) module: Its input end is connected to the output end of the background data fusion and recognition module. It receives target state information and basic identification information from the fused features. It uses Kalman filtering algorithm combined with Hungarian algorithm to associate multi-frame data. Based on the three-dimensional spatial coordinates, radial velocity, acceleration and other parameters in the target state information, it predicts and corrects the real-time state of the target. At the same time, it achieves optimal matching of multiple targets through indicators such as spatial coordinate deviation and static attribute similarity. Finally, it generates complete time-series trajectory data of each target object from entering the monitoring area to leaving, and updates the target state information in real time. 6. Abnormal Behavior Detection and Early Warning Module: Its input end is connected to the output end of the Multi-Target Tracking (MOT) module. It receives complete time-series trajectory data of the target and real-time updated target status information. It combines the two to determine abnormal traffic events. The trajectory data reflects the spatiotemporal path of the target, and the status information supplements the behavioral details of each node on the path (such as speed, attitude, relative distance, etc.). Through a dual mechanism of rule model and machine learning, it identifies abnormal events such as running red lights, driving against traffic, and sudden acceleration and deceleration, and generates early warning information that includes the anomaly type, occurrence timestamp, on-site coordinates, and associated trajectory segments. 7. Traffic Control Center Visualization Platform: Its input end is connected to the output end of the abnormal behavior detection and early warning module, and it is used to receive and visualize early warning information, target trajectory and status data to support traffic control decisions.

[0139] The above-mentioned units are connected in sequence through data transmission links to realize the entire data flow process of "data acquisition - synchronous calibration - feature processing - fusion recognition - tracking judgment - early warning display".

[0140] The entire process can be summarized into the following steps: Step 1: Multi-sensor time synchronization and spatial data acquisition Each sensor (camera, lidar, millimeter-wave radar, etc.) autonomously calibrates its local clock based on computer time synchronization protocols (such as NTP, GPS) to ensure consistent timestamps. The sensors collect spatial data (image data, point cloud data, echo signal data) of the traffic scene and send it to the system with the calibrated timestamps.

[0141] Step 2: Filter spatial data within the same time period Based on the corrected timestamps of each sensor, the system filters out multi-source spatial data whose time difference falls within a preset window (e.g., 50 milliseconds) to ensure that the selected data corresponds to the same spatiotemporal state of the traffic scenario, laying a foundation for time consistency for subsequent fusion.

[0142] Step 3: Spatial data preprocessing (coordinate unification and viewpoint alignment) By using external parameter calibration and ICP algorithm, the lidar point cloud data and millimeter-wave radar coordinate data are converted to a unified coordinate system consistent with the camera, realizing spatial alignment of multi-sensor data and eliminating viewpoint deviation (such as misalignment between point cloud and image target positions).

[0143] Step 4: Extract multimodal features of the target object Extract semantic features (object category, bounding box, pose) from image data; Extract geometric features (length, width, height, and outline) and spatial clustering features (point cloud cluster information) from point cloud data. Extract motion state features (radial velocity, acceleration, and relative distance) from echo signal data.

[0144] Step 5: Multimodal feature fusion to generate fused features Semantic features, geometric features, spatial clustering features, and motion state features are concatenated into a one-dimensional concatenated feature. By splicing features into a multimodal attention mechanism deep fusion network, cross-modal attention feature maps (fusion features) are generated, and the weights of key features are strengthened (such as higher weights for motion state features in speed detection).

[0145] Step Six: Preliminary Judgment of Single-Frame Anomalies (Optional, Single-Frame Level) Cross-modal attention feature maps are sliced ​​according to data type to obtain semantic feature slices, geometric morphology + spatial clustering feature slices, and motion state feature slices. Each slice is input into the corresponding single-mode decision module (image / point cloud / radar adaptation model), and the single-frame single-mode detection result is output. The weighted fusion of single-mode results yields a preliminary anomaly determination at the single-frame level (such as "whether the vehicle in the current frame is speeding").

[0146] Step 7: Multi-target tracking (Connecting the single-frame feature / preliminary judgment results from step six, proceeding to temporal-level processing) Kalman filter algorithm: Based on the target's state information (3D coordinates, velocity, acceleration) in the previous frame, predict the theoretical state of the target in the current frame (such as estimated coordinates and velocity). Hungarian algorithm matching: Calculate the cost value (Euclidean distance of spatial location × 0.7 + absolute difference of velocity × 0.3) between the observed features of the current frame (multimodal features from step four) and the predicted state of the previous frame, and construct a matching cost matrix; when the cost value is less than a preset threshold (such as 0.5), it is determined to be the same target, realizing accurate association of target information in multiple frames; Generate time-series data: Based on the "same target" determination result of the Hungarian algorithm, the discrete single-frame fusion features are concatenated to generate complete time-series trajectory data of each target from entering the monitoring area to leaving, and the target status information (corrected coordinates, velocity, acceleration, etc.) is updated in real time.

[0147] Step 8: Final Judgment of Temporal Anomalies (Fused Tracking Results) Based on the complete trajectory data generated in step seven, plus real-time status information, and combined with a dual-determination mechanism: Rule-based model: By analyzing the spatiotemporal path of the trajectory (such as whether it enters a red light area) and status information (such as dwell time and driving direction), rule-based anomalies such as running red lights, driving against traffic, and illegal parking are determined. Machine learning model: By combining trajectory time-series features (such as rate of change of speed and trajectory deflection angle) with state information (such as acceleration fluctuations), it can identify behavioral anomalies such as rapid acceleration and deceleration and irregular lane changes. Output the final result: Generate early warning information containing the anomaly type, occurrence timestamp, on-site coordinates, and associated trajectory segments, and report it to the traffic control center.

[0148] To address the problem that existing multi-sensor fusion technologies struggle to achieve high-precision time synchronization of data from multiple different or similar sensors, thus hindering ITS from accurately detecting abnormal traffic events based on sensor data, this application provides a detection device for traffic monitoring systems, based on the same inventive concept.

[0149] A schematic diagram of the specific structure of the device is shown below. Figure 3 As shown, it includes the following functional units: The spatial data acquisition unit 31 is used to acquire spatial data collected by different sensors within the same time period based on the acquisition timestamps of the spatial data collected by different sensors; the acquisition timestamps are timestamps corrected based on computer time synchronization protocols. Feature extraction unit 32 is used to extract features of the target object from spatial data collected by different sensors within the same time period. The fusion and detection unit 33 is used to fuse the features of the separately extracted target objects to obtain fused features; based on the fused features, it detects whether there are abnormal traffic events.

[0150] In one optional implementation, the different sensors include: different types of sensors; the spatial data collected by the different sensors within the same time period includes: spatial data of different data types collected by the different sensors within the same time period.

[0151] In one optional implementation, the spatial data of different data types include: image data, point cloud data, and echo signal data; then, the feature extraction unit 32 can specifically be used for: Extracting semantic features about the target object from image data; Extracting geometric morphological features and spatial clustering features of target objects from point cloud data; Extract motion state features of the target object from echo signal data.

[0152] In one optional implementation, the fusion feature includes a cross-modal attention feature map; then, the fusion and detection unit 33 can specifically be used for: Semantic features, geometric morphological features, spatial clustering features, and motion state features are concatenated to obtain concatenated features; The spliced ​​features are input into a deep fusion network based on a multimodal attention mechanism, and the deep fusion network generates a cross-modal attention feature map based on the spliced ​​features.

[0153] In one alternative implementation, the deep fusion network performs the following: The cross-modal attention feature map is sliced ​​to obtain feature slices of different data types; The feature slices of the different data types are respectively input into each pre-set single-mode decision module adapted to the corresponding data type to obtain the single-mode detection results generated by each single-mode decision module; the single-mode detection results represent: the decision result of the corresponding single-mode decision module on whether there is an abnormal traffic event in the target object; All single-mode detection results are fused to obtain the final fusion decision result used to characterize whether the target object has an abnormal traffic event.

[0154] In an optional implementation, the apparatus provided in this application embodiment may further include: a target tracking and detection unit, used for: Based on the features of the target object extracted from the spatial data collected by the different sensors in a continuous acquisition period, the three-dimensional spatial coordinates, radial velocity, acceleration and corresponding acquisition timestamp of the target object are obtained. Based on the three-dimensional spatial coordinates, radial velocity, acceleration, and corresponding acquisition timestamps of the target object, as well as the unique identifier assigned to the target object, the motion trajectory and state information of the target object are generated by combining the Kalman filter algorithm with the Hungarian algorithm. Based on the movement trajectory and status information, as well as the preset abnormal traffic event determination conditions, it is determined whether an abnormal traffic event exists.

[0155] The device described in this application ensures data consistency through time synchronization, and then achieves multi-source information complementarity through feature extraction and fusion, forming a closed loop of "time synchronization-feature extraction-fusion detection", which fundamentally solves the problem of inaccurate detection caused by insufficient time synchronization accuracy in the prior art.

[0156] Based on the same inventive concept as the foregoing embodiments of this application, this application also provides a computing device to solve the problem that existing multi-sensor fusion technologies have difficulty in achieving high-precision time synchronization of data from multiple different or similar types of sensors, which makes it difficult for ITS to accurately detect abnormal traffic events based on sensor data.

[0157] like Figure 4 As shown, the computing device includes a memory 41 and a processor 42. The memory 41 can be configured to store various other data to support operation on the electronic device. Examples of such data include instructions for any application or method used to operate on the electronic device. The memory 41 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0158] The processor 42, coupled to the memory 41, is used to execute the computer program stored in the memory 41 to perform a detection method applied to a traffic monitoring system according to Embodiment 1 of this application.

[0159] When the processor 42 executes the computer program in the memory 41, in addition to the functions described above, it can also perform other functions, as detailed in the descriptions of the preceding embodiments.

[0160] Furthermore, such as Figure 4 As shown, the computing device also includes other components such as a display 44, a communication component 43, a power supply component 45, and an audio component 46. Figure 4 The diagram only shows some components and does not mean that the computing device includes only these components. Figure 4 The components shown.

[0161] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a computer, can implement the methods provided in the above embodiments.

[0162] Accordingly, this application also provides a computer program product that stores instructions which, when executed by a computer, cause the computer to implement the methods provided in the above embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0163] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0164] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

[0165] The entire process of model training and application involved in this application strictly follows Article 5 of the Patent Law of the People's Republic of China (hereinafter referred to as "A5") and the relevant examination standards of the Patent Examination Guidelines (2023 Edition) to ensure that the technical solution does not violate the law, social morality and public interest, and that the data acquisition and utilization and model training process all comply with ethical compliance requirements. Specific details are as follows: I. Explanation of whether the data used in the model meets A5 requirements 1. Data Source Legality: All datasets used in training this model were obtained through legal means, covering three categories: publicly authorized data, data authorized by partners, and self-collected compliant data. Publicly authorized data comes from compliant data sources following open-source licenses such as Apache 2.0, with complete copyright attribution and authorization scope clearly marked, and no unauthorized open-source code or data reuse. Data authorized by partners has a formal data usage agreement clearly defining the scope, duration, and confidentiality obligations, with complete authorization chain proof. For self-collected data involving personal information, strict informed consent procedures have been followed, and anonymization processes (including but not limited to field masking, feature anonymization, and differential privacy technology application) have been used to remove personally identifiable information, fully complying with the requirements of the "Interim Measures for the Administration of Generative Artificial Intelligence Services," the "Personal Information Protection Law," and other relevant laws and regulations.

[0166] 2. Data Content Compliance: This dataset has undergone multiple screening and cleaning processes to remove all content that may violate social morality or harm public interests. It contains no obscene, pornographic, violent, discriminatory, or information that endangers national or public safety, nor does it involve the illegal acquisition or use of genetic resources. For data in sensitive fields (such as medical and financial fields), an additional privacy-preserving computation module (including federated learning and secure multi-party computation technologies) ensures that the data is "usable but not visible," avoiding compliance risks during the original data transmission process and ensuring that the data application scenarios and uses comply with public order and good morals and industry regulatory requirements.

[0167] 3. Data Governance Compliance: Establish a complete data traceability system to automatically record the source, collection time, annotation process, cleaning rules, and permission allocation of training data, generating traceable compliance reports to ensure data is verifiable throughout its entire lifecycle. The dataset annotation process is completed by a professional human R&D team, clearly defining the proportion of human creative contributions, avoiding reliance on AI-generated data that has not undergone substantial human modification, and complying with the "human main contribution" examination requirements in AI patent applications.

[0168] II. Explanation of Model Training Process Meeting A5 Requirements 1. Compliance of Training Objectives and Schemes: The training objectives of this model focus on [specific technical scenarios that can be supplemented, such as intelligent driving decision optimization, multimodal information interaction, etc.]. The training scheme and the final output results do not violate any mandatory provisions of laws and administrative regulations, do not harm the public interest or the legitimate rights and interests of others, and do not pose any potential risks of being used for illegal activities, infringing on privacy, or undermining public safety. The model strictly adheres to the ethical principle of "intelligent for good".

[0169] 2. Compliance Management of Training Process: A closed-loop training framework is adopted to ensure compliance and controllability of the training process. The specific process is as follows: First, training samples are obtained through compliant data sources. After the aforementioned data cleaning and desensitization, they are input into the first neural network model to generate preliminary training results. Second, an expert system is introduced to verify the preliminary results. Based on preset rules and human expert experience, the feasibility of the results is evaluated, and outputs that may pose ethical risks or compliance hazards are corrected (such as removing decision logic that violates public order and good morals, and adjusting model parameters that do not comply with safety regulations). Finally, the loss function weights are dynamically optimized based on expert system feedback to strengthen the model's learning of compliant results, avoid overfitting errors or non-compliant labels, and form a closed-loop management system of "data input - model training - expert verification - parameter optimization - result feedback" to ensure that the entire training process complies with A5 ethical review requirements.

[0170] 3. Compliance of Training Environment and Tools: Model training is implemented on a compliant training platform. All open-source frameworks and components used in the training process have obtained the corresponding licenses, and copyright statements and patent citation information are fully retained, with no infringement or reuse. The training environment is built using virtual devices (containers / virtual machines) with fixed random seeds and initial parameter configurations to ensure the reproducibility of the training process. At the same time, through access control and operation log recording, risks such as data leakage and parameter tampering during training are prevented, ensuring the security and compliance of the training process.

[0171] 4. Ethical verification of training results: After the model is trained, it will undergo an additional third-party ethical compliance assessment and algorithm filing review to verify that the model output does not violate social morality or harm public interests. For potentially sensitive scenarios (such as public services and intelligent decision-making), a special result verification mechanism will be established to ensure that the model always complies with A5 and relevant laws and regulations in practical applications.

[0172] In summary, the data and training process used in this application model strictly comply with the relevant provisions of Article 5 of the Patent Law and the Patent Examination Guidelines (2023 Edition), and there are no violations of laws, social ethics, public interests, or illegal use of genetic resources. Therefore, it fully meets the compliance requirements for patent authorization.

Claims

1. A detection method applied to a traffic monitoring system, characterized in that, include: Based on the acquisition timestamps of spatial data collected by different sensors, spatial data collected by different sensors within the same time period is obtained. The timestamp collected is a timestamp corrected based on a computer time synchronization protocol. Features of the target object are extracted from spatial data collected by different sensors within the same time period. The features extracted from the target objects are fused together to obtain fused features; Based on fused features, the system detects whether abnormal traffic events exist.

2. The method according to claim 1, characterized in that: Different sensors include: different types of sensors; Spatial data collected by different sensors within the same time period includes spatial data of different data types collected by different sensors within the same time period.

3. The method according to claim 2, characterized in that, Spatial data of different types includes: image data, point cloud data, and echo signal data; therefore, Features of the target object are extracted from spatial data collected by different sensors within the same time period, including: Spatial alignment is performed on the spatial data of the different data types to obtain spatially aligned image data, point cloud data, and echo signal data. Extracting semantic features about the target object from spatially aligned image data; Extracting geometric morphological features and spatial clustering features of target objects from spatially aligned point cloud data; Extract motion state features of the target object from spatially aligned echo signal data.

4. The method according to claim 3, characterized in that, The fusion feature includes a cross-modal attention feature map; therefore, the features of the extracted target objects are fused to obtain the fusion feature, including: Semantic features, geometric morphological features, spatial clustering features, and motion state features are concatenated to obtain concatenated features; The spliced ​​features are input into a deep fusion network based on a multimodal attention mechanism, and the deep fusion network generates a cross-modal attention feature map based on the spliced ​​features.

5. The method as described in claim 4, characterized in that, Based on fused features, the system detects the existence of abnormal traffic events, including: Performed by the deep fusion network: The cross-modal attention feature map is sliced ​​to obtain feature slices of different data types; The feature slices of the different data types are respectively input into each pre-set single-mode decision module adapted to the corresponding data type to obtain the single-mode detection results generated by each single-mode decision module; the single-mode detection results represent: the decision result of the corresponding single-mode decision module on whether there is an abnormal traffic event in the target object; All single-mode detection results are fused to obtain the final fusion decision result used to characterize whether the target object has an abnormal traffic event.

6. The method as described in claim 1, characterized in that, The method further includes: Based on the features of the target object extracted from the spatial data collected by the different sensors in a continuous acquisition period, the three-dimensional spatial coordinates, radial velocity, acceleration and corresponding acquisition timestamp of the target object are obtained. Based on the three-dimensional spatial coordinates, radial velocity, acceleration, and corresponding acquisition timestamps of the target object, as well as the unique identifier assigned to the target object, the motion trajectory and state information of the target object are generated by combining the Kalman filter algorithm with the Hungarian algorithm. Based on the movement trajectory and status information, as well as the preset abnormal traffic event determination conditions, it is determined whether an abnormal traffic event exists.

7. A detection device applied to a traffic monitoring system, characterized in that, include: The spatial data acquisition unit is used to acquire spatial data collected by different sensors within the same time period based on the acquisition timestamps of the spatial data collected by different sensors. The timestamp collected is a timestamp corrected based on a computer time synchronization protocol. The feature extraction unit is used to extract the features of the target object from the spatial data collected by different sensors within the same time period. The fusion and detection unit is used to fuse the features of the separately extracted target objects to obtain fused features; Based on fused features, the system detects whether abnormal traffic events exist.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 6.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, When the processor executes the program, it implements the steps of the method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product stores instructions that, when executed by a computer, cause the computer to perform the method as described in any one of claims 1 to 6.

Images