Semi-automated data annotation system for intelligent transportation

Abstract

The application relates to the technical field of video recognition, and particularly relates to a semi-automatic data labeling system for intelligent traffic. The system comprises a feature extraction module, which is used for respectively extracting features of multi-modal data to obtain multi-modal features of detection targets in a detection area; an information fusion module, which is used for correlating the multi-modal features of the detection targets, determining the confidence of each modal feature under the current environment according to environmental information of the detection area, and performing multi-modal feature fusion based on the current confidence of each modal feature; a fusion detection module, which is used for performing conflict detection on the multi-modal fusion information of the detection targets, identifying feature conflicts between different modes of the detection targets, and performing conflict labeling on the detection targets with the feature conflicts; and an automatic labeling module, which is used for performing data labeling on the detection targets according to the fusion information through a pre-trained feature labeling model when there is no conflict between the multi-modal features.

Description

Technical Field

[0001] This invention relates to the field of video recognition technology, and more specifically to a semi-automated data annotation system for intelligent transportation. Background Technology

[0002] With the development of intelligent transportation systems, video and image data play a crucial role in traffic management and monitoring. For example, video cloud networking platforms aggregate video and image data from various transportation sectors, including operating highways, highways under construction, national and provincial trunk roads, passenger transport, and water transport. Utilizing the rapid algorithm iteration and intelligent analysis capabilities of an AI platform, video event monitoring and analysis are achieved, covering the monitoring of violations and irregularities on highways, effectively improving the efficiency and accuracy of traffic management. However, video cloud data needs to be labeled to form structured data to support the algorithm iteration of the AI ​​platform. This process remains time-consuming and labor-intensive, currently relying mainly on manual labeling, which suffers from high costs, low efficiency, and difficulty in guaranteeing accuracy.

[0003] Currently, in intelligent transportation systems, the annotation of video data mainly includes target detection and initial localization: identifying potential targets (vehicles, pedestrians, cyclists, traffic signs, traffic lights, cones, etc.) in images or point clouds and providing rough bounding boxes; target tracking: associating the same target in a continuous video sequence and assigning it a unique ID; basic classification and attribute recognition: classifying detected targets into broad categories (e.g., cars, trucks, pedestrians, bicycles) or identifying some significant attributes (e.g., red vehicles, large vehicles); lane line / road marking detection: detecting the position and type of road markings such as lane lines, stop lines, pedestrian crossings, and arrows; and traffic light status recognition. In the past, this information annotation was mostly done manually, but in practical applications, manual annotation is inefficient. Existing technologies also include some automatic annotation methods that use pre-trained recognition models to identify targets in images. For example, CN111476324B discloses an automatic traffic data annotation method. However, in practical applications, it has been found that traffic information often presents many complex scenarios, such as targets that are severely obscured and only a small part is visible, targets that are truncated at the edge of the image, and small targets at a distance; or targets that are densely interspersed. The automatic annotation model has a low ability to understand complex scenarios and often requires manual review and correction in the later stages, which also leads to low efficiency. Summary of the Invention

[0004] The technical problem solved by this invention is to provide a semi-automatic data annotation system for intelligent transportation, which can identify targets that can be automatically annotated and targets that cannot be automatically annotated in the detection area, and use automatic annotation and manual annotation methods to annotate the detection targets respectively, thereby reducing the time for manual review and improving the efficiency of data annotation.

[0005] The basic solution provided by this invention is a semi-automatic data annotation system for intelligent transportation, comprising an information sensing device, an environmental sensing device, and a server. The information sensing device is located in the detection area and is used to collect multimodal data of the detection area. The information sensing device includes a camera, a lidar, and a millimeter-wave radar. The environmental sensing device is used to collect environmental information of the detection area. The server includes a feature extraction module, an information fusion module, a fusion detection module, an automatic annotation module, and a manual annotation module. The feature extraction module is used to extract features from the multimodal data to obtain the multimodal features of the detected target in the detection area. The multimodal features include image features, point cloud features, and radar features. The information fusion module is used to determine the confidence level of each modal feature under the current environment based on the environmental information of the detection area, and to perform multimodal feature fusion based on the current confidence level of each modal feature to obtain the multimodal fusion information of the detection target; The fusion detection module is used to perform conflict detection on the multimodal fusion information of the target, identify feature conflicts between different modalities of the target, and perform conflict labeling on the target with feature conflicts. The automatic annotation module is used to annotate the detection target based on the fusion information using a pre-trained feature annotation model when there is no conflict between multimodal features; The manual labeling module is used to distribute detection targets with conflict annotations to manual labelers.

[0006] The principle and advantages of this invention are as follows: Multimodal data of the detection area is collected using multi-source sensors, including a camera, LiDAR, and millimeter-wave radar. The camera detects and collects images of the target in the area; the LiDAR collects point cloud data of the target; and the millimeter-wave radar collects velocity and distance information of the target. After spatiotemporal preprocessing of the collected multimodal data, feature extraction is performed. Image features of the target are extracted from the image data collected by the camera, and point cloud features are extracted from the point cloud data collected by the LiDAR. The velocity and distance information collected by the millimeter-wave radar are converted into radar features. Then, multimodal feature fusion is performed, such as associating and fusing the 2D bounding boxes detected by the camera and the 3D bounding boxes detected by the LiDAR, and fusing the position and velocity of the target detected by the millimeter-wave radar with the category and appearance of the target detected by the camera to obtain complementary information.

[0007] After feature fusion is completed, conflict detection is performed on the fused information. This includes checking for situations such as complete discrepancies between multimodal results, single-modal anomalies, or blurred recognition boundaries. Specifically, one modality might detect a target, while another modality either fails to detect it at the corresponding location or the detected target information deviates significantly. Multimodal conflicts indicate complex scenarios in the current detection area. For example, in foggy weather, a vehicle's speed and position might be detected by millimeter-wave radar, but the point cloud data from lidar might be incomplete, or the contours of the video image might be missing. In such cases, manual annotation is required. If there are no conflicts, the system can automatically annotate the data.

[0008] Compared to existing technologies, it has the following advantages: 1. In this solution, identification and annotation are performed through multimodal feature fusion. It simultaneously processes multi-source data, including camera video, laser point cloud, and millimeter-wave radar data. By utilizing the complementarity of different modal information, such as the texture of the image combined with the precise 3D position of the point cloud and the detection range of the millimeter-wave radar, compared with the automatic annotation of target information after traditional single-modal data acquisition, it can achieve more robust and accurate automatic pre-annotation. The annotation accuracy is higher in more complex environments such as night, rain, fog, and occlusion, reducing the difficulty of subsequent review and correction.

[0009] 2. Collision detection identifies targets that can be automatically labeled and those that cannot. Based on the collision detection results, the system chooses to automatically label the targets or provide manual assistance. This human-machine collaboration perfectly combines the system's strengths in data processing, pattern recognition, and rapid computation with the human expertise in high-level understanding, handling ambiguous scenarios, and final decision-making. Furthermore, by utilizing collision detection, which is essential after multimodal fusion, the system determines whether manual intervention is necessary. Without introducing excessive steps, it can identify targets in complex scenarios and hand them over to manual labeling, further reducing the difficulty of subsequent review and correction.

[0010] Furthermore, the feature extraction module includes an image feature module, a point cloud feature module, and a radar feature module, and the information fusion module includes an environmental feature analysis module and a feature association module; The image feature module is used to segment the key frames of the video captured by the camera into various detection targets using an image segmentation algorithm, add detection boxes to each detection target in the image, and determine the image features of the detection targets in the boxes through image recognition. The image features include image coordinate information, texture information, and contour information. The point cloud feature module is used to identify the point cloud features of each detection target in the detection area based on the point cloud data collected by the lidar. The point cloud features include spatial location information and shape information. The radar feature module is used to obtain the radar features of the detected target based on the speed and distance information of the detected target collected by the millimeter-wave radar. The radar features include radar position information, attitude information, and speed information.

[0011] The environmental feature analysis module is used to generate confidence scores for each mode under the current environment based on environmental features, and to determine the primary mode and secondary mode based on the confidence scores; The feature association module is used to determine the location information of the detected target by using the image coordinate information of image features, the spatial location information of point cloud features, and the radar location information of radar data, respectively. A spatial coordinate system is established with the location information of the main mode as the origin, and the location information of the secondary mode is mapped into the spatial coordinate system.

[0012] The image feature module extracts image coordinates, texture, and contour information of the detected target from video footage; the point cloud feature module extracts spatial location and shape information from LiDAR point cloud data; and the radar feature module extracts position, attitude, and velocity information from millimeter-wave radar data. The information fusion module includes an environmental feature analysis module and a feature association module. The former evaluates the reliability of each modality and determines the primary and secondary modalities based on environmental information, while the latter maps features from different modalities to the same spatial coordinate system using unified position information, achieving alignment and association of multimodal data. The principle of this design is to process data from different sensors in modules to ensure that various features are fully extracted, then dynamically adjust the weights of each modality through environmental perception, and finally achieve accurate matching and fusion of multimodal information based on the spatial coordinate system.

[0013] Its advantages lie in the fact that, compared to existing technologies that use single or simple superposition of multimodal processing methods, this solution significantly improves the completeness of feature extraction and the accuracy of fusion through modular division of labor and spatial alignment mechanisms. For example, in real-world traffic scenarios, cameras may lose texture details due to insufficient lighting at night, LiDAR may suffer from sparse point clouds due to rain or fog, and millimeter-wave radar may lack sensitivity to static targets. This solution uses an environmental feature analysis module to evaluate the confidence level of each modality in real time and establishes a coordinate system with the master modality as a reference during the feature association stage. This allows the system to guide other modalities to complete data completion and calibration even when the quality of some modalities deteriorates. This approach not only enhances the system's adaptability to complex environments but also provides a more reliable multimodal fusion information foundation for subsequent conflict detection and automatic annotation, thereby reducing the frequency of manual intervention and improving overall annotation efficiency.

[0014] Furthermore, the conflict detection module includes a location conflict detection module, an attribute conflict detection module, and an existence conflict detection module; The position conflict detection module is used to calculate the position deviation of the same detection target in different modes in a spatial coordinate system. When the deviation exceeds a preset distance threshold, it is marked as a position conflict. The attribute conflict detection module is used to compare the attribute recognition results of different modalities for the same target, and to mark the inconsistencies as attribute conflicts. The existence detection module is used to detect whether there is a behavior where one modality recognizes the target while another modality does not. If such a behavior exists, it is marked as an existence conflict.

[0015] Inconsistencies between multimodal information are identified from three dimensions: location, attribute, and existence. The location conflict detection module calculates the positional deviation of the same target in different modalities to determine if it exceeds a reasonable range. The attribute conflict detection module compares the target attribute identification results of different modalities to check for discrepancies in attribute descriptions such as category and color. The existence conflict detection module is used to detect situations where one modality detects a target while another does not. These three detection methods together constitute a multi-layered conflict identification system, ensuring that the system can comprehensively capture anomalies and contradictions in multimodal data.

[0016] Its advantage lies in the fact that existing conflict detection technologies are often limited to a single dimension, such as focusing only on positional deviations or simple consensus voting, making it difficult to handle the complex and intertwined situations of multi-source sensor data in intelligent transportation scenarios. This solution, through a triple detection mechanism, can more finely distinguish conflicts of different natures. For example, in rainy or foggy weather, the position of LiDAR point clouds may drift due to moisture interference, while camera images can still maintain the target outline. In this case, positional conflict detection can identify deviations, attribute conflict detection can verify whether the target categories are consistent, and existence detection can confirm whether the target has been captured by both parties. This refined conflict recognition capability allows the system not only to detect obvious anomalies but also to capture potential inconsistencies, thereby more accurately filtering out complex targets that truly require human intervention, avoiding the misjudgment of simple conflicts as automatically labelable, and further improving the efficiency and labeling quality of human-machine collaboration.

[0017] Furthermore, the environmental feature analysis module includes an environmental parameter analysis module and a confidence score module.

[0018] The environmental parameter analysis module is used to generate normalized environmental impact factors based on environmental information. These environmental impact factors include visibility factors, illumination factors, weather factors, and target motion factors. The confidence score module is used to generate confidence scores for each modality based on environmental impact factors.

[0019] in This represents the confidence score for mode m. Visibility factor Light factor, As a weather factor, For the target motion factor, , , , These are the modal sensitivity coefficients of mode m with respect to visibility factor, illumination factor, weather factor, and target motion factor, respectively. + + + =1.

[0020] The environmental feature analysis module quantifies the impact of environmental factors on the data quality of each modality, enabling a scientific assessment of the confidence level of each modality. This module first extracts visibility, illumination, weather, and target motion factors from environmental information and normalizes these factors. Then, based on the sensitivity of each modality to different environmental factors, it calculates the confidence score for each modality using a weighted summation method. The weight coefficients reflect the degree of dependence of each modality on environmental factors; for example, cameras are sensitive to changes in illumination, lidar is sensitive to changes in visibility, and millimeter-wave radar is more sensitive to target motion. Through this mathematical model, the system can dynamically adjust the confidence level of each modality's data under different environmental conditions, providing a basis for subsequent feature fusion and conflict resolution.

[0021] Its advantage lies in the fact that traditional multimodal systems often assume that each sensor has fixed reliability in any environment, or select modes based solely on simple rules, making it difficult to adapt to the changing environmental conditions in traffic scenarios. This solution introduces a quantitative evaluation mechanism based on environmental factors and modal sensitivity coefficients, enabling the system to automatically adjust the weights of each mode based on real-time environmental data, achieving more adaptive multimodal fusion. For example, during dusk, as illumination gradually weakens, the confidence score of cameras may decrease, while LiDAR and millimeter-wave radar are less affected by illumination, and their scores remain relatively stable. The system will then automatically reduce the proportion of image modes in the fusion, relying more on point cloud and radar data. This confidence evaluation method based on environmental perception significantly improves the robustness of the system in complex scenarios such as severe weather, low light, and high-speed movement, allowing the automatic annotation process to maintain high accuracy even when the environment changes, reducing the number of conflicting annotations caused by environmental interference, and thus reducing the burden of manual review.

[0022] Furthermore, the fusion detection module is characterized by including a master-slave modality repair module, a missing data repair module, and a spatiotemporal repair module; The master-slave modality repair module is used to correct the conflict features of the secondary modality when there is a conflict between different modalities of the detected target. If the confidence score difference between the master modality and this module is higher than a preset first threshold, the detection result of the master modality shall be used as the standard. When the score difference between the master modality and the secondary modality is lower than the preset first threshold but higher than the preset second threshold, the recognition result consistent with the majority modality shall be selected as the final labeling basis. The missing feature repair module is used to complete the feature by using features from other modalities when there are missing modal features. The spatiotemporal repair module is used to perform spatiotemporal smoothing and correction on the conflict features of the current frame when the detected target has continuous frame motion trajectories and historical features.

[0023] Building upon conflict detection, an automatic repair mechanism is introduced to attempt to autonomously correct some repairable conflicts before human intervention. The master-slave modality repair module determines the conflict resolution strategy based on the confidence score difference. If the score difference between the master and slave modalities is large, the low-confidence modality is corrected based on the high-confidence modality; if the scores are close and most modalities are consistent, the majority result is adopted. The missing feature repair module uses features from other modalities to infer and complete the missing features of a particular modality. The spatiotemporal repair module uses the target's motion trajectory and historical features in consecutive frames to smooth and correct the conflict information of the current frame in both time and space. These repair mechanisms together constitute a progressive conflict handling process, aiming to resolve inconsistencies in multimodal data as much as possible through the system's own capabilities.

[0024] Its advantage lies in the fact that existing semi-automatic annotation systems typically hand over the task to manual processing immediately after a conflict is detected, lacking the ability to autonomously handle repairable conflicts, resulting in a heavy and inefficient manual annotation workload. This solution, by introducing a multi-level repair module, can selectively and automatically correct conflicts when they occur, thereby reducing unnecessary transfers. For example, when a point cloud is missing due to a brief obstruction of the LiDAR, the missing data repair module can use historical and real-time data from the camera and millimeter-wave radar to complete the shape and position information; when a target's trajectory is smooth in consecutive video frames but a positional conflict occurs in a certain frame due to noise, the spatiotemporal repair module can correct the anomaly based on information from preceding and following frames. This repair mechanism not only reduces the burden of manual annotation but also improves the system's continuous annotation capability and stability in dynamic scenes, allowing more edge cases to be automatically processed by the system, further optimizing the efficiency of human-machine collaboration and the overall intelligence level of the annotation process.

[0025] Furthermore, the server also includes a spatiotemporal synchronization module, which is used to perform timestamp calibration and spatial coordinate system alignment on the multimodal data collected by the information sensing device and the environmental information collected by the environmental sensing device before the feature extraction module performs feature extraction, so as to unify the spatiotemporal reference of each data. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of an embodiment of the present invention. Detailed Implementation

[0027] The following detailed description illustrates the specific implementation method: The basic implementation examples are as follows: Figure 1 As shown: A semi-automated data annotation system for intelligent transportation includes an information sensing device, an environmental sensing device, and a server. The information sensing device is located in the detection area and is used to collect multimodal data of the detection area. The information sensing device includes a camera, a lidar, and a millimeter-wave radar. The environmental sensing device is used to collect environmental information of the detection area. The server includes a feature extraction module, an information fusion module, a fusion detection module, an automatic annotation module, and a manual annotation module. The feature extraction module is used to extract features from the multimodal data to obtain the multimodal features of the detected target in the detection area. The multimodal features include image features, point cloud features, and radar features. The information fusion module is used to determine the confidence level of each modal feature under the current environment based on the environmental information of the detection area, and to perform multimodal feature fusion based on the current confidence level of each modal feature to obtain the multimodal fusion information of the detection target; The fusion detection module is used to perform conflict detection on the multimodal fusion information of the target, identify feature conflicts between different modalities of the target, and perform conflict labeling on the target with feature conflicts. The automatic annotation module is used to annotate the detection target based on the fusion information using a pre-trained feature annotation model when there is no conflict between multimodal features; The manual labeling module is used to distribute detection targets with conflict annotations to manual labelers.

[0028] The feature extraction module includes an image feature module, a point cloud feature module, and a radar feature module; the information fusion module includes an environmental feature analysis module and a feature association module. The image feature module is used to segment the key frames of the video captured by the camera into various detection targets using an image segmentation algorithm, add detection boxes to each detection target in the image, and determine the image features of the detection targets in the boxes through image recognition. The image features include image coordinate information, texture information, and contour information. The point cloud feature module is used to identify the point cloud features of each detection target in the detection area based on the point cloud data collected by the lidar. The point cloud features include spatial location information and shape information. The radar feature module is used to obtain the radar features of the detected target based on the speed and distance information of the detected target collected by the millimeter-wave radar. The radar features include radar position information, attitude information, and speed information.

[0029] The environmental feature analysis module is used to generate confidence scores for each mode under the current environment based on environmental features, and to determine the primary mode and secondary mode based on the confidence scores; The feature association module is used to determine the location information of the detected target by using the image coordinate information of image features, the spatial location information of point cloud features, and the radar location information of radar data, respectively. A spatial coordinate system is established with the location information of the main mode as the origin, and the location information of the secondary mode is mapped into the spatial coordinate system.

[0030] Specifically, after the system starts, the camera acquires a video stream of the detection area at a rate of 25 frames per second, the LiDAR scans the surrounding environment at a frequency of 10 Hz to generate point cloud data, and the millimeter-wave radar outputs the speed and distance information of the detected targets at a frequency of 20 Hz. The image feature module extracts 5 keyframes per second from the video stream and runs the Mask R-CNN image segmentation algorithm for each keyframe. This algorithm is pre-trained on the COCO dataset and fine-tuned in a highway scene. After segmentation, a pixel-level mask is obtained for each detected target. Then, a rectangular detection box is added around the mask. The coordinates of the upper left and lower right corners of the detection box are recorded in pixels. At the same time, the gray-level co-occurrence matrix is ​​extracted from the mask area as texture features, and Fourier descriptors are extracted from the mask boundary as contour features. After receiving the point cloud frames from the lidar, the point cloud feature module first performs voxel filtering downsampling with a voxel grid side length of 0.2 meters. Then, it runs an Euclidean clustering algorithm with a clustering distance threshold of 0.5 meters, a minimum cluster size of 10, and a maximum cluster size of 500. Each clustering result corresponds to a detected target. The centroid coordinates of the target are calculated from its point cloud set as its spatial position information, and the length, width, and height of its bounding box are calculated as its shape information. After receiving the target list from the millimeter-wave radar, the radar feature module receives the radial range, azimuth, elevation, and radial velocity for each target. These parameters are converted to a Cartesian coordinate system with the radar installation location as the origin to obtain the radar position coordinates of the target. The velocity vector is calculated using the trigonometric function relationship between radial velocity and azimuth, and the attitude information is estimated by the position difference between two consecutive frames. The environmental feature analysis module receives temperature, humidity, light intensity, visibility, and rainfall data for the current detection area from the environmental sensing device. Based on this data, it calculates the confidence score for each modality. After comparing the scores of the three modalities, the modality with the highest score is determined as the primary modality, and the other two modalities are determined as secondary modalities. After determining the primary modality, the feature association module establishes a three-dimensional spatial coordinate system with the location information of a target detected by the primary modality as the origin. The three axes of this coordinate system point to the longitudinal, lateral, and vertical directions of the road, respectively. Then, by reading image coordinate information from image features and combining it with the intrinsic and extrinsic parameter matrices of the camera, it calculates the projected position of the target in three-dimensional space. It also directly obtains the target's position in three-dimensional space by reading spatial position information from point cloud features. Furthermore, it calculates the target's position in three-dimensional space by reading radar position information from radar features and combining it with the installation position and attitude parameters of the millimeter-wave radar. The position coordinates of these two secondary modalities are mapped to the coordinate system established with the primary modality as the origin through translation and rotation matrices, thus ensuring that the position descriptions of the target by the three modalities are under the same spatial reference.

[0031] The conflict detection module includes a location conflict detection module, an attribute conflict detection module, and an existence conflict detection module. The position conflict detection module is used to calculate the position deviation of the same detection target in different modes in a spatial coordinate system. When the deviation exceeds a preset distance threshold, it is marked as a position conflict. The attribute conflict detection module is used to compare the attribute recognition results of different modalities for the same target, and to mark the inconsistencies as attribute conflicts. The existence detection module is used to detect whether there is a behavior where one modality recognizes the target while another modality does not. If such a behavior exists, it is marked as an existence conflict.

[0032] Specifically, the fusion detection module starts after the feature association module completes the spatial coordinate mapping. The position conflict detection module first extracts the 3D position coordinates of the same detected target in the main mode and each sub-mode from the association results output by the feature association module. It then calculates the distance difference between the main mode and the image sub-mode, the distance difference between the main mode and the point cloud sub-mode, and the distance difference between the main mode and the radar sub-mode. The distance difference is calculated using Euclidean distance. The preset distance threshold for this module is 3 meters, which is determined based on the standard lane width of 3.75 meters on a highway and the average size of vehicles, allowing for a certain positioning error. When the position deviation between a sub-mode and the main mode exceeds 3 meters, the module marks the target as a position conflict, records the conflict type as position conflict, and records the name of the mode with the largest deviation. When the attribute conflict detection module detects the same target in different modalities, it reads the target category label output by the image feature module, such as car, truck, pedestrian, or bicycle; reads the target category label output by the point cloud feature module through the point cloud shape classifier; and reads the target category label inferred by the radar feature module through velocity and radar cross section. The three labels are compared one by one. If there is an inconsistency, it is marked as an attribute conflict. For example, if the image identifies it as a car but the radar identifies it as a truck, an attribute conflict marking is triggered.

[0033] The existence conflict detection module receives target lists output by three feature extraction modules and performs cross-modal matching for each target. Specifically, it checks whether each target in the primary modality has a matching entry in the target list of the secondary modality of the image. The matching conditions are that the spatial distance between the two targets is less than 2 meters and the target sizes are similar. If no matching entry is found, the target is marked as missing in the image modality. Similarly, it checks whether the target has a matching entry in the target list of the radar modality. If not, it is marked as missing in the radar modality. When any target has at least one missing modality, the module marks the target as having an existence conflict. The three conflict detection modules run in parallel and finally output a conflict marking result. This result contains three Boolean variables representing whether a positional conflict, attribute conflict, and existence conflict exist, as well as the corresponding conflict details.

[0034] The environmental feature analysis module includes an environmental parameter analysis module and a confidence score module.

[0035] The environmental parameter analysis module is used to generate normalized environmental impact factors based on environmental information. These environmental impact factors include visibility factors, illumination factors, weather factors, and target motion factors. The confidence score module is used to generate confidence scores for each modality based on environmental impact factors.

[0036] in This represents the confidence score for mode m. Visibility factor Light factor, As a weather factor, For the target motion factor, , , , These are the modal sensitivity coefficients of mode m with respect to visibility factor, illumination factor, weather factor, and target motion factor, respectively. + + + =1.

[0037] Specifically, the environmental parameter analysis module receives real-time environmental data of the detection area from the environmental sensing device. Visibility data comes from a visibility sensor, with a measurement range of 0 to 1000 meters; light intensity data comes from a light intensity sensor, with a measurement range of 0 to 100,000 lux; weather data comes from a rain sensor and a temperature and humidity sensor, outputting a weather type code: 0 for sunny, 1 for cloudy, 2 for light rain, 3 for moderate rain, 4 for heavy rain, and 5 for fog; the target motion factor comes from the statistical average of the absolute values ​​of the speeds of all targets within the detection area, in meters per second. This module normalizes the raw data. The visibility factor is obtained by dividing the current visibility value by 1000 meters, ranging from 0 to 1, and is forcibly set to 0 when visibility is below 50 meters. The illumination factor is obtained by dividing the current illumination intensity value by 100,000 lux, ranging from 0 to 1, and is forcibly set to 0.1 when illumination intensity is below 100 lux to simulate the lowest visible light at night. The weather factor is obtained by querying a preset weather code mapping table, with 1 for sunny days, 0.9 for cloudy days, 0.7 for light rain, 0.5 for moderate rain, 0.3 for heavy rain, and 0.2 for foggy days. The target motion factor is converted to a value between 0 and 1 by dividing the statistical average of the absolute values ​​of all target speeds by 120 kilometers per hour, and is set to 1 when the average speed exceeds 120 kilometers per hour. After obtaining the four environmental impact factors, the confidence scoring module calculates a confidence score for each modality. For the image mode, the sensitivity coefficients for visibility factor, illumination factor, weather factor, and target motion factor are set to 0.2, 0.5, 0.2, and 0.1, respectively, because image quality is most sensitive to changes in illumination, and a confidence score is calculated. For the lidar mode, the sensitivity coefficients for visibility factor, illumination factor, weather factor, and target motion factor are set to 0.3, 0.1, 0.4, and 0.2, respectively, because lidar is highly sensitive to changes in weather and visibility, and a confidence score is calculated. For the millimeter-wave radar mode, the sensitivity coefficients for visibility factor, illumination factor, weather factor, and target motion factor are set to 0.1, 0.1, 0.2, and 0.6, respectively, because millimeter-wave radar is most sensitive to target motion, and a confidence score is calculated. The confidence score module normalizes the scores of the three modes, compares their magnitudes, and determines the mode with the highest score as the dominant mode under the current environmental conditions.

[0038] The fusion detection module is characterized by further including a master-slave modality repair module, a missing data repair module, and a spatiotemporal repair module; The master-slave modality repair module is used to correct the conflict features of the secondary modality when there is a conflict between different modalities of the detected target. If the confidence score difference between the master modality and this module is higher than a preset first threshold, the detection result of the master modality shall be used as the standard. When the score difference between the master modality and the secondary modality is lower than the preset first threshold but higher than the preset second threshold, the recognition result consistent with the majority modality shall be selected as the final labeling basis. The missing feature repair module is used to complete the feature by using features from other modalities when there are missing modal features. The spatiotemporal repair module is used to perform spatiotemporal smoothing and correction on the conflict features of the current frame when the detected target has continuous frame motion trajectories and historical features.

[0039] Specifically, the master-slave modality repair module begins operation after the fusion detection module detects a conflict. This module first reads the confidence scores of the three modalities in the current environment and calculates the score difference between the master modality and each sub-modality. A first threshold of 0.3 and a second threshold of 0.1 are preset; these thresholds were obtained through extensive experimental statistics and can effectively distinguish the reliability differences between modalities. When the score difference between the master modality and a sub-modality is higher than 0.3, the module considers the detection result of the master modality to have an overwhelming advantage. In this case, the position coordinates of the sub-modality are replaced with the position coordinates of the master modality, and the attribute labels of the sub-modality are replaced with the attribute labels of the master modality. The repaired features are used as the final annotation basis. When the score difference between the primary modality and a secondary modality is less than 0.3 but greater than 0.1, this module considers the reliability difference between the primary and secondary modalities to be small. In this case, the recognition results of any two modalities that are consistent are read as the final labeling basis. For example, if the image modality and the LiDAR modality both identify it as a sedan, but the LiDAR modality identifies it as a truck, then the sedan category label is adopted. If the recognition results of the three modalities are different, then the result of the primary modality is used. The missing object repair module is activated when a conflict is detected. When a modality is missing the detection information of a certain target, this module first checks whether the missing target has been detected in both the primary modality and other secondary modalities. If so, it fills in the position coordinates of the missing modality by averaging the position information of the primary modality and other secondary modalities, and fills in the attribute label of the missing modality by majority voting on the attribute categories of the primary modality and other secondary modalities. When multiple modalities are missing the same target at the same time, this module directly copies and fills in the missing modality with the features of the remaining uniquely detected modality.

[0040] The spatiotemporal repair module maintains a target historical trajectory cache of 10 frames. When a target is detected to have a positional or attribute conflict in the current frame, this module reads the target's historical features from the previous 5 and next 5 frames. For positional information, it uses median filtering, taking the median of the target's coordinates across consecutive frames as the corrected position for the current frame. For attribute information, it uses majority voting, selecting the category with the highest frequency of the target's attribute label across consecutive frames as the corrected attribute for the current frame. The three repair modules execute sequentially: the master-slave modality repair module processes first, the missing data repair module second, and the spatiotemporal repair module last. The final output is repaired, conflict-free fusion information for use by the automatic annotation module. If the conflict remains unresolved after processing by the three repair modules, the target is still marked as conflicted and distributed to the manual annotation module.

[0041] The server also includes a spatiotemporal synchronization module, which is used to perform timestamp calibration and spatial coordinate system alignment on the multimodal data collected by the information sensing device and the environmental information collected by the environmental sensing device before the feature extraction module performs feature extraction, so as to unify the spatiotemporal reference of each data.

[0042] Specifically, the spatiotemporal synchronization module operates continuously during system operation, serving as the first step in data preprocessing. This module receives video streams from the camera and their corresponding image acquisition timestamps (accurate to milliseconds) from the information sensing device; it also receives LiDAR point cloud frames and their corresponding scan start and end timestamps; it receives the millimeter-wave radar target list and its corresponding detection timestamps; and it receives environmental information from the environmental sensing device and its corresponding sampling timestamps. Because the three sensors have different acquisition frequencies—25 frames per second (40 milliseconds per frame) for the camera, 10 Hz (100 milliseconds per frame) for the LiDAR, and 20 Hz (50 milliseconds per frame) for the millimeter-wave radar—this module uses the LiDAR scan time as the reference time for timestamp calibration. Specifically, for each LiDAR point cloud frame's scan midpoint, it finds the two nearest frames from the camera timestamp sequence and generates a virtual image frame for that moment through linear interpolation; similarly, it finds the two nearest detection times from the millimeter-wave radar target list timestamp sequence and generates a virtual radar target list for that moment through linear interpolation. After time calibration, all data is unified to the LiDAR scanning time. Regarding spatial coordinate system alignment, this module performs a calibration during system installation, recording the translation vector and rotation matrix of the camera installation position relative to the LiDAR installation position, and recording the translation vector and rotation matrix of the millimeter-wave radar installation position relative to the LiDAR installation position. In actual operation, this module reads the spatial coordinates from each data source, transforming the pixel coordinates of the camera detection box to the camera coordinate system using the camera intrinsic parameter matrix, and then transforming them to the LiDAR coordinate system using the calibrated rotation matrix and translation vector. Similarly, it transforms the polar coordinates of the target detected by the millimeter-wave radar to the millimeter-wave radar coordinate system using the millimeter-wave radar installation position and attitude parameters, and then transforming them to the LiDAR coordinate system using the calibrated rotation matrix and translation vector. After spatial coordinate alignment, the target position descriptions for all modalities are unified in the LiDAR coordinate system. This module outputs timestamp-aligned and spatially coordinate-unified multimodal data packets. Each data packet contains image feature data, point cloud feature data, and radar feature data at the same time, as well as environmental information data at that time. These data packets are sequentially sent to the feature extraction module for further processing in chronological order.

[0043] The above are merely embodiments of the present invention. Commonly known structures and characteristics are not described in detail here. Those skilled in the art are aware of all common technical knowledge in the field prior to the application date or priority date, are aware of all existing technologies in that field, and have the ability to apply conventional experimental methods prior to that date. Those skilled in the art can, under the guidance of this application, improve and implement this solution in combination with their own capabilities. Some typical known structures or methods should not be obstacles for those skilled in the art to implement this application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the structure of the present invention. These should also be considered within the scope of protection of the present invention, and will not affect the effectiveness of the implementation of the present invention or the practicality of the patent. The scope of protection claimed in this application should be determined by the content of its claims, and the specific embodiments described in the specification can be used to interpret the content of the claims.

Claims

1. A semi-automated data annotation system for intelligent transportation, characterized in that: The system includes an information sensing device, an environmental sensing device, and a server. The information sensing device is located in the detection area and is used to collect multimodal data of the detection area. The information sensing device includes a camera, a lidar, and a millimeter-wave radar. The environmental sensing device is used to collect environmental information of the detection area. The server includes a feature extraction module, an information fusion module, a fusion detection module, an automatic annotation module, and a manual annotation module. The feature extraction module is used to extract features from the multimodal data to obtain the multimodal features of the detected target in the detection area. The multimodal features include image features, point cloud features, and radar features. The information fusion module is used to associate the multimodal features of each detected target, determine the confidence level of each modal feature under the current environment based on the environmental information of the detection area, and perform multimodal feature fusion based on the current confidence level of each modal feature to obtain the multimodal fusion information of the detected target; The fusion detection module is used to perform conflict detection on the multimodal fusion information of the target, identify feature conflicts between different modalities of the target, and perform conflict labeling on the target with feature conflicts. The automatic annotation module is used to annotate the detection target based on the fusion information using a pre-trained feature annotation model when there is no conflict between multimodal features; The manual labeling module is used to distribute detection targets with conflict labels to manual labelers; The feature extraction module includes an image feature module, a point cloud feature module, and a radar feature module; the information fusion module includes an environmental feature analysis module and a feature association module. The image feature module is used to segment the key frames of the video captured by the camera into various detection targets using an image segmentation algorithm, add detection boxes to each detection target in the image, and determine the image features of the detection targets in the boxes through image recognition. The image features include image coordinate information, texture information, and contour information. The point cloud feature module is used to identify the point cloud features of each detection target in the detection area based on the point cloud data collected by the lidar. The point cloud features include spatial location information and shape information. The radar feature module is used to obtain the radar features of the detected target based on the speed and distance information of the detected target collected by the millimeter-wave radar. The radar features include radar position information, attitude information and speed information. The environmental feature analysis module is used to generate confidence scores for each mode under the current environment based on environmental features, and to determine the primary mode and secondary mode based on the confidence scores; The feature association module is used to determine the location information of the detected target by using the image coordinate information of image features, the spatial location information of point cloud features, and the radar location information of radar data, respectively. A spatial coordinate system is established with the location information of the main mode as the origin, and the location information of the secondary mode is mapped into the spatial coordinate system. The environmental feature analysis module includes an environmental parameter analysis module and a confidence score module; The environmental parameter analysis module is used to generate normalized environmental impact factors based on environmental information. These environmental impact factors include visibility factors, illumination factors, weather factors, and target motion factors. The confidence score module is used to generate confidence scores for each modality based on environmental impact factors. in This represents the confidence score for mode m. Visibility factor Light factor, As a weather factor, For the target motion factor, , , , These are the modal sensitivity coefficients of mode m with respect to visibility factor, illumination factor, weather factor, and target motion factor, respectively. + + + = 1; The fusion detection module also includes a master-slave modality repair module, a missing data repair module, and a spatiotemporal repair module; The master-slave modality repair module is used to correct the conflict features of the secondary modality when there is a conflict between different modalities of the detected target. If the confidence score difference between the master modality and this module is higher than a preset first threshold, the detection result of the master modality shall be used as the standard. When the score difference between the master modality and the secondary modality is lower than the preset first threshold but higher than the preset second threshold, the recognition result consistent with the majority modality shall be selected as the final labeling basis. The missing feature repair module is used to complete the feature by using features from other modalities when there are missing modal features. The spatiotemporal repair module is used to perform spatiotemporal smoothing and correction on the conflict features of the current frame when the detected target has continuous frame motion trajectories and historical features.

2. The semi-automatic data annotation system for intelligent transportation according to claim 1, characterized in that: The fusion detection module includes a location conflict detection module, an attribute conflict detection module, and an existence conflict detection module. The position conflict detection module is used to calculate the position deviation of the same detection target in different modes in a spatial coordinate system. When the deviation exceeds a preset distance threshold, it is marked as a position conflict. The attribute conflict detection module is used to compare the attribute recognition results of different modalities for the same target, and to mark the inconsistencies as attribute conflicts. The existence detection module is used to detect whether there is a behavior where one modality recognizes the target while another modality does not. If such a behavior exists, it is marked as an existence conflict.

3. The semi-automatic data annotation system for intelligent transportation according to claim 1, characterized in that: The server also includes a spatiotemporal synchronization module, which is used to perform timestamp calibration and spatial coordinate system alignment on the multimodal data collected by the information sensing device and the environmental information collected by the environmental sensing device before the feature extraction module performs feature extraction, so as to unify the spatiotemporal reference of each data.

Images