A semi-automatic traffic light annotation method based on mutual information of image point cloud

Abstract

The present invention discloses a semi-automatic traffic light annotation method based on image point cloud mutual information, comprising the following steps: (1) collecting a traffic light dataset, and performing data annotation and enhancement after preprocessing; (2) using the data from step (1) to train a target detection model, wherein the target detection model adopts an improved YOLOv4 model with anchor box optimization and attention mechanism; using the trained target detection model to search road video images and find images containing road traffic lights; (3) through the joint calibration of laser radar and camera, back-projecting the pixels of the 2D image into 3D point cloud data of the laser radar coordinate system; (4) manually verifying and correcting the automatically annotated traffic light positions. The present invention is helpful to improve the efficiency and accuracy of high-precision map production and provide more accurate traffic light positioning information for autonomous driving systems.

Description

Technical Field

[0001] The present invention belongs to the technical field of vehicle automatic driving, and in particular relates to a semi-automatic marking method for traffic lights based on image point cloud mutual information. Background Art

[0002] With the continuous advancement of technology, autonomous driving technology is becoming a new trend in the automotive industry. Within this field, the importance of automatic traffic light recognition technology is self-evident. Accurately identifying traffic lights is crucial for autonomous vehicles, helping to ensure road safety and improve driving efficiency.

[0003] High-precision point cloud maps can significantly reduce traffic risks and improve driving safety. By leveraging traffic light location information from high-precision maps, autonomous vehicles can more accurately identify and process traffic signals, providing strong support for autonomous vehicles. However, extracting traffic light information presents several challenges.

[0004] Currently, the main methods for extracting traffic lights include manual and traditional methods, but these methods have certain limitations and shortcomings. Traditional methods are not adaptable to complex scenarios and have low recognition accuracy, which poses a significant challenge to the practical application of autonomous vehicles. While manual methods have a certain degree of accuracy, they are costly and inefficient, making them unsuitable for large-scale deployment.

[0005] Traditional methods using YOLOv4 to detect traffic lights only yield two-dimensional target detection information, with limited spatial information, and fail to capture the three-dimensional spatial position of traffic lights. Furthermore, using YOLOv4 alone can be affected by factors such as lighting and occlusion, resulting in reduced detection performance.

[0006] Chinese patent document publication number CN112733640A discloses a traffic light positioning and extraction method and system based on a high-precision point cloud map. The method trains a target detection network by producing road traffic light training sample data, and uses the target detection network to find pictures containing road traffic lights. Then, based on the correspondence between the picture and the point cloud, the approximate area of the point cloud set corresponding to the picture containing the road traffic light is found in the point cloud high-precision map, and the point cloud is projected two-dimensionally. The target is segmented on the two-dimensional projection view of the point cloud through a target semantic segmentation network, and the target area of the traffic light in the two-dimensional projection view of the point cloud is determined, and then the location information of the traffic light is obtained based on the point cloud coordinates.

[0007] This method requires two models: target detection and semantic segmentation for detection. The model structure is relatively complex, the detection speed is slow, and the accuracy needs to be further improved. Summary of the Invention

[0008] The present invention provides a semi-automatic traffic light annotation method based on the mutual information of image point clouds, which helps to improve the efficiency and accuracy of high-precision map production and provides more accurate traffic light positioning information for autonomous driving systems.

[0009] A semi-automatic traffic light annotation method based on image point cloud mutual information includes the following steps:

[0010] (1) Collect traffic light data sets, and perform data annotation and enhancement after preprocessing;

[0011] (2) using the data from step (1) to train a target detection model, wherein the target detection model adopts an improved YOLOv4 model with anchor box optimization and attention mechanism;

[0012] Use the trained object detection model to search road video images and find images containing road traffic lights;

[0013] (3) Through the joint calibration of the lidar and camera, the pixels of the 2D image are back-projected into the 3D point cloud data of the lidar coordinate system;

[0014] (4) Manually verify and correct the automatically marked traffic light positions.

[0015] In step (1), the preprocessing is specifically as follows: converting the image to a uniform size and format; using image processing tools to compress, crop, and flip the image to improve data diversity; and grayscale processing or normalization of the image to reduce the amount of calculation and improve model training efficiency.

[0016] Data annotation specifically includes: marking the location, type (red light, yellow light, green light, etc.) and status (on or off) of each traffic light; data enhancement specifically includes: increasing data diversity through image transformation technology.

[0017] Finally, the sorted data set is divided into 70% training set, 15% validation set, and 15% test set.

[0018] In step (2), the YOLOv4 model includes a backbone network, a neck network, and a head network;

[0019] The backbone network uses CSPDarknet53 for feature extraction from input images, while the neck network utilizes the SPP module and PANet network to broaden the receptive field and achieve more effective feature fusion. YOLOv4 introduces PANet (Path Aggregation Network) and BiFPN (Bi-directional Feature Pyramid Network) for feature fusion. These two structures improve the quality of the feature pyramid and achieve more accurate detection.

[0020] The head structure uses three-scale detection heads, responsible for detecting large, medium, and small targets respectively. Each detection head outputs a feature map containing the target's location, category, and confidence information; and introduces the SPP (Spatial Pyramid Pooling) module and SAM (Spatial Attention Module) module to enhance the performance of the detection head.

[0021] The output of the detection head is decoded to obtain the predicted object location, category, and confidence. YOLOv4 uses the CIoU (Complete Intersection over Union) loss to improve the accuracy of bounding box positioning. To remove duplicate detection results, YOLOv4 uses the non-maximum suppression (NMS) algorithm to filter the final object detection results.

[0022] The specific optimization of adding anchor boxes to the YOLOv4 model is as follows:

[0023] Cluster all the annotation boxes to determine K cluster centers, generate anchor boxes based on the cluster centers, and then train based on the anchor boxes.

[0024] The specific steps of adding attention mechanism to the YOLOv4 model are:

[0025] The efficient channel attention (ECA) module is introduced to perform weight training on the channel dimension of the feature layer extracted by the backbone network, making the model pay more attention to useful information.

[0026] Among them, the ECA module replaces the two fully connected operations of the SE module with 1D convolution. In the ECA module, the 1D convolution kernel size k and the number of channels C show a nonlinear proportional relationship, that is,

[0027] C=φ(k)=2 (γ×k-b)

[0028] After determining the channel dimension C, the convolution kernel size k is:

[0029]

[0030] Where γ and b are adjustment parameters; odd represents the nearest odd number.

[0031] The specific process of step (3) is as follows:

[0032] (3-1) Obtain the camera's intrinsic parameter matrix and distortion parameters, and use the camera's intrinsic parameter matrix and distortion parameters to correct the input 2D image;

[0033] (3-2) Back-project the corrected 2D image pixels to 3D points in the camera coordinate system, and then use the relative transformation matrix T between the lidar and the camera to transform the 3D points in the camera coordinate system to the lidar coordinate system;

[0034] (3-3) For a given 2D pixel point, find its corresponding 3D coordinate in the lidar coordinate system based on the established correspondence.

[0035] In step (3-1), the internal parameter matrix M includes the focal length (f x , f y ) and the optical center coordinates (c x , c y ), the distortion parameter D includes radial distortion parameters (k1, k2, k3) and tangential distortion parameters (p1, p2); the original pixel coordinates are (x, y), r is the distance from the pixel to the optical center, and the corrected pixel coordinates are (x′, y′), and the formula is as follows:

[0036] x′=x+(xc x )(k1r 2 +k2r 4 +k3r 6 )+2p1xy+p2(r 2 +2x 2 )

[0037] y′=y+(yc y )(k1r 2 +k2r 4 +k3r 6 )+2p2xy+p1(r 2 +2y 2 )

[0038] In step (3-2), the back projection formula is as follows:

[0039]

[0040] Among them, (u, v) is the 2D pixel coordinate, (X C , Y C , Z C ) is the 3D point back-projected from the 2D pixel point to the camera coordinate system;

[0041] The relative transformation matrix T includes the rotation matrix R and the translation vector t. The formula for converting the 3D point in the camera coordinate system to the lidar coordinate system is as follows:

[0042]

[0043] In the formula, (X L ,Y L ,Z L ) are the 3D point coordinates in the lidar coordinate system.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] 1. This invention helps improve the efficiency and accuracy of traffic light annotation in high-precision maps, providing more accurate traffic light positioning information for autonomous driving systems. It has high value in practical applications and provides strong support for the development of autonomous driving technology.

[0046] 2. The present invention is low-cost and highly accurate. Traffic light location information is pre-marked for the marking personnel, who only need to verify and check, thus saving manpower and material resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a flow chart of a semi-automatic traffic light annotation method based on image point cloud mutual information of the present invention;

[0048] Figure 2 This is an example diagram of the results of traffic light detection in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The present invention will be described in further detail below with reference to the accompanying drawings and examples. It should be noted that the following examples are intended to facilitate understanding of the present invention and do not have any limiting effect on the present invention.

[0050] like Figure 1 As shown in FIG, a semi-automatic traffic light annotation method based on the mutual information of image point clouds includes the following steps:

[0051] S01, create a traffic light dataset, perform preprocessing, data annotation and enhancement.

[0052] (1) Data collection

[0053] Traffic light images were collected from a variety of sources, including online image libraries, open source datasets, camera images, drone images, etc., to ensure the diversity of the dataset, including various types of traffic lights, images in different weather conditions, and images from different time periods and angles.

[0054] (2) Data preprocessing

[0055] Convert the collected images to a uniform size and format, and use image processing tools to perform operations such as image compression, cropping, and flipping to increase data diversity. Perform grayscale processing or normalization on the images to reduce computational effort and improve model training efficiency.

[0056] (3) Labeled data

[0057] Annotate the collected images to determine the location, type (red, yellow, green, etc.) and status (on, off) of each traffic light, and use annotation tools (such as Labelbox, RectLabel, etc.) for auxiliary annotation.

[0058] (4) Data enhancement

[0059] Increase data diversity and improve the generalization ability of the model through image transformation technology (rotation, translation, scaling, flipping, lighting changes, etc.).

[0060] (5) Divide the data set

[0061] The organized dataset is divided into a training set (70%), a validation set (15%), and a test set (15%). The training set is used to train the model, the validation set is used to adjust hyperparameters and evaluate model performance, and the test set is used to ultimately evaluate the model's generalization ability.

[0062] S02: Use the traffic light data to train an improved YOLOv4 model, and use the trained model to search road video images to find pictures containing road traffic lights.

[0063] The structure of the YOLOv4 network can be divided into three parts: backbone network (Backbone), neck network (Neck) and head (Head).

[0064] In this paper, the backbone network adopts the CSPDarknet53 structure, which significantly improves the performance of the model through effective feature extraction and feature fusion. CSPDarknet53 has higher computational efficiency and achieves real-time performance while maintaining accuracy.

[0065] The neck network uses the SPP module and the PANet network, which can widen the receptive field and achieve more effective feature fusion. The SPP module enhances the model's perception of objects of different scales through multi-scale pooling operations, while the PANet network achieves effective fusion between high-level and low-level features, thereby improving model performance.

[0066] The head structure is the same as YOLOv3, using three-scale output to detect objects of different sizes. This enables YOLOv4 to achieve accurate detection of multi-scale objects while maintaining real-time performance.

[0067] In terms of performance, YOLOv4's inference speed is twice that of EfficientDet, demonstrating excellent real-time performance. Compared to YOLOv3, YOLOv4 improves accuracy (AP) and frame rate (FPS) by 10% and 12%, respectively.

[0068] This paper uses the K-means++ algorithm to optimize anchor boxes for target samples. The YOLOv4 network presets nine target boxes, totaling three categories, suitable for predicting bounding boxes for targets using YOLO detection heads at three different scales. These anchor boxes are based on bounding box clustering from the PASCAL VOC dataset and cover objects of multiple scales. While the original preset anchor boxes are well-suited for object detection tasks in common scenarios, they perform poorly for objects of extremely large sizes, which can affect model performance.

[0069] This paper uses the K-means++ clustering algorithm to cluster the anchor boxes of targets in the VEDAI dataset. Compared with the traditional K-means, K-means++ optimizes the initial point selection, significantly improves the error, obtains an anchor box size more suitable for small target datasets, and improves detection accuracy. The algorithm first randomly selects the initial cluster center, calculates the distance between the sample and the cluster center and the probability of selecting it as the next cluster center, that is,

[0070]

[0071] The next cluster center is then selected using the roulette wheel method, and the distance and probability calculations are repeated until K target boxes are obtained. Finally, the distance from the sample to the cluster center is repeatedly calculated, the sample is divided into the class with the smallest distance, and the cluster center is updated until the size of the anchor box no longer changes.

[0072] To improve the bounding box detection rate, we perform bounding box clustering analysis based on the object size in the dataset. This method can better capture the scale changes of the object and generate more appropriate anchor boxes for objects of different sizes.

[0073] Bounding box clustering analysis helps to filter out the appropriate bounding box when calculating IOU, thereby improving model performance. By using anchor boxes optimized for specific datasets, the YOLOv4 network can achieve higher object detection accuracy in various scenarios.

[0074] In order to further improve the performance of YOLOv4 object detection, the present invention introduces an efficient channel attention (ECA) module into its structure.

[0075] This module focuses on weighting the channel dimensions of the feature layer extracted by the backbone network. By introducing the ECA module, the YOLOv4 model can focus more on useful information, thereby improving performance. While maintaining low computational complexity and parameter count, the ECA module significantly enhances the network's expressive power, enabling the model to better capture and focus on task-relevant information.

[0076] The ECA module is an improved version of the Squeeze-and-Excitation (SE) module. The SE module strengthens feature representation by capturing inter-channel dependencies, and the ECA module further optimizes this.

[0077] The ECA module believes that capturing the dependencies of all channels is inefficient and unnecessary. Therefore, the ECA module reduces the computational complexity while maintaining good expressive power by selectively focusing on the local dependencies between channels.

[0078] The ECA module replaces the two fully connected operations of the SE module with 1D convolution. This improvement enables the ECA module to reduce computational complexity and the number of parameters while still maintaining good cross-channel information acquisition capabilities.

[0079] In the ECA module, the 1D convolution kernel size k has an important impact on the coverage of cross-channel interactions. Therefore, choosing an appropriate k value becomes very critical. The appropriate k value can effectively balance the relationship between cross-channel information acquisition capabilities and computational complexity. At the same time, in the ECA module, when the number of channels C is large, the model requires stronger long-term interactions; and when the number of channels C is small, the model requires stronger short-term interactions. This nonlinear proportional relationship helps to achieve more effective information interaction under different channel numbers. The 1D convolution kernel size k and the number of channels C show a nonlinear proportional relationship, that is,

[0080] C=φ(k)=2 (γ×k-b)

[0081] After determining the channel dimension C, the convolution kernel size k is:

[0082]

[0083] γ, b are adjustment parameters; odd represents the nearest odd number.

[0084] S03, through the joint calibration of the laser radar and the camera, the actual pixel position of the corresponding target detection is obtained.

[0085] (1) Calculate camera intrinsic parameters and distortion parameters

[0086] Use the calibration algorithm to calculate the camera's intrinsic parameter matrix M and distortion parameters D. The intrinsic parameter matrix K includes the focal length (f x , f y ) and the optical center coordinates (c x , c y ), the distortion parameter D includes radial distortion parameters (k1, k2, k3) and tangential distortion parameters (p1, p2), as shown below

[0087] D=(k1,k2,k3,p1,p2),

[0088] (2) Image correction

[0089] The captured image is corrected using the intrinsic parameter matrix M and the distortion parameter D. For each input 2D image, the above formula is applied to each pixel in the image. This will eliminate radial and tangential distortion in the image, making the image closer to the ideal undistorted state. The corrected pixel coordinates are (x', y'), the original pixel coordinates are (x, y), and r is the distance from the pixel to the optical center. The formula is as follows:

[0090] x′=x+(xc x )(k1r 2 +k2r 4 +k3r 6 )+2p1xy+p2(r 2 +2x 2 )

[0091] y′=y+(yc y )(k1r 2 +k2r 4 +k3r 6 )+2p2xy+p1(r 2 +2y 2 )

[0092] (3) Back-projection from 2D pixel points to 3D points in the camera coordinate system

[0093] To find the corresponding 3D point cloud data (X L , Y L , Z L ), we need to first back-project the 2D pixel point to the 3D point in the camera coordinate system (X C , Y C , Z C ), and then transform the 3D point in the camera coordinate system to the lidar coordinate system.

[0094]

[0095] The back projection formula is as follows:

[0096]

[0097] We can find X C and Y C :

[0098]

[0099] (4) Convert the 3D point in the camera coordinate system to the lidar coordinate system

[0100] The relative transformation matrix T (including the rotation matrix R and the translation vector t) calculated by joint calibration is used to transform the 3D point (X C ,Y C ,Z C ) is converted to a 3D point (X L ,Y L ,Z L ):

[0101]

[0102] After completing the above steps, you can find the corresponding point in the 3D lidar coordinate system from the 2D pixel point.

[0103] S04, manually verifying and correcting the automatically marked traffic light positions.

[0104] like Figure 2 As shown, the automatic annotation of traffic lights during the mapping process is more accurate than simply manually annotating their locations and heights. Traffic light annotation relies on visual detection and radar positioning, using LiDAR to register depth information in the image to determine the actual pixel locations of the corresponding target detections. Because joint LiDAR and camera calibration cannot guarantee highly accurate registration results for every point, deviations in detection and positioning are inevitable. When traffic lights are far away, positioning errors are large, and even the probability of depth registration errors is high.

[0105] Therefore, this work focuses on determining which traffic lights are correctly located, while filtering out those with incorrect or significant deviations. Semi-automated traffic light labeling requires post-processing by human labelers. Testing has shown that while the probability of missed detections is low, false detections do occur, and the error for properly located traffic lights is within 1 meter.

[0106] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A semi-automatic traffic light annotation method based on image point cloud mutual information, characterized in that: The steps include: (1) Collect traffic light data sets, and perform data annotation and enhancement after preprocessing; (2) using the data from step (1) to train a target detection model, wherein the target detection model adopts an improved YOLOv4 model with anchor box optimization and attention mechanism; Use the trained object detection model to search road video images and find images containing road traffic lights; The YOLOv4 model consists of a backbone network, a neck network, and a head network. The backbone network uses CSPDarknet53 for feature extraction from the input image. The neck network uses the SPP module and PANet network to widen the receptive field and achieve more effective feature fusion. The head structure uses three-scale detection heads, responsible for detecting large, medium, and small objects respectively. Each detection head outputs a feature map containing the object's location, category, and confidence information. The SPP module and SAM module are introduced to enhance the performance of the detection head. The anchor box optimization is added to the YOLOv4 model as follows: cluster all the annotation boxes to determine K cluster centers, generate anchor boxes based on the cluster centers, and then train based on the anchor boxes; The attention mechanism is added to the YOLOv4 model by introducing the efficient channel attention (ECA) module to perform weight training on the channel dimension of the feature layer extracted by the backbone network. Among them, the ECA module replaces the two fully connected operations of the SE module with 1D convolution. In the ECA module, the 1D convolution kernel size k and the number of channels C show a nonlinear proportional relationship, that is, C=φ(k)=2 (γ×k-b) After determining the channel dimension C, the convolution kernel size k is: Where γ and b are adjustment parameters; odd represents the nearest odd number; (3) Through the joint calibration of the lidar and camera, the pixels of the 2D image are back-projected into the 3D point cloud data of the lidar coordinate system; (4) Manually verify and correct the automatically marked traffic light positions.

2. The semi-automatic traffic light labeling method based on image point cloud mutual information according to claim 1 is characterized in that: In step (1), the preprocessing is specifically as follows: converting the image to a uniform size and format; using image processing tools to compress, crop, and flip the image to improve data diversity; and grayscale processing or normalization of the image to reduce the amount of calculation and improve model training efficiency.

3. The semi-automatic traffic light labeling method based on image point cloud mutual information according to claim 1 is characterized in that: In step (1), data annotation specifically includes: marking the location, type and status of each traffic light; data enhancement specifically includes: increasing the diversity of data through image transformation technology.

4. The semi-automatic traffic light labeling method based on image point cloud mutual information according to claim 1 is characterized in that: The specific process of step (3) is as follows: (3-1) Obtain the camera's intrinsic parameter matrix and distortion parameters, and use the camera's intrinsic parameter matrix and distortion parameters to correct the input 2D image; (3-2) Back-project the corrected 2D image pixels to 3D points in the camera coordinate system, and then use the relative transformation matrix T between the lidar and the camera to transform the 3D points in the camera coordinate system to the lidar coordinate system; (3-3) For a given 2D pixel point, find its corresponding 3D coordinate in the lidar coordinate system based on the established correspondence.

5. The semi-automatic traffic light labeling method based on image point cloud mutual information according to claim 4 is characterized in that: In step (3-1), the internal parameter matrix M includes the focal length (f x ,f y ) and the optical center coordinates (c x ,c y ), the distortion parameter D includes radial distortion parameters (k1, k2, k3) and tangential distortion parameters (p1, p2); the original pixel coordinates are (x, y), r is the distance from the pixel to the optical center, and the corrected pixel coordinates are (x', y'). The formula is as follows:

6. The semi-automatic traffic light labeling method based on image point cloud mutual information according to claim 5 is characterized in that: In step (3-2), the back projection formula is as follows: Among them, (u, v) is the 2D pixel coordinate, (X C ,Y C ,Z C ) is the 3D point back-projected from the 2D pixel point to the camera coordinate system; The relative transformation matrix T includes the rotation matrix R and the translation vector t. The formula for converting the 3D point in the camera coordinate system to the lidar coordinate system is as follows: In the formula, (X L ,Y L ,Z L ) are the 3D point coordinates in the lidar coordinate system.

Images