Traffic light semi-automatic labeling method

Abstract

The application discloses a traffic light semi-automatic labeling method, and relates to the technical field of automatic driving. The traffic light semi-automatic labeling method mainly comprises the following steps: according to an input image, a detection model is used for traffic light detection to obtain a traffic light image and a two-dimensional external frame thereof; according to the two-dimensional external frame and camera pose parameters, traffic light instances in a high-definition map are projected to an image plane to obtain a projection external frame and three-dimensional geometric attributes of the traffic light; the two-dimensional external frame, the projection external frame and the three-dimensional geometric attributes are fused and matched to obtain a final position and size of the traffic light; and a traffic light color state is obtained according to the traffic light image, a direction category and a distance of the traffic light. The traffic light semi-automatic labeling method provided by the application can generate more complete semantic information, reduce the burden of manual supplementary labeling and adapt to more complex traffic light structures.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and more specifically, to a semi-automatic traffic light marking method. Background Technology

[0002] In autonomous driving perception systems, traffic light information, as a crucial component of road traffic control, plays a vital role in vehicle longitudinal control, intersection passage decisions, and overall safety. Autonomous vehicles typically rely on forward-facing cameras and deep learning models to detect the position, state, and geometric attributes of traffic lights. In typical urban roads, complex intersections, occluded scenarios, and situations with multiple traffic light clusters, accurately identifying the state of traffic lights directly impacts the planning module's decisions regarding stopping, proceeding, or slowing down. Therefore, high-quality traffic light datasets are fundamental for training autonomous driving perception models, and their annotation accuracy and consistency significantly affect the model's reliability and generalization ability in long-tail scenarios.

[0003] With the development of end-to-end autonomous driving algorithms in recent years, integrated perception and planning models increasingly rely on more refined data structures. For example, in intersection decision-making tasks, models need to explicitly understand the binding relationship between traffic lights and corresponding lanes to support more accurate right-of-way judgments and trajectory generation. This binding relationship requires providing the location of the traffic lights, their status information, and the lanes or turning directions they control during the annotation process. However, existing semi-automated processes typically only output detection boxes or traffic light status inference results, failing to automatically deduce the association between traffic lights and lane lines. Manual annotation based on scene semantics and road structure is still required, further increasing annotation time.

[0004] In addition to the binding relationship between traffic lights and lane lines, the semantic categories of traffic lights also need to be subdivided. Common traffic lights in real-world roads include positive lights, side lights, and backlights, each differing in shape, orientation, image appearance, and impact on vehicle driving decisions. For example, while backlights are visible in images, they have no constraint effect on the vehicle; side lights may belong to lateral traffic unrelated to the vehicle; only positive lights are used for actual decision-making. Most existing semi-automated labeling systems can only provide basic detection results and cannot automatically distinguish the orientation category of the lights, requiring manual identification and classification of each traffic light, further increasing labor costs.

[0005] Currently, common semi-automated traffic light annotation methods in the industry mainly include pre-annotation based on detection models, cross-frame association based on tracking algorithms, and rule-based light status judgment auxiliary modules. For example, candidate traffic light location boxes are generated using object detection models, and then cross-frame matching is performed in the video sequence using multi-object tracking algorithms to reduce repetitive manual work; some processes also attempt to use brightness thresholds, color distribution features, or geometric rules to assist in judging the light status. These methods improve annotation efficiency to some extent, but still have several shortcomings: (1) The pre-labeling results are unstable in complex scenarios, such as backlight, nighttime glare, rain and snow, and densely populated areas with multiple light heads. There are still many false detections and missed detections, and the cost of manual correction is still high. (2) Cross-frame tracking algorithms are prone to drift or incorrect association under conditions such as occlusion, violent movement, rapid changes in target size, and viewpoint switching, resulting in insufficient cross-frame consistency. Manual correction of cross-frame relationships is still required. (3) Rule-based lamp status assisted judgment is sensitive to scene changes and is difficult to adapt to different regions, different lamp shapes and different imaging conditions, which limits the degree of automation; (4) Existing methods cannot automatically generate light-lane binding relationships, and manual judgment is required to determine the controlled lanes corresponding to traffic lights; (5) Existing methods are difficult to automatically distinguish between front lights, side lights and back lights, and still require manual classification of each light head.

[0006] Therefore, the automation level and scenario adaptability of existing semi-automated annotation processes are still limited, and the overall efficiency still relies heavily on human participation. It is difficult to meet the needs of autonomous driving data production for large-scale, high-precision and structured output. There is an urgent need for a semi-automated annotation technology that can generate more complete semantic information, reduce the burden of manual annotation, and adapt to complex traffic light structures. Summary of the Invention

[0007] The purpose of this invention is to provide a semi-automatic traffic light labeling method that can generate more complete semantic information, reduce the burden of manual labeling, and adapt to more complex traffic light structures.

[0008] This invention provides a semi-automatic traffic light marking method, comprising the following steps: S1: Based on the input image, use the detection model to detect traffic lights and obtain the traffic light image and its two-dimensional bounding box; S2: Obtain camera pose parameters. Based on the two-dimensional bounding box and the camera pose parameters, project the traffic light instance in the high-precision map onto the image plane to obtain the projected bounding box and three-dimensional geometric attributes of the traffic light. S3: The two-dimensional bounding box, the projected bounding box, and the three-dimensional geometric attributes are fused and matched to obtain the final position and size of the traffic light; S4: Based on the traffic light image, the direction category of the traffic light, and the distance, obtain the traffic light color status.

[0009] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described semi-automatic traffic light marking method.

[0010] The semi-automatic traffic light marking method provided by this invention has the following beneficial effects: This invention transforms the traditional problems of false detection and missed detection into the more easily judged problem of "whether there is occlusion" by fusing detection results with high-precision map projection results. Furthermore, it improves the accuracy of results through 2D-3D fusion and multi-camera fusion modules, thereby significantly reducing the cost of manual correction. Compared with tracking algorithms that rely on cross-frame association, this invention utilizes the stable priors provided by high-precision maps, greatly improving cross-frame consistency, requiring almost no manual correction of cross-frame relationships. This invention employs a color recognition method based on a classification model, achieving more robust color determination results under different lighting, exposure, and weather conditions. This invention utilizes the binding relationships between traffic lights and lanes in the high-precision map for annotation, eliminating the need for manual establishment of light-lane associations. This invention utilizes the three-dimensional orientation information of traffic lights provided by the high-precision map, automatically distinguishing between forward-facing, lateral, and backward-facing lights, avoiding manual annotation of orientation.

[0011] In summary, this invention utilizes the joint correction of detection results and high-precision map projection results to achieve precise positioning, attribute recognition, and color labeling of traffic lights. While ensuring labeling consistency, it reduces manual intervention, improves data production efficiency, and outputs structured traffic light semantic information. Attached Figure Description

[0012] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a flowchart of the semi-automatic traffic light marking method provided by the present invention; Figure 2 This is a flowchart of the semi-automated traffic light labeling process provided by the present invention; Figure 3 This is a closed-loop diagram of the color extraction module data provided by the present invention. Detailed Implementation

[0013] To provide a clearer understanding of the technical features, objectives, and effects of the present invention, specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0014] Figure 1 A schematic diagram of the semi-automatic traffic light marking method of this embodiment is shown. In this embodiment, the semi-automatic traffic light marking method includes the following steps: S1: Based on the input image, use the detection model to detect traffic lights and obtain the traffic light image and its two-dimensional bounding box.

[0015] In one exemplary embodiment, the detection model is YOLOX or Co-DETR.

[0016] S2: Obtain camera pose parameters. Based on the two-dimensional bounding box and the camera pose parameters, project the traffic light instance in the high-precision map onto the image plane to obtain the projected bounding box and three-dimensional geometric attributes of the traffic light.

[0017] In one exemplary embodiment, the three-dimensional geometric properties of the traffic light include the traffic light's orientation category and the distance between the traffic light and the vehicle.

[0018] In one exemplary embodiment, the orientation categories include front, side, and back.

[0019] S3: The two-dimensional bounding box, the projected bounding box, and the three-dimensional geometric attributes are fused and matched to obtain the final position and size of the traffic light.

[0020] In one exemplary embodiment, the fusion matching process includes: Calculate the intersection-union ratio and cost matrix of the two-dimensional bounding box and the projected bounding box; According to the cost matrix, the Hungarian algorithm matches the two-dimensional bounding box with the projected bounding box. If the two-dimensional bounding box is not matched, it is considered a false detection of the model. If the projected bounding box is not matched, it can be determined that the traffic light is blocked. The final size of the traffic light is determined by the dimensions of the matched projection bounding box; the final position of the traffic light is determined by the center coordinates of the matched two-dimensional bounding box.

[0021] In one exemplary embodiment, the formula for calculating the crossover-union ratio is:

[0022] in, Indicates the first A two-dimensional outer frame; Indicates the first One projection outer frame; Represents a two-dimensional bounding box With projection frame The intersection region; Represents a two-dimensional bounding box With projection frame The union region; This indicates that the area of ​​the specified region will be calculated. This represents the intersection-union ratio, which ranges from [0,1].

[0023] In one exemplary embodiment, the cost matrix is ​​calculated using the following formula:

[0024] in, Let $\mathbf{i}$ represent the matching cost between the $i$-th 2D bounding box and the $j$-th projected bounding box.

[0025] S4: Based on the traffic light image, the direction category of the traffic light, and the distance, obtain the traffic light color status.

[0026] In one exemplary embodiment, step S4 specifically includes: Traffic light color data is obtained based on the HSV features of the traffic light image, the orientation category of the traffic light, and the distance. The color classification model is trained using the color data to obtain a trained color classification model; The trained color classification model is used to determine the color of the traffic light image to obtain the color status of the traffic light.

[0027] In one exemplary embodiment, the color classification model is a convolutional neural network model based on a ResNet backbone.

[0028] In one exemplary embodiment, step S4 further includes: manually correcting the color state of the traffic light.

[0029] In one exemplary embodiment, the method further includes: performing complementary fusion of the detection results from the telephoto camera and the wide-angle camera, wherein when the two cameras make inconsistent color determinations for the same traffic light, the color result from the telephoto camera is taken as the final color state.

[0030] In one exemplary embodiment, the method further includes: smoothing and correcting abrupt color changes and abnormal jumps in the color state of traffic lights in a single frame within a continuous series of frames.

[0031] In some embodiments, the above-described semi-automatic traffic light marking method can also be implemented in the following ways.

[0032] like Figure 2 The diagram shows a semi-automatic traffic light labeling process; in this embodiment, the semi-automatic traffic light labeling method includes the following steps: 1. Using a detection model, traffic light detection is performed on the input image to obtain the two-dimensional bounding box information of the traffic lights; 2. Using the projection module, based on the camera pose, traffic light instances in the high-precision map are projected onto the image plane, and their three-dimensional geometric properties are obtained; 3. Using the 2D-3D fusion module, the detection results and projection results are fused and matched in two dimensions and three dimensions, and the final position and size of the traffic lights are jointly corrected. 4. The color extraction module determines the color status of traffic lights based on image information; 5. Through a multi-camera fusion module, results from different cameras (such as telephoto and wide-angle) are complementaryly fused; 6. The timing fusion module is used to smooth the single-frame results and correct abnormal jumps using timing information.

[0033] As an exemplary embodiment, since this invention utilizes high-precision map projection to provide stable and accurate prior locations for traffic lights, the accuracy requirements for the traffic light detection model are relatively low. Existing general object detection networks can be directly used without additional training for this scenario. For example, detection models such as YOLOX and Co-DETR can be used to obtain the two-dimensional bounding boxes of traffic lights from the image. Furthermore, the confidence threshold can be appropriately reduced; this invention retains detection results with a confidence level greater than 0.3.

[0034] As an exemplary embodiment, the 3D positions of traffic lights in a high-precision map are projected onto the image plane using camera pose parameters to obtain a projection bounding box. Based on the angle θ between the camera's optical axis and the traffic light's normal vector, the orientation of the traffic light can be inferred (θ>= 135° for forward, θ<= 45° for sideways, or θ<= 45° for backward). Based on the spatial relationship between the camera position and the 3D coordinates of the traffic light, the distance between the traffic light and the vehicle can also be calculated.

[0035] As an exemplary embodiment, a cost matrix is ​​constructed using the Interchange of Units (IoU), and the Hungarian algorithm is employed to match the detection boxes and projected boxes; an IoU > 0 indicates a match. A detection box not matching is considered a false detection; a projected box not matching indicates that the traffic light is obstructed. Considering that the size of the detection box is susceptible to fluctuations due to model stability, this invention uses the size of the projected box in the final annotation; however, the position of the projected box may be affected by positioning errors, so the final position uses the center coordinates of the detection box, ensuring both stability and accuracy in the overall result.

[0036] As an exemplary embodiment, this invention initially employs a rule-based approach to extract traffic light colors. The rules are determined based on image HSV features, traffic light orientation, and distance conditions. The H threshold is (0, 10)∪(160, 180) for red, (18, 35) for yellow, and (40, 100) for green. Since traffic light brightness is distance-dependent, the V threshold is 64 within 20 meters and 100 beyond 20 meters. The S threshold remains constant at 64. The backlight color is fixed as black. After accumulating high-quality color data with manual corrections, a color classification model, such as a convolutional neural network model based on a ResNet backbone, is further trained, thereby significantly improving the robustness of color recognition. Figure 3 The diagram shown is a closed-loop diagram of the color extraction module data.

[0037] As an exemplary embodiment, the detection or projection results from a telephoto camera and a wide-angle camera are fused to reduce missed detections and supplement missing information. When the two cameras disagree on the color of the same traffic light, the present invention prioritizes the color result from the telephoto camera due to its clearer imaging capability. When the same 3D traffic light is detected by only one of the two cameras, the undetected image is considered a missed detection. The projection frame position is applied to the undetected image, and the detected color result is used.

[0038] As an exemplary embodiment, temporal consistency verification is performed on the detection results of consecutive frames. Based on the characteristic that the color state of traffic lights has temporal continuity in the real physical environment, unreasonable color abrupt changes, such as repeated jumps between red and green in a short period of time, are smoothed and repaired, thereby improving the temporal stability of the final annotation results.

[0039] This embodiment provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described semi-automatic traffic light marking method.

[0040] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A semi-automatic method for marking traffic lights, characterized in that, Includes the following steps: S1: Based on the input image, use the detection model to detect traffic lights and obtain the traffic light image and its two-dimensional bounding box; S2: Obtain camera pose parameters. Based on the two-dimensional bounding box and the camera pose parameters, project the traffic light instance in the high-precision map onto the image plane to obtain the projected bounding box and three-dimensional geometric attributes of the traffic light. S3: The two-dimensional bounding box, the projected bounding box, and the three-dimensional geometric attributes are fused and matched to obtain the final position and size of the traffic light; S4: Based on the traffic light image, the direction category of the traffic light, and the distance, obtain the traffic light color status.

2. The semi-automatic traffic light marking method according to claim 1, characterized in that, Also includes: The detection results from telephoto and wide-angle cameras are complemented and fused. When the two cameras make inconsistent color judgments for the same traffic light, the color result from the telephoto camera is taken as the final color state.

3. The semi-automatic traffic light marking method according to claim 1, characterized in that, Also includes: Smoothing and abnormal jump correction are performed on the color abrupt changes of traffic light color status in a single frame within a continuous frame.

4. The semi-automatic traffic light marking method according to claim 1, characterized in that, The detection model is YOLOX or Co-DETR.

5. The semi-automatic traffic light marking method according to claim 1, characterized in that, The three-dimensional geometric properties of the traffic light include the traffic light's orientation category and the distance between the traffic light and the vehicle.

6. The semi-automatic traffic light marking method according to claim 1, characterized in that, The fusion matching process includes: calculating the intersection-union ratio and cost matrix of the two-dimensional bounding box and the projected bounding box; according to the cost matrix, the Hungarian algorithm matches the two-dimensional bounding box and the projected bounding box. If the two-dimensional bounding box is not matched, it is considered a false detection of the model. If the projected bounding box is not matched, it can be determined that the traffic light is blocked; the size of the matched projected bounding box is used as the final size of the traffic light; and the center coordinates of the matched two-dimensional bounding box are used as the final position of the traffic light.

7. The semi-automatic traffic light marking method according to claim 6, characterized in that, The formula for calculating the crossover-union ratio is: in, Indicates the first A two-dimensional outer frame; Indicates the first One projection outer frame; Represents a two-dimensional bounding box With projection frame The intersection region; Represents a two-dimensional bounding box With projection frame The union region; This indicates that the area of ​​the specified region will be calculated. This represents the intersection-union ratio, which ranges from [0,1].

8. The semi-automatic traffic light marking method according to claim 6, characterized in that, The formula for calculating the cost matrix is ​​as follows: in, Let $\mathbf{i}$ represent the matching cost between the $i$-th 2D bounding box and the $j$-th projected bounding box.

9. The semi-automatic traffic light marking method according to claim 1, characterized in that, Step S4 specifically includes: obtaining traffic light color data based on the HSV features of the traffic light image, the orientation category of the traffic light, and the distance; training a color classification model using the color data to obtain a trained color classification model; and using the trained color classification model to perform color discrimination on the traffic light image to obtain the traffic light color state.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the semi-automatic traffic light marking method as described in any one of claims 1-9.

Images