Traffic light detection and positioning method based on time sequence information fusion
By using a temporal information fusion method, the actual coordinates of traffic lights are calculated using the vehicle pose change matrix. This solves the problems of cumbersome traffic light detection and inaccurate positioning in existing technologies, achieving efficient and low-cost accurate traffic light perception and improving the safety of autonomous driving.
Patent Information
- Application Number
- CN202510943702.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-09
- Publication Date
- 2025-11-21
AI Technical Summary
Existing traffic light detection methods are cumbersome, expensive, and cannot achieve precise positioning, which affects the safety and efficiency of autonomous driving.
By extracting multiple sets of detection data from the same traffic light, calculating the actual coordinates of the traffic light using the vehicle pose change matrix, and combining multi-frame data for temporal information fusion, the system avoids expensive hardware and tedious manual annotation, thus achieving accurate detection and positioning of traffic lights.
It improves the accuracy and precision of traffic light detection, reduces costs, and enhances the accuracy and safety of traffic light perception in autonomous driving scenarios.
Smart Images
Figure CN120997794A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of automatic driving, and in particular to a traffic light detection and positioning method based on time sequence information fusion. BACKGROUND
[0002] With the development of automatic driving technology, accurate perception of traffic lights has become a key challenge to realize urban road automatic driving. In this process, traffic light detection and positioning is one of the core technologies of automatic driving and intelligent transportation system, and its accuracy and real-time performance are directly related to the safe passage and traffic efficiency of vehicles.
[0003] There are several methods for detecting traffic lights in the prior art:
[0004] 1. Traditional method based on computer vision: using color segmentation (HSV / RGB space), shape matching (circle, arrow template) and edge detection (Canny operator) to recognize traffic lights; the specific scheme is: (1) color threshold segmentation of candidate regions; (2) morphological filtering to remove noise; (3) Hough transform to detect circular or arrow shapes; (4) state switching according to time sequence analysis (such as red light to green light). The advantages of this method are simple principle and high real-time performance; the disadvantages are poor robustness, dependence on lighting conditions, susceptibility to similar objects (such as taillights, billboards), poor dynamic scene adaptability, and no actual traffic light position is given.
[0005] 2. Detection method based on deep learning: using neural network model to detect directly output traffic light position and state; the specific scheme is: using CNN or Transformer series neural network model to detect the image collected by the camera, directly giving the position and category state of the traffic light in the image. The advantages of this method are simple logic, high real-time performance and high accuracy; the disadvantages are the need for a large amount of data for training, and no actual traffic light position is given.
[0006] 3. Multi-sensor fusion scheme: combining camera, LiDAR, HD Map and V2X data to improve robustness. The specific scheme is: (1) camera + LiDAR: using LiDAR point cloud to locate the three-dimensional position of traffic lights, reducing occlusion false detection; (2) camera + HD Map: narrowing down the detection area through map prior information to improve efficiency; (3) V2X cooperation: receiving road side unit (RSU) signal light countdown data to enhance redundancy. The advantages of this method are the fusion of spatial and semantic information, the adaptation to complex weather (rain and fog) and dynamic occlusion scenarios; the disadvantages are the need for expensive hardware or a large amount of manual pre-annotation of traffic light positions on HD Map.
[0007] Obviously, the prior art mainly explores and applies the detection of traffic lights, and has problems such as complicated detection, high price, etc.; in addition, the traffic light is not positioned at the same time as the detection, and the dynamic and efficient positioning of the position cannot be realized.
[0008] Therefore, the prior art still needs to be improved. SUMMARY
[0009] In view of the above problems of the prior art, the purpose of the present application is to provide a traffic light detection and positioning method based on time sequence information fusion, which aims to solve the problems of complicated traffic light detection method, high price, etc. in the prior art; and to solve the problem that the position of the traffic light cannot be accurately positioned after the traffic light is detected in the prior art.
[0010] The technical scheme of the present application is as follows: a traffic light detection and positioning method based on time sequence information fusion, the method comprising:
[0011] Extracting a plurality of sets of detection data of the same traffic light, and obtaining the coordinates (u1, v1) and (u2, v2) of the two position points of the same traffic light when the vehicle moves from the first pose point to the second pose point. That is, the position point coordinates of the traffic light recorded by the vehicle at the first pose point are (u1, v1), and the position point coordinates of the same traffic light recorded by the vehicle at the second pose point are (u2, v2).
[0012] Calculating the normalized coordinates of (u1, v1) and (u2, v2) in the camera coordinate system, and calculating the actual coordinates of the traffic light according to the relative pose change matrix of the vehicle from the first pose point to the second pose point and the calculated normalized coordinates.
[0013] Therefore, the traffic light detection and positioning method based on time sequence information fusion in the present application can further process the detection data of the same traffic light, gradually calculate the actual coordinates of the traffic light, and avoid complicated manual pre-labeling without using expensive hardware conditions, so as to obtain the actual positioning of the traffic light, effectively reduce the cost, improve the accurate perception of the traffic light in the automatic driving scene, and improve the safety of the unmanned operation.
[0014] In addition, the final calculated actual coordinate position of the traffic light can be sent to the vehicle for decision-making of the traffic light passing logic, for example, the traffic light cannot be passed when it is red, and the traffic light can be passed when it is green.
[0015] In an embodiment, the process of extracting a plurality of sets of detection data of the same traffic light is:
[0016] Real-time detection of the acquired traffic light image information, vehicle positioning data, and output of the traffic light position information, category information. The vehicle is constantly collecting and updating the traffic light image information and vehicle positioning data.
[0017] The traffic light position information detected in the front and rear frames is tracked, the traffic light information corresponding to the same traffic light belonging to the same spatial position in the front and rear frames is matched, and the same ID information is assigned. Generally, after matching the traffic light information corresponding to the same traffic light belonging to the same spatial position in the front and rear frames, the ID of the latter frame is identified, and the ID value is the same as that of the former frame, that is, the same ID value is identified for the same traffic light in the same spatial position; if it is not matched, it means a new traffic light, and the ID of the current frame of the traffic light is assigned. Obviously, the application can realize the acquisition of traffic light detection data, realize the accurate detection of traffic lights, and compare the traffic light data of the front and rear frames in the detection process. The historical data can be effectively combined to realize the combination of historical data, ensure higher accuracy of detection data, provide accurate data for subsequent positioning calculation of traffic light position, and prevent misjudgment and other problems.
[0018] According to the ID information and timestamp of each frame of data, a plurality of groups of detection data of the same traffic light are extracted from the historical queue. Of course, the ID information in this process is generally a specific ID value.
[0019] In an embodiment, the normalized coordinates of (u1, v1) and (u2, v2) calculated in the camera coordinate system are x1 and x2, respectively, wherein x1=(un1, vn1), and x2=(un2, vn2). Specifically, the coordinates are projected to the normalized plane after distortion by the camera intrinsic parameters.
[0020] In an embodiment, the relative pose change matrix of the vehicle when moving from the first pose point to the second pose point is Mv=[R|t], wherein R is a 3*3 rotation matrix, and t is a 3*1 translation vector.
[0021] In an embodiment, the calculation process of the actual coordinates of the traffic light is as follows:
[0022] According to the calculated normalized coordinates x1 and x2 and the relative pose change matrix Mv=[R|t], an equation is established.
[0023]
[0024] Wherein, s1 and s2 are the first scale factor and the second scale factor, respectively, and M1 is the vehicle pose matrix of the first pose point; P is the actual three-dimensional coordinates of the traffic light, P=[Xw, Yw, Zw] T ;
[0025] The equations are solved simultaneously to obtain:
[0026] s2·x2=s1·R·x1+t.
[0027] In an embodiment, the detected traffic light position information of the front and rear frames is tracked, and the matching rule used when matching the traffic light information corresponding to the same traffic light in the same spatial position of the front and rear frames is: calculating the intersection over union IoU of the detection boxes of the front and rear frames, and the class consistency to associate the same traffic light. The class consistency refers to the class of the detected traffic light being the same class, for example, the detected traffic light in the two consecutive frames is a red light, and it is considered that the class is consistent.
[0028] In an embodiment, the traffic light image information is obtained by an image perception device on the vehicle; the traffic light image information includes camera calibration parameters and image data. The image perception device is an image sensor fixed on the vehicle, which is used to collect the traffic light image information of the environment in front of the vehicle in real time.
[0029] In an embodiment, the traffic light position information includes pixel coordinates of the traffic light in the image; and the class information includes red light, green light, yellow light, and countdown number.
[0030] In an embodiment, a tracking and matching algorithm is used to track the detected traffic light position information of the front and rear frames; the tracking and matching algorithm is Kalman filtering algorithm, Hungarian matching algorithm, sort tracking and matching algorithm, and / or deepsort tracking and matching algorithm. In addition, when tracking the detected traffic light position information of the front and rear frames by using the tracking and matching algorithm, the parameters corresponding to the coordinates of the captured traffic light in each frame are used as the state vector (including the target pixel coordinates, the change amount of the target pixel value with time, i.e. the speed of the target pixel), that is, the target state vector is defined as [u, v, du / dt, dv / dt], for example, for the position point coordinates (u1, v1) of the recorded traffic light of the vehicle at the first pose point, the defined target state vector is [u1, v1, du1 / dt, dv1 / dt].
[0031] In summary: the traffic light detection and positioning method based on time sequence information fusion proposed in this paper has the following beneficial effects:
[0032] 1. Comparing the traffic light data of the front and rear frames in the detection process can effectively combine multiple frames of data, realize the combination of historical data, ensure the high accuracy of the detection data, and effectively improve the accuracy of the traffic light detection;
[0033] 2. The actual coordinates of the traffic light can be accurately calculated without using expensive hardware conditions, and manual pre-labeling and other tedious processes are avoided, thereby effectively reducing the cost;
[0034] 3. The final calculated actual coordinate position of the traffic light can be sent to the vehicle for decision-making of the traffic light passing logic, improving the accurate perception of the traffic light in the automatic driving scene and improving the safety of unmanned operation. BRIEF DESCRIPTION OF DRAWINGS
[0035] The application will be further described below in conjunction with the drawings and embodiments. In the drawings:
[0036] Fig. 1 The flowchart is for steps S1-S2 of the application.
[0037] Fig. 2 The flowchart is for steps S10-S12 of the application.
[0038] Fig. 3 The schematic diagram is for the vehicle driving from m1 to m2, and the vehicle and the traffic light. DETAILED DESCRIPTION
[0039] To make the purpose, technical scheme and effect of the application clearer and more explicit, the application will be further described in detail below. It should be understood that the specific embodiments described herein are only used to explain the application and not to limit the application. The embodiments of the application will be described below in conjunction with the drawings.
[0040] Please refer to Figs. 1-3 A traffic light detection and positioning method based on time sequence information fusion can accurately detect the traffic light, extract several sets of detection data of the same traffic light, further process the detection data of the same traffic light, gradually calculate the actual coordinates of the traffic light, and obtain the actual positioning of the traffic light without using expensive hardware conditions and avoiding tedious manual pre-labeling, thereby effectively reducing the cost, improving the accurate perception of the traffic light in the automatic driving scene, and improving the safety of unmanned operation. Specifically, the method comprises the following steps:
[0041] S1: Extracting several sets of detection data of the same traffic light and obtaining the coordinates (u1, v1) and (u2, v2) of the two position points of the same traffic light when the vehicle moves from the first pose point to the second pose point.
[0042] Specifically, the position point coordinate of the traffic light recorded by the vehicle at the first pose point is (u1, v1), and the position point coordinate of the same traffic light recorded at the first pose point at the second pose point is (u2, v2). In this embodiment, the first pose point of the vehicle is m1, the second pose point is m2, and the actual position of the traffic light is point P.
[0043] Specifically, in this embodiment, the process of extracting several sets of detection data of the same traffic light is:
[0044] S10: Real-time detection of the acquired traffic light image information, vehicle positioning data, and output of the traffic light position information, category information.
[0045] Specifically, the traffic light image information is acquired by the image perception device on the vehicle; in this embodiment, the image perception device is an image sensor fixed on the vehicle, which is configured as: (1) monocular camera: resolution 1920x1280, frame rate 30FPS, fixedly installed in front of the vehicle, optical axis parallel to the vehicle forward direction; (2) positioning unit: integrated GNSS (Global Navigation Satellite System) and IMU (Inertial Measurement Unit), outputting vehicle pose (position, attitude angle) and motion parameters (speed, angular velocity); (3) computing platform: vehicle embedded industrial computer (such as NVIDIA Jetson AGX Orin). The monocular camera is connected to the industrial computer on the vehicle through the GigE interface to realize the transmission of traffic light image information; the integrated GNSS (Global Navigation Satellite System) and IMU (Inertial Measurement Unit) communicate with the industrial computer of the vehicle through CAN bus or USB interface, generally at a frequency of 100HZ, outputting positioning data (actual coordinates of the vehicle); the vehicle positioning data is acquired by the GPS module or SLAM positioning module on the vehicle, and through the vehicle positioning data, the specific real-time position of the unmanned vehicle, vehicle pose matrix M1 (calculated and processed by the existing data processing module on the vehicle), and other data can be obtained.
[0046] Specifically, in this embodiment, the traffic light image information includes camera calibration parameters and image data; the traffic light position information includes pixel coordinates of the traffic light in the image; and the category information includes red light, green light, yellow light, and countdown numbers (identified by an OCR module). The monocular camera continuously collects and updates the traffic light image information and performs real-time calculation. In addition, it should be noted that in the detection process of step S10, the model architecture used is: the anchor box size and feature fusion layer are optimized for small target traffic lights, for example, the YOLO series target detection network; the input data: the original image of the monocular camera is in RGB format, and the resolution is scaled to 640x640; the output result is the traffic light position information and category information, and confidence information is also output, which shows the credibility of the detection result, and the threshold value of the credibility is generally set to 0.25.
[0047] S11: Track the traffic light position information detected in the previous and subsequent frames, match the traffic light information corresponding to the same traffic light in the same spatial position in the previous and subsequent frames, and assign the same ID information to the same traffic light.
[0048] Specifically, in the present embodiment, the tracking matching algorithm is used to track the traffic light position information detected by the front and rear frames; the tracking matching algorithm is Kalman filtering algorithm and / or Hungarian matching algorithm and / or sort tracking matching algorithm and / or deepsort tracking matching algorithm. In addition, when tracking the traffic light position information detected by the front and rear frames by using the tracking matching algorithm, the parameters corresponding to the coordinates of the traffic light captured in each frame are used as the state vector (including the target pixel coordinates, the change amount of the target pixel value over time, i.e. the speed of the target pixel), that is, the target state vector is defined as [u, v, du / dt, dv / dt], for example, for the position point coordinates (u1, v1) of the traffic light recorded by the vehicle at the first pose point, the defined target state vector is [u1, v1, du1 / dt, dv1 / dt].
[0049] In addition, it should be noted that generally after matching the traffic light information corresponding to the same traffic light in the same spatial position between the front and rear frames, the ID of the latter frame is identified, and the ID value is the same as that of the former frame, that is, the same ID value is identified for the same traffic light in the same spatial position; if it is not matched, it means that it is a new traffic light, and the ID of the current frame of the traffic light is assigned; and after assigning the ID of each new traffic light, if the same traffic light in the same spatial position is not matched for the next 5 consecutive frames, the ID is cancelled.
[0050] S12: Extracting a plurality of groups of detection data of the same traffic light from the historical queue according to the ID information and the time stamp of each frame data. Of course, the ID information in this process is generally a specific ID value. Specifically, the recorded traffic light image information, traffic light position information, category information, etc. of each traffic light detected by the previous frame and the traffic light image information, traffic light position information, category information tracked by the current frame (the next frame) can constitute a time sequence frame data. Specifically, the time stamps of the traffic light image information and the vehicle positioning data can be aligned by using the message_filters module of ROS2.
[0051] Specifically, the structure of the time sequence frame data is generally as follows:
[0052]
[0053] Obviously, in the calculation process of steps S10-S12, the traffic light image information is acquired in real time by the image sensor, and the tracking of the traffic light position information detected by the front and rear frames is realized by the tracking algorithm. In the whole process, the traffic light related data of the front and rear frames is compared, the combination of multiple frame data is effectively combined, the combination of the current data and the historical data is realized, the detection data accuracy is higher, the accurate data is provided for the subsequent positioning calculation of the traffic light position, and the misjudgment and other problems are prevented.
[0054] S2: calculate the normalized coordinates of the (u1, v1), (u2, v2) in the camera coordinate system, and calculate the actual coordinates of the traffic light according to the relative pose change matrix of the vehicle when moving from the first pose point to the second pose point, the calculated normalized coordinates.
[0055] In addition, the finally calculated actual coordinate position of the traffic light can be sent to the vehicle for decision-making of the traffic light passing logic, for example, not passing when it is a red light, and passing when it is a green light. The whole process is simple, fast, efficient, and real-time, effectively improving the accurate perception of the traffic light in the automatic driving scene and improving the safety of unmanned operation.
[0056] Specifically, in the embodiment, the normalized coordinates of the two position points corresponding to the same traffic light are x1 and x2 when the vehicle moves from the first pose point m1 to the second pose point m2, and the x1=(un1, vn1), and the x2=(un2, vn2). Specifically, the coordinates are projected to the normalized plane after removing the distortion of the camera intrinsic parameter. It should be noted that the camera coordinate system is the coordinate system in which the image sensor fixed on the vehicle is located, and the camera intrinsic parameter is the camera intrinsic parameter of the image sensor fixed on the vehicle. Since (u1, v1) and (u2, v2) are known, the related values of the normalized coordinates x1 and x2 can be directly calculated.
[0057] Specifically, in the embodiment, the relative pose change matrix of the vehicle when moving from the first pose point to the second pose point is Mv=[R|t], wherein R is a 3*3 rotation matrix (Euler angle conversion), and t is a 3*1 translation vector (vehicle displacement).
[0058] Specifically, in the embodiment, the calculation process of the actual coordinates of the traffic light is:
[0059] According to the calculated normalized coordinates x1 and x2, the relative pose change matrix Mv=[R|t], and the camera projection model, an equation can be established:
[0060]
[0061] Wherein, s1 and s2 are respectively a first scale factor and a second scale factor, M1 is a vehicle pose matrix of the first pose point; P is an actual three-dimensional coordinate of the traffic light, P=[Xw, Yw, Zw] T ;
[0062] The equations are solved together to obtain:
[0063] s2·x2=s1·R·x1+t;
[0064] After calculation, s1 and s2 can be solved, and finally substituted back into the original equation, P, i.e. the actual coordinates of the traffic light, can be solved. In addition, multiple time sequence frame data can be solved simultaneously to construct an overdetermined equation group, and the least square method can be used to solve more accurate P coordinates.
[0065] Specifically, in the present embodiment, the traffic light position information detected in the front and rear frames is tracked, and the matching rule adopted when matching the traffic light information corresponding to the same traffic light belonging to the same spatial position in the front and rear frames is: calculating the intersection over union IoU of the detection boxes of the front and rear frames, and the category consistency, to associate the same traffic light. The category consistency refers to that the category of the detected traffic light is the same category, for example, the traffic light detected in the front and rear two frames is a red light, and it is considered that the category is consistent. In addition, in the process of tracking the traffic light position information detected in the front and rear frames, the calculation of the actual coordinate position of the traffic light is triggered only when the same traffic light accumulates ≥2 frame data. Therefore, when only one frame of data related to the same traffic light is detected, the calculation of the actual coordinate position of the traffic light will not be triggered, effectively avoiding the waste of computing resources.
[0066] It should be understood that the application of the present application is not limited to the above examples, and those skilled in the art can improve or change it according to the above description, and all these improvements and changes shall belong to the protection scope of the appended claims of the present application.
Claims
1. A traffic light detection and positioning method based on time sequence information fusion, characterized in that, The method comprises: extracting a plurality of sets of detection data of the same traffic light, and obtaining coordinates (u1, v1) and (u2, v2) of two position points of the same traffic light when the vehicle moves from a first pose point to a second pose point; calculating normalized coordinates of the (u1, v1) and (u2, v2) in a camera coordinate system, and calculating actual coordinates of the traffic light according to a relative pose change matrix of the vehicle when the vehicle moves from the first pose point to the second pose point and the calculated normalized coordinates. 2.The traffic light detection and positioning method based on time sequence information fusion according to claim 1, wherein, The process of extracting a plurality of sets of detection data of the same traffic light comprises: real-time detecting acquired traffic light image information and vehicle positioning data, and outputting traffic light position information and category information; tracking the traffic light position information detected in front and back frames, matching traffic light information corresponding to the same traffic light belonging to the same spatial position in the front and back frames, and assigning the same ID information to the same traffic light; extracting a plurality of sets of detection data of the same traffic light from a historical queue according to ID information and time stamps of each frame of data. 3.The traffic light detection and positioning method based on time sequence information fusion according to claim 1, wherein, The calculated normalized coordinates of the (u1, v1) and (u2, v2) in the camera coordinate system are x1 and x2 respectively, wherein the x1=(un1, vn1), and the x2=(un2, vn2).
4. The traffic light detection and positioning method based on time sequence information fusion according to claim 3, characterized in that, The relative pose change matrix of the vehicle when the vehicle moves from the first pose point to the second pose point is Mv=[R|t], wherein R is a 3*3 rotation matrix, and t is a 3*1 translation vector.
5. The traffic light detection and positioning method based on time-series information fusion according to claim 4, characterized in that, The process of calculating the actual coordinates of the traffic light comprises: establishing an equation according to the calculated normalized coordinates x1 and x2 and the relative pose change matrix Mv=[R|t]; Wherein, s1, s2 are respectively the first scale factor, the second scale factor, M1 is the vehicle pose matrix of the first pose point; P is the actual three-dimensional coordinate of the traffic light, P = [Xw, Yw, Zw] T ; solving the equation to obtain: s2·x2=s1·R·x1+t.
6. The traffic light detection and positioning method based on time sequence information fusion according to claim 2, characterized in that, When tracking the traffic light position information detected in front and back frames and matching traffic light information corresponding to the same traffic light belonging to the same spatial position in the front and back frames, a matching rule adopted is to calculate an intersection over union IoU and category consistency of detection boxes in the front and back frames to associate the same traffic light.
7. The traffic light detection and positioning method based on time-series information fusion according to claim 2, characterized in that, The traffic light image information is acquired by an image perception device on the vehicle; and the traffic light image information comprises camera calibration parameters and image data.
8. The traffic light detection and positioning method based on time sequence information fusion according to claim 2, characterized in that, The traffic light position information comprises pixel coordinates of the traffic light in an image; and the category information comprises a red light, a green light, a yellow light, and countdown numbers.
9. The traffic light detection and positioning method based on time-series information fusion according to claim 2, characterized in that, The tracking matching algorithm is a Kalman filtering algorithm, a Hungarian matching algorithm, a sort tracking matching algorithm, and / or a deepsort tracking matching algorithm.