Multi-camera coordinated traffic light and traffic sign multi-task perception system

Abstract

The application discloses a multi-camera cooperative traffic light and traffic sign multi-task perception system, and relates to the technical field of traffic environment perception. According to the position information of the intersection traffic light in the high-precision map and the real-time pose information of the vehicle, the distance between the current position of the vehicle and the target intersection is calculated, and the camera with the corresponding field of view is selected according to the comparison result of the distance and the preset threshold value. When the vehicle approaches the intersection, the steering state of the vehicle and the driving route in the high-precision map are combined to further determine whether to call the surround-view camera, and the target camera video stream for subsequent processing is obtained. The application realizes multi-view coverage through adaptive camera selection, improves the far and near target continuous perception capability and reduces the blind area, simultaneously constructs a unified multi-task perception framework, combines the regional guidance and the space-time fusion mechanism, realizes the stable identification of the traffic light, the countdown and the traffic sign, and thus improves the overall perception efficiency and the complex environment adaptability.

Description

Technical Field

[0001] This application relates to the field of traffic environment perception technology, specifically to a multi-camera collaborative multi-task perception system for traffic lights and traffic signs. Background Technology

[0002] With the continuous development of autonomous driving technology, the vehicle's ability to perceive the external traffic environment has become a key foundation for ensuring driving safety and decision-making accuracy. Traffic lights and signs, as important carriers of road traffic rules, directly impact the performance of autonomous driving systems in areas such as intersection navigation, route planning, and behavioral decision-making. Current technologies typically employ deep learning-based target detection algorithms to process image data collected by forward-looking cameras, thereby enabling the detection and classification of traffic lights and signs. For example, convolutional neural networks are used to locate targets in images and identify the color state of traffic lights or the category of traffic signs. Furthermore, some systems combine multi-sensor data (such as LiDAR and millimeter-wave radar) to assist environmental perception; however, visual sensors still dominate in traffic light and sign recognition tasks. Overall, most existing autonomous driving perception systems rely on a single forward-looking camera for target detection or implement traffic light and sign recognition as independent task modules.

[0003] However, the aforementioned existing technologies still have many shortcomings in practical applications. First, most solutions do not fully consider the collaborative perception problem among multiple cameras. Different cameras have different fields of view (FOV) and observation ranges, making it difficult for a single camera to cover all key targets in a complex intersection environment, easily leading to missed or false detections of traffic lights or traffic signs. Second, existing traffic light detection methods typically only identify the color state of the traffic light, ignoring the detection and parsing of countdown information. This results in the system being unable to obtain the remaining time of the traffic light, thus limiting the downstream decision-making module's ability to finely control the timing of passage. Third, in autonomous vehicles, computing and storage resources are usually limited, and existing solutions often build independent models for tasks such as traffic light detection and traffic sign recognition, resulting in high system resource consumption and difficulty in balancing real-time performance and other requirements. Furthermore, since traffic lights and traffic signs are usually small-scale targets in images, existing detection algorithms have low recognition accuracy in complex lighting, occlusion, or long-distance scenes, and lack a unified multi-task perception framework to simultaneously process multiple types of targets and achieve optimized resource allocation.

[0004] Therefore, how to achieve multi-camera collaboration, multi-task fusion, and high-precision small target detection under limited resources has become an urgent technical problem to be solved.

[0005] The information disclosed in the background section is only intended to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The purpose of this application is to provide a multi-camera collaborative traffic light and traffic sign multi-task perception system to solve the problems mentioned in the background art.

[0007] To achieve the above objectives, this application provides the following technical solution: a multi-camera collaborative traffic light and traffic sign multi-task perception system, including a camera selection module, a two-dimensional target detection module, a fusion post-processing module, and a 3D position output module; The camera selection module calculates the distance between the vehicle's current position and the target intersection based on the location information of traffic lights at the intersection in the high-precision map and the real-time pose information of the vehicle. It then selects the camera with the corresponding field of view based on the comparison result between the distance and a preset threshold. When the vehicle approaches the intersection, it further determines whether to call the surround-view camera by combining the vehicle's turning state and the driving route in the high-precision map, thereby obtaining the target camera video stream for subsequent processing. The 2D target detection module processes the video stream from the target camera, detects the type and 2D location information of traffic lights, countdown lights, and traffic signs, and projects the spatial location information of traffic lights at the corresponding intersections in the high-precision map onto the image plane through camera parameters to generate target areas. Within the target areas, the detection results are matched to determine the traffic lights of interest. When a countdown light is present in the target area, the countdown area is cropped and its digital information is identified. When no countdown light is detected in the target area, candidate areas are adaptively generated based on the spatial distribution of traffic lights to perform countdown detection again, thereby obtaining 2D detection results of traffic light status, countdown information, and traffic sign information. The fusion post-processing module, based on the two-dimensional detection results, fuses traffic light, countdown light and traffic sign information from multiple cameras, and combines target tracking algorithm and state switching logic to update the traffic light state and countdown information in a timely manner to obtain stable multi-target perception results; The 3D position output module performs depth mapping on the two-dimensional position information in the fused multi-target perception results based on the depth information generated by the LiDAR point cloud, calculates the spatial position of each target, and maps the target to a three-dimensional spatial coordinate system in combination with the camera's intrinsic and extrinsic parameters, thereby outputting the three-dimensional position information of traffic lights, countdown lights, and traffic signs.

[0008] The camera selection module includes a multi-dimensional state determination unit. This unit combines the vehicle's latitude and longitude coordinates, heading angle change rate, and lateral acceleration to form a composite state vector. It then generates camera activation commands through a preset determination function. The activation conditions for telephoto cameras include the matching relationship between distance parameters and viewing angle coverage. The activation conditions for wide-angle cameras include the integrity constraints of the field of view coverage. The activation conditions for surround-view cameras include the lateral target coverage requirements. Through the joint constraints of multiple conditions, a dynamic camera scheduling mechanism is formed, thereby realizing the continuous switching of target perception capabilities in different spatial areas.

[0009] The two-dimensional target detection module includes a region generation unit guided by a high-precision map. This region generation unit performs projection calculations using the three-dimensional coordinates of the traffic light and the camera's intrinsic parameters to form multiple candidate projection regions. It then obtains a unified detection region through geometric envelope operations and performs proportional expansion processing on the region boundaries to construct an enhanced detection region containing potential small targets. Combined with the detection model, targets within the region are prioritized for identification to improve the ability to recognize traffic lights at long distances.

[0010] The two-dimensional target detection module further includes a countdown recognition subunit. This subunit improves the digital region feature representation capability by cropping the area near the traffic light and performing multi-scale image preprocessing, including brightness normalization, contrast enhancement, and local noise reduction. It also parses the countdown sequence through digital sorting relationships to infer the remaining time of the traffic signal.

[0011] The two-dimensional target detection module includes a countdown region adaptive selection mechanism. This mechanism constructs candidate regions based on the spatial distribution of traffic lights. When there are multiple traffic lights, a minimum bounding region generation strategy is adopted. When there are fewer traffic lights, a dynamically expanding region centered on the traffic light center is constructed. Through multiple region expansion and detection cycles, the countdown region is gradually approached, thereby improving the success rate of weak feature target recognition.

[0012] The fusion post-processing module includes a multi-source target association unit. This unit constructs target matching relationships through spatial distance constraints and category consistency constraints, assigns unified labels to detection results from different cameras, and filters and fuses duplicate targets through a target confidence weight allocation mechanism, thereby forming a unique target representation set.

[0013] The fusion post-processing module includes a time series filtering unit, which uses a Kalman filtering model to estimate the state of traffic lights and traffic signs. It achieves target trajectory smoothing through prediction and update cycles, and combines an anomaly detection mechanism to remove data from abrupt change locations, thereby reducing the impact of detection noise on the stability of the results.

[0014] The fusion post-processing module further includes a state logic constraint unit. This unit restricts the color change path by pre-setting traffic light state transition rules, allowing only state transition relationships that conform to traffic rules. At the same time, it combines the countdown decrement law to perform consistency verification on the results of consecutive frames, thereby avoiding unreasonable state jump phenomena.

[0015] The 3D position output module includes a depth estimation unit, which generates a depth map by projecting LiDAR point cloud data and performs statistical processing on the depth value set within the target area. It uses the median estimation method to obtain stable depth results and combines multi-frame depth information for weighted fusion to improve spatial positioning accuracy.

[0016] The 3D position output module includes a multi-camera coordinate unification unit. This unit completes coordinate transformation through the calibration parameters of each camera, maps the target position obtained from different perspectives to the vehicle coordinate system, and performs spatial fusion processing in combination with the multi-view observation results to obtain a more consistent three-dimensional target position expression.

[0017] The technical effects and advantages provided by this application in the above technical solution are as follows: This application introduces an adaptive camera selection mechanism based on vehicle operating status and high-precision map information to dynamically allocate perception resources at different distances and in different spatial directions, thereby continuously maintaining the effective coverage of traffic lights and traffic signs during vehicle movement. Through the reasonable switching and combination of multi-field-of-view cameras, the perception blind spot problem caused by a single viewpoint can be avoided, while simultaneously improving the continuous observation capability of both distant and near targets. This results in more complete acquisition of traffic control information, providing more continuous and reliable environmental perception results for subsequent decision-making modules.

[0018] This application constructs a unified multi-task perception framework, integrating traffic light status recognition, countdown parsing, and traffic sign detection into a single processing system. It enhances small-scale target recognition capabilities through region guidance and target selection mechanisms, while reducing resource consumption issues caused by independent operation of multiple models. By combining temporal fusion and spatial localization strategies, it maintains stable output of detection results under complex environmental conditions, ensuring consistency of perceived information in both temporal and spatial dimensions. This improves the overall system efficiency and enhances adaptability to complex scenarios. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0020] Figure 1 This is a system architecture diagram for this application.

[0021] Figure 2 Flowchart for selecting the camera for this application.

[0022] Figure 3 This is a flowchart of the testing process for traffic lights and traffic signs in this application.

[0023] Figure 4 This is a flowchart of the countdown testing process for this application. Detailed Implementation

[0024] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that the description of this disclosure will be more complete and fully convey the concept of the exemplary embodiments to those skilled in the art.

[0025] like Figures 1 to 4 As shown, this application provides a multi-camera collaborative traffic light and traffic sign multi-task perception system, including a camera selection module, a two-dimensional target detection module, a fusion post-processing module, and a 3D position output module; The camera selection module calculates the distance between the vehicle's current position and the target intersection based on the location information of traffic lights at the intersection in the high-precision map and the real-time pose information of the vehicle. It then selects the camera with the corresponding field of view based on the comparison result between the distance and a preset threshold. When the vehicle approaches the intersection, it further determines whether to call the surround-view camera by combining the vehicle's turning state and the driving route in the high-precision map, thereby obtaining the target camera video stream for subsequent processing. The 2D target detection module processes the video stream from the target camera, detects the type and 2D location information of traffic lights, countdown lights, and traffic signs, and projects the spatial location information of traffic lights at the corresponding intersections in the high-precision map onto the image plane through camera parameters to generate target areas. Within the target areas, the detection results are matched to determine the traffic lights of interest. When a countdown light is present in the target area, the countdown area is cropped and its digital information is identified. When no countdown light is detected in the target area, candidate areas are adaptively generated based on the spatial distribution of traffic lights to perform countdown detection again, thereby obtaining 2D detection results of traffic light status, countdown information, and traffic sign information. The fusion post-processing module, based on the two-dimensional detection results, fuses traffic light, countdown light and traffic sign information from multiple cameras, and combines target tracking algorithm and state switching logic to update the traffic light state and countdown information in a timely manner to obtain stable multi-target perception results; The 3D position output module performs depth mapping on the two-dimensional position information in the fused multi-target perception results based on the depth information generated by the LiDAR point cloud, calculates the spatial position of each target, and maps the target to a three-dimensional spatial coordinate system in combination with the camera's intrinsic and extrinsic parameters, thereby outputting the three-dimensional position information of traffic lights, countdown lights, and traffic signs.

[0026] Implementation method one: such as Figure 1 As shown, this embodiment provides a multi-camera collaborative traffic light and traffic sign multi-task perception system. The system mainly consists of a camera selection module, a 2D target detection module, a fusion post-processing module, and a 3D position output module. These modules are sequentially connected via data streams to form a closed-loop processing link, enabling high-precision perception and spatial positioning of traffic light status, countdown information, and traffic sign type when a vehicle is traveling straight through an intersection. This embodiment focuses on the typical application scenario of vehicles traveling straight through an intersection along a predetermined lane. In this scenario, traffic participants are relatively concentrated but their distances vary significantly, placing high demands on both long-range recognition capabilities and short-range coverage capabilities.

[0027] In this embodiment, the camera selection module first receives data from a high-precision map and real-time vehicle operation status data. The high-precision map data includes the latitude, longitude, and altitude information of traffic lights, turning paths, road geometry, etc., while the real-time vehicle operation status data includes the latitude and longitude coordinates of the vehicle's current position, heading angle, longitudinal velocity, lateral acceleration, and angular velocity. Based on the above information, the camera selection module performs a distance calculation process, that is, by calculating the spatial distance between the vehicle's current position and the positions of traffic lights at each candidate intersection in the high-precision map, it selects the target intersection with the smallest distance as the current focus, and further selects the camera mode based on the relationship between distance and a preset threshold.

[0028] like Figure 2 As shown, when the vehicle is far from the target intersection, the forward-looking telephoto camera is prioritized. This camera has a narrower field of view but higher pixel density and stronger long-distance resolution, effectively improving the recognition accuracy of traffic lights at a distance. As the vehicle approaches the target intersection, when the distance is less than a preset threshold, the system automatically switches to a forward-looking wide-angle camera to expand the field of view coverage, ensuring that multiple traffic lights and related traffic signs can be fully captured after entering the intersection area. In straight-ahead scenarios, since the vehicle does not exhibit significant turning behavior, the camera selection module maintains only the forward-looking wide-angle camera at close range, without activating the surround-view camera, thereby reducing computational resource consumption while ensuring the integrity of perception. Finally, the camera selection module outputs the target camera video stream matching the current driving stage, providing input data for subsequent visual perception processing.

[0029] Subsequently, the 2D target detection module processes the video stream output by the camera selection module in real time, such as... Figure 3 As shown, this module, based on a deep learning object detection model deployed on the edge, jointly detects traffic lights, countdown lights, and traffic signs in video frames, outputting the two-dimensional location, category information, and confidence score for each type of object. Unlike traditional full-image detection methods, this implementation introduces a spatial constraint mechanism based on a high-precision map. By projecting the three-dimensional spatial coordinates of the traffic lights at the target intersection in the high-precision map in conjunction with camera intrinsic parameters, the projection area of ​​the traffic lights on the image plane is obtained. Furthermore, multiple projection areas are merged to generate a Region of Interest (ROI) containing all candidate traffic lights. This ROI spatially corresponds highly to the actual traffic light locations, effectively narrowing the detection search range, improving detection efficiency, and reducing the false detection rate.

[0030] Within the ROI area, the 2D target detection module filters and matches the detection results, associating detected traffic light targets with traffic light information in the high-precision map to determine the traffic light objects that need to be focused on. If a countdown light is detected, the system crops the countdown light area and calls the digit recognition sub-model to identify the numbers within the area. By analyzing the horizontal arrangement of the numbers, the system calculates the current countdown seconds, thus obtaining the remaining time information for the traffic light. When no countdown light is detected within the ROI area, the system further calls the countdown area adaptive selection sub-module, such as... Figure 4 As shown, potential countdown regions are generated based on the spatial distribution of traffic lights within the ROI area. For example, when multiple traffic lights are detected, the smallest enclosing area formed by these lights is used as the candidate region; when the number of traffic lights is small, a circular search area is constructed based on the center point of the traffic lights, and the search range is dynamically adjusted according to the spacing between the traffic lights. The countdown region is gradually located through multiple cropping and detection processes. Through this adaptive mechanism, the probability of obtaining countdown information can still be improved even if the countdown lights are not identified in the initial detection.

[0031] After obtaining the two-dimensional detection results of traffic lights, countdown lights, and traffic signs, the fusion post-processing module fuses the data from consecutive time frames. This module first constructs target associations based on target location and category information, then uses a Kalman filter algorithm to perform state estimation and trajectory tracking for each target, smoothing target position changes and suppressing detection noise. Simultaneously, traffic light state switching logic is introduced to constrain traffic light color changes; for example, the transition from red to green must conform to the actual traffic rule time sequence, thus avoiding state jumps caused by detection errors. The fusion post-processing module also performs consistency checks on countdown information from different time frames, correcting abnormal readings through time continuity constraints to improve the stability of the countdown results. Finally, the module outputs multi-target perception results optimized in both time and space.

[0032] After fusing the 2D perception results, the 3D position output module spatially locates the target. This module first uses the LiDAR onboard the autonomous vehicle to acquire environmental point cloud data and generates a corresponding depth map through point cloud projection, aligning the 2D image coordinates with the depth information. For each detected target, the system extracts the depth value of the corresponding region from the depth map and uses the median calculation method to determine the target's depth distance, reducing the impact of noise points. Subsequently, combining the camera's intrinsic and extrinsic parameters, the target coordinates and depth values ​​on the image plane are back-projected to obtain the target's 3D position in the vehicle coordinate system. This process enables precise positioning of traffic lights, countdown lights, and traffic signs in 3D space, providing reliable spatial information input for subsequent path planning and decision control modules.

[0033] This implementation uses a camera selection module to dynamically schedule sensing resources, a 2D target detection module to perform joint detection and countdown recognition of multiple target categories, a fusion post-processing module to optimize the timing and stabilize the output of multi-frame data, and a 3D position output module to achieve spatial positioning of targets. These modules work together to construct a highly efficient, stable, and resource-friendly multi-camera collaborative sensing system suitable for straight-ahead intersection scenarios. This system not only improves the detection accuracy of traffic lights and signs but also achieves multi-task fusion and high-precision 3D positioning under limited computing resources, demonstrating strong engineering application value and promotional significance.

[0034] Implementation Method Two: (e.g.) Figure 1As shown, this embodiment constructs a multi-camera collaborative perception system for vehicle turning through intersections. The overall architecture still consists of a camera selection module, a 2D target detection module, a fusion post-processing module, and a 3D position output module. These modules are sequentially connected via data flow to form a unified perception and processing link. Compared to straight-ahead scenarios, turning scenarios are more complex, mainly in that the perception range expands from a single forward area to lateral and even rearward areas. Additionally, traffic lights are more diverse in their locations, with some potentially located to the side or even vertically. Therefore, higher demands are placed on the camera's field-of-view coverage, multi-view fusion capabilities, and target matching stability. This embodiment focuses on solving the problem of complete perception and stable tracking of traffic lights and signs during turning, thereby providing a reliable basis for the autonomous driving system's turning decision-making process.

[0035] In this embodiment, the camera selection module first receives high-precision map data and real-time vehicle pose data. The high-precision map data includes the latitude, longitude, and altitude information of traffic lights, turning paths, road geometry, etc., while the real-time vehicle pose data includes dynamic parameters such as current position coordinates, heading angle change rate, lateral acceleration, and steering wheel angle. Based on the above data, the camera selection module analyzes the vehicle's motion state to determine whether it is about to enter a turning process. Specifically, when the heading angle change rate exceeds a preset threshold or the lateral acceleration reaches a certain range, and the current path in the high-precision map contains a turning section, it is determined that the vehicle is in the turning preparation stage or the turning execution stage.

[0036] like Figure 2 As shown, the camera selection strategy differs significantly from that in the straight-ahead scenario during this stage. When the vehicle is far from the intersection, a forward-looking telephoto camera can still be used to detect traffic lights at a distance to obtain traffic signal status in advance. As the vehicle approaches the intersection and enters the turning judgment condition, the system no longer relies solely on the forward-looking camera but simultaneously activates a forward-looking wide-angle camera and a surround-view camera. The forward-looking wide-angle camera covers the area in front of the vehicle and part of the side front, while the surround-view camera captures information about the vehicle's sides and nearby areas, thus forming multi-view joint coverage. In this way, the problem of side traffic lights being difficult to capture by a single forward-looking camera during turning can be effectively solved. The camera selection module ultimately outputs multiple video streams, providing the input basis for subsequent multi-view perception processing.

[0037] After acquiring multiple video streams, the 2D object detection module processes each video stream independently, such as... Figure 3As shown, each video stream is input to the same or shared weighted target detection model to detect traffic lights, countdown lights, and traffic signs, outputting the corresponding 2D location, category information, and confidence score. Due to differences in the viewing angle and resolution of different cameras, the detection results exhibit spatial variations. Therefore, a spatial constraint mechanism based on a high-precision map is introduced during the detection phase. Specifically, the 3D coordinates of the traffic lights in the high-precision map are projected onto the corresponding image planes using the intrinsic parameters of each camera, generating ROI regions for each video stream. These ROI regions have different locations in different cameras but correspond to the same traffic light target in physical space.

[0038] Within each Region of Interest (ROI), the system filters and matches detection results, prioritizing traffic light targets located within the ROI and possessing high confidence levels to reduce the risk of false detections due to background interference. For countdown detection, when a countdown light is successfully detected in a video stream, the affected area is directly cropped and a digit recognition operation is performed to obtain the countdown timer value. If no countdown light is detected in a video stream, the countdown region adaptive selection submodule is invoked, such as... Figure 4 As shown, candidate regions are constructed based on the spatial distribution of traffic lights within the ROI. For example, when one or more positive traffic lights are detected, a circular or rectangular search region is constructed based on the center position of these traffic lights, and the region size is dynamically adjusted according to the spacing between the traffic lights. By performing secondary detection on the candidate regions, the countdown recognition probability is improved. In multi-camera scenarios, the countdown detection results from different cameras can complement each other, thereby improving the robustness of the overall system.

[0039] After each camera completes its independent detection, the fusion post-processing module performs unified fusion processing on the multi-channel detection results. This module first performs spatial fusion, mapping the detection results from different cameras to a unified vehicle coordinate system. Targets with similar spatial locations and consistent categories are associated, eliminating duplicate detections and unifying target identification. Subsequently, a temporal fusion process is performed, tracking targets in consecutive time frames and using a Kalman filter algorithm to predict and update target positions, smoothing target trajectories and reducing the impact of detection noise. For traffic light state processing, state machine-based logical constraints are introduced. For example, traffic light state changes must conform to a reasonable switching sequence between red, yellow, and green, thus avoiding unreasonable state jumps due to detection errors. Simultaneously, countdown information undergoes time consistency verification. If the countdown changes in consecutive frames do not conform to the time decrease rule, abnormal data is corrected or removed, ensuring the stability and reliability of the output results.

[0040] After the fusion process is complete, the 3D position output module performs spatial localization on the fused 2D target. This module uses LiDAR to acquire environmental point cloud data and projects the point cloud to generate a depth map, establishing a correspondence between image coordinates and depth information. For each target, the depth value set of the corresponding region is extracted from the depth map, and the median is used as the target depth estimate to reduce interference from outliers. Subsequently, combining the intrinsic and extrinsic parameters of each camera, the 2D position and depth information of the target in the image are back-projected to calculate the 3D position of the target in the vehicle coordinate system. Since multiple cameras participate in detection, the same target may have observation results in different cameras. Therefore, multiple observation results can be fused during the 3D localization process, for example, by using weighted averaging or confidence-weighted methods to improve localization accuracy. Finally, the 3D coordinates of traffic lights, countdown lights, and traffic signs are output in a unified manner, providing accurate spatial perception information for the autonomous driving system.

[0041] Through the coordinated operation of the aforementioned modules, this implementation method achieves multi-view coverage, complementary target information, and high-precision spatial positioning during vehicle turning, effectively solving the perception blind spot problem caused by the limited field of view of a single camera, while improving the success rate of countdown recognition and the stability of traffic light status judgment. In complex intersection environments, even if some traffic lights are obscured or in a lateral position, the system can still obtain complete perception results through multi-camera fusion, thereby significantly improving the safety and reliability of the autonomous driving system in turning scenarios. Furthermore, through a reasonable camera selection strategy and multi-task fusion mechanism, the system effectively controls computational resource consumption while ensuring perception performance, giving the system good engineering application value and scalability.

[0042] Implementation Method 3: For example Figure 1 As shown, this embodiment provides a multi-camera collaborative perception system for complex lighting conditions and long-distance small target detection scenarios. The system consists of a camera selection module, a 2D target detection module, a fusion post-processing module, and a 3D position output module. These modules work collaboratively within a unified framework, forming a complete processing chain from perception data acquisition, target detection, information fusion to spatial positioning. This embodiment focuses on addressing complex environmental challenges such as small-sized traffic lights at long distances, drastic changes in lighting conditions, and the susceptibility of targets to occlusion. It improves detection accuracy and system robustness by introducing a multi-level optimization mechanism.

[0043] In this implementation, the camera selection module first acquires traffic light distribution information for distant intersections from a high-precision map and then selects a perception strategy based on the vehicle's current operating status. Since distant traffic lights are small in size in the image, they are prone to detection failure due to insufficient resolution. Therefore, when the vehicle is far from the target intersection, a forward-looking telephoto camera is prioritized. Figure 2As shown, this camera boasts a high pixel density, significantly improving the imaging quality of small targets at a distance. As the vehicle approaches the target intersection, the camera selection module dynamically adjusts its perception strategy based on distance changes. When the distance decreases to a certain range, a forward-looking wide-angle camera is gradually introduced to expand the field of view and prevent target loss due to limited field of view. In complex lighting environments, such as backlighting, strong light, or low-light nighttime scenes, the camera selection module can also dynamically adjust exposure parameters based on image brightness statistics or prioritize cameras with a stronger dynamic range, thereby ensuring stable input video stream quality.

[0044] After completing the video stream selection, the 2D object detection module processes the input image, such as... Figure 3 As shown, this module employs an edge-side real-time target detection algorithm to identify traffic lights, countdown lights, and traffic signs. In long-distance scenes, since traffic lights occupy only a small number of pixels in the image, traditional full-image detection methods are prone to missed detections. Therefore, this implementation introduces a spatial guidance mechanism based on a high-precision map. Specifically, the three-dimensional positions of traffic lights in the high-precision map are projected onto the image plane using camera intrinsic parameters to generate initial ROI regions. These ROI regions are then expanded proportionally to ensure sufficient coverage of potential small target areas, thereby improving the detection model's ability to perceive small targets.

[0045] Within the ROI area, the 2D object detection module performs a fine-grained detection process, identifying traffic light types, color states, and traffic sign categories, and outputting the corresponding 2D positions. When the detection results include countdown lights, the system crops the countdown area and extracts digital information using a digit recognition model, calculating the current countdown timer based on the digit arrangement. In complex lighting environments, the countdown numbers may exhibit uneven brightness or partial occlusion. To improve recognition accuracy, the cropped area can be preprocessed, such as through brightness normalization, contrast enhancement, or local noise reduction, thereby improving the digit recognition effect.

[0046] When no countdown light is detected within the ROI area, the system further calls the countdown area adaptive selection submodule, such as... Figure 4 As shown, candidate detection regions are constructed by analyzing the spatial distribution of traffic lights within the ROI area. In scenarios involving small targets at a distance, initial detection often fails to directly identify the countdown region. Therefore, a progressive expansion strategy is adopted, starting with the center of the traffic light to construct the initial search region, and gradually expanding the search range according to a preset ratio. After each expansion, the target detection operation is re-executed until the countdown region is detected or the maximum search range is reached. This strategy can effectively improve the recall rate of countdown detection, especially suitable for situations where the target size is small or partially occluded.

[0047] After two-dimensional detection, the fusion post-processing module integrates and optimizes the detection results from multiple frames. Since complex lighting conditions can cause fluctuations in the detection results over time—for example, the same traffic light may exhibit positional shifts or unstable state recognition in different frames—a Kalman filtering algorithm is introduced to predict and update the target position. Time series modeling smooths the target trajectory and reduces the impact of noise. Simultaneously, the detection results are constrained by the traffic light state switching logic, for example, by disallowing unreasonable state jumps within a short period, thereby improving the stability of the recognition results. In countdown processing, time consistency checks are performed on the countdown information in consecutive frames to correct abnormal readings and ensure that the countdown changes conform to the actual decreasing pattern.

[0048] In complex lighting and long-distance scenes, targets may be occluded or have low detection confidence. Therefore, the fusion post-processing module can also combine information from multiple frames for target compensation. For example, if a target is not detected in a certain frame, the tracking results of the preceding and following frames can be used to predict its position, thereby maintaining target continuity. In addition, multiple detection results can be fused by weighting based on confidence levels to improve overall perception reliability.

[0049] After the fusion process is completed, the 3D position output module performs spatial localization of the target. This module uses LiDAR to collect environmental point cloud data and generates a depth map aligned with the image. Under complex lighting conditions, visual information may be affected by interference, while the depth information provided by LiDAR has high stability. Therefore, fusing visual and LiDAR data can significantly improve positioning accuracy. In the specific implementation, for each detected target, the depth value set of the corresponding region is extracted from the depth map, and the target depth is obtained using the median calculation method, thereby reducing the influence of outliers. Subsequently, combined with camera calibration parameters, the two-dimensional image coordinates and depth information are back-projected to obtain the target's three-dimensional position in the vehicle coordinate system.

[0050] In long-range scenarios, due to the low resolution of depth information, single measurement results may contain errors. Therefore, multiple frames of depth data can be fused, and the accuracy of depth estimation can be improved through weighted averaging or filtering in the time dimension. Furthermore, even in the presence of occlusion, and despite the instability of visual detection, spatial compensation can still be performed using LiDAR point cloud data, thereby ensuring the continuity and accuracy of target localization.

[0051] This implementation utilizes a camera selection module to dynamically adjust perception strategies in long-distance and complex lighting environments, a 2D target detection module combined with high-precision map guidance to enhance small target detection capabilities, an adaptive countdown region selection mechanism to strengthen countdown recognition capabilities, a fusion post-processing module to optimize and stabilize multi-frame information output, and a 3D position output module to achieve high-precision spatial positioning. This constructs a multi-camera collaborative perception system with high robustness in complex environments. This system not only effectively addresses the challenge of detecting small targets at long distances but also maintains stable performance under conditions of drastic lighting changes or occlusion, demonstrating significant engineering practical value and potential for widespread application.

[0052] The foregoing has only described certain exemplary embodiments of this application by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of this application. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of this application.

Claims

1. A multi-camera collaborative traffic light and traffic sign multi-task perception system, characterized in that, It includes a camera selection module, a 2D target detection module, a fusion post-processing module, and a 3D position output module; The camera selection module calculates the distance between the vehicle's current position and the target intersection based on the location information of traffic lights at the intersection in the high-precision map and the real-time pose information of the vehicle. It then selects the camera with the corresponding field of view based on the comparison result between the distance and a preset threshold. When the vehicle approaches the intersection, it further determines whether to call the surround-view camera by combining the vehicle's turning state and the driving route in the high-precision map, and obtains the target camera video stream for subsequent processing. The 2D target detection module processes the video stream from the target camera, detects the type and 2D location information of traffic lights, countdown lights, and traffic signs, and projects the spatial location information of traffic lights at the corresponding intersections in the high-precision map onto the image plane through camera parameters to generate target areas. Within the target areas, the detection results are matched to determine the traffic lights of interest. When a countdown light is present in the target area, the countdown area is cropped and its digital information is identified. When no countdown light is detected in the target area, candidate areas are adaptively generated based on the spatial distribution of traffic lights to perform countdown detection again, obtaining 2D detection results of traffic light status, countdown information, and traffic sign information. The fusion post-processing module, based on the two-dimensional detection results, fuses traffic light, countdown light and traffic sign information from multiple cameras, and combines target tracking algorithm and state switching logic to update the traffic light state and countdown information in time sequence to obtain multi-target perception results; The 3D position output module performs depth mapping on the two-dimensional position information in the fused multi-target perception results based on the depth information generated by the LiDAR point cloud, calculates the spatial position of each target, and maps the target to a three-dimensional spatial coordinate system in combination with the camera's intrinsic and extrinsic parameters, outputting the three-dimensional position information of traffic lights, countdown lights, and traffic signs.

2. The multi-camera collaborative traffic light and traffic sign multi-task perception system according to claim 1, characterized in that, The camera selection module includes a multi-dimensional state determination unit. This unit combines the vehicle's latitude and longitude coordinates, heading angle change rate, and lateral acceleration to form a composite state vector. It then generates camera activation commands through a preset determination function. The activation conditions for telephoto cameras include the matching relationship between distance parameters and viewing angle coverage. The activation conditions for wide-angle cameras include constraints on the integrity of the field of view coverage. The activation conditions for surround-view cameras include lateral target coverage requirements. Through the joint constraints of multiple conditions, a dynamic camera scheduling mechanism is formed to achieve continuous switching of target perception capabilities in different spatial areas.

3. The multi-camera collaborative traffic light and traffic sign multi-task perception system according to claim 1, characterized in that, The two-dimensional target detection module includes a region generation unit guided by a high-precision map. The region generation unit performs projection calculations using the three-dimensional coordinates of traffic lights and camera intrinsic parameters to form multiple candidate projection regions. It then obtains a unified detection region through geometric envelope operations and performs proportional expansion processing on the region boundaries to construct an enhanced detection region containing potential small targets. Combined with the detection model, targets within the region are prioritized for identification.

4. A multi-camera collaborative traffic light and traffic sign multi-task perception system according to claim 3, characterized in that, The two-dimensional target detection module further includes a countdown recognition subunit. This subunit improves the digital region feature representation capability by cropping the area near the traffic light and performing multi-scale image preprocessing, including brightness normalization, contrast enhancement, and local noise reduction. It also parses the countdown sequence through digital sorting relationships to infer the remaining time of the traffic signal.

5. A multi-camera collaborative traffic light and traffic sign multi-task perception system according to claim 4, characterized in that, The two-dimensional target detection module includes an adaptive selection mechanism for the countdown region. This mechanism constructs candidate regions based on the spatial distribution of traffic lights. When there are multiple traffic lights, a minimum bounding region generation strategy is adopted. When the number of traffic lights is small, a dynamically expanding region centered on the traffic light center is constructed, and the countdown region is gradually approached through multiple region expansion and detection cycles.

6. A multi-camera collaborative traffic light and traffic sign multi-task perception system according to claim 1, characterized in that, The fusion post-processing module includes a multi-source target association unit. This unit constructs target matching relationships through spatial distance constraints and category consistency constraints, assigns unified labels to detection results from different cameras, and filters and fuses duplicate targets through a target confidence weight allocation mechanism to form a unique target representation set.

7. A multi-camera collaborative traffic light and traffic sign multi-task perception system according to claim 6, characterized in that, The fusion post-processing module includes a time series filtering unit, which uses a Kalman filtering model to estimate the state of traffic lights and traffic signs. It achieves target trajectory smoothing through prediction and update cycles, and combines an anomaly detection mechanism to remove data from abrupt change locations.

8. A multi-camera collaborative traffic light and traffic sign multi-task perception system according to claim 7, characterized in that, The fusion post-processing module further includes a state logic constraint unit. The state logic constraint unit restricts the color change path by setting the traffic light state transition rules, allowing only state transition relationships that conform to traffic rules. At the same time, it combines the countdown decrementing law to perform consistency verification on the results of consecutive frames.

9. A multi-camera collaborative traffic light and traffic sign multi-task perception system according to claim 1, characterized in that, The 3D position output module includes a depth estimation unit. The depth estimation unit generates a depth map by projecting the LiDAR point cloud and performs statistical processing on the set of depth values ​​in the target area. It uses the median estimation method to obtain stable depth results and combines multi-frame depth information for weighted fusion to improve spatial positioning accuracy.

10. A multi-camera collaborative traffic light and traffic sign multi-task perception system according to claim 9, characterized in that, The 3D position output module includes a multi-camera coordinate unification unit. This unit performs coordinate transformation through the calibration parameters of each camera, mapping the target position obtained from different perspectives to the vehicle coordinate system. It also performs spatial fusion processing based on the multi-view observation results, thereby obtaining a more consistent 3D target position representation.

Images