Traffic signal lamp state identification method, device, equipment, medium and program product
By using visual language models and thought chain reasoning mechanisms, detailed natural language text recognition results are generated, solving the problems of accuracy and interpretability in traffic light recognition in complex scenarios, and improving the recognition performance and user trust of autonomous driving systems.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NAVINFO
- Filing Date
- 2026-01-23
- Publication Date
- 2026-05-01
AI Technical Summary
In the field of autonomous driving, the accuracy, real-time performance, and interpretability of traffic light recognition in complex scenarios are challenging. In particular, traditional vision algorithms are prone to missed detections or false detections when there are multiple lights, obstructions, strong light, or interference from similar colored objects, and the output lacks interpretability.
By introducing the Visual Language Model (VLM) and the CoT reasoning mechanism, and through end-to-end training, it generates detailed recognition results in the form of natural language text, including a complete logical reasoning process of scene type inference, target traffic light screening, and light status analysis, simulating the cognitive process of human drivers.
It significantly improves the accuracy and interpretability of traffic light recognition in complex scenarios, enhances the vehicle's understanding of the traffic environment, and improves driving safety and decision-making accuracy.
Smart Images
Figure CN121963143A_ABST
Abstract
Description
Traffic signal light status recognition methods, devices, equipment, media, and program products Technical Field
[0001] This application relates to the field of computer vision technology, and in particular to a method, apparatus, device, medium and program product for recognizing the status of traffic lights. Background Technology
[0002] In the field of autonomous driving, vehicles need to perceive the status (red, yellow, green) and directional indications of surrounding traffic lights in real time to determine whether to stop, change lanes, or pass through intersections, thereby ensuring driving safety and efficiency. In the production of dynamic traffic data for maps, traffic light status recognition is a crucial step in constructing real-time traffic information maps.
[0003] However, real-world applications present complex and varied challenges: for example, when multiple traffic lights coexist, it's necessary to identify the traffic light that the current vehicle needs to focus on; differences in size between near and far traffic lights can lead to target detection and localization errors; traditional vision algorithms are prone to missed or false detections in obstructed (e.g., other vehicles or trees) or strong / backlight conditions; interference from similarly colored objects (e.g., billboards or illuminated decorations) can cause misjudgments of light states. Complex scenarios place stringent demands on the accuracy, real-time performance, and interpretability of traffic light recognition. Summary of the Invention
[0004] This application provides a traffic signal light status recognition method, apparatus, device, medium, and program product to improve the accuracy of traffic signal light recognition in complex scenarios and enhance the interpretability of the recognition results.
[0005] In a first aspect, embodiments of this application provide a traffic light status recognition method, including:
[0006] Acquire images from the vehicle's driving perspective;
[0007] Based on the image, a traffic light status recognition is performed using a preset visual language model to generate a traffic light status recognition result. The traffic light status recognition result is natural language text, which includes a logical reasoning process that derives the current scene type of the vehicle based on image features, a logical reasoning process that filters out target traffic lights related to driving decisions based on scene type conclusions, and a logical reasoning process that analyzes the current status of the target traffic light.
[0008] Secondly, embodiments of this application provide a traffic signal light status recognition device, comprising:
[0009] The acquisition module is used to acquire images from the vehicle's driving perspective.
[0010] The generation module is used to perform light status recognition based on the image and a preset visual language model to generate traffic light status recognition results. The traffic light status recognition results are natural language text, including a logical reasoning process to deduce the current scene type of the vehicle based on image features, a logical reasoning process to filter out target traffic lights related to driving decisions based on scene type conclusions, and a logical reasoning process to analyze the current status of the target traffic light.
[0011] Thirdly, embodiments of this application provide a traffic signal light status recognition device, including: a memory and a processor;
[0012] The memory stores computer-executed instructions;
[0013] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.
[0014] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.
[0015] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.
[0016] The traffic light status recognition method, apparatus, device, medium, and program products provided in this application achieve efficient recognition of traffic light status by analyzing images from the vehicle's driving perspective and utilizing a preset visual language model. This method can generate detailed recognition results in natural language text form. These results not only include scene type derivation and target traffic light selection, but also cover the complete logical reasoning process of light status analysis, thereby greatly enhancing the vehicle's understanding of the traffic environment and improving driving safety and decision-making accuracy. This method simulates the cognitive process of a human driver, leveraging the contextual reasoning capabilities of the visual language model to enable the model to combine global scene information (such as lane lines, vehicle positions, and intersection structures) for step-by-step reasoning, identifying traffic lights directly related to driving decisions, and determining their current status. This technology significantly improves the accuracy of traffic light recognition in complex scenarios. Attached Figure Description
[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0018] Figure 1 is a flowchart illustrating the traffic light status recognition method provided in this application.
[0019] Figure 2 is a flowchart illustrating the traffic light status recognition method provided in this application.
[0020] Figure 3 is a schematic diagram of the process of generating complete attribute description text provided in this application;
[0021] Figure 4 is a schematic diagram of the process of generating the screening reason description text and the light status description text provided in this application;
[0022] Figure 5 is a schematic diagram of the process of generating analytical reasoning process text and formatted answer text for model training provided in this application;
[0023] Figure 6 is a schematic diagram of the traffic light status recognition results generated based on the preset visual language model provided in this application;
[0024] Figure 7 is a schematic diagram of the traffic signal light status recognition device provided in this application;
[0025] Figure 8 is a schematic diagram of the traffic signal status recognition device provided in this application.
[0026] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concepts of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation
[0027] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.
[0028] Traffic light status recognition is a key task in autonomous driving and intelligent transportation systems. Its purpose is to detect the location of traffic lights (usually represented by bounding box coordinates) from images and identify their current state (e.g., red, yellow, green, or other states). This task is typically achieved using computer vision techniques, especially object detection models based on convolutional neural networks (CNNs), such as popular models like YOLOv5 and YOLOv8. There are generally two technical approaches to recognizing traffic light states:
[0029] The first approach is a two-stage scheme. First, an object detection model is used to identify traffic lights in the image and their locations (i.e., Regions of Interest) are marked with bounding boxes. These ROI regions are then extracted from the image and fed into a separate classification model for attribute classification to determine the light status. This scheme relies on accurate object detection and image cropping, breaking down the recognition task into two independent stages: localization and classification.
[0030] The second approach involves simultaneously identifying the geometric location and attributes of traffic lights within a unified model, achieving a one-step identification process. For example, in the YOLO network architecture, this can be achieved by adding a branch for light status classification, thus outputting the position and status of the traffic lights simultaneously within a single network.
[0031] Pure vision models typically focus on the visual features of images, but they have limitations in scene reasoning and semantic understanding, which may lead to the inability to accurately identify key traffic lights that influence current driving decisions in specific driving scenarios. Due to the diversity and complexity of scenes, traditional pure vision models face challenges in accurately identifying the light states and directions of traffic lights. In complex scenes with multiple traffic lights, obstructions, strong or backlighting, and interference from similarly colored objects (such as billboards), the object detection stage may experience missed detections, false detections, or inaccurate localization, which can lead to the failure of subsequent classification tasks.
[0032] Furthermore, traditional methods typically output only bounding box coordinates and category labels, lacking explanation of the reasoning process. This not only makes the results unintuitive for non-experts but also hinders natural interaction with systems that require detailed explanations to make advanced driving decisions. Therefore, to improve the reliability of autonomous driving systems and user trust, more advanced models are needed to provide detailed reasoning processes and richer output information.
[0033] The traffic light status recognition method provided in this application effectively solves the challenges faced by traditional pure visual models in recognizing traffic lights in complex scenes by introducing a Visual Language Model (VLM) and a Chain of Thought (CoT) reasoning mechanism. VLM possesses excellent visual understanding and contextual reasoning capabilities. It can comprehensively consider global scene information, such as the vehicle's current position, lane distribution, and intersection structure, thereby assisting in determining which traffic lights require attention in the current scene. By introducing the CoT reasoning mechanism, the VLM is trained to learn to reason step-by-step like a human. The model first understands the current driving scene, then locates the traffic lights that need attention, and finally determines their status. This step-by-step reasoning process allows the model to directly output the target traffic lights that the driver needs to pay attention to, rather than simply detecting and classifying them. This invention redefines the traffic light status recognition task as a text sequence generation problem, achieving end-to-end training and inference. This end-to-end solution avoids the propagation and accumulation of errors between different modules (such as object detection, image cropping, and attribute classification) in traditional solutions, thereby improving overall recognition performance. VLM outputs recognition results in the form of a readable text chain, which not only includes the final recognition results (such as light status and direction attributes), but also demonstrates the complete reasoning logic from scene understanding to target localization and state determination. This output, which includes the reasoning process, is more reliable, easier to integrate, and easier for users to understand and trust.
[0034] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.
[0035] Figure 1 is a flowchart illustrating the traffic light status recognition method provided in this application. As shown in Figure 1, the method includes:
[0036] S101. Acquire an image from the vehicle's driving perspective.
[0037] In this step, an onboard camera or other image capture device is used to acquire real-time images of the road ahead of the vehicle. These images reflect the vehicle's perspective while it is in motion, including various elements on the road such as traffic lights, road signs, other vehicles, pedestrians, and the condition of the road itself.
[0038] S102. Based on the image, combined with the preset visual language model, the traffic light status is recognized to generate traffic light status recognition results. The traffic light status recognition results are natural language text, including the logical reasoning process of inferring the current scene type of the vehicle based on image features, the logical reasoning process of selecting the target traffic light related to driving decision based on the scene type conclusion, and the logical reasoning process of analyzing the current status of the target traffic light.
[0039] In this step, a pre-defined Visual Language Model (VLM) is used to analyze the acquired images. Based on the extracted image features, the model infers the current scene type of the vehicle. This reasoning process helps determine which traffic lights might be relevant to the current driving decision. Based on the scene type conclusion, the model further filters out target traffic lights relevant to the driving decision. For example, when approaching an intersection, the model identifies the traffic lights at that intersection as targets. The model analyzes the selected target traffic lights and infers their current state, such as red, yellow, green, or other states. Finally, the model transforms the above reasoning process and results into natural language text, which describes the complete logical reasoning process from image feature extraction to scene type inference, and then to target traffic light selection and light state analysis.
[0040] The traffic light status recognition method provided in this application achieves efficient recognition of traffic light status by analyzing images from the vehicle's driving perspective and utilizing a preset visual language model. This method generates detailed recognition results in natural language text form, including not only scene type derivation and target traffic light selection, but also the complete logical reasoning process of light status analysis. This significantly enhances the vehicle's understanding of the traffic environment, improving driving safety and decision-making accuracy. This method simulates the cognitive process of a human driver, leveraging the contextual reasoning capabilities of the visual language model to enable the model to combine global scene information (such as lane lines, vehicle positions, and intersection structures) for step-by-step reasoning, identifying traffic lights directly related to driving decisions and determining their current status. This technique significantly improves the accuracy of traffic light recognition in complex scenarios.
[0041] Figure 2 is a flowchart illustrating the traffic light status recognition method provided in this application. As shown in Figure 2, this embodiment, based on the embodiment in Figure 1, provides a detailed description of the traffic light status recognition method, which includes:
[0042] S201. Acquire an image from the vehicle's driving perspective.
[0043] S202. Based on the image, combined with the preset visual language model, the traffic light status is recognized to generate traffic light status recognition results. The traffic light status recognition results are natural language text, including the logical reasoning process of inferring the current scene type of the vehicle based on image features, the logical reasoning process of selecting the target traffic light related to driving decision based on the scene type conclusion, and the logical reasoning process of analyzing the current status of the target traffic light.
[0044] In one possible implementation, the pre-defined visual language model is trained in the following manner:
[0045] Obtain the labeled dataset, which contains images from the vehicle's driving perspective and corresponding labeled data. The labeled data includes scene type labeled data and traffic light labeled data.
[0046] Based on scene type annotation data and traffic light annotation data, generate text for the analysis and reasoning process of the corresponding vehicle driving perspective image;
[0047] The model is trained using images from the vehicle's driving perspective and the corresponding text of the analysis and reasoning process to obtain a pre-defined visual language model.
[0048] In this embodiment, the labeled dataset contains images from the vehicle's driving perspective, reflecting the road conditions seen by the vehicle during travel. Each image is equipped with corresponding labeled data, including scene type labeled data and traffic light labeled data. Using the scene type labeled data and traffic light labeled data, a descriptive text describing the analysis and reasoning process is generated for each image. The vehicle driving perspective images and their corresponding analysis and reasoning process texts are used to train the model to obtain a preset visual language model. Through training, the model learns how to extract features from images and generate corresponding analysis and reasoning process texts. This text is the product of the model's analysis and understanding of the image content, reflecting the model's recognition results of the scene and traffic light status.
[0049] Specifically, 2000 images taken from a vehicle's perspective can be collected, covering different traffic scenarios. These images are then categorized and labeled into four types: Before a one-way intersection, Before a regular / large intersection, Passing an intersection, and In the middle of the road, generating scene type labeling data. "Before a one-way intersection" refers to a vehicle approaching a one-way intersection, meaning the vehicle can only travel in one direction through the intersection. In this case, traffic lights typically only control traffic flow in one direction, so the vehicle needs to recognize the traffic light status of the upcoming intersection to decide whether to stop or continue. "Before a regular / large intersection" refers to a vehicle approaching a regular or large intersection, which may have traffic flow in multiple directions, including straight, left-turn, and right-turn. In this complex traffic environment, the vehicle needs to recognize all relevant traffic lights, including those controlling straight and turning traffic, to make the correct driving decisions. "Passing an intersection" means the vehicle is currently crossing an intersection. This may mean the vehicle is in the middle of the intersection or just entering it. In this scenario, vehicles need to recognize the current traffic light status at the intersection to ensure safe passage. The road description refers to vehicles traveling in the non-intersection section of the road, meaning they are not near any intersection and therefore cannot see the traffic lights for the next intersection. In this case, vehicles may not need to focus on traffic lights but rather on other road users and signs.
[0050] Additionally, the collected images can be labeled with the locations and related attributes of traffic lights, generating traffic light annotation data. Rectangular bounding boxes can be used to identify the geometric location of each traffic light. This helps the model learn how to detect traffic lights from images. For each traffic light, its related attributes are labeled, such as type, orientation, relevance, pattern, direction of travel, light status, and attachments. Type refers to the type of traffic participant the traffic light is designed for. Orientation describes the direction the traffic light faces, which is crucial for determining the visibility and relevance of the light to vehicles. Relevance refers to whether the traffic light is relevant to the vehicle's current travel path and whether it is a light that the driver needs to pay attention to. For example, at multi-lane or multi-directional intersections, only the lights for specific directions are relevant to the current vehicle. Pattern refers to the patterns on the traffic light, such as digital countdowns, which provide additional information, such as indicating waiting time. Direction of travel refers to the direction of travel controlled by the traffic light, such as straight ahead, left turn, right turn, etc. This is crucial for vehicles to understand which directions of travel are permitted or prohibited under a specific traffic light. The light status describes the current state of the traffic light, such as red, yellow, green, or other special states (such as flashing yellow). The attachment refers to the object or structure to which the traffic light is attached, such as a pole, cantilever, or power line. This helps determine the physical location and installation method of the traffic light, and is helpful for its detection and identification.
[0051] In one possible implementation, based on scene type annotation data and traffic light annotation data, an analysis and reasoning process text corresponding to the vehicle driving perspective image is generated, which may specifically include the following steps:
[0052] Generate corresponding scene description text based on scene type annotation data;
[0053] Generate corresponding attribute description text for each traffic light in the image based on traffic light annotation data;
[0054] Based on the scene description text and attribute description text, select the traffic lights that are relevant to the decision and generate the corresponding description text of the reason for selection and the description text of the light status of the traffic lights that are relevant to the decision.
[0055] This is a text describing the analysis and reasoning process of generating corresponding vehicle driving perspective images based on scene description text, attribute description text, selection reason description text, and light status description text.
[0056] In this embodiment, scene description text is generated for each vehicle driving perspective image using scene type annotation data. Based on traffic light annotation data, a detailed attribute description text is generated for each traffic light in the image. This description text includes attributes such as the type, orientation, pattern, direction of indication, light status, and attached objects of the traffic light. Combining the scene description text and attribute description text, traffic lights relevant to the current driving decision are selected. Simultaneously, a selection rationale description text is generated, explaining why certain traffic lights are considered relevant to the decision. For the selected decision-relevant traffic lights, a text describing their current light status is generated. Finally, the scene description text, attribute description text, selection rationale description text, and light status description text are integrated to form a complete analysis and reasoning process text. This text not only describes the final recognition result but also demonstrates the complete logical reasoning process from image feature extraction to scene understanding, target traffic light selection, and light status analysis.
[0057] In one possible implementation, generating corresponding attribute description text for each traffic light in the image based on traffic light annotation data may specifically include the following steps:
[0058] For each traffic light, obtain its annotation attribute information from the traffic light annotation data;
[0059] Based on the acquired labeled attribute information, multi-dimensional attribute description text for traffic lights is generated; the multiple dimensions include one or more of type, installation method, orientation, pattern, and relevance.
[0060] The attribute description texts from multiple dimensions are concatenated to form the complete attribute description text for each traffic light.
[0061] Referring to Figure 3, which illustrates the process of generating complete attribute description text according to this application, the process first iterates through each traffic light instance in the image. For each traffic light, its index and annotation attribute information in the annotation dataset are obtained. Based on the obtained annotation attribute information, attribute description text for the traffic light is generated. This text covers multiple dimensions. The attribute description texts of each dimension are concatenated to form the complete attribute description text for each traffic light. For example, the generated complete attribute description text can be expressed as: "Traffic light {idx+1}: This is a traffic light of {type description}, {installation description}. It faces {orientation description} and displays {pattern description}. {relevance description}." This text format is not only clear and organized but also rich in information, providing a detailed and easy-to-understand description of the traffic light.
[0062] Furthermore, based on scene description text and attribute description text, traffic lights relevant to the decision can be filtered out, and corresponding filtering reason description text and light status description text for the decision-related traffic lights can be generated. Referring to Figure 4, which is a schematic diagram of the process of generating filtering reason description text and light status description text provided in this application, traffic lights relevant to the current driving decision are first filtered out. For the filtered decision-related traffic lights, a text describing their current state is generated. This description includes key information such as the color and pattern of the traffic light. For example, the generated light status description text can be "The color of the traffic light is red, and the pattern is currently unrecognizable." Furthermore, inference information can be generated using the light status description text, such as "Since the pattern is unrecognizable, the indicated direction is recorded as 'NA'." This inference process enhances the understanding of the traffic light status, enabling reasonable decisions to be made even when some information is incomplete. In addition, if the traffic light displays countdown numbers, a description related to the traffic light countdown can also be generated, and this key information can be expressed in natural language, providing the driver or autonomous driving system with more comprehensive traffic light status information.
[0063] In one possible implementation, the model is trained using vehicle driving perspective images and their corresponding analytical reasoning process text to obtain a preset visual language model, which may specifically include the following steps:
[0064] Generate formatted answer text based on the answer annotation data in the labeled dataset;
[0065] The model is trained using images from the vehicle's driving perspective, along with the corresponding analytical reasoning process text and formatted answer text, to obtain a pre-defined visual language model.
[0066] Referring to Figure 5, which illustrates the process of generating the analytical reasoning process text and formatted answer text for model training provided in this application, the process involves three steps using a labeled dataset: Step 1 generates scene description text, Step 2 generates attribute description text, and Step 3 generates selection reason description text and light status description text. These three generated texts are then merged to form a thought chain, i.e., the analytical reasoning process text. Simultaneously, formatted answer text is generated based on the answer annotation data in the labeled dataset. Finally, the analytical reasoning process text and formatted answer text (i.e., the thought chain dataset) are used together for model training to improve the accuracy and reliability of the model's traffic light status recognition. Supervised fine-tuning is performed on the model, with the training goal of enabling the model to learn to generate corresponding reasoning text sequences based on input images. The model needs to learn how to map image features to text descriptions. After supervised fine-tuning, the model initially possesses CoT reasoning capabilities for traffic light status recognition tasks. This means that the model can generate detailed analytical reasoning process text based on input traffic scene images, explaining how it identifies and understands the traffic light status.
[0067] In one possible implementation, the model is trained using vehicle driving perspective images and their corresponding analytical reasoning process text and formatted answer text to obtain a preset visual language model, which may specifically include the following steps:
[0068] The model is trained using images from the vehicle's driving perspective as input, and the model output is obtained.
[0069] The light state recognition conclusion in the model output is compared and verified with the formatted answer text used as the learning target to obtain the first verification result;
[0070] The consistency of the reasoning text in the model output with the analysis and reasoning process text used as the learning target is verified to obtain a second verification result.
[0071] Verify whether the model output contains predefined labels and fields to obtain a third verification result;
[0072] Based on the first, second, and third verification results, the model parameters are optimized and updated to obtain the preset visual language model.
[0073] In this implementation, the traffic light status recognition conclusion output by the model is compared with the formatted answer text used as the learning target. This comparison verifies the model's recognition accuracy, i.e., whether the model correctly identifies the traffic light status. The result of this step is called the first verification result. Next, the inference text output by the model is checked for consistency with the analysis and inference process text used as the learning target. This step evaluates whether the inference text generated by the model matches the expected inference logic, ensuring that the model can reasonably interpret its recognition results. The result of this step is called the second verification result. Finally, the model's output is checked to ensure that it contains all predefined labels and fields. This is to ensure that the model's output format is correct and contains all necessary information. The result of this step is called the third verification result. Based on the above three verification results, the model parameters are optimized and updated. This step aims to improve the model's performance, enabling it to provide more accurate, consistent, and complete outputs in future predictions. After such optimization, the final model is the preset visual language model.
[0074] Specifically, the improvement in model performance is achieved through a reward function that evaluates the overall quality of the text sequence Y generated by the model. This reward function R consists of three parts, each corresponding to an evaluation of a specific aspect of the model output:
[0075] R(Y) =λ1×R_accuracy +λ2×R_coherence +λ3×R_format;
[0076] Here, R_accuracy represents the accuracy reward, which is based on whether the model correctly identifies the state of the traffic light; R_coherence represents the consistency reward, which is based on whether the reasoning text generated by the model is logically sound, and can be evaluated using a small discriminator model or rules; R_format represents the format reward, which is based on whether the model's output format is correct, whether it contains the "think" and "answer" labels, and whether the "answer" part contains all the necessary fields; λ1, λ2, and λ3 are weight parameters used to adjust the contribution of each reward component to the total reward. In this way, the model is trained to generate text sequences that can obtain higher rewards, thereby incentivizing the model to output more accurate, logically coherent, and formatted results.
[0077] Referring to Figure 6, this is a schematic diagram of the traffic light state recognition result generated based on a preset visual language model provided in this application, including the thought process and the answer. First, the task objective is clarified: to identify the traffic lights that the driver should pay attention to, output the light color, and infer their direction. If the traffic light contains countdown numbers, the corresponding countdown numbers also need to be confirmed. The model breaks down the task into three steps. In step 1, the model analyzes that the current scene is located in front of a regular / large intersection, and the screen shows two opposing lanes, traffic lights, and a pedestrian crossing, therefore it is a regular / large intersection, not a one-way street. In step 2, the model analyzes each group of traffic lights one by one and determines their relevance to the current driving decision. In step 3, the model outputs the light color of the traffic light that needs attention and infers the direction. Finally, an answer text conforming to the predefined format requirements is generated, including the bounding box coordinates (bbox_2d) of the traffic light, the direction of indication (indications), and the state (state).
[0078] S203. Perform vehicle control or map updates based on traffic signal status recognition results.
[0079] In this step, based on the identified traffic light status, corresponding vehicle control operations can be performed. For example, if the identification indicates that the traffic light ahead is red, a stop operation will be automatically executed; if it is green, the vehicle can continue forward or accelerate; if it is yellow, it may need to slow down and prepare to stop. Traffic light status recognition results can also be used to update digital maps. For example, if a change in the status of a traffic light is detected (such as changing from red to green), this information can be uploaded to the map service provider's server in real time. By applying traffic light status recognition results to vehicle control and map updates, the user's driving experience can be significantly improved.
[0080] This invention redefines traffic light state recognition as a sequence generation problem. Utilizing a high-quality thought chain dataset and based on a visual language model, it significantly improves the model's end-to-end output capability through a training strategy combining supervised fine-tuning (SFT) and policy-based reinforcement learning (GRPO). The model can directly output the coordinates of traffic lights influencing driving decisions from a driving perspective, along with their light states and directions. It outputs the traffic light recognition results step-by-step in text form, demonstrating the results of the large model's end-to-end reasoning. This method effectively addresses the complexity of traffic light recognition in various scenarios, significantly improving the accuracy and generalization ability of traffic light state recognition.
[0081] During training and inference, the model is required not only to output the final result but also the intermediate inference steps at each step, mimicking the cognitive process of a human driver: first understanding the overall scene, then identifying the points of interest, and finally determining the state. The model can be applied in the vehicle, while its training can be conducted in the cloud.
[0082] By employing a training strategy that combines Supervised Fine-Tuning (SFT) with Policy Optimization-Based Reinforcement Learning (GRPO), the SFT stage enables the model to learn basic CoT inference patterns, while the GRPO stage uses a carefully designed reward function to finely calibrate and optimize the model's output, ensuring that the generated results are not only correct but also reliable and efficient. This training strategy not only improves the model's performance but also enhances its practicality and reliability in real-world applications.
[0083] The traffic light status recognition method provided in this application achieves efficient recognition of traffic light status by analyzing images from the vehicle's driving perspective and utilizing a preset visual language model. This method generates detailed recognition results in natural language text form, including not only scene type derivation and target traffic light selection, but also the complete logical reasoning process of light status analysis. This significantly enhances the vehicle's understanding of the traffic environment, improving driving safety and decision-making accuracy. This method simulates the cognitive process of a human driver, leveraging the contextual reasoning capabilities of the visual language model to enable the model to combine global scene information (such as lane lines, vehicle positions, and intersection structures) for step-by-step reasoning, identifying traffic lights directly related to driving decisions and determining their current status. This technique significantly improves the accuracy of traffic light recognition in complex scenarios.
[0084] Figure 7 is a schematic diagram of the traffic light status recognition device provided in this application. As shown in Figure 7, the traffic light status recognition device 70 provided in this embodiment includes:
[0085] Acquisition module 701 is used to acquire images from the vehicle's driving perspective;
[0086] The generation module 702 is used to perform light status recognition based on images and combined with a preset visual language model to generate traffic light status recognition results. The traffic light status recognition results are natural language text, including the logical reasoning process of inferring the current scene type of the vehicle based on image features, the logical reasoning process of filtering out target traffic lights related to driving decisions based on scene type conclusions, and the logical reasoning process of analyzing the current state of the target traffic lights.
[0087] In one possible implementation, a model training device is also included, which is used for:
[0088] Obtain the labeled dataset, which contains images from the vehicle's driving perspective and corresponding labeled data. The labeled data includes scene type labeled data and traffic light labeled data.
[0089] Based on scene type annotation data and traffic light annotation data, generate text for the analysis and reasoning process of the corresponding vehicle driving perspective image;
[0090] The model is trained using images from the vehicle's driving perspective and the corresponding text of the analysis and reasoning process to obtain a pre-defined visual language model.
[0091] In one possible implementation, the model training device is specifically used for:
[0092] Generate corresponding scene description text based on scene type annotation data;
[0093] Generate corresponding attribute description text for each traffic light in the image based on traffic light annotation data;
[0094] Based on the scene description text and attribute description text, select the traffic lights that are relevant to the decision and generate the corresponding description text of the reason for selection and the description text of the light status of the traffic lights that are relevant to the decision.
[0095] This is a text describing the analysis and reasoning process of generating corresponding vehicle driving perspective images based on scene description text, attribute description text, selection reason description text, and light status description text.
[0096] In one possible implementation, the model training device is specifically used for:
[0097] For each traffic light, obtain its annotation attribute information from the traffic light annotation data;
[0098] Based on the acquired labeled attribute information, multi-dimensional attribute description text for traffic lights is generated; the multiple dimensions include one or more of type, installation method, orientation, pattern, and relevance.
[0099] The attribute description texts from multiple dimensions are concatenated to form the complete attribute description text for each traffic light.
[0100] In one possible implementation, the model training device is specifically used for:
[0101] Generate formatted answer text based on the answer annotation data in the labeled dataset;
[0102] The model is trained using images from the vehicle's driving perspective, along with the corresponding analytical reasoning process text and formatted answer text, to obtain a pre-defined visual language model.
[0103] In one possible implementation, the model training device is specifically used for:
[0104] The model is trained using images from the vehicle's driving perspective as input, and the model's output is obtained.
[0105] The light state recognition conclusion in the model output is compared and verified with the formatted answer text used as the learning target to obtain the first verification result;
[0106] The consistency of the reasoning text in the model output with the analysis and reasoning process text used as the learning target is verified to obtain a second verification result.
[0107] Verify whether the model output contains predefined labels and fields to obtain a third verification result;
[0108] Based on the first, second, and third verification results, the model parameters are optimized and updated to obtain the preset visual language model.
[0109] In one possible implementation, the traffic light status recognition device is further used for:
[0110] Vehicle control or map updates are based on traffic light status recognition results.
[0111] The traffic light status recognition device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.
[0112] Figure 8 is a schematic diagram of the traffic light status recognition device provided in this application. As shown in Figure 8, the traffic light status recognition device 80 provided in this embodiment includes at least one processor 801 and a memory 802. Optionally, the device 80 further includes a communication component 803. The processor 801, memory 802, and communication component 803 are connected via a bus.
[0113] In a specific implementation, at least one processor 801 executes computer execution instructions stored in memory 802, causing at least one processor 801 to perform the above-described method.
[0114] The specific implementation process of processor 801 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.
[0115] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.
[0116] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.
[0117] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.
[0118] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.
[0119] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.
[0120] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.
[0121] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.
[0122] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.
[0123] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0124] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0125] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0126] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.
[0127] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.
Claims
1. A method for recognizing the status of traffic lights, characterized in that, include: Acquire images from the vehicle's driving perspective; Based on the image, a traffic light status recognition result is generated by combining a preset visual language model to identify the light status. The traffic light status recognition result is natural language text, which includes a logical reasoning process to deduce the current scene type of the vehicle based on image features, a logical reasoning process to filter out target traffic lights related to driving decisions based on scene type conclusions, and a logical reasoning process to analyze the current status of the target traffic light.
2. The method according to claim 1, characterized in that, The preset visual language model is trained as follows: A labeled dataset is acquired, comprising vehicle driving perspective images and corresponding labeled data, including scene type labeled data and traffic light labeled data; based on the scene type labeled data and the traffic light labeled data, analysis and reasoning process text corresponding to the vehicle driving perspective images is generated; the model is trained using the vehicle driving perspective images and their corresponding analysis and reasoning process text to obtain the preset visual language model.
3. The method according to claim 2, characterized in that, The step of generating analysis and reasoning process text for the corresponding vehicle driving perspective image based on the scene type annotation data and the traffic light annotation data includes: generating corresponding scene description text based on the scene type annotation data; generating corresponding attribute description text for each traffic light in the image based on the traffic light annotation data; filtering decision-related traffic lights based on the scene description text and the attribute description text, and generating corresponding filtering reason description text and light status description text for the decision-related traffic lights; and generating analysis and reasoning process text for the corresponding vehicle driving perspective image based on the scene description text, the attribute description text, the filtering reason description text, and the light status description text.
4. The method according to claim 3, characterized in that, The step of generating corresponding attribute description text for each traffic light in the image based on the traffic light annotation data includes: for each traffic light, obtaining its annotation attribute information in the traffic light annotation data; generating attribute description text for the traffic light in multiple dimensions based on the obtained annotation attribute information; the multiple dimensions include one or more of type, installation method, orientation, pattern, and relevance; and concatenating the attribute description text in multiple dimensions to form the complete attribute description text for each traffic light.
5. The method according to claim 3, characterized in that, The step of training the model using the vehicle driving perspective image and its corresponding analysis and reasoning process text to obtain the preset visual language model includes: generating formatted answer text based on the answer annotation data in the labeled dataset; and training the model using the vehicle driving perspective image and its corresponding analysis and reasoning process text and formatted answer text to obtain the preset visual language model.
6. The method according to claim 5, characterized in that, The step of training a model using the vehicle driving perspective image and its corresponding analysis and reasoning process text and formatted answer text to obtain the preset visual language model includes: training the model using the vehicle driving perspective image as input and obtaining the model output result; comparing and verifying the light status recognition conclusion in the model output result with the formatted answer text used as the learning target to obtain a first verification result; performing consistency verification between the reasoning text in the model output result and the analysis and reasoning process text used as the learning target to obtain a second verification result; verifying whether the model output result contains predefined labels and fields to obtain a third verification result; and optimizing and updating the model parameters based on the first verification result, the second verification result, and the third verification result to obtain the preset visual language model.
7. The method according to any one of claims 1-6, characterized in that, After generating traffic light status recognition results by performing light status recognition based on the image and combining it with a preset visual language model, the method further includes: performing vehicle control or map updating based on the traffic light status recognition results.
8. A traffic signal light status recognition device, characterized in that, include: The acquisition module is used to acquire images from the vehicle's driving perspective; The generation module is used to perform light status recognition based on the image and a preset visual language model to generate traffic light status recognition results. The traffic light status recognition results are natural language text, including a logical reasoning process to deduce the current scene type of the vehicle based on image features, a logical reasoning process to filter out target traffic lights related to driving decisions based on scene type conclusions, and a logical reasoning process to analyze the current status of the target traffic light.
9. A traffic signal light status recognition device, characterized in that, include: Memory, processor; The memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-7.
10. A computer-readable storage medium or computer program product, characterized in that, The computer-readable storage medium stores computer-executable instructions that, when executed by a processor, are used to implement the method as described in any one of claims 1-7; and / or, the computer program product includes a computer program that, when executed by a processor, implements the method as described in any one of claims 1-7.