Traffic light cross-view matching methods, devices, equipment, media, and procedures.

Abstract

This application provides a method, apparatus, device, medium, and program product for cross-view matching of traffic lights. The method includes: acquiring a driving view and a corresponding map view; inputting the image pair of the driving view and map view, along with text prompts, into a visual language model trained with thought chain instructions; explicitly constraining the model's output reasoning process and output format, enabling the model to directly extract effective information from the original views and match traffic light instances in different views accordingly. This process does not rely on view transformation, effectively preserving original image information, and through thought chain instructions, the model possesses fine-grained instance matching capabilities. Therefore, while avoiding information loss, it efficiently and accurately completes fine-grained instance-level traffic light matching, improving the accuracy and robustness of cross-view matching of traffic lights.

Description

Technical Field

[0001] This application relates to the field of computer vision, and in particular to a method, apparatus, device, medium, and program product for cross-view matching of traffic lights. Background Technology

[0002] With the rapid development of autonomous driving technology, cross-view traffic light instance matching has become one of the core requirements of intelligent transportation systems. In dynamic traffic data production scenarios, traffic light matching between the driving view and the map view is a key step in building real-time traffic status perception.

[0003] Currently, the main method for cross-view matching of traffic lights between driving view and map view is to train a deep neural network to convert the driving view into a bird's-eye view representation, and then achieve matching through feature point alignment. This method relies on a large amount of pixel-level labeled data and is prone to introducing geometric distortion of ray direction and loss of height information during the conversion process, resulting in a decrease in matching accuracy. Summary of the Invention

[0004] This application provides a method, apparatus, device, medium, and program product for cross-view matching of traffic lights, in order to improve the accuracy of cross-view matching of traffic lights.

[0005] In a first aspect, embodiments of this application provide a traffic light cross-view matching method, including:

[0006] Get the driving view and the corresponding map view, each containing at least one traffic light instance to be matched;

[0007] The preset text prompts and image pairs containing driving view and map view are input into a pre-trained visual language model; wherein, the visual language model is trained via a thought chain format instruction dataset, so that the visual language model performs step-by-step reasoning according to explicit constraints.

[0008] Receive and parse the inference results output by the visual language model to obtain the matching relationship between traffic light instances in the driving view and the map view.

[0009] In some possible implementations, step-by-step reasoning includes the following steps:

[0010] Based on image pairs, traffic light instances in each view are identified, and the identified traffic light attributes are recorded as text content.

[0011] Based on the text content, perform association matching between traffic light instances in the driving view and the map view.

[0012] In the above method, an object detection algorithm can be used to identify traffic light instances and convert the visual attributes of traffic lights in the driving view and map view into text descriptions. This can map the traffic light features of the two different views into a unified text semantic space, reduce the information loss of image conversion between different views, and use the powerful understanding of text features by the visual language model to accurately achieve cross-view feature matching.

[0013] In some possible implementations, the identified traffic light attributes include the shape and / or position of the traffic light instance in each view.

[0014] In the above method, shape can refer to the shape of the traffic light indicator pattern (such as a circular light or an arrow indicator), and position can refer to the relative position of the traffic light and the vehicle. By converting these two types of key visual attributes into structured text, the text description can have category distinction and spatial correspondence information, thereby aligning the key features of different views at the semantic level and significantly improving the accuracy and scene adaptability of traffic light cross-view matching.

[0015] In some possible implementations, the instruction dataset in the mind chain format contains multiple training samples, including system prompts, model inference mind chain labels, and image pairs containing driving view and map view.

[0016] The system prompts are used to construct multiple sequentially arranged reasoning steps; the model reasoning thought chain labels are used as truth values ​​for the reasoning steps.

[0017] In the above approach, by constructing a training dataset containing structured system prompts and corresponding truth labels of the thought chain, a clear reasoning paradigm can be set for the model, guiding the model to decompose the complex cross-view matching task into multiple single-step sub-tasks with a fixed order. This can reduce the randomness of the model's reasoning and improve the stability and interpretability of traffic light instance matching.

[0018] In some possible implementations, multiple sequentially arranged inference steps are constructed, including defining the inference goal and output format of the inference steps.

[0019] In the above approach, by defining the reasoning objectives and output formats for each reasoning step, the output of each step in the model's step-by-step reasoning matching of cross-view traffic light instances conforms to a standardized preset format. This ensures the accuracy and consistency of information transmission between steps and avoids the final matching failure due to ambiguity or format deviation in the output of intermediate steps, thereby improving the stability and reliability of the entire matching process.

[0020] In some possible implementations, multiple sequentially arranged reasoning steps include:

[0021] Analyze and output the first orientation information of traffic light instances in the driving view relative to the shooting angle, and analyze and output the second orientation information of traffic light instances in the map view relative to the reference point;

[0022] Based on the first location information, the second location information, and the attributes of the traffic light instances, determine the matching relationship between the traffic light instances in the driving view and the map view.

[0023] In the above method, by limiting the relative position of traffic light instances in the view, the problem of inconsistent spatial reference systems in cross-view matching can be solved, and a simpler and more effective spatial association logic can be constructed.

[0024] In some possible implementations, the training process of a visual language model includes:

[0025] The visual language model to be trained is selected as the initial model;

[0026] The initial model is trained using a thought chain format instruction dataset until the model generates reasoning process text that conforms to the thought chain format, and the correct traffic light instance matching result is obtained based on the reasoning process text.

[0027] In the above approach, selecting a visual language model for training can reduce training difficulty by utilizing the multimodal processing capabilities of the visual language model itself. Furthermore, by enabling the model to generate inference process text during training, the model's cross-view matching decisions for traffic light instances can become traceable, thereby improving the transparency of the model's inference process.

[0028] Secondly, embodiments of this application provide a traffic light cross-view matching device, comprising:

[0029] The acquisition module is used to acquire the driving view and the corresponding map view. Each of the driving view and the map view contains at least one traffic light instance to be matched.

[0030] The input module is used to input preset text prompts and image pairs containing driving view and map view into a pre-trained visual language model; wherein, the visual language model is trained via a thought chain format instruction dataset so that the visual language model performs step-by-step reasoning according to explicit constraints.

[0031] The output module is used to receive and parse the inference results output by the visual language model to obtain the matching relationship between traffic light instances in the driving view and the map view.

[0032] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0033] The memory stores instructions that the computer executes;

[0034] The processor executes computer execution instructions stored in memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0035] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0036] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0037] The traffic light cross-view matching method, apparatus, device, medium, and program products provided in this application acquire a driving view and a corresponding map view. The image pairs of the driving view and map view, along with text prompts, are input into a visual language model trained with thought chain instructions. Explicit constraints can be placed on the model's inference process and output format, enabling the model to directly extract effective information from the original views and match traffic light instances in different views accordingly. This process does not rely on view transformation, effectively preserving original image information. Furthermore, thought chain instructions enable the model to possess fine-grained instance matching capabilities, thereby efficiently and accurately completing fine-grained instance-level traffic light matching while avoiding information loss, thus improving the accuracy and robustness of cross-view traffic light matching. Attached Figure Description

[0038] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0039] Figure 1 A flowchart illustrating a traffic light cross-view matching method provided as an example in this application;

[0040] Figure 2 A schematic diagram of an image pair including a driving view and a map view is provided as an example of this application;

[0041] Figure 3 A schematic diagram illustrating a model-based matching application process is provided as an example for this application;

[0042] Figure 4 A schematic diagram of the structure of a traffic light cross-view matching device is provided as an example in this application;

[0043] Figure 5 This is a schematic diagram of the structure of an electronic device provided as an example in this application.

[0044] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0045] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0046] Below, we will first explain some of the terms used in this application.

[0047] Driving view: refers to the horizontal view of the road captured by onboard cameras and other equipment, dynamically reflecting the scene around the vehicle, and is the direct data source for traffic light sensing.

[0048] BEV (Bird's-Eye-View): refers to a shooting perspective that is directly shooting vertically downwards from above, or a perspective that is represented by converting forward-looking or other images into a vertically downward-looking view through algorithms.

[0049] Map view: refers to a fixed-view image (mostly a top-down view of the BEV map) obtained from high-precision maps, which provides global and precise location and layout information of road elements such as traffic lights.

[0050] Traffic light example: refers to a single, specific traffic light entity that can be independently detected and identified in an image.

[0051] Visual language model: refers to a deep learning model that can process image and text input simultaneously, understand their semantic relationships, and generate text responses.

[0052] With the rapid development of autonomous driving technology, cross-view traffic light instance matching has become one of the core requirements of intelligent transportation systems. Cross-view traffic light instance matching, especially for driving views and map views, is crucial for traffic light change timing and countdown prediction, as well as autonomous driving and other fields.

[0053] Currently, cross-view matching of traffic lights mainly employs feature point matching and view transformation to train deep neural networks for camera pose estimation, bridging the domain differences between the driving view and the map view. Feature point matching primarily utilizes contrastive learning to map the image features of the driving view and the map view to the same representation space, converting the driving view image into a BEV (bird's-eye view) representation. Then, by matching feature points between this BEV image and the map view, the relative pose is estimated, thereby locating the traffic light instance.

[0054] The aforementioned feature-point matching-based methods suffer from the following drawbacks. First, these methods heavily rely on a large amount of pixel-level labeled data to train the pose estimation network, but such labels are often difficult to obtain in practical applications, limiting the model's generalization ability. Second, the process of converting the driving view image to a BEV (Battery Electric Vehicle) image is not lossless; it introduces geometric distortion along the ray direction and loses information along the height dimension, leading to a decrease in BEV image quality and consequently affecting its alignment accuracy with the map view image. This problem is particularly prominent in complex scenarios such as unknown or dynamically changing camera extrinsic parameters, severely impacting the accuracy and robustness of target matching.

[0055] Furthermore, deep learning-based view transformation methods typically employ non-weight-sharing pseudo-Twin networks to extract image features from different views and design effective domain adaptation strategies to reduce view differences. This approach is limited to image-level or category-level information extraction and is difficult to apply to instance-level matching within the same category, such as traffic lights. It cannot distinguish which real traffic light corresponds to a specific traffic light instance in the driving view, thus failing to meet the practical application needs of dynamic traffic data production and traffic light countdown prediction.

[0056] Based on this, a technical concept is proposed: constructing a command dataset in the form of a thought chain, and explicitly constraining the model to perform step-by-step reasoning (such as traffic light orientation analysis and attribute matching), enabling the model to learn the logical chain from detection to instance-level matching. For example, in the command dataset, the model is required to analyze the orientation, type, and direction of traffic lights step by step, and finally output instance-level matching results (e.g., which traffic light in the driving view matches which traffic light in the map view).

[0057] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0058] Figure 1 This application provides an exemplary flowchart of a traffic light cross-view matching method, as shown below. Figure 1 As shown, the method includes:

[0059] Step S101: Obtain the driving view and the corresponding map view.

[0060] Each of the driving view and the map view contains at least one traffic light instance to be matched.

[0061] In this embodiment of the application, the vehicle can be captured in real time by a dashcam in the driving view, and the vehicle's latitude and longitude coordinates, heading angle and other pose information can be recorded. Based on the pose information, the real geographic coordinates, orientation, type and status of traffic lights in the area can be obtained from the map database, and a top view of the map centered on the vehicle's position can be generated.

[0062] Step S102: Input the image pair containing the driving view and the map view, along with the preset text prompts, into the pre-trained visual language model.

[0063] The visual language model is trained on a thought chain-format instruction dataset and is subjected to explicit constraints to perform step-by-step reasoning.

[0064] In this embodiment of the application, an open-source visual language model (such as Qwen3-VL) can be selected as the initial model. Image pairs of driving view images and corresponding map view images under different traffic scenarios are collected in advance, and instructions in the form of thought chains are generated. The initial model is trained using image pairs and instructions as training samples, so that the model learns the reasoning process in the form of thought chains.

[0065] After obtaining the trained visual language model, it can be deployed in vehicles or the cloud. The driving view images collected in real time by the vehicle's dashcam are paired with the map view images and then fed into the visual language model. It can support the synchronous input of text prompts with a format similar to the instructions in the training samples based on data sources such as user voice.

[0066] Step S103: Receive and parse the inference results output by the visual language model to obtain the matching relationship between traffic light instances between the driving view and the map view.

[0067] Based on the input image pairs and text prompts, the visual language model can output inference results that include a step-by-step reasoning process and matching relationships between traffic light instances. Application modules such as vehicle navigation systems can obtain the inference results from the visual language model and implement practical application requirements such as traffic light countdown prediction based on the matching relationships between traffic light instances.

[0068] In the above embodiments, a driving view and the corresponding map view are obtained. The image pairs of the driving view and the map view, along with text prompts, are input into a visual language model trained with thought chain instructions. Explicit constraints can be placed on the model's inference process and output format, enabling the model to directly extract effective information from the original views and match traffic light instances in different views accordingly. This process does not rely on view transformation, effectively preserving the original image information. Furthermore, the thought chain instructions enable the model to possess fine-grained instance matching capabilities, thereby efficiently and accurately completing fine-grained instance-level traffic light matching while avoiding information loss, thus improving the accuracy and robustness of cross-view traffic light matching.

[0069] In one embodiment, step-by-step reasoning includes the following steps:

[0070] Traffic light instances in each view are identified based on image pairs, and the identified traffic light attributes are recorded as text content. Based on the text content, association matching is performed between traffic light instances in the driving view and the map view.

[0071] The identified traffic light attributes include the shape and / or position of the traffic light instance in each view.

[0072] Shape can refer to the shape of the traffic light's indicator pattern (such as a circular light or an arrow indicator), and position can refer to the relative position of the traffic light and the vehicle. By transforming these two key visual attributes into structured text, the text description can possess category differentiation and spatial correspondence information, thereby aligning the key features of different views at the semantic level and significantly improving the accuracy and scene adaptability of cross-view matching of traffic lights.

[0073] In some possible implementations, the location of a traffic light instance can include its relative orientation to a vehicle, such as being to the left or right front of the vehicle.

[0074] In this embodiment, the visual language model parses the driving view and the map view respectively, and converts the visual features of each traffic light instance (such as the relative position in the driving view, the coordinate orientation in the map view, and the shape of the light body) into a structured text description. Then, the model performs semantic-based logical reasoning by comparing the text descriptions of traffic light attributes between the two views (such as positional consistency and shape similarity), thereby completing instance-level association matching without relying on direct conversion of geometric coordinates.

[0075] Figure 2 This application provides an exemplary schematic diagram of an image pair including a driving view and a map view.

[0076] Reference Figure 2To determine the relative position of a traffic light instance in the driving view, the pixel coordinates of the traffic light instance in the driving view can be identified first using open-source object detection algorithms (such as YOLO or GroundingDINO), and then the relative positional relationship between the traffic light instance and the vehicle can be further analyzed by a visual language model.

[0077] To determine the coordinates and orientation of a traffic light instance in the map view, the vehicle orientation can be determined first in the map view based on the vehicle's pose information, such as its steering angle, and then the relationship between the traffic light instance's orientation and the vehicle's orientation can be determined based on the map database.

[0078] like Figure 2 As shown, in this image pair, the left image is the view from the vehicle's perspective (such as the view from a dashcam). You can see a set of traffic light instances in the left front of this view (the green box represents the detection bounding box of the object detection algorithm), and another set of traffic light instances in the right front. The right image of the image pair is the map view generated from the map data. This image represents the vehicle's position and driving perspective (if using a dashcam in front of the vehicle, the perspective is aligned with the vehicle's direction), and also shows the traffic light instances around the vehicle (e.g., lights 0, 1, and 2 in this image represent one set of traffic light instances, and lights 3 and 4 represent another set). This image can also include color indicators for different traffic light types; for example, pink lights (lights 1, 2, and 4) can represent circular traffic lights, and orange-yellow lights (lights 0 and 3) can represent left-turn arrow lights.

[0079] In the above embodiments, prior information such as traffic light detection boxes, types, and light states output by the target detection algorithm is introduced to enable the model to adapt to fine-grained instance-level matching tasks. By converting the visual attributes of traffic lights (such as shape and position) into text descriptions and combining them with the relative positional relationship between traffic lights and vehicles, the matching process can be transformed from traditional geometric spatial calculations to semantic-level logical reasoning, effectively avoiding geometric distortion and information loss in BEV projection. At the same time, the robustness of matching is enhanced by comprehensive judgment of multiple attributes, especially in complex scenes (such as partial occlusion, viewpoint differences, etc.) to still achieve accurate traffic light instance-level association.

[0080] In one embodiment, the instruction dataset in the mind chain format contains multiple training samples, including system prompts, user prompts, model inference mind chain labels, and image pairs containing driving views and map views.

[0081] The system prompts are used to construct multiple sequentially arranged reasoning steps; the model reasoning thought chain labels are used as truth values ​​for the reasoning steps.

[0082] In this embodiment of the application, the system prompt can provide a preset, fixed global instruction framework. It constructs a standardized logical flow for the model to process traffic light matching tasks by clearly defining multiple reasoning steps that must be executed in sequence (e.g., first analyze the location, then determine the direction, and finally perform the matching), thereby ensuring the structured and consistent nature of the model's reasoning process.

[0083] User prompts are specific operation instructions issued for the currently input image within the framework of system prompts. Their function is to guide the model's general processing capabilities to this specific matching task instance, and they can be concise task trigger statements.

[0084] Model reasoning chain labels are standard text outputs in the training data that correspond to image pairs and prompt words. Their content follows the steps of the system prompt words, fully demonstrating each step of the correct reasoning process and intermediate results from initial observation to final traffic light matching. They serve as supervisory labels during model training, guiding the model to learn and generate outputs with the same logical structure.

[0085] In the above embodiments, the construction of a thought chain-formatted instruction dataset aims to improve the traditional model reasoning method from image input to result output to a thought chain reasoning method from image input to guided reasoning process and then to result output. This transforms traffic light instance information from different views into the same text semantic space, reducing information loss during view-to-view conversion and improving matching accuracy. Furthermore, adding prompt words to the training samples sets a clear reasoning paradigm for the model, guiding it to decompose the complex cross-view matching task into multiple single-step sub-tasks with a fixed order. This reduces the randomness of model reasoning and improves the stability and interpretability of traffic light instance matching.

[0086] In one embodiment, multiple sequentially arranged inference steps are constructed, including defining the inference goal and output format of the inference steps.

[0087] In this embodiment, by defining the reasoning objective and output format for each reasoning step, the output of each step in the model's multi-step reasoning matching of cross-view traffic light instances conforms to a standardized preset format. For example, when determining the traffic light's location in the previous step, the model can represent it using one of the directions (front, back, left, right) from the vehicle's perspective, or it can represent it using one of the directions (east, west, south, north) from the vehicle's perspective. By limiting the objective and output format of the model's reasoning steps, format deviations can be avoided in the intermediate processing results of multiple steps. This not only ensures the accuracy and consistency of information transmission between steps but also improves reasoning efficiency, thereby enhancing the stability and reliability of the entire matching process.

[0088] For example, multiple sequentially arranged reasoning steps include:

[0089] S301, analyze and output the first orientation information of the traffic light instance in the driving view relative to the shooting angle, and analyze and output the second orientation information of the traffic light instance in the map view relative to the reference point.

[0090] S302, based on the first location information, the second location information, and the attributes of the traffic light instance, determine the matching relationship between the traffic light instances in the driving view and the map view.

[0091] The table below provides an example containing system prompts, user prompts, and model inference thought chain labels for training samples.

[0092]

[0093] The model output matching results can be in formats such as JSON. For example, if the model is set to output matching results in JSON format, step 4 of the above system prompt can also include the following prompt: ### Prompt 1: Matching result output format: <answer> [{\"bbox_st\": [xx, xx, xx, xx], \"num_map\": 2}, {\"bbox_st\":[yy, yy, yy, yy], \"num_map\": 3}, {\"bbox_st\": [zz, zz, zz, zz], \"num_map\": -1}]< / answer> ###Tip 2: bbox_st represents the coordinates of the detection box for the light in the dashcam's view, and num_map represents the number of the matched light in the map's top-down view. When a match fails, num_map is -1. These tips can further constrain the model's output format, facilitating its use by navigation and other software or services.

[0094] It should be noted that although images are not shown in the table above, the training samples may include image pairs of map views and driving views. Image pairs can be referenced. Figure 2 The image may contain traffic light instances to be matched.

[0095] In the above embodiments, by constructing reasoning steps that include orientation analysis, attribute comparison, and relationship determination, the cross-view matching task can be decomposed into a standardized process that is executed sequentially. This constrains the model to perform ordered and discrete logical processing, effectively reducing overfitting during model training, ensuring that each matching decision of the model is based on verified intermediate conclusions, and improving the interpretability of the matching process and the credibility of the matching results.

[0096] In one embodiment, the training process of a visual language model includes:

[0097] A visual language model to be trained is selected as the initial model; the initial model is trained using a thought chain format instruction dataset until the model generates reasoning process text that conforms to the thought chain format, and the correct traffic light instance matching result is obtained based on the reasoning process text.

[0098] In this embodiment, an open-source visual language model (such as Qwen3-VL) can be selected as the initial model, and full fine-tuning can be performed using the instruction dataset constructed in the above embodiments. Optionally, the model hyperparameters (such as maximum sequence length, image embedding size, etc.) can be adjusted according to task requirements, and high-performance training / inference frameworks such as DeepSpeed ​​and vLLM can be introduced to improve efficiency. In addition, evaluation metrics for matching tasks can be defined to monitor the model training effect.

[0099] In the above embodiments, selecting a visual language model for training leverages its multimodal processing capabilities, reducing training difficulty. Furthermore, by generating reasoning process text during training, the model's cross-view matching decisions for traffic light instances become traceable, enhancing the transparency of the reasoning process. Fine-tuning the visual language model using a thought chain-formatted instruction dataset transforms manually defined logical judgment processes (such as locating, analyzing attributes, and then associating) into the model's inherent processing paradigm. This significantly improves the verifiability and generalization ability of the model's output in complex real-world traffic scenarios, ultimately achieving high-precision instance-level matching of traffic lights without relying on geometric view transformations.

[0100] Figure 3 This application provides an exemplary schematic diagram of a model-based matching application process, which is illustrated below in conjunction with... Figure 3 The specific application of the method in this application will be explained.

[0101] like Figure 3As shown, this process involves modules and corresponding processing steps such as data preprocessing, object detection, instruction fine-tuning dataset construction, model fine-tuning, and inference matching. The data preprocessing module can include a dashcam device and a proprietary map database for collecting dashcam video and map data. Based on this data, video decoders and image processing tools are used to extract frames from the dashcam video and generate map view images. The object detection module uses an open-source open-set object detection model (such as GroundingDINO) to detect traffic lights in the dashcam view images, outputting basic information such as detection box coordinates, type, and light status for instruction dataset generation. The instruction dataset construction module builds training samples based on the thought chain format. This module includes prompt word generation templates and traffic light basic information embedding. The model fine-tuning module uses an open-source visual language model (such as Qwen3-VL) and performs full fine-tuning using the instruction fine-tuning dataset. This module integrates the DeepSpeed ​​framework to accelerate training. The inference matching module receives dashcam and map view images and directly outputs traffic light matching results, including instance-level correspondences, through the fine-tuned model. This module integrates the vLLM framework to accelerate batch model inference. The modules are connected via data streams: the data preprocessing module outputs dual-view image pairs and basic traffic light information; the object detection module extracts the location and attributes of the traffic lights; the instruction fine-tuning dataset construction module generates training data; the model fine-tuning module produces the optimized model; and the inference matching module completes the final application.

[0102] Below is a specific example of the above application:

[0103] The first step involves collecting driving video using devices such as a dashcam and GPS sensors, while also recording the vehicle's latitude and longitude coordinates, orientation angle, and other information. Keyframes containing traffic lights are extracted from the video, and the images are cropped and scaled to a uniform resolution. Using a proprietary map database, a map top-down view image and a dashcam view image are generated from the ground truth values ​​of traffic lights within a 200-meter radius of the vehicle's location and a 90-degree viewing angle.

[0104] The second step involves applying the pre-trained target detection model GroundingDINO to perform target detection on the aforementioned driving view image, and outputting key information such as the coordinates, type, and status of the traffic light detection box.

[0105] The third step involves constructing a thought chain format dataset based on the above image pairs and basic information. This dataset includes system prompts, user prompts, the model's output thought chain content, and dual-view images. During the construction process, explicit constraints are applied to the model's output thought process and format. The total number of samples in the dataset can be 2600.

[0106] The fourth step involves using the open-source visual language model Qwen3-VL-4B as the base model and performing full fine-tuning on the dataset using the instructions described above. Hyperparameter settings include a maximum truncation length of 8192, a maximum image pixel count of 16,000,000, a maximum token count of 2048, an initial learning rate of 1e-5, a cosine decay learning rate strategy, a warm-up step ratio of 0.1, a batch size of 1, and a gradient accumulation step count of 4. The DeepSpeed ​​ZeRO-3 optimizer is used, and the model is trained for 3 epochs on 4 PPU-ZW810E GPUs. The visual encoder and multimodal fusion module are then frozen.

[0107] The fifth step involves using a new driving view and map view image pair, along with corresponding text prompts, and leveraging vLLM to accelerate model inference. The model can directly output the traffic light matching results between the two views in JSON format. The entire process is completely end-to-end, requiring no view conversion or other operations, and efficiently and accurately completes fine-grained instance-level traffic light matching.

[0108] Figure 4 A schematic diagram of a traffic light cross-view matching device provided in this application is shown below. Figure 4 As shown, the traffic light cross-view matching device 400 provided in this embodiment includes:

[0109] The acquisition module 401 is used to acquire the driving view and the corresponding map view, each of which contains at least one traffic light instance to be matched;

[0110] The input module 402 is used to input preset text prompts and image pairs containing driving view and map view into a pre-trained visual language model; wherein the visual language model is trained via a thought chain format instruction dataset so that the visual language model performs step-by-step reasoning according to explicit constraints.

[0111] Output module 403 is used to receive and parse the inference results output by the visual language model to obtain the matching relationship between traffic light instances in the driving view and the map view.

[0112] In some possible implementations, the input module 402 may include a step-by-step guidance unit, which can be used to enable the visual language model to perform the following steps in steps: identify traffic light instances in each view based on image pairs, record the identified traffic light attributes as text content; and perform association matching between traffic light instances in the driving view and the map view based on the text content.

[0113] In some possible implementations, the input module 402 may include an attribute recognition unit, which can be used to enable the visual language model to recognize the shape and / or position of the traffic light instance in the view.

[0114] In some possible implementations, the input module 402 may also include a sample construction unit, which can be used to construct an instruction dataset in the form of a thought chain. This instruction dataset contains multiple training samples, including system prompts, model inference thought chain labels, and image pairs containing driving views and map views. The system prompts are used to construct multiple sequentially arranged inference steps, and the model inference thought chain labels are used as truth values ​​for the inference steps.

[0115] In some possible implementations, the input module 402 may also include a step constraint unit, which can be used to define the reasoning objective and output format of the reasoning steps when multiple sequentially arranged reasoning steps are constructed using system prompts.

[0116] In some possible implementations, the input module 402 may also include a step-by-step reasoning unit, which can be used to perform the following steps in steps: analyze and output the first orientation information of the traffic light instance in the driving view relative to the shooting angle, and analyze and output the second orientation information of the traffic light instance in the map view relative to the reference point; determine the matching relationship of the traffic light instance between the driving view and the map view based on the first orientation information, the second orientation information and the attributes of the traffic light instance.

[0117] In some possible implementations, the input module 402 may also include a model training unit, which can be used to: select a visual language model to be trained as an initial model; train the initial model using a thought chain format instruction dataset until the model generates reasoning process text that conforms to the thought chain format, and obtain the correct traffic light instance matching result based on the reasoning process text.

[0118] The traffic light cross-view matching device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0119] Figure 5 This is a schematic diagram of the structure of an electronic device provided in this application. Figure 5 As shown, the electronic device 50 provided in this embodiment includes at least one processor 501 and a memory 502. Optionally, the device 50 further includes a communication component 503. The processor 501, memory 502, and communication component 503 are connected via a bus 504.

[0120] In a specific implementation, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to perform the above-described method.

[0121] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0122] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0123] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0124] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0125] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0126] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0127] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0128] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0129] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0130] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0131] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0132] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0133] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0134] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A method for cross-view matching of traffic lights, characterized in that, include: Obtain a driving view and a corresponding map view, each of which contains at least one traffic light instance to be matched; The preset text prompts and image pairs containing the driving view and the map view are input into a pre-trained visual language model; wherein the visual language model is trained via a thought chain format instruction dataset so that the visual language model performs step-by-step reasoning according to explicit constraints. Receive and parse the inference results output by the visual language model to obtain the matching relationship between traffic light instances between the driving view and the map view.

2. The method according to claim 1, characterized in that, The step-by-step reasoning includes the following steps: Based on the image pairs, traffic light instances in each view are identified, and the identified traffic light attributes are recorded as text content. Based on the text content, perform association matching between traffic light instances in the driving view and the map view.

3. The method according to claim 2, characterized in that, The identified traffic light attributes include the shape and / or position of the traffic light instance in each view.

4. The method according to any one of claims 1 to 3, characterized in that, The instruction dataset in the mind chain format contains multiple training samples, which include system prompts, model inference mind chain labels, and image pairs containing driving view and map view. The system prompts are used to construct multiple sequentially arranged reasoning steps; the model reasoning thought chain labels are used as the truth values ​​of the reasoning steps.

5. The method according to claim 4, characterized in that, The construction of multiple sequentially arranged reasoning steps includes defining the reasoning objective and output format of the reasoning steps.

6. The method according to claim 4, characterized in that, The multiple sequentially arranged reasoning steps include: Analyze and output the first orientation information of the traffic light instance in the driving view relative to the shooting angle, and analyze and output the second orientation information of the traffic light instance in the map view relative to the reference point; Based on the first location information, the second location information, and the attributes of the traffic light instance, the matching relationship between the traffic light instances in the driving view and the map view is determined.

7. The method according to any one of claims 1 to 3, characterized in that, The training process of the visual language model includes: The visual language model to be trained is selected as the initial model; The initial model is trained using the instruction dataset in the mind chain format until the model generates reasoning process text that conforms to the mind chain format, and the correct traffic light instance matching result is obtained based on the reasoning process text.

8. A traffic light cross-view matching device, characterized in that, include: The acquisition module is used to acquire a driving view and a corresponding map view, wherein each of the driving view and the map view contains at least one traffic light instance to be matched; An input module is used to input preset text prompts and image pairs containing the driving view and the map view into a pre-trained visual language model; wherein the visual language model is trained via a thought chain format instruction dataset so that the visual language model performs step-by-step reasoning according to explicit constraints; The output module is used to receive and parse the inference results output by the visual language model to obtain the matching relationship between traffic light instances between the driving view and the map view.

9. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium / computer program product, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 7; and / or, The computer program product includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

Images