Signal light state tracking method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a signal lamp state tracking method and device, electronic equipment and storage medium, the method comprising: acquiring a current frame image; performing signal lamp detection on the current frame image to obtain at least one detection box, each detection box including a target signal lamp; for any detection box, based on the spatial position similarity and the appearance feature similarity between the detection box and a plurality of prediction boxes, matching the detection box with the plurality of prediction boxes to determine a target prediction box matched with the detection box; wherein the plurality of prediction boxes correspond one by one to a plurality of historical motion trajectories, and the historical motion trajectory is used to represent the change of the position and state category of the same signal lamp in a plurality of consecutive historical images; updating the historical motion trajectory corresponding to the target prediction box to obtain a target motion trajectory corresponding to the target signal lamp in the detection box. The present application can improve the accuracy of tracking multiple signal lamps.

Description

[0001] This application claims priority to Chinese Patent Application No. 202511842337.1, filed on December 8, 2025, entitled “Method, Apparatus, Device and Medium for Determining the Light Status of a Traffic Light”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of image processing technology, and in particular to a method, apparatus, electronic device, and storage medium for tracking the status of traffic lights. Background Technology

[0003] In urban transportation, traffic lights, as crucial traffic control devices, directly impact vehicle passage decisions and traffic flow stability through their state transitions. For example, in the field of intelligent driving, vehicles sense changes in traffic light status to execute driving operations such as starting, stopping, or changing lanes; in the field of traffic signal control, real-time detection of traffic light status optimizes signal timing strategies to improve traffic efficiency; and in the field of map and navigation services, the location and status of traffic lights are dynamically updated to provide more accurate navigation services.

[0004] In related technologies, during the dynamic tracking of traffic lights, intersection images are manually labeled, and the labeled data is used to train a recognition model to detect and track the status of traffic lights. However, this method relies on manual labor, has low processing efficiency, high cost, and is difficult to meet the needs of large-scale data processing; furthermore, the visual algorithms used have low accuracy and poor stability. Summary of the Invention

[0005] This application provides a traffic light status tracking method, apparatus, electronic device, and storage medium, which can improve the accuracy of tracking multiple traffic lights.

[0006] In a first aspect, embodiments of this application provide a traffic light state tracking method, the method comprising: acquiring a current frame image; performing traffic light detection on the current frame image to obtain at least one detection box, each detection box including a target traffic light; for any detection box, matching the detection box with multiple prediction boxes based on the spatial position similarity and appearance feature similarity between the detection box and multiple prediction boxes to determine a target prediction box matching the detection box; wherein, the multiple prediction boxes correspond one-to-one with multiple historical motion trajectories, the historical motion trajectories being used to characterize the changes in the position and state category of the same traffic light in consecutive frames of historical images; updating the historical motion trajectory corresponding to the target prediction box to obtain the target motion trajectory corresponding to the target traffic light in the detection box.

[0007] In one possible implementation, the above-mentioned matching of a detection box with multiple prediction boxes based on the spatial similarity and appearance feature similarity between the detection box and multiple prediction boxes to determine the target prediction box that matches the detection box includes: for each prediction box, determining the fusion cost value between the detection box and the prediction box based on the spatial similarity and appearance feature similarity between the detection box and the prediction box; if there is a target fusion cost value among the multiple fusion cost values ​​of the detection box and the multiple prediction boxes, then the prediction box corresponding to the target fusion cost value is determined as the target prediction box that matches the detection box; wherein, the target fusion cost value is the fusion cost value among the multiple fusion cost values ​​that is less than a preset cost threshold; or, the target fusion cost value is the smallest among the multiple fusion cost values, and the target fusion cost value is less than the preset cost threshold.

[0008] In one possible implementation, the above-mentioned determination of the fusion cost of the detection box and the prediction box based on the spatial similarity and appearance feature similarity between the detection box and the prediction box for each predicted box includes: for each predicted box, determining a location cost based on the spatial similarity between the detection box and the prediction box, wherein the location cost is negatively correlated with the spatial similarity between the detection box and the prediction box; determining an appearance cost based on the appearance feature similarity between the detection box and the prediction box, wherein the appearance cost is negatively correlated with the appearance feature similarity between the detection box and the prediction box; and performing a weighted fusion based on the location cost and the appearance cost to obtain the fusion cost of the detection box and the prediction box, wherein the location cost corresponds to a first weight parameter, the appearance cost corresponds to a second weight parameter, and the sum of the first weight parameter and the second weight parameter is 1.

[0009] In one possible implementation, if the displacement of the current acquisition position corresponding to the current frame image relative to the previous acquisition position corresponding to the previous frame image is greater than a preset displacement threshold, the first weight parameter is greater than the second weight parameter; if the displacement of the current acquisition position corresponding to the current frame image relative to the previous acquisition position corresponding to the previous frame image is less than the preset displacement threshold, the first weight parameter is less than the second weight parameter.

[0010] In one possible implementation, the method further includes: adjusting the area cross-union ratio of the detection box and the prediction box based on the width cross-union ratio and the height cross-union ratio of the detection box and the prediction box to obtain an adjusted area cross-union ratio; and using the adjusted area cross-union ratio as the spatial similarity between the detection box and the prediction box.

[0011] In one possible implementation, adjusting the area cross-union ratio of the detection box and the prediction box based on the width cross-union ratio and the height cross-union ratio of the detection box and the prediction box to obtain the adjusted area cross-union ratio includes: compensating the area cross-union ratio of the detection box and the prediction box when the width cross-union ratio of the detection box and the prediction box is greater than a first cross-union ratio threshold, and / or when the height cross-union ratio of the detection box and the prediction box is greater than a second cross-union ratio threshold, to obtain the adjusted area cross-union ratio.

[0012] In one possible implementation, the above-mentioned traffic light detection of the current frame image to obtain at least one detection box includes: using a traffic light detection model, performing traffic light detection on the current frame image to obtain at least one detection box.

[0013] In one possible implementation, the above-described signal light detection model for detecting signal lights in the current frame image to obtain at least one detection box includes: using the signal light detection model to extract features from the current frame image at multiple scales to obtain multiple image features at different scales, wherein the multiple image features include at least a first image feature and a second image feature, and the scale of the first image feature is larger than that of the second image feature; using the first image feature as a reference and fusing it with the second image feature to obtain a first fused feature; using the second image feature as a reference and fusing it with the first fused feature to obtain a second fused feature; and performing signal light detection on the current frame image based on the first fused feature and the second fused feature to obtain at least one detection box.

[0014] In one possible implementation, after updating the historical motion trajectory corresponding to the target prediction box to obtain the target motion trajectory corresponding to the target traffic light in the detection box, the method further includes: generating key frame data of the target traffic light in the detection box based on the target motion trajectory, wherein the key frame data is image data representing the change in the state category of the target traffic light in the detection box; and outputting the key frame data.

[0015] Secondly, embodiments of this application provide a traffic light status tracking device, comprising:

[0016] The acquisition module is used to acquire the image of the current frame;

[0017] The detection module is used to detect traffic lights in the current frame image and obtain at least one detection box, each detection box including a target traffic light;

[0018] The matching module is used to match any detection box with multiple prediction boxes based on the spatial similarity and appearance feature similarity between the detection box and multiple prediction boxes, and to determine the target prediction box that matches the detection box. Among them, multiple prediction boxes correspond one-to-one with multiple historical motion trajectories, and the historical motion trajectories are used to characterize the changes in the position and state category of the same traffic light in multiple consecutive historical images.

[0019] The update module is used to update the historical motion trajectory corresponding to the target prediction box, so as to obtain the target motion trajectory corresponding to the target traffic light in the detection box.

[0020] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0021] The memory stores the instructions that the computer executes;

[0022] The processor executes computer execution instructions stored in memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0023] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0024] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0025] In this embodiment, by detecting traffic lights in the current frame image, at least one detection box corresponding to at least one target traffic light is obtained. Based on the spatial similarity and appearance feature similarity between the detection box and each prediction box, the detection box is matched with multiple prediction boxes to determine the target prediction box that matches the detection box. Then, the historical motion trajectory corresponding to the target prediction box is updated to obtain the target motion trajectory corresponding to the target traffic light in the detection box. In this way, by fusing the spatial similarity and appearance feature similarity between the detection box and the prediction box, interference from other objects with similar appearances to traffic lights (such as billboards and road signs) can be reduced, improving the accuracy of tracking multiple traffic lights. Furthermore, by fusing and matching spatial similarity and appearance feature similarity, even if the appearance features are distorted due to changes in illumination, the spatial position information can still provide a reliable matching basis, thus maintaining high detection and tracking robustness under different lighting conditions. Attached Figure Description

[0026] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0027] Figure 1 This is a schematic diagram of the implementation environment of a traffic light state tracking method according to an embodiment of this application;

[0028] Figure 2 A flowchart illustrating a traffic light state tracking method provided in an embodiment of this application;

[0029] Figure 3 A schematic diagram of a detection frame provided in an embodiment of this application;

[0030] Figure 4 A flowchart illustrating another traffic light state tracking method provided in this application embodiment;

[0031] Figure 5 A schematic diagram of a traffic light detection model provided in an embodiment of this application;

[0032] Figure 6 A schematic diagram of another traffic light detection model provided in an embodiment of this application;

[0033] Figure 7 A flowchart illustrating another traffic light state tracking method provided in this application embodiment;

[0034] Figure 8 A schematic diagram illustrating traffic light tracking as provided in an embodiment of this application;

[0035] Figure 9 This is a schematic diagram of the structure of a traffic light status tracking device provided in an embodiment of this application;

[0036] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0037] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described below do not represent all embodiments consistent with this application. They are merely examples of systems and methods consistent with some aspects of this application as detailed in the claims.

[0038] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0039] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0040] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0041] The term "module" refers to any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0042] In urban traffic, traffic lights (also known as traffic signals or signal lights) are crucial traffic control devices, and their state transitions directly affect vehicle passage decisions and traffic flow stability. For example, in the field of intelligent driving, vehicles sense changes in the state of traffic lights to perform driving operations such as starting, stopping, or changing lanes; in the field of traffic signal control, real-time detection of traffic light states optimizes signal timing strategies to improve traffic efficiency; and in the field of map and navigation services, the location and state of traffic lights are dynamically updated to provide more accurate navigation services.

[0043] In related technologies, during the dynamic tracking of traffic lights, intersection images are manually labeled, and the labeled data is used to train a recognition model to detect and track the status of traffic lights. However, this method relies on manual labeling, resulting in low processing efficiency, high cost, and difficulty in meeting the needs of large-scale data processing. Furthermore, the matching strategy of the visual algorithm used is relatively simple, making it difficult to distinguish traffic lights from other similar objects (such as road signs), leading to low processing accuracy and poor stability.

[0044] In view of this, embodiments of this application provide a traffic light state tracking method. By detecting traffic lights in the current frame image, at least one detection box corresponding to at least one target traffic light is obtained. Based on the spatial position similarity and appearance feature similarity between the detection box and each prediction box, the detection box is matched with multiple prediction boxes to determine the target prediction box that matches the detection box. Then, the historical motion trajectory corresponding to the target prediction box is updated to obtain the target motion trajectory corresponding to the target traffic light in the detection box. In this way, by fusing the spatial position similarity and appearance feature similarity between the detection box and the prediction boxes, interference from other objects with similar appearances to traffic lights (such as billboards and road signs) can be reduced, improving the accuracy of tracking multiple traffic lights. Furthermore, by fusing and matching spatial position similarity and appearance feature similarity, even if the appearance features are distorted due to changes in illumination, the spatial position information can still provide a reliable matching basis, thus maintaining high detection and tracking robustness under different lighting conditions.

[0045] Before introducing the traffic light state tracking method provided in the embodiments of this application, the application scenarios of the traffic light state tracking method will be explained first.

[0046] The traffic light state tracking method provided in this application can be applied to fields such as intelligent driving, traffic signal control, vehicle-road cooperative equipment, and dynamic updating of high-precision maps.

[0047] Scenario 1 (Intelligent Driving Scenario): During vehicle operation, video data (i.e., multi-frame intersection images) is continuously acquired via onboard cameras. Traffic lights in each frame are detected and tracked, and keyframe data representing changes in traffic light status is extracted. Based on this keyframe data, the vehicle is controlled to perform driving operations such as starting, stopping, or changing lanes.

[0048] Scenario 2 (Traffic Signal Control Scenario): Roadside sensing devices collect video data (i.e., multi-frame intersection images) via cameras, detect and track traffic lights in each frame, and extract keyframe data representing changes in traffic light status. Based on the keyframe data, each traffic light is dynamically controlled to optimize the traffic light timing strategy.

[0049] Scenario 3 (Map Update Scenario): Video data (i.e., multi-frame intersection images) is collected via cameras. Traffic lights in each frame are detected and tracked, and keyframe data representing changes in traffic light status is extracted. Based on the keyframe data, the status of traffic lights on the map is updated in real time.

[0050] Figure 1 This is a schematic diagram illustrating the implementation environment of a traffic light state tracking method according to an embodiment of this application.

[0051] like Figure 1As shown, the implementation environment of this traffic light state tracking method includes an image acquisition device 10 and a server 20. The server 20 is communicatively connected to the image acquisition device 10. The image acquisition device 10 continuously acquires multiple frames of intersection images and transmits them to the server 20. The server 20 performs traffic light detection on the current frame image, obtaining the detection boxes for each traffic light in the current frame image. Then, it performs multi-dimensional feature matching between the motion trajectories of multiple traffic lights (i.e., the historical motion trajectories mentioned below) and the multiple detection boxes of the current frame image, and updates the motion trajectories of the traffic lights matched by each detection box according to the matching results, so as to realize the dynamic tracking of multiple traffic lights. It can be understood that the multiple frames of intersection images include the current frame image and multiple historical frames. The traffic light motion trajectories can be generated based on the multiple historical frames.

[0052] Furthermore, after updating the motion trajectory of the traffic lights matched by each detection box according to the matching results, key frame data representing the state changes of the traffic lights can be extracted based on the tracking data of the traffic lights, which can be used for intelligent driving, traffic signal control, dynamic map updates, etc.

[0053] In some examples, when matching the motion trajectory of a traffic light with the detection boxes in the current frame image, the server 20 first predicts the expected position of the traffic light's motion trajectory in the current frame image (i.e., at the current moment) for each traffic light motion trajectory and generates the corresponding prediction box; then, it matches each prediction box with each detection box in the current frame image.

[0054] More specifically, server 20 constructs a fusion cost matrix based on the spatial overlap (i.e., spatial similarity) between the predicted bounding box and the detection bounding box, and the similarity of appearance features between the image regions corresponding to the predicted bounding box and the image regions corresponding to the detection bounding box. Based on the fusion cost matrix, the server determines the motion trajectory of the traffic light matched by each detection bounding box, and uses the detection bounding box to update the motion trajectory of the matched traffic light, thereby realizing continuous dynamic tracking of multiple traffic lights.

[0055] In some examples, a traffic light detection model can be deployed on server 20, which includes a small object detection head for traffic lights. Server 20 can use the traffic light detection model to detect traffic lights on intersection images (such as the current frame image).

[0056] It is understandable that in the aforementioned intelligent driving scenarios, the image acquisition device 10 can be an image acquisition module such as a camera on the vehicle, and the subsequent detection and tracking of traffic lights can also be performed by the processing module on the vehicle. In the aforementioned traffic signal control scenario, the image acquisition device 10 can be a roadside perception device, and the server 20 is a roadside edge computing device. In the aforementioned map update scenario, the image acquisition device 10 can be a camera on the vehicle or a roadside perception device, and the server 20 can be a cloud-based map service platform.

[0057] Of course, the signal light status tracking method provided in this application embodiment can also be executed by an electronic device.

[0058] The above is a schematic description of the application scenarios and implementation environment of the traffic light state tracking method provided in the embodiments of this application. The traffic light state tracking method provided in the embodiments of this application will now be described in detail with reference to the accompanying drawings and application scenarios.

[0059] Figure 2 This is a flowchart illustrating a traffic light state tracking method provided in an embodiment of this application. Figure 2 The traffic light status tracking method may include the following steps:

[0060] S201, Obtain the current frame image.

[0061] Specifically, during vehicle operation, multiple frames of images (i.e., raw video data) are continuously captured by image acquisition devices mounted on the vehicle. These multiple frames include the current frame and multiple historical frames. Electronic devices can obtain the current frame image from the image acquisition devices.

[0062] The current frame image can be an intersection image captured by the image acquisition device at the current moment. The current frame image may include one or more target traffic lights.

[0063] For example, a data collection vehicle equipped with a self-developed micro-object detection device (including a high-definition camera and an anti-backlight lens) can be used to capture video of the intersection during the morning rush hour (8:00-9:00) on a sunny day to obtain raw video data.

[0064] S202, perform traffic light detection on the current frame image to obtain at least one detection box, and each detection box includes a target traffic light.

[0065] Specifically, after acquiring the current frame image from the image acquisition device, the electronic device performs traffic light detection on the current frame image to obtain the detection box corresponding to each target traffic light in the current frame image, that is, to obtain at least one detection box.

[0066] The detection box is used to indicate the position of the corresponding target traffic light in the current frame image. For example, the detection box can have a regular shape or an irregular shape. For instance, the detection box can be a rectangle, a circle, a polygon, or any other geometric shape. It is understood that the detection box is obtained by detecting traffic lights in the current frame image; therefore, the detection box here can be referred to as the current detection box.

[0067] The detection result corresponding to the detection box can include the current position information of the detection box, the state category of the target traffic light in the detection box, and the confidence score of the detection box. The current position information of the detection box includes the center point position (i.e., (x, y)) and size information (i.e., width w, height h). The state category of the target traffic light indicates the state of the target traffic light in the current frame image, i.e., the current signal color of the target traffic light. The state category of the traffic light can include red, green, and yellow. The confidence score of the detection box represents the probability that the corresponding target traffic light belongs to that state type.

[0068] For example, the detection result of the detection box can be represented as:

[0069] [Current position information of the detection box: [x, y, width, height], confidence level, state category].

[0070] For example, such as Figure 3 As shown, traffic light detection is performed on the current frame image, resulting in four detection boxes, such as detection box 1 (corresponding to traffic light 1), detection box 2 (corresponding to traffic light 2), detection box 3 (corresponding to traffic light 3), and detection box 4 (corresponding to traffic light 4).

[0071] Among them, detection box 1: [[520, 180, 60, 80], 0.92, "green"];

[0072] Detection box 2: [[520, 190, 60, 80], 0.88, "green"].

[0073] In addition, for each detection box, the current frame image also includes a status identifier (i.e., a status label) for the detection box, indicating the status category of the corresponding traffic light. Of course, the current frame image may also include the center point position of the detection box (not shown in the figure).

[0074] Optionally, the size of the detection box is larger than the size of the actual image area of ​​the target traffic light in the current frame image, and the distance between the border of the detection box and the corresponding boundary of the actual image area is greater than a preset distance.

[0075] For example, the distance between the left (or right) border of the detection box and the left (or right) boundary of the actual image area is a first preset distance, and the distance between the top (or bottom) border of the detection box and the top (or bottom) boundary of the actual image area is a second preset distance.

[0076] The first preset distance can be represented in pixels, such as 15 pixels. Similarly, the second preset distance can also be represented in pixels, such as 25 pixels.

[0077] Specifically, when generating the detection box, the actual image area (i.e., region of interest) of the target traffic light is identified. The actual image area is then expanded outward by a first preset distance (e.g., 15 pixels) along the width direction and a second preset distance (e.g., 25 pixels) along the height direction to obtain the detection box. In this way, the detection box can cover the local background features around the target traffic light, allowing for the extraction of more features during subsequent appearance feature matching, thus improving the accuracy of subsequent traffic light tracking.

[0078] It is understandable that the traffic light detection model used in this embodiment can directly output the expanded detection frame.

[0079] The following is a schematic illustration of traffic light detection and training of a traffic light detection model for the current frame image.

[0080] In some examples, when performing traffic light detection on the current frame image, multi-scale feature extraction can be performed on the current frame image to obtain multiple image features at different scales; the multiple image features at different scales are cross-fused, and the fused image features are used to perform traffic light detection on the current frame image to obtain at least one detection box.

[0081] For example, such as Figure 4 As shown, performing traffic light detection on the current frame image to obtain at least one detection box may include the following steps:

[0082] S401, perform multi-scale feature extraction on the current frame image to obtain multiple image features at different scales. The multiple image features include at least a first image feature and a second image feature, and the scale of the first image feature is larger than the scale of the second image feature.

[0083] For example, image features at different scales can include large-scale image features, medium-scale image features, and small-scale image features. Large-scale image features, such as 160×160 pixels, have a small receptive field but contain rich detail information, used for detecting small targets; medium-scale image features, such as 80×80 pixels, balance detail and semantics, used for detecting medium-sized targets; and small-scale image features, such as 20×20 pixels, have a large receptive field and rich semantic information, used for detecting large targets. The first image feature can be a large-scale image feature; the second image feature can include both medium-scale and small-scale image features.

[0084] In this embodiment, the traffic lights to be detected are usually small targets. Therefore, a first image feature of a larger scale is used, and features of other scales are fused together to detect the traffic lights.

[0085] S402, the first image feature is used as a reference and fused with the second image feature to obtain the first fused feature.

[0086] This fusion process can be achieved through upsampling, splicing, or weighted summation to inject mesoscale semantic information into large-scale features.

[0087] S403, using the second image feature as a reference, and fusing it with the first fusion feature to obtain the second fusion feature.

[0088] This step further enhances feature representation, allowing mesoscale features to also receive detailed supplementation from large-scale features.

[0089] S404, based on the first fusion feature and the second fusion feature, perform traffic light detection on the current frame image to obtain at least one detection box.

[0090] In some examples, a pre-trained traffic light detection model can be used to detect traffic lights in the current frame image and obtain at least one detection box.

[0091] For example, the traffic light detection model can employ deep learning-based object detection algorithms, such as the YOLO series (e.g., YOLOv5, YOLOv7), Faster R-CNN, RetinaNet, or CenterNet. In this embodiment, the specific type of object detection algorithm is not limited.

[0092] For example, such as Figure 5 and Figure 6 As shown, the traffic light detection model includes an input, a backbone network, a neck network, and an output. Specifically, the input receives the original image (such as the current frame image), adjusts the original image to a uniform size (such as 640×640×3), and then sends it to the backbone network.

[0093] The backbone network downsamples the image layer by layer through a series of convolutional layers (Conv) and C2f modules to extract deep features at multiple scales. Specifically, the image is processed sequentially through convolutions and C2f layers of 320×320×64, 160×160×128, 80×80×256, and 40×40×512, ultimately outputting a 20×20×1024 deepest feature map. In this process, the C2f module acts as an improved feature extraction unit, enhancing feature reuse through cross-layer connections, while the convolutional layers are responsible for progressively reducing the resolution and increasing the number of channels, enabling the network to capture both detailed and semantic information simultaneously.

[0094] The feature maps at multiple scales output by the backbone network are then fed into the neck network for feature pyramid fusion. The neck network employs a top-down approach, progressively restoring the resolution of deep feature maps through upsampling operations and concatenating them with corresponding shallow feature maps. These concatenated maps are then fused using the C2f module, ensuring that each feature map possesses both deep semantics and shallow details. Specifically, the fusion path is as follows: the 20×20×1024 layer is upsampled and fused with the 40×40×512 layer to obtain a 40×40×512 fused feature; this layer is then upsampled and fused with the 80×80×256 layer to obtain an 80×80×256 fused feature; finally, it is upsampled and fused with the 160×160×128 layer to obtain a high-resolution 160×160×128 fused feature. This process constructs the feature pyramid, laying the foundation for subsequent multi-scale detection.

[0095] The fused multi-scale feature maps are used for object detection at the input and output ends. The output end contains multiple parallel detection heads, each corresponding to a feature map of a different scale, responsible for detecting targets of different sizes. Specifically, considering that traffic lights are typically small targets, a micro-object detection head is set on a high-resolution feature map of 160×160×128 to improve the recall rate for small targets. Each detection head employs a decoupled head structure, separating the classification and regression tasks into different branches: the classification branch outputs the probability of the target belonging to each category (e.g., red, yellow, green), supervised by the classification loss (e.g., cross-entropy loss); the regression branch outputs the position coordinates of the detection box (center point x, y, width w, height h), supervised by the bounding box regression loss (e.g., CIoU loss). This decoupled design avoids mutual interference between the classification and regression tasks, effectively improving detection accuracy.

[0096] Ultimately, the model outputs detection results at three scales: 160×160 scale for small targets (such as distant traffic lights), 80×80 scale for medium-sized targets, and 40×40 scale for large targets. If necessary, it can also use an additional path to detect extremely large targets at a 20×20 scale. These detection boxes and their corresponding state categories (red / yellow / green) are output for use by the subsequent tracking module.

[0097] Through the above-mentioned multi-scale feature extraction, pyramid fusion, and decoupled detection head design, this model can effectively cope with the challenges of traffic lights in images such as small scale, large illumination changes, and complex backgrounds, providing high-quality detection input for subsequent stable tracking.

[0098] The Bottleneck module is a lightweight residual unit designed to deepen the network while controlling the number of parameters. Its basic structure is: Conv → Conv2d → BN → SiLU, where: Conv: typically a 1×1 convolution used for dimensionality reduction and computational cost reduction; Conv2d: a 3×3 convolution used to extract spatial features; BN: Batch Normalization, accelerating training convergence; SiLU: Sigmoid Linear Unit activation function, enhancing nonlinear expressive power.

[0099] When a module is configured with a shortcut, the input is directly added to the output to form a residual connection, which effectively alleviates the gradient vanishing problem and allows the network to be deeper.

[0100] The C2f module is a highly efficient feature extraction unit with the following structure: Conv → Split → Bottleneck × N → Conv. The specific process is as follows: the input first undergoes preliminary processing through a Conv module; the feature map is split into two parts along the channel dimension; one part is directly passed, while the other part is sequentially processed through multiple Bottleneck modules (the number N is configurable) for deep feature extraction; finally, the two feature parts are concatenated (Concat) and then fused through another Conv module for output. This design enhances feature expressiveness while maintaining lightweight design through cross-stage feature reuse and branching structures, and is widely used for feature extraction and fusion in backbone and neck networks.

[0101] The SPPF module is used to capture multi-scale contextual information. Its structure is: Conv → MaxPool → MaxPool → MaxPool → Conv. The input first passes through a Conv layer to adjust the number of channels; then it passes through three cascaded max-pooling (MaxPool) layers (the pooling kernel size is usually 5×5). With each pooling, the receptive field expands progressively. The original features are concatenated with the features obtained from the three pooling layers (the concatenation is not explicitly shown in the figure, but it is typically done in implementations), and finally, it passes through a Conv layer for fusion and output. This module can extract contextual features at different scales without significantly increasing computational cost, enhancing the model's robustness to changes in the target scale.

[0102] The decoupled head is the core detection unit at the output end, and its structure is as follows:

[0103] Conv → Conv → Conv2d → Bbox Loss (Regression Branch)

[0104] Conv → Conv → Conv2d → Cls Loss (Classification Branch)

[0105] The input feature map is first processed through two shared Conv2d methods. Then, it branches into two independent branches: a regression branch, which outputs the location coordinates of the detection box (e.g., center point x, y, width w, height h) through a Conv2d method and is supervised by a bounding box regression loss (e.g., CIoU Loss); and a classification branch, which outputs the probability of the target belonging to each class (e.g., red, yellow, green) through a Conv2d method and is supervised by a classification loss (Cls Loss, e.g., cross-entropy loss). This separation of branches avoids interference between classification and regression tasks, effectively improving detection accuracy.

[0106] In the model, each scale of feature map is connected to a decoupling head, which is responsible for object detection at that scale. In particular, the decoupling head connected to the high-resolution feature map (such as 160×160) for small object detection is the micro-object detection head.

[0107] In addition, during the training phase, traffic lights are manually labeled, such as by using rectangular detection boxes to label traffic lights. The initial traffic light detection model is iteratively trained using manually labeled sample image data to obtain the traffic light detection model.

[0108] In this embodiment, based on the characteristics of the original model, a small object detection head is added to improve the detection and recognition resolution, making the model detection more accurate. At the same time, widening the detection box during the manual annotation of training data allows the model to learn various features around the traffic lights, improving the accuracy of model detection and recognition.

[0109] S203: For any given detection box, based on the spatial similarity and appearance feature similarity between the detection box and multiple prediction boxes, match the detection box with multiple prediction boxes to determine the target prediction box that matches the detection box.

[0110] In this system, multiple predicted bounding boxes correspond one-to-one with multiple historical motion trajectories. The historical motion trajectories characterize the changes in the position and state category of the same traffic light (i.e., the traffic light to be matched) across multiple consecutive historical images; these trajectories are determined based on multiple historical images. A predicted bounding box indicates the possible position of the corresponding traffic light to be matched in the current frame image. The traffic light to be matched can be understood as the traffic light identified from the previous frame image (or the previous n frames).

[0111] In some examples, before matching the detection box with multiple prediction boxes, traffic lights are detected in the previous frame image to determine the previous detection box of each traffic light to be matched in the previous frame image, and the possible position of the corresponding traffic light to be matched in the current frame image is predicted based on the previous detection box, thus obtaining the prediction box corresponding to each traffic light to be matched.

[0112] For example, a Kalman filter can be used as a motion model to predict the possible location of the traffic light to be matched in the current frame image.

[0113] In some examples, the electronic device may acquire the detection box first and then the prediction box, or acquire the detection box and the prediction box simultaneously. In the embodiments of this application, the acquisition order of the detection box and the prediction box is not specifically limited.

[0114] After determining multiple predicted boxes, for each detection box, the detection box and the predicted boxes are matched based on the spatial similarity between the detection box and each predicted box, as well as the appearance feature similarity between the detection box and each predicted box, to determine the target predicted box that matches the detection box.

[0115] The spatial similarity between the detection box and the prediction box characterizes the degree of overlap between them, that is, the degree of overlap between the position of the target traffic light and the position of the traffic light to be matched in the current frame image. The greater the spatial similarity between the detection box and the prediction box, the higher the degree of overlap between them.

[0116] For example, taking any predicted bounding box as an example, the spatial similarity between the detected bounding box and the predicted bounding box can be determined based on the area intersection-union ratio between the detected bounding box and the predicted bounding box.

[0117] Specifically, the intersection-union ratio of the areas of the detection box and the prediction box can be determined according to the following formula (1).

[0118] (1)

[0119] in, This represents the area intersection-union ratio (IUU) of the detection bounding box and the predicted bounding box. The prediction box represents the traffic light to be matched corresponding to the historical motion trajectory; Represents the detection box; This represents the area of ​​the intersection region between the detection box and the prediction box, i.e., the pixel area of ​​the overlapping part; This represents the area of ​​the union region between the detection box and the predicted box.

[0120] Optionally, the area cross-union ratio between the detection box and the prediction box can be adjusted based on the width cross-union ratio and the height cross-union ratio, and the adjusted area cross-union ratio can be used as the spatial similarity between the detection box and the prediction box.

[0121] The width intersection-union ratio (CIU) of the detection and prediction boxes indicates the degree of overlap between them in the width direction. Specifically, the CIU is the ratio of the intersection width of the detection and prediction boxes in the width direction to the union width of the detection and prediction boxes in the width direction.

[0122] The height intersection-over-union ratio (IoU) of the detection and prediction boxes indicates the degree of overlap between them in the height direction. Specifically, the IoU is the ratio of the height of the intersection of the detection and prediction boxes in the width direction to the height of their union in the width direction.

[0123] For example, if the width cross-union ratio of the detection box and the prediction box is greater than a first cross-union ratio threshold, and / or the height cross-union ratio of the detection box and the prediction box is greater than a second cross-union ratio threshold, the area cross-union ratio of the detection box and the prediction box is compensated to obtain the adjusted area cross-union ratio, which is the spatial similarity between the detection box and the prediction box.

[0124] If the width cross-union ratio of the detection box and the prediction box is less than or equal to the first cross-union ratio threshold, and the height cross-union ratio of the detection box and the prediction box is less than or equal to the second cross-union ratio threshold, the area cross-union ratio of the detection box and the prediction box is used as the spatial similarity between the detection box and the prediction box.

[0125] The first intersection-union (IU) threshold is used to determine whether the overlap between the detection box and the prediction box is large in the width direction. For example, the first IU threshold can be 0.8. The second IU threshold is used to determine whether the overlap between the detection box and the prediction box is large in the height direction. For example, the second IU threshold can be 0.8. Of course, the second IU threshold can be different from the first IU threshold. The specific values ​​of the first IU threshold and the second IU threshold are not limited in the embodiments of this application.

[0126] More specifically, taking the first cross-union threshold of 0.8 and the second cross-union threshold of 0.8 as an example, the spatial similarity between the detection box and the prediction box can be determined according to the following formula (2).

[0127] (2)

[0128] in, This represents the augmented intersection-over-union ratio (AUC) between the detected bounding box and the predicted bounding box, which is the spatial similarity between the two boxes. This represents the area intersection-union ratio (IUU) of the detection bounding box and the predicted bounding box. Indicates the crossover and union ratio compensation value; Indicates an indicator function; This represents the intersection-over-union ratio (IoU) of the widths of the detection bounding box and the predicted bounding box. This represents the height intersection-over-union ratio (IoU) of the detection box and the prediction box.

[0129] For example, the maximum value for the crossover-union ratio (CUI) compensation is 1. For instance, the CUI compensation value could be 0.2.

[0130] Indicator Function The decision bars are used to characterize the width cross-union ratio (CUNR) and height cross-union ratio (CUNR) of the detected and predicted bounding boxes. When the width CUNR of the detected and predicted bounding boxes is greater than a first CUNR threshold (e.g., 0.8), and / or the height CUNR of the detected and predicted bounding boxes is greater than a second CUNR threshold (e.g., 0.8), the width CUNR and height CUNR of the detected and predicted bounding boxes meet the conditions, and the indicator function is then activated. The value is 1; otherwise, the indicator function... The value is 0. That is, when the width cross-union ratio of the detection box and the prediction box is greater than the first cross-union ratio threshold (e.g., 0.8), and / or the height cross-union ratio of the detection box and the prediction box is greater than the second cross-union ratio threshold (e.g., 0.8), the cross-union ratio compensation value is used as a reward and added to the area cross-union ratio of the detection box and the prediction box to obtain the adjusted area cross-union ratio, which is the spatial similarity between the detection box and the prediction box.

[0131] For example, the intersection width of the detection box and the prediction box in the width direction can be determined according to the following formula (3).

[0132] (3)

[0133] in, This represents the width of the intersection between the detection box and the prediction box in the width direction; This represents the right boundary of the prediction box, specifically the x-coordinate (i.e., pixel value) of the right boundary of the prediction box. This represents the left boundary of the prediction box, specifically the x-coordinate (i.e., pixel value) of the left boundary of the prediction box. This represents the right boundary of the detection box, specifically the x-coordinate (i.e., pixel value) of the right boundary of the detection box. This represents the left boundary of the detection box, specifically the x-coordinate (i.e., pixel value) of the left boundary of the detection box.

[0134] For example, the union width of the detection box and the prediction box in the width direction can be determined according to the following formula (4).

[0135] (4)

[0136] in, This represents the width of the union of the detection box and the prediction box in the width direction; Indicates the width of the prediction box; This indicates the width of the detection box.

[0137] Similarly, the intersection height of the detection box and the prediction box in the height direction can be determined according to the following formula (5).

[0138] (5)

[0139] in, This represents the height of the intersection between the detection box and the predicted box in the height direction; This represents the lower boundary of the prediction box, specifically the y-coordinate (i.e., pixel value) of the lower boundary of the prediction box. This represents the upper boundary of the prediction box, specifically the y-coordinate (i.e., pixel value) of the upper boundary of the prediction box. This represents the lower boundary of the detection box, specifically the y-coordinate (i.e., pixel value) of the lower boundary of the detection box. This represents the upper boundary of the detection box, specifically the y-coordinate (i.e., pixel value) of the upper boundary of the detection box.

[0140] For example, the union height of the detection box and the prediction box in the height direction can be determined according to the following formula (6).

[0141] (6)

[0142] in, This represents the union height of the detection box and the predicted box in the height direction; Indicates the height of the prediction box; This indicates the height of the detection frame.

[0143] In this embodiment, the appearance feature similarity between the detection box and the prediction box can characterize the degree of similarity between the image features (such as shape, color, and texture) of the detection box and the prediction box.

[0144] For example, a similarity detection model can be used to determine the appearance feature similarity between the detection box and the predicted box. Specifically, the information of the current frame image, the detection box, and the predicted box is input into a pre-trained similarity detection model. The image regions corresponding to the detection box and the predicted box are extracted from the current frame image, respectively. Based on the image features of these two image regions, the appearance feature similarity between the detection box and the predicted box is output.

[0145] As a specific implementation method, the similarity detection model can use the MobileNetv2 network. After fine-tuning the MobileNetv2 network, the classification layer is removed, and the output of the bottleneck layer is used as the feature vector of the image region. The cosine similarity between the feature vector corresponding to the detection box and the feature vector corresponding to the prediction box is calculated, which gives the appearance feature similarity between the detection box and the prediction box.

[0146] As another specific implementation, the similarity detection model can also adopt the Transformer architecture, which extracts the feature vectors of image regions through the self-attention mechanism and outputs the appearance feature similarity.

[0147] In this embodiment, after determining the spatial similarity and appearance feature similarity between the detection box and each prediction box, the detection box is matched with multiple prediction boxes based on the spatial similarity and appearance feature similarity between the detection box and each prediction box to determine the target prediction box that matches the detection box.

[0148] In some examples, such as Figure 7 As shown, matching the detection box with multiple prediction boxes to determine the target prediction box that matches the detection box can include the following steps:

[0149] S701, for each predicted bounding box, determine the fusion value of the detected bounding box and the predicted bounding box based on the spatial similarity and appearance feature similarity between them.

[0150] The lower the fusion value, the higher the matching degree between the detected bounding box and the predicted bounding box. The higher the fusion value, the lower the matching degree between the detected bounding box and the predicted bounding box.

[0151] For example, when determining the fusion cost of the detection box and the predicted box, the location cost is determined based on the spatial similarity between the detection box and the predicted box; the appearance cost is determined based on the appearance feature similarity between the detection box and the predicted box; and the fusion cost of the detection box and the predicted box is obtained by weighted fusion based on the location cost and the appearance cost.

[0152] The location cost is negatively correlated with the spatial similarity between the detection box and the predicted box. The greater the spatial similarity between the detection box and the predicted box, the smaller the location cost; conversely, the smaller the spatial similarity between the detection box and the predicted box, the greater the location cost.

[0153] The appearance cost is negatively correlated with the appearance feature similarity between the detection box and the predicted box. The greater the appearance feature similarity between the detection box and the predicted box, the smaller the appearance cost; conversely, the smaller the appearance feature similarity between the detection box and the predicted box, the greater the appearance cost.

[0154] Optionally, the location cost corresponds to the first weight parameter, and the appearance cost corresponds to the second weight parameter. The sum of the first weight parameter and the second weight parameter is 1.

[0155] For example, the fusion cost of the detection box and the prediction box can be determined according to the following formula (7).

[0156] (7)

[0157] in, This represents the fusion cost of the predicted bounding box corresponding to the i-th historical motion trajectory (i.e., the i-th predicted bounding box) and the j-th detection box; This represents the spatial similarity between the i-th predicted bounding box and the j-th detected bounding box; This represents the similarity in appearance features between the i-th predicted bounding box and the j-th detected bounding box; Indicates the first weight parameter; This represents the second weighting parameter.

[0158] In one possible implementation, if the displacement of the current acquisition position corresponding to the current frame image relative to the previous acquisition position corresponding to the previous frame image is greater than a preset displacement threshold, the first weight parameter is greater than the second weight parameter.

[0159] For example, when the displacement of the current acquisition position relative to the previous acquisition position is greater than the preset displacement threshold, the first weight parameter is set to 0.7 and the second weight parameter is set to 0.3.

[0160] In another possible implementation, if the displacement of the current acquisition position corresponding to the current frame image relative to the previous acquisition position corresponding to the previous frame image is less than a preset displacement threshold, the first weight parameter is less than the second weight parameter.

[0161] For example, when the displacement of the current acquisition position relative to the previous acquisition position is less than a preset displacement threshold, the first weight parameter is set to 0.3 and the second weight parameter is set to 0.7.

[0162] In other words, when the vehicle is in motion, it means the traffic light to be identified is also in motion relative to the vehicle, i.e., the position of the traffic light in the image changes significantly. In this case, a larger first weight parameter (corresponding to the positional cost) and a smaller second weight parameter (corresponding to the appearance cost) are selected, so that the fusion cost is mainly determined by the positional cost. Conversely, when the vehicle is stationary, it means the traffic light to be identified is stationary relative to the vehicle, i.e., the position of the traffic light in the image changes little. In this case, a smaller first weight parameter and a larger second weight parameter are selected, so that the fusion cost is mainly determined by the appearance cost. Thus, by adaptively adjusting the weight parameters of the positional cost and appearance cost, and matching the detection box and the prediction box based on the fusion cost, the accuracy of the matching can be improved.

[0163] S702, if a target fusion value exists among the multiple fusion values ​​of the detection box and multiple prediction boxes, then the prediction box corresponding to the target fusion value is determined as the target prediction box that matches the detection box.

[0164] The target fusion cost value is the fusion value among multiple fusion values ​​that is less than a preset cost threshold; or, the target fusion cost value is the smallest among multiple fusion values, and the target fusion value is less than the preset cost threshold.

[0165] For example, a cost matrix can be constructed based on the fusion cost of the detection boxes and the predicted boxes. This cost matrix indicates the degree of matching between the detection boxes and the predicted boxes corresponding to each historical motion trajectory. Based on the cost matrix of the detection boxes and the predicted boxes, the target predicted box matching each detection box is determined.

[0166] More specifically, the Hungarian algorithm can be used to determine the target prediction box that matches each detection box based on the cost matrix between the detection box and the prediction box.

[0167] Based on the fusion cost matrix, the Hungarian algorithm is used for optimal matching to find the minimum cost match between the detection box and the prediction box. The Hungarian algorithm ensures that the matching process is globally optimal. If the fusion cost of the detection box and the prediction box is less than the preset cost threshold, it is considered a successful match, and the prediction box is determined as the target prediction box for the detection box.

[0168] S204, update the historical motion trajectory corresponding to the target prediction box to obtain the target motion trajectory corresponding to the target traffic light in the detection box.

[0169] Specifically, after determining the target prediction box that matches the detection box, i.e., determining the historical motion trajectory that matches the detection box, the historical motion trajectory corresponding to the target prediction box is updated according to the detection result of the detection box, so as to obtain the target motion trajectory corresponding to the target traffic light, and the tracking data of the target motion trajectory is output.

[0170] For example, the tracking data of the target motion trajectory may include the image number of each frame, the position information of the detection box, and the status category of the target signal light in the detection box.

[0171] For example, the tracking data of the target's motion trajectory can be represented as follows:

[0172] {

[0173] "Trajectory number of the target movement trajectory of the target signal light": [

[0174] ["Current frame image number", "[x-coordinate of the center point of the detection box, y-coordinate of the center point of the detection box, height of the detection box, width of the detection box]", "color of the target traffic light in the detection box"]

[0175] ["Frame 2 image number", "[x-coordinate of the center point of the detection box, y-coordinate of the center point of the detection box, height of the detection box, width of the detection box]", "color of the target traffic light in the detection box"]

[0176] ... ]

[0178] }

[0179] For example, "Motion Trajectory 1": [

[0180] [1,[510,175,570,255],"red"],

[0181] [2,[520,185,570,255],"green"]

[0182] ... ]

[0184] }

[0185] For example, such as Figure 8 As shown, in the first frame, the position and state of the traffic lights (i.e., traffic lights) are detected; in the second frame, as the vehicle moves forward, the image moves upward, and the detection boxes continuously track it. Even if there are multiple detection boxes in the image, their respective detection boxes are matched during the tracking phase; in the third frame, the color of the traffic lights changes during continuous tracking, and the state of the detection boxes changes accordingly.

[0186] In some embodiments, after determining the target prediction box that matches the detection box, i.e., determining the historical motion trajectory that matches the detection box, the Kalman filter corresponding to the historical motion trajectory is updated according to the position information of the detection box, so as to use the updated Kalman filter to predict the possible position of the target traffic light in the next frame image, thereby improving the accuracy of prediction and thus improving the accuracy of traffic light tracking.

[0187] In some embodiments, if no target fusion value is found among the multiple fusion values ​​of the detection box and multiple predicted boxes (i.e., no target predicted box matching the detection box exists among the multiple predicted boxes), candidate motion trajectories are generated based on the position information of the detection box. Then, the next frame image is acquired, and traffic light detection is performed on the next frame image to obtain multiple next detection boxes. Based on the candidate motion trajectories, the next predicted box of the target traffic light in the next frame image is predicted. If a next detection box matching the next predicted box exists among the multiple next detection boxes, the candidate motion trajectory is determined to be a successful match. If the candidate motion trajectories are all successfully matched in subsequent preset number of frames, the candidate motion trajectory is determined to be valid, and is thus used as a historical motion trajectory for subsequent traffic light tracking.

[0188] For example, if the detection box corresponding to the target traffic light fails to match the historical motion trajectory, a candidate motion trajectory is generated based on the position information of the detection box. If the candidate motion trajectory matches successfully in the next 5 frames, that is, the survival time of the candidate motion trajectory exceeds 5 frames, the candidate motion trajectory is determined to be valid, and the candidate motion trajectory is used as the historical motion trajectory for subsequent traffic light tracking.

[0189] This allows for the removal of invalid short tracks caused by false detections or brief occurrences, while retaining valid traffic light tracks with a sufficiently long duration (generally at least 80 frames), thereby filtering out low-quality tracks.

[0190] In some embodiments, the method further includes: if there are unmatched historical motion trajectories among multiple historical motion trajectories, determining the disappearance area of ​​the traffic light according to the driving operation type of the vehicle; if the traffic light to be matched corresponding to the historical motion trajectory disappears from other areas besides the disappearance area, then the historical motion trajectory is deleted.

[0191] Understandably, during normal vehicle operation, when a vehicle is going straight, turning, or making a U-turn, traffic lights typically gradually move out of the image from the top or left / right sides, rather than suddenly disappearing from the center or bottom. Based on this, dynamic borders can be pre-defined at the top and left / right sides of the image as disappearance zones. During tracking, if a historical trajectory fails to match, and the last position of the traffic light corresponding to that trajectory is not within the disappearance zone, the historical trajectory is determined to be a low-quality trajectory and deleted. This dynamic border filtering mechanism effectively filters out sudden changes in traffic light position caused by false detections or anomalies, further improving the reliability of the tracking results.

[0192] In some embodiments, after updating the historical motion trajectory corresponding to the target prediction box to obtain the target motion trajectory corresponding to the target traffic light in the detection box, the method further includes: generating key frame data of the target traffic light in the detection box based on the target motion trajectory, wherein the key frame data is image data representing the change in the state category of the target traffic light in the detection box; and outputting the key frame data.

[0193] For example, the output format of keyframe data can be: "Video ID of raw video data": [{"Color switching order": frame number}, ...].

[0194] For example, the keyframe data is: "250519073842166430": [{"red_green": 516}, {"red_green": 516}]. This indicates that target traffic light 1 changes from red to green in frame 516, and target traffic light 2 changes from red to green in frame 516.

[0195] In some examples, the target motion trajectory is smoothed and filtered before generating keyframe data of the target traffic light within the detection box based on the target motion trajectory. The target motion trajectory includes the position information and state category of the target traffic light within the detection box in each frame image.

[0196] For example, a preset sliding window is used to smooth the state categories. Specifically, the state category (i.e., color) of the target traffic light (i.e., the detection box) in multiple consecutive frames of images is obtained from the target motion trajectory. It is then detected whether there are any abnormal state categories in the target traffic light in multiple consecutive frames of images. If abnormal state categories exist, they are modified.

[0197] For example, taking a preset sliding window length of 5 frames as an example, the color of the target traffic light in each frame of the preset sliding window (i.e., the current frame and the two frames before and after) is projected onto the screen, and the mode of the voting results is taken as the color of the target traffic light in the current frame. For example, in the sequence [red, red, green, red, red], the color green in the 3rd frame is misidentified and is changed to red.

[0198] In this embodiment, single-frame misidentification is effectively suppressed, the fault tolerance rate of color recognition is enhanced, and the state jump caused by detection noise is reduced, so that the color state change of traffic lights is more in line with the actual timing pattern.

[0199] In some examples, if the keyframe data indicates that the state switching duration and sequence of the target traffic light in the detection frame meet preset switching conditions, then the keyframe data is output. Otherwise, the keyframe data is deleted.

[0200] The preset switching conditions indicate that the state switching duration and switching sequence of the target traffic lights comply with traffic regulations.

[0201] For example, according to the "Specifications for the Setting and Installation of Road Traffic Signal Lights", the duration of a yellow light is usually 3 to 5 seconds. Considering detection errors, a tolerance range can be set. For example, trajectories with a yellow light duration exceeding 6 seconds can be judged as abnormal and filtered; red or green lights with excessively short durations (such as less than 4 seconds) are also considered abnormal.

[0202] For example, the sequence of traffic light transitions should conform to regulations. For instance, the traffic light sequence for motor vehicles should be red → green → yellow → red, and the yellow light should not immediately turn green. If an illegal transition such as "yellow → green" or "green → red" is detected, it is considered abnormal and filtered out.

[0203] In this way, by filtering out noise from single-frame detection and correcting abnormal sequences using the standard traffic light switching logic of "green→yellow→red," the problem of instantaneous false detections is solved. Pattern verification ensures that state changes conform to objective laws, and the dual mechanisms work together to improve the reliability of state recognition. Furthermore, by utilizing the physical characteristics of the traffic light's motion patterns, the legitimacy of the target state transition is comprehensively judged. This mechanism specifically filters out sudden changes in traffic light activity, improving tracking stability.

[0204] In this embodiment, by detecting traffic lights in the current frame image, at least one detection box corresponding to at least one target traffic light is obtained. Based on the spatial similarity and appearance feature similarity between the detection box and each prediction box, the detection box is matched with multiple prediction boxes to determine the target prediction box that matches the detection box. Then, the historical motion trajectory corresponding to the target prediction box is updated to obtain the target motion trajectory corresponding to the target traffic light in the detection box. In this way, by fusing the spatial similarity and appearance feature similarity between the detection box and the prediction box, interference from other objects with similar appearances to traffic lights (such as billboards and road signs) can be reduced, improving the accuracy of tracking multiple traffic lights. Furthermore, by fusing and matching spatial similarity and appearance feature similarity, even if the appearance features are distorted due to changes in illumination, the spatial position information can still provide a reliable matching basis, thus maintaining high detection and tracking robustness under different lighting conditions.

[0205] Furthermore, spatial similarity is used to reflect the positional continuity of traffic lights, while appearance feature similarity distinguishes different traffic lights. Cascaded matching preserves short-term occlusion trajectories. Multi-dimensional feature fusion covers the needs of different tracking scenarios, thus reducing trajectory interruptions and ensuring trajectory stability.

[0206] Furthermore, during vehicle movement, as the vehicle approaches or moves away from the traffic light, the traffic light typically disappears from one side of the image. The position of the traffic light in adjacent frames may shift. Therefore, the overlap between the detection and prediction boxes of the same traffic light in adjacent frames not only varies in area but also in the width and height directions. Consequently, based on the width and height cross-union ratios of the detection and prediction boxes, the area cross-union ratio is compensated. Then, based on the compensated area cross-union ratio, the spatial similarity between the detection and prediction boxes is determined. This adds a reward to the area cross-union ratio, thereby strengthening the matching priority for targets with highly similar shapes but whose area overlap may slightly decrease due to translation. Through this compensation mechanism, the spatial similarity between the detection and prediction boxes can be more accurately characterized, avoiding mismatches caused by solely relying on the area cross-union ratio, thus improving the accuracy and robustness of traffic light tracking.

[0207] In some embodiments, after acquiring the keyframe data of the target traffic light in the detection frame, the keyframe data of the target traffic light in the detection frame can be used for intelligent driving, traffic signal control, vehicle-road cooperative equipment, and dynamic updates of high-precision maps.

[0208] For example, real-time scene monitoring and anomaly warning: keyframe data can accurately mark the position, status category, and dynamic changes of traffic lights in the video, and combined with visual video, intuitively present the overall scene. In application, it can monitor in real time whether traffic lights at intersections are displaying normally, whether vehicles are illegally crossing the line, and other abnormal situations. When an abnormality is detected in the target location or status (such as a traffic light suddenly going out or a vehicle running a red light), the system can trigger an alert based on keyframe data to assist managers in responding quickly.

[0209] Business data statistics and decision support: After structured processing, keyframe data can generate quantitative indicators (such as the number of vehicles passing through an intersection per unit time, the duration of each traffic light state, etc.), while visualized videos provide contextual evidence for the data. This information can be used in decision-making scenarios such as traffic flow analysis and traffic light timing optimization. For example, by statistically analyzing vehicle traffic data at different times and combining it with visualized congestion conditions in videos, data support can be provided for adjusting traffic light durations and planning traffic management schemes.

[0210] System Iteration and Optimization, and Problem Tracing: During the application phase, the accuracy of keyframe data (such as target localization deviation and category misclassification) can be visually verified through video visualization. When there are discrepancies between the detection results and the actual targets in the video, the deficiencies of the algorithm in complex scenes (such as backlighting and occlusion) can be identified by comparing the positions of the detection boxes in the keyframe data with the video frame. Simultaneously, the accumulated visualized video data with keyframe annotations can serve as a sample library for continuous iterative training of the model, improving the robustness of this technical solution in real-world scenarios.

[0211] In summary, this application, through multi-dimensional feature fusion matching and adaptive weight adjustment, demonstrates high robustness in the face of challenges such as occlusion, multi-target interference, similar appearance, and changes in illumination, meeting the high requirements for traffic light tracking stability in practical applications such as autonomous driving and traffic monitoring. Furthermore, this method inherently supports multi-target tracking scenarios, capable of processing multiple traffic light targets in the same frame simultaneously, exhibiting good scalability and practicality.

[0212] This application also provides a traffic light status tracking device. For example... Figure 9 As shown, the traffic light state tracking device 900 includes an acquisition module 901, a detection module 902, a matching module 903, and an update module 904. The acquisition module 901 acquires the current frame image. The detection module 902 performs traffic light detection on the current frame image, obtaining at least one detection box, each detection box including a target traffic light. The matching module 903, for any given detection box, matches it with multiple prediction boxes based on the spatial similarity and appearance feature similarity between the detection box and multiple prediction boxes, determining the target prediction box that matches the detection box; wherein, the multiple prediction boxes correspond one-to-one with multiple historical motion trajectories, and the historical motion trajectories represent the changes in the position and state category of the same traffic light in consecutive frames of historical images. The update module 904 updates the historical motion trajectory corresponding to the target prediction box, obtaining the target motion trajectory corresponding to the target traffic light in the detection box.

[0213] In some embodiments, the matching module 903 is specifically used to determine the fusion cost value between the detection box and the prediction box based on the spatial similarity and appearance feature similarity between the detection box and the prediction box for each prediction box; if there is a target fusion cost value among the multiple fusion cost values ​​of the detection box and multiple prediction boxes, then the prediction box corresponding to the target fusion cost value is determined as the target prediction box that matches the detection box; wherein, the target fusion cost value is the fusion cost value among the multiple fusion cost values ​​that is less than a preset cost threshold; or, the target fusion cost value is the smallest among the multiple fusion cost values, and the target fusion cost value is less than the preset cost threshold.

[0214] In some embodiments, the matching module 903 is specifically used to, for each predicted box, determine a location cost value based on the spatial similarity between the detected box and the predicted box, wherein the location cost value is negatively correlated with the spatial similarity between the detected box and the predicted box; determine an appearance cost value based on the appearance feature similarity between the detected box and the predicted box, wherein the appearance cost value is negatively correlated with the appearance feature similarity between the detected box and the predicted box; and perform a weighted fusion of the location cost value and the appearance cost value to obtain a fused cost value between the detected box and the predicted box, wherein the location cost value corresponds to a first weight parameter, the appearance cost value corresponds to a second weight parameter, and the sum of the first weight parameter and the second weight parameter is 1.

[0215] In some embodiments, when the displacement of the current acquisition position corresponding to the current frame image relative to the previous acquisition position corresponding to the previous frame image is greater than a preset displacement threshold, the first weight parameter is greater than the second weight parameter; when the displacement of the current acquisition position corresponding to the current frame image relative to the previous acquisition position corresponding to the previous frame image is less than the preset displacement threshold, the first weight parameter is less than the second weight parameter.

[0216] In some embodiments, the traffic light state tracking device 900 further includes a determination module, which adjusts the area cross-union ratio of the detection box and the prediction box according to the width cross-union ratio and the height cross-union ratio of the detection box and the prediction box to obtain an adjusted area cross-union ratio; and uses the adjusted area cross-union ratio as the spatial position similarity between the detection box and the prediction box.

[0217] In some embodiments, the determining module is specifically used to compensate the area cross-union ratio of the detection box and the prediction box when the width cross-union ratio of the detection box and the prediction box is greater than a first cross-union ratio threshold, and / or the height cross-union ratio of the detection box and the prediction box is greater than a second cross-union ratio threshold, so as to obtain an adjusted area cross-union ratio.

[0218] In some embodiments, the detection module 902 is specifically used to perform traffic light detection on the current frame image using a traffic light detection model to obtain at least one detection box.

[0219] In some embodiments, the detection module 902 is specifically used to perform multi-scale feature extraction on the current frame image using a traffic light detection model to obtain multiple image features of different scales, wherein the multiple image features include at least a first image feature and a second image feature, and the scale of the first image feature is larger than the scale of the second image feature; the first image feature is used as a reference and fused with the second image feature to obtain a first fused feature; the second image feature is used as a reference and fused with the first fused feature to obtain a second fused feature; and traffic light detection is performed on the current frame image based on the first fused feature and the second fused feature to obtain at least one detection box.

[0220] In some embodiments, the traffic light state tracking device 900 further includes a key frame extraction module, which generates key frame data of the target traffic light in the detection frame based on the target motion trajectory. The key frame data is image data representing changes in the state category of the target traffic light in the detection frame; and outputs the key frame data.

[0221] The electronic device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0222] Figure 10 A schematic diagram of the structure of the electronic device provided in this application. Figure 10 As shown, the electronic device 100 provided in this embodiment includes at least one processor 1001 and a memory 1002. Optionally, the electronic device 100 further includes a communication component 1003. The processor 1001, memory 1002, and communication component 1003 are connected via a bus 1004.

[0223] In a specific implementation, at least one processor 1001 executes computer execution instructions stored in memory 1002, causing at least one processor 1001 to perform the above-described method.

[0224] The specific implementation process of processor 1001 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0225] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0226] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0227] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0228] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0229] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0230] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0231] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0232] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0233] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0234] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0235] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0236] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0237] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A method for tracking the state of a traffic light, characterized in that, The method includes: Get the current frame image; Traffic light detection is performed on the current frame image to obtain at least one detection box, and each detection box includes a target traffic light; For any given detection box, based on the spatial similarity and appearance feature similarity between the detection box and multiple prediction boxes, the detection box is matched with the multiple prediction boxes to determine the target prediction box that matches the detection box; wherein, the multiple prediction boxes correspond one-to-one with multiple historical motion trajectories, and the historical motion trajectories are used to characterize the changes in the position and state category of the same traffic light in multiple consecutive historical images; The historical motion trajectory corresponding to the target prediction box is updated to obtain the target motion trajectory corresponding to the target traffic light in the detection box.

2. The method according to claim 1, characterized in that, For any given detection box, based on the spatial similarity and appearance feature similarity between the detection box and multiple predicted boxes, the detection box is matched with the multiple predicted boxes to determine the target predicted box that matches the detection box, including: For each predicted bounding box, the fusion value of the detected bounding box and the predicted bounding box is determined based on the spatial similarity and appearance feature similarity between the detected bounding box and the predicted bounding box. If a target fusion value exists among the multiple fusion values ​​of the detection box and the multiple prediction boxes, then the prediction box corresponding to the target fusion value is determined as the target prediction box that matches the detection box. Wherein, the target fusion cost value is the fusion generation value among the plurality of fusion generation values ​​that is less than a preset cost threshold; or, the target fusion cost value is the smallest among the plurality of fusion generation values, and the target fusion generation value is less than the preset cost threshold.

3. The method according to claim 2, characterized in that, For each predicted bounding box, the fusion value between the detected bounding box and the predicted bounding box is determined based on the spatial similarity and appearance feature similarity between them, including: For each predicted bounding box, a location cost is determined based on the spatial similarity between the detected bounding box and the predicted bounding box. The location cost is negatively correlated with the spatial similarity between the detected bounding box and the predicted bounding box. The appearance cost is determined based on the appearance feature similarity between the detection box and the prediction box, and the appearance cost is negatively correlated with the appearance feature similarity between the detection box and the prediction box. The location value and the appearance value are weighted and fused to obtain the fusion value of the detection box and the prediction box, wherein the location value corresponds to a first weight parameter, the appearance value corresponds to a second weight parameter, and the sum of the first weight parameter and the second weight parameter is 1.

4. The method according to claim 3, characterized in that, If the displacement of the current acquisition position corresponding to the current frame image relative to the previous acquisition position corresponding to the previous frame image is greater than a preset displacement threshold, the first weight parameter is greater than the second weight parameter. If the displacement of the current acquisition position corresponding to the current frame image relative to the previous acquisition position corresponding to the previous frame image is less than a preset displacement threshold, the first weight parameter is less than the second weight parameter.

5. The method according to claim 1, characterized in that, The method further includes: Based on the width intersection-union ratio and height intersection-union ratio of the detection box and the prediction box, the area intersection-union ratio of the detection box and the prediction box is adjusted to obtain the adjusted area intersection-union ratio; the adjusted area intersection-union ratio is used as the spatial similarity between the detection box and the prediction box. or, The step of performing traffic light detection on the current frame image to obtain at least one detection box includes: Using a traffic light detection model, the current frame image is used to detect traffic lights, resulting in at least one detection box.

6. The method according to claim 5, characterized in that, The step of adjusting the area intersection ratio of the detection box and the prediction box based on the width intersection ratio and the height intersection ratio of the detection box and the prediction box to obtain the adjusted area intersection ratio includes: If the width intersection-union ratio of the detection box and the prediction box is greater than a first intersection-union ratio threshold, and / or the height intersection-union ratio of the detection box and the prediction box is greater than a second intersection-union ratio threshold, the area intersection-union ratio of the detection box and the prediction box is compensated to obtain the adjusted area intersection-union ratio.

7. The method according to claim 1, characterized in that, The step of performing traffic light detection on the current frame image to obtain at least one detection box includes: Using a traffic light detection model, multi-scale feature extraction is performed on the current frame image to obtain multiple image features at different scales. Among the multiple image features, at least a first image feature and a second image feature are included, and the scale of the first image feature is larger than the scale of the second image feature. The first image feature is used as a reference and fused with the second image feature to obtain the first fused feature; Based on the second image feature, it is fused with the first fusion feature to obtain the second fusion feature; Based on the first fusion feature and the second fusion feature, traffic light detection is performed on the current frame image to obtain at least one detection box.

8. The method according to any one of claims 1-7, characterized in that, After updating the historical motion trajectory corresponding to the target prediction box to obtain the target motion trajectory corresponding to the target traffic light in the detection box, the method further includes: Based on the target motion trajectory, key frame data of the target traffic light in the detection frame is generated. The key frame data is image data representing the change in the state category of the target traffic light in the detection frame. Output the keyframe data.

9. A signal light status tracking device, characterized in that, include: The acquisition module is used to acquire the image of the current frame; The detection module is used to detect traffic lights in the current frame image and obtain at least one detection box, each detection box including a target traffic light; The matching module is used to match any detection box with the multiple prediction boxes based on the spatial similarity and appearance feature similarity between the detection box and the multiple prediction boxes, and to determine the target prediction box that matches the detection box; wherein, the multiple prediction boxes correspond one-to-one with multiple historical motion trajectories, and the historical motion trajectories are used to characterize the changes in the position and state category of the same traffic light in multiple consecutive historical images; The update module is used to update the historical motion trajectory corresponding to the target prediction box to obtain the target motion trajectory corresponding to the target traffic light in the detection box.

10. An electronic device / computer-readable storage medium / computer program product, characterized in that, include: Memory and processor; The memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-8; And / or, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-8; and / or, The computer program product includes a computer program that, when executed by a processor, is used to implement the method as described in any one of claims 1-8.

Images