Method of detecting traffic lights and electronic device

Abstract

The present disclosure provides a method and an electronic device for detecting traffic lights. The method can include: collecting vehicle motion information and a current frame image of a vehicle during driving, wherein the frame image is captured by a camera installed on the vehicle; selecting a previous frame image adjacent to the current frame image, and identifying at least one traffic light to be tracked from the previous frame image; predicting a predicted position of the traffic light to be tracked in the current frame image based on position information of the traffic light to be tracked in the previous frame image and the vehicle motion information of the vehicle; determining a plurality of candidate traffic lights in the current frame image based on the predicted position of the traffic light to be tracked in the current frame image; and determining a target traffic light from the plurality of candidate traffic lights. The present disclosure solves the technical problem of low reliability of traffic light detection.

Description

Technical Field

[0001] This disclosure relates to the fields of artificial intelligence and pattern recognition, and in particular to methods and electronic devices for detecting traffic lights. Background Technology

[0002] Currently, when detecting traffic lights, the traffic light information associated with the lane where the vehicle is located is first obtained through a high-precision map. This traffic light frame is then projected onto the image plane, and a larger Region of Interest (ROI) is expanded based on the projected frame. Within this ROI, the model is invoked to detect the traffic lights. Subsequently, the detection frame is matched with the projected frame, and the result is output. However, this method relies entirely on the high-precision map. If the high-precision map of the current vehicle's driving area is unavailable, or if the traffic lights do not match the information in the high-precision map due to road maintenance or replacement, the traffic light detection module will fail and cannot perform traffic light detection normally. Therefore, there is a technical problem of low reliability in traffic light detection.

[0003] There is currently no effective solution to the aforementioned technical problem of low reliability in traffic light detection. Summary of the Invention

[0004] This invention provides a method and electronic device for detecting traffic lights, thereby at least solving the technical problem of low reliability in traffic light detection.

[0005] According to one aspect of the present invention, a method for detecting traffic lights is provided, comprising: a vehicle acquiring vehicle motion information and a current frame image during its driving process, wherein the frame image is obtained by capturing images using a camera mounted on the vehicle; selecting a previous frame image adjacent to the current frame image, and identifying at least one traffic light to be tracked from the previous frame image; predicting the predicted position of the traffic light to be tracked in the current frame image based on the position information of the traffic light to be tracked in the previous frame image and the vehicle motion information of the vehicle; determining a plurality of candidate traffic lights in the current frame image based on the predicted position of the traffic light to be tracked in the current frame image, wherein the traffic light to be tracked and the plurality of candidate traffic lights in the current frame have a one-to-many matching relationship; and determining a target traffic light from the plurality of candidate traffic lights, wherein the target traffic light represents the traffic light that matches the traffic light to be tracked in the current frame image.

[0006] Optionally, based on the position information of the traffic light in the previous frame image and the vehicle motion information, the predicted position of the traffic light to be tracked in the current frame image is predicted. The method includes: calculating the vehicle displacement change information and vehicle orientation angle change information between the current frame image and the previous frame image based on the vehicle motion information during the vehicle's movement; and determining the predicted position of the traffic light to be tracked in the current frame image based on the position information, vehicle displacement change information, and vehicle orientation angle change information of the traffic light to be tracked in the previous frame image.

[0007] Optionally, the tracking frame of the traffic light to be tracked is displayed in the previous frame image, wherein the tracking frame is used to characterize the traffic light frame information of the traffic light to be tracked.

[0008] Optionally, based on the predicted position of the traffic light to be tracked in the current frame image, multiple candidate traffic lights in the current frame image are determined. The method includes: displaying a tracking box at the predicted position in the current frame image; calling a full-image detection model or a ROI detection model to detect the current frame image and identify multiple candidate traffic lights in the current frame image.

[0009] Optionally, the full-image detection model is used to detect at least one candidate traffic light in the current frame image that is associated with the tracking box, and to identify the candidate traffic light pairs with the detection box.

[0010] Optionally, the ROI detection model is used to expand the region based on the location of the tracking box to obtain an expanded box, identify at least one candidate traffic light located in the expanded box, and mark the candidate traffic light pair detection box.

[0011] Optionally, the method further includes: filtering multiple candidate traffic lights, the step of which includes at least one of the following: sorting the multiple candidate traffic lights according to their confidence level, and obtaining traffic lights with a confidence level exceeding a threshold according to the sorting result; deleting traffic lights located below the horizon in the current frame image.

[0012] Optionally, the target traffic light is determined from multiple candidate traffic lights. The method includes: displaying a tracking box and multiple detection boxes in the current frame image, wherein the tracking box is used to represent the traffic light frame information of the traffic light to be tracked in the current frame image, and the detection boxes are used to represent the traffic light frame information of the candidate traffic lights in the current frame image; calling a type recognition model to classify the multiple detection boxes; using a Hungarian algorithm to sequentially match the classification results with the tracking box to determine the target detection box that matches the tracking box from the multiple detection boxes; and identifying the target traffic light located in the current frame image based on the target detection box.

[0013] Optionally, the method further includes: determining a newly identified target traffic light in the current frame image based on at least one traffic light that did not match successfully, and creating tracking information for the target traffic light.

[0014] According to another aspect of the present invention, an apparatus for detecting traffic lights is provided, comprising: a data acquisition unit for acquiring vehicle motion information and a current frame image during vehicle operation, wherein the frame image is captured by a camera mounted on the vehicle; a selection unit for selecting a previous frame image adjacent to the current frame image and identifying at least one traffic light to be tracked from the previous frame image; a prediction unit for predicting the predicted position of the traffic light to be tracked in the current frame image based on the position information of the traffic light to be tracked in the previous frame image and the vehicle motion information of the vehicle; a first determination unit for determining multiple candidate traffic lights in the current frame image based on the predicted position of the traffic light to be tracked in the current frame image, wherein the traffic light to be tracked and the multiple candidate traffic lights in the current frame have a one-to-many matching relationship; and a second determination unit for determining a target traffic light from the multiple candidate traffic lights, wherein the target traffic light represents the traffic light that matches the traffic light to be tracked in the current frame image.

[0015] According to another aspect of the present invention, an electronic device is also provided, which is installed in a vehicle. The electronic device includes: one or more processors; a storage device for storing one or more programs; and when the one or more programs are executed by the one or more processors, the one or more processors perform the method for detecting traffic lights according to the present disclosure.

[0016] According to another aspect of the present invention, a vehicle is also provided in which a stored program is run, wherein, when the program is running, the processor of the device controls the execution of the method for detecting traffic lights according to the present disclosure.

[0017] In this embodiment of the invention, vehicle motion information and the current frame image are acquired during vehicle operation. The frame image is captured by a camera mounted on the vehicle. A previous frame image adjacent to the current frame image is selected, and at least one traffic light to be tracked is identified from the previous frame image. Based on the position information of the traffic light to be tracked in the previous frame image and the vehicle motion information, the predicted position of the traffic light to be tracked in the current frame image is predicted. Based on the predicted position of the traffic light to be tracked in the current frame image, multiple candidate traffic lights are determined, wherein the traffic light to be tracked and the multiple candidate traffic lights in the current frame have a one-to-many matching relationship. The target traffic light is determined from the multiple candidate traffic lights, wherein the target traffic light represents the traffic light that matches the traffic light to be tracked in the current frame image. In other words, in this application, a frame-by-frame detection model is designed. The input of this model is an ROI generated based on the predicted value, and a Measurement Hash Tree is used. The MHT (Multi-Hypergraph Hierarchical Tree) method performs one-to-many matching, then candidate box filtering, and finally one-to-one matching. This matching scheme uses basic computer vision principles such as pinhole imaging and camera vanishing point for prediction, tracking, and filtering, reducing the demand for and dependence on computing resources. This solves the technical problem of low reliability in traffic light detection and achieves the technical effect of improving the reliability of traffic light detection.

[0018] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0019] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0020] Figure 1 This is a flowchart of a method for detecting traffic lights according to an embodiment of the present disclosure;

[0021] Figure 2 This is a schematic diagram illustrating the relationship between an image coordinate system, a camera coordinate system, and a world coordinate system according to an embodiment of the present disclosure;

[0022] Figure 3 This is a flowchart of a traffic light detection and recognition method for autonomous driving without a high-precision map, according to an embodiment of the present disclosure;

[0023] Figure 4A This is a schematic diagram of the previous frame image of a traffic light to be tracked according to an embodiment of the present disclosure;

[0024] Figure 4BThis is a schematic diagram of the current frame image of a traffic light to be tracked according to an embodiment of the present disclosure;

[0025] Figure 4C This is a schematic diagram illustrating the application effect of a traffic light detection and recognition method for autonomous driving without a high-precision map, according to an embodiment of the present disclosure.

[0026] Figure 5 This is an apparatus diagram of a traffic light detection method according to an embodiment of the present disclosure. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0028] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] Example 1

[0030] The method for detecting traffic lights according to embodiments of this disclosure will now be described.

[0031] In existing technologies, when detecting traffic lights, the traffic light information associated with the current vehicle's lane is first obtained through a high-precision map. This traffic light frame is then projected onto the image plane, and a larger Region of Interest (ROI) is expanded based on the projected frame. Within this ROI, a model is invoked to detect the traffic lights. Subsequently, the detection frame is matched with the projected frame, and the result is output. However, this method relies entirely on high-precision maps. If the high-precision map for the current vehicle's driving area is unavailable, or if the traffic lights do not match the information in the high-precision map due to road repairs or replacements, the traffic light detection module will fail and cannot perform traffic light detection normally. Furthermore, such solutions require the purchase or self-construction of high-precision maps, increasing development costs and difficulty. Real-time maintenance and updates of the map are also required, increasing time and manpower costs.

[0032] In addition, the model designs used in existing traffic light detection schemes are mainly divided into two categories: one type detects the traffic light and outputs its color at the same time, but this scheme has poor robustness and cannot handle the situation where multiple bulbs are lit at the same time; the other type uses a combination of detection model and classification model, first using the detection model to determine the position of the traffic light, and then using the classification model to judge the color and category of the bulbs inside the traffic light. The disadvantage of this method is that it consumes a lot of computing resources and it is difficult to guarantee real-time performance.

[0033] In this invention, an additional small ROI model that can quickly detect traffic lights is designed. Based on this, the traffic light full-image detection model can be called every other frame. The traffic light frames detected by the model are combined with vehicle motion information to predict their positions, and the small ROI model is called in the next frame for detection. This alternating detection design significantly reduces the computational resource requirements, effectively improves operating efficiency, and ensures the real-time performance of the traffic light module.

[0034] Figure 1 This is a flowchart of a method for detecting traffic lights according to an embodiment of the present disclosure, as follows: Figure 1 As shown, the method may include the following steps:

[0035] Step S101: Collect vehicle motion information and current frame image during vehicle operation, wherein the frame image is obtained by a camera installed on the vehicle.

[0036] In the technical solution provided by step S101 of this disclosure, vehicle motion information and current frame image during the driving process can be collected. For example, vehicle motion information and current frame image during the driving process can be obtained by using a camera installed on the vehicle.

[0037] In this embodiment, vehicle motion information can be vehicle speed, yaw rate of change, etc., and is not limited thereto.

[0038] In this embodiment, the current frame image can be an image containing multiple traffic lights.

[0039] Step S102: Select the previous frame image adjacent to the current frame image, and identify at least one traffic light to be tracked from the previous frame image.

[0040] In the technical solution provided by step S102 of this disclosure, a previous frame image adjacent to the current frame image can be selected, and at least one traffic light to be tracked can be identified from the previous frame image. For example, in the full-image detection model, the previous frame image adjacent to the current frame image is used as the input of the model to detect traffic lights in the entire image.

[0041] Step S103: Based on the position information of the traffic light to be tracked in the previous frame image and the vehicle motion information, predict the predicted position of the traffic light to be tracked in the current frame image.

[0042] In the technical solution provided by step S103 of this disclosure, when predicting the predicted position of the traffic light to be tracked in the current frame image, the traffic light is detected within the range of the previous frame image adjacent to the current frame image by a full-image detection model or a ROI detection model, and traffic light frame information is output. The traffic light frame information is used to represent the position information of the traffic light to be tracked in the previous frame image. Then, the vehicle displacement change information and vehicle orientation angle change information are calculated, and then the predicted position of the traffic light to be tracked in the current frame image is predicted.

[0043] In this embodiment, the vehicle motion information can be vehicle displacement change information and vehicle orientation angle change information. The vehicle displacement change information and vehicle orientation angle change information can be calculated using information such as the vanishing point of the camera, the yaw rate of the vehicle, and the lateral position of the traffic light in the image.

[0044] Step S104: Based on the predicted position of the traffic light to be tracked in the current frame image, determine multiple candidate traffic lights in the current frame image, wherein the traffic light to be tracked and the multiple candidate traffic lights in the current frame have a one-to-many matching relationship.

[0045] In this embodiment, multiple candidate traffic lights in the current frame image can be determined based on the predicted position of the traffic light to be tracked in the current frame image. For example, based on the configuration, a full-image detection model or a ROI detection model can be called to detect the current frame. After obtaining the results, all detection boxes are sorted according to confidence, and the maximum number of output detections is preset to ensure the accuracy of detection.

[0046] In this embodiment, when determining multiple candidate traffic lights in the current frame image, multiple candidate traffic lights in the previous frame image can be obtained, and after sorting all detection boxes according to confidence level, all candidate detection boxes can be initially screened.

[0047] For example, firstly, all detection boxes with confidence scores below a threshold are deleted. Secondly, all detection boxes whose lower endpoints are a certain distance below the vanishing point are deleted. Based on common sense about traffic, traffic lights do not appear below the horizon in the image. After completing the initial screening, an MHT tree structure is constructed by performing a two-level traversal of the tracking and detection boxes to achieve a one-to-many matching relationship between the tracking box from the previous frame and the current detection box. Before matching, the aspect ratio and area of ​​the tracking and detection boxes are calculated and compared. If the difference is greater than a threshold, it indicates that the two boxes are significantly different, and they will not be matched.

[0048] Step S105: Determine the target traffic light from multiple candidate traffic lights, wherein the target traffic light represents the traffic light that matches the traffic light to be tracked in the current frame image.

[0049] In this embodiment, the target traffic light can be determined from multiple candidate traffic lights. For example, after obtaining all traffic light frames to be detected, a classification model is called to detect and classify the bulbs inside the detected traffic light frames, completing a one-to-one matching between the traffic light tracking frame and the detection frame, that is, determining the target traffic light from multiple candidate traffic lights.

[0050] In this embodiment, the Hungarian algorithm can be used to match the classification results with the tracking boxes in sequence, determine the target detection box that matches the tracking box from multiple detection boxes, and identify the target traffic light located in the current frame image based on the target detection box.

[0051] Through steps S101 to S106, vehicle motion information and the current frame image during vehicle operation are collected. The frame image is obtained by capturing images using a camera mounted on the vehicle. A previous frame image adjacent to the current frame image is selected, and at least one traffic light to be tracked is identified from the previous frame image. Based on the position information of the traffic light to be tracked in the previous frame image and the vehicle motion information, the predicted position of the traffic light to be tracked in the current frame image is predicted. Based on the predicted position of the traffic light to be tracked in the current frame image, multiple candidate traffic lights are determined in the current frame image, wherein the traffic light to be tracked... The traffic light has a one-to-many matching relationship with multiple candidate traffic lights in the current frame. The target traffic light is determined from the multiple candidate traffic lights. In other words, in this application, an inter-frame detection model is designed. The input of this model is the ROI generated based on the predicted value. One-to-many matching is performed using an MHT tree, followed by candidate box filtering, and finally one-to-one matching. Basic computer vision principles such as pinhole imaging and camera vanishing point are used for prediction, tracking, and filtering, reducing the demand for and dependence on computing resources. This solves the technical problem of low reliability in traffic light detection and achieves the technical effect of improving the reliability of traffic light detection.

[0052] The method described in this embodiment will now be described in further detail.

[0053] As an optional implementation, step S103 involves predicting the predicted position of the traffic light to be tracked in the current frame image based on the position information of the traffic light in the previous frame image and the vehicle motion information. This method includes: calculating the vehicle displacement change information and vehicle orientation angle change information between the current frame image and the previous frame image based on the vehicle motion information during vehicle movement; and determining the predicted position of the traffic light to be tracked in the current frame image based on the position information, vehicle displacement change information, and vehicle orientation angle change information of the traffic light to be tracked in the previous frame image.

[0054] In this embodiment, the predicted position of the traffic light to be tracked in the current frame image can be determined based on the position information, vehicle displacement change information, and vehicle orientation angle change information of the traffic light to be tracked in the previous frame image. For example, the traffic light can be detected within the range of the previous frame image adjacent to the current frame image by using a full-image detection model or a ROI detection model, and the traffic light frame information can be output. The traffic light frame information is used to represent the position information of the traffic light to be tracked in the previous frame image. Then, the vehicle displacement change information and vehicle orientation angle change information are calculated, and thus, the predicted position of the traffic light to be tracked in the current frame image is predicted.

[0055] In this embodiment, vehicle displacement change information and vehicle orientation angle change information can be calculated using information such as the vanishing point of the camera, the vehicle's yaw rate, and the lateral position of the traffic light in the image.

[0056] For example, firstly, the vehicle's distance traveled between two frames and the change in its orientation angle are calculated using vehicle kinematics information. Then, based on the principle of similar triangles, the distance between the traffic light detected in the previous frame and the vehicle body in the vehicle coordinate system (VCS coordinate system) is estimated, as follows:

[0057]

[0058] Where, x vcs Let f be the distance between the traffic light and the vehicle body in the VCS coordinate system. v s represents the image distance of the camera used. H The height of the traffic light frame in the VCS coordinate system (which can be estimated based on empirical values), s imH This represents the height of the traffic light frame in the image coordinate system.

[0059] Similarly, by acquiring information such as the vanishing point of the camera, the yaw rate of the vehicle, and the lateral position of the traffic light in the image, the lateral position of the traffic light in the VCS coordinate system can be calculated. After obtaining the position information of the traffic light in the VCS coordinate system of the previous frame, the possible position of the traffic light frame in the image coordinate system of the current frame can be estimated again using the vehicle's kinematic information and the principle of similar triangles, as shown in the following formula:

[0060]

[0061] Among them, y im Let y be the predicted ordinate of the center point of the traffic light frame in the current frame within the image coordinate system. vp The vanishing point of the camera. This represents the ordinate of the center point of the traffic light detection box in the previous frame within the image coordinate system. yaw represents the position of the traffic light on the x-axis in the VCS coordinate system of the previous frame, L represents the distance the vehicle travels between the two frames. rate This represents the rate of change of the vehicle's yaw angle between two frames.

[0062] Similarly, the x-coordinate of the predicted center point of the traffic light frame in the current frame can be calculated in the image coordinate system, thus ultimately obtaining the possible location of the current frame detection box predicted by the detection box in the previous frame.

[0063] Subsequently, the above process is repeated to predict the traffic light detection box obtained from the previous frame, and the result is stored in an array of member variables. The predicted box will then be used for detection and matching.

[0064] As an optional implementation, the tracking frame of the traffic light to be tracked is displayed in the previous frame image, wherein the tracking frame is used to characterize the traffic light frame information of the traffic light to be tracked.

[0065] In this embodiment, the tracking frame of the traffic light to be tracked can be displayed in the previous frame image. For example, the position, color, and type (circular light, arrow light, etc.) of the traffic light to be tracked can be displayed in the previous frame image.

[0066] In this embodiment, the traffic light frame information can be the color and category of the traffic light, and there are no restrictions on this.

[0067] As an optional implementation, based on the predicted position of the traffic light to be tracked in the current frame image, multiple candidate traffic lights are determined in the current frame image. The method includes: displaying a tracking box at the predicted position in the current frame image; calling a full-image detection model or a ROI detection model to detect the current frame image and identify multiple candidate traffic lights in the current frame image.

[0068] In this embodiment, a tracking box can be displayed at the predicted position in the current frame image, for example, a traffic light tracking box can be displayed at the predicted position in the current frame image.

[0069] In this embodiment, a full-image detection model or a ROI detection model can be invoked to detect the current frame image and identify multiple candidate traffic lights in the current frame image. For example, based on the configuration, a full-image detection model or a ROI detection model can be invoked to detect the current frame. After obtaining the results, all detection boxes are sorted according to confidence level, and a maximum output detection number is preset to ensure the accuracy of detection.

[0070] In this embodiment, candidate traffic lights can be obtained by performing preliminary screening on all candidate detection boxes. For example, all detection boxes with confidence scores below the threshold and all detection boxes whose lower endpoints are a certain distance below the vanishing point can be deleted. Based on common sense about traffic, traffic lights will not appear below the horizon in the image.

[0071] As an optional implementation, the full-image detection model is used to detect at least one candidate traffic light in the current frame image that is associated with the tracking box, and to identify the candidate traffic light pair detection box.

[0072] In this embodiment, the input to the full-image detection model can be the original image, and it will detect traffic lights across the entire image, outputting only the traffic light frame information. This model is called every other frame, and the calling period can also be configured to be longer. In image frames where the full-image detection model is not called, the ROI detection model will be called to perform supplementary detection.

[0073] As an optional implementation, the ROI detection model is used to expand the region based on the position of the tracking box to obtain an expanded box, identify at least one candidate traffic light located in the expanded box, and mark the candidate traffic light pair detection box.

[0074] In this embodiment, the input to the ROI detection model can be an ROI, which is obtained by predicting the possible location of the traffic light frame in the current frame and appropriately expanding its range. The model only outputs traffic light frame information, and its function is basically the same as that of the full-image detection model. However, this model has high computational efficiency, low computational resource requirements, and can quickly complete the detection, thereby improving the module's operating efficiency.

[0075] As an optional implementation, the method further includes: screening multiple candidate traffic lights, the step of which includes at least one of the following: sorting the multiple candidate traffic lights according to their confidence level, and obtaining traffic lights with a confidence level exceeding a threshold according to the sorting result; deleting traffic lights located below the horizon in the current frame image.

[0076] In this embodiment, multiple candidate traffic lights can be sorted according to their confidence level, and traffic lights with confidence levels exceeding a threshold can be obtained based on the sorting results. For example, when performing preliminary screening on all candidate detection boxes, all detection boxes can be sorted according to their confidence level, and all detection boxes with confidence levels below the threshold can be deleted, thereby obtaining traffic lights with confidence levels exceeding the threshold.

[0077] In this embodiment, traffic lights located below the horizon in the current frame image can be deleted. For example, when performing preliminary screening of all candidate detection boxes, all detection boxes whose lower endpoints are a certain distance below the vanishing point can be deleted: based on common sense about traffic, traffic lights will not appear below the horizon in the image.

[0078] As an optional implementation, step S105 involves determining the target traffic light from multiple candidate traffic lights. This method includes: displaying a tracking box and multiple detection boxes in the current frame image, wherein the tracking box is used to represent the traffic light frame information of the traffic light to be tracked in the current frame image, and the detection boxes are used to represent the traffic light frame information of the candidate traffic lights in the current frame image; calling a type recognition model to classify the multiple detection boxes; using a Hungarian algorithm to sequentially match the classification results with the tracking boxes to determine the target detection box that matches the tracking box from the multiple detection boxes; and identifying the target traffic light located in the current frame image based on the target detection box.

[0079] In this embodiment, a tracking box and multiple detection boxes can be displayed in the current frame image. For example, an MHT tree structure can be constructed by traversing the tracking box and the detection box in two layers to realize a one-to-many matching relationship between the tracking box in the previous frame of the traffic light and the current detection box, thereby displaying the tracking box and multiple detection boxes in the current frame image.

[0080] In this embodiment, a type recognition model can be invoked to classify multiple detection boxes. For example, the classification model can obtain the outer frame of the traffic light box through a full-image detection model or an ROI detection model. This outer frame will be used as the input of this model. In this model, the light bulbs inside the light box will be detected and classified according to their color and category (round light, arrow light, etc.), and finally the detection results will be output.

[0081] In this embodiment, the Hungarian algorithm can be used to sequentially match the classification results with the tracking boxes to determine the target detection boxes that match the tracking boxes from multiple detection boxes. For example, the Hungarian algorithm can be used to further refine one-to-many matching into one-to-one matching. The Hungarian algorithm can be a combinatorial optimization algorithm that solves the task allocation problem in polynomial time. Through this algorithm and the set cost function, the one-to-one matching between tracking boxes and detection boxes can be generated globally in the optimal way. The distance between the center points of the two boxes is used as the cost function to match each pair of tracking boxes and detection boxes that are closest to each other.

[0082] In this embodiment, the target traffic light in the current frame image can be identified based on the target detection box. For example, the distance between the center points of two boxes can be used as the cost function to match each pair of tracking boxes and detection boxes that are closest to each other, thereby identifying the target traffic light in the current frame image.

[0083] As an optional implementation, the method further includes: determining a newly identified target traffic light in the current frame image based on at least one traffic light that did not match successfully, and creating tracking information for the target traffic light.

[0084] In this embodiment, a newly identified target traffic light in the current frame image can be determined based on at least one traffic light that has not been successfully matched, and tracking information of the target traffic light can be created. For example, for the previously generated candidate detection box list, if there are detection boxes that have not been matched, a voting queue is generated for them and a tracking box is created. The voting queue is a sliding window of fixed length, which is used to reduce false detections and improve the accuracy of attribute output.

[0085] For example, creating a voting queue of length 5 requires that for all traffic light objects, there must be a matching detection result for at least 5 consecutive frames. The final color output will be determined by the attribute with the most votes from the 5 voting outputs. For instance, if the voting results for a detected traffic light for 5 consecutive frames are: black, yellow, black, green, green, then the final output green will be used as the detection result. After obtaining the color attribute, a Kalman filter will be used, taking the width and height of the tracking box as input, to predict its width and height in the current frame. The detection result will then be used as the observation value, and the two will be combined using Kalman gain to obtain the final detection box size information.

[0086] In the embodiments described above, the vehicle displacement change information and vehicle orientation angle change information between the current frame image and the previous frame image are calculated based on the vehicle motion information during the vehicle's driving process. Based on the position information, vehicle displacement change information, and vehicle orientation angle change information of the traffic light to be tracked in the previous frame image, the predicted position of the traffic light to be tracked in the current frame image is determined. A tracking box and multiple detection boxes are displayed in the current frame image, wherein the tracking box represents the traffic light frame information of the traffic light to be tracked in the current frame image, and the detection boxes represent the traffic light frame information of candidate traffic lights in the current frame image. A type recognition model is invoked to classify the multiple detection boxes. A Hungarian... The algorithm sequentially matches the classification results with the tracking boxes to determine the target detection box that matches the tracking box from multiple detection boxes. Based on the target detection box, the target traffic light located in the current frame image is identified. In other words, in this application, an inter-frame detection model is designed. The input of this model is the ROI generated based on the predicted value. One-to-many matching is performed using an MHT tree, followed by candidate box filtering, and finally one-to-one matching. Basic computer vision principles such as pinhole imaging and camera vanishing point are used for prediction, tracking, and filtering, reducing the demand for and dependence on computing resources. This solves the technical problem of low reliability in traffic light detection and achieves the technical effect of improving the reliability of traffic light detection.

[0087] Example 2

[0088] The traffic light detection method of this disclosure will be further described below with reference to preferred embodiments.

[0089] Currently, traffic light detection requires obtaining the location information of the traffic lights through high-precision maps and then projecting them onto an image plane through coordinate transformation for subsequent processing. This reliance on high-precision maps significantly increases vehicle costs. Furthermore, this approach demands real-time accuracy from the high-precision maps; changes in the location or type of traffic lights can lead to functional failures and pose significant security risks.

[0090] The solution designed in this invention is independent of high-precision maps. By statistical voting on detection results, using historical frame information and vehicle kinematics information for prediction and tracking, and designing separate detection and classification models, it achieves stable detection of traffic lights without high-precision maps, meeting the safety requirements of autonomous driving while significantly reducing development costs.

[0091] The proposed solution in this invention mainly consists of three parts: preprocessing, detection process, and post-processing. For the first frame of image information, it is detected, and the detection results are saved in a list without being output, awaiting processing in conjunction with subsequent detection results. When historical detection information already exists, the complete traffic light detection process can be invoked.

[0092] The main task of preprocessing is to predict the possible locations of the two types of targets in the current frame based on the detection results of traffic light boxes and traffic bulbs obtained in the previous frame. During prediction, it is necessary to obtain the vehicle's kinematic information, including vehicle speed and yaw rate of change, and then predict the possible location of the detection box in the next frame based on the principle of similar triangles.

[0093] The preprocessing flow is as follows:

[0094] Figure 2 This is a schematic diagram illustrating the relationship between an image coordinate system, a camera coordinate system, and a world coordinate system according to an embodiment of this disclosure, such as... Figure 2 As shown in the figure, the relationship between the image coordinate system, camera coordinate system, and world coordinate system is illustrated. In the world coordinate system, point P is projected onto point P in the image plane through the pinhole imaging principle of the camera. Because image points are obtained through the pinhole imaging principle, the position of objects in the world coordinate system in the image plane can be calculated using similar triangles. Based on this principle, the following calculation process is performed:

[0095] First, the vehicle's distance traveled between two frames and the change in its orientation angle were calculated using vehicle kinematics information. Then, based on the principle of similar triangles, the distance between the traffic light detected in the previous frame and the vehicle body in the VCS coordinate system (vehicle coordinate system) was estimated as follows:

[0096]

[0097] Where, x vcs Let fv be the distance between the traffic light and the vehicle body in the VCS coordinate system, fv be the image distance of the camera used, and s be the distance between the traffic light and the vehicle body. H The height of the traffic light frame in the VCS coordinate system (which can be estimated based on empirical values), s imH This represents the height of the traffic light frame in the image coordinate system.

[0098] Similarly, by acquiring information such as the vanishing point of the camera, the yaw rate of the vehicle, and the lateral position of the traffic light in the image, the lateral position of the traffic light in the VCS coordinate system can be calculated. After obtaining the position information of the traffic light in the VCS coordinate system of the previous frame, the possible position of the traffic light frame in the image coordinate system of the current frame can be predicted again using the vehicle's kinematic information and the principle of similar triangles. The specific implementation scheme is as follows:

[0099]

[0100] Among them, y im Let y be the predicted ordinate of the center point of the traffic light frame in the current frame within the image coordinate system. vp The vanishing point of the camera. This represents the ordinate of the center point of the traffic light detection box in the previous frame within the image coordinate system. yaw represents the position of the traffic light on the x-axis in the VCS coordinate system of the previous frame, L represents the distance the vehicle travels between the two frames. rate This represents the rate of change of the vehicle's yaw angle between two frames.

[0101] Similarly, the x-coordinate of the predicted center point of the traffic light frame in the current frame can be calculated in the image coordinate system, thus ultimately obtaining the possible location of the current frame detection box predicted by the detection box in the previous frame.

[0102] Subsequently, the above process is repeated to predict the traffic light detection box obtained from the previous frame, and the result is stored in an array of member variables. The predicted box will then be used for detection and matching.

[0103] The testing process is as follows:

[0104] This method uses three detection models, as follows:

[0105] 1) Full-Image Detection Model: The model takes the original image as input and will detect traffic lights across the entire image, outputting only the traffic light frame information. This model is called every other frame, and the calling period can also be configured to be longer. In image frames where the full-image detection model has not been called, the ROI detection model will be called to perform supplementary detection.

[0106] 2) ROI Detection Model: The model takes ROI as input and obtains it by predicting the possible location of traffic light frames in the current frame and appropriately expanding its range. This model only outputs traffic light frame information, and its function is basically the same as the full-image detection model. However, this model has high computational efficiency, low computational resource requirements, and can quickly complete the detection, thereby improving the module's operating efficiency.

[0107] 3) Classification Model: The outer frame of the traffic light box is obtained through the full-image detection model or the ROI detection model. This outer frame will be used as the input of this model. In this model, the light bulbs inside the light box will be detected and classified according to their color and category (round light, arrow light, etc.). Finally, the detection results will be output.

[0108] In the detection section, the first step is to call either a full-image detection model or a ROI detection model based on the configuration to detect the current frame. After obtaining the results, all detection boxes are sorted according to their confidence scores, and a maximum output detection count is preset to ensure detection accuracy. Subsequently, all candidate detection boxes undergo preliminary screening, as follows:

[0109] Step 1: Delete all detections with confidence scores below the threshold;

[0110] Step 2: Delete all detection boxes whose lower endpoints are a certain distance below the vanishing point: Based on common sense about traffic, traffic lights will not appear below the horizon in the image.

[0111] After initial screening, an MHT tree structure is constructed by traversing the tracking and detection boxes in a two-level manner to achieve a one-to-many matching relationship between the tracking box in the previous frame and the current detection box. Before matching, the aspect ratio and area of ​​the tracking and detection boxes are calculated and compared. If the difference is greater than a threshold, it indicates that the two boxes are significantly different, and they will not be matched.

[0112] Next, the distance between the center points of the two bounding boxes is calculated and compared with a preset threshold. If it is less than the threshold, the current detection box is associated with the tracking box. After traversing all tracking and detection boxes, the MHT tree is initially constructed. However, if the number of detection results exceeds a preset value, all detection results need to be reordered according to the following rules (sorted by importance), as follows:

[0113] 1) Whether the aspect ratio of the detection frame is within the threshold range (whether it is a regular traffic light shape);

[0114] 2) Is the detection box located at the image edge? (Image edges are distorted, and generally, traffic lights at the image edge are not associated with the current driving direction. Detection boxes that are not at the image edge are output first.)

[0115] 3) Whether the length and width of the detection box are greater than the threshold (smaller length and width mean that the current traffic light is smaller in the image, that is, the actual distance is farther, and traffic lights that are closer are output first);

[0116] 4) Is the area of ​​the detection box greater than the threshold? (A smaller area means a greater actual distance, so traffic lights that are closer will be output first.)

[0117] 5) Is the detection box close to the center of the image? (Detection boxes that are closer to the center of the image are more relevant to the current driving situation and should be output first).

[0118] After reordering according to the above rules, the boxes are added to the candidate list in descending order of importance until the detection limit is reached. If the detection limit is still not reached, additional boxes can be added based on the tracking bounding box information, as follows:

[0119] Traverse all tracking boxes and associated detection boxes in the MHT tree, find all detection boxes of suitable size that are not at the edge of the image, calculate the IOU between the box and the associated tracking box, and if it is less than the threshold and the tracking box has existed for more than 1 frame, add the current detection box as a candidate box to the output list until the list reaches the upper limit.

[0120] At this point, all traffic light frames to be detected are obtained. Then, the classification model is called to detect and classify the light bulbs inside the detected traffic light frames.

[0121] The post-processing workflow is as follows:

[0122] The main function of the post-processing module is to complete the one-to-one matching of traffic light tracking boxes and detection boxes, add tracking box and detection box information for newly appearing traffic lights, vote and store detection attributes, correct the width and height of the output detection boxes through Kalman filtering, and finally output all detection results.

[0123] First, the one-to-many matching is further refined into one-to-one matching using the Hungarian algorithm. The Hungarian algorithm is a combinatorial optimization algorithm that solves the task allocation problem in polynomial time. Using this algorithm and a defined cost function, a globally optimal one-to-one matching of tracking boxes and detection boxes can be generated. In this module, the distance between the center points of two boxes is used as the cost function to match each pair of tracking boxes and detection boxes with the closest relative distance.

[0124] Subsequently, for the previously generated list of candidate detection boxes, if there are any detection boxes that have not been matched, a voting queue is generated for them, and a tracking box is created.

[0125] The voting queue is a fixed-length sliding window designed to reduce false positives and improve the accuracy of attribute output. For example, if a voting queue of length 5 is created, all traffic light objects require at least 5 consecutive frames with matching detection results. The final color output will be determined by the attribute with the most votes from these 5 frames. For instance, if a detected traffic light has 5 consecutive frames with the following voting results: black, yellow, black, green, green, then green will be the final output as the detection result.

[0126] After obtaining the color attributes, a Kalman filter is applied, using the width and height of the tracking bounding box as input to predict its width and height in the current frame. The detection result is then used as the observation value, and the two are combined using Kalman gain to obtain the final detection bounding box size information.

[0127] The specific Kalman filter process is as follows:

[0128] (Predicting state changes);

[0129] (Predicting and estimating the covariance matrix);

[0130] (Measurement residuals);

[0131] (Measurement of residual covariance);

[0132] (Optimal Kalman gain);

[0133] (Updated state estimate);

[0134] P k|k =(IK k H h )P k|k-1 (Updated covariance estimate).

[0135] This completes the entire traffic light processing procedure and outputs the final traffic light test results.

[0136] Figure 3 This is a flowchart of a traffic light detection and recognition method for autonomous driving without a high-precision map, according to an embodiment of this disclosure. Figure 3 As shown, the steps of this method may include the following:

[0137] Step S301: Predict the position of the tracking box obtained in the previous frame;

[0138] Step S302: Expand the ROI for the predicted location and call the detection model;

[0139] Step S303: Filter the detection boxes based on prior information;

[0140] Step S304: Establish a one-to-many matching of tracking boxes and detection boxes based on the MHT tree structure;

[0141] Step S305: Select the detection box to be identified;

[0142] Step S306: Call the type recognition model for fine detection and classification;

[0143] Step S307: Establish a one-to-one match using the Hungarian algorithm;

[0144] Step S308: Establish tracking information for the newly appearing traffic lights;

[0145] Step S309: Adjust the size of the detection frame using Kalman filtering;

[0146] Step S310: Output the result.

[0147] Figure 4A This is a schematic diagram of the previous frame image of a traffic light to be tracked according to an embodiment of the present disclosure, such as... Figure 4A As shown, the tracking box of the traffic light to be tracked is displayed in the previous frame image, where the tracking box is used to represent the traffic light frame information of the traffic light to be tracked.

[0148] Figure 4BThis is a schematic diagram of the current frame image of a traffic light to be tracked according to an embodiment of the present disclosure, such as... Figure 4B As shown, a tracking box and multiple detection boxes are displayed in the current frame image. The tracking box represents the traffic light frame information of the traffic light to be tracked in the current frame image, and the detection boxes represent the traffic light frame information of the candidate traffic lights in the current frame image. Then, a type recognition model is called to classify the multiple detection boxes. The classification results are then matched with the tracking box sequentially using the Hungarian algorithm to determine the target detection box that matches the tracking box from the multiple detection boxes. Finally, based on the target detection box, the target traffic light located in the current frame image is identified.

[0149] Figure 4C This is a schematic diagram illustrating the application effect of a traffic light detection and recognition method for autonomous driving without high-precision maps according to an embodiment of the present disclosure. As shown in 4C, by designing a small ROI model that can quickly detect traffic lights, the full-map traffic light detection model can be called every other frame. The traffic light frames detected by the model are combined with vehicle motion information to predict their positions, and the small ROI model is called in the next frame for detection. Through this alternating detection design, the computational resource requirements are significantly reduced, the operating efficiency is effectively improved, and the real-time performance of the traffic light module is guaranteed.

[0150] In this embodiment, a frame-separated detection model is designed. The input of the model is the ROI generated based on the predicted value. The model uses MHT tree for one-to-many matching, then performs candidate box filtering, and finally performs one-to-one matching. It uses basic computer vision principles such as pinhole imaging and camera vanishing point for prediction, tracking and filtering, thereby reducing the demand for and dependence on computing resources. This solves the technical problem of low reliability in traffic light detection and achieves the technical effect of improving the reliability of traffic light detection.

[0151] It should be noted that the steps in the above embodiments of this disclosure do not necessarily have to be performed in the above order. For example, the order of steps S606 and S607 can be interchanged.

[0152] Example 3

[0153] This disclosure also provides an embodiment for performing Figure 1 The illustrated embodiment describes a method for detecting traffic lights and an apparatus for detecting traffic lights.

[0154] Figure 5 This is a schematic diagram of a traffic light detection device according to an embodiment of the present disclosure, as shown below. Figure 5 As shown, the device 50 for detecting traffic lights may include: a data acquisition unit 51, a selection unit 52, a prediction unit 53, a first determination unit 54, and a second determination unit 55.

[0155] The acquisition unit 51 acquires vehicle motion information and current frame images during the vehicle's operation, wherein the frame images are obtained by a camera installed on the vehicle.

[0156] The selection unit 52 selects the previous frame image adjacent to the current frame image and identifies at least one traffic light to be tracked from the previous frame image.

[0157] The prediction unit 53 predicts the position of the traffic light to be tracked in the current frame image based on the position information of the traffic light to be tracked in the previous frame image and the vehicle motion information of the vehicle.

[0158] The first determining unit 54 determines multiple candidate traffic lights in the current frame image based on the predicted position of the traffic light to be tracked in the current frame image, wherein the traffic light to be tracked and the multiple candidate traffic lights in the current frame have a one-to-many matching relationship.

[0159] The second determining unit 55 determines the target traffic light from a plurality of candidate traffic lights, wherein the target traffic light represents the traffic light that matches the traffic light to be tracked in the current frame image.

[0160] Optionally, the prediction unit 53 includes: a calculation module and a first determination module, wherein the first determination module includes: a first display module, wherein the first display module includes: a display subunit and a first recognition subunit, wherein the first recognition subunit includes: a first recognition submodule, a second recognition submodule, a sorting submodule, and a deletion submodule. The calculation module is used to calculate, based on the vehicle motion information during the vehicle's movement, the vehicle displacement change information and the vehicle orientation angle change information between the current frame image and the previous frame image; the first determination module is used to determine the predicted position of the traffic light to be tracked in the current frame image based on the position information, vehicle displacement change information, and vehicle orientation angle change information of the traffic light to be tracked in the previous frame image; the display module is used to display the tracking box of the traffic light to be tracked in the previous frame image, wherein the tracking box is used to represent the traffic light frame information of the traffic light to be tracked; the display subunit is used to display the tracking box at the predicted position in the current frame image; the first recognition subunit is used to call the full-image detection model or the ROI detection model. The system comprises the following components: a first identification submodule, which can be a full-image detection model, for detecting at least one candidate traffic light associated with a tracking box in the current frame image and identifying the candidate traffic light pair detection boxes; a second identification submodule, which can be a ROI detection model, for expanding the region based on the position of the tracking box to obtain an expanded box, identifying at least one candidate traffic light located within the expanded box, and identifying the candidate traffic light pair detection boxes; a sorting submodule, for sorting the multiple candidate traffic lights according to their confidence levels, and for obtaining traffic lights with confidence levels exceeding a threshold according to the sorting results; and a deletion submodule, for deleting traffic lights located below the horizon in the current frame image.

[0161] Optionally, the second determining unit 55 includes: a second display module, a calling module, a second determining module, and a recognition module, wherein the second determining module includes: a first determining subunit. The second display module is used to display a tracking box and multiple detection boxes in the current frame image, wherein the tracking box represents the traffic light frame information of the traffic light to be tracked in the current frame image, and the detection boxes represent the traffic light frame information of candidate traffic lights in the current frame image; the calling module is used to call a type recognition model to classify the multiple detection boxes; the second determining module is used to use a Hungarian algorithm to sequentially match the classification results with the tracking boxes, determining the target detection box that matches the tracking box from the multiple detection boxes; the recognition module is used to identify the target traffic light located in the current frame image based on the target detection box; the first determining subunit is used to determine the newly identified target traffic light in the current frame image based on at least one traffic light that did not match successfully, and create tracking information for the target traffic light.

[0162] In the traffic light detection device of this embodiment, the acquisition unit 51 acquires vehicle motion information and the current frame image during vehicle operation, wherein the frame image is captured by a camera installed on the vehicle; the selection unit 52 selects the previous frame image adjacent to the current frame image and identifies at least one traffic light to be tracked from the previous frame image; the prediction unit 53 predicts the predicted position of the traffic light to be tracked in the current frame image based on the position information of the traffic light to be tracked in the previous frame image and the vehicle motion information of the vehicle; the first determination unit 54 determines multiple candidate traffic lights in the current frame image based on the predicted position of the traffic light to be tracked in the current frame image, wherein the traffic light to be tracked and the multiple candidate traffic lights in the current frame have a one-to-many matching relationship; the second determination unit 55 determines the target traffic light from the multiple candidate traffic lights, wherein the target traffic light represents the traffic light that matches the traffic light to be tracked in the current frame image, thereby solving the technical problem of low reliability of traffic light detection and achieving the technical effect of improving the reliability of traffic light detection.

[0163] Example 4

[0164] According to embodiments of the present invention, an electronic device is also provided. The electronic device is installed in a vehicle and includes: one or more processors; a storage device for storing one or more programs; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to perform the method for detecting traffic lights according to embodiments of the present disclosure.

[0165] Example 5

[0166] According to embodiments of this disclosure, this disclosure also provides a vehicle including a method for detecting traffic lights according to embodiments of this disclosure.

[0167] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0168] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0169] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be indirect couplings or communication connections through some interfaces, units, or models, and may be electrical or other forms.

[0170] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0171] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0172] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0173] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method of detecting a traffic light, characterized by, The method comprises the following steps: Collecting vehicle motion information and a current frame image during driving of a vehicle, wherein the frame image is captured by a camera installed on the vehicle; Selecting a previous frame image adjacent to the current frame image, and identifying at least one traffic light to be tracked from the previous frame image; Based on position information of the traffic light to be tracked in the previous frame image and vehicle motion information of the vehicle, a predicted position of the traffic light to be tracked in the current frame image is predicted; Based on the predicted position of the traffic light to be tracked in the current frame image, a plurality of candidate traffic lights in the current frame image are determined, wherein the traffic light to be tracked and the plurality of candidate traffic lights in the current frame are in a one-to-many matching relationship; A target traffic light is determined from the plurality of candidate traffic lights, wherein the target traffic light represents a traffic light matched with the traffic light to be tracked in the current frame image; The determination of the target traffic light from the plurality of candidate traffic lights comprises: constructing a metric hash tree structure through double-layer traversal of a tracking box and a plurality of detection boxes, realizing a one-to-many matching relationship between the tracking box in the previous frame image and the plurality of detection boxes in the current frame image, displaying the tracking box and the plurality of detection boxes in the current frame image, wherein the tracking box is used to represent traffic light box information of the traffic light to be tracked in the current frame image, the detection box is used to represent traffic light box information of the candidate traffic light in the current frame image, and the traffic light box information is used to at least represent position information of a traffic light in the current frame image; a type recognition model is called to classify the plurality of detection boxes to obtain a classification result; a Hungarian algorithm is used, a distance between a center point of the tracking box and a center point of the detection box is taken as a cost function, the classification result is matched with the tracking box in sequence, a target detection box matched with the tracking box is determined from the plurality of detection boxes by matching the tracking box with the detection box with the closest relative distance; and based on the target detection box, the target traffic light located in the current frame image is identified; The method further comprises: based on at least one traffic light that fails to match, determining a newly identified target traffic light in the current frame image, and creating tracking information of the newly identified target traffic light. The predicted position of the traffic light to be tracked in the current frame image is predicted based on the position information of the traffic light in the previous frame image and vehicle motion information of the vehicle, including: calculating vehicle displacement change information and vehicle orientation angle change information of the vehicle between the current frame image and the previous frame image based on vehicle motion information of the vehicle during driving, wherein the vehicle displacement change information and the vehicle orientation angle change information are calculated by the vanishing point of the camera, the yaw rate of the vehicle, and the lateral position of the traffic light in the frame image; determining the predicted position of the traffic light to be tracked in the current frame image based on the position information of the traffic light to be tracked in the previous frame image, the vehicle displacement change information and the vehicle orientation angle change information.

2. The method of claim 1, wherein, The tracking frame of the traffic light to be tracked is displayed in the previous frame image, wherein the tracking frame is used to represent traffic light frame information of the traffic light to be tracked.

3. The method according to claim 1 or 2, characterized in that, Based on the predicted position of the traffic light to be tracked in the current frame image, a plurality of candidate traffic lights in the current frame image are determined, including: The tracking frame is displayed at the predicted position in the current frame image; A full-image detection model or an ROI detection model is called to detect the current frame image to identify a plurality of candidate traffic lights in the current frame image.

4. The method of claim 3, wherein, The full-image detection model is used to detect at least one candidate traffic light in the current frame image associated with the tracking frame and identify a detection frame of the candidate traffic light.

5. The method of claim 3, wherein, The ROI detection model is used to expand the region based on the position of the tracking frame to obtain an expanded frame, identify at least one candidate traffic light located in the expanded frame, and identify a detection frame of the candidate traffic light.

6. The method of claim 3, wherein, The method further includes: The plurality of candidate traffic lights are screened, including at least one of the following: The plurality of candidate traffic lights are sorted according to confidence, and traffic lights with confidence exceeding a threshold value are obtained according to the sorting result. Traffic lights below the horizon in the current frame image are deleted.

7. An electronic device, comprising: The electronic device is installed in a vehicle, and the electronic device includes: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors execute the method for detecting traffic lights according to any one of claims 1-6.

8. A vehicle characterized by comprising: The stored program is run in the vehicle, and when the program is running, the processor of the device controls the execution of the method for detecting traffic lights according to any one of claims 1-6.

Images