A traffic light shielding tracking method and device and an unmanned vehicle

Abstract

The present application relates to the technical field of artificial intelligence, in particular to a traffic light occlusion tracking method and device and an unmanned vehicle, which mainly acquires a detection frame of a traffic light in a current frame image through a target detection algorithm, then acquires a first prediction frame of the traffic light in a previous frame image which is lost for a preset number of frames and mapped to the current frame image, and finally matches the detection frame with the first prediction frame, if the detection frame and the first prediction frame are matched successfully, it is determined that the traffic light corresponding to the first prediction frame is tracked successfully, and the state of the detection frame is occluded. Thus, the occluded traffic light is quickly identified, and the safety of the unmanned vehicle is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a traffic light occlusion tracking method and device and an unmanned vehicle. BACKGROUND

[0002] With the continuous development of the field of artificial intelligence, unmanned vehicles are also widely used, but in the process of unmanned vehicle movement, how to track the current traffic light is a big problem. In the process of the unmanned vehicle, the traffic light may be occluded, so that the unmanned vehicle cannot make a judgment at the first time, which reduces the safety of the unmanned vehicle. SUMMARY

[0003] The embodiments of the present application mainly provide a traffic light occlusion tracking method, device and unmanned vehicle, aiming at solving the technical problems of slow recognition speed and low safety performance of the existing technology.

[0004] To solve the above technical problems, one technical scheme adopted by the embodiments of the present application is to provide a traffic light occlusion tracking method applied to an unmanned vehicle, the method comprising:

[0005] obtaining a detection box of a traffic light in a current frame image based on a target detection algorithm;

[0006] obtaining a first prediction box of the traffic light in the current frame image, which is mapped from a previous frame image missing a preset number of frames;

[0007] matching the detection box with the first prediction box, if the detection box and the first prediction box are matched successfully, determining that the traffic light corresponding to the first prediction box is tracked successfully, and the state of the detection box is occluded.

[0008] Optionally, the matching of the detection box with the first prediction box, if the detection box and the first prediction box are matched successfully, determining that the traffic light corresponding to the first prediction box is tracked successfully, and the state of the detection box is occluded, comprises:

[0009] obtaining the categories corresponding to the detection box and the first prediction box;

[0010] obtaining the confidence of the detection box, and dividing the detection box into a high-score box and a low-score box according to the confidence;

[0011] matching the high-score box and the first prediction box with the same category to obtain a first matching result according to the categories of the high-score box and the first prediction box;

[0012] According to the category of the high-scored bounding box and the first prediction box, the high-scored bounding box and the first prediction box of the same category are matched to obtain a matching result;

[0013] A detection box not matched with the first prediction box is obtained, and the detection box is matched with the first prediction box not matched with the low-scored bounding box to obtain a third matching result;

[0014] The first matching result, the second matching result and the third matching result are integrated to determine a tracking result of a detection box occluded in the current frame image.

[0015] Optionally, the matching of the high-scored bounding box and the first prediction box of the same category according to the category of the high-scored bounding box and the first prediction box comprises:

[0016] According to the category of the detection box and the first prediction box, a high-scored bounding box of the same category as the first prediction box is obtained;

[0017] First position information and second position information of the high-scored bounding box and the first prediction box of the same category in the current frame image are obtained;

[0018] The coincidence degree between the first position information and the second position information is calculated, and the high-scored bounding box and the first prediction box with a coincidence degree greater than a first preset threshold value and the high-scored bounding box and the first prediction box with a coincidence degree equal to a second preset threshold value are obtained;

[0019] The high-scored bounding box and the first prediction box with a coincidence degree greater than the first preset threshold value are matched, and if the high-scored bounding box and the first prediction box are matched successfully, the high-scored bounding box is confirmed to be in a tracked state;

[0020] The Euclidean distance of the high-scored bounding box and the first prediction box with a coincidence degree equal to the second preset threshold value is calculated, and when the Euclidean distance is less than a preset distance, the high-scored bounding box is confirmed to be in a tracked state.

[0021] Optionally, the matching of the low-scored bounding box and the first prediction box not matched with the high-scored bounding box of the same category according to the category of the low-scored bounding box and the first prediction box comprises:

[0022] According to the category of the detection box and the first prediction box, a low-scored bounding box of the same category as the first prediction box is obtained;

[0023] Third position information and fourth position information of the low-scored bounding box and the first prediction box not matched with the high-scored bounding box of the same category in the current frame image are obtained;

[0024] calculate coincidence between the third position information and the fourth position information, and obtain the low frame and the first prediction frame which are not matched with the high frame and whose coincidence is greater than a third preset threshold, and the low frame and the first prediction frame which are not matched with the high frame and whose coincidence is equal to a second preset threshold;

[0025] match the low frame and the first prediction frame which are not matched with the high frame and whose coincidence is greater than the third preset threshold, and if the low frame and the first prediction frame which are not matched with the high frame are matched successfully, confirm that the low frame is in a tracked state;

[0026] calculate the Euclidean distance of the low frame and the first prediction frame which are not matched with the high frame and whose coincidence is equal to the second preset threshold, and if the Euclidean distance is less than the preset distance, confirm that the low frame is in the tracked state.

[0027] Optionally, the method further comprises:

[0028] obtain the fifth position information of the detection frame which is not matched with the first prediction frame in the current frame image, wherein the detection frame which is not matched with the first prediction frame comprises a high frame and a low frame which are not matched with the first prediction frame;

[0029] obtain the first prediction frame which is not matched with the low frame, and the sixth position information of the first prediction frame in the current frame image;

[0030] calculate the coincidence of the fifth position information and the sixth position information, and obtain the first prediction frame and the detection frame whose coincidence is greater than a fourth preset threshold, and the first prediction frame and the detection frame whose coincidence is equal to the second preset threshold;

[0031] match the detection frame and the first prediction frame whose coincidence is greater than the fourth preset threshold, and if the detection frame and the first prediction frame are matched successfully, confirm that the detection frame is in the tracked state;

[0032] calculate the Euclidean distance of the detection frame and the first prediction frame whose coincidence is equal to the second preset threshold, and if the Euclidean distance is less than the preset distance, confirm that the detection frame is in the tracked state.

[0033] Optionally, the method further comprises:

[0034] obtain the detection frame of the current frame image and the second prediction frame of the tracked traffic light in the previous frame image which is mapped to the current frame image;

[0035] The second prediction box is matched with the detection box to obtain the second prediction box that does not match the detection box. The traffic light corresponding to the unmatched second prediction box is a traffic light that has lost a preset number of frames.

[0036] The second prediction box that failed to match is used as the prediction box for the traffic light in the next frame of the current frame image, and the traffic light in the next frame image is tracked based on the prediction box.

[0037] Optionally, after obtaining the detection bounding box of the traffic light in the current frame image, the method further includes:

[0038] Obtain all detection boxes in the current frame image;

[0039] The overlap of the detection boxes is calculated to determine whether there are overlapping detection boxes in the current frame image;

[0040] When there are overlapping detection boxes in the current frame image, the detection boxes with higher confidence are retained based on the confidence level of the detection boxes.

[0041] To solve the above-mentioned technical problems, another technical solution adopted in the embodiments of the present invention is: providing a tracking device for traffic light obstruction, applied to an autonomous vehicle, the device comprising:

[0042] The first acquisition module is used to acquire the detection boxes of traffic lights in the current frame image based on the target detection algorithm;

[0043] The second acquisition module is used to acquire the first prediction box of the traffic lights that have lost a preset number of frames in the previous frame image and map them to the current frame image.

[0044] The matching module is used to match the detection box with the first prediction box. If the detection box and the first prediction box are successfully matched, it is determined that the traffic light corresponding to the first prediction box is successfully tracked, and the state of the detection box is occluded.

[0045] To solve the above-mentioned technical problems, another technical solution adopted in the embodiments of the present invention is: providing an unmanned vehicle, comprising:

[0046] At least one processor; and,

[0047] A memory communicatively connected to the at least one processor; wherein,

[0048] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method as described above.

[0049] To solve the above technical problems, another technical solution adopted by the embodiment of the present application is to provide a non-volatile computer readable storage medium, which stores computer executable instructions, and when the computer executable instructions are executed by a processor, the processor executes the method as described above.

[0050] Different from the related art, the embodiment of the present application provides a traffic light occlusion tracking method and device and an unmanned vehicle, which mainly acquires a detection box of a traffic light in a current frame image through a target detection algorithm, then acquires a first prediction box of the traffic light in a previous frame image which is lost for a preset number of frames and mapped to the current frame image, and finally matches the detection box with the first prediction box, if the detection box and the first prediction box are matched successfully, it is determined that the traffic light corresponding to the first prediction box is tracked successfully, and the state of the detection box is occluded. Thus, the occluded traffic light is quickly identified, and the safety of the unmanned vehicle is improved. BRIEF DESCRIPTION OF DRAWINGS

[0051] One or more embodiments are illustrated by way of example with reference to the accompanying drawings, which are schematic and not intended to be limiting of the embodiments, and in which like reference numerals designate similar items in the figures, and wherein the drawings are not necessarily to scale, unless otherwise indicated. In the drawings:

[0052] Figure 1 is a method flowchart of a traffic light occlusion tracking method provided by the embodiment of the present application;

[0053] Figure 2 is a method flowchart of a traffic light occlusion tracking method provided by another embodiment of the present application;

[0054] Figures 3a-3e is a schematic diagram of continuous frame images acquired in a traffic light tracking process provided by the embodiment of the present application;

[0055] Figure 4 is a structural block diagram of a traffic light occlusion tracking device provided by the embodiment of the present application;

[0056] Figure 5 is a structural diagram of an unmanned vehicle provided by the embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to make the objectives, technical solutions and advantages of the present application clearer, further detailed description will be made to the present application in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0058] It should be noted that the various features of the embodiments of the present application can be combined, if not in conflict, and are within the scope of the present application. In addition, although the division of functional modules is made in the device schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a manner different from the module division in the device schematic diagram or the order in the flowchart.

[0059] Unless otherwise defined, all technical and scientific terms used in the present application have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. The term "and / or" used in the present application includes any and all combinations of one or more related listed items.

[0060] Please refer to Figure 1 , Figure 1 is a method flowchart of a traffic light occlusion tracking method provided by an embodiment of the present application. The traffic light occlusion tracking method is applied to an unmanned vehicle, as shown in Figure 1 The method comprises the following steps.

[0061] S11, based on a target detection algorithm, obtaining a detection box of a traffic light in a current frame image.

[0062] Specifically, the unmanned vehicle comprises a camera. During the travel of the unmanned vehicle, the camera is used to capture images of a current road section in real time, thereby obtaining a current frame image of the current road section. The current road section has a traffic light. Then, the current frame image is input into a target detection algorithm, thereby obtaining a detection box of the traffic light in the current frame image. Further, the detection box comprises a position, a category and a confidence of the traffic light. The position refers to coordinates, height and width of the traffic light in the frame image. The confidence refers to the importance of the detection box in the frame image. The category refers to the color of the traffic light.

[0063] The target detection algorithm can be divided into one-stage and two-stage. The two-stage algorithm includes the R-CNN (Region-CNN) series, which is a target detection technology based on a convolutional neural network (CNN), linear regression, and the like. The input image is processed through a network, and then a detection box is generated, which includes position and category information. The one-stage algorithm includes the Yolo (You Only Look Once) series, which is a target detection system based on a single neural network. The input image is processed through a candidate box generation network to generate a candidate box, and then the image is processed through a classification network to classify the content of the candidate box. Preferably, the Yolov5 network is used to process the current frame image to obtain the detection box of the traffic light in the current frame image.

[0064] After obtaining the detection box of the traffic light in the current frame image, the detection box needs to be converted to obtain the specific position, category, and score of the detection box corresponding to the traffic light in the frame image. According to the coordinates, height, and width of the detection box in the frame image, the specific position (x1, y1, x2, y2) of the detection box in the frame image is obtained, and the score of the detection box is obtained according to the confidence score, which can be in the range of 0-1. The specific position, category, and score of the detection box can be stored in a preset data structure, such as a strack (stack) that stores the specific position, category, and score of the detection box. The stack refers to a specific register, a data structure arranged in order, and only one end of the data item can be inserted and deleted. It strictly follows the principle of "first-in, last-out" to access and is used to temporarily store data and addresses.

[0065] In some embodiments, TensorRT can also be constructed in the target detection algorithm. TensorRT refers to an inference framework that can run on various GPU (graphics processing unit) hardware platforms. By using a model trained by a framework, the model is converted into the format of TensorRT, and then the model is run by using the TensorRT inference engine to improve the speed of the model running on the GPU. Therefore, by constructing TensorRT, the detection speed of the target detection algorithm on the frame image can be improved, thereby achieving the purpose of real-time detection of the frame image without missing frames.

[0066] In some embodiments, after the detection box of the traffic light in the current frame image is obtained, all detection boxes in the current frame image can be obtained, and then the coincidence degree of the detection boxes is calculated to determine whether there is a coincident detection box in the current frame image, and when there is a coincident detection box in the current frame image, the detection box with high confidence in the coincident detection box is retained based on the confidence of the detection box. Thus, the same traffic light having two detection boxes due to detection errors is avoided, and the accuracy of traffic light detection is improved.

[0067] S12, obtaining a first prediction box of a traffic light missing a preset number of frames in a previous frame image and mapped to a current frame image.

[0068] The first prediction box can be obtained by a Kalman filter. Specifically, a detection box of a traffic light missing a preset number of frames in a previous frame image is obtained, and the detection box is input into the Kalman filter, and then an estimated position of the traffic light missing the preset number of frames in the current frame image is obtained, thereby obtaining the first prediction box. The first prediction box includes a position, a category and a confidence, the position is the estimated position, and the category and the confidence are the category and the confidence of the detection box corresponding to the traffic light missing the preset number of frames. The Kalman filter is an algorithm for optimal estimation of system state by using linear system state equation and observing data of system input and output. Preferably, the preset number of frames can be 30 frames, that is, the traffic light missing 30 frames is mapped to the current frame image by the Kalman filter, thereby obtaining the first prediction box.

[0069] The state of the detection frame can be determined according to the state of the traffic light corresponding to the detection frame in the current frame. After the matching of the detection frame in the current frame image with the first prediction frame is completed, the state of the detection frame in the current frame image can be obtained. The state of the detection frame includes a tracked state, an added state, a lost state and a removed state. The added state refers to a detection frame newly appearing in the current frame image. The tracked state refers to the traffic light appearing in at least two frames of images. The lost state refers to that the first prediction frame is not matched in the current frame image matching. The removed state refers to that the detection frame in the lost state is not matched within a preset number of frames. That is, in the matching process of the detection frame and the first prediction frame, if the detection frame is successfully matched with the first prediction frame, the state of the detection frame is the tracked state. If the detection frame is not successfully matched with the first prediction frame, the state of the detection frame is the added state, and the state of the first prediction frame is the lost state. If the first prediction frame in the lost state is not matched within a preset number of frames, the state of the first prediction frame is the removed state. For example, when the first prediction frame in the lost state is not matched for more than 30 frames, the first prediction frame in the lost state is updated to the first prediction frame in the removed state.

[0070] S13, match the detection frame with the first prediction frame. If the detection frame is successfully matched with the first prediction frame, it is determined that the tracking of the traffic light corresponding to the first prediction frame is successful, and the state of the detection frame is occluded.

[0071] Specifically, in the process of tracking the traffic light, if the traffic light is lost for a preset number of frames, it can be considered that the traffic light is occluded. At this time, the first prediction frame corresponding to the traffic light lost for a preset number of frames is matched with the detection frame. If the matching is successful, it is considered that the traffic light lost for a preset number of frames is tracked in the current frame image, that is, the occluded traffic light is successfully tracked.

[0072] Please refer to Figure 2 , Figure 2 is a method flow chart of a traffic light occlusion tracking method provided by another embodiment of the application, as Figure 2 shown, comprising:

[0073] S131, obtaining the categories corresponding to the detection frame and the first prediction frame.

[0074] S132, obtaining the confidence of the detection frame, and dividing the detection frame into high-score frames and low-score frames according to the confidence.

[0075] According to the score of the detection frame, a detection threshold is set, and then the score of the detection frame in the stack is compared with the detection threshold, and finally the detection frame with the score greater than or equal to the detection threshold is determined as a high-score frame, and the detection frame with the score less than the detection threshold is determined as a low-score frame. Wherein, the detection threshold is an optimal value obtained after multiple experiments, by setting the detection threshold, the priority of the object similar to the tracked traffic light is ensured in the target detection of the current frame image, thereby improving the matching accuracy. Preferably, the detection threshold can be 0.7.

[0076] S133、According to the category of the high-score frame and the first prediction frame, the high-score frame and the first prediction frame with the same category are matched to obtain a first matching result.

[0077] After obtaining the category of the detection frame and the first prediction frame, according to the category, the high-score frame and the first prediction frame with the same category are matched to obtain a matching result. For example, the category of the detection frame is red light, and the first prediction frame includes red light and green light. When matching, the red light detection frame is compared with the red light first prediction frame to obtain a matching result.

[0078] Specifically, after obtaining the first prediction frame according to the previous frame image or the previous N frame images, the first prediction frame with the same category as the high-score frame is obtained, and then the high-score frame and the first prediction frame with the same category are matched. The trajectory matching refers to obtaining the matching value of the detection frame and the first prediction frame corresponding to the red-green light lost for a preset number of frames in the previous N frame images based on the Intersection over Union (IOU). When the matching value is greater than a first preset threshold, the high-score frame is subjected to a Hungarian matching algorithm with the first prediction frame, thereby obtaining a first matching result, wherein the first matching result refers to that the high-score frame and the first prediction frame are successfully matched. The Hungarian matching algorithm is a combinatorial optimization algorithm for solving task allocation problems in polynomial time. The first preset threshold is an optimal value obtained according to multiple experiments. By the first preset threshold, the accuracy of tracking the high-score frame is improved while avoiding detection errors caused by too high values. Preferably, the first preset threshold can be 0.8.

[0079] In some embodiments, since the camera also moves when the unmanned vehicle moves, the traffic light tracking is tracking a static traffic light by a dynamic camera. When tracking the traffic light, each movement of the camera will cause the traffic light to have a large pixel displacement in the image, so when the high-score frame and the first prediction frame are matched, if the matching value of the high-score frame and the first prediction frame is a second preset threshold, the Euclidean distance between the high-score frame and the first prediction frame can be calculated, and the Euclidean distance is compared with a preset distance, when the Euclidean distance is less than the preset distance, it is determined that the state of the high-score frame is tracked. And the degree of coincidence between the high-score frame and the first prediction frame is modified to a third preset threshold, so as to avoid the state of the tracked high-score frame being filtered out due to the numerical setting, thereby improving the accuracy of traffic light tracking. Wherein, the third preset threshold is less than the first preset threshold, and the second preset threshold and the third preset threshold are optimal values obtained through multiple experiments. Since the Euclidean distance is the distance between the high-score frame and the first prediction frame in the current frame image, the unit of the Euclidean distance is pixel. And the preset distance is set according to the speed of the current unmanned vehicle, preferably, the second preset threshold is 0, and the third preset threshold can be 0.5, when the speed of the unmanned vehicle is 30-40 kilometers per hour, the preset distance is 50 pixels.

[0080] Specifically, the first position information of the high-score frame in the current frame image and the second position information of the first prediction frame corresponding to the current frame image are obtained, and then the matching value of the high-score frame and the first prediction frame of the same category is calculated according to the first position information and the second position information, so as to obtain the matching value of the high-score frame and the first prediction frame. If the matching value is less than the first preset threshold, the high-score frame and the first prediction frame with the matching value of the second preset threshold are obtained, and the Euclidean distance between the high-score frame and the first prediction frame is calculated, and when the Euclidean distance is less than the preset distance, the high-score frame is confirmed as a tracked trajectory. When the matching value is greater than the first preset threshold, the high-score frame and the first prediction frame are matched by the Hungarian matching algorithm, if the high-score frame and the first prediction frame are matched successfully, it is determined that the first prediction frame is tracked successfully, and the traffic light corresponding to the detection frame is a blocked traffic light.

[0081] S134, according to the category of the low-score frame and the first prediction frame, the low-score frame and the first prediction frame which is not matched with the high-score frame are matched to obtain a second matching result.

[0082] In the first prediction frame, the first prediction frame not matched with the high-scored frame is obtained, and based on the categories of the detection frame and the first prediction frame, the third position information and the fourth position information of the low-scored frame and the first prediction frame not matched with the high-scored frame in the current frame image are obtained, the overlap degree of the low-scored frame and the first prediction frame not matched with the high-scored frame is calculated according to the third position information and the fourth position information respectively, and the low-scored frame and the first prediction frame not matched with the high-scored frame with the overlap degree greater than a third preset threshold and the low-scored frame and the first prediction frame not matched with the high-scored frame with the overlap degree equal to a second preset threshold are obtained. The low-scored frame and the first prediction frame not matched with the high-scored frame with the overlap degree greater than the third preset threshold are subjected to Hungarian matching, if the low-scored frame and the first prediction frame not matched with the high-scored frame are matched successfully, the low-scored frame is determined to be in a tracked state, and the traffic light corresponding to the first prediction frame is a blocked traffic light. Then, the Euclidean distance of the low-scored frame and the first prediction frame not matched with the high-scored frame with the overlap degree equal to the second preset threshold is calculated, and when the Euclidean distance is less than the preset distance, the low-scored frame is determined to be in a tracked state, and the traffic light corresponding to the first prediction frame is a blocked traffic light.

[0083] S135, the detection frame not matched with the first prediction frame is obtained, and the detection frame and the first prediction frame not matched with the low-scored frame are matched to obtain a third matching result.

[0084] Since the detection frame and the first prediction frame are matched according to the category, if the category of the tracked detection frame in the previous frame image is red, the category of the first prediction frame mapped to the current frame after the Kalman filter is also red. At this time, if the category of the detection frame in the current frame image changes from red to green, there may be the same traffic light in different frames with different categories during the matching, and thus the high-scored frame and the low-scored frame are not matched successfully. Therefore, the detection frame and the first prediction frame need to be matched again, so as to avoid tracking errors of the traffic light caused by the change of the category, and thus the accuracy of traffic light tracking is improved.

[0085] Specifically, after the low-score bounding box is matched with the first prediction bounding box which fails to match with the high-score bounding box, a detection bounding box which fails to match with the first prediction bounding box is obtained, wherein the detection bounding box includes a high-score bounding box which fails to match with the first prediction bounding box and a low-score bounding box, and a first preset bounding box which fails to match with the detection bounding box is obtained. Fifth position information and sixth position information of the detection bounding box which fails to match and the first prediction bounding box which fails to match in the current frame image are obtained, and coincidence degree calculation is performed based on the fifth position information and the sixth position information, so as to obtain the detection bounding box which fails to match and the first prediction bounding box which fails to match with a coincidence degree greater than a fourth preset threshold value, and the detection bounding box which fails to match and the first prediction bounding box which fails to match with a coincidence degree equal to a second preset threshold value. The detection bounding box which fails to match and the first prediction bounding box which fails to match with the coincidence degree greater than the fourth preset threshold value are subjected to Hungarian matching, if the detection bounding box matches with the first prediction bounding box successfully, it is confirmed that the detection bounding box is in a tracked state, and the traffic light corresponding to the first prediction bounding box is a blocked traffic light. Finally, the Euclidean distance of the detection bounding box and the first prediction bounding box with the coincidence degree equal to the second preset threshold value is calculated, and when the Euclidean distance is less than the preset distance, it is confirmed that the detection bounding box is in the tracked state, and the traffic light corresponding to the first prediction bounding box is the blocked traffic light. Preferably, the fourth preset threshold value can be 0.7.

[0086] S136, integrate the first matching result, the second matching result and the third matching result, and determine the tracking result of the blocked detection bounding box in the current frame image.

[0087] After three times of matching, a detection bounding box which matches with the first prediction bounding box and a first prediction bounding box which fails to match with the detection bounding box are obtained, wherein the traffic light corresponding to the first prediction bounding box is a blocked traffic light, and when the first prediction bounding box matches with the detection bounding box successfully, it is considered that the blocked traffic light is tracked successfully. The first prediction bounding box which fails to match is considered as the traffic light corresponding to the first prediction bounding box is still a blocked traffic light.

[0088] In some embodiments, the method further comprises obtaining a second prediction box in which a tracked traffic light in a previous frame image is mapped to a current frame image, matching the second prediction box with the detection box to obtain the second prediction box that fails to be matched with the detection box, wherein the traffic light corresponding to the second prediction box that fails to be matched is a traffic light that has been lost for a preset number of frames, taking the second prediction box that fails to be matched as a prediction box of a traffic light in a next frame image of the current frame image, and tracking the traffic light in the next frame image according to the prediction box. If, in the previous frame image, there are both a detection box in a tracked state and a detection box in a lost state, and after the detection box in the tracked state and the detection box in the lost state are mapped to the current frame image by Kalman filters, the detection box in the lost state and the tracked detection box both fail to be matched in the current frame state, then in the next frame image, the detection box in the tracked state and the detection box in the lost state are both detection boxes in a lost state.

[0089] In some embodiments, referring to Figures 3a-3e , Figures 3a-3e is a schematic diagram of consecutive frame images obtained in a traffic light tracking process provided by an embodiment of the present application, as shown in Figures 3a-3eAs shown in the first frame image, four detection boxes in the first frame image can be obtained by the target detection algorithm. Then the four detection boxes in the first frame image are input into the Kalman filter, so as to obtain four prediction boxes in the second frame image. Then the detection boxes of the traffic lights in the second frame image are obtained by the target detection algorithm. Since the traffic lights are partially blocked at this time, only two detection boxes in the second frame image can be obtained after the target detection algorithm. Then, according to the confidence of the detection boxes, the detection boxes with a confidence higher than 0.7 are set as high-score boxes, and the detection boxes with a confidence lower than 0.7 are set as low-score boxes. Then, the prediction boxes of the same category as the high-score boxes are obtained for overlap calculation. If the overlap between the high-score box and the prediction box is greater than or equal to 0.8, the high-score box and the prediction box are matched by the Hungarian method, and after successful matching, it is determined that the high-score box is successfully tracked. If the overlap is equal to 0, the Euclidean distance between the high-score box and the prediction box is calculated. When the speed of the unmanned vehicle is 30-40 km / h, if the Euclidean distance is less than 50 pixels, it is considered that the high-score box is successfully tracked. Then, the low-score boxes that are not matched with the high-score boxes are calculated for overlap calculation, and the low-score boxes with an overlap greater than or equal to 0.5 are matched with the first prediction box by the Hungarian method. If the matching is successful, it is determined that the low-score box is successfully tracked. If the overlap is equal to 0, the Euclidean distance between the low-score box and the first prediction box is calculated. When the Euclidean distance is less than 50 pixels, the low-score box is successfully tracked. The first prediction box that is not successfully matched with the low-score box is calculated for overlap calculation with the detection box that is not successfully matched. The detection box and the prediction box with an overlap greater than 0.6 are matched by the Hungarian method. If the matching is successful, it is determined that the detection box is successfully tracked. If the overlap is equal to 0, the Euclidean distance between the detection box and the prediction box is calculated. When the Euclidean distance is less than 50 pixels, it is determined that the detection box is tracked. After the above three matching algorithms, the two detection boxes in the second frame image can be successfully matched. At this time, the two prediction boxes that are not successfully matched are in a lost state. Then, in the third frame image, the two prediction boxes in the lost state are matched with the detection boxes, and the detection boxes in the tracked state in the second frame image are mapped to the third frame image for matching. As shown in Figure 3c After the matching is completed, there are three prediction boxes that are not successfully matched in the third frame image. As shown in Figure 3d After the matching of the detection boxes and the three lost prediction boxes in the fourth frame image is completed, the prediction boxes in the fourth frame are still output in a lost state. In the fifth frame image, four detection boxes can be obtained after target detection. The four detection boxes are matched with the three prediction boxes that are lost for a preset number of frames. After successful matching, the three detection boxes that are lost for a preset number of frames are successfully tracked, that is, the traffic lights that are blocked for a preset number of frames are successfully tracked.

[0090] The embodiment of the present application provides a traffic light shielding tracking method and device and an unmanned vehicle, which mainly obtains a detection frame of a traffic light in a current frame image through a target detection algorithm, then obtains a first prediction frame of the traffic light in a previous frame image which is lost for a preset number of frames and mapped to the current frame image, and finally matches the detection frame with the first prediction frame, if the detection frame and the first prediction frame are matched successfully, it is determined that the traffic light corresponding to the first prediction frame is tracked successfully, and the state of the detection frame is shielding. Thus, the shielded traffic light can be quickly identified, and the safety of the unmanned vehicle is improved.

[0091] Please refer to Figure 4 , Figure 4 The embodiment of the present application provides a traffic light shielding tracking device, which is a structural block diagram, and is applied to an unmanned vehicle, as shown in Figure 4 The traffic light shielding tracking device 400 comprises a first obtaining module 41, a second obtaining module 42 and a matching module 43.

[0092] The first obtaining module 41 is used for obtaining a detection frame of a traffic light in a current frame image based on a target detection algorithm.

[0093] The second obtaining module 42 is used for obtaining a first prediction frame of a traffic light in a previous frame image which is lost for a preset number of frames and mapped to the current frame image.

[0094] The matching module 43 is used for matching the detection frame with the first prediction frame, if the detection frame and the first prediction frame are matched successfully, it is determined that the traffic light corresponding to the first prediction frame is tracked successfully, and the state of the detection frame is shielding.

[0095] It should be noted that the traffic light shielding tracking device can execute the traffic light shielding tracking method provided by the embodiment of the present application, has the function modules and beneficial effects corresponding to the execution method. The technical details not described in detail in the traffic light shielding tracking device embodiment can be referred to the traffic light shielding tracking method provided by the embodiment of the present application.

[0096] Please refer to Figure 5 The embodiment of the present application provides an unmanned vehicle 30, which comprises at least one processor 31, Figure 5 taking the processor 31 as an example; the at least one processor 31 is in communication connection with a memory 32, Figure 5 taking the bus connection as an example.

[0097] The memory 32 stores instructions executable by the at least one processor 31, and the instructions are executed by the at least one processor 31 to enable the at least one processor 31 to perform the tracking method of traffic light occlusion described above.

[0098] The memory 32 is a non-volatile computer readable storage medium, which can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the tracking method of traffic light occlusion in the embodiments of the present application. The processor 31 performs various functional applications and data processing of the unmanned vehicle 30 by running the non-volatile software programs, instructions and modules stored in the memory 32, that is, implements the tracking method of traffic light occlusion in the method embodiments described above.

[0099] The memory 32 can include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required by a function. In addition, the memory 32 can include a high-speed random access memory and can also include a non-volatile memory. For example, it includes at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state memory device. In some embodiments, the memory 32 can optionally include a memory remotely arranged with respect to the processor 31.

[0100] The one or more modules are stored in the memory 32, and when executed by the one or more processors 31, perform the tracking method of traffic light occlusion in any of the method embodiments described above, for example, perform the method steps in the above description Figure 1 , Figure 2 .

[0101] The above-described unmanned vehicle can execute the method provided by the embodiments of the present application, and has a corresponding functional module for executing the method. Technical details not described in detail in the present embodiment can refer to the method provided by the embodiments of the present application.

[0102] The embodiments of the present application also provide a non-volatile computer readable storage medium, which stores computer executable instructions, and the computer executable instructions are executed by one or more processors, for example, perform the method steps in the above description Figure 1 and Figure 2 , to realize the functions of each module in Figure 4 .

[0103] The device embodiments described above are merely illustrative, wherein the units described as separated components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment.

[0104] Through the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be implemented by means of software plus a general hardware platform, and of course can also be implemented by hardware. Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware, and the program can be stored in a computer readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiment methods. The storage medium can be a magnetic disc, an optical disc, a read-only memory (ROM) or a random access memory (RAM), etc.

[0105] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, and not to limit them; under the idea of the present application, the technical features of the above examples or different examples can also be combined, the steps can be implemented in any order, and there are many other changes of the different aspects of the present application as described above. In order to be brief, they are not provided in detail; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A tracking method for traffic light occlusion, applied to autonomous vehicles, characterized in that, The method includes: Based on the object detection algorithm, obtain the detection box of the traffic light in the current frame image; Obtain the first prediction box of the traffic lights that have lost a preset number of frames in the previous frame and map them to the current frame image; The detection box is matched with the first prediction box. If the detection box and the first prediction box are successfully matched, it is determined that the traffic light corresponding to the first prediction box is successfully tracked, and the state of the detection box is that it is occluded. The step of matching the detection box with the first prediction box, and if the detection box and the first prediction box match successfully, determines that the traffic light corresponding to the first prediction box has been successfully tracked, and the state of the detection box is occluded, includes: Obtain the categories corresponding to the detection box and the first prediction box; Obtain the confidence score of the detection box, and divide the detection box into high-scoring boxes and low-scoring boxes based on the confidence score; Based on the categories of the high-resolution bounding boxes and the first predicted bounding boxes, the high-resolution bounding boxes and the first predicted bounding boxes of the same category are matched to obtain a first matching result; Based on the categories of the low-scoring bounding boxes and the first predicted bounding boxes, the low-scoring bounding boxes of the same category and the first predicted bounding boxes that do not match the high-scoring bounding boxes are matched to obtain a second matching result; Obtain detection boxes that do not match the first predicted box, and match the detection boxes with the first predicted boxes that do not match the low-scoring boxes to obtain a third matching result; By integrating the first matching result, the second matching result, and the third matching result, the tracking result of the occluded detection box in the current frame image is determined.

2. The method according to claim 1, characterized in that, The step of matching high-resolution bounding boxes and the first predicted bounding box of the same category to obtain matching results includes: Based on the categories of the detection box and the first predicted box, obtain a high-scoring box that is the same as the category of the first predicted box; Obtain the first and second location information of the high-resolution bounding boxes of the same category and the first predicted bounding box in the current frame image; Calculate the overlap between the first location information and the second location information, and obtain the high-resolution bounding box and the first predicted bounding box with an overlap greater than a first preset threshold, and the high-resolution bounding box and the first predicted bounding box with an overlap equal to a second preset threshold; The high-resolution bounding box with an overlap greater than the first preset threshold is matched with the first predicted bounding box. If the high-resolution bounding box and the first predicted bounding box are successfully matched, the high-resolution bounding box is confirmed to be in a tracked state. Calculate the Euclidean distance between the high-resolution bounding box and the first predicted bounding box when the overlap is equal to the second preset threshold, and confirm that the high-resolution bounding box is tracked when the Euclidean distance is less than the preset distance.

3. The method according to claim 2, characterized in that, The step of matching low-scoring boxes and first predicted boxes of the same category with first predicted boxes that do not match the high-scoring boxes to obtain a second matching result includes: Based on the categories of the detection box and the first predicted box, obtain the low-scoring box that is the same as the category of the first predicted box; Obtain the third and fourth position information of the low-scoring bounding boxes of the same category and the first predicted bounding boxes that do not match the high-scoring bounding boxes in the current frame image; Calculate the overlap between the third location information and the fourth location information, and obtain the low-scoring box and the first predicted box that does not match the high-scoring box if the overlap is greater than the third preset threshold, and the low-scoring box and the first predicted box that does not match the high-scoring box if the overlap is equal to the second preset threshold. The low-scoring boxes with an overlap greater than the third preset threshold and the first predicted boxes that do not match the high-scoring boxes are matched. If the low-scoring boxes and the first predicted boxes that do not match the high-scoring boxes are successfully matched, the low-scoring boxes are confirmed to be in a tracked state. Calculate the Euclidean distance between the low-scoring bounding box whose overlap is equal to the second preset threshold and the first predicted bounding box that does not match the high-scoring bounding box, and confirm that the low-scoring bounding box is tracked when the Euclidean distance is less than the preset distance.

4. The method according to claim 3, characterized in that, The step of obtaining a detection box that does not match the first predicted box and matching the detection box with the first predicted box that does not match the low-scoring box to obtain a third matching result includes: Obtain the fifth position information of the detection box that does not match the first prediction box in the current frame image, wherein the detection box that does not match the first prediction box includes high-resolution boxes and low-resolution boxes that do not match the first prediction box; Obtain a first predicted bounding box that does not match the low-resolution bounding box, and the sixth position information of the first predicted bounding box in the current frame image; The overlap degree of the fifth position information and the sixth position information is calculated, and the first prediction box and the detection box with an overlap degree greater than the fourth preset threshold, and the first prediction box and the detection box with an overlap degree equal to the second preset threshold are obtained. The detection box and the first prediction box with an overlap greater than the fourth preset threshold are matched. If the detection box and the first prediction box are successfully matched, the detection box is confirmed to be in a tracked state. Calculate the Euclidean distance between the detection box and the first prediction box whose overlap is equal to the second preset threshold, and confirm that the detection box is tracked when the Euclidean distance is less than the preset distance.

5. The method according to any one of claims 1-4, characterized in that, The method further includes: Obtain the detection bounding box of the current frame image and map the tracked traffic light in the previous frame image to the second prediction bounding box in the current frame image; The second prediction box is matched with the detection box to obtain the second prediction box that does not match the detection box. The traffic light corresponding to the unmatched second prediction box is the traffic light that has lost a preset number of frames. The second prediction box that failed to match is used as the prediction box for the traffic light in the next frame of the current frame image, and the traffic light in the next frame image is tracked based on the prediction box.

6. The method according to claim 1, characterized in that, After obtaining the detection bounding box of the traffic light in the current frame image, the method further includes: Obtain all detection boxes in the current frame image; The overlap of the detection boxes is calculated to determine whether there are overlapping detection boxes in the current frame image; When there are overlapping detection boxes in the current frame image, the detection boxes with higher confidence are retained based on the confidence of the detection boxes.

7. A tracking device for traffic light obstruction, applied to autonomous vehicles, characterized in that, The device includes: The first acquisition module is used to acquire the detection boxes of traffic lights in the current frame image based on the target detection algorithm; The second acquisition module is used to acquire the first prediction box of the traffic lights that have lost a preset number of frames in the previous frame image and map them to the current frame image. The matching module is used to match the detection box with the first prediction box. If the detection box and the first prediction box are successfully matched, it is determined that the traffic light corresponding to the first prediction box is successfully tracked, and the state of the detection box is occluded. Specifically, the matching module is used to: obtain the categories corresponding to the detection box and the first prediction box; Obtain the confidence score of the detection box, and divide the detection box into high-scoring boxes and low-scoring boxes based on the confidence score; Based on the categories of the high-resolution bounding boxes and the first predicted bounding boxes, the high-resolution bounding boxes and the first predicted bounding boxes of the same category are matched to obtain a first matching result; Based on the categories of the low-scoring bounding boxes and the first predicted bounding boxes, the low-scoring bounding boxes of the same category and the first predicted bounding boxes that do not match the high-scoring bounding boxes are matched to obtain a second matching result; Obtain detection boxes that do not match the first predicted box, and match the detection boxes with the first predicted boxes that do not match the low-scoring boxes to obtain a third matching result; By integrating the first matching result, the second matching result, and the third matching result, the tracking result of the occluded detection box in the current frame image is determined.

8. An unmanned vehicle, characterized in that, The driverless vehicles include: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1-6.

9. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer-executable instructions for causing a computer to perform the method as described in any one of claims 1-6.

Images