Cyclist tracking method, device, equipment, storage medium and vehicle

Abstract

The present disclosure relates to a cyclist tracking method, apparatus, device, storage medium, and vehicle. After acquiring multiple frames of images with pedestrian and cyclist detection frames labeled for cyclists, the method selects one of the pedestrian and cyclist detection frames as a tracking detection frame using frame loss parameters of the pedestrian and cyclist detection frames. The tracking detection frame is then used to associate cyclists in two adjacent image frames, thereby tracking the cyclist in the multiple image frames. This solves the problem of tracking the cyclist's target separately in the two detection frames, which may lead to erroneous tracking results.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and in particular to a cyclist tracking method, apparatus, device, storage medium, and vehicle. Background Art

[0002] As the automotive industry continues to develop, autonomous driving has always been a goal that people have been committed to achieving. One of the technical challenges is tracking other moving objects on the road.

[0003] Multi-target tracking has always been an important research topic in the field of computer vision technology. Multi-target tracking can not only measure the speed and distance of multiple moving objects, but also predict the trajectories of moving objects. Its application in autonomous driving technology can enable vehicles to avoid other moving objects, making route planning and adjustment safer.

[0004] Currently, multi-target tracking is mostly achieved by capturing road videos with a camera mounted on a car. The target is then detected in each frame, and tracking is achieved by correlating the detected targets in the previous and next frames. The accuracy of target detection directly impacts the overall tracking performance. In practical applications, when tracking a cyclist, there are two detection frames for the cyclist, while there is only one actual target. If the cyclist is tracked separately based on the two detection frames, the tracking result will be erroneous. Summary of the Invention

[0005] In order to solve the above technical problems, the present disclosure provides a cyclist tracking method, device, equipment, storage medium and vehicle.

[0006] In a first aspect, the present disclosure provides a cyclist tracking method, comprising:

[0007] Acquire multiple frames of images containing a cyclist, wherein the cyclist in the multiple frames of images is marked with a pedestrian detection frame and a cyclist detection frame, and the pedestrian detection frame and the cyclist detection frame have corresponding numbers of lost frames;

[0008] In every two adjacent frames of the plurality of image frames, each cyclist in a first image frame is combined with each cyclist in a second image frame to obtain a plurality of pre-matching groups, where the second image frame is a previous image frame of the first image frame;

[0009] For each pre-matching group, respectively calculating a first frame loss rate based on the pedestrian detection frame and a second frame loss rate based on the cyclist detection frame between the two cyclists according to the number of frame losses;

[0010] selecting, in the pre-matching group, one of the pedestrian detection frame and the cyclist detection frame as a tracking detection frame according to the first frame loss rate and the second frame loss rate;

[0011] Based on the similarity of the tracking detection frames between the two cyclists in each pre-matching group, the cyclist in the first image frame and the cyclist in the second image frame are associated to obtain an association result.

[0012] Optionally, for each pre-matching group, respectively calculating, based on the number of frame losses, a first frame loss rate based on the pedestrian detection frame and a second frame loss rate based on the cyclist detection frame between two cyclists, includes:

[0013] For the two cyclists in the pre-matching group, calculating a first difference between the number of frame drops corresponding to the two pedestrian detection frames, and a second difference between the number of frame drops corresponding to the two cyclist detection frames;

[0014] The first difference is used as a first frame loss rate, and the second difference is used as a second frame loss rate.

[0015] Optionally, selecting one of the pedestrian detection frame and the cyclist detection frame in the pre-matching group as the tracking detection frame according to the first frame loss rate and the second frame loss rate includes:

[0016] If the first frame loss rate between two cyclists in the pre-matching group is less than the second frame loss rate, selecting the pedestrian detection frame as the tracking detection frame;

[0017] If the first frame loss rate between two cyclists in the pre-matching group is greater than or equal to the second frame loss rate, the cyclist detection frame is selected as the tracking detection frame.

[0018] Optionally, associating the cyclist in the first image frame with the cyclist in the second image frame based on the similarity of the tracking detection frames between the two cyclists in each pre-matching group to obtain an association result includes:

[0019] Calculating an intersection-over-union (IoU) ratio for the selected tracking detection frames between the two cyclists in each pre-matching group, and using the IoU ratio as the similarity;

[0020] According to the intersection-over-union ratio of two tracking detection frames in each pre-matching group, the cyclist in the first image frame is assigned to the cyclist in the second image frame as an association object to obtain an association result.

[0021] Optionally, assigning the cyclist in the first image frame as an association object based on the intersection-over-union ratio of two tracking detection frames in each pre-matching group to the cyclist in the second image frame to obtain an association result includes:

[0022] Selecting a pre-matching group whose intersection-over-union ratio is greater than a first preset threshold as a candidate matching group;

[0023] Placing the cyclists in the candidate matching group and the corresponding intersection-union ratios into a weight matrix, and matching the cyclists in the weight matrix using the Hungarian algorithm to obtain a matching result;

[0024] The matching result is used as the association result.

[0025] Optionally, the cyclist in the multiple frames of images also has corresponding appearance feature information;

[0026] The method further comprises:

[0027] Calculating feature similarity between two cyclists in the pre-matching group based on the appearance feature information;

[0028] The pre-matching group whose feature similarity is greater than a second preset threshold is selected as the candidate matching group.

[0029] In a second aspect, the present disclosure provides a cyclist tracking device, comprising:

[0030] an acquisition module configured to acquire a plurality of frames of images containing a cyclist, wherein the cyclist in the plurality of frames of images is marked with a pedestrian detection frame and a cyclist detection frame, and the pedestrian detection frame and the cyclist detection frame have corresponding numbers of lost frames;

[0031] a grouping module configured to combine, in every two adjacent frames of the plurality of image frames, each cyclist in a first image frame with each cyclist in a second image frame to obtain a plurality of pre-matching groups, where the second image frame is a previous image frame of the first image frame;

[0032] a calculation module, configured to calculate, for each pre-matching group, a first frame loss rate based on the pedestrian detection frame and a second frame loss rate based on the cyclist detection frame between two cyclists according to the number of frame losses;

[0033] a selection module, configured to select, in the pre-matching group, one of the pedestrian detection frame and the cyclist detection frame as a tracking detection frame based on the first frame loss rate and the second frame loss rate;

[0034] The association module is configured to associate the cyclist in the first image frame with the cyclist in the second image frame based on the similarity of the tracking detection frames between the two cyclists in each pre-matching group to obtain an association result.

[0035] Optionally, when the calculation module calculates the first frame loss rate based on the pedestrian detection frame and the second frame loss rate based on the cyclist detection frame between two cyclists for each pre-matching group according to the frame loss number, the calculation module is specifically used to calculate the first difference between the frame loss numbers corresponding to the two pedestrian detection frames and the second difference between the frame loss numbers corresponding to the two cyclists in the pre-matching group; and use the first difference as the first frame loss rate and the second difference as the second frame loss rate.

[0036] Optionally, when the selection module selects one of the pedestrian detection frame and the cyclist detection frame in the pre-matching group as the tracking detection frame based on the first frame loss rate and the second frame loss rate, it is specifically used to select the pedestrian detection frame as the tracking detection frame if the first frame loss rate between two cyclists in the pre-matching group is less than the second frame loss rate; if the first frame loss rate between two cyclists in the pre-matching group is greater than or equal to the second frame loss rate, the cyclist detection frame is selected as the tracking detection frame.

[0037] Optionally, the association module associates the cyclist in the first image frame with the cyclist in the second image frame based on the similarity of the tracking detection frames between the two cyclists in each pre-matching group to obtain an association result, and is specifically used to calculate the intersection-over-union (IoU) of the tracking detection frames selected between the two cyclists in each pre-matching group, and use the IoU as the similarity; according to the IoU of the two tracking detection frames in each pre-matching group, assign the cyclist in the first image frame to the cyclist in the second image frame as an association object to obtain an association result.

[0038] Optionally, the association module assigns the cyclist in the second image frame as an association object to the cyclist in the first image frame according to the intersection-and-union ratio of the two tracking detection frames in each pre-matching group, and when obtaining the association result, is specifically used to select the pre-matching group whose intersection-and-union ratio is greater than a first preset threshold as a candidate matching group; place the cyclists in the candidate matching group and the corresponding intersection-and-union ratio into a weight matrix, use the Hungarian algorithm to match the cyclists in the weight matrix, and obtain a matching result; and use the matching result as the association result.

[0039] Optionally, the cyclist in the multiple frames of images also has corresponding appearance feature information;

[0040] The association module is further configured to calculate a feature similarity between two cyclists in the pre-matching group based on the appearance feature information; and select a pre-matching group whose feature similarity is greater than a second preset threshold as the candidate matching group.

[0041] In a third aspect, the present disclosure provides an electronic device, comprising:

[0042] Memory;

[0043] processor; and

[0044] computer programs;

[0045] The computer program is stored in the memory and is configured to be executed by the processor to implement the method as described in the first aspect.

[0046] In a fourth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method as described in the first aspect.

[0047] In a fifth aspect, the present disclosure provides a vehicle comprising the cyclist tracking device as described in the second aspect.

[0048] The cyclist tracking method, apparatus, device, storage medium, and vehicle provided by the embodiments of the present disclosure, after acquiring multiple frames of images with pedestrian detection frames and cyclist detection frames labeled for the cyclist, utilize the frame loss parameters of the pedestrian detection frames and the cyclist detection frames to select one type of detection frame from the pedestrian detection frames and the cyclist detection frames as the tracking detection frame. The tracking detection frame is then used to associate the cyclist in two adjacent frames of images, thereby achieving target tracking of the cyclist in the multiple frames of images. This solves the problem of tracking the two detection frames of the cyclist target separately, which may lead to erroneous tracking results. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A flow chart of a cyclist tracking method provided in an embodiment of the present disclosure;

[0050] Figure 2 A schematic diagram of an application scenario provided by an embodiment of the present disclosure;

[0051] Figure 3 A schematic structural diagram of a cyclist tracking device provided in an embodiment of the present disclosure;

[0052] Figure 4 Schematic diagram of the structure of an electronic device in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0053] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features therein can be combined with each other in the absence of conflict.

[0054] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0055] At present, multi-target tracking is mostly achieved by shooting a video of the road through a camera installed on a car, then detecting the target in each frame of the video, and finally tracking the target by associating the targets detected in the previous and next frames. The accuracy of target detection directly affects the overall effect of target tracking. In actual applications, there are usually many similar individuals in a shooting scene. When tracking the cyclist target, the shared bicycles on the roadside and pedestrians passing by the shared bicycles may often be mistaken by the machine as cyclists, affecting the effect of tracking the cyclist target. In response to the above problems, the embodiments of the present disclosure provide a cyclist tracking method, which is introduced below in conjunction with specific embodiments.

[0056] Figure 1 This is a flow chart of a cyclist tracking method provided by an embodiment of the present disclosure. The method can be executed by a cyclist tracking device. The cyclist tracking device can be implemented in software and / or hardware. The cyclist tracking device can be configured in an electronic device, such as a server or a terminal, wherein the terminal specifically includes a vehicle computer, a computer, etc. In addition, the method can be applied to Figure 2 The application scenario shown includes image frame 210 and image frame 220, where image frame 210 is the previous image frame of image frame 220. It is understandable that the cyclist tracking method provided in the embodiment of the present disclosure can also be applied in other scenarios.

[0057] The following combination Figure 2 The application scenario shown is Figure 1 The cyclist tracking method shown in FIG. 1 is introduced, and the specific steps of the method are as follows:

[0058] S101: Acquire multiple frames of images containing a cyclist, wherein the cyclist in the multiple frames of images is marked with a pedestrian detection frame and a cyclist detection frame, and the pedestrian detection frame and the cyclist detection frame have corresponding numbers of lost frames.

[0059] In the process of tracking cyclists, the car's external camera will capture video images on the road, and then input multiple frames of the video images into the preset convolutional neural network for processing, detect the cyclist target in each frame, and output the cyclist with a pedestrian detection frame and a cyclist detection frame marked, for example Figure 2In the image frame 210 shown, a cyclist 211 is annotated with a pedestrian detection frame 212 and a cyclist detection frame 213. The annotated pedestrian and cyclist detection frames have corresponding parameters, including a missing frame count (missing_detection_tracking), which indicates the number of times the detection frame missed tracking during the tracking process. When implementing the target tracking task, target tracking in multiple frames is divided into the task of associating targets in two adjacent frames. Targets that are associated in the previous and next frames are considered to be the same target, thereby achieving target tracking. For example, when associating cyclists in the previous and current frames, the intersection-and-union ratio is calculated using the cyclist detection frames to determine the degree of association between the cyclists in the two frames. If the cyclist detection frame of a cyclist in the current frame is not associated with the tracked cyclist target in the previous frame, it is considered that the cyclist in the current frame is not identified as associated with the cyclist in the previous frame, indicating that the association failed. The missing frame count corresponding to the cyclist detection frame of the cyclist is then increased by 1 and used in the cyclist association task between the current and next frames.

[0060] S102 . In every two adjacent frames of the multi-frame image, combine each cyclist in the first image frame with each cyclist in the second image frame to obtain a plurality of pre-matching groups, where the second image frame is the previous image frame of the first image frame.

[0061] The task of tracking cyclist targets in multiple image frames can be the task of associating targets in two adjacent image frames. Therefore, when associating cyclists in two adjacent image frames, it is necessary to traverse and compare each cyclist in the current image frame with each cyclist in the previous image frame to determine whether the targets in the two image frames are the same cyclist.

[0062] Exemplarily, image frame 210 and image frame 220 are two adjacent frames in a multi-frame image, where the second image frame is image frame 210 and the first image frame is image frame 220. Cyclist 221 and cyclist 224 in image frame 220 are sequentially compared with cyclist 211 and cyclist 214 in image frame 210 to determine which of them are the same cyclist. In this way, there are four pre-matching groups in the comparison process, namely (221, 211), (221, 214), (224, 211), and (224, 214).

[0063] S103 . For each pre-matching group, calculate, based on the number of frame losses, a first frame loss rate based on the pedestrian detection frame and a second frame loss rate based on the cyclist detection frame between two cyclists.

[0064] Because each cyclist is labeled with both a pedestrian detection frame and a cyclist detection frame, when comparing cyclists in two image frames, you need to select one type of detection frame for comparison. Otherwise, tracking the cyclist based on both detection frames will result in incorrect tracking results.

[0065] The embodiment of the present disclosure selects the tracking detection frame by comparing the number of frame drops of the two types of detection frames between the two cyclists. Therefore, it is necessary to calculate the frame drop rate of the two types of detection frames between the two cyclists. Figure 2 For example, for pre-matching group (221, 211), the first frame loss rate based on the pedestrian detection frame is: the frame loss rate between the number of frame losses of pedestrian detection frame 222 and the number of frame losses of pedestrian detection frame 212; the second frame loss rate based on the cyclist detection frame is: the frame loss rate between the number of frame losses of cyclist detection frame 223 and the number of frame losses of cyclist detection frame 213. Similarly (the following detection frame numbers are examples, and the text descriptions of the detection frames are omitted), the first frame loss rate of pre-matching group (221, 214) is (222-215), and the second frame loss rate is (223-216); the first frame loss rate of pre-matching group (224, 211) is (225-212), and the second frame loss rate is (226-213); the first frame loss rate of pre-matching group (224, 214) is (225-215), and the second frame loss rate is (226-216).

[0066] S104 : Selecting, in the pre-matching group, one of a pedestrian detection frame and a cyclist detection frame as a tracking detection frame according to the first frame loss rate and the second frame loss rate.

[0067] Based on the first frame loss rate and the second frame loss rate calculated for each pre-matching group, the type of detection frame of the pre-matching group is selected as the tracking detection frame. For example, if the first frame loss rate (222-215) of the pre-matching group (221, 214) is smaller than the second frame loss rate (223-216), the pedestrian detection frames of both sides in the pre-matching group (221, 214) are selected as the tracking detection frames. That is, when comparing cyclist 221 with cyclist 214, the pedestrian detection frame 222 of cyclist 221 and the pedestrian detection frame 215 of the pedestrian detection frame are selected for comparison. If the second frame loss rate is smaller than the first frame loss rate, the cyclist detection frame is selected as the tracking detection frame. If the second frame loss rate is equal to the first frame loss rate, the cyclist detection frame is preferentially selected as the tracking detection frame.

[0068] S105 : Associating the cyclist in the first image frame with the cyclist in the second image frame based on the similarity of the tracking detection frames between the two cyclists in each pre-matching group to obtain an association result.

[0069] The similarity between the tracking detection frames of the two cyclists in each pre-matching group is used to determine the degree of association between the cyclist in the first image frame and the cyclist in the second image frame. The cyclist in the first image frame and the cyclist in the second image frame are then associated to obtain the association result of the cyclists between the two adjacent image frames. The associated cyclists in the two image frames are determined to be the same cyclist. The association is then repeated for each two adjacent image frames in multiple image frames, thus achieving tracking of the cyclist target in the multiple image frames.

[0070] The disclosed embodiment, after acquiring multiple frames of images with pedestrian detection frames and cyclist detection frames marked on the cyclist, uses the frame loss parameters of the pedestrian detection frames and the cyclist detection frames to select one type of detection frame from the pedestrian detection frames and the cyclist detection frames as a tracking detection frame. The tracking detection frame is then used to associate the cyclist in two adjacent frames of images to achieve target tracking of the cyclist in the multiple frames of images. This solves the problem of tracking the cyclist target separately in the two detection frames, which may lead to erroneous tracking results.

[0071] Based on the above embodiment, for each pre-matching group, a first frame loss rate based on the pedestrian detection frame and a second frame loss rate based on the cyclist detection frame are calculated between the two cyclists according to the number of frame losses, including: for the two cyclists in the pre-matching group, a first difference between the number of frame losses corresponding to the two pedestrian detection frames and a second difference between the number of frame losses corresponding to the two cyclist detection frames are calculated; the first difference is used as the first frame loss rate, and the second difference is used as the second frame loss rate.

[0072] Target tracking requires associating the target in the current image frame with the target in the previous image frame to determine the relationship between the target in the current image frame and the tracked target in the previous image frame. Target tracking is achieved by identifying two targets in the current image frame that have been successfully associated with the previous image frame as the same target. Because each cyclist is annotated with a pedestrian detection frame and a cyclist detection frame, when comparing cyclists in two image frames, it is necessary to select one type of detection frame for comparison. Otherwise, tracking the cyclist based on the two detection frames of the cyclist target separately will result in incorrect tracking results.

[0073] The disclosed embodiment calculates the first difference in the number of frame losses corresponding to the two pedestrian detection frames between the two cyclists in each pre-matched group formed when traversing and comparing each cyclist in the current image frame with the cyclists included in the previous image frame, and calculates the second difference in the number of frame losses corresponding to the two cyclist detection frames as the second frame loss rate based on the cyclist detection frame. Thus, the first frame loss rate and the second frame loss rate can be used to compare which type of detection frame between the two cyclists has a closer number of frame losses. Then, in the subsequent steps, the detection frame of the type with the closer number of frame losses is selected to determine the comparison result between the cyclists, making the comparison result more accurate.

[0074] On the basis of the above embodiment, according to the first frame loss rate and the second frame loss rate, one of the pedestrian detection frame and the cyclist detection frame is selected in the pre-matching group as the tracking detection frame, including: if the first frame loss rate between two cyclists in the pre-matching group is less than the second frame loss rate, the pedestrian detection frame is selected as the tracking detection frame; if the first frame loss rate between two cyclists in the pre-matching group is greater than or equal to the second frame loss rate, the cyclist detection frame is selected as the tracking detection frame.

[0075] Between the two cyclists in each pre-matched group formed, the corresponding detection frame with the smaller frame loss rate between the pedestrian detection frame and the cyclist detection frame is selected as the tracking detection frame. From the above embodiment, it can be seen that the first frame loss rate is the first difference in the number of frame loss corresponding to the two pedestrian detection frames, and the second frame loss rate is the second difference in the number of frame loss corresponding to the two cyclist detection frames. The judgment logic is: if the first difference is less than the second difference, the pedestrian detection frame is selected as the tracking detection frame; if the first difference is greater than or equal to the second difference, the cyclist detection frame is selected as the tracking detection frame. This means that when the number of frame loss of the two pedestrian detection frames is closer, the pedestrian detection frame is selected as the tracking detection frame; when the number of frame loss of the two cyclist detection frames is closer, the cyclist detection frame is selected as the tracking detection frame; when the first difference is equal to the second difference, the cyclist detection frame is preferentially selected as the tracking detection frame.

[0076] Because the number of lost frames represents the number of missed detections corresponding to the cyclist's tracking detection frame when tracking the cyclist in the image frame before the current image frame, for example, if the number of lost frames for cyclist detection frame 213 corresponding to cyclist 211 in image frame 210 is 1, it means that during the association process between an image frame before image frame 210 and the previous image frame, the cyclist detection frame of cyclist 211 in the image frame was selected as the tracking detection frame, but the cyclist detection frame of cyclist 211 was not associated with the tracked cyclist detection frame in the previous image frame. Therefore, the number of lost frames is increased by 1, indicating that the cyclist target was not tracked between the two adjacent image frames.

[0077] The disclosed embodiment calculates, for two cyclists in a pre-matched group, a first difference in the number of frame losses corresponding to two pedestrian detection frames and a second difference in the number of frame losses corresponding to two cyclist detection frames, respectively, to determine a first frame loss rate and a second frame loss rate. Then, based on the first and second frame loss rates corresponding to the two cyclists, the type of detection frame with the smaller frame loss rate is selected as the tracking detection frame. When the frame loss rates are equal, the cyclist detection frame is preferentially selected as the tracking detection frame. This avoids the situation where a cyclist target is tracked separately according to two different types of detection frames, and improves tracking accuracy by selecting the type of detection frame with the smaller frame loss rate for tracking.

[0078] Based on the above embodiment, based on the similarity of the tracking detection frames between the two cyclists in each pre-matching group, the cyclist in the first image frame and the cyclist in the second image frame are associated to obtain an association result, including: calculating the intersection-over-union (IoU) of the tracking detection frames selected between the two cyclists in each pre-matching group, and using the IoU as the similarity; and assigning the cyclist in the first image frame to the cyclist in the second image frame as an association object based on the IoU of the two tracking detection frames in each pre-matching group, to obtain an association result.

[0079] The intersection-over-union (IOU) is the ratio of the area where two frames intersect to the area where the two frames are parallel. The similarity of the tracking detection frames is determined by calculating the IOU between the two tracking detection frames in each pre-matching group. Because the time of one frame is relatively short and the speed of the cyclist is relatively low, the distance moved by the same cyclist between two adjacent frames is small. Therefore, we can assume that the overlap rate of the two detection frames of the same cyclist in two adjacent frames will be relatively high. Conversely, it can be assumed that the higher the overlap rate of the two tracking detection frames, the greater the possibility that the two tracking detection frames are tracking the same cyclist. Therefore, the IOU of the two tracking detection frames can be used as an important parameter for cyclist similarity.

[0080] Based on the intersection-over-union ratio calculated from all pre-matching groups, the similarity between each tracking detection frame in the first image frame and each tracking detection frame in the second image frame can be obtained. Since target tracking is a one-to-one tracking task, that is, the tracking target in the previous image frame will only determine a corresponding tracking target in the current image frame, this similarity can be used to assign the cyclist in the first image frame to the cyclist in the second image frame as an associated object, thereby obtaining an association result.

[0081] In this case, according to the intersection-and-union ratio of two tracking detection frames in each pre-matching group, the cyclist in the first image frame is assigned to the cyclist in the second image frame as an association object to obtain an association result, including: selecting a pre-matching group with an intersection-and-union ratio greater than a first preset threshold as a candidate matching group; placing the cyclists in the candidate matching group and the corresponding intersection-and-union ratio into a weight matrix, using the Hungarian algorithm to match the cyclists in the weight matrix to obtain a matching result; and using the matching result as the association result.

[0082] Since the task of tracking cyclists requires high timeliness, we perform subsequent matching by screening some pre-matching groups whose intersection-union ratios meet the requirements. Because when the intersection-union ratio calculated in a pre-matching group is less than the first preset threshold, it can be considered that the probability that the two cyclists in the pre-matching group are the same tracking target is very low, so these pre-matching groups with intersection-union ratios less than the first preset threshold can be omitted from subsequent associated matching, thereby reducing the amount of calculation. In addition, the height ratio between the two tracking detection frames can also be used as a screening parameter, and a height ratio threshold can be set. The results of the pre-matching groups whose intersection-union ratios and height ratios meet the corresponding requirements are selected for subsequent matching.

[0083] For example, the results of the intersection-over-union ratio and the height ratio meeting the threshold requirements are stored in a database named res_vec. The parameters contained in the database include the tracking detection frame in the first image frame, the tracking detection frame in the second image frame, and the intersection-over-union ratio between the two tracking detection frames. The weight matrix is ​​then updated using the intersection-over-union ratio and the tracking detection frames stored in the database. Finally, the Hungarian algorithm is used to match the cyclists in the weight matrix to obtain the optimal match between the cyclist in the first image frame and the cyclist in the second image frame. The result of this matching is the association result. For example, Figure 2 In the corresponding pre-matching groups (221, 211), (221, 214), (224, 211), and (224, 214), the pre-matching groups (221, 214) and (224, 211) do not meet the threshold requirement. Therefore, the results of the pre-matching groups (221, 211) and (224, 214) are used as candidate matching groups. Then, the intersection-over-union ratio of the cyclist detection frame 213 and the cyclist detection frame 223, and the intersection-over-union ratio of the cyclist detection frame 216 and the cyclist detection frame 226 in the candidate matching groups (221, 211) and (224, 214) are used to update the weight matrix. The information in the weight matrix and the Hungarian algorithm are used to obtain the optimal match, that is, the cyclist 211 is matched with the cyclist 221, and the cyclist 214 is matched with the cyclist 224. The final association result is obtained, and it is determined that the cyclist 211 and the cyclist 221 are the same tracking target, and the cyclist 214 and the cyclist 224 are the same tracking target.

[0084] The disclosed embodiment selects pre-matching groups whose intersection-over-union ratios are greater than a preset threshold as candidate matching groups to eliminate combinations in the pre-matching groups with a low probability of two cyclists being the same tracking target, thereby reducing the computational effort required to obtain a matching result between the cyclist in the first image and the cyclist in the second image. The weight matrix is ​​then updated using the results from the candidate matching groups, and the Hungarian algorithm is used to obtain a matching result for the cyclist in the weight matrix, thereby improving the real-time performance of the cyclist tracking algorithm.

[0085] Based on the above embodiment, the cyclists in the multiple-frame images also have corresponding appearance feature information; the method also includes: calculating the feature similarity between two cyclists in the pre-matching group based on the appearance feature information; and selecting the pre-matching group whose feature similarity is greater than a second preset threshold as a candidate matching group.

[0086] When screening candidate matching groups, the cyclists' appearance feature information can be further incorporated into the screening process. This is because appearance feature information can help determine the likelihood that two cyclists are the same tracking target. If the feature similarity calculated based on the appearance feature information is less than a second preset threshold, the probability that the two cyclists are the same tracking target is considered low, and the two cyclists are determined to be unmatched. If the feature similarity is greater than the second preset threshold, the pre-match group is selected as a candidate matching group. The weight matrix is ​​then updated based on the feature similarity and the intersection-over-union ratio, further improving the accuracy of screening candidate matching groups.

[0087] The embodiment of the present disclosure calculates feature similarity using appearance feature information corresponding to the cyclist, and selects pre-matching groups with feature similarity greater than a second preset threshold as candidate matching groups, thereby further improving the accuracy of screening candidate matching groups.

[0088] Figure 3 This is a schematic diagram of the structure of a cyclist tracking device provided in an embodiment of the present disclosure. The cyclist tracking device may be the terminal described above, or the cyclist tracking device may be a component or assembly in the terminal. The cyclist tracking device provided in an embodiment of the present disclosure may execute the processing flow provided in an embodiment of the cyclist tracking method, such as Figure 3As shown, the cyclist tracking device 300 includes: an acquisition module 301 for acquiring multiple frames of images containing cyclists, wherein the cyclists in the multiple frames of images are annotated with pedestrian detection frames and cyclist detection frames, and the pedestrian detection frames and cyclist detection frames have corresponding frame loss numbers; a grouping module 302 for combining each cyclist in a first image frame with each cyclist in a second image frame in every two adjacent frames of the multiple frames to obtain multiple pre-matching groups, where the second image frame is the image frame preceding the first image frame; a calculation module 303 for calculating, for each pre-matching group, a first frame loss rate based on the pedestrian detection frame and a second frame loss rate based on the cyclist detection frame between the two cyclists based on the frame loss numbers; a selection module 304 for selecting one of the pedestrian detection frame and the cyclist detection frame in the pre-matching group as a tracking detection frame based on the first frame loss rate and the second frame loss rate; and an association module 305 for associating the cyclist in the first image frame with the cyclist in the second image frame based on the similarity between the tracking detection frames of the two cyclists in each pre-matching group to obtain an association result.

[0089] In some embodiments, when the calculation module 303 calculates the first frame loss rate based on the pedestrian detection frame and the second frame loss rate based on the cyclist detection frame between two cyclists for each pre-matching group according to the number of frame losses, it is specifically used to calculate, for the two cyclists in the pre-matching group, a first difference between the number of frame losses corresponding to the two pedestrian detection frames and a second difference between the number of frame losses corresponding to the two cyclists detection frames; the first difference is used as the first frame loss rate, and the second difference is used as the second frame loss rate.

[0090] In some embodiments, when the selection module 304 selects one of the pedestrian detection frame and the cyclist detection frame in the pre-matching group as the tracking detection frame based on the first frame loss rate and the second frame loss rate, it is specifically used to select the pedestrian detection frame as the tracking detection frame if the first frame loss rate between the two cyclists in the pre-matching group is less than the second frame loss rate; if the first frame loss rate between the two cyclists in the pre-matching group is greater than or equal to the second frame loss rate, the cyclist detection frame is selected as the tracking detection frame.

[0091] In some embodiments, the association module 305 associates the cyclist in the first image frame with the cyclist in the second image frame based on the similarity of the tracking detection frames between the two cyclists in each pre-matching group to obtain an association result, which is specifically used to calculate the intersection-over-union (IoU) of the tracking detection frames selected between the two cyclists in each pre-matching group, and use the IoU as the similarity; according to the IoU of the two tracking detection frames in each pre-matching group, the cyclist in the first image frame is assigned to the cyclist in the second image frame as an association object to obtain an association result.

[0092] In some embodiments, the association module 305 assigns the cyclist in the second image frame as an association object to the cyclist in the first image frame based on the intersection-and-union ratio of the two tracking detection frames in each pre-matching group. When obtaining the association result, the association module 305 is specifically used to select the pre-matching group whose intersection-and-union ratio is greater than a first preset threshold as the candidate matching group; place the cyclists in the candidate matching group and the corresponding intersection-and-union ratio into a weight matrix, use the Hungarian algorithm to match the cyclists in the weight matrix, and obtain a matching result; and use the matching result as the association result.

[0093] In some embodiments, the cyclists in the multiple frames of images also have corresponding appearance feature information; the association module 305 is further used to calculate the feature similarity between the two cyclists in the pre-matching group based on the appearance feature information; and select the pre-matching group whose feature similarity is greater than the second preset threshold as the candidate matching group.

[0094] Figure 3 The cyclist tracking device of the illustrated embodiment can be used to implement the technical solution of the above-mentioned method embodiment. Its implementation principle and technical effects are similar and will not be described in detail here.

[0095] Figure 4 This is a schematic diagram of the structure of an electronic device in the embodiment of the present disclosure. Figure 4 , which shows a structural diagram of an electronic device 400 suitable for implementing the embodiments of the present disclosure. Figure 4 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0096] like Figure 4 As shown, the electronic device 400 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes to implement the cyclist tracking method of the embodiment described in the present disclosure according to the program stored in the read-only memory (ROM) 402 or the program loaded from the storage device 408 to the random access memory (RAM) 403. Various programs and data required for the operation of the electronic device 400 are also stored in the RAM 403. The processing device 401, ROM 402, and RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0097] Typically, the following devices may be connected to the I / O interface 405: an input device 406 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 408 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 409. The communication device 409 may allow the electronic device 400 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 4 The electronic device 400 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0098] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart, thereby implementing the cyclist tracking method described above. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 409, or installed from the storage device 408, or installed from the ROM 402. When the computer program is executed by the processing device 401, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0099] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0100] In some embodiments, the client and server can communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0101] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0102] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device:

[0103] Acquire multiple frames of images containing a cyclist, wherein the cyclist in the multiple frames of images is annotated with a pedestrian detection frame and a cyclist detection frame, and the pedestrian detection frame and the cyclist detection frame have corresponding numbers of lost frames;

[0104] In every two adjacent frames of the multiple image frames, each cyclist in the first image frame is combined with each cyclist in the second image frame to obtain a plurality of pre-matching groups, where the second image frame is the previous image frame of the first image frame;

[0105] For each pre-matching group, calculate the first frame loss rate based on the pedestrian detection frame and the second frame loss rate based on the cyclist detection frame between the two cyclists based on the number of frame losses.

[0106] Selecting, according to the first frame loss rate and the second frame loss rate, one of the pedestrian detection frame and the cyclist detection frame in the pre-matching group as the tracking detection frame;

[0107] Based on the similarity of the tracking detection frames between the two cyclists in each pre-matching group, the cyclist in the first image frame is associated with the cyclist in the second image frame to obtain an association result.

[0108] Optionally, when the above one or more programs are executed by the electronic device, the electronic device may also execute other steps described in the above embodiments.

[0109] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0110] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0111] The units involved in the embodiments described in this disclosure may be implemented in software or hardware, wherein the name of a unit does not necessarily limit the unit itself.

[0112] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0113] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0114] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0115] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0116] Although the subject matter has been described using language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims. Furthermore, embodiments of the present disclosure further provide a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the cyclist tracking method described in the above embodiments.

[0117] In addition, the present disclosure also provides a vehicle, which includes: Figure 3 The cyclist tracking device shown.

[0118] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a..." do not exclude the presence of other identical elements in the process, method, article or device that includes the elements.

Claims

1. A cyclist tracking method, characterized in that: include: Acquire multiple frames of images containing a cyclist, wherein the cyclist in the multiple frames of images is marked with a pedestrian detection frame and a cyclist detection frame, and the pedestrian detection frame and the cyclist detection frame have corresponding numbers of lost frames; In every two adjacent frames of the plurality of image frames, each cyclist in a first image frame is combined with each cyclist in a second image frame to obtain a plurality of pre-matching groups, where the second image frame is a previous image frame of the first image frame; For each pre-matching group, respectively calculating a first frame loss rate based on the pedestrian detection frame and a second frame loss rate based on the cyclist detection frame between the two cyclists according to the number of frame losses; selecting, in the pre-matching group, one of the pedestrian detection frame and the cyclist detection frame as a tracking detection frame according to the first frame loss rate and the second frame loss rate; Based on the similarity of the tracking detection frames between the two cyclists in each pre-matching group, the cyclist in the first image frame and the cyclist in the second image frame are associated to obtain an association result.

2. The method according to claim 1, wherein For each pre-matching group, respectively calculating a first frame loss rate based on the pedestrian detection frame and a second frame loss rate based on the cyclist detection frame between two cyclists according to the number of frame losses, including: For the two cyclists in the pre-matching group, calculating a first difference between the number of frame drops corresponding to the two pedestrian detection frames, and a second difference between the number of frame drops corresponding to the two cyclist detection frames; The first difference is used as a first frame loss rate, and the second difference is used as a second frame loss rate.

3. The method according to claim 1, wherein The selecting, according to the first frame loss rate and the second frame loss rate, one of the pedestrian detection frame and the cyclist detection frame in the pre-matching group as the tracking detection frame includes: If the first frame loss rate between two cyclists in the pre-matching group is less than the second frame loss rate, selecting the pedestrian detection frame as the tracking detection frame; If the first frame loss rate between two cyclists in the pre-matching group is greater than or equal to the second frame loss rate, the cyclist detection frame is selected as the tracking detection frame.

4. The method according to claim 1, wherein The associating the cyclist in the first image frame with the cyclist in the second image frame based on the similarity of the tracking detection frames between the two cyclists in each pre-matching group to obtain an association result includes: Calculating an intersection-over-union (IoU) ratio for the selected tracking detection frames between the two cyclists in each pre-matching group, and using the IoU ratio as the similarity; According to the intersection-over-union ratio of two tracking detection frames in each pre-matching group, the cyclist in the first image frame is assigned to the cyclist in the second image frame as an association object to obtain an association result.

5. The method according to claim 4, wherein The step of assigning the cyclist in the first image frame as an association object based on the intersection-over-union ratio of the two tracking detection frames in each pre-matching group, and obtaining an association result, includes: Selecting a pre-matching group whose intersection-over-union ratio is greater than a first preset threshold as a candidate matching group; Placing the cyclists in the candidate matching group and the corresponding intersection-union ratios into a weight matrix, and matching the cyclists in the weight matrix using the Hungarian algorithm to obtain a matching result; The matching result is used as the association result.

6. The method according to claim 5, wherein The cyclist in the multiple frames of images also has corresponding appearance feature information; The method further comprises: Calculating feature similarity between two cyclists in the pre-matching group based on the appearance feature information; The pre-matching group whose feature similarity is greater than a second preset threshold is selected as the candidate matching group.

7. A cyclist tracking device, characterized in that: include: an acquisition module configured to acquire a plurality of frames of images containing a cyclist, wherein the cyclist in the plurality of frames of images is marked with a pedestrian detection frame and a cyclist detection frame, and the pedestrian detection frame and the cyclist detection frame have corresponding numbers of lost frames; a grouping module configured to combine, in every two adjacent frames of the plurality of image frames, each cyclist in a first image frame with each cyclist in a second image frame to obtain a plurality of pre-matching groups, where the second image frame is a previous image frame of the first image frame; a calculation module, configured to calculate, for each pre-matching group, a first frame loss rate based on the pedestrian detection frame and a second frame loss rate based on the cyclist detection frame between two cyclists according to the number of frame losses; a selection module, configured to select, in the pre-matching group, one of the pedestrian detection frame and the cyclist detection frame as a tracking detection frame based on the first frame loss rate and the second frame loss rate; The association module is configured to associate the cyclist in the first image frame with the cyclist in the second image frame based on the similarity of the tracking detection frames between the two cyclists in each pre-matching group to obtain an association result.

8. An electronic device comprising: Memory; processor; as well as computer programs; The computer program is stored in the memory and configured to be executed by the processor to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A vehicle, characterized in that: The device comprises the cyclist tracking device as claimed in claim 7.

Images