Online multi-target tracking method and system based on feature fusion and iterative association

Abstract

The invention relates to an online multi-target tracking method and system based on feature fusion and iterative association, and relates to the technical field of target tracking. The online multi-target tracking method comprises the following steps: performing fusion updating on confidence coefficients by utilizing position information and the confidence coefficients corresponding to possible targets in each frame of image in a given video sequence to obtain fusion updating confidence coefficients; determining a plurality of thresholds corresponding to the fusion update confidence by using an iterative algorithm or an adaptive threshold method; dividing the fusion update confidence into a plurality of target detection frame intervals based on the plurality of thresholds; and tracking a possible target in each frame of image in the given video sequence by using the position information corresponding to the plurality of target detection frame intervals. According to the embodiment of the invention, online multi-target tracking can be realized.

Description

Technical Field

[0001] The present disclosure relates to the technical field of target tracking, and in particular to an online multi-target tracking method and system based on feature fusion and iterative association. Background Art

[0002] Autonomous driving has high requirements for safety and high efficiency. Pedestrian tracking technology can detect pedestrians and other vehicles around the vehicle in a timely manner to avoid potential traffic accidents, especially in emergencies. Therefore, the high safety and high efficiency development of pedestrian tracking technology plays an important role in autonomous driving technology, especially in solving serious occlusion, clustering, blur and other problems.

[0003] The development time of video target tracking is relatively short. Early algorithms such as mean shift and particle filtering have low accuracy and are not suitable for practical applications. In recent years, with the development of target tracking, the mainstream R-CNN series and YOLO series target detection algorithms have extremely high accuracy. MOT based on target detection has developed rapidly and gradually become a mainstream framework, which has greatly promoted the development of multi-target tracking. With the hot application of attention mechanisms such as Transformer in computer vision, researchers have recently proposed a multi-target tracking framework based on attention mechanism. In this paper, because it specializes in processing sequence information, it can perfectly complete the frame-by-frame operation of video sequences. In addition, there is also a graph convolutional network (GCN) to solve the MOT problem. It regards a group of targets detected from consecutive frames as a node and completes matching through the Hungarian algorithm, which is an offline tracking method.

[0004] Specifically, the framework based on Tracking-by-detection (TBD) has made significant progress in multi-target tracking. Existing methods usually first detect the target frame by frame in the video sequence, use a box to capture the target, and then convert it into an association problem between the previous and next frames. However, when the target object faces occlusion, especially occlusion of the same type of target, the reliability of matching will be reduced, especially in the case of severe occlusion;

[0005] During the tracking process, it can be done through techniques such as Kalman filters or deep learning models, which allow accurate prediction of the object position corresponding to the possible target in the next frame. For example, in 2016, Alex Bewley et al. proposed SORT, an online real-time multi-target tracking method that uses convolutional neural networks to detect pedestrians. This is one of the first multi-target tracking algorithms. In addition, the SORT algorithm uses Faster-RCNN for target detection to obtain the corresponding position coordinates (position information), classification, and credibility (confidence), and feeds the target's state (such as position coordinates, classification, and credibility) back to the Kalman filter for target association. However, it ignores the appearance features of the detected object and loses the tracking value in actual application scenarios. DeepSORT is an improvement on SORT, introducing appearance features in data association to improve the robustness of tracking.

[0006] MOTDT is similar to DeepSORT, and its main purpose is to enhance the association efficiency of SORT and DeepSORT. Various data association methods such as SORT, DeepSORT, FairMOT, MeMOT, etc. use position and motion clues or appearance features in different ways to achieve robust tracking results.

[0007] Bytetrack proposed a universal association method that associates almost every detection box, not just the high-scoring detection boxes. OC-SORT designed a tracker based on a pure motion model, which can improve tracking accuracy in crowded scenes. Some short-term trackers use independent re-identification (reID) mechanisms for long-term association.

[0008] In practical applications, how to efficiently and quickly solve MOT is a challenging task due to target occlusion, loss, clustering, complexity transformation, etc. in crowded scenes. Among these influencing factors, target loss caused by high clustering occlusion has a greater impact on the performance of MOT trackers. Summary of the invention

[0009] The present invention discloses a technical solution of an online multi-target tracking method and system based on feature fusion and iterative association.

[0010] According to one aspect of the present disclosure, there is provided an online multi-target tracking based on feature fusion and iterative association, comprising: fusing and updating the confidence using position information and confidence corresponding to possible targets in each frame image in a given video sequence to obtain a fused updated confidence;

[0011] Determine the multiple thresholds corresponding to the fusion update confidence by using an iterative algorithm or an adaptive threshold method;

[0012] Based on the multiple thresholds, dividing the fused updated confidence into multiple target detection frame intervals;

[0013] The position information corresponding to the multiple target detection frame intervals is used to track possible targets in each frame image in the given video sequence.

[0014] Preferably, the method of fusing and updating the confidence using the position information and confidence corresponding to the possible targets in each frame image in a given video sequence to obtain the fused updated confidence includes: determining the corresponding target bounding box or target detection box based on the position information corresponding to the possible targets in each frame image; calculating the target bounding box or target detection box corresponding to the possible target and the height corresponding to each frame image to determine the corresponding height difference; fusing and optimizing the confidence based on the height difference to obtain the fused updated confidence; and updating the fused optimized confidence using the fused updated confidence.

[0015] Preferably, the method of calculating the target bounding box or target detection box corresponding to the possible target and the height corresponding to each frame image, and determining the corresponding height difference includes: determining the vertical coordinate corresponding to the lowest point or the highest point of the target bounding box or target detection box corresponding to the possible target; calculating the first difference between the height corresponding to each frame image and the vertical coordinate, and determining the corresponding height difference.

[0016] Preferably, the method of fusing and optimizing the confidence based on the height difference to obtain a fused and updated confidence includes: obtaining the information range level of the target bounding box or the target detection box corresponding to the possible target; multiplying the information range level by the height difference to obtain the height difference corresponding to the information range level; calculating a second difference between 1 and the height difference corresponding to the information range level; and based on the second difference, fusing and optimizing the confidence to obtain a fused and updated confidence.

[0017] Preferably, the method of fusing and optimizing the confidence based on the second difference to obtain a fused updated confidence includes: obtaining an adjustable weight parameter; adjusting the confidence using the adjustable weight parameter to obtain an adjusted confidence; and adding the adjusted confidence to the second difference to obtain a fused updated confidence.

[0018] Preferably, the method of adjusting the confidence level by using the adjustable weight parameter to obtain the adjusted confidence level comprises: multiplying the adjustable weight parameter by the confidence level to obtain the adjusted confidence level.

[0019] Preferably, the method of determining the multiple thresholds corresponding to the fusion update confidence by using an iterative algorithm includes: establishing an inter-class variance objective function corresponding to the multiple thresholds to be determined Iteratively calculate the change average fusion update confidence U corresponding to each change threshold in the multiple thresholds to be determined i or μ i The average confidence U corresponding to the fusion update confidence T or μ T The sum of squares corresponding to the multiple third differences between the two; in the process of the iterative calculation, the inter-class variance objective function The multiple thresholds corresponding to the maximum are determined as the corresponding multiple thresholds; wherein the sum of the multiple thresholds is configured to be 1.

[0020] Preferably, the establishment of the inter-class variance objective function corresponding to the multiple thresholds to be determined The process also includes: obtaining a preset threshold, establishing an inter-class variance objective function including the preset threshold and the multiple thresholds to be determined; wherein the preset threshold is configured as the threshold corresponding to the non-possible target closest to the background in each frame image in the given video sequence; wherein the sum of the preset threshold and the multiple thresholds is configured to be 1.

[0021] Preferably, the iterative calculation of the change average fusion update confidence U corresponding to each change threshold value in the multiple threshold values ​​to be determined is i or μ i The average confidence U corresponding to the fusion update confidence T or μ T The method for summing the squares of the third differences between the two comprises: obtaining a set number of iterations, a plurality of initial change thresholds corresponding to the plurality of thresholds to be determined, an initial change average fusion update confidence U i or μ i and its corresponding first set change rate, and the second set change rate corresponding to the multiple initial thresholds; calculate the average confidence U corresponding to the fusion update confidence T or μ T Based on the multiple initial change thresholds corresponding to the multiple thresholds to be determined, the initial change average fusion update confidence U i or μ i and its corresponding first set change rate, the second set change rate corresponding to the multiple initial thresholds, and the average fusion update confidence U corresponding to each change threshold in the multiple thresholds to be determined within the set number of iterations i or μ i The average confidence U corresponding to the fusion update confidence T or μ TThe sum of squares corresponding to the third multiple differences between them.

[0022] Preferably, when the number of iterations is greater than 1, the change average fusion update confidence U corresponding to each change threshold in the multiple thresholds to be determined is i or μ i The configuration is that the average fusion update confidence of the change in the previous iteration is multiplied by the first set change rate; each change threshold value in the multiple thresholds to be determined is configured as the change threshold value in the previous iteration multiplied by the second set change rate.

[0023] Preferably, the method of using the adaptive threshold method to determine the multiple thresholds corresponding to the fusion update confidence includes: sorting the fusion update confidence according to size to obtain the sorted fusion update confidence; based on the sorted fusion update confidence and the set aggregation degree, determining the aggregation interval corresponding to the set aggregation degree; using the aggregation interval and the numbers corresponding to the multiple thresholds, determining the multiple thresholds corresponding to the fusion update confidence.

[0024] Preferably, the method of determining the multiple thresholds corresponding to the fusion update confidence by utilizing the clustering intervals and the numbers corresponding to the multiple thresholds includes: determining the multiple clustering interval coefficients corresponding to the multiple thresholds based on the minimum fusion update confidence and / or the maximum fusion update confidence corresponding to the sorted fusion update confidence, the clustering intervals and the numbers corresponding to the multiple thresholds; determining the multiple thresholds on the curve corresponding to the sorted fusion update confidence based on the multiple clustering interval coefficients; wherein the interval range corresponding to each clustering interval coefficient in the multiple clustering interval coefficients includes the clustering interval.

[0025] Preferably, the method for determining multiple clustering interval coefficients corresponding to the multiple thresholds based on the minimum fusion update confidence and / or the maximum fusion update confidence corresponding to the sorted fusion update confidence, the clustering interval and the number corresponding to the multiple thresholds includes: taking the minimum fusion update confidence and / or the maximum fusion update confidence as the starting point, respectively determining the multiple clustering interval coefficients corresponding to the multiple thresholds including the clustering interval.

[0026] Preferably, based on the multiple thresholds corresponding to the fusion update confidence determined by using the iterative algorithm, the multiple thresholds corresponding to the fusion update confidence determined by using the adaptive threshold method are verified; and / or, the number corresponding to the multiple thresholds is configured to be 2, and the number corresponding to the multiple target detection box intervals into which the fusion update confidence is divided is 3.

[0027] Preferably, when the number of target detection frame intervals into which the fusion update confidence is divided is 3 or the number corresponding to the multiple thresholds is configured to be 2, the minimum fusion update confidence and the maximum fusion update confidence are used as starting points, and the two clustering interval coefficients corresponding to the two thresholds including the clustering interval are determined respectively; wherein the numerical values ​​corresponding to the two clustering interval coefficients are the same, and the numerical values ​​corresponding to the two clustering interval coefficients are configured to be 2 / 3.

[0028] Preferably, the method of dividing the fusion update confidence into multiple target detection frame intervals based on the multiple thresholds includes: configuring the interval from the minimum fusion update confidence in the fusion update confidence to the minimum threshold in the multiple thresholds as a first detection frame interval; respectively configuring the intervals corresponding to adjacent thresholds in the multiple thresholds as multiple second detection frame intervals; and configuring the interval from the maximum threshold in the multiple thresholds to the maximum fusion update confidence in the fusion update confidence as a third detection frame interval.

[0029] Preferably, the number corresponding to the multiple thresholds is configured as 2, and the 2 thresholds are respectively configured as the first threshold and the second threshold; the first threshold and the second threshold are used to divide the fusion update confidence into 3 target detection frame intervals, and the 3 target detection frame intervals are respectively configured as a high-score detection frame interval (third detection frame interval), a medium-score detection frame interval (second detection frame interval) and a low-score detection frame interval (first detection frame interval).

[0030] Preferably, the method of tracking possible targets in each frame image in the given video sequence by using the position information corresponding to the multiple target detection frame intervals includes: determining the position information corresponding to the trajectory mark detection frame interval used to initialize the trajectory from the position information corresponding to the multiple target detection frame intervals; initializing the track list being tracked by using the position information corresponding to the track mark detection frame interval; and tracking possible targets in each frame image in the given video sequence based on the initialized track list being tracked and the position information corresponding to the multiple target detection frame intervals.

[0031] Preferably, the method for determining the position information corresponding to the trajectory mark detection frame interval used to initialize the trajectory from the position information corresponding to the multiple target detection frame intervals, includes: respectively configuring the intervals corresponding to adjacent thresholds among the multiple thresholds as multiple second detection frame intervals; configuring the interval corresponding to the maximum threshold among the multiple thresholds to the maximum fusion update confidence in the fusion update confidence as a third detection frame interval; determining the position information corresponding to the trajectory mark detection frame interval used to initialize the trajectory from the position information corresponding to the third detection frame interval and / or any one or more detection frame intervals among the multiple second detection frame intervals.

[0032] Preferably, the method for tracking possible targets in each frame image in the given video sequence based on the initialized list of tracks being tracked and the position information corresponding to the multiple target detection frame intervals comprises: predicting the tracks in the initialized list of tracks being tracked to obtain corresponding predicted position information; tracking possible targets in each frame image in the given video sequence based on the predicted position information and the position information corresponding to the multiple target detection frame intervals.

[0033] Preferably, the method for tracking possible targets in each frame of the given video sequence based on the predicted position information and the position information corresponding to the multiple target detection frame intervals includes: constructing a corresponding first cost matrix using the predicted position information and the position information corresponding to the third detection frame interval from the maximum threshold among multiple thresholds in the multiple target detection frame intervals to the maximum fusion update confidence in the fusion update confidence; matching the predicted position information in the first cost matrix with the position information to obtain a first matching trajectory corresponding to a first optimal matching combination, a first unmatched detection value, and the predicted position information or the first unmatched trajectory corresponding to the list of trajectories being tracked; wherein the first unmatched detection value is configured as the position information corresponding to the third detection frame interval; constructing multiple second cost matrices using the position information corresponding to multiple second detection frame intervals corresponding to intervals corresponding to adjacent thresholds in the multiple thresholds, the first unmatched detection value, and the first unmatched trajectory; matching the first unmatched trajectory in the multiple second cost matrices with the position information , obtain the second matching trajectory corresponding to the corresponding second optimal matching combination, the second unmatched detection value, and the second unmatched trajectory corresponding to the first unmatched trajectory; wherein the second unmatched detection value is configured as the position information corresponding to the second detection frame interval; use the position information corresponding to the first detection frame interval with the minimum fusion update confidence in the fusion update confidence to the minimum threshold among the multiple thresholds and the second unmatched trajectory to construct a third cost matrix; match the second unmatched trajectory in the third cost matrix with the position information to obtain the third matching trajectory corresponding to the third optimal matching combination, the third unmatched detection value, and the third unmatched trajectory corresponding to the second unmatched trajectory; wherein the first unmatched detection value is configured as the position information corresponding to the third detection frame interval; based on the first matching trajectory, the second matching trajectory and the third matching trajectory, track possible targets in each frame image in the given video sequence; and / or, use the first matching trajectory, the second matching trajectory and the third matching trajectory to update the trajectory in the trajectory list being tracked.

[0034] Preferably, the method of constructing multiple second cost matrices by respectively using the position information corresponding to multiple second detection frame intervals corresponding to intervals corresponding to adjacent thresholds among the multiple thresholds, the first unmatched detection value and the first unmatched trajectory, includes: configuring the multiple second detection frame intervals as second detection frame intervals whose fusion update confidences are successively reduced and cascaded; constructing a first sub-cost matrix among the multiple second cost matrices by using the position information corresponding to the second sub-detection frame interval corresponding to the highest fusion update confidence among the intervals corresponding to adjacent thresholds among the multiple thresholds, the first unmatched detection value and the first unmatched trajectory; constructing a second sub-cost matrix among the multiple second cost matrices by using the position information corresponding to the second sub-unmatched trajectory and the second sub-unmatched detection value corresponding to the first sub-cost matrix, and the position information corresponding to the detection frame interval cascaded with the second sub-detection frame interval corresponding to the highest fusion update confidence; and constructing the sub-cost matrix to be constructed this time by using the position information corresponding to the unmatched trajectory and the unmatched detection value corresponding to the sub-cost matrix constructed last time and the position information corresponding to the detection frame interval of the sub-cost matrix to be constructed this time.

[0035] Preferably, the method for matching the first unmatched trajectory in the multiple second cost matrices with the position information includes: respectively determining the first rectangular frame coordinates corresponding to the first unmatched trajectory and the second rectangular frame coordinates describing the position information in the multiple second cost matrices; respectively determining the corresponding intersection-and-joint ratio and width intersection-and-joint ratio based on the first rectangular frame coordinates and the second rectangular frame coordinates; respectively determining the corresponding width modulation intersection-and-joint ratio based on the intersection-and-joint ratio and the width intersection-and-joint ratio; and performing width modulation intersection-and-joint ratio matching on the predicted position information and the position information in the second cost matrix.

[0036] Preferably, the method for determining the corresponding width intersection-and-joint ratio based on the first rectangular frame coordinates and the second rectangular frame coordinates, respectively, includes: respectively determining the first point coordinates corresponding to the first rectangular frame coordinates and the second point coordinates corresponding to the second rectangular frame coordinates; wherein the first point coordinates and the second point coordinates are in a diagonal relationship; respectively determining the maximum horizontal coordinate and the minimum horizontal coordinate of the first horizontal coordinate in the first point coordinates and the second horizontal coordinate in the second point coordinates; and respectively determining the corresponding width intersection-and-joint ratio based on the maximum horizontal coordinate and the minimum horizontal coordinate.

[0037] Preferably, the method for determining the corresponding intersection-and-joint ratio and width intersection-and-joint ratio based on the maximum horizontal coordinate and the minimum horizontal coordinate, respectively, includes: respectively calculating the first horizontal coordinate difference corresponding to the minimum horizontal coordinate and the maximum horizontal coordinate; respectively calculating the second horizontal coordinate difference corresponding to the maximum horizontal coordinate and the minimum horizontal coordinate; respectively calculating the ratio of the first horizontal coordinate difference to the second horizontal coordinate difference, and determining the corresponding width intersection-and-joint ratio.

[0038] Preferably, before performing width modulation intersection-and-parallel matching on the predicted position information and the position information in the second cost matrix, it includes: adjusting the width modulation intersection-and-parallel ratio using a preset adjustment parameter to obtain an adjusted width modulation intersection-and-parallel ratio; and performing adjusted width modulation intersection-and-parallel matching on the predicted position information and the position information in the second cost matrix.

[0039] Preferably, the method for width modulated intersection-and-parallel matching of the adjusted predicted position information and position information in the second cost matrix includes: obtaining a Hungarian matching algorithm, and using the Hungarian matching algorithm to perform width modulated intersection-and-parallel matching of the adjusted predicted position information and position information in the second cost matrix.

[0040] Preferably, the method of adjusting the width modulation I / O ratio by using a preset adjustment parameter to obtain an adjusted width modulation I / O ratio comprises: calculating the preset adjustment parameter power corresponding to the width modulation I / O ratio to obtain the adjusted width modulation I / O ratio.

[0041] Preferably, the method for determining the preset adjustment parameter includes: respectively obtaining adjacent thresholds among the multiple thresholds corresponding to each second detection frame interval in the multiple second detection frame intervals; calculating the difference between the adjacent thresholds; and determining the preset adjustment parameter.

[0042] Preferably, the method for calculating the difference between the adjacent thresholds is configured as a numerical value greater than or equal to 0; or, the method for calculating the difference between the adjacent thresholds includes: determining the maximum threshold and the minimum threshold corresponding to the adjacent thresholds; and calculating the difference between the maximum threshold and the minimum threshold.

[0043] Preferably, the method for determining the preset adjustment parameters includes: respectively determining set order of magnitude parameters of the intersection-and-union ratio or width intersection-and-union ratio corresponding to multiple second cost matrices; and determining the preset adjustment parameters based on the difference between the adjacent thresholds and the set order of magnitude parameters.

[0044] Preferably, the method for determining the preset adjustment parameter based on the difference between the adjacent threshold values ​​and the set order of magnitude parameter, respectively, includes: respectively calculating the ratio of the difference between the adjacent threshold values ​​to the set order of magnitude parameter; and respectively configuring or determining the ratio as the corresponding preset adjustment parameter.

[0045] Preferably, the method of determining the corresponding width modulation I / O ratio based on the I / O ratio and the width I / O ratio respectively comprises: multiplying the I / O ratio by the corresponding width I / O ratio respectively to determine the corresponding width modulation I / O ratio.

[0046] Preferably, the first point coordinates and the second point coordinates in a diagonal relationship are configured as the upper left corner corresponding to the first rectangular frame coordinates and the lower right corner corresponding to the second rectangular frame coordinates; or, the first point coordinates and the second point coordinates in a diagonal relationship are configured as the lower left corner corresponding to the first rectangular frame coordinates and the upper right corner corresponding to the second rectangular frame coordinates; or, the first point coordinates and the second point coordinates in a diagonal relationship are configured as the upper right corner corresponding to the first rectangular frame coordinates and the lower left corner corresponding to the second rectangular frame coordinates; or, the first point coordinates and the second point coordinates in a diagonal relationship are configured as the lower right corner corresponding to the first rectangular frame coordinates and the upper left corner corresponding to the second rectangular frame coordinates.

[0047] Preferably, the method for matching the predicted position information in the first cost matrix with the position information comprises: obtaining a Hungarian matching algorithm, and performing intersection-and-union matching on the predicted position information in the first cost matrix with the position information using the Hungarian matching algorithm.

[0048] Preferably, the method for matching the second unmatched trajectory in the third cost matrix with the position information comprises: obtaining a Hungarian matching algorithm, and using the Hungarian matching algorithm to match the second unmatched trajectory in the third cost matrix with the position information to perform intersection-and-union matching.

[0049] Preferably, before fusing and updating the confidence using the position information and confidence corresponding to the possible target in each frame image in a given video sequence, it also includes: acquiring or detecting the position information and confidence corresponding to the possible target in each frame image in a given video sequence.

[0050] Preferably, the method for detecting position information and confidence corresponding to possible targets in each frame image in a given video sequence includes: obtaining a target detector; and using the target detector to detect position information and confidence corresponding to possible targets in each frame image in a given video sequence.

[0051] According to one aspect of the present disclosure, an online multi-target tracking device / system based on feature fusion and iterative association is provided, comprising:

[0052] A fusion updating unit, used to fuse and update the confidence by using the position information and confidence corresponding to the possible target in each frame image in a given video sequence to obtain a fusion updated confidence;

[0053] A threshold determination unit, used to determine a plurality of thresholds corresponding to the fusion update confidence determination by using an iterative algorithm;

[0054] A target detection frame interval division unit, configured to divide the fused updated confidence into a plurality of target detection frame intervals based on the plurality of thresholds;

[0055] The target tracking unit is used to track possible targets in each frame image in the given video sequence by using the position information corresponding to the multiple target detection frame intervals.

[0056] According to one aspect of the present disclosure, an online multi-target tracking system based on feature fusion and iterative association is provided, comprising: a processor; a memory for storing processor executable instructions; wherein the processor is configured to call the instructions stored in the memory to execute the above-mentioned online multi-target tracking method.

[0057] According to one aspect of the present disclosure, there is provided an online multi-target tracking system based on feature fusion and iterative association, comprising: a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions implement the above-mentioned online multi-target tracking method when executed by a processor.

[0058] According to one aspect of the present disclosure, an online multi-target tracking system based on feature fusion and iterative association is provided, including: a computer program product, including a computer program / instruction, which implements the above-mentioned online multi-target tracking method when executed by a processor.

[0059] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to: execute the above-mentioned online multi-target tracking method.

[0060] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the above-mentioned online multi-target tracking method is implemented.

[0061] In the disclosed embodiments, the present disclosure proposes a technical solution of an online multi-target tracking method and system based on feature fusion and iterative association to solve at least one technical problem that the existing associations in challenging scenarios are prone to mismatching and the distinction provided by the appearance information when two adjacent targets are highly overlapped is limited.

[0062] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

[0063] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.

[0065] Figure 1 A flowchart of an online multi-target tracking method based on feature fusion and iterative association according to an embodiment of the present disclosure is shown;

[0066] Figure 2 A block diagram corresponding to an online multi-target tracking method based on feature fusion and iterative association according to an embodiment of the present disclosure is shown;

[0067] Figure 3 A schematic diagram showing depth information in each frame of an image in a given video sequence according to an embodiment of the present disclosure;

[0068] Figure 4 A threshold determination schematic diagram corresponding to an adaptive threshold example according to an embodiment of the present disclosure is shown;

[0069] Figure 5 A schematic diagram showing the principle of the width modulation IOU according to an embodiment of the present disclosure;

[0070] Figure 6 A schematic diagram of the position relationship between the detection frame and the trajectory in different extreme cases according to an embodiment of the present disclosure is shown;

[0071] Figure 7 A schematic diagram of MOTA comparison on each subset of the validation set of two data sets according to an embodiment of the present disclosure is shown;

[0072] Figure 8 is a block diagram of an electronic device 800 according to an exemplary embodiment;

[0073] Fig. 9 is a block diagram of an electronic device 1900 according to an exemplary embodiment. DETAILED DESCRIPTION

[0074] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0075] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0076] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.

[0077] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0078] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not go into details.

[0079] In addition, the present disclosure also provides an online multi-target tracking device / system based on feature fusion and iterative association, an electronic device, a computer-readable storage medium, and a program, all of which can be used to implement any one of the online multi-target tracking methods based on feature fusion and iterative association provided by the present disclosure. The corresponding technical solutions and descriptions can be found in the corresponding records in the online multi-target tracking method based on feature fusion and iterative association section, and will not be repeated here.

[0080] Figure 1 A flowchart of an online multi-target tracking method based on feature fusion and iterative association according to an embodiment of the present disclosure is shown. Figure 1As shown, the online multi-target tracking method includes: step S101: using the position information and confidence corresponding to the possible target in each frame image in a given video sequence to fuse and update the confidence to obtain the fused updated confidence; step S102: using an iterative algorithm or an adaptive threshold method to determine the multiple thresholds corresponding to the fused updated confidence; step S103: based on the multiple thresholds, dividing the fused updated confidence into multiple target detection frame intervals; step S104: using the position information corresponding to the multiple target detection frame intervals to track the possible target in each frame image in the given video sequence. To solve at least one technical problem that the existing association in challenging scenes is prone to mismatching and the appearance information provided when two adjacent targets are highly overlapped is limited.

[0081] Figure 2 FIG. 2 shows a block diagram corresponding to an online multi-target tracking method based on feature fusion and iterative association according to an embodiment of the present disclosure. Figure 2 As shown, the present invention takes the bytetrack algorithm as a benchmark, strengthens the feature expression of the detection results, and makes adaptive and detailed divisions. Similar to bytetrack, it assumes that the detection results are near the tracking values, so IoU is used to calculate the distance between the trajectory and the detection value. It adds three data association stages on the basis of bytetrack. Specifically, the first time, the detection values ​​that are determined to be targets and easy to match (high confidence) are matched with the trajectory, the second time, the detection values ​​that are determined to be targets but easy to confuse (medium confidence) are matched with the trajectory, and the third time is associated with the remaining low confidence. The method proposed by the present invention is highlighted in the dotted box.

[0082] Among them, the real-time MOT tracker / tracking method is based on the TBD framework. The video (given video sequence) is in frames. The target detector (hereinafter referred to as the detector) first detects possible targets in all video frames and marks the corresponding probability values. Then, the existing or updated trajectories are matched based on the probability values ​​corresponding to the possible targets, and the matched trajectories are updated to achieve real-time multi-target tracking.

[0083] like Figure 2 As shown in FIG. 1 , the number of target detection box intervals is 3. The basic framework of this algorithm or method is Bytetrack, and several improvements are made (such as Figure 1The three dotted boxes correspond to the depth information fusion, the adaptive division threshold according to the confidence, and the cost matrix (WMIoU distance) shown in FIG. 1 ). Each frame of a given video sequence (hereinafter referred to as a video frame) is first detected by the target detector, and the position information and confidence corresponding to the possible target in each frame are output. The position information and confidence corresponding to the possible target in each frame are updated by the depth information fusion module to obtain the corresponding fused updated confidence.

[0084] like Figure 2 As shown, an adaptive threshold division module divides the fusion update confidence corresponding to the possible target in each frame image into three confidence intervals (high score detection frame, medium score detection frame, low score detection frame) according to the confidence adaptive division threshold method. The trajectory in the trajectory list being tracked obtains the corresponding predicted position information after being predicted by the Kalman filter algorithm; the corresponding cost matrix (cost matrix) is constructed using the predicted position information and the position information output by the target detector; the predicted position information in the cost matrix is ​​matched with the position information output by the target detector by IoU using the Hungarian matching algorithm to obtain the matching trajectory (matched trajectory), unmatched detection value (position information output by the target detector) and unmatched trajectory (trajectory in the trajectory list being tracked) corresponding to the optimal matching combination.

[0085] like Figure 2 As shown in the figure, the three confidence intervals (high-score detection frame, medium-score detection frame, and low-score detection frame) are updated with the parameters corresponding to the position information of the Kalman filter algorithm through the corresponding cost matrix and the optimal matching combination (matched trajectory) obtained by the Hungarian matching algorithm; and the trajectory in the trajectory list being tracked is updated using the optimal matching combination (matched trajectory) corresponding to the three confidence intervals (high-score detection frame, medium-score detection frame, and low-score detection frame).

[0086] like Figure 2 As shown in the figure, the second data association match uses the width-based WMIoU match, and the detection values ​​that are temporarily not matched in the first and second data association matches are added to the third data association match, and finally the trajectory update of one frame is completed. The unmatched trajectories in the third data association are marked as lost trajectories, and the storage time is set to a set frame (for example, 30 frames). During the storage period, they participate in the next target matching. If they are still not matched after the time, they are discarded.

[0087] Step S101: The confidence is updated by fusing the position information and confidence corresponding to the possible target in each frame image in a given video sequence to obtain a fused updated confidence.

[0088] In an embodiment of the present disclosure, before fusing and updating the confidence using the position information and confidence corresponding to the possible target in each frame image in a given video sequence, it also includes: acquiring or detecting the position information and confidence corresponding to the possible target in each frame image in a given video sequence.

[0089] In an embodiment of the present disclosure, the given video sequence is configured as a video sequence corresponding to a vehicle and / or a pedestrian. For example, a video camera or a camera may be used to shoot a vehicle and / or a pedestrian to obtain a video sequence corresponding to the vehicle and / or the pedestrian.

[0090] In an embodiment of the present disclosure, the method for detecting position information and confidence corresponding to possible targets in each frame image in a given video sequence includes: obtaining a target detector (setting a target detector or detectors); using the target detector to detect position information and confidence corresponding to possible targets in each frame image in a given video sequence.

[0091] In the embodiments of the present disclosure and other possible embodiments, the target detector (set target detector or detector) corresponding to the target detection based on the TBD (tracking-by-detection) paradigm / framework of target tracking detects the target of interest in each frame of the given video sequence, and obtains the corresponding position coordinates (position information), classification, and credibility (confidence).

[0092] Specifically, in the embodiments of the present disclosure and other possible embodiments, based on the TBD (tracking-by-detection) paradigm / framework of target tracking, target detection is divided into two parts. First, the first part uses an efficient detector to detect the target of interest in each frame, and obtains the corresponding position coordinates (position information), classification, and credibility (confidence). The second part associates the same target in two adjacent frames. A common association method is to calculate the similarity between the objects framed by each positioning frame between the previous and next frames, and give the identity ID corresponding to each detection frame in the current frame according to the similarity matrix, and finally achieve tracking.

[0093] However, in challenging scenarios, such as occlusion and clustering, associations are prone to mismatches, and when two adjacent targets are highly overlapped, the distinction provided by appearance information is limited. In order to overcome the above shortcomings, based on advanced detectors, the present invention intends to design a new model, method, device or system under the framework of TBD, which can effectively promote the combination of detection and tracking, and efficiently handle target matching in complex scenarios.

[0094] In the embodiments of the present disclosure and other possible embodiments, based on tracking-by-detection, object detections (position information and confidence corresponding to possible targets) obtained from a detector are used to track the same object in subsequent frames.

[0095] In an embodiment of the present disclosure, the method of fusing and updating the confidence using the position information and confidence corresponding to the possible targets in each frame image in a given video sequence to obtain the fused updated confidence includes: determining the corresponding target bounding box or target detection box based on the position information corresponding to the possible targets in each frame image; calculating the target bounding box or target detection box corresponding to the possible target and the height corresponding to each frame image, and determining the corresponding height difference (H-y0); fusing and optimizing the confidence S based on the height difference (H-y0) to obtain the fused updated confidence S0; and updating the fused optimized confidence using the fused updated confidence.

[0096] In an embodiment of the present disclosure, the method of calculating the target bounding box or target detection box corresponding to the possible target and the height corresponding to each frame image, and determining the corresponding height difference includes: determining the vertical coordinate corresponding to the lowest point or the highest point of the target bounding box or target detection box corresponding to the possible target; calculating the first difference between the height corresponding to each frame image and the vertical coordinate, and determining the corresponding height difference.

[0097] In an embodiment of the present disclosure, the method of fusing and optimizing the confidence based on the height difference to obtain a fused updated confidence includes: obtaining an information range level 10 of a target bounding box or a target detection box corresponding to the possible target; -3 ; The information scope level 10 -3 Multiply by the height difference (H-y0) to obtain the height difference (H-y0) corresponding to the information range level * 10 -3 ; Calculate the second difference 1-(H-y0) between 1 and the height difference corresponding to the information range level * 10 -3 ; Based on the second difference 1-(H-y0) * 10 -3 , the confidence is fused and optimized to obtain a fused updated confidence S0.

[0098] In an embodiment of the present disclosure, the method of fusing and optimizing the confidence based on the second difference to obtain a fused updated confidence includes: obtaining an adjustable weight parameter α; adjusting the confidence S using the adjustable weight parameter α to obtain an adjusted confidence; adding the adjusted confidence to the second difference 1-(H-y0) *10 -3 , and obtain the fusion update confidence S0.

[0099] In an embodiment of the present disclosure, the method of adjusting the confidence level by using the adjustable weight parameter to obtain the adjusted confidence level comprises: multiplying the adjustable weight parameter α by the confidence level S to obtain the adjusted confidence level S * α.

[0100] In the embodiments of the present disclosure and other possible embodiments, the multi-target tracker under the TBD paradigm is generally composed of two parts: a target detector and a trajectory association discriminator. The overall tracking performance depends largely on the performance of the detector. The development of target detectors such as YOLOX has shown strong performance in detection and has good performance in complex scenarios. However, an obvious disadvantage of this paradigm is that the association between the two parts is not obvious, which leads to the inability to reflect the high efficiency and high performance of the detector in the target tracking problem. The matching efficiency fluctuations in crowded or highly clustered scenarios further aggravate this phenomenon. Therefore, making more use of the high-confidence information output by the detector in the associated part can greatly improve the overall tracking effect.

[0101] In the embodiments of the present disclosure and other possible embodiments, taking the YOLOX algorithm as an example, the information output when used as a detector is the detection target box and the confidence score. This can be regarded as trustworthy information in the entire tracking process, combined with the purpose of solving the problem of reduced tracking performance in complex scenes. The present invention finds that the location information of the target in the image can also reflect the priority during association matching.

[0102] Figure 3 FIG. 2 is a schematic diagram showing depth information in each frame of an image in a given video sequence according to an embodiment of the present disclosure. Figure 3 As shown in the figure, the vertical distance between the bottom edge of the pedestrian bounding box and the bottom edge of the image after target detection is used as the depth position information of the target. Assuming that the camera sensor position is fixed and all detected targets are on the same horizontal line, the Euclidean distance between the target and the next fixed point of the camera is positively correlated with the depth position information. Although it cannot correctly represent the actual position of the target in the 3D scene, it can promote the target segmentation in complex scenes. Under the same confidence level, the depth position information is negatively correlated with the feature distinction of the target itself, which is reflected in the tracking process as a higher degree of matching. After combining the depth information with the confidence level, the new confidence level replaces the confidence level (hereinafter referred to as confidence level or fusion updated confidence level). When dividing the threshold in the subsequent process, it is easier to divide the clusters, reduce the mutual interference during cascade matching, and improve the robustness of the overall tracking.

[0103] In the embodiment of the present disclosure and other possible embodiments, in the output information of the target detector, the confidence range is 0 to 1, the information range of the target bounding box is 10-3 level, assuming that the confidence is S, the lower right corner position coordinates of the bounding box (target bounding box) corresponding to the position information are assumed to be (x0, y0), the height of each frame image is H, and the confidence S0 of the fusion update is calculated as follows:

[0104] S0=S * α+(1-(H-y0) * 10 -3 )

[0105] Among them, α is an adjustable weight parameter, y0 is the height of the target bounding box, and H is the height of the entire image corresponding to each frame of the image. The present invention believes that the depth information has a high reliability, so the depth information is reduced to an order of magnitude of 10 0 After the level is reached, the correlation between the depth position information and the target’s own feature distinction is adjusted, and finally combined with the original confidence, S0 replaces S with the new confidence, and participates in the subsequent matching association steps.

[0106] Step S102: using an iterative algorithm or an adaptive threshold method to determine multiple thresholds corresponding to the fusion update confidence.

[0107] In the embodiments of the present disclosure and other possible embodiments, in the early image segmentation picture processing process, when the operation such as binarization is performed before laying, the desired result is to be able to retain all the information in the corresponding area of ​​the picture. In the laboratory environment, the corresponding material is templated, but when the laboratory method is applied to the real environment, the present invention will find that the influence of the light and shadow environment on the effect is actually very large. Processing in this case will make the result unsatisfactory: one piece is black, one piece is white, and the features of the black area cannot be extracted. At this time, the adaptive threshold algorithm is particularly important. Unlike the global threshold, it pays more attention to the contextual relationship, divides the original picture into smaller areas for judgment, and greatly reduces the impact of the shadow on the picture itself. In fact, the essence of this algorithm is the local threshold. Generally, the weighted average method is used to obtain the local threshold, which can obtain a more accurate segmented image as a whole. Similarly, the present invention also hopes that the algorithm can "see" targets that are easier to distinguish during the association process of the multi-target tracking algorithm. Therefore, the present invention attempts to transfer this idea to multi-target tracking to further improve the performance of the algorithm.

[0108] In the embodiments of the present disclosure and other possible embodiments, in the bytetrack algorithm, the author divides different confidence intervals by preset thresholds. The setting of thresholds requires a large number of repeated experiments. Different data sets have different thresholds according to the complexity of the data (for example, the MOT17 data set is more scattered, and the threshold is set to 0.6, while the MOT20 data set is denser than the MOT17 data set, and the threshold is set to 0.3). For a new data set, the corresponding threshold can also be set by knowing the density in advance. In order to have better experimental data, different subsets of the same data set will also have different threshold divisions. Although ByteTrack decouples low-score occlusions from the scene, low-score targets are still crowded. In this case, IoU-based data association is prone to matching errors, thereby limiting the tracking ability to handle occlusions. However, under actual scene changes, the density of future moments is difficult to predict. The present invention requires a method for adaptively classifying low-confidence and high-confidence detections output by deep learning detectors.

[0109] In the embodiments of the present disclosure and other possible embodiments, in the field of image segmentation, the target area and the background area are segmented from the grayscale image. For a color-balanced image, the thresholding of the image can be completed by directly using a threshold. However, sometimes the color of the image is uneven. At this time, if only one threshold is used, a clear and effective threshold-segmented image cannot be obtained. Therefore, an adaptive thresholding method, also called a local thresholding method, has emerged. Its idea is not to calculate the threshold of the global image, but to obtain the threshold by calculating the weighted average of the adjacent areas of each pixel axis, and use the threshold to process the current pixel. Different thresholds can be calculated adaptively. Inspired by this, it is also effective to apply its method to the thresholding of target tracking.

[0110] First, the threshold is not a "dividing line" for the quantity ratio; second, the threshold should have a "mathematical property" that describes the data trend. In the video sequence input to the detector as a whole, each frame of the image is reflected as a local part. The local complexity transformation is difficult to effectively segment the confidence by setting a fixed threshold for the whole. Therefore, it is necessary to calculate the threshold separately for each frame of the image, which is the so-called adaptive threshold.

[0111] In an embodiment of the present disclosure, the method of using an iterative algorithm to determine the multiple thresholds corresponding to the fusion update confidence includes: establishing an inter-class variance objective function corresponding to the multiple thresholds to be determined Iteratively calculate the change average fusion update confidence U corresponding to each change threshold in the multiple thresholds to be determined i or μ i The average confidence U corresponding to the fusion update confidence Tor μ T The sum of squares corresponding to the multiple third differences between the two; in the process of the iterative calculation, the inter-class variance objective function The multiple thresholds corresponding to the maximum are determined as the corresponding multiple thresholds; wherein the sum of the multiple thresholds is configured to be 1.

[0112] In the embodiment of the present disclosure, the establishment of the inter-class variance objective function corresponding to the multiple thresholds to be determined The process also includes: obtaining a preset threshold, establishing an inter-class variance objective function including the preset threshold and the multiple thresholds to be determined; wherein the preset threshold is configured as the threshold corresponding to the non-possible target closest to the background in each frame image in the given video sequence; wherein the sum of the preset threshold and the multiple thresholds is configured to be 1.

[0113] In the embodiment of the present disclosure, the iterative calculation of the change average fusion update confidence U corresponding to each change threshold value in the multiple threshold values ​​to be determined i or μ i The average confidence U corresponding to the fusion update confidence T or μ T The method for summing the squares of the third differences between the two comprises: obtaining a set number of iterations, a plurality of initial change thresholds corresponding to the plurality of thresholds to be determined, an initial change average fusion update confidence U i or μ i and its corresponding first set change rate, and the second set change rate corresponding to the multiple initial thresholds; calculate the average confidence U corresponding to the fusion update confidence T or μ T Based on the multiple initial change thresholds corresponding to the multiple thresholds to be determined, the initial change average fusion update confidence U i or μ i and its corresponding first set change rate, the second set change rate corresponding to the multiple initial thresholds, and the average fusion update confidence U corresponding to each change threshold in the multiple thresholds to be determined within the set number of iterations i or μ i The average confidence U corresponding to the fusion update confidence T or μ T The sum of squares of the corresponding multiple third differences between them.

[0114] In the embodiment of the present disclosure and other possible embodiments, when the number of iterations is greater than 1, the change average fusion update confidence U corresponding to each change threshold in the multiple thresholds to be determined is i or μ iThe configuration is that the average fusion update confidence of the change in the previous iteration is multiplied by the first set change rate; each change threshold value in the multiple thresholds to be determined is configured as the change threshold value in the previous iteration multiplied by the second set change rate.

[0115] In the embodiment of the present disclosure and other possible embodiments, the fusion update confidence is divided into a plurality of target detection frame intervals corresponding to a number of 3 or the plurality of thresholds corresponding to a number of 2 particles for illustration.

[0116] In the embodiment of the present disclosure and other possible embodiments, the measurement standard used is the maximum inter-class variance (inter-class variance objective function) Among them, the difference between classes The larger it is, the smaller the intra-class distance is, and the easier it is to classify the data, thus reducing the probability of subsequent errors. The specific calculation of the maximum inter-class variance is as follows:

[0117] ω0+ω1=1

[0118]

[0119] Among them, ω=[ω0,ω1] represents the proportion of the two parts after the threshold is determined, μ=[μ0,μ1] represents the average value of each part, and μ T is the overall average, when When the maximum, the thresholds for dividing ω0 and ω1 are the optimal thresholds, corresponding to the first threshold and the second threshold in the multiple thresholds respectively. Wherein, μ0 represents the average fusion update confidence corresponding to the first threshold; μ1 represents the average fusion update confidence corresponding to the second threshold.

[0120] In the embodiment of the present disclosure and other possible embodiments, within the threshold range of [0, 1], two values ​​need to be found to separate the three layers corresponding to high, medium, and low (three confidence intervals, i.e., high score detection frame, medium score detection frame, and low score detection frame). At this time, the inter-class variance is expressed as follows:

[0121]

[0122] After simplification, the inter-class variance objective function corresponding to the final inter-class variance formula is as follows:

[0123]

[0124] Among them, ω0 is the low confidence layer of the non-possible target closest to the background in the image, ω 1、 ω2 is the first threshold and the second threshold corresponding to the medium and high confidence layers (high score detection frame, medium score detection frame), μ i is the average value of each part. The present invention requires two thresholds, so the number of layers / intervals is configured as 3. iis the mean confidence value of the currently selected layer, U T is the overall mean, W i =[ω 0， ω 1， ω2], the current layer number current / interval ratio, let ω0=0, then when When ω is the maximum 1， ω2 is the first threshold and the second threshold in the optimal threshold correspondence.

[0125] In the embodiment of the present disclosure and other possible embodiments, the previous method finds the optimal threshold by traversing all possible thresholds, and its time complexity is O(n2). This method obviously cannot meet the real-time requirements.

[0126] In an embodiment of the present disclosure, the method of using an adaptive threshold method to determine the multiple thresholds corresponding to the fusion update confidence includes: sorting the fusion update confidences according to size to obtain sorted fusion update confidences; based on the sorted fusion update confidences and set aggregation, determining the aggregation interval corresponding to the set aggregation; using the aggregation interval and the numbers corresponding to the multiple thresholds, determining the multiple thresholds corresponding to the fusion update confidence.

[0127] In an embodiment of the present disclosure, the method of determining the multiple thresholds corresponding to the fusion update confidence by utilizing the clustering intervals and the numbers corresponding to the multiple thresholds includes: determining the multiple clustering interval coefficients corresponding to the multiple thresholds based on the minimum fusion update confidence and / or the maximum fusion update confidence corresponding to the sorted fusion update confidence, the clustering intervals and the numbers corresponding to the multiple thresholds; determining the multiple thresholds on the curve corresponding to the sorted fusion update confidence based on the multiple clustering interval coefficients; wherein the interval range corresponding to each clustering interval coefficient in the multiple clustering interval coefficients includes the clustering interval.

[0128] In an embodiment of the present disclosure, the method for determining multiple clustering interval coefficients corresponding to the multiple thresholds based on the minimum fusion update confidence and / or the maximum fusion update confidence corresponding to the sorted fusion update confidence, the clustering interval and the number corresponding to the multiple thresholds includes: taking the minimum fusion update confidence and / or the maximum fusion update confidence as the starting point, respectively determining the multiple clustering interval coefficients corresponding to the multiple thresholds including the clustering interval.

[0129] In an embodiment of the present disclosure, based on the multiple thresholds corresponding to the fusion update confidence determined by using an iterative algorithm, the multiple thresholds corresponding to the fusion update confidence determined by using an adaptive threshold method are verified.

[0130] In an embodiment of the present disclosure, the number corresponding to the multiple thresholds is configured to be 2, and the number corresponding to the multiple target detection frame intervals into which the fusion update confidence is divided is 3.

[0131] In the embodiments of the present disclosure and other possible embodiments, when the number of target detection frame intervals into which the fusion update confidence is divided is 3 or the number corresponding to the multiple thresholds is configured as 2, the minimum fusion update confidence and the maximum fusion update confidence are used as starting points, and two clustering interval coefficients corresponding to the two thresholds including the clustering interval are determined respectively; wherein the values ​​corresponding to the two clustering interval coefficients are the same, and the values ​​corresponding to the two clustering interval coefficients are configured as 2 / 3.

[0132] Figure 4 A threshold determination schematic diagram corresponding to an adaptive threshold example according to an embodiment of the present disclosure is shown. Figure 4 As shown, the number of target detection frame intervals corresponding to the fusion update confidence division is 3 or the number of thresholds corresponding to the configuration is 2 as an example, a simple and effective method is used to obtain two thresholds for each frame, that is, by simply averaging, it can be highly close to the optimal threshold interval, and at this time the threshold is at the steepest point of the steepest descent method, such as Figure 4 As shown. After the confidences are arranged in order, a clustering interval coefficient α is set to determine the curve interval corresponding to the sorted fusion update confidences used to divide the thresholds. The two intervals intersect at the middle confidence level, with the purpose of reconfirming the clearer confidences and reducing the error rate. After that, it is converted into two single threshold divisions. This method can also be flexibly applied to the threshold division of other trackers, eliminating the tedious steps of repeated experiments to obtain appropriate thresholds.

[0133] in, Figure 4 (a) is a subset of MOT20-06 in the test set of the MOT20 dataset. Figure 4 (b) is a subset of MOT20-03 in the test set of the MOT17 dataset. The alpha coefficient of the datasets with higher complexity (MOT20) and lower complexity (MOT17) is 2 / 3. The red circles show the first threshold and the second threshold corresponding to the two thresholds. The confidence level is higher than the high red circle, which is the high-scoring detection frame interval corresponding to the high confidence level. The area between the two circles is the medium confidence area (medium-scoring detection frame interval), and the rest is the low confidence area (low-scoring detection frame interval).

[0134] Step S103: Based on the multiple thresholds, the fused updated confidence is divided into multiple target detection frame intervals.

[0135] In an embodiment of the present disclosure, the method of dividing the fusion update confidence into multiple target detection frame intervals based on the multiple thresholds includes: configuring the interval corresponding to the minimum fusion update confidence in the fusion update confidence to the minimum threshold in the multiple thresholds as a first detection frame interval; respectively configuring the intervals corresponding to adjacent thresholds in the multiple thresholds as multiple second detection frame intervals; and configuring the interval corresponding to the maximum fusion update confidence in the multiple thresholds to the maximum fusion update confidence in the fusion update confidence as a third detection frame interval.

[0136] In the embodiments of the present disclosure and other possible embodiments, the number corresponding to the multiple thresholds is configured as 2, and the two thresholds are respectively configured as the first threshold and the second threshold; the first threshold and the second threshold are used to divide the fusion update confidence into three target detection frame intervals, and the three target detection frame intervals are respectively configured as a high-score detection frame interval (third detection frame interval), a medium-score detection frame interval (second detection frame interval) and a low-score detection frame interval (first detection frame interval).

[0137] Step S104: using the position information corresponding to the multiple target detection frame intervals, track possible targets in each frame image in the given video sequence.

[0138] In the embodiments of the present disclosure and other possible embodiments, target tracking generally uses Kalman filtering and Hungarian algorithm to achieve accurate and efficient results. Due to measurement errors, the boundary frames obtained by the target detector usually contain noise and inaccuracies. The Kalman filter solves this problem by using data fusion between the predicted values ​​and the measured values ​​using a mathematical model, while the Hungarian algorithm performs data fusion between the estimated values ​​and the observed values.

[0139] In the embodiments of the present disclosure and other possible embodiments, ByteTrack is based on the YOLOX object detector. First, the detector obtains information such as the location, category, and confidence of the object, which is sent to the input of the tracker BYTE. Similar to the SORT algorithm, BYTE uses Kalman filtering to predict the location of existing tracks in the current frame. Then, it calculates the intersection over union (IoU) between the detection frame and the predicted frame, and uses the Hungarian algorithm to match the detection frame to the optimal track and assign an ID to each tracked track.

[0140] In the embodiments of the present disclosure and other possible embodiments, the main feature of BYTE is to use low-score detection frames for two matches with existing trajectories, effectively optimizing pedestrian trajectory loss and identity switching problems caused by pedestrian occlusion and motion blur. In order to ensure real-time performance, BYTE only uses motion features and position information instead of appearance features for data association. Therefore, its tracking effect is highly dependent on the quality of the detection model.

[0141] Based on the above, in an embodiment of the present disclosure, the method of tracking possible targets in each frame image in the given video sequence using the position information corresponding to the multiple target detection frame intervals includes: determining the position information corresponding to the trajectory mark detection frame interval used to initialize the trajectory from the position information corresponding to the multiple target detection frame intervals; initializing the track list being tracked using the position information corresponding to the track mark detection frame interval; and tracking possible targets in each frame image in the given video sequence based on the initialized track list being tracked and the position information corresponding to the multiple target detection frame intervals.

[0142] In an embodiment of the present disclosure, the method for determining the position information corresponding to the trajectory mark detection frame interval for initializing the trajectory from the position information corresponding to the multiple target detection frame intervals includes: respectively configuring the intervals corresponding to adjacent thresholds among the multiple thresholds as multiple second detection frame intervals; configuring the interval corresponding to the maximum threshold among the multiple thresholds to the maximum fusion update confidence among the fusion update confidence as a third detection frame interval; determining the position information corresponding to the trajectory mark detection frame interval for initializing the trajectory from the position information corresponding to the third detection frame interval and / or any one or more detection frame intervals among the multiple second detection frame intervals.

[0143] In an embodiment of the present disclosure, the method for tracking possible targets in each frame image in the given video sequence based on the initialized list of tracks being tracked and the position information corresponding to the multiple target detection frame intervals includes: predicting the tracks in the initialized list of tracks being tracked to obtain corresponding predicted position information; tracking possible targets in each frame image in the given video sequence based on the predicted position information and the position information corresponding to the multiple target detection frame intervals.

[0144] In the embodiments of the present disclosure and other possible embodiments, the method of predicting the trajectories in the initialized list of trajectories being tracked to obtain corresponding predicted position information includes: obtaining a Kalman filter algorithm; and using the Kalman filter algorithm to predict the trajectories in the initialized list of trajectories being tracked to obtain corresponding predicted position information.

[0145] In an embodiment of the present disclosure, the method for tracking possible targets in each frame image in the given video sequence based on the predicted position information and the position information corresponding to the multiple target detection frame intervals includes: using the predicted position information and the position information corresponding to the third detection frame interval from the maximum threshold among multiple thresholds in the multiple target detection frame intervals to the maximum fusion update confidence in the fusion update confidence to construct a corresponding first cost matrix; matching the predicted position information in the first cost matrix with the position information to obtain a first matching trajectory corresponding to a first optimal matching combination, a first unmatched detection value, and the predicted position information or the first unmatched trajectory corresponding to the list of trajectories being tracked; wherein the first unmatched detection value is configured as the position information corresponding to the third detection frame interval; constructing multiple second cost matrices using the position information corresponding to multiple second detection frame intervals corresponding to intervals corresponding to adjacent thresholds in the multiple thresholds, the first unmatched detection value, and the first unmatched trajectory; The first unmatched trajectories in the multiple second cost matrices are matched with the position information to obtain the second matching trajectory corresponding to the corresponding second optimal matching combination, the second unmatched detection value, and the second unmatched trajectory corresponding to the first unmatched trajectory; wherein the second unmatched detection value is configured as the position information corresponding to the second detection frame interval; using the position information corresponding to the first detection frame interval with the minimum fusion update confidence in the fusion update confidence to the minimum threshold among the multiple thresholds and the second unmatched trajectory to construct a third cost matrix; matching the second unmatched trajectory in the third cost matrix with the position information to obtain the third matching trajectory corresponding to the third optimal matching combination, the third unmatched detection value, and the third unmatched trajectory corresponding to the second unmatched trajectory; wherein the first unmatched detection value is configured as the position information corresponding to the third detection frame interval; based on the first matching trajectory, the second matching trajectory and the third matching trajectory, possible targets in each frame image in the given video sequence are tracked;

[0146] In an embodiment of the present disclosure, the method further includes: updating the trajectory in the tracking trajectory list using the first matching trajectory, the second matching trajectory and the third matching trajectory.

[0147] In the embodiment of the present disclosure and other possible embodiments, the method further includes: updating parameters corresponding to the position information in the Kalman filter algorithm using the first matching trajectory, the second matching trajectory and the third matching trajectory.

[0148] In the embodiments of the present disclosure and other possible embodiments, the method of constructing multiple second cost matrices by respectively using the position information corresponding to multiple second detection frame intervals corresponding to intervals corresponding to adjacent thresholds among the multiple thresholds, the first unmatched detection value and the first unmatched trajectory, includes: configuring the multiple second detection frame intervals as second detection frame intervals whose fusion update confidences are successively reduced and cascaded; constructing a first sub-cost matrix among the multiple second cost matrices by using the position information corresponding to the second sub-detection frame interval corresponding to the highest fusion update confidence among the intervals corresponding to adjacent thresholds among the multiple thresholds, the first unmatched detection value and the first unmatched trajectory; constructing a second sub-cost matrix among the multiple second cost matrices by using the position information corresponding to the second sub-unmatched trajectory and the second sub-unmatched detection value corresponding to the first sub-cost matrix, and the position information corresponding to the detection frame interval cascaded with the second sub-detection frame interval corresponding to the highest fusion update confidence; and constructing the sub-cost matrix to be constructed this time by using the position information corresponding to the unmatched trajectory and the unmatched detection value corresponding to the sub-cost matrix constructed last time and the position information corresponding to the detection frame interval of the sub-cost matrix to be constructed this time.

[0149] In the embodiments of the present disclosure and other possible embodiments, the spatial stability characteristics of the identified target are also one of the key issues in multi-target tracking (MOT). In addition to the confidence state, the height state also provides informative weak clues to help make up for the lack of strong clues.

[0150] Figure 5 FIG. 2 shows a schematic diagram of the principle corresponding to the width modulation IOU according to an embodiment of the present disclosure. Width modulation (WMIOU) is the main work of the present invention in this part. Figure 5 As shown in the figure, WMIoU only adds the width of the rectangular box corresponding to the pedestrian detection value to enhance the calculation of IoU. The center position, scale and shape of the original detection and trajectory have not changed, but the length of the entire matching space has been extended.

[0151] like Figure 5 As shown, the IoU in camera space is used. In the method of the present invention, the present invention transforms the coordinates of the detected bounding boxes by projecting them onto the ground plane, and by calculating the IoU using these transformed rectangles, the present invention increases the refinement of the cost matrix in effect and can more effectively match objects and occlusions.

[0152] In an embodiment of the present disclosure, the method for matching the first unmatched trajectory in the multiple second cost matrices with the position information includes: respectively determining the first rectangular box coordinates A corresponding to the first unmatched trajectory and the second rectangular box coordinates B describing the position information in the multiple second cost matrices; respectively determining the corresponding intersection-over-union ratio IoU and width intersection-over-union ratio WIoU based on the first rectangular box coordinates A and the second rectangular box coordinates B; respectively determining the corresponding width modulation intersection-over-union ratio WMIoU based on the intersection-over-union ratio IoU and the width intersection-over-union ratio WIoU; and performing width modulation intersection-over-union ratio matching on the predicted position information and the position information in the second cost matrix.

[0153] In the embodiment of the present disclosure and other possible embodiments, the formula corresponding to the intersection over union (IoU) is configured as follows:

[0154]

[0155] In an embodiment of the present disclosure, the method for determining the corresponding width intersection-and-union ratio based on the first rectangular frame coordinates and the second rectangular frame coordinates, respectively, includes: respectively determining the first point coordinate b1 corresponding to the first rectangular frame coordinates and the second point coordinate b2 corresponding to the second rectangular frame coordinates; wherein the first point coordinate b1 is in a diagonal relationship with the second point coordinate b2; respectively determining the maximum horizontal coordinate and the minimum horizontal coordinate of the first horizontal coordinate in the first point coordinates and the second horizontal coordinate in the second point coordinates; and respectively determining the corresponding width intersection-and-union ratio based on the maximum horizontal coordinate and the minimum horizontal coordinate.

[0156] In the embodiments of the present disclosure and other possible embodiments, the first point coordinates and the second point coordinates in a diagonal relationship are configured as the upper left corner corresponding to the first rectangular frame coordinates and the lower right corner corresponding to the second rectangular frame coordinates; or, the first point coordinates and the second point coordinates in a diagonal relationship are configured as the lower left corner corresponding to the first rectangular frame coordinates and the upper right corner corresponding to the second rectangular frame coordinates; or, the first point coordinates and the second point coordinates in a diagonal relationship are configured as the upper right corner corresponding to the first rectangular frame coordinates and the lower left corner corresponding to the second rectangular frame coordinates; or, the first point coordinates and the second point coordinates in a diagonal relationship are configured as the lower right corner corresponding to the first rectangular frame coordinates and the upper left corner corresponding to the second rectangular frame coordinates.

[0157] For example, the present invention defines these two boxes as and in Represents the upper left corner, Represents the lower right corner.

[0158] In an embodiment of the present disclosure, the method for determining the corresponding intersection-over-unit ratio and width intersection-over-unit ratio based on the maximum horizontal coordinate and the minimum horizontal coordinate respectively includes: calculating the minimum horizontal coordinate respectively With the maximum horizontal coordinate The corresponding first horizontal coordinate difference; respectively calculate the maximum horizontal coordinate With the minimum horizontal coordinate The corresponding second horizontal coordinate difference; respectively calculating the ratio of the first horizontal coordinate difference to the second horizontal coordinate difference, and determining the corresponding width intersection-combination ratio.

[0159] In the embodiment of the present disclosure and other possible embodiments, the formula corresponding to the width intersection-over-combination ratio is configured as:

[0160]

[0161] In the embodiments of the present disclosure and other possible embodiments, the method of determining the corresponding width modulation cross-over-coupled ratio based on the cross-over-coupled ratio and the width cross-over-coupled ratio, respectively, includes: multiplying the cross-over-coupled ratio by the corresponding width cross-over-coupled ratio, respectively, to determine the corresponding width modulation cross-over-coupled ratio.

[0162] In the embodiment of the present disclosure and other possible embodiments, the formula corresponding to the width modulation intersection-and-parallel ratio is configured as follows:

[0163] WMIoU=IoU·WIoU

[0164] In the embodiments of the present disclosure and other possible embodiments, in order to better utilize the height state, the present invention combines the width IoU (WIoU) with the conventional IoU and introduces the width modulated IoU (WMIoU). At the same time, considering that WIoU represents the width state, which is a weak clue, and IoU represents the spatial information, which is a strong clue, the present invention uses WIoU to modulate IoU, and through the multiplication of elements, realizes the enhanced recognition of occluded or clustered objects. At the same time, the present invention finds that IoU is involved in the matching of targets with low confidence and targets with severe occlusion.

[0165] Figure 6 A schematic diagram of the positional relationship between the detection frame and the trajectory in different extreme cases according to an embodiment of the present disclosure is shown. Figure 6 As shown, Figure 6 In (a), the trajectory has no intersection with the target box and the IoU value is 0. Figure 6 When one party in (b) and (c) completely includes the other party, WIoU cannot reflect the position information represented by the included party. This is because no matter which party is included, there will always be a consistent WIoU value, which is obviously inconsistent with the position information of the two.

[0166] Based on the above, in an embodiment of the present disclosure, before performing width modulation intersection-and-joint ratio matching on the predicted position information and the position information in the second cost matrix, it includes: adjusting the width modulation intersection-and-joint ratio using a preset adjustment parameter to obtain an adjusted width modulation intersection-and-joint ratio; and performing adjusted width modulation intersection-and-joint ratio matching on the predicted position information and the position information in the second cost matrix.

[0167] In the embodiments of the present disclosure and other possible embodiments, the method for width modulated intersection-and-parallel matching of the adjusted predicted position information and position information in the second cost matrix includes: obtaining a Hungarian matching algorithm, and using the Hungarian matching algorithm to perform width modulated intersection-and-parallel matching of the adjusted predicted position information and position information in the second cost matrix.

[0168] In the embodiments of the present disclosure and other possible embodiments, the method of adjusting the width modulation intersection-and-parallel ratio using a preset adjustment parameter to obtain an adjusted width modulation intersection-and-parallel ratio includes: calculating the preset adjustment parameter power corresponding to the width modulation intersection-and-parallel ratio to obtain the adjusted width modulation intersection-and-parallel ratio.

[0169] In an embodiment of the present disclosure, the method for determining the preset adjustment parameter includes: respectively obtaining adjacent threshold values ​​[n, m] from the multiple threshold values ​​corresponding to each second detection frame interval in the multiple second detection frame intervals; calculating the difference mn between the adjacent threshold values; and determining the preset adjustment parameter.

[0170] In the embodiments of the present disclosure and other possible embodiments, the method for calculating the difference between the adjacent thresholds is configured as a value greater than or equal to 0; or, the method for calculating the difference between the adjacent thresholds includes: determining the maximum threshold m and the minimum threshold n corresponding to the adjacent thresholds [n, m]; calculating the difference mn between the maximum threshold m and the minimum threshold n.

[0171] In an embodiment of the present disclosure, the method for determining the preset adjustment parameter includes: respectively determining a set order of magnitude parameter d of the intersection-and-union ratio or width intersection-and-union ratio corresponding to multiple second cost matrices; and determining the preset adjustment parameter based on the difference mn between the adjacent thresholds and the set order of magnitude parameter d.

[0172] In an embodiment of the present disclosure, the method for determining the preset adjustment parameter based on the difference between the adjacent thresholds and the set order of magnitude parameter, respectively, comprises: calculating the ratio of the difference mn between the adjacent thresholds to the set order of magnitude parameter d respectively. The ratios are Configure or determine the corresponding preset adjustment parameters.

[0173] In the embodiments of the present disclosure and other possible embodiments, the present invention divides the intervals according to the confidence value of the detection to avoid the occurrence of such a phenomenon. In order to reflect the effectiveness, the original IoU is not used in the interval, and the interval is adaptively adjusted according to the interval size. The final calculation formula is as follows:

[0174]

[0175] Among them, S is the fusion update confidence corresponding to the detection value in the current frame, m and n are the high and low thresholds (multiple thresholds, such as the first threshold and the second threshold) divided by the above adaptation, and d is the set order of magnitude parameter for adjusting IoU and WIoU.

[0176] The above method can stabilize the stable target separated from the complex and changing environment, reduce the possibility of confusion caused by its entry into the next stage (low confidence matching), and reduce the complexity of error-prone situations such as severe occlusion and background from the generation path, thereby improving the overall tracking accuracy.

[0177] In the embodiments of the present disclosure and other possible embodiments, the method for matching the predicted position information in the first cost matrix with the position information includes: obtaining a Hungarian matching algorithm, and using the Hungarian matching algorithm to perform intersection-and-union matching on the predicted position information in the first cost matrix with the position information.

[0178] In the embodiments of the present disclosure and other possible embodiments, the method of matching the second unmatched trajectory in the third cost matrix with the position information includes: obtaining a Hungarian matching algorithm, and using the Hungarian matching algorithm to match the second unmatched trajectory in the third cost matrix with the position information to perform intersection-and-union matching.

[0179] In summary, this paper takes the bytetrack algorithm as a benchmark, strengthens the feature expression of the detection results, and makes adaptive and detailed divisions. It proposes a new tracking method and system based on bytetrack. By more accurately distinguishing different confidence scores and dividing intervals, it reduces the interference between adjacent targets during matching and significantly filters the background. It introduces a generalized confidence output, combines the original confidence score with the target position information, and forms a strong association between the detector and the tracker in the tracking-by-detection framework. An adaptive threshold division method is used in the tracking model to remove the limitations of the fixed threshold on the algorithm in changing scenarios.

[0180] Specifically, in order to alleviate the technical problems raised by the present invention, based on the high reliability of target detection, by strengthening the inheritance of the target detection module and the association module, an online multi-target tracking method based on feature fusion and iterative association is proposed, which mainly includes: strengthening the output of the target detection module to provide more reliable confidence information;) adopting a new iterative association step to significantly improve the performance of tracking targets with uniform appearance; designing an adaptive parameter setting method, which not only reduces the parameter debugging steps and makes the algorithm more concise, but also improves the robustness of the algorithm in dealing with scene complexity changes. Through this series of improvements to the algorithm, the method of the present invention can solve the problem of tracking interruption caused by occlusion in online tracking scenes. A large number of experimental results show that the method of the present invention has achieved promising performance levels in both unoccluded and occluded tracking scenes.

[0181] The execution subject of the online multi-target tracking method based on feature fusion and iterative association may be an image processing device, for example, the online multi-target tracking method based on feature fusion and iterative association may be executed by a terminal device or a server or other processing device, wherein the terminal device may be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementations, the online multi-target tracking method based on feature fusion and iterative association may be implemented by a processor calling computer-readable instructions stored in a memory.

[0182] Those skilled in the art will understand that, in the above-mentioned online multi-target tracking method based on feature fusion and iterative association in the specific implementation mode, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0183] The disclosed embodiment also proposes an online multi-target tracking device / system based on feature fusion and iterative association, including: a fusion update unit, used to fuse and update the confidence using the position information and confidence corresponding to the possible targets in each frame image in a given video sequence, so as to obtain a fused updated confidence; a threshold determination unit, used to determine the multiple thresholds corresponding to the fused updated confidence using an iterative algorithm; a target detection frame interval division unit, used to divide the fused updated confidence into multiple target detection frame intervals based on the multiple thresholds; and a target tracking unit, used to track the possible targets in each frame image in the given video sequence using the position information corresponding to the multiple target detection frame intervals.

[0184] The disclosed embodiment also proposes an online multi-target tracking device / system based on feature fusion and iterative association, comprising: a processor; a memory for storing processor executable instructions; wherein the processor is configured to call the instructions stored in the memory to execute the above-mentioned online multi-target tracking method based on feature fusion and iterative association.

[0185] The disclosed embodiment also proposes an online multi-target tracking system based on feature fusion and iterative association, comprising: a processor; a memory for storing processor executable instructions; wherein the processor is configured to call the instructions stored in the memory to execute the above-mentioned online multi-target tracking method.

[0186] The disclosed embodiment also proposes an online multi-target tracking system based on feature fusion and iterative association, including: a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions implement the above-mentioned online multi-target tracking method when executed by a processor.

[0187] The disclosed embodiment also proposes an online multi-target tracking system based on feature fusion and iterative association, including: a computer program product, including a computer program / instruction, which implements the above-mentioned online multi-target tracking method when executed by a processor.

[0188] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the online multi-target tracking method based on feature fusion and iterative association described in the above method embodiment. Its specific implementation can refer to the description of the above online multi-target tracking method based on feature fusion and iterative association embodiment. For the sake of brevity, it will not be repeated here.

[0189] The present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the online multi-target tracking method based on feature fusion and iterative association is implemented. The computer-readable storage medium may be a non-volatile computer-readable storage medium.

[0190] The embodiment of the present disclosure also proposes an electronic device, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to implement the online multi-target tracking method based on feature fusion and iterative association. The electronic device can be provided as a terminal, a server, or other forms of devices.

[0191] The disclosed embodiment also provides a computer program product, including a computer program / instruction, which implements the above-mentioned online multi-target tracking method when executed by a processor.

[0192] The experimental results of the proposed algorithm are described below, and they are compared with other state-of-the-art methods on the Crowdhuman dataset and the MOTChallenge (MOT17and MOT20) dataset. In addition, sufficient ablation experiments are designed to test and verify the performance of the proposed module. Like most pioneers, HOTA and MOTA are used to evaluate the proposed algorithm, and the experimental results show that it outperforms the baseline method.

[0193] The MOT17 dataset and the MOT20 dataset are both public datasets for target tracking. The MOT17 dataset contains the tracking of human targets, including 7 sequences, covering various scenarios and applications. The MOT20 dataset is an upgraded version of the MOT17 dataset, adding more sequences and more target categories. These datasets provide a benchmark for the study of target tracking algorithms and can be used to compare and evaluate different algorithms. At the same time, these datasets are also helpful for studying problems in the fields of human behavior analysis, video analysis, and object detection. The CrowdHuman dataset is a large benchmark dataset designed for target detection tasks in crowd scenes. The CrowdHuman dataset is divided into three parts: training set, validation set, and test set. Among them, the training set contains 15,000 images, the validation set contains 4,370 images, and the test set contains 5,000 images. These images are all from real scenes, containing a variety of different lighting conditions, perspectives, and backgrounds.

[0194] The experimental platform is configured with RTX 3060 GPU (10G). The operating system of the experimental platform is Ubuntu 18.04, the deep learning framework is PyTorch 1.9.0, the Cuda version is 11.1, and the Python version is 3.8.

[0195] The experiment was conducted using a transfer learning strategy. The experiment was first pre-trained on the CrowedHuman dataset and half of the MOT17 dataset, and then trained on the MOT17 dataset using the pre-trained weights. The hyperparameters were set to batch size 16, epoch 30, initial learning rate 0.001, and weight decay 0.0005.

[0196] The performance of the proposed algorithm is evaluated using the High-Order Tracking Accuracy (HOTA) and CLEAR MOT metrics as standard protocols, which include Multi-Object Tracking Accuracy (MOTA), Identity F1 (IDF1) score, Main Tracking (MT), Main Loss (ML), and IDS.

[0197] HOTA.HOTA is defined as the geometric mean of localization accuracy (LocA), detection accuracy (DetA), and association accuracy (AssA). As a high-level tracking metric, HOTA combines detection, localization, and association performance into a unified metric. All three components (localization, detection, and association) are important for tracking success, so HOTA is important for measuring overall tracking accuracy.

[0198] MOTA.MOTA is the most widely used and commonly used official indicator for evaluating MOT target detection and tracking performance. MOTA is defined as:

[0199]

[0200] Where t represents the frame, FN T , FP T 、IDSW T and GT T denote the number of false negatives, false positives, IDS and true bounding boxes of frame t, respectively.

[0201] IDF1.IDF1 can be used to compute the bijection between the true and predicted trajectories and determine whether the model can robustly maintain the identity of the object throughout the sequence. IDF1 can be expressed as:

[0202]

[0203] Where IDTP, IDFN and IDFP are the number of true positive IDs, false negative IDs and false positive IDs of frame t, respectively.

[0204] Table 1 Results on the MOT17 test set

[0205]

[0206] Table 2 Results on the MOT20 test set

[0207]

[0208] Table 3 Comparison of the tracking speed between the method of the present invention and several existing methods

[0209]

[0210] For fair comparison, the methods in the bottom block use detections generated by yolo-x and achieve similar MOTA to state-of-the-art methods on mot17.

[0211] In order to verify the effectiveness of the method proposed in the present invention, a comparative analysis was conducted between the present invention and other state-of-the-art trackers on the MOT17 and MOT20 datasets. According to the tracking process, these trackers are all advanced post-detection trackers at each stage, and the detectors are all consistent with bytetrack, using YOLOX-X.

[0212] As shown in Table 1 and Table 2, the method of the present invention outperforms other methods on the MOT17 dataset and the MOT20 dataset in most indicators. Among them, the highest MOTA (76.4%) is achieved on the MOT17 dataset, and at the same time, it also has the lowest FP (5.9%) and IDSW (0.4%). This is because the method of the present invention distinguishes different categories of targets in the matching process, resulting in different types of targets being unable to affect each other, greatly reducing the tracker's false detection and ID switching. It is worth noting that the present invention is also tested on the more complex dataset MOT20 dataset. The scene density in the MOT20 dataset is larger and the clustering is serious. In the test set of MOT20, the average number of pedestrians per image is 170. The method of the present invention has a more obvious weakening effect on IDSW, and is not behind the most advanced trackers in terms of accuracy (MOTA78.5%, IDF172.27%, HOTA63,63%). Tracking speed is usually quantified in frames per second (FPS) and is a key indicator for evaluating the performance of trackers in practical applications. As shown in Table 3, the tracking speed of the algorithm of the present invention is at the middle level of all algorithms. Obviously, this is due to the fact that the algorithm of the present invention uses a deeper cascade matching layer in the data association stage. However, considering the improvement brought by the algorithm and the requirements of multi-target tracking in practical applications, the algorithm of the present invention is still acceptable. In summary, by observing these results, it is proved that the algorithm of the present invention can show competitive strength compared with the most advanced methods on multiple data sets.

[0213] The present invention described below conducts an ablation experiment on the proposed method to study the effects of different training data, model frameworks, hyperparameters, and different modules on the proposed method.

[0214] 1) Influence of different levels. In order to verify the effectiveness of the three-layer cascade of the overall framework of the algorithm of the present invention, the present invention conducted the following comparative experiments. As shown in Table 4, as a general method, the N=2 version of the present invention is basically equivalent to OC-SORT and BYTE on the half-value benchmark, and the N=3 version of the present invention is slightly higher than the N=2 version. Compared with several existing models, it has the highest MOTA of 82.5%, the lowest missed detection (FP) and IDs, without adjusting the tracking parameters. This proves the overall general ability of the method of the present invention. The results of the comparison of the present invention come directly from the Open MMLabmmtracking software library.

[0215] Table 4 Comparison of different partition intervals with the most advanced methods on half-value mot17 using the same detector

[0216]

[0217] 2) The impact of adaptive threshold on the data set. Taking the MOT data set as an example, the baseline obtained the best threshold after repeated experiments. Although the overall data is very good, the results of each subset are quite different due to the difference in complexity. Figure 5 As shown in the figure, the lower the subset MOTA score is, the higher its complexity is. As shown in the figure, the method of the present invention has a better MOTA improvement effect on the subset with high complexity, and the difference is more obvious than the overall effect comparison. It is not weaker than the existing method on the subset with low complexity, which means that the adaptive threshold method of the present invention can perform better in dealing with various situations, especially complex environments.

[0218] 3) Ablation experiment of WMIOU on the MOT17 test set. Table 6 shows the impact of different IOU calculation methods on the tracker. In the table, N=1 means that WMIOU only participates in one association, and it is the high threshold part. Because in the matching of the low threshold part, the background and occluded targets with higher probability and higher ratio are contained, and the trajectory does not intersect or completely contain the detection box, resulting in WIOU greater than 1 or far from expectations. This is also the reason why the application strategy of WMIOU is to improve the accuracy of the high threshold association part to reduce the pressure of the low threshold part. The detection shows the best performance when WMIOU is used and only participates in one high threshold match, with a HOTA gain of 0.5%, a MOTA gain of 0.2%, and an IDF1 gain of 0.8%. It verifies that WMIOU mentioned earlier in this article strengthens the association of high matching values ​​and should not be used in threshold intervals with lower complexity. At the same time, the present invention also adds a set of two-stage matching in which WMIOU participates (N=2). The final result of each tracking index is slightly lower than that of the method with N=1. This is because the method of the present invention is not fully applicable to the low threshold environment. Through the overall framework, the strategy is adjusted and WMIOU matching is only used in the high threshold. While improving the accuracy, it can also reduce the matching difficulty of the low threshold. Overall, another solution to the low threshold is realized. Similarly, other more suitable methods can be used instead.

[0219] 3) Tracking effect. As shown in Table 5, the present invention explores the effects of the proposed adaptive threshold and WMIOU matching modules compared with the baseline model (the baseline model includes a deep information fusion module). It can be seen that each module can improve the tracking performance of the baseline model to varying degrees. The combination of WMIOU and adaptive threshold partitioning modules is optimal in most indicators. Specifically, MOTA increased by 1.2% and IDs decreased by 0.2%. The proposed WMIOU module helps the network to be sensitive to objects that are highly likely to be identified, thereby reducing the interference of uncertain matching items. The adaptive threshold can accurately distinguish between targets with different confidence levels. By combining these two modules, the network can further improve the tracking performance, which verifies the effectiveness and superiority of the method proposed in the present invention.

[0220] Table 5 Comparison with the baseline model

[0221]

[0222] Table 6 Comparison of WMIoU with GIOU, DIOU, etc. using Bytetrack as a unified framework

[0223]

[0224] In Table 6, all IOUs participate in one match of bytetrack (high threshold), and a WMIOU participating in both stages is added. N represents the number of times a track participates in the match. The present invention obtains the following results, where the WMIOU participating in only one high threshold stage shows the highest performance.

[0225] Figure 7 A schematic diagram of MOTA comparison on each subset of the validation set of two data sets according to an embodiment of the present disclosure is shown. Figure 7 As shown, the figure shows the MOTA comparison on each subset of the validation set of the two data sets MOT17 (right) and MOT20 (left). The experiment only uses the adaptive threshold module to compare with the baseline bytetrack. It can be found that the lower the MOTA value of each subset of the baseline and the more complex the data, the greater the improvement shown by the adaptive threshold of the present invention.

[0226] 4) Interaction between modules. Previous experience has confirmed that the baseline bytetrack used in the present invention is already a simple and effective model. The goal of the algorithm design of the present invention is to solve the same problem, that is, multi-target tracking under complex conditions, by improving several modules in different tracking processes but interrelated based on bytetrack. In fact, the present invention is also designed according to this idea. The deep information fusion module realizes the cross-application of confidence scores and strong and weak clues of depth information, providing a clearer boundary for the three-stage threshold set division. The adaptive threshold division is upgraded to a three-stage adaptive division based on the original two stages. The three stages divided out the most complex and error-prone parts, and WMIOU is used to match them in a targeted manner. The core of WMIOU is to use width information for precise matching, and the confidence score output by the detector is fully utilized with the deep information fusion module, and the width and height information are fully utilized, which strengthens the association between detection and tracking of post-detection tracking (TBD). As shown in Table 5, the tracking effect is effectively improved.

[0227] In a visualization experiment corresponding to the performance of the algorithm of the present invention in complex scenarios, the method of the present invention has excellent performance in ID retention in terms of several important indicators affecting multi-target tracking technology, namely missed detection (FN), false detection (FP) and ID switching (IDs).

[0228] In summary, the technical solution proposed in the present invention is used for accurate tracking of pedestrian targets in complex environments. Experiments were conducted on relevant MOT datasets, and the performance of the tracker of the present invention is significantly better than most existing methods of the same type. These results show that the method of the present invention can accurately track pedestrian targets in dense environments and reduce the probability of false tracking. The excellent performance of the tracker of the present invention is attributed to three improvements: first, the correlation strength during trajectory matching is enhanced; second, the tracking efficiency in different environments and environmental changes is stabilized by adaptive thresholds to enhance the overall tracking effect; third, the estimation bias in matching is reduced by wmIoU. In addition, the tracker of the present invention uses bytetrack as the baseline, and the post-processing part of the data after trajectory matching is consistent with the baseline. First, it is easy to prove the effectiveness of the method; second, the improved part is concise and modular, which can be combined with more advanced methods. In addition, the effect of detectors has improved by leaps and bounds in recent years, and more effective solutions can be explored in the future.

[0229] Figure 8 8 is a block diagram of an electronic device 800 according to an exemplary embodiment. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, or the like.

[0230] Reference Figure 8 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0231] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. The memory 804 is configured to store various types of data to support operations on the electronic device 800. Examples of these data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The power supply component 806 provides power to the various components of the electronic device 800. The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP).

[0232] The audio component 810 is configured to output and / or input audio signals. The I / O interface 812 provides an interface between the processing component 802 and the peripheral interface module, which can be a keyboard, a click wheel, a button, etc. The sensor component 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 804 including computer program instructions, which can be executed by a processor 820 of the electronic device 800 to complete the above method.

[0233] Fig. 9 1 is a block diagram of an electronic device 1900 according to an exemplary embodiment. For example, the electronic device 1900 may be provided as a server. Fig. 9 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0234] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output (I / O) interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

[0235] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0236] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0237] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0238] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0239] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0240] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0241] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0242] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0243] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0244] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. An online multi-target tracking method based on feature fusion and iterative association, characterized in that: include: The confidence is updated by fusing the position information and confidence corresponding to the possible target in each frame image in a given video sequence to obtain a fused updated confidence; Determine the multiple thresholds corresponding to the fusion update confidence by using an iterative algorithm or an adaptive threshold method; Based on the multiple thresholds, dividing the fused updated confidence into multiple target detection frame intervals; The position information corresponding to the multiple target detection frame intervals is used to track possible targets in each frame image in the given video sequence.

2. The online multi-target tracking method based on feature fusion and iterative association according to claim 1 is characterized in that: The method of fusing and updating the confidence by using the position information and confidence corresponding to the possible target in each frame image in a given video sequence to obtain the fused updated confidence includes: determining the corresponding target bounding box or target detection box based on the position information corresponding to the possible target in each frame image; calculating the target bounding box or target detection box corresponding to the possible target and the height corresponding to each frame image, and determining the corresponding height difference; fusing and optimizing the confidence based on the height difference to obtain the fused updated confidence; using the fused updated confidence to update the fused optimized confidence; and / or, The method of calculating the target bounding box or target detection box corresponding to the possible target and the height corresponding to each frame of the image to determine the corresponding height difference includes: determining the ordinate corresponding to the lowest point or the highest point of the target bounding box or target detection box corresponding to the possible target; calculating the first difference between the height corresponding to each frame of the image and the ordinate to determine the corresponding height difference; and / or, The method of fusing and optimizing the confidence based on the height difference to obtain a fused updated confidence includes: obtaining the information range level of the target bounding box or the target detection box corresponding to the possible target; multiplying the information range level by the height difference to obtain the height difference corresponding to the information range level; calculating a second difference between 1 and the height difference corresponding to the information range level; based on the second difference, fusing and optimizing the confidence to obtain a fused updated confidence; and / or, The method of fusing and optimizing the confidence based on the second difference to obtain a fused updated confidence includes: obtaining an adjustable weight parameter; adjusting the confidence using the adjustable weight parameter to obtain an adjusted confidence; adding the adjusted confidence to the second difference to obtain a fused updated confidence; and / or, The method of adjusting the confidence level by using the adjustable weight parameter to obtain the adjusted confidence level includes: multiplying the adjustable weight parameter by the confidence level to obtain the adjusted confidence level.

3. The online multi-target tracking method based on feature fusion and iterative association according to claim 1 or 2, characterized in that: The method of determining the multiple thresholds corresponding to the fusion update confidence by using an iterative algorithm includes: establishing an inter-class variance objective function corresponding to the multiple thresholds to be determined Iteratively calculate the change average fusion update confidence U corresponding to each change threshold in the multiple thresholds to be determined i or μ i The average confidence U corresponding to the fusion update confidence T or μ T The sum of squares corresponding to the multiple third differences between the two; in the process of the iterative calculation, the inter-class variance objective function The multiple thresholds corresponding to the maximum are determined as the corresponding multiple thresholds; wherein the sum of the multiple thresholds is configured to be 1; and / or, The establishment of the inter-class variance objective function corresponding to the multiple thresholds to be determined The process also includes: obtaining a preset threshold, establishing an inter-class variance objective function including the preset threshold and the multiple thresholds to be determined; wherein the preset threshold is configured as a threshold corresponding to the non-possible target closest to the background in each frame image in the given video sequence; wherein the sum of the preset threshold and the multiple thresholds is configured to be 1; and / or, The iterative calculation of the change average fusion update confidence U corresponding to each change threshold value in the multiple threshold values ​​to be determined i or μ i The average confidence U corresponding to the fusion update confidence T or μ T The method of calculating the sum of squares corresponding to the third differences between the multiples includes: Get the set number of iterations, the multiple initial change thresholds corresponding to the multiple thresholds to be determined, and the initial change average fusion update confidence U i or μ i and its corresponding first set change rate, and the second set change rate corresponding to the multiple initial thresholds; calculate the average confidence U corresponding to the fusion update confidence T or μ T Based on the multiple initial change thresholds corresponding to the multiple thresholds to be determined, the initial change average fusion update confidence U i or μ i and its corresponding first set change rate, the second set change rate corresponding to the multiple initial thresholds, and the average fusion update confidence U corresponding to each change threshold in the multiple thresholds to be determined within the set number of iterations i or μ i The average confidence U corresponding to the fusion update confidence T or μ T The sum of squares of the corresponding multiple third differences between them.

4. The online multi-target tracking method based on feature fusion and iterative association according to claim 1 or 2, characterized in that: The method of using the adaptive threshold method to determine the multiple thresholds corresponding to the fusion update confidence includes: sorting the fusion update confidence according to size to obtain the sorted fusion update confidence; based on the sorted fusion update confidence and the set aggregation degree, determining the aggregation interval corresponding to the set aggregation degree; using the aggregation interval and the number corresponding to the multiple thresholds, determining the multiple thresholds corresponding to the fusion update confidence; and / or, The method of determining the multiple thresholds corresponding to the fusion update confidence by using the clustering intervals and the numbers corresponding to the multiple thresholds includes: determining the multiple clustering interval coefficients corresponding to the multiple thresholds based on the minimum fusion update confidence and / or the maximum fusion update confidence corresponding to the sorted fusion update confidence, the clustering intervals and the numbers corresponding to the multiple thresholds; determining the multiple thresholds on the curve corresponding to the sorted fusion update confidence based on the multiple clustering interval coefficients; wherein the interval range corresponding to each clustering interval coefficient in the multiple clustering interval coefficients includes the clustering interval; and / or, The method for determining multiple clustering interval coefficients corresponding to the multiple thresholds based on the minimum fusion update confidence and / or the maximum fusion update confidence corresponding to the sorted fusion update confidence, the clustering interval and the number corresponding to the multiple thresholds comprises: taking the minimum fusion update confidence and / or the maximum fusion update confidence as the starting point, respectively determining the multiple clustering interval coefficients corresponding to the multiple thresholds including the clustering interval; and / or, Based on the multiple thresholds corresponding to the fusion update confidence determined by the iterative algorithm, the multiple thresholds corresponding to the fusion update confidence determined by the adaptive threshold method are verified; and / or, the number corresponding to the multiple thresholds is configured to be 2, and the number corresponding to the multiple target detection box intervals into which the fusion update confidence is divided is 3.

5. The online multi-target tracking method based on feature fusion and iterative association according to any one of claims 1 to 4, characterized in that: The method of dividing the fusion update confidence into multiple target detection frame intervals based on the multiple thresholds includes: configuring an interval from a minimum fusion update confidence in the fusion update confidence to a minimum threshold in the multiple thresholds as a first detection frame interval; respectively configuring intervals corresponding to adjacent threshold values ​​among the plurality of threshold values ​​as a plurality of second detection frame intervals; An interval from a maximum threshold among the multiple thresholds to a maximum fusion update confidence among the fusion update confidences is configured as a third detection frame interval.

6. The online multi-target tracking method based on feature fusion and iterative association according to any one of claims 1 to 5, characterized in that: The method for tracking possible targets in each frame image in the given video sequence by using the position information corresponding to the multiple target detection frame intervals includes: determining the position information corresponding to the track mark detection frame interval used to initialize the track from the position information corresponding to the multiple target detection frame intervals; initializing the track list being tracked by using the position information corresponding to the track mark detection frame interval; tracking possible targets in each frame image in the given video sequence based on the initialized track list being tracked and the position information corresponding to the multiple target detection frame intervals; and / or, The method for determining the position information corresponding to the trajectory mark detection frame interval used to initialize the trajectory from the position information corresponding to the multiple target detection frame intervals, includes: respectively configuring the intervals corresponding to adjacent thresholds among the multiple thresholds as multiple second detection frame intervals; configuring the interval corresponding to the maximum threshold among the multiple thresholds to the maximum fusion update confidence among the fusion update confidence as a third detection frame interval; and determining the position information corresponding to the trajectory mark detection frame interval used to initialize the trajectory from the position information corresponding to the third detection frame interval and / or any one or more detection frame intervals among the multiple second detection frame intervals.

7. The online multi-target tracking method based on feature fusion and iterative association according to claim 6, characterized in that: The method for tracking possible targets in each frame image in the given video sequence based on the initialized list of trajectories being tracked and the position information corresponding to the multiple target detection frame intervals comprises: predicting the trajectories in the initialized list of trajectories being tracked to obtain corresponding predicted position information; tracking possible targets in each frame image in the given video sequence based on the predicted position information and the position information corresponding to the multiple target detection frame intervals; and / or, The method for tracking possible targets in each frame image in the given video sequence based on the predicted position information and the position information corresponding to the multiple target detection frame intervals includes: Constructing a corresponding first cost matrix using the predicted position information and the position information corresponding to the third detection frame interval from the maximum threshold among the multiple thresholds in the multiple target detection frame intervals to the maximum fusion update confidence among the fusion update confidences; Matching the predicted position information in the first cost matrix with the position information to obtain a first matching trajectory corresponding to a first optimal matching combination, a first unmatched detection value, the predicted position information or a first unmatched trajectory corresponding to the list of trajectories being tracked; wherein the first unmatched detection value is configured as the position information corresponding to the third detection frame interval; Constructing a plurality of second cost matrices using the position information corresponding to a plurality of second detection frame intervals corresponding to intervals corresponding to adjacent threshold values ​​among the plurality of threshold values, the first unmatched detection value, and the first unmatched trajectory respectively; Matching the first unmatched trajectories in the plurality of second cost matrices with the position information to obtain a second matching trajectory corresponding to the corresponding second optimal matching combination, a second unmatched detection value, and a second unmatched trajectory corresponding to the first unmatched trajectory; wherein the second unmatched detection value is configured as the position information corresponding to the second detection frame interval; Constructing a third cost matrix using the position information corresponding to the first detection frame interval from the minimum fusion update confidence in the fusion update confidence to the minimum threshold among the multiple thresholds and the second unmatched trajectory; Matching the second unmatched trajectory in the third cost matrix with the position information to obtain a third matching trajectory corresponding to a third optimal matching combination, a third unmatched detection value, and a third unmatched trajectory corresponding to the second unmatched trajectory; wherein the first unmatched detection value is configured as the position information corresponding to the third detection frame interval; Based on the first matching trajectory, the second matching trajectory and the third matching trajectory, possible targets in each frame of the given video sequence are tracked; and / or, the trajectories in the tracking trajectory list are updated using the first matching trajectory, the second matching trajectory and the third matching trajectory.

8. The online multi-target tracking method based on feature fusion and iterative association according to claim 7, characterized in that: The method for matching the first unmatched trajectory in the plurality of second cost matrices with the position information comprises: respectively determining the first rectangular frame coordinates corresponding to the first unmatched trajectory and the second rectangular frame coordinates describing the position information in the plurality of second cost matrices; respectively determining the corresponding intersection-and-union ratio and width intersection-and-union ratio according to the first rectangular frame coordinates and the second rectangular frame coordinates; respectively determining the corresponding width modulation intersection-and-union ratio based on the intersection-and-union ratio and the width intersection-and-union ratio; performing width modulation intersection-and-union ratio matching on the predicted position information and the position information in the second cost matrix; and / or, The method for determining the corresponding width intersection-and-joint ratio according to the first rectangular frame coordinates and the second rectangular frame coordinates respectively comprises: determining the first point coordinates corresponding to the first rectangular frame coordinates and the second point coordinates corresponding to the second rectangular frame coordinates respectively; wherein the first point coordinates and the second point coordinates are in a diagonal relationship; determining the maximum horizontal coordinate and the minimum horizontal coordinate of the first horizontal coordinate in the first point coordinates and the second horizontal coordinate in the second point coordinates respectively; determining the corresponding width intersection-and-joint ratio based on the maximum horizontal coordinate and the minimum horizontal coordinate respectively; and / or, The method for determining the corresponding intersection-and-joint ratio and width intersection-and-joint ratio based on the maximum horizontal coordinate and the minimum horizontal coordinate, respectively, comprises: respectively calculating the first horizontal coordinate difference corresponding to the minimum horizontal coordinate and the maximum horizontal coordinate; respectively calculating the second horizontal coordinate difference corresponding to the maximum horizontal coordinate and the minimum horizontal coordinate; respectively calculating the ratio of the first horizontal coordinate difference to the second horizontal coordinate difference to determine the corresponding width intersection-and-joint ratio; and / or, Before performing width modulation intersection-and-parallel matching on the predicted position information and the position information in the second cost matrix, the method includes: adjusting the width modulation intersection-and-parallel ratio using a preset adjustment parameter to obtain an adjusted width modulation intersection-and-parallel ratio; performing width modulation intersection-and-parallel matching on the predicted position information and the position information in the second cost matrix; and / or, The method for determining the preset adjustment parameter comprises: respectively obtaining adjacent threshold values ​​among the multiple threshold values ​​corresponding to each second detection frame interval in the multiple second detection frame intervals; calculating the difference between the adjacent threshold values; determining the preset adjustment parameter; and / or, The method for determining the preset adjustment parameter comprises: respectively determining set order of magnitude parameters of intersection-over-combination ratios or width intersection-over-combination ratios corresponding to a plurality of second cost matrices; respectively determining the preset adjustment parameter based on the difference between the adjacent thresholds and the set order of magnitude parameters; and / or, The method for determining the preset adjustment parameter based on the difference between the adjacent threshold values ​​and the set order of magnitude parameter, respectively, includes: respectively calculating the ratio of the difference between the adjacent threshold values ​​to the set order of magnitude parameter; and respectively configuring or determining the ratio as the corresponding preset adjustment parameter.

9. The online multi-target tracking method based on feature fusion and iterative association according to any one of claims 1 to 8, characterized in that: Before fusing and updating the confidence by using the position information and confidence corresponding to the possible target in each frame of the given video sequence, the method further includes: acquiring or detecting the position information and confidence corresponding to the possible target in each frame of the given video sequence; and / or, the given video sequence is configured as a video sequence corresponding to a vehicle and / or a pedestrian; The method for detecting position information and confidence corresponding to possible targets in each frame image in a given video sequence includes: obtaining a target detector; using the target detector to detect position information and confidence corresponding to possible targets in each frame image in a given video sequence; and / or,.

10. An online multi-target tracking system based on feature fusion and iterative association, characterized in that: include: A fusion updating unit, used to fuse and update the confidence by using the position information and confidence corresponding to the possible target in each frame image in a given video sequence to obtain a fusion updated confidence; A threshold determination unit, used to determine a plurality of thresholds corresponding to the fusion update confidence determination by using an iterative algorithm; A target detection frame interval division unit, configured to divide the fused updated confidence into a plurality of target detection frame intervals based on the plurality of thresholds; A target tracking unit, configured to track possible targets in each frame of the given video sequence by using the position information corresponding to the multiple target detection frame intervals; or, The invention comprises: a processor; a memory for storing instructions executable by the processor; The processor is configured to call the instructions stored in the memory to execute the online multi-target tracking method based on feature fusion and iterative association as described in any one of claims 1 to 9; or The invention comprises: a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions, when executed by a processor, implement the online multi-target tracking method based on feature fusion and iterative association as claimed in any one of claims 1 to 9; or, The invention comprises: a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the online multi-target tracking method based on feature fusion and iterative association as described in any one of claims 1 to 9.

Images