An online multi-target tracking method combined with lightweight deep appearance extraction

Abstract

The present application relates to computer vision technology, in particular to a kind of online multi-target tracking method combined with lightweight deep appearance extraction;Input current time video frame, obtain the detection bounding box of all tracking targets through detector;All detection bounding boxes are divided into high-confidence detection bounding box and low-confidence detection bounding box, and the state of all detection bounding boxes at current time is predicted;Based on high-confidence detection bounding box segmentation, obtain the image block of all high-confidence targets in video frame, and input the image block into lightweight re-identification network FastOSNet, obtain the appearance embedding of all high-confidence targets;The appearance state of all matched tracks is updated using hierarchical adaptive exponential moving average HAEMA;The technical scheme provided by the present application can effectively overcome the defects of low occlusion noise resistance, slow inference speed and large memory occupation existing in the prior art.

Description

TECHNICAL FIELD

[0001] The present application relates to computer vision technology, and in particular to an online multi-object tracking method combined with lightweight deep appearance extraction. BACKGROUND

[0002] Online multi-object tracking (MOT) is widely used in video surveillance, pose detection and unmanned driving fields. Thanks to the maturity of deep learning detectors, the tracking accuracy and precision of motion-based methods have made great progress, but they are difficult to deal with non-linear motion, camera motion and long-term occlusion of targets.

[0003] Introducing appearance clues can effectively improve the performance of this method in the face of the above difficulties: the document [Du Y, Wan J, Zhao Y, et al. Giaotracker: A comprehensive framework for mcmot with global information and optimizing strategies in visdrone 2021 [C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2021: 2809-2819.] inherits the OSNet network in the field of pedestrian re-identification based on the motion method to realize the appearance embedding of the target; the existing EMA method saves all trajectory appearance states in the recent time window of the track to alleviate the pollution of occlusion noise; the document [Du Y, Zhao Z, Song Y, et al. Strongsort: Make deepsort great again [J]. IEEE Transactions on Multimedia, 2023.] inherits the BOT model in the field of pedestrian re-identification and combines the traditional EMA method as its deep appearance extraction scheme.

[0004] In order to pursue better tracking performance, the existing multi-object tracking method usually adopts an independent deep appearance extraction scheme and directly inherits the appearance embedding network proposed in the field of re-identification, such as OSNet network, BOT model, etc. However, these networks do not consider the difference between the re-identification task and the multi-object tracking task when designing. Specifically, the former has fewer training samples, smaller intra-class similarity and greater inter-class ambiguity, and requires the network to distinguish finer-grained features; while the latter requires the network to cope with noise interference caused by similar backgrounds, and faster inference speed and smaller memory occupation.

[0005] Furthermore, existing trajectory appearance updating methods, such as the feature library and its variant EMA method, need to save a large number of appearance embeddings of trajectories over time, resulting in large memory occupation and high computational complexity, which is not friendly to edge devices with limited computing power and memory. On the other hand, the fixed weight setting of the EMA method makes it difficult to cope with changing environments, leading to the easy pollution of trajectory appearance states by occlusion noise. Therefore, in view of the needs of time-critical application scenarios and edge devices, a deep appearance extraction scheme for multi-target tracking tasks is urgently needed, which needs to have small memory occupation, fast inference speed, and strong resistance to background noise. SUMMARY

[0006] (One) technical problems solved

[0007] In view of the above shortcomings of the prior art, the present application provides an online multi-target tracking method combining lightweight deep appearance extraction, which can effectively overcome the defects of low resistance to occlusion noise, slow inference speed and large memory occupation of the prior art.

[0008] (Two) technical solutions

[0009] To achieve the above purpose, the present application is implemented by the following technical solutions:

[0010] An online multi-target tracking method combining lightweight deep appearance extraction, comprising the following steps:

[0011] S1, input the video frame at the current time, and obtain the detection bounding box of all tracking targets through the detector;

[0012] S2, divide all detection bounding boxes into high-confidence detection bounding boxes and low-confidence detection bounding boxes, and predict the state of all detection bounding boxes at the current time;

[0013] S3, based on the high-confidence detection bounding box, divide to obtain the image block of all high-confidence targets in the video frame, and input the image block into the lightweight re-identification network Fast OSNet to obtain the appearance embedding of all high-confidence targets;

[0014] S4, input the bounding box and appearance state of all trajectories at the previous time, construct a fusion cost matrix, and based on the fusion cost matrix, obtain the high-confidence detection bounding box matched with the trajectory at the previous time, the first unmatched trajectory and the unmatched high-confidence detection bounding box;

[0015] S5, construct a matching cost matrix according to the first unmatched trajectory, and based on the matching cost matrix, obtain the low-confidence detection bounding box matched with the trajectory at the previous time, the second unmatched trajectory and the unmatched low-confidence detection bounding box;

[0016] S6, updating the appearance state of all matched trajectories by using hierarchical adaptive exponentially weighted moving average (HAEMA);

[0017] S7, performing trajectory management on matched high-confidence detection bounding boxes, low-confidence detection bounding boxes, unmatched high-confidence detection bounding boxes, second-time unmatched trajectories, and unmatched low-confidence detection bounding boxes;

[0018] S8, obtaining the trajectory state of all tracking targets in the current video frame by integrating the bounding boxes and appearance states of the updated trajectories.

[0019] Preferably, in S4, the bounding boxes and appearance states of all trajectories at the previous time are input, a fusion cost matrix is constructed, and high-confidence detection bounding boxes matched with the trajectories at the previous time, first-time unmatched trajectories, and unmatched high-confidence detection bounding boxes are obtained based on the fusion cost matrix, including:

[0020] The intersection-over-union distance between the trajectories of the high-confidence detection bounding boxes and the trajectories at the previous time, and the cosine distance of the appearance embedding are calculated to construct the fusion cost matrix.

[0021] Based on the fusion cost matrix, the first association is performed by using the Hungarian algorithm to obtain the high-confidence detection bounding boxes matched with the trajectories at the previous time, the first-time unmatched trajectories, and the unmatched high-confidence detection bounding boxes.

[0022] Preferably, in S5, a matching cost matrix is constructed according to the first-time unmatched trajectories, and low-confidence detection bounding boxes matched with the trajectories at the previous time, second-time unmatched trajectories, and unmatched low-confidence detection bounding boxes are obtained based on the matching cost matrix, including:

[0023] The intersection-over-union similarity between the trajectories of the first-time unmatched trajectories and the low-confidence detection bounding boxes is calculated to construct the matching cost matrix.

[0024] Based on the matching cost matrix, the second association is performed by using the Hungarian algorithm to obtain the low-confidence detection bounding boxes matched with the trajectories at the previous time, the second-time unmatched trajectories, and the unmatched low-confidence detection bounding boxes.

[0025] Preferably, in S6, the appearance state of all matched trajectories is updated by using hierarchical adaptive exponentially weighted moving average (HAEMA), including:

[0026] S61, obtaining matched pairs at time t by using a data association algorithm, and obtaining the target confidence score at time t and the trajectory appearance score at the previous time obtaining the appearance update weight of trajectory i at time t by subtracting the two linear transformations as shown in the following formula:

[0027]

[0028] where η ∈ [0, 0.5) is the adaptive gradient, by adjusting η, so that the appearance update weight has different properties in two cases: when , the appearance update weight should be appropriately increased to absorb the appearance embedding of the target with low occlusion noise; otherwise, the appearance update weight should be reduced to suppress the features with high occlusion noise, thereby improving the correctness of trajectory matching.

[0029] S62, update the appearance state of trajectory i according to the appearance update weight and the appearance embedding of the jth target at time t , as shown in the following formula:

[0030]

[0031] where is the appearance state of trajectory i at the previous time, t s is the initial frame of the trajectory;

[0032] S63, perform EMA update on the trajectory appearance score, as shown in the following formula:

[0033]

[0034] where q is the score fusion weight.

[0035] Preferably, the trajectory management of the matched high-confidence detection bounding box, low-confidence detection bounding box, unmatched high-confidence detection bounding box, second-time unmatched trajectory, and unmatched low-confidence detection bounding box in S7 includes:

[0036] For the matched high-confidence detection bounding box and low-confidence detection bounding box, update the trajectory state through Kalman filtering;

[0037] For the second-time unmatched trajectory, if the age is less than the maximum age threshold, do not update the state, otherwise delete it;

[0038] For the unmatched high-confidence detection bounding box and unmatched low-confidence detection bounding box, if the confidence of the unmatched high-confidence detection bounding box is greater than the initialization threshold, initialize it as a new trajectory, otherwise delete it together with all unmatched low-confidence detection bounding boxes.

[0039] Preferably, in the lightweight re-identification network Fast OSNet, after the Conv5 stage, 4 part-level feature vectors h1, h2, … h4 with spatial information are obtained through part-level average pooling PAP, and a global feature vector h is obtained through global-level average pooling GAP.g ;

[0040] The lightweight re-identification network Fast OSNet only outputs the appearance embedding f during inference:

[0041] f = [h1, h2, …, h4, h g ] 1×(d×5)

[0042] wherein, d is the dimension of the final feature map, represents the real field.

[0043] Preferably, the lightweight re-identification network Fast OSNet comprises a simplified overall scale residual block SOSRB, which contains channel-by-channel convolution branches with convolution kernel sizes of 3×3, 5×5 and 7×7 respectively, each branch only contains one convolution layer, and a shared channel attention aggregation gate AG is arranged after the convolution layer of each convolution branch, and the three branch channels are aggregated and fused with full-scale features through a 1×1 convolution layer.

[0044] Preferably, the loss function of the lightweight re-identification network Fast OSNet is a joint loss function combining a multi-output branch ID loss and a circle loss, and the joint loss function L total is expressed as follows:

[0045] L total = L multi-ID +w c L circle

[0046] wherein, L multi-ID is the multi-output branch ID loss, L circle is the circle loss, and w c is the circle loss adjustment weight.

[0047] Preferably, the multi-output branch ID loss L multi-ID is expressed as follows:

[0048]

[0049] wherein, is the part-level feature ID loss, which is expressed as follows:

[0050]

[0051] In the above formula, N c is the ID number of the data set, p is the part-level output branch number, W p is the classification weight matrix of the pth part-level feature vector, respectively, are the Wp The yth i row, j-th row, y i For positive sample numbers, These are the p-th component-level feature vectors of the positive and negative instances, respectively.

[0052] The global feature ID loss is expressed by the following formula:

[0053]

[0054] In the above formula, N c W represents the number of dataset IDs. g For the global feature vector h g The classification weight matrix, W respectively g The yth i row, j-th row, y i For positive sample numbers, These are the global feature vectors for positive and negative instances, respectively.

[0055] Preferably, the circular loss L circle It can be expressed as follows:

[0056]

[0057] Where r is the scaling factor, m is the margin boundary, used to promote similarity separation, [·] + For hinge functions;

[0058] To perform metric learning, an L×K mini-batch of training samples needs to be constructed, containing L classes, with K samples for each class. For the appearance embedding f a Embedded with the appearance of positive and negative samples Cosine distance between It can be expressed as follows:

[0059]

[0060] (III) Beneficial Effects

[0061] Compared with existing technologies, the online multi-target tracking method combining lightweight depth appearance extraction provided by this invention has the following advantages:

[0062] 1) Compared with the OSNet network in the traditional pedestrian re-identification field, the lightweight re-identification network Fast OSNet can provide more discriminative appearance embedding for the multi-target tracking process, and thanks to the structural improvement of the simplified full-scale residual block SOSRB, the forward inference speed of the lightweight re-identification network Fast OSNet is improved by 69.0% compared with the OSNet network, which is more friendly to time-critical online tasks;

[0063] 2) The method combining part-level and global-level average pooling is used for the final feature map output by the lightweight re-identification network Fast OSNet, instead of single feature pooling, which can introduce prior constraints to the network, improve its robustness to complex similar backgrounds, and is more suitable for application in multi-target tracking tasks, and by introducing a circle loss weight adjustment, the multi-output branch ID loss and the circle loss are combined to construct a joint loss function, which takes into account the representation and metric learning of the network;

[0064] 3) The hierarchical adaptive exponential moving average HAEMA can improve the accuracy and robustness of multi-target tracking in complex scenes at a lower computational cost, and the hierarchical adaptive exponential moving average HAEMA is equivalent to performing EMA twice in layers, the first adaptive EMA updates the appearance state of the track, and the second updates the appearance score of the track, so it can be compatible with all deep appearance extraction schemes using the traditional EMA method;

[0065] 4) The deep appearance extraction method of the present application can be used as a plug-in to access the multi-target tracking method based on the detection and tracking paradigm, which can improve the overall tracking performance in complex scenes while ensuring real-time requirements. BRIEF DESCRIPTION OF DRAWINGS

[0066] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0067] Figure 1 The flowchart of the present application;

[0068] Figure 2 The structural diagram of the lightweight re-identification network Fast OSNet in the present application;

[0069] Figure 3 The structural diagram of the simplified full-scale residual block SOSRB in the lightweight re-identification network Fast OSNet;

[0070] Figure 4 A structural schematic diagram of a full-scale residual block OSRB. DETAILED DESCRIPTION

[0071] For the purposes of the embodiments of the present application, the technical solutions and advantages, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0072] An online multi-target tracking method combined with lightweight deep appearance extraction, as shown in Figure 1 ①input the video frame at the current time, and obtain the detection bounding boxes of all tracking targets through the detector.

[0073] ②divide all the detection bounding boxes into high-confidence detection bounding boxes and low-confidence detection bounding boxes (divided by setting a hard score threshold), and predict the state of all the detection bounding boxes at the current time (predicted by Kalman filtering).

[0074] ③divide the high-confidence targets in the video frame based on the high-confidence detection bounding boxes to obtain image blocks of all the high-confidence targets, and input the image blocks into the lightweight re-identification network Fast OSNet to obtain the appearance embeddings of all the high-confidence targets.

[0075] In the technical solutions of the present application, the structure of the lightweight re-identification network Fast OSNet is as shown in Figure 2 , which can extract discriminative appearance embeddings of tracking targets with small memory occupation. The image blocks obtained by cropping according to the high-confidence detection bounding boxes are aligned to be 128x64x3 in size, and are input into the lightweight re-identification network Fast OSNet. The size changes of the feature maps between the layers of the network and the specific structure of the network are as shown in the following table:

[0076] Table 1: Lightweight re-identification network Fast OSNet structure and comparison table of output feature map size of each layer

[0077]

[0078]

[0079] In the above table, Spatial attention is a spatial attention module responsible for weighting the pixels of the feature map one by one; Channel attention is a channel attention module responsible for weighting the channels of the feature map one by one; SOSRB is a simplified full-scale residual block, and its structure is as shown in Figure 3As shown, each channel-wise convolution branch contains a convolution layer with a kernel size of 3x3, 5x5, and 7x7 respectively, and a shared channel attention aggregation gate AG is arranged after the convolution layer of each convolution branch. The three branch channels are aggregated by a 1x1 convolution layer to fuse the full-scale features.

[0080] As shown in Figure 4 , the full-scale residual block OSRB has 3x3, 5x5, 7x7, and 9x9 convolution kernel branches and a residual branch. Multiple branches improve network performance while also bringing additional overhead, reducing computational efficiency. Considering that the input resolution of the deep appearance extractor in the multi-target tracking task is usually only 128x64, which is smaller than the common 256x128 in Re-ID, an excessively large receptive field does not have a significant effect. Therefore, the 9x9 branch is deleted in the present application. In addition, the receptive field of the full-scale residual block OSRB is realized by stacking multiple depthwise convolutions (DWConv), which significantly increases the memory access cost, thereby greatly reducing the inference speed. Recent research has found that directly using large kernel convolution can significantly increase the actual receptive field of the network, thereby improving the network performance. Therefore, the 5x5 and 7x7 branches are replaced by a single large kernel DWConv to obtain the simplified full-scale residual block SOSRB in the present application.

[0081] After the feature map passes through the Conv5 stage, 4 component-level feature vectors h1, h2, …, h4 with spatial information are obtained through part-level average pooling PAP, and a global feature vector h g As shown in Figure 2 , the left dashed box in the middle;

[0082] To perform metric learning, all feature vectors need to be spliced and projected to the unit hypersphere to form the appearance embedding f. The lightweight re-identification network Fast OSNet only outputs the appearance embedding f during inference:

[0083] f = [h1, h2, …, h4, h g ] 1×(d×5)

[0084] wherein, d is the dimension of the final feature map (d = 256), represents the real number field.

[0085] During training, each output branch has an independent ID loss, as shown in the right dashed box in Figure 2 . The loss function of the lightweight re-identification network Fast OSNet is a joint loss function combining the ID loss and the circle loss of multiple output branches. The joint loss function L totalIt can be expressed as follows:

[0086] L total =L multi-ID +w c L circle

[0087] Among them, L multi-ID For representation learning, the multi-output branch ID loss, L circle For circular loss, w c Adjusting the weights for circular loss (w) c =0.001).

[0088] 1) Multi-output branch ID loss L multi-ID It can be expressed as follows:

[0089]

[0090] in, The component-level feature ID loss is expressed by the following formula:

[0091]

[0092] In the above formula, N c Number of dataset IDs (N) c =751), p is the component-level output branch number, W p Let p be the classification weight matrix for the p-th component-level feature vector. W respectively p The yth i row, j-th row, y i For positive sample numbers, These are the p-th component-level feature vectors of the positive and negative instances, respectively.

[0093] The global feature ID loss is expressed by the following formula:

[0094]

[0095] In the above formula, N c Number of dataset IDs (N) c =751), W g For the global feature vector h g The classification weight matrix, W respectively g The yth i row, j-th row, y i For positive sample numbers, These are the global feature vectors for positive and negative instances, respectively.

[0096] 2) Circular loss L circle It can be expressed as follows:

[0097]

[0098] where r is a scale factor (r = 128), m is a margin (m = 0.3) to promote similarity separation, [·] + is a hinge function;

[0099] To perform metric learning, a small batch of training samples of L x K (L = 4, K = 16) is constructed, containing L classes, each with K samples. For appearance embedding f a , the cosine distance between the positive and negative sample appearance embeddings is expressed as follows:

[0100]

[0101] In the technical solution of the application, the lightweight re-identification network Fast OSNet is trained for a total of 120 iterations, the optimizer is Adam, the initial learning rate is 3.5e -4 , the learning rate is decayed to 7e -5 at the 75th round, and the learning rate is decayed to 1.4e -5 at the 100th round. In order to avoid overfitting of the network, some data enhancement techniques are used, such as random horizontal flipping, cutting, color jittering and erasing.

[0102] (4) input the bounding boxes and appearance states of all trajectories at the last time, construct a fusion cost matrix, and obtain high-confidence detection bounding boxes matched with the trajectories at the last time, first unmatched trajectories and unmatched high-confidence detection bounding boxes based on the fusion cost matrix, specifically including:

[0103] Calculate the intersection-over-union distance between the trajectories of the high-confidence detection bounding boxes and the trajectories at the last time, and the cosine distance of the appearance embedding, and construct a fusion cost matrix;

[0104] Based on the fusion cost matrix, the first association is performed by the Hungarian algorithm, and the high-confidence detection bounding boxes matched with the trajectories at the last time, the first unmatched trajectories and the unmatched high-confidence detection bounding boxes are obtained.

[0105] (5) construct a matching cost matrix according to the first unmatched trajectories, and obtain low-confidence detection bounding boxes matched with the trajectories at the last time, second unmatched trajectories and unmatched low-confidence detection bounding boxes based on the matching cost matrix, specifically including:

[0106] Calculate the intersection-over-union similarity between the first unmatched trajectories and the trajectories of the low-confidence detection bounding boxes, and construct a matching cost matrix;

[0107] ​Based on the matching cost matrix, a second association is performed by the Hungarian algorithm to obtain low-confidence detection bounding boxes matched with the track at the previous time, second-time unmatched tracks, and unmatched low-confidence detection bounding boxes.

[0108] 6. The appearance state of all matched tracks is updated using hierarchical adaptive exponential moving average (HAEMA), which specifically includes:

[0109] S61. The matching pairs (high-confidence detection bounding boxes matched with the track at the previous time, low-confidence detection bounding boxes) at time t are obtained using a data association algorithm, and the target confidence of the track i at time t is calculated as follows: The appearance score of the track at the previous time The difference is calculated, and then a two-segment linear transformation is performed to obtain the appearance update weight of the track i at time t as shown in the following formula:

[0110]

[0111] wherein η ∈ [0, 0.5) is an adaptive gradient (η = 0.05), and by adjusting η, the appearance update weight has different properties in two cases: when , the appearance update weight is appropriately increased to absorb the target appearance embedding of low occlusion noise; otherwise, the appearance update weight should be reduced to suppress high-occlusion noise features, thereby improving the correctness of track matching.

[0112] S62. The appearance state of the track i is updated according to the appearance update weight and the appearance embedding of the jth target at time t as shown in the following formula:

[0113]

[0114] wherein is the appearance state of the track i at the previous time, t s is the initial frame of the track;

[0115] S63. The EMA update is performed on the track appearance score as shown in the following formula:

[0116]

[0117] wherein q is a score fusion weight (q = 0.9).

[0118] In the technical solution of the present application, the fixed update weight of the exponential moving average EMA cannot adapt to the dynamic tracking process, thereby easily causing pollution of occlusion noise to the track appearance state. Therefore, on the basis of the exponential moving average EMA, the track appearance score and the multi-segment linear transformation are introduced, and a hierarchical adaptive exponential moving average (HAEMA) method is proposed to adaptively update the track appearance state.

[0119] ⑦Track management is performed on the matched high-confidence detection bounding box, low-confidence detection bounding box, and unmatched high-confidence detection bounding box, second-time unmatched track, and unmatched low-confidence detection bounding box, specifically including:

[0120] For the matched high-confidence detection bounding box and low-confidence detection bounding box, the track state is updated through Kalman filtering;

[0121] For the second-time unmatched track, if it is less than the maximum life threshold, the state is not updated, otherwise it is deleted;

[0122] For the unmatched high-confidence detection bounding box and unmatched low-confidence detection bounding box, if the confidence of the unmatched high-confidence detection bounding box is greater than the initialization threshold, it is initialized as a new track, otherwise it is deleted together with all unmatched low-confidence detection bounding boxes.

[0123] ⑧The bounding box and appearance state of the updated track are comprehensively obtained to obtain the track state of all tracking targets in the video frame at the current time.

[0124] The above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An online multi-target tracking method combined with lightweight deep appearance extraction, characterized in that: The method comprises the following steps: S1, input the video frame at the current time, and obtain the detection bounding boxes of all tracking targets through a detector; S2, divide all the detection bounding boxes into high-confidence detection bounding boxes and low-confidence detection bounding boxes, and predict the states of all the detection bounding boxes at the current time; S3, divide the high-confidence detection bounding boxes to obtain image blocks of all high-confidence targets in the video frame, and input the image blocks into a lightweight re-identification network Fast OSNet to obtain appearance embeddings of all the high-confidence targets; S4, input the bounding boxes and appearance states of all the trajectories at the previous time, construct a fusion cost matrix, and obtain the high-confidence detection bounding boxes matched with the trajectories at the previous time, the first unmatched trajectories and the unmatched high-confidence detection bounding boxes based on the fusion cost matrix; S5, construct a matching cost matrix according to the first unmatched trajectories, and obtain the low-confidence detection bounding boxes matched with the trajectories at the previous time, the second unmatched trajectories and the unmatched low-confidence detection bounding boxes based on the matching cost matrix; S6, update the appearance states of all the matched trajectories by using a hierarchical adaptive exponential moving average (HAEMA), comprising: S61, get the matching pair of t moment by data association algorithm, target confidence of t moment The track appearance score of the previous moment Difference, then get the appearance update weight of track i at t moment through two linear transformations As shown in the following formula: where η ∈ [0, 0.5) is the adaptive gradient, which is adjusted to make the appearance update weight has different properties in two cases: when , it is appropriate to increase to absorb the target appearance embedding of low occlusion noise; otherwise, it should be reduced to suppress high occlusion noise features, thereby improving the correctness of trajectory matching; S62, update the appearance weight according to the appearance and the appearance embedding of the jth target at time t update the appearance state of trajectory i as follows: wherein, is the appearance state of the trajectory i at the previous time instant, t s is the initial frame of the trajectory; S63, perform EMA update on the trajectory appearance score, as shown in the following formula: Wherein, q is a score fusion weight; S7, perform trajectory management on the matched high-confidence detection bounding boxes, low-confidence detection bounding boxes, and unmatched high-confidence detection bounding boxes, second unmatched trajectories and unmatched low-confidence detection bounding boxes; S8, comprehensively obtain the trajectory states of all the tracking targets in the video frame at the current time by combining the updated bounding boxes and appearance states of the trajectories; In the lightweight re-identification network Fast OSNet, after the Conv5 stage, 4 part-level feature vectors h1, h2, … h4 with spatial information are obtained through part-level average pooling PAP, and a global feature vector h is obtained through global-level average pooling GAP g ; The lightweight re-identification network Fast OSNet only outputs appearance embeddings f during inference: f = [h1, h2,..., h4, h g ] 1×(d×5) wherein, d is the dimension of the final feature map, denotes the real number field; The lightweight re-identification network Fast OSNet comprises a simplified full-scale residual block (SOSRB) containing three channel-wise convolution branches with convolution kernel sizes of 3x3, 5x5 and 7x7 respectively, each branch containing only one convolution layer, and a shared channel attention aggregation gate (AG) being arranged after the convolution layer of each convolution branch, aggregating the channels of the three branches through a 1x1 convolution layer and fusing full-scale features.

2. The method of claim 1, wherein the online multi-object tracking with light-weight deep appearance extraction is characterized by: In S4, the bounding boxes and appearance states of all the trajectories at the previous time are inputted to construct a fusion cost matrix, and the high-confidence detection bounding boxes matched with the trajectories at the previous time, the first unmatched trajectories and the unmatched high-confidence detection bounding boxes are obtained based on the fusion cost matrix, comprising: The intersection over union distance between the trajectories of the high-confidence detection bounding boxes and the trajectories at the previous time, and the cosine distance of the appearance embeddings are calculated to construct the fusion cost matrix; Based on the fusion cost matrix, the first association is performed through the Hungarian algorithm to obtain the high-confidence detection bounding boxes matched with the trajectories at the previous time, the first unmatched trajectories and the unmatched high-confidence detection bounding boxes.

3. The method of claim 2, wherein: In S5, the matching cost matrix is constructed according to the first unmatched trajectories, and the low-confidence detection bounding boxes matched with the trajectories at the previous time, the second unmatched trajectories and the unmatched low-confidence detection bounding boxes are obtained based on the matching cost matrix, comprising: Calculate the intersection-over-union similarity between the first unmatched track and the track of the low-confidence bounding box, and construct a matching cost matrix; Based on the matching cost matrix, perform the second association by the Hungarian algorithm to obtain the low-confidence bounding box matched with the track of the last time, the second unmatched track and the unmatched low-confidence bounding box.

4. The method of claim 1, wherein: In S7, the matched high-confidence bounding box, the low-confidence bounding box, the unmatched high-confidence bounding box, the second unmatched track and the unmatched low-confidence bounding box are managed, including: For the matched high-confidence bounding box and the low-confidence bounding box, the track state is updated by Kalman filtering; For the second unmatched track, if it is less than the maximum life threshold, the state is not updated, otherwise it is deleted; For the unmatched high-confidence bounding box and the unmatched low-confidence bounding box, if the confidence of the unmatched high-confidence bounding box is greater than the initialization threshold, it is initialized as a new track, otherwise it is deleted together with all the unmatched low-confidence bounding boxes.

5. The method of claim 1, wherein: The loss function of the lightweight re-identification network Fast OSNet is a joint loss function combining a multi-output branch ID loss and a circle loss, and the joint loss function L total is expressed by the following formula: L total = L multi-ID + w c L circle where L multi-ID is the multi-output branch ID loss, L circle is the circle loss, w c is the circle loss adjustment weight.

6. The method of claim 5, wherein: The multi-output branch ID loss L multi-ID is expressed by the following formula: wherein, is the component-level feature ID loss, which is represented by the following equation: In the above formula, N c is the number of data set IDs, p is the component level output branch number, W p is the classification weight matrix of the pth component level feature vector, is the y p th row and jth row of W i , respectively, y i is the positive sample number, is the pth component level feature vector of the positive and negative instances, respectively; For the global feature ID loss, the following equation is used: In the above formula, N c W represents the number of dataset IDs. g For the global feature vector h g The classification weight matrix, W respectively g The yth i row, j-th row, y i For positive sample numbers, These are the global feature vectors for positive and negative instances, respectively.

7. The method of claim 5, wherein: The circle loss L circle is expressed by the following equation: where r is a scale factor, m is a margin used to promote similarity separation, [·] + is a hinge function; To perform metric learning, a small batch of training samples of size L x K is constructed, containing L classes, each with K samples, for which the appearance embedding f a is computed. The cosine distance between the positive and negative sample appearance embeddings is computed. The cosine distance between the positive and negative sample appearance embeddings is computed. The cosine distance between the positive and negative sample appearance embeddings

Images