Transform-based long-short-term memory delay perception multi-target tracking method and system

Abstract

The invention discloses a long and short term memory delay perception multi-target tracking method and system based on Transform. The method comprises the following steps: embedding a parallel adaptive module into a residual network of freezing parameters, and taking the residual network as a backbone network; embedding the KAN into a gating circulation unit, selectively forgetting and updating the long-term memory vector by using reset gating and update gating of the gating circulation unit, and fusing the joint features of the long-term memory vector and the current frame by using high interpretability and data fitting capability of the KAN; a time prediction module is introduced, target information of a current frame moment is effectively acquired by using joint features of adjacent frames and long-term memory vectors, and information of a target at the current moment is predicted in combination with airborne computing power, so that the model effectively utilizes long-term information of a video stream and is more suitable for being applied to an online tracking task with limited computing power. According to the method, an effective solution is provided for solving the problem of tracking failure caused by target identity exchange of the tracker in a severe shielding and strenuous movement scene. The target ID exchange probability can be reduced, the model stability and robustness are improved, and the method can be applied to online airborne multi-target tracking tasks.

Description

Technical Field

[0001] The present invention belongs to the technical field of multi-object tracking, and particularly relates to a long short-term memory delay-aware multi-object tracking method and system based on Transformer. Background Art

[0002] Visual object tracking is the basis for many unmanned aerial vehicle applications. Most existing trackers are developed and evaluated for existing datasets in an offline environment. However, in practical applications, the online delay caused by the processing time of the tracker cannot be ignored because the environment has changed when the tracker finishes processing the current frame. In particular, it is more prominent in challenging tracking scenarios with limited on-board computing power, such as aircraft and unmanned vehicles.

[0003] Multi-object tracking aims to detect multiple objects in a video stream and continuously identify the object identities. The multi-object tracking task can be divided into two parts: object detection and association. The main purpose of object detection is to extract useful information from image information, identify and locate the position of the object in the image, and provide a reliable basis for subsequent processing and analysis in fields such as intelligent transportation and medical imaging. The main purpose of association is to match multiple objects between frames and maintain the identity of each object to continuously identify the object.

[0004] At present, the existing multi-object tracking research mainly includes two categories: detection tracking and query tracking. Detection tracking obtains object trajectories by associating a given set of detection boxes in the video stream. Detection tracking methods utilize existing detectors with proven excellent performance. By using some simple object association methods such as overlapping IoU and Kalman filters, they can perform excellently in pedestrian tracking scenarios with relatively regular movements and large differences in appearance features. However, they perform mediocrely when dealing with complex scenarios where object movements are irregular and appearance features are less different. Typical research works such as SORT apply Kalman filters to each instance and use the intersection over union (IoU) between the predicted boxes of the Kalman filters and the detection boxes for matching. DeepSORT introduced an appearance feature extraction network on this basis and used cosine distance to match features. In addition, JDE, FairMOT, Unicorn, etc. further explored the joint training architecture of object detection and appearance embedding, while ByteTrack achieved state-of-the-art performance through an enhanced version of SORT. Its key improvement lies in associating low-confidence detection boxes. BoT-SORT designed better Kalman filter states, camera motion compensation, and ReID feature fusion methods. OCSORT relaxed the linear motion assumption and adopted a learnable motion model to improve the robustness of trajectory recovery. Query tracking usually does not require an additional module to associate detection results. Different from the detection tracking paradigm, query tracking methods use tracking queries to gradually decode the positions of the objects being tracked. Most existing methods use learnable object queries for newborn object detection and tracking queries to locate the positions of the objects being tracked. Typical research works such as TransMOT and GTR applied spatio-temporal transformers to the calculation of the assignment matrix to enhance feature interaction and aggregation of historical information. Some research such as TransTrack, TrackFormer also explored the potential of introducing Transformers in the detection matching task. Dendorfer et al. introduced more complex motion estimation methods such as S-GAN through the trajectory prediction task to more accurately model pedestrian trajectories. These algorithms perform excellently in classic pedestrian tracking scenarios because the objects have obvious appearance features and regular motion patterns. However, when dealing with complex scenarios with irregular movements, the performance of these methods is often not ideal because unpredictable motion patterns can lead to the failure of the trajectory estimation and prediction modules. Current query-based methods usually only utilize the information of adjacent frames to achieve object tracking. Although the tracking queries can be continuously updated over time, most methods still do not explicitly utilize long-term information. This may lead to the situation that when a part of an object is occluded by other objects or the background, the model can only extract partial features and ignore the object or identify the object as another object; it may also lead to the situation that when an object that the model has already recognized reappears, it cannot well recover its identity but simply regards it as a newborn object. These phenomena are difficult problems that need to be solved urgently in frame-level object association.

[0005] In summary, although good research results have been achieved in the study of offline multi-object tracking tasks, in the case of limited airborne computing power, the computational delay of the offline tracker cannot be ignored, resulting in a mismatch between its analysis results and the real world, which will lead to a decline in tracking performance; when the target is occluded, it cannot extract the target features well, which may lead to target tracking failure; the existing methods fail to make good use of the long-term information of the video sequence, resulting in the possible failure to recover the identity of the target well when the target reappears. The following problems still need to be further studied: (1) The tracker output is always out of date, and there may be a mismatch between the tracker result and the current state. (2) The tracker can only process the latest frame, making the non-real-time tracker may skip some frames, making the object move more greatly. (3) When a part of the target is occluded by other targets or the background, the model can only extract partial features and ignore the target or identify the target as other targets. (4) When the target that the model has recognized reappears, it cannot recover its identity well, but simply regards it as a newly born target. Summary of the Invention

[0006] The purpose of the present invention is to provide a Transformer-based long short-term memory delay-aware multi-object tracking method for the problems existing in the above-mentioned prior art, so as to effectively realize multi-object tracking of an intelligent agent in the case of limited airborne computing power.

[0007] To achieve the technical solution of the purpose of the present invention: on the one hand, a Transformer-based long short-term memory delay-aware multi-object tracking method is provided, and the method includes the following steps:

[0008] Step 1, for the collected video stream, convert the graphic signal of each frame I t into the corresponding digital signal f t , as the input of the backbone network PARN, and extract the feature vector b t of the image target representation;

[0009] Step 2, input the feature vector b t into the Transformer-based feature encoder D enc to obtain the estimated value of the image feature coding embedding vector

[0010] Step 3, after the feature vector b t passes through the parallel adaptive network, it is concatenated with the estimate of the image coding embedding vector to obtain the image feature coding embedding vector The image feature coding embedding vector is concatenated with the randomly initialized learnable detection query vector Q detAs the input of the Transformer-based detection decoder D det The corresponding detection embedding vector is obtained through the detection decoder

[0011] Step 4. Embed the image feature encoding vector Detection embedding vector Tracking embedding vector After linear mapping, they are respectively used as the input query matrix Q, key matrix K, and value matrix V of the cross-attention-based joint decoder D joint Calculate the cross-attention to obtain the joint features of the detected newborn targets and the tracked targets The tracking embedding vector Is a vector initialized with a matrix of all zeros;

[0012] Step 5. Construct a long-term memory model. Based on this model, update the long-term temporal features of the tracked targets and combine with the image features of the current video frame to obtain the estimated value of the long-term memory vector at the next moment

[0013] Step 6. Construct a time prediction module. Combine the output of Step 5 and use this time prediction module to predict the tracking boxes and confidences of each target tracked at the moment when the airborne device completes the calculation

[0014] Furthermore, in Step 1, the backbone network PARN is an improved parallel adaptive network PARN based on the residual network, which is composed of multiple PARN sub-blocks connected in series. The PARN sub-block is formed by adding a parallel adaptive module to the Resnet sub-block;

[0015] Let x and y represent the input and output of the PARN sub-block respectively, res represent the residual block function, norm represent the normalization function, and nlf represent the non-linear activation function. The calculation method of the PARN sub-block is shown in the following formula:

[0016] y = res(x) + norm(W up * nlf(W down * x))

[0017] Among them, the non-linear activation function nlf uses the relu function, and W up and W down Represent the learnable convolution weight matrices for upsampling and downsampling respectively. The input x is transformed into a low-dimensional feature through downsampling, and after activation, upsampling restores its dimension, normalizes and adds the result of the residual block to obtain the output y

[0018] Further, the parallel adaptive network described in step 3 is a parallel adaptive structure based on a convolutional neural network, including a backbone convolutional layer and a multi-scale processing path in parallel with it; the multi-scale processing path can adaptively extract and fuse feature information of different scales.

[0019] Further, the method for obtaining the detection embedding vector described in step 3 is expressed as:

[0020]

[0021] In the formula, PA(*) represents the parallel adaptive network, W K represents the key weight matrix, W V represents the value weight matrix, Q det represents the detection query.

[0022] Further, the long-term memory model described in step 5 is:

[0023]

[0024] In the formula, represents the long-term memory vector initialized based on a matrix of all zeros, the target label l ∈ [1, L], the motion feature represents the tracking target position, represents the tracking target motion speed, represents the tracking target motion direction, and the three jointly describe the motion characteristics of the target in the video stream; the appearance feature represents the three-dimensional shape feature of the tracking target, represents the state feature of the target, represents the deep semantic information of the target behavior trend feature, and the three are used to describe the appearance shape, actions, and attributes of the tracking target.

[0025] Further, the update rule of the long-term memory vector in step 5 is:

[0026]

[0027] In the formula, respectively represent the long-term memory vectors corresponding to the (t - 1)th moment and the tth moment, η represents the learning rate, l′ represents the loss of self-supervised learning,

[0028] Further, the time prediction module in step 6 includes a short-term memory fusion layer, a memory attention layer, and a KAN layer;

[0029] The short-term memory fusion layer is used to fuse the joint feature memories of two consecutive frames to obtain the short-term memory O t ;

[0030] The memory attention layer is used to calculate the short-term memory O t and the estimated value of the long-term memory vector for cross-attention;

[0031] The KAN layer is used to calculate the output embedding vector using the cross-attention result for predicting the tracking box and confidence of each target tracked at the moment when the airborne device completes the calculation.

[0032] Furthermore, combining the output of step 5 in step 6, using this time prediction module to predict the tracking box and confidence of each target tracked at the moment when the airborne device completes the calculation, specifically including:

[0033] Step 6-1, applying the parallel adaptive network PA to encode the joint features at time t and fusing it with the joint features at time t-1 to obtain the short-term memory O of the target to be tracked t , expressed as:

[0034]

[0035] In the formula, the non-linear activation function nlf uses the relu function, W up and W down respectively represent the learnable convolution weight matrices for upsampling and downsampling, and norm represents the normalization function;

[0036] Step 6-2, using the long-term memory vector as the hidden state and the joint features at time t as the input at the current moment, through the deep learning model KAN_GRU based on the KAN network and the gated recurrent unit GRU, resetting the gate and updating the gate, so as to obtain the estimated value of the long-term memory vector expressed as:

[0037]

[0038] In the formula, R t represents the reset gate, Z t represents the update gate, represents the candidate hidden state, e represents the bitwise multiplication operation, H t represents the hidden state at the time step, {W xr ,W xz ,W xh}, {W hr ,W hz ,W hh}, {b r ,b z ,b h} represent the input transformation matrix, hidden transformation matrix, and bias vector for reset, update, and candidate gating respectively;

[0039] Step 6-3: Use the short-term memory O of the target to be tracked t , the joint feature at time t and the estimated value of the long-term memory vector as the query matrix, key matrix, and value matrix of the memory attention layer respectively, calculate the attention among the three, and fuse it with the long-term memory vector through a multi-layer perceptron to obtain the output embedding vector, expressed as:

[0040]

[0041] In the formula, represents the output embedding vector, which is used for updating the tracking embedding vector at the next time stamp, i.e., time t+1, and predicting the tracking box and confidence of each trajectory; d represents the dimension of the key vector. k

[0042] On the other hand, a long-short-term memory delay-aware multi-object tracking system based on Transformer is provided. The system includes:

[0043] The first module is used to convert the graphic signal of each frame I t in the collected video stream into the corresponding digital signal f t , which is used as the input of the backbone network PARN to extract the feature vector b t of the image target representation;

[0044] The second module is used to input the feature vector b t into the Transformer-based feature encoder D enc to obtain the estimated value of the image feature encoding embedding vector

[0045] The third module is used to pass the feature vector b t through a parallel adaptive network and then splice it with the estimated value of the image coding embedding vector to obtain the image feature coding embedding vector The image feature coding embedding vector and the randomly initialized learnable detection query vector Q det are used as the input of the Transformer-based detection decoder D det , and the corresponding detection embedding vector

[0046] The fourth module is used to use the image feature coding embedding vector the detection embedding vector Tracking embedding vector After linear mapping, they are respectively used as the input query matrix Q, key matrix K, and value matrix V of the joint decoder D based on cross-attention to calculate cross-attention and obtain the joint features of the detected new targets and the tracked targets joint The tracking embedding vector is a vector initialized with a zero matrix

[0047] The fifth module is used to construct a long-term memory model. Based on this model, the long-term temporal features of the tracked targets are updated, and combined with the image features of the current video frame to obtain the estimated value of the long-term memory vector at the next moment

[0048] The sixth module constructs a time prediction module. Combining the output of the fifth module, this time prediction module is used to predict the tracking boxes and confidences of each target being tracked at the moment when the airborne device completes the calculation

[0049] Compared with the prior art, the significant advantages of the present invention are

[0050] 1) This method provides an effective solution to solve the problem that the tracker fails to track due to target identity exchange in severely occluded and violently moving scenarios. It can reduce the probability of target ID exchange, improve the stability and robustness of the model, and can be applied to online airborne multi-target tracking tasks

[0051] 2) The present invention embeds a parallel adaptive module into the residual network with frozen parameters and uses it as the backbone network. Compared with the full-parameter fine-tuning of existing work, it can effectively avoid catastrophic forgetting of the pre-trained feature extraction network, and can make the network better applicable to multi-target tracking tasks with only a small number of parameters updated. At the same time, it improves the parallelism of the program during the inference process

[0052] 3) Embed KAN into the gated recurrent unit, use the reset gate and update gate of the gated recurrent unit to selectively forget and update the long-term memory vector, and at the same time use the high interpretability and data fitting ability of KAN to fuse the long-term memory vector and the joint features of the current frame

[0053] 4) The introduced time prediction module uses the joint features of adjacent frames and the long-term memory vector to effectively obtain the target information at the current frame moment, and combines the airborne computing power to predict the information of the target at the current moment, enabling the model to effectively utilize the long-term information of the video stream and be more suitable for online tracking tasks with limited computing power

[0054] ​5) The present invention collects relevant video stream data from the multi-object tracking field dataset, preprocesses it, and constructs a dataset; a delay-aware multi-object tracking method based on Transformer long short-term memory and a long-term memory enhanced online object tracking method are proposed, providing an effective method to solve the problem of tracking failure caused by ID swapping in the tracking of objects in severe occlusion and violent movement scenarios, reducing the probability of object ID swapping, improving the stability and robustness of the model, and can be applied to online airborne multi-object tracking tasks.

[0055] The present invention will be further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a network framework diagram of the delay-aware multi-object tracking method based on Transformer long short-term memory of the present invention.

[0057] Figure 2 It is a structure diagram of the time prediction module in an embodiment.

[0058] Figure 3 It is a structure diagram of the parallel adaptive network based on the residual network in an embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0059] In order to make the objectives, technical solutions and advantages of the present application more clear, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] It should be noted that if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the quantity of the technical features indicated. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of such features. In addition, the technical solutions between various embodiments can be combined with each other, but it must be based on the fact that those skilled in the art can implement them. When the combination of technical solutions appears to be contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the protection scope required by the present invention.

[0061] In one embodiment, in combination with Figure 1 , a delay-aware multi-object tracking method based on Transformer long short-term memory is provided, and the method includes the following steps:

[0062] Step 1, for the collected video stream, convert the graphic signal of each frame I t into the corresponding digital signal f t, as the input of the backbone network PARN, extract the feature vector b of the image target representation t ;

[0063] Designed a parallel adaptive network PARN based on the residual network as the backbone network to reduce the model training complexity, prevent catastrophic forgetting in the full-scale fine-tuning of the model, and improve the adaptability of the model in scenarios of drastic target movement and severe occlusion. In the video stream, convert the graphic signal of frame I t into a digital signal f t , as the input of the backbone network PARN, extract the feature vector b of the image target representation t , let t represent the timestamp, f t represent the frame corresponding to the t-th moment in the video stream, and b t represent the image representation features extracted by the backbone network PARN from the frame at the t-th moment of the video stream. Its expression is as follows:

[0064] b t = PARN(f t )

[0065] The backbone network PARN is an improved parallel adaptive network PARN based on the residual network. As Figure 3 shown, it is composed of multiple PARN sub-blocks connected in series. The PARN sub-block is formed by adding a parallel adaptive module to the Resnet sub-block; only a small number of parameters are fine-tuned to improve the adaptability of the model to the feature extraction of specific sub-tasks and reduce the possibility of catastrophic forgetting of the model.

[0066] Let x and y represent the input and output of the PARN sub-block respectively, res represent the residual block function, norm represent the normalization function, and nlf represent the non-linear activation function. The calculation method of the PARN sub-block is as follows:

[0067] y = res(x) + norm(W up * nlf(W down * x))

[0068] Among them, the non-linear activation function nlf uses the relu function, and W up and W down represent the learnable convolution weight matrices for upsampling and downsampling respectively. The input x is transformed into low-dimensional features after downsampling, and after activation, upsampling restores its dimension, normalizes it, and adds it to the result of the residual block to obtain the output y. During the training process, freeze the parameters of the pre-trained residual network, and only update the weight matrices W up and W down , maintain the consistency and adaptability of the model's feature extraction ability, avoid catastrophic forgetting of the network compared with full-parameter fine-tuning, and improve the efficiency at the same time.

[0069] Here, the proposed Parallel Adaptive Residual Network (PARN) adds a parallel adaptive module compared with the existing technology; the convolutional layer and pooling layer are applied to process the digital signal f t of the frame I t in the video stream, and the feature map is obtained through n cascaded PARN sub-blocks; among them, in each sub-block, the residual block with frozen parameters and the parallel adaptive block with trainable parameters can be calculated in parallel, which improves the parallelism and adaptability of the model and reduces the complexity of model training at the same time.

[0070] Step 2: Input the feature vector b t into the Transformer-based feature encoder D enc to obtain an estimated value of the image feature encoding embedding vector

[0071]

[0072] For each layer of the encoder D enc , let the input be the vector matrix A, and multiply A by the three weight matrices W Q , W k , W V respectively to obtain the matrices Q, K, and V. The specific calculation method is shown in the following formula:

[0073]

[0074] Then, pass the matrices Q, K, and V through the linear layer to obtain the matrices Q i , K i , V i corresponding to each head, i = 1,..., h:

[0075]

[0076] Perform the normalized dot-product self-attention calculation for each head to obtain the self-attention matrix:

[0077]

[0078] Among them, d is the dimension of Q i , and after passing through the softmax function, it takes the inner product with V i . Concatenate the self-attention matrices att i together and then perform a linear transformation to obtain the output matrix B:

[0079] B = concat(att1,..., att h )W

[0080] In the above algorithm, the feature matrix Q of each head is obtained from the matrix Ai , K i , V i After two adjacent linear transformations, unnecessary calculations will be performed by the program. During program operation, the same purpose can be achieved by directly performing a linear transformation on A to obtain the target dimension and then making a division.

[0081] Step 3, eigenvector b t After passing through the parallel adaptive network, it is estimated with the image coding embedding vector Concatenated to obtain the image feature coding embedding vector The image feature coding embedding vector And the randomly initialized learnable detection query vector Q det Serves as the input of the Transformer-based detection decoder D det To obtain the corresponding detection embedding vector through the detection decoder The specific calculation is as follows:

[0082]

[0083] In the formula, PA represents the parallel adaptive network, and through the linear mapping of the key weight matrix W K And the value weight matrix W V , the fusion embedding vector Is mapped into a key matrix and a value matrix, and through the detection decoder D det Calculate its cross-attention weight with the detection query Q det To obtain the detection embedding vector

[0084] Step 4, the image feature coding embedding vector The detection embedding vector The tracking embedding vector After linear mapping, they are respectively used as the input query matrix Q, key matrix K, and value matrix V of the cross-attention-based joint decoder D joint To calculate the cross-attention and obtain the joint features of the detected new targets and the tracked targets The tracking embedding vector Is a vector initialized with a zero matrix;

[0085] Step 5, construct a long-term memory model. Based on this model, update the long-term temporal features of the tracked targets, and combine the image features of the current video frame to obtain the estimated value of the long-term memory vector at the next moment

[0086] The long-term memory model is:

[0087]

[0088] In the formula, represents the long-term memory vector initialized based on the all-zero matrix, the target label l ∈ [1, L], and the motion feature represents the tracked target position, represents the motion speed of the tracked target, represents the motion direction of the tracked target. The three together describe the motion characteristics of the target in the video stream; the appearance feature represents the three-dimensional shape feature of the tracked target, including surface features such as the length, width, and height of the target shape and the depth semantic features of the shape, represents the state feature of the target, including the depth semantic features of states such as the target flying normally, climbing, and diving; represents the depth semantic information of the target behavior trend feature. The three are used to describe the appearance shape, actions, and attributes of the tracked target. The long-term memory vector has a clear physical meaning, enabling the model to accurately infer the shape and actions of the target in the next frame and match the corresponding target to the correct trajectory through the time prediction module, effectively reducing the probability of identity exchange events.

[0089] The update rule of the long-term memory vector is:

[0090]

[0091] In the formula, respectively represent the long-term memory vectors corresponding to the (t - 1)th moment and the tth moment, η represents the learning rate, and l′ represents the loss of self-supervised learning. The long-term memory vector is updated using the gradient descent method during the application model training process, making the long-term memory vector itself a learnable model. During testing, the parameters are updated to actively learn the motion features and appearance features of the tracked target in a specific scenario. f represents the calculation formula of the deep learning model KAN_GRU based on the KAN network and the gated recurrent unit GRU.

[0092] Step 6: Construct a time prediction module. Combining the output of Step 5, use this time prediction module to predict the tracking box and confidence of each target being tracked at the moment when the airborne device completes the calculation.

[0093] Combined with Figure 2 , the time prediction module includes a short-term memory fusion layer, a memory attention layer, and a KAN layer;

[0094] The short-term memory fusion layer is used to fuse the joint feature memories of two consecutive frames to obtain the short-term memory O t ;

[0095] The memory attention layer is used to calculate the short-term memory Ot Cross-attention with the estimated value of the long-term memory vector ;

[0096] The KAN layer is used to calculate the output embedding vector using the cross-attention result for predicting the tracking box and confidence of each target tracked at the moment when the airborne device completes the calculation.

[0097] The calculation methods of the short-term memory, the calculation method of the long-term memory estimated value, and the calculation method of the output embedding vector in the time prediction module are shown as follows:

[0098]

[0099] In the formula, represents the joint feature at time t, represents the joint feature at time t-1, O t represents the short-term memory at time t, and in formula (12) represents the estimated value of the long-term memory vector, applying parallel adaptive network coding and fusing it with to obtain the short-term memory O t ; The Memory Attention layer (MA) calculates the cross-attention among the short-term memory, the encoded joint feature, and the long-term memory estimated value to obtain the output embedding vector and uses it for the update of the tracking embedding vector at the next time stamp and the prediction of the tracking box and confidence of each trajectory.

[0100] The specific process of time prediction includes:

[0101] Step 6-1, applying parallel adaptive network PA to encode the joint feature at time t fusing it with the joint feature at time t-1 to obtain the short-term memory O of the target to be tracked t , expressed as:

[0102]

[0103] In the formula, the non-linear activation function nlf uses the relu function, W up and W down respectively represent the learnable convolution weight matrices for upsampling and downsampling, and norm represents the normalization function;

[0104] Step 6-2, using the long-term memory vector as the hidden state, and using the joint feature at time t As the input at the current moment, through the deep learning model KAN_GRU based on the Kolmogorov - Arnold network (KAN) and the Gated Recurrent Unit (GRU), the reset gate and the update gate are reset to obtain an estimated value of the long - term memory vector It is expressed as:

[0105]

[0106] In the formula, R t represents the reset gate, Z t represents the update gate, represents the candidate hidden state, e represents the bit - by - bit multiplication operation, H t represents the hidden state at the time step, {W xr , W xz , W xh}, {W hr , W hz , W hh}, {b r , b z , b h} respectively represent the input transformation matrix, the hidden transformation matrix and the bias vector of the reset, update and candidate gates;

[0107] Step 6 - 3: Take the short - term memory O t of the target to be tracked, the joint feature at time t and the estimated value of the long - term memory vector as the query matrix, the key matrix and the value matrix of the memory attention layer respectively, calculate the attention among the three, and fuse it with the long - term memory vector through a multi - layer perceptron to obtain the output embedding vector, which is expressed as:

[0108]

[0109] In the formula, represents the output embedding vector, which is used for the update of the tracking embedding vector at the next time stamp, i.e., time t + 1, and the prediction of the tracking box and confidence of each trajectory; d represents the dimension of the key vector. k

[0110] In one embodiment, a long - short - term memory delay - aware multi - object tracking system based on Transformer is provided. The system includes:

[0111] A first module for converting the graphic signal of each frame I t in the collected video stream into the corresponding digital signal f t t, as the input of the backbone network PARN, extract the feature vector b of the image target representation t ;

[0112] The second module is used to input the feature vector b t into the Transformer-based feature encoder D enc to obtain an estimated value of the image feature encoding embedding vector

[0113] The third module is used to pass the feature vector b t through a parallel adaptive network and then concatenate it with the estimated value of the image encoding embedding vector to obtain the image feature encoding embedding vector Take the image feature encoding embedding vector and the randomly initialized learnable detection query vector Q det as the input of the Transformer-based detection decoder D det and obtain the corresponding detection embedding vector through the detection decoder

[0114] The fourth module is used to linearly map the image feature encoding embedding vector the detection embedding vector the tracking embedding vector and respectively use them as the input query matrix Q, key matrix K, and value matrix V of the cross-attention-based joint decoder D joint to calculate the cross-attention and obtain the joint features of the detected new targets and the tracked targets The tracking embedding vector is a vector initialized with a all-zero matrix;

[0115] The fifth module is used to build a long-term memory model. Based on this model, update the long-term temporal features of the stored tracked targets and combine them with the image features of the current video frame to obtain an estimated value of the long-term memory vector at the next moment

[0116] The sixth module builds a time prediction module. Combining the output of the fifth module, use this time prediction module to predict the tracking boxes and confidences of each target being tracked at the moment when the airborne device completes the calculation.

[0117] For the specific limitations of the Transformer-based long short-term memory delay-aware multi-object tracking system, reference can be made to the limitations of the Transformer-based long short-term memory delay-aware multi-object tracking method in the foregoing text, which will not be elaborated herein. Each module in the above-mentioned Transformer-based long short-term memory delay-aware multi-object tracking system can be implemented in whole or in part by software, hardware, and their combination. Each of the above modules can be embedded in the processor of the computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each of the above modules.

[0118] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the following is implemented:

[0119] Step 1. For the collected video stream, convert the graphic signal of each frame I t into the corresponding digital signal f t , which is used as the input of the backbone network PARN, and extract the feature vector b t of the image target representation;

[0120] Step 2. Input the feature vector b t into the Transformer-based feature encoder D enc to obtain the estimated value of the image feature encoding embedding vector

[0121] Step 3. After the feature vector b t passes through the parallel adaptive network, it is concatenated with the estimated value of the image encoding embedding vector to obtain the image feature encoding embedding vector The image feature encoding embedding vector and the randomly initialized learnable detection query vector Q det are used as the input of the Transformer-based detection decoder D det , and the corresponding detection embedding vector

[0122] Step 4. The image feature encoding embedding vector the detection embedding vector the tracking embedding vector After linear mapping, they are respectively used as the input query matrix Q, key matrix K, and value matrix V of the cross-attention-based joint decoder D joint , calculate the cross-attention, and obtain the joint features of the detected new targets and the tracked targets The tracking embedding vector is a vector initialized with a zero matrix;

[0123] Step 5: Construct a long-term memory model. Based on this model, update the long-term temporal features of the tracked target, and combine the image features of the current video frame to obtain the estimated value of the long-term memory vector at the next moment.

[0124] Step 6: Construct a time prediction module. Combine the output of Step 5 and use this time prediction module to predict the tracking box and confidence of each target being tracked at the moment when the airborne device completes the calculation.

[0125] For the specific limitations of each step, reference can be made to the limitations of the long short-term memory delay-aware multi-object tracking method based on Transformer in the above text, which will not be elaborated here.

[0126] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, it realizes:

[0127] Step 1: For the collected video stream, convert the graphic signal of each frame I t into the corresponding digital signal f t , which is used as the input of the backbone network PARN to extract the feature vector b t of the image target representation;

[0128] Step 2: Input the feature vector b t into the Transformer-based feature encoder D enc to obtain the estimated value of the image feature coding embedding vector

[0129] Step 3: After the feature vector b t passes through the parallel adaptive network, it is concatenated with the estimated value of the image coding embedding vector to obtain the image feature coding embedding vector The image feature coding embedding vector and the randomly initialized learnable detection query vector Q det are used as the input of the Transformer-based detection decoder D det to obtain the corresponding detection embedding vector through the detection decoder

[0130] Step 4: After linearly mapping the image feature coding embedding vector the detection embedding vector and the tracking embedding vector they are respectively used as the input of the cross-attention-based joint decoder Djoint For the input query matrix Q, key matrix K, and value matrix V, calculate the cross-attention to obtain the joint features of the detected new targets and the tracked targets. The tracked embedding vector is a vector initialized with a matrix of all zeros;

[0131] Step 5: Construct a long-term memory model. Based on this model, update the stored long-term temporal features of the tracked targets, and combine the image features of the current video frame to obtain the estimated value of the long-term memory vector at the next moment.

[0132] Step 6: Construct a time prediction module. Combine the output of Step 5 and use this time prediction module to predict the tracking boxes and confidences of each target being tracked at the moment when the airborne device finishes the calculation.

[0133] For the specific limitations of each step, reference can be made to the limitations of the Transformer-based long short-term memory delay-aware multi-object tracking method in the above text, which will not be elaborated here.

[0134] As a specific example, in one of the embodiments, the present invention is further described in detail.

[0135] Combined with Figure 1 , the present invention proposes a Transformer-based long short-term memory delay-aware multi-object tracking method. The specific verification includes the following steps:

[0136] Step 1: Construct a dataset. For the publicly available and general multi-object tracking datasets MOT17 (a total of 42 video sequences, including 21 video sequences as the training set and 21 video sequences as the detection set) and DanceTrack (a total of 100 video sequences), adjust the short side of the input image to 800, and the maximum size limit is 1536.

[0137] Step 2: On the PyTorch framework, implement the Transformer-based long short-term memory delay-aware multi-object tracking method. Adopt ResNet50 pre-trained on the COCO dataset and freeze its parameters. Use two learnable convolutional layers with a convolution kernel of 1×1 as parallel adaptive modules to be embedded in ResNet50. Use Deformable-DETR to construct the Transformer encoder-decoder module, and its weights also come from the results pre-trained by PyTorch official on the COCO dataset.

[0138] Step 3: Set the training parameters. The minibatch is set to 1. Each minibatch contains a video clip with multiple frames. Sample video frames at random intervals from 1 to 10. Use the AdamW optimizer. The initial learning rate is set to 2×10 -4 , and the MOT17 training set was trained for 80 epochs. The learning rate was adjusted to 2×10 -5 in the last five epochs. It was trained on the DanceTrack training set for 16 epochs, and the learning rate was reduced by 10 times in the 12th epoch.

[0139] Step 4: Data augmentation method. Add the CrowdHuman dataset as a supplement to the MOT17 training set to reduce the probability of model overfitting. Randomly resize, crop, and rotate the image frames.

[0140] Step 5: Use the Higher-Order Metric for Evaluating Multi-Object Tracking (HOTA) and Association Accuracy (AssA) to evaluate the method of the present invention. At the same time, the CLEARMOT metrics DetA, MOTA, and the IDF1 metric are listed in the experimental results.

[0141] Step 6: Verify the model generalization on the MOT17 test set. As shown in Table 1, the method of the present invention slightly improves the performance on the MOT17 test set, reaching 65.5% HOTA. As a method that also utilizes the memory mechanism, the method of the present invention achieves higher AssA (7% and 3.8% higher respectively) and IDF1 (19.3% and 6.8% higher respectively) compared to MeMOT and MeMOTR.

[0142] Table 1 Evaluation results of comparison with existing methods on the MOT17 dataset (the method with the best performance is marked in bold)

[0143]

[0144] Step 7: Verify the model generalization on the DanceTrack test set. As shown in Table 2, the method of the present invention achieves 59.4% AssA, which is 1% higher than MeMOTR. It also achieves a competitive 66.2% in the HOTA metric.

[0145] Table 2 Evaluation results of comparison with existing methods on the DanceTrack dataset (the method with the best performance is marked in bold)

[0146]

[0147] Step 8, ablation experiments were carried out on the DanceTrack dataset. As shown in Table 3 and Table 4, the results in Table 3 show that embedding a parallel adaptive module in the backbone network can significantly improve the HOTA, AssA, and IDF1 metrics, by 2.8%, 5.6%, and 2.5% respectively, indicating that the module designed in the present invention plays a role in the adaptability of feature extraction. Embedding a parallel adaptive module in the time prediction module can also improve the HOTA, AssA, and IDF1 metrics, which are 2.7%, 2.7%, and 4.1% respectively. The results in Table 4 show that using long-term memory and applying the KAN-improved gated recurrent unit has a huge performance improvement compared to the original method. Among them, the HOTA metric has increased by 14.8%, the AssA metric has increased by 9.1%, and the IDF1 metric has increased by 12.5%. Reasonably using the long-term memory vector can make the model have better association performance, with performance improvements of 1.6%, 4.6%, and 3.5% in the three metrics of HOTA, AssA, and IDF1 respectively. Using KAN can help the model better detect objects, with the performance of the DetA and MOTA metrics increasing by 4.1% and 3.7% respectively, and it also has a certain help in the association performance, with performance improvements of 1.0%, 1.5%, and 0.8% in the three evaluation metrics of HOTA, AssA, and IDF1 respectively.

[0148] Table 3 Comparison of different configurations of whether to embed a parallel adaptive module in the backbone network and the time prediction module (the method with the best performance is marked in bold)

[0149]

[0150] Table 4 Ablation experiment on whether to use KAN improvement in the long-term memory vector and the gated recurrent unit based on KAN (the method with the best performance is marked in bold)

[0151]

[0152] Here, the original method refers to the method of directly feeding the joint embedding vector at time t encoded by the parallel adaptive network and fused with the joint features at time t - 1 into KAN to obtain the required tracking embedding vector.

[0153] In summary, the present invention provides a method of embedding a vector parallel adaptive module in the backbone network; based on the time prediction module, the tracking is regressed to the target tracking box at the current moment. The target tracking method based on Transformer has high prediction accuracy and can be effectively applied to the online airborne multi-target tracking task.

[0154] The foregoing has shown and described the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. What is described in the above embodiments and the specification only illustrates the principles of the present invention. Without departing from the spirit and scope of the present invention, any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A Transformer-based long short-term memory delay perception multi-target tracking method, characterized in that: The method comprises the following steps: Step 1: for the collected video stream, each frame I t The graphic signal is converted into the corresponding digital signal f t , as the input of the backbone network PARN, extract the feature vector b of the image target representation t ; Step 2: transform the feature vector b t Input to the Transformer-based feature encoder D enc Get the estimated value of the image feature encoding embedding vector Step 3, feature vector b t After the parallel adaptive network, the estimated embedding vector of the image code Splicing to get the image feature encoding embedding vector Encode image features into vectors With a randomly initialized learnable detection query vector Q det As a Transformer-based detection decoder D det The corresponding detection embedding vector is obtained through the detection decoder. Step 4: Encode the image features into embedding vectors Detection Embedding Vector Keep track of the embedding vectors After linear mapping, they are used as joint decoders D based on cross attention. joint The input query matrix Q, key matrix K, and value matrix V are used to calculate the cross attention and obtain the joint features of the detected new target and the tracked target. The tracking embedding vector is a vector initialized with an all-zero matrix; Step 5: Build a long-term memory model. Based on this model, update and store the long-term temporal features of the tracked target, combine the image features of the current video frame, and obtain the estimated value of the long-term memory vector at the next moment. Step 6, construct a time prediction module, and combine it with the output of step 5, and use the time prediction module to predict the tracking box and confidence of each target tracked at the moment when the airborne device completes the calculation.

2. The Transformer-based long short-term memory delay-aware multi-target tracking method according to claim 1 is characterized in that: In step 1, the backbone network PARN is an improved parallel adaptive network PARN based on residual network, which is composed of multiple PARN sub-blocks connected in series, and the PARN sub-block is formed by adding a parallel adaptive module to the Resnet sub-block; Let x, y represent the input and output of the PARN sub-block respectively, res represents the residual block function, norm represents the normalization function, and nlf represents the nonlinear activation function. The calculation method of the PARN sub-block is shown as follows: y=res(x)+norm(W up *nlf(W down *x)) Among them, the nonlinear activation function nlf uses the relu function, W up and W down They represent the learnable convolution weight matrices for upsampling and downsampling respectively. The input x is transformed into a low-dimensional feature after downsampling. After activation, the upsampling restores its dimension, normalizes it and adds it to the result of the residual block to get the output y.

3. The Transformer-based long short-term memory delay perception multi-target tracking method according to claim 1 is characterized in that: The parallel adaptive network described in step 3 is a parallel adaptive structure based on a convolutional neural network, including a backbone convolution layer and a multi-scale processing path parallel to the backbone convolution layer; the multi-scale processing path can adaptively extract and fuse feature information of different scales.

4. The Transformer-based long short-term memory delay perception multi-target tracking method according to claim 3 is characterized in that: The multi-scale processing path is composed of a downsampling convolution layer, an upsampling convolution layer and a normalization layer connected in series.

5. The Transformer-based long short-term memory delay perception multi-target tracking method according to claim 1, characterized in that: The method for obtaining the detection embedding vector in step 3 is expressed as: Where PA(*) represents the parallel adaptive network, W K represents the key weight matrix, W V represents the value weight matrix, Q det Represents a detection query.

6. The Transformer-based long short-term memory delay perception multi-target tracking method according to claim 1, characterized in that: The long-term memory model described in step 5 is: In the formula, represents the long-term memory vector initialized based on the all-zero matrix, the target label l∈[1,L], the motion feature Indicates the tracking target position. Indicates the target moving speed. Indicates the direction of the target's motion. The three together describe the target's motion characteristics in the video stream; appearance features Represents the three-dimensional shape features of the tracking target, Indicates the state characteristics of the target, The deep semantic information representing the target behavior trend characteristics is used to describe the appearance, shape, action and attributes of the tracked target.

7. The Transformer-based long short-term memory delay-aware multi-target tracking method according to claim 1, characterized in that: The update rule for the long-term memory vector in step 5 is: In the formula, They represent the long-term memory vectors corresponding to time t-1 and time t respectively, η represents the learning rate, l′ represents the loss of self-supervised learning, f represents the calculation formula of the deep learning model KAN_GRU based on the KAN network and the gated recurrent unit GRU.

8. The Transformer-based long short-term memory delay perception multi-target tracking method according to claim 1, characterized in that: The time prediction module in step 6 includes a short-term memory fusion layer, a memory attention layer and a KAN layer; The short-term memory fusion layer is used to fuse the joint feature memory of two consecutive frames to obtain the short-term memory O t ; The memory attention layer is used to calculate the short-term memory O t and the estimated value of the long-term memory vector Cross-attention between The KAN layer is used to calculate the output embedding vector using the cross attention result, so as to predict the tracking box and confidence of each target tracked at the moment when the onboard device completes the calculation.

9. The Transformer-based long short-term memory delay perception multi-target tracking method according to claim 8, characterized in that: In step 6, the output of step 5 is combined to use the time prediction module to predict the tracking frame and confidence of each target tracked at the time when the airborne device completes the calculation, specifically including: Step 6-1, apply the parallel adaptive network PA to encode the joint features at time t Combine it with the joint feature at time t-1 Fusion obtains the short-term memory O of the tracked target t , expressed as: In the formula, the nonlinear activation function nlf uses the relu function, W up and W down They represent the learnable upsampling and downsampling convolution weight matrices respectively, and norm represents the normalization function; Step 6-2: Take the long-term memory vector as the hidden state and the joint feature at time t As the input at the current moment, the deep learning model KAN_GRU based on the KAN network and the gated recurrent unit GRU is used to reset the gate and update the gate to obtain the estimated value of the long-term memory vector It is expressed as: In the formula, R t Represents the reset gate, Z t represents the update gate, represents the candidate hidden state, e represents the bitwise multiplication operation, H t represents the time step hidden state, {W xr ,W xz ,W xh }、{W hr ,W hz ,W hh }、{b r ,b z ,b h } represent the input transformation matrix, hidden transformation matrix and bias vector of reset, update and candidate gating respectively; Step 6-3, the short-term memory O of the tracked target t , the joint features at time t and an estimate of the long-term memory vector They are used as the query matrix, key matrix and value matrix of the memory attention layer respectively, and the attention between them is calculated. They are then fused with the long-term memory vector through a multi-layer perceptron to obtain the output embedding vector, which is expressed as: In the formula, Represents the output embedding vector, which is used to track the embedding vector at the next timestamp, i.e., time t+1 Update of the tracking box and confidence prediction of each trajectory; d k Indicates the dimension of the key vector.

10. The Transformer-based long short-term memory delay perception multi-target tracking system according to any one of claims 1 to 9, characterized in that: The system comprises: The first module is used to convert each frame I in the collected video stream t The graphic signal is converted into the corresponding digital signal f t , as the input of the backbone network PARN, extract the feature vector b of the image target representation t ; The second module is used to convert the feature vector b t Input to the Transformer-based feature encoder D enc Get the estimated value of the image feature encoding embedding vector The third module is used to transform the feature vector b t After the parallel adaptive network, the estimated embedding vector of the image code Splicing to get the image feature encoding embedding vector Encode image features into vectors With a randomly initialized learnable detection query vector Q det As a Transformer-based detection decoder D det The corresponding detection embedding vector is obtained through the detection decoder. The fourth module is used to encode image features into vectors Detection Embedding Vector Keep track of the embedding vectors After linear mapping, they are used as joint decoders D based on cross attention. joint The input query matrix Q, key matrix K, and value matrix V are used to calculate the cross attention and obtain the joint features of the detected new target and the tracked target. The tracking embedding vector is a vector initialized with an all-zero matrix; The fifth module is used to build a long-term memory model. Based on this model, the long-term temporal features of the tracked target are updated and stored, and the image features of the current video frame are combined to obtain the estimated value of the long-term memory vector at the next moment. The sixth module constructs a time prediction module, and combines the output of the fifth module to use the time prediction module to predict the tracking box and confidence of each target tracked at the moment when the airborne device completes the calculation.

Images