A multi-target tracking method, system, device and readable storage medium

Abstract

The application discloses a kind of multi-target tracking method, system, equipment and readable storage medium, the multi-target tracking method includes the following steps: step 1, obtaining N frame video sequence to be processed;Step 2, the N frame video sequence to be processed is frame by frame into tracking framework and is processed to multi-target tracking, and multi-target tracking result is output.The application provides a kind of multi-target tracking method based on long-term motion feature, can solve the technical problems of identity recognition error and trajectory interruption caused by crowded crowd and serious occlusion existing in prior art;Wherein, by extracting " interactive motion feature " to obtain short-term correlation result, by extracting " historical trajectory motion feature " to obtain long-term correlation result, the final tracking result is obtained by combining two, can enhance the identity recognition ability of frame by frame in tracking process and the identity re-identification ability across frame.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision, and particularly relates to a multi-target tracking method, system, device and readable storage medium. BACKGROUND

[0002] The multi-target tracking task is to locate each target in a video sequence and identify the continuous identity of each target. The multi-target tracking task is a very important task in the field of computer vision and is a core technology in many other fields. For example, automatic driving, behavior recognition, intelligent monitoring and medical influence analysis.

[0003] At the current stage, the main challenge in the multi-target tracking method comes from the identity recognition error and track interruption problem caused by crowded groups and severe occlusion. Among them, the existing motion model independently models each target without considering the interaction between targets, and cannot well handle the identity recognition error caused by crowded groups.

[0004] For the track interruption problem caused by severe occlusion, the common existing solution is to introduce an appearance feature model to re-identify the identity of each target using appearance features. However, the reliability of the appearance feature can be easily weakened by some factors, such as low resolution, pose change, small target, etc., especially when the target is occluded and the complete appearance feature cannot be extracted when the target enters the crowded group. SUMMARY

[0005] The present application aims to provide a multi-target tracking method, system, device and readable storage medium to solve one or more of the above technical problems. The present application specifically provides a multi-target tracking method based on long-term and short-term motion features, which can solve the technical problems of identity recognition error and track interruption caused by crowded groups and severe occlusion existing in the prior art.

[0006] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0007] The first aspect of the present application provides a multi-target tracking method, comprising the following steps:

[0008] Step 1, obtaining N frames of video sequences to be processed; wherein N>2;

[0009] Step 2, transmitting the N frames of video sequences to be processed into a tracking framework frame by frame for multi-target tracking processing, and outputting a multi-target tracking result;

[0010] Wherein, the step of transmitting the N frames of video sequences to be processed into the tracking framework frame by frame for multi-target tracking processing comprises:

[0011] obtaining a preliminary tracking result based on the first frame of video and the second frame of video;

[0012] For the third frame video to the t-th frame video in the N-th frame video, the target detection result of the t-th frame video is obtained; based on the t-2-th frame video and the t-1-th frame video, the offset of each tracking target is obtained; the obtained offset is input into the pre-trained interaction module to model the interaction between targets and predict the offset of the t-th frame video; based on the t-1-th frame video target detection result, the t-th frame video and the offset of the t-th frame video, the short-term association result of the t-th frame video is obtained;

[0013] The target whose state is lost in the process of obtaining the short-term association result of the t-th frame video and the target detection result which is not matched in the algorithm are input into the pre-trained long-distance recovery module for lost target recovery processing, and the long-term association result of the t-th frame video is output.

[0014] The short-term association result of the t-th frame video is merged with the long-term association result, and finally the multi-target tracking result of the t-th frame video is obtained.

[0015] The further improvement of the application is that the step of transmitting the first frame video and the second frame video into the tracking framework for multi-target tracking processing specifically comprises:

[0016] The first frame video and the second frame video are transmitted into the tracking framework for target detection, and the target detection result of the first frame video and the target detection result of the second frame video are obtained respectively; wherein the target detection result comprises the position information and the size information of the preset target; the similarity between targets is calculated based on the obtained position information and size information, and the Hungarian algorithm is used to obtain the preliminary tracking result;

[0017] The third frame of video to the Nth frame of video is input into the tracking framework frame by frame, and multi-target tracking processing is performed to output multi-target tracking results;Among them, for the tth frame of video in the third frame of video to the Nth frame of video, target detection is performed to obtain the target detection result of the tth frame of video;Based on the position information and size information in the target detection results of the t-2th frame of video and the t-1th frame of video, the offset of each tracking target is calculated and obtained;The calculated offset is input into the pre-trained interaction module to model the interaction between targets, and the offset of the tth frame of video is predicted and output;Based on the position information, size information in the target detection results of the tth frame of video and the t-1th frame of video, and the offset of the tth frame of video, the similarity between targets is calculated, and the Hungarian algorithm or greedy algorithm is used to obtain the short-term association result of the tth frame of video;Determine whether the tth frame of video has a target in a lost state and whether there is a target detection result that is not matched in the algorithm;If yes, input the target in a lost state and the unmatched detection result into the pre-trained long-distance recovery module to perform lost target recovery processing, and output the long-term association result of the tth frame of video;Merge the short-term association result and the long-term association result of the tth frame of video, and finally obtain the multi-target tracking result of the tth frame of video;If not, the short-term association result of the tth frame of video is taken as the final multi-target tracking result of the tth frame of video.

[0018] Further improvement of the application is that the expression for calculating and obtaining the offset of each tracking target based on the position information and size information in the target detection results of the t-2th frame of video and the t-1th frame of video is

[0019]

[0020] In the formula, (x, y) represents the center point coordinates of the target, w represents the width of the target, h represents the height of the target, t and j represent the jth target of the tth frame, and Δ represents the change amount.

[0021] Further improvement of the application is that in the pre-trained interaction module, the step of modeling the interaction between targets and predicting and outputting the offset includes:

[0022] (1) preliminary extraction of interaction information, including: for the tth frame of video, based on the offset O t of each tracking target obtained from the t-2th frame of video and the t-1th frame of video t , and the absolute coordinates are spliced into I t , the interaction between the trajectories of the targets is calculated using the self-attention mechanism, and the calculation expression of the self-attention is

[0023] E t = φ(I t , W E )

[0024] Q t = φ(E t , W Q )

[0025] K t = φ(E t , W K );

[0026]

[0027] where φ(·,·) represents a linear transformation, W E , W Q and W K represent weights when performing linear transformation, I t is an input formed by concatenating offset and absolute coordinates, E t represents an embedding vector formed after feature mapping, Q t represents a query vector, K t represents a key vector, Softmax represents a normalized exponential function, d represents a scale factor, represents a final formed self-attention score matrix; wherein the self-attention score matrix is an asymmetric matrix, and each element in the matrix represents the interaction between the corresponding two positions of the target;

[0028] (2) Extract high-order interaction information, including: on the basis of the self-attention score matrix A atte , higher-order interaction features are extracted by using asymmetric convolution, and the expression is,

[0029] A l = δ(conv(A l-1 , K 1×k ) + conv(A l-1 , K κ×1 ));

[0030] where K 1×κ and K κ×1 represent convolution kernels when performing convolution, δ represents an activation function, and conv represents a convolution operation; wherein L convolution layers are set, and A atte is initialized by using the previous self-attention score matrix A l-1 , the self-attention score matrix A l generated each time is used for the calculation of A L , and A L is the output result after L convolution operations;

[0031] (3) screening effective interaction information, including: screening high-order interaction information extracted in step (2) to obtain an interaction matrix A containing final interaction information adjc , the expression is,

[0032]

[0033] In the formula, represents a sigmoid function; represents an element-wise multiplication operation; sgn is a step function used in conjunction with threshold value ξ to complete the screening of interaction information; A mask is used to represent which interactions are effective and which interactions are ineffective;

[0034] (4) aggregating interaction information, including: using the interaction matrix A adjc obtained in step (3) to aggregate interaction information between each target, the expression is,

[0035] A=δ(φ(A adjc ·O t ,W G ));

[0036] In the formula, δ represents an activation function, φ(·,·) represents a linear transformation, W G represents the weight when performing linear transformation, and A represents the features of each target after aggregation;

[0037] (5) using a multilayer perceptron to predict the offset of the features A obtained in step (4), the expression is,

[0038] P offs =MLP(A);

[0039] In the formula, MLP represents a multilayer perceptron; P offs is the predicted offset, which is used for the next matching.

[0040] Further improvement of the application is that in the training step of the pre-trained interaction module,

[0041] Three consecutive images in the sample data set are taken as a training sample, the difference between the first two frames is taken as the input, and the third frame is taken as the label;

[0042] The loss function expression is,

[0043]

[0044] In the formula, P coor represents the coordinate information of the prediction result, P gt represents the coordinate information of the ground truth in the data set, and IoU represents the overlap calculation of the rectangular frame.

[0045] Further improvements of the present application are that in the pre-trained long-distance retrieval module, the step of performing the lost target re-retrieval processing comprises:

[0046] 1) Feature extraction is performed on the target trajectory in the lost state, and the expression is,

[0047]

[0048] In the formula, K κ×1 and K 1×κ represent the convolution kernel when convolution is performed, conv represents the convolution operation, pool represents the pooling operation, and δ represents the activation function; wherein, L convolution layers are set in total, and the input T lost is composed of all the target trajectories in the lost state; l-1 The self-attention score matrix T l generated each time is used for the calculation of the next step T L ;

[0049] 2) Feature extraction is performed on the target detection result that is not matched in the algorithm, and the expression is,

[0050]

[0051] In the formula, φ(·,·) represents linear transformation, and W D represents the weight when linear transformation is performed;

[0052] 3) After the features extracted in steps 1) and 2) are spliced, binary classification is performed to obtain a correlation matrix Each element represents the possibility of being connected; according to the correlation matrix, long-term correlation is completed, and the lost target is re-retrieved; wherein, all the correlations lower than the threshold in the correlation matrix are screened out, and then the result of binary matching is obtained by using the greedy algorithm; the result of binary matching is taken as a pair of long-term correlations, the trajectory during the occlusion is inferred, and the error between the detection matched and the guessed position is back-propagated forward, and the expression is,

[0053]

[0054] In the formula, t1 represents the time when the trajectory is lost, t2 represents the time when the trajectory is re-retrieved, t p represents each frame calculated in the middle, d t represents the coordinate information of the detection result matched, p t represents the coordinate information of the predicted result of the lost trajectory, represents the predicted result of the t p th frame, represents the predicted result of the t pError-corrected result.

[0055] The further improvement of the present application is that in the training step of the pre-trained long-distance retrieval module,

[0056] In all the tracks of a sample video sequence, optionally two tracks are combined as a group; based on the obtained combined sample track and a detection as input, the condition for being positive and negative samples is whether the track and the detection come from the same track, and the loss function expression is,

[0057]

[0058] In the formula, c i represents the predicted correlation score, y i represents the corresponding training label, and n represents the number of samples.

[0059] The second aspect of the present application provides a multi-target tracking system, comprising:

[0060] A data acquisition module is configured to acquire N frames of video sequences to be processed; wherein N>2;

[0061] A result acquisition module is configured to input the N frames of video sequences to be processed into a tracking framework frame by frame for multi-target tracking processing, and output multi-target tracking results.

[0062] The step of inputting the N frames of video sequences to be processed into the tracking framework frame by frame for multi-target tracking processing comprises:

[0063] Based on the first frame of video and the second frame of video, a preliminary tracking result is obtained;

[0064] For the t-th frame of video in the third frame of video to the N-th frame of video, the target detection result of the t-th frame of video is acquired; the offset of each tracking target is acquired based on the t-2-th frame of video and the t-1-th frame of video; the acquired offset is input into a pre-trained interaction module for interaction modeling between targets and prediction to acquire the offset of the t-th frame of video; the short-term association result of the t-th frame of video is acquired based on the target detection result of the t-1-th frame of video, the t-th frame of video and the offset of the t-th frame of video.

[0065] The target detection result of the target lost in the process of acquiring the short-term association result of the t-th frame of video and the target not matched in the algorithm is input into a pre-trained long-distance retrieval module for lost target retrieval processing, and the long-term association result of the t-th frame of video is output.

[0066] The short-term association result of the t-th frame of video is merged with the long-term association result to finally obtain the multi-target tracking result of the t-th frame of video.

[0067] The electronic device provided in the third aspect of the present application comprises:

[0068] at least one processor; and

[0069] a memory in communication with the at least one processor; wherein

[0070] the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the multi-target tracking method according to any one of the first aspect of the present application.

[0071] The computer-readable storage medium provided in the fourth aspect of the present application stores a computer program, and the computer program is executed by a processor to implement the multi-target tracking method according to any one of the first aspect of the present application.

[0072] Compared with the prior art, the present application has the following beneficial effects:

[0073] The present application specifically provides a multi-target tracking method based on long-term and short-term motion features, which can solve the technical problems of identity recognition errors and track interruption caused by crowded people and severe occlusion in the prior art; wherein the present application extracts "interactive motion features" to obtain short-term association results and extracts "historical trajectory motion features" to obtain long-term association results, and the combination of the two obtains the final tracking result, which can enhance the frame-by-frame identity recognition ability and cross-frame identity re-identification ability in the tracking process.

[0074] Further specifically, the present application provides a simple and effective online tracking method for multi-target tracking, which combines an interactive module and a long-distance recovery module to extract "interactive motion features" and "historical trajectory motion features" to guide short-term association and long-term association in multi-target tracking, respectively; the use of the above technical means not only makes the motion features of each target more distinctive when the crowd is crowded, thereby making the tracker have stronger identity maintenance ability, but also has the ability to recover after the target experiences severe occlusion and the track is interrupted, thereby making the multi-target tracking technology have higher accuracy when dealing with more complex scenes and improving the safety when used in the corresponding field. BRIEF DESCRIPTION OF DRAWINGS

[0075] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following briefly introduces the drawings needed to be used in the embodiments or prior art description; obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0076] Figure 1 is a flowchart of a multi-target tracking method provided by an embodiment of the present application;

[0077] Figure 2 is a flowchart of a tracking framework in an embodiment of the present application;

[0078] Figure 3 is a structural diagram of an interaction module in an embodiment of the present application;

[0079] Figure 4 is a structural diagram of a long-distance recovery module in an embodiment of the present application;

[0080] Figure 5 is a diagram of a specific case in an embodiment of the present application;

[0081] Figure 6 is a diagram of a multi-target tracking system provided by an embodiment of the present application. DETAILED DESCRIPTION

[0082] In order to make the personnel in the art better understand the present application scheme, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by the personnel in the art without creative labor should belong to the scope of protection of the present application.

[0083] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0084] The present application will be described in further detail below in conjunction with the drawings:

[0085] Referring to Figure 1 The multi-target tracking method provided by the embodiment of the present application includes the following steps:

[0086] Step 1, obtaining N frames of video sequences to be processed; wherein N≥2;

[0087] Step 2, the N frames of video sequences to be processed are transmitted frame by frame into a tracking framework for multi-target tracking processing, and multi-target tracking results are output.

[0088] The step of transmitting the frames into the tracking framework for multi-target tracking processing includes:

[0089] When N = 2, the first frame of video and the second frame of video are transmitted frame by frame into the tracking framework for target detection, and target detection results of the first frame of video and the second frame of video are obtained respectively; the target detection results include position information and size information of preset targets; similarity between targets is calculated based on the obtained position information and size information, and a tracking result is obtained by using a Hungarian algorithm;

[0090] When N > 2, the first frame of video and the second frame of video are transmitted frame by frame into the tracking framework for target detection, and target detection results of the first frame of video and the second frame of video are obtained respectively; the target detection results include position information and size information of preset targets; similarity between targets is calculated based on the obtained position information and size information, and a preliminary tracking result is obtained by using a Hungarian algorithm;

[0091] The third frame of video to the Nth frame of video are transmitted frame by frame into the tracking framework for multi-target tracking processing, and multi-target tracking results are output; for the tth frame of video among the third frame of video to the Nth frame of video, target detection is performed to obtain target detection results of the tth frame of video; offset of each tracking target is calculated based on position information and size information in the target detection results of the t-2th frame of video and the t-1th frame of video (two frames of video before the current frame of video); the offset is input into a pre-trained interaction module for interaction modeling between targets, and offset of the tth frame of video (the current frame) is predicted and output; similarity between targets is calculated based on position information, size information in the target detection results of the tth frame of video and the t-1th frame of video, and the offset of the tth frame of video, and a short-term association result of the tth frame of video is obtained by using a Hungarian algorithm (or a greedy algorithm capable of realizing bipartite graph matching); whether there is a target in a lost state and whether there is a target detection result unmatched in the Hungarian algorithm in the tth frame of video is determined; if yes, the target in the lost state and the unmatched detection result are input into a pre-trained long-distance recovery module for lost target recovery processing, and a long-term association result of the tth frame of video is output; the short-term association result of the tth frame of video is merged with the long-term association result, and finally a multi-target tracking result of the tth frame of video is obtained.

[0092] The multi-target tracking method provided in the embodiments of the application can obtain relatively accurate tracking results in the face of special situations such as crowded people or severe occlusion.

[0093] In the embodiment of the present application, the expression for calculating the offset of each tracking target based on the position information and size information in the target detection result of the t-2th frame video and the t-1th frame video (two frames of video before the current frame video) is

[0094]

[0095] In the formula, (x, y) represents the center point coordinates of the target, w represents the width of the target, h represents the height of the target, t and j represent the jth target in the tth frame, and Δ represents the change amount.

[0096] In the embodiment of the present application, the step of modeling the interaction between targets in the pre-trained interaction module includes:

[0097] 1) Generating the previous offset from the track library; specifically, in the embodiment of the present application, the tracking system sets a track library to store the tracks of each target, and when a target appears for the first time in the video, an id is assigned to it, and then the id is used in the subsequent video until the track library is removed after the target leaves the scene for more than a certain time. Specifically, the track library is recorded as Each track is recorded as The position of the target at each time point after the target is initialized as a new target is recorded.

[0098] In the embodiment of the present application, the state of the target being continuously tracked is defined as a survival state, the tracking failure is called a lost state, and the target is removed by the track library after a period of time after the loss occurs, which is called a death state.

[0099] The previous offset of each target is obtained by subtracting the positions of the previous two frames of each target, as shown below:

[0100] In the formula, (x, y) represents the center point coordinates of the target, w represents the width of the target, h represents the height of the target, t and j represent the jth target in the tth frame, and Δ represents the change amount.

[0101] The offsets of all targets are combined to be the input of the next step.

[0102] 2) Preliminary extraction of interaction information.

[0103] In the embodiment of the present application, for the tth frame video, a track offset O t is extracted based on the t-2th frame video and the t-1th frame video, and after being spliced with the absolute coordinates, it becomes I t The self-attention mechanism is used to preliminarily calculate the interaction between the tracks of the targets, and the calculation expression of the self-attention is:

[0104] Et = φ(I t , W E ),

[0105] Q t = φ(E t , W Q ),

[0106] K t = φ(E t , W K ),

[0107]

[0108] where φ(·,·) represents a linear transformation, W E , W Q and W K represent weights in the linear transformation, I t is an input formed by concatenating an offset and an absolute coordinate, E t represents an embedding vector formed after feature mapping, Q t represents a query vector, K t represents a key vector, Softmax represents a normalized exponential function, d represents a scale factor, and A atte represents a final self-attention score matrix.

[0109] In the embodiments of the present application, the self-attention score matrix A is obtained by self-attention. This is an asymmetric matrix, and each element in the matrix represents the interaction between the corresponding two positions of the target; because of the asymmetry, the interaction is initiative, and the mutual interaction between the two targets is not necessarily symmetrical, but is initiative and passive, and the present application selects self-attention which produces asymmetric interaction instead of directly using the distance between the targets as the basis for the degree of interaction, which makes the interaction more accurate and effective.

[0110] 3) Extract high-order interaction information.

[0111] On the basis of the self-attention score matrix A atte , a non-symmetric convolution is used to further extract higher-order interaction features, and the expression is:

[0112] A l = δ(conv(A l-1 , K 1×κ ) + conv(A l-1 , K κ×1 ));

[0113] where K 1×κ and K κ×1The convolution kernel when expressing convolution, δ represents the activation function, and conv represents the convolution operation; L convolution layers are configured, and the previous self-attention score matrix A is used atte Initialize A0, and generate a self-attention score matrix A each time l-1 , which is used for the calculation of the next step A l , A L is the output result after undergoing L-layer convolution operation;

[0114] Interpretation, the interaction generated by self-attention only considers the interaction between single targets, and the extraction of high-order information considers the event of multiple people.

[0115] 4) Screen the effective interaction information.

[0116] The high-order interaction information obtained in the last step, but not every interaction is effective, and it needs to be screened, and the expression is:

[0117]

[0118] A adjc =A mask ⊙A atte ,

[0119] In the formula, represents the sigmoid function, ⊙ represents the element-wise multiplication operation, sgn is a step function, and cooperates with the threshold value ξ to complete the screening of interaction information, A mask represents which interactions are effective and which interactions are ineffective, and finally obtains the interaction matrix A adjc containing the final interaction information.

[0120] 5) Aggregate interaction information.

[0121] After obtaining the interaction information, use graph convolution to aggregate the interaction information between each target, and the expression is:

[0122] A=δ(φ(A adjc ·O t ,W G )),

[0123] In the formula, δ represents the activation function, φ(·,·) represents the linear transformation, W G represents the weight when doing linear transformation, and A represents the features of each target after aggregation.

[0124] 6) Predict the offset according to the interaction information.

[0125] Finally, use a multilayer perceptron to predict the offset from the features, and the expression is:

[0126] P offs =MLP(A),

[0127] wherein MLP represents a multi-layer perceptron, and P is a predicted offset. offs is a predicted offset needed for the next matching.

[0128] Referring to Figures 2 to 4 A multi-target tracking training method based on long-term and short-term motion features according to an embodiment of the present application includes the following steps:

[0129] Step 1: Video data acquisition; exemplary, generally derived from video data by a fixed camera, a vehicle-mounted camera, or some application scenarios.

[0130] According to an embodiment of the present application, an exemplary real-time online tracking method is adopted, and a tracking result corresponding to the current input image is obtained for each input image. Real-time video collected by a data source is transmitted into the framework frame by frame, and after completing a complete tracking, the next frame is continuously input in real time.

[0131] Step 2: Target detection and positioning in the image, including: performing target detection on the current frame image, and using the obtained target detection result for the following data association.

[0132] According to an embodiment of the present application, an open-source target detection model YOLOX and an open-source model weight are obtained, the image obtained in step 1 is input into the target detection model, and the corresponding target and corresponding information in the image are obtained, and the result is recorded as Each detection result includes (x, y, w, h, s); wherein (x, y) is the center coordinate of the detection frame of the target, w is the width, h is the height, and s is the confidence of the target.

[0133] Step 3: Reasoning the offset of each trajectory in the trajectory library to the current frame, including the following steps:

[0134] 1) Generating the previous offset from the trajectory library;

[0135] 2) Preliminary extraction of interaction information;

[0136] 3) Extraction of high-order interaction information;

[0137] 4) Screening out effective interaction information;

[0138] 5) Aggregating interaction information;

[0139] 6) Predicting the offset according to the interaction information.

[0140] Step 4: Completing the short-term association between the trajectory of the target and the new frame detection result, including:

[0141] In the embodiment of the present application, the data association is divided into short-term association and long-term association, wherein the short-term association mainly processes frame-by-frame matching, and the long-term association mainly processes long-term matching across frames.

[0142] In the embodiment of the present application, the target detection result of the current frame and the offset of each trajectory predicted in the previous trajectory library are obtained based on the above steps, and a bipartite matching is performed between the two, the offset is added to the position of the last frame of the trajectory, and then the predicted position and the detection frame of the target detection are calculated by IoU, as follows:

[0143]

[0144] In the formula, A and B represent the predicted frame and the detection frame respectively, and the obtained IoU score is regarded as the similarity between the two, and in this process, the interaction information is fused into the similarity calculation; then the Hungarian algorithm is used to complete the bipartite matching, and the short-term association tracking result is obtained.

[0145] Step 5, correlation calculation between lost trajectories and unmatched detections, specifically including:

[0146] The unmatched detections in step 4 and the trajectory segments in the trajectory library which still have no matching after step 4 are matched, so that some lost trajectories can be re-identified when they reappear.

[0147] In the embodiment of the present application, after step 4, the input The input of all lost trajectories Wherein the detection contains five values (t, x, y, w, h), which respectively represent time, center point coordinates and width and height, and the lost trajectory extracts thirty frames of historical trajectories as input, and for U unmatched detections and S lost trajectories, two-by-two combination is made into UxS combinations, and correlation calculation is performed.

[0148] First, feature extraction is performed on the lost trajectory:

[0149] T l =δ(conv(T l-1 ,K κ×i )),

[0150] F traj =pool(δ(conv(T L ,K 1×κ ))),

[0151] In the formula, K κ×1 and K 1×κThe convolution kernel when representing convolution, conv represents the convolution operation, pool represents the pooling operation, delta represents the activation function, L convolution layers are set, and the input T composed of all missing tracks lost Initialize T0, and generate a self-attention score matrix T each time l-1 , which is used for the calculation of the next step T l ; T L is the output result after undergoing L-layer convolution operation.

[0152] Then, the extracted features are detected:

[0153]

[0154] In the formula, phi (·, ·) represents a linear transformation, W D represents the weight when performing linear transformation.

[0155] After completing twice feature extraction, the features of the two are spliced, and then binary classification is performed to obtain a correlation matrix Wherein each element represents the possibility of being connected.

[0156] Step 6, according to the correlation matrix, complete long-term association, specifically including:

[0157] From the correlation matrix, the effective connection is screened out, and the corresponding unmatched detection and missing track are connected to complete the re-association of the track after being occluded.

[0158] In the embodiment of the application, first, all the correlations less than 0.9 in the correlation matrix are screened out, and then a binary matching is obtained by using a greedy algorithm, and the result of the binary matching is a corresponding long-term association pair. After obtaining the association pair, the track during the occlusion needs to be inferred, and the error between the detection on the new matching and the guessed position is back propagated:

[0159]

[0160] In the formula, t1 represents the time when the track is lost, t2 represents the time when it is found back, t p represents each frame calculated in the middle, d t represents the coordinate information of the detection result on the matching, p t represents the coordinate information of the prediction result of the missing track, represents the prediction result of the t p th, represents the result after error correction of the t p th.

[0161] Compared with the previous method, in the embodiment of the application, the track after correction is removed instead of generating a new track to obtain a more accurate position during the occlusion.

[0162] Step 7, training of the interaction module and the long distance recall module.

[0163] 1) calculating the loss function of the interaction module.

[0164] 2) calculating the loss function of the long distance recall module.

[0165] 3) optimizing the network parameters.

[0166] In step 7 described above, the training steps of the interaction module and the long distance recall module include:

[0167] 1) calculating the loss function of the interaction module;

[0168] In the embodiment of the application, three consecutive images in the sample data set are taken as a training sample, the difference between the first two frames is taken as the input, and the third frame is taken as the label; the loss function expression is:

[0169]

[0170] In the formula, P coor represents the coordinate information of the prediction result, P gt represents the coordinate information of the ground truth in the data set, and IoU represents the overlap calculation of the rectangular frame.

[0171] 2) calculating the loss function of the long distance recall module.

[0172] In the embodiment of the application, all trajectories in a sample video sequence are adjusted out, and two trajectories are selected as a combination, a section of trajectory and a detection are sampled from the above as input, and the condition of positive and negative samples is whether the trajectory and the detection come from the same trajectory, and the loss function expression is:

[0173]

[0174] In the formula, c i represents the correlation score of the prediction, y i represents the corresponding training label, and n represents the number of samples.

[0175] 3) optimizing the network parameters.

[0176] The Adam optimizer is used, the interaction module is iterated 100 times, the long distance recall module is iterated 10 times, and the learning rate is slowly decayed with the increase of the number of iterations when training the interaction module.

[0177] In the technical solution of the embodiment of the present application, the interaction module uses the previous offset as input, uses an asymmetric matrix to describe the interaction between trajectories, uses a self-attention mechanism as the basis for interaction extraction, uses an asymmetric convolution to extract higher-order interactions, filters out effective interactions, that is, "interaction motion features", to guide the motion prediction of the next frame, and obtains more accurate matching results by obtaining more accurate prediction. In a crowded scene, neither the motion features nor the appearance features have high distinguishability, the interaction module introduces "interaction motion features", so that the crowded targets are more distinguishable, thereby reducing identity recognition errors and enhancing tracking performance.

[0178] In the technical solution of the embodiment of the present application, the long-distance recovery module calculates the correlation between the short-term matching unmatching detection and the lost trajectory to determine whether they should be connected together. After serious occlusion occurs, existing methods cannot handle long-distance occlusion, and can only assign a new identity to the target when it reappears after occlusion; the long-distance recovery module introduced by the present application uses "history trajectory motion features" to guide the re-identification of the occluded trajectory, which can effectively solve the occlusion problem and improve the tracking ability and robustness of the tracker.

[0179] In the specific embodiment of the present application, qualitative and quantitative comparative experimental analysis is performed on the method proposed in the embodiment of the present application and existing methods, and the effectiveness of the method is verified on two public data sets MOT17 and MOT20. Table 1 and Table 2 are quantitative experimental results of the present application; Table 1 is the experimental results of the method on the MOT17 data set, and Table 2 is the experimental results of the method on the MOT20 data set.

[0180] Table 1. Experimental results on the MOT17 data set

[0181]

[0182] Table 2. Experimental results on the MOT20 data set

[0183]

[0184]

[0185] As shown in Table 1 and Table 2, the bolded results are the optimal results; the multi-target tracking method based on long-term and short-term motion features proposed in the embodiment of the application achieves the optimal performance in multiple indicators, including IDF1, HOTA, AssA, DetA, FN, and IDs; wherein the score of MOTA is only 0.1 lower than the highest P3AFormer, but is obviously higher than it in other indicators; wherein IDs represents the number of identity transitions in the tracking process, which is reduced by 40% compared with the previous optimal method in MOT17, and is reduced by 13% compared with the most optimal method in the MOT20 dataset; wherein HOTA comprehensively reflects the tracking performance, and the method of the embodiment of the application is the highest, which shows the robustness of the tracker, and IDF1 and AssA reflect the association performance of the tracker, which is greatly improved compared with the previous method. Please refer to Figure 5 , Figure 5 The interaction condition is visualized, and the arrow reflects the influence of the target on other targets.

[0186] In summary, the embodiment of the application discloses a multi-target tracking method and system based on long-term and short-term motion features, which belongs to the technical field of computer vision; unlike other methods that only focus on each target itself and ignore the interaction between targets, which causes the identity recognition misplacement problem in crowded scenes and the occlusion problem caused by the low reliability, which cannot solve the track interruption problem. The embodiment of the application provides a simple and effective online tracking method for multi-target tracking, which combines the interaction module and the long-distance recovery module to extract "interactive motion features" and "historical trajectory motion features" to guide short-term association and long-term association during multi-target tracking, respectively. The above-mentioned method not only makes the motion features of each target more distinguishable when the crowd is crowded, so that the tracker has stronger identity maintaining ability, but also has the ability to recover after the target experiences serious occlusion and the track is interrupted, so that the multi-target tracking technology can have higher accuracy when dealing with more complex scenes, and the safety when used in the corresponding field is improved.

[0187] The following is an apparatus embodiment of the application, which can be used to execute the method embodiment of the application. For details not disclosed in the apparatus embodiment, please refer to the method embodiment of the application.

[0188] Please refer to Figure 6 , the embodiment of the application provides a multi-target tracking system, which comprises:

[0189] A data acquisition module is configured to acquire N frames of video sequences to be processed; wherein N>2;

[0190] A result acquisition module is configured to input the N frames of video sequences to be processed into a tracking framework frame by frame for multi-target tracking processing, and output multi-target tracking results;

[0191] The step of performing multi-target tracking processing on the frame-by-frame incoming tracking framework includes:

[0192] Based on the first frame video and the second frame video, a preliminary tracking result is obtained.

[0193] For the tth frame video in the third frame video to the Nth frame video, a target detection result of the tth frame video is obtained; based on the t-2th frame video and the t-1th frame video, an offset of each tracking target is obtained; the obtained offset is input into a pre-trained interaction module to model the interaction between targets and predict the offset of the tth frame video; based on the t-1th frame video target detection result, the tth frame video and the offset of the tth frame video, a short-term association result of the tth frame video is obtained.

[0194] In the process of obtaining the short-term association result of the tth frame video, the target whose state is lost and the target detection result that is not matched in the algorithm are input into a pre-trained long-distance recovery module for lost target recovery processing, and a long-term association result of the tth frame video is output.

[0195] The short-term association result and the long-term association result of the tth frame video are merged, and finally a multi-target tracking result of the tth frame video is obtained.

[0196] As a further preferred, the system of the embodiment of the application can specifically include:

[0197] The data acquisition module acquires a video to be processed and inputs the video into the tracking framework in real time.

[0198] The short-term association module inputs the trajectories in the trajectory library into a pre-trained interaction module and outputs a prediction result; the short-term association is completed in combination with the current frame detection result.

[0199] The long-term association module inputs the lost trajectories and unmatched detection in the trajectory library into a pre-trained long-distance recovery module and outputs a long-term association result.

[0200] The results of the final short-term association and the long-term association are merged to obtain the final tracking result.

[0201] The interaction module includes:

[0202] The interaction feature extraction network is used for inputting the trajectory offset and outputting the interaction information between targets.

[0203] The prediction network is used for inputting the interaction information and outputting the predicted value of the next frame offset of the trajectory.

[0204] In still another embodiment of the present application, a computer device is provided, which comprises a processor and a memory for storing a computer program comprising program instructions, and the processor is configured to execute the program instructions stored in the computer storage medium. The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, etc., which are the computing core and control core of the terminal and are suitable for implementing one or more instructions, and are specifically suitable for loading and executing one or more instructions in the computer storage medium to implement a corresponding method flow or a corresponding function; the processor in the embodiments of the present application can be used for the operation of the multi-target tracking method.

[0205] In still another embodiment of the present application, the present application further provides a storage medium, specifically a computer readable storage medium (Memory), which is a memory device in the computer device and is used for storing programs and data. It can be understood that the computer readable storage medium herein can include the built-in storage medium in the computer device, and of course can also include the extended storage medium supported by the computer device. The computer readable storage medium provides a storage space, which stores the operating system of the terminal. Moreover, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer readable storage medium herein can be a high-speed RAM memory, or a non-volatile memory such as at least one disk memory. One or more instructions stored in the computer readable storage medium can be loaded and executed by the processor to implement the corresponding steps of the multi-target tracking method in the above embodiments.

[0206] Those skilled in the art will appreciate that embodiments of the application can be devised for a method, a system, or a computer program product. Accordingly, the present application can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0207] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0208] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0209] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0210] Finally, it should be noted that the above-mentioned embodiments are merely intended for describing the technical solutions of the present application, but not for limiting it. Although the present application has been described in detail with reference to the above-mentioned embodiments, those skilled in the art should understand that the technical solutions of the present application can still be modified or equivalent replaced without departing from the spirit and scope of the present application, and any modification or equivalent replacement should be covered in the protection scope of the claims of the present application.

Claims

1. A multi-target tracking method characterized by, The method comprises the following steps: Step 1, obtaining an N-frame video sequence to be processed; wherein N>2; Step 2, transmitting the N-frame video sequence to be processed into a tracking framework frame by frame for multi-target tracking processing, and outputting a multi-target tracking result; Wherein the step of transmitting the N-frame video sequence to be processed into the tracking framework frame by frame for multi-target tracking processing comprises: Based on the first frame video and the second frame video, obtaining a preliminary tracking result; For the tth frame video in the third frame video to the Nth frame video, obtaining a target detection result of the tth frame video; based on the t-2th frame video and the t-1th frame video, obtaining an offset of each tracking target; inputting the obtained offset into a pre-trained interaction module for interaction modeling between targets and predicting an offset of the tth frame video; based on the t-1th frame video target detection result, the tth frame video and the offset of the tth frame video, obtaining a short-term association result of the tth frame video; In the process of obtaining the short-term association result of the tth frame video, inputting the target detection result of the target in the lost state and the target not matched in the algorithm into a pre-trained long-distance recovery module for lost target recovery processing, and outputting a long-term association result of the tth frame video; Merging the short-term association result of the tth frame video with the long-term association result to finally obtain a multi-target tracking result of the tth frame video; In the pre-trained interaction module, the step of modeling the interaction between targets and predicting and outputting the offset comprises: (1) preliminarily extracting interaction information, including: for the tth frame video, based on the offset of each tracking target obtained from the t-2th frame video and the t-1th frame video, concatenating the offset with the absolute coordinates to become, using a self-attention mechanism to preliminarily calculate the interaction between the trajectories of the targets, and obtaining a self-attention score matrix; (2) extracting high-order interaction information, including: based on the self-attention score matrix, using an asymmetric convolution to extract higher-order interaction features; (3) screening effective interaction information, including: screening the higher-order interaction features extracted in step (2) to obtain an interaction matrix containing the final interaction information; (4) aggregating interaction information, including: based on the interaction matrix obtained in step (3), using a graph convolution to aggregate the interaction information between the targets; (5) using a multi-layer perception to predict the offset from the features obtained in step (4).

2. The multi-target tracking method of claim 1, wherein, The step of transmitting the N-frame video sequence to be processed into the tracking framework frame by frame for multi-target tracking processing comprises: Transmitting the first frame video and the second frame video into the tracking framework frame by frame for target detection, and obtaining a target detection result of the first frame video and a target detection result of the second frame video frame by frame, respectively; wherein the target detection result comprises position information and size information of a preset target; calculating the similarity between the targets based on the obtained position information and size information, and using a Hungarian algorithm to obtain a preliminary tracking result; The third frame of video to the Nth frame of video is input into the tracking framework frame by frame, and multi-target tracking processing is performed to output multi-target tracking results; wherein, for the tth frame of video in the third frame of video to the Nth frame of video, target detection is performed to obtain target detection results of the tth frame of video; based on position information and size information in the target detection results of the t-2th frame of video and the t-1th frame of video, offset of each tracking target is calculated and obtained; the calculated offset is input into the pre-trained interaction module to model the interaction between targets, and the offset of the tth frame of video is predicted and output; based on the position information, the size information in the target detection results of the tth frame of video and the t-1th frame of video, and the offset of the tth frame of video, the similarity between targets is calculated, and the Hungarian algorithm or the greedy algorithm is used to obtain the short-term association result of the tth frame of video; it is judged whether the tth frame of video has a target in a lost state and whether there is a target detection result that is not matched in the algorithm; if yes, the target in the lost state and the unmatched detection result are input into the pre-trained long-distance recovery module for lost target recovery processing, and the long-term association result of the tth frame of video is output; the short-term association result of the tth frame of video is merged with the long-term association result, and finally the multi-target tracking result of the tth frame of video is obtained; if not, the short-term association result of the tth frame of video is taken as the final multi-target tracking result of the tth frame of video.

3. The multi-target tracking method of claim 2, wherein, The expression for calculating and obtaining the offset of each tracking target based on the position information and size information in the target detection results of the t-2th frame of video and the t-1th frame of video is, ； In the formula, (x, y) represents the coordinate of the center point of the target, w represents the width of the target, h represents the height of the target, t and j represent the jth target in the tth frame, represents the change amount.

4. The multi-target tracking method of claim 1, wherein, In the pre-trained interaction module, the step of modeling the interaction between targets and predicting and outputting the offset, In step (1), the calculation expression of self-attention is: ； In the formula, represents a linear transformation, and respectively represent weights when doing linear transformation, is an input formed by splicing the offset and the absolute coordinate, represents an embedding vector formed after feature mapping, represents a query vector, represents a key vector, represents a normalized exponential function, and d represents a proportion factor, represents a finally formed self-attention score matrix; wherein the self-attention score matrix is an asymmetric matrix, and each element in the matrix represents an interaction between targets at corresponding two positions; In step (2), the expression for extracting higher-order interaction features using asymmetric convolution is: ； In the formula, and indicates the convolution kernel when convolution, indicates the activation function, and conv indicates the convolution operation; wherein, L convolution layers are set in common, and the previous self-attention score matrix Initialization , the self-attention score matrix generated each time is used for the calculation of the next step , is the output result after L layer convolution operation; In step (3), the interaction matrix containing the final interaction information is expressed as, ； wherein represents a sigmoid function; represents an element-wise multiplication operation; is a step function used in conjunction with a threshold to complete the filtering of the interaction information; to represent which interactions are valid and which are not. In step (4), the expression for using graph convolution to aggregate interaction information between targets is: ； In the formula, represents an activation function, represents a linear transformation, represents a weight when the linear transformation is performed, represents a feature of each target after aggregation; In step (5), the expression for predicting the offset from the features obtained in step (4) is: ； In the formula, represents a multi-layer perceptron; is a predicted offset for the next matching step.

5. The multi-target tracking method of claim 1, wherein, In the training step of the pre-trained interaction module, Three consecutive images in the sample data set are taken as a training sample, the difference between the first two frames is taken as the input, and the third frame is taken as the label; The loss function expression is: ； In the formula, coordinate information representing a prediction result, coordinate information representing a ground truth in the data set, and IoU represents an overlap degree calculation of a rectangular frame.

6. The multi-target tracking method of claim 1, wherein, In the pre-trained long-distance recovery module, the step of performing lost target recovery processing includes: 1) Feature extraction is performed on the target trajectory in the lost state, and the expression is: ； In the formula, and The convolution kernel when the convolution is represented, and the convolution operation is represented, and the pooling operation is represented, The activation function is represented; wherein, L convolution layers are set in common, and the input is composed of the target track of all lost states Initialization , the self-attention score matrix generated each time , for the calculation of the next step ; It is the result of undergoing L-layer convolution operation output; 2) Feature extraction is performed on the target detection result that is not matched in the algorithm, and the expression is: ； wherein represents a linear transformation, denotes the weight when doing the linear transformation; 3) splice the features extracted in step 1) and step 2), and then perform binary classification to obtain a correlation matrix where each element represents the possibility of being connected; according to the correlation matrix, long-term association is completed to realize the re-finding of the lost target; wherein, all the correlations lower than the threshold value in the correlation matrix are excluded, and then a result of two-part matching is obtained by using a greedy algorithm; the result of two-part matching is taken as a long-term association pair, and the trajectory during the occlusion is inferred, and the error between the detection on the new matching and the guessed position is back-propagated, and the expression is ； In the formula, indicates the time of trajectory loss, indicates the time of recovery, indicates each frame of intermediate calculation, indicates the coordinate information of the detection result of matching, indicates the coordinate information of the prediction result of lost trajectory, indicates the prediction result of the first indicates the prediction result of the second error correction result.​ 7. The multi-target tracking method of claim 1, wherein, In the training step of the pre-trained long-distance recovery module, In all trajectories of a sample video sequence, two trajectories are selected as a combination; based on the obtained combination, a piece of trajectory and a detection are sampled as input, and the condition for being positive and negative samples is whether the trajectory and the detection come from the same trajectory, and the loss function expression is ； wherein denotes the predicted correlation score, denotes the corresponding training label, n denotes the number of samples.

8. A multi-target tracking system, characterized by It comprises: A data acquisition module is configured to acquire N frames of video sequences to be processed; wherein, N>2. An result obtaining module is configured to input the N frames of video sequences to be processed into a tracking framework frame by frame for multi-target tracking processing, and output multi-target tracking results. The step of inputting the N frames of video sequences to be processed into the tracking framework frame by frame for multi-target tracking processing comprises: obtaining a preliminary tracking result based on the first frame of video and the second frame of video; For the t-th frame of video among the third frame of video to the N-th frame of video, obtaining a target detection result of the t-th frame of video; obtaining an offset of each tracking target based on the t-2-th frame of video and the t-1-th frame of video; inputting the obtained offset into a pre-trained interaction module for interaction modeling between targets and predicting an offset of the t-th frame of video; obtaining a short-term association result of the t-th frame of video based on the target detection result of the t-1-th frame of video, the t-th frame of video and the offset of the t-th frame of video; inputting a target detection result of a target in a lost state in the process of obtaining the short-term association result of the t-th frame of video and a target not matched in the algorithm into a pre-trained long-distance recovery module for lost target recovery processing, and outputting a long-term association result of the t-th frame of video; merging the short-term association result of the t-th frame of video and the long-term association result to finally obtain a multi-target tracking result of the t-th frame of video; In the pre-trained interaction module, the step of modeling the interaction between targets and predicting and outputting the offset comprises: (1) preliminarily extracting interaction information, including: for the t-th frame of video, based on the offset of each tracking target obtained based on the t-2-th frame of video and the t-1-th frame of video, concatenating the offset and the absolute coordinates to become, using a self-attention mechanism to preliminarily calculate the interaction between the trajectories of the targets to obtain a self-attention score matrix; (2) extracting high-order interaction information, including: based on the self-attention score matrix, using an asymmetric convolution to extract higher-order interaction features; (3) screening effective interaction information, including: screening the higher-order interaction features extracted in step (2) to obtain an interaction matrix containing final interaction information; (4) aggregating interaction information, including: based on the interaction matrix obtained in step (3), using a graph convolution to aggregate the interaction information between the targets; (5) using a multi-layer perception to predict the offset from the features obtained in step (4).

9. An electronic device, comprising: comprise: at least one processor; and a memory in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the multi-target tracking method according to any one of claims 1 to 7.

10. A computer readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to implement the multi-target tracking method according to any one of claims 1 to 7. The computer program is executed by the processor to implement the multi-target tracking method according to any one of claims 1 to 7.

Images