Multi-target tracking method, electronic equipment and storage medium

Abstract

The invention discloses a multi-target tracking method, electronic equipment and a storage medium, and the method comprises the steps: obtaining a current tracking image and a target text which is used for describing the information of a to-be-tracked target; performing target detection on the current tracking image based on the target text by using a multi-target tracking model, and determining a plurality of current targets and target detection information corresponding to the current targets; and for each current target, selecting a historical target matched with the current target for association by using the target detection information of the current target and the historical detection information corresponding to the historical target. According to the scheme, the target tracking efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a multi-target tracking method, electronic device, and storage medium. Background Art

[0002] Currently, there are few existing open-vocabulary multi-object tracking methods. The main research approaches rely on heavy-duty image and text encoders, associating text with images and using video stream data to learn the distinguishing features of objects, thereby developing open-vocabulary multi-object trackers. However, these studies generally suffer from heavy computational burdens and difficulties in deployment and application. Furthermore, video data acquisition is also difficult. Summary of the Invention

[0003] This application at least provides a multi-target tracking method, electronic device, and storage medium, which can improve the efficiency of target tracking.

[0004] The first aspect of the present application provides a multi-target tracking method, which includes: obtaining a current tracking image and a target text, where the target text is used to describe information about the target to be tracked; using a multi-target tracking model to perform target detection on the current tracking image based on the target text, and determining several current targets and target detection information corresponding to the current targets; for each current target, using the target detection information of the current target and the historical detection information corresponding to the historical targets, select the historical target that matches the current target for association.

[0005] Among them, the target detection information also includes at least one of the position information, type information and appearance feature information of the current target; the multi-target tracking model is used to perform target detection on the current tracking image based on the target text, and several current targets and target detection information corresponding to the current targets are determined, including: feature extraction on the current tracking image to obtain a multi-scale feature map; feature extraction on the target text to obtain text features; fusing the multi-scale feature map and the text features to obtain a first multi-scale fused feature map; predicting the first multi-scale fused feature map to obtain at least one of the position information and category information of the current target; and / or performing deformable fusion processing on the multi-scale feature map to obtain appearance feature information of the current target.

[0006] Among them, the multi-scale feature map is deformably fused to obtain the appearance feature information of the current target, including: fusing the multi-scale feature map to obtain a second multi-scale fused feature map; using variable convolution to fuse the second multi-scale fused feature map to obtain a third multi-scale fused feature map; and performing feature extraction on the third multi-scale fused feature map to obtain appearance feature information.

[0007] wherein the multi-scale feature map includes a first resolution feature map, a second resolution feature map and a third resolution feature map, and the resolutions of the first resolution feature map, the second resolution feature map and the third resolution feature map decrease in sequence; fusing the multi-scale feature map to obtain a second multi-scale fused feature map, including: upsampling the third resolution feature map to obtain a first upsampling feature map; fusing the first upsampling feature map and the second resolution feature map to obtain a first sampling fusion feature map; upsampling the first sampling fusion feature map to obtain a second upsampling feature map; fusing the second upsampling feature map and the first resolution feature map to obtain a second sampling fusion feature map; downsampling the second sampling fusion feature map to obtain a first downsampling feature map; fusing the first downsampling feature map and the first sampling fusion feature map to obtain a third sampling fusion feature map; downsampling the third sampling fusion feature map to obtain a second downsampling feature map; fusing the second downsampling feature map and the third resolution feature map to obtain a fourth sampling fusion feature map; and fusing the second sampling fusion feature map, the third sampling fusion feature map and the fourth sampling fusion feature map to obtain a second multi-scale fused feature map.

[0008] The step of performing feature extraction on the third multi-scale fusion feature map to obtain appearance feature information includes: performing region recognition on the third multi-scale fusion feature map to obtain a feature region of interest; and performing quasi-dense feature extraction on the feature region of interest to obtain appearance feature information.

[0009] Among them, feature extraction is performed on the current tracking image to obtain a multi-scale feature map, including: performing feature extraction processing on the current tracking image for a preset number of stages to obtain several feature maps; performing pooling processing on the feature map obtained in the final stage to obtain a pooled feature map; combining at least one feature map and the pooled feature map to obtain a multi-scale feature map; and / or, the position information represents the position of the current bounding box corresponding to the current target; predicting the first multi-scale fusion feature map to obtain at least one of the position information and category information of the current target, including at least one of the following steps: performing bounding box regression processing on the first multi-scale fusion feature map to obtain the position of the current bounding box; performing text normalization processing on the first multi-scale fusion feature map to obtain the category information of the current target.

[0010] The multi-scale feature map includes a first resolution feature map, a second resolution feature map and a third resolution feature map, and the resolutions of the first resolution feature map, the second resolution feature map and the third resolution feature map are sequentially reduced; the multi-scale feature map and the text feature are fused to obtain a first multi-scale fused feature map, including: upsampling the third resolution feature map to obtain a third up-sampled feature map; fusing the third up-sampled feature map, the second resolution feature map and the text feature to obtain a first fused feature map; upsampling the first fused feature map to obtain a fourth up-sampled feature map; fusing the fourth up-sampled feature map The sampling feature map, the first resolution feature map and the text feature are fused to obtain a second fused feature map; the second fused feature map is downsampled to obtain a third downsampled feature map; the third downsampled feature map, the first fused feature map and the text feature are fused to obtain a third fused feature map; the third fused feature map is downsampled to obtain a fourth downsampled feature map; the fourth downsampled feature map, the third resolution feature map and the text feature are fused to obtain a fourth fused feature map; the second fused feature map, the third fused feature map and the fourth fused feature map are fused to obtain a first multi-scale fused feature map.

[0011] Among them, the multi-target tracking model includes a backbone network module, a text encoding module, a vision-language aggregation module, a prediction module and a matching tracking module; among them, the backbone network module is used to extract features of the current tracking image, the text encoding module is used to extract features of the target text, the vision-language aggregation module is used to fuse multi-scale feature maps and text features, the prediction module is used to predict the first multi-scale fused feature map, and the matching tracking module is used to perform deformable fusion processing on the multi-scale feature map.

[0012] Among them, the target detection information includes the position information and current appearance feature information of the current target, and the historical detection information includes historical position information and historical appearance feature information; the target detection information of the current target and the historical detection information corresponding to the historical target are used to associate the selected historical targets that match the current target, including: using the current appearance feature information to perform similarity calculations with each historical appearance feature information to obtain an appearance similarity score; using the position information and the historical position information to perform distance calculations to obtain a position score; constructing a cost matrix based on each appearance similarity score and each position score; based on the cost matrix, determining the historical target that matches the current target; and associating the matched current target with the historical target.

[0013] Among them, the current appearance feature information is used to calculate the similarity with each historical appearance feature information respectively to obtain an appearance similarity score, including: using the bidirectional similarity of the current appearance feature information and the historical appearance feature information to fuse to obtain a first similarity score; and based on the difference between the cosine similarity between the current appearance feature information and the historical appearance feature information and a preset value, obtaining a second similarity score; obtaining a central tendency statistic of the first similarity score and the second similarity score to obtain an appearance similarity score; and / or, the position information represents the position of the current bounding box corresponding to the current target, and the historical position information represents the historical bounding box corresponding to the historical target The method comprises the steps of: determining the position of a frame; performing distance calculation using the position information and the historical position information to obtain a position score, including: taking the historical bounding box closest to the current bounding box as the nearest bounding box; obtaining an inter-frame distance value based on the distance between a preset position point in the nearest bounding box and a preset position point in the current bounding box; in response to the inter-frame distance value being less than a target distance threshold, obtaining a position score based on a difference between the inter-frame distance value and a target distance threshold, wherein the target distance threshold is obtained based on a scaling factor and a preset distance threshold, and the scaling factor is related to the frame difference data; or, in response to the inter-frame distance value being greater than or equal to the target distance threshold, the position score is a preset value.

[0014] Among them, based on each appearance similarity score and each position score, a cost matrix is constructed, including: fusing the appearance similarity score and the position score corresponding to the same historical target to obtain the fusion score corresponding to the historical target; using the fusion scores corresponding to different historical targets as different elements in the cost matrix; and / or, based on the cost matrix, determining the historical target that matches the current target, including: selecting the largest element in the cost matrix as the target element; in response to the numerical value corresponding to the target element being greater than a preset matching value, determining that the historical target corresponding to the target element matches the current target.

[0015] Among them, the target detection information also includes the location information of the current target and the location confidence corresponding to the location information; for each current target, using the target detection information of the current target and the historical detection information corresponding to the historical target, before associating the selected historical target that matches the current target, it includes: deleting the current target whose confidence does not meet the preset confidence requirements; and / or, among the currently remaining current targets, taking the current target with the highest confidence as the first target, and the remaining current targets as the second targets; based on the location information, obtaining the first intersection-and-union ratio between the current target and each second target respectively; deleting the second target whose first intersection-and-union ratio is greater than the preset screening value; repeating the current target with the highest confidence as the first target and the remaining current targets as the second targets and its subsequent steps among the currently remaining current targets, until only one current target is left as the first target or all current targets are taken as the first target.

[0016] The training method of the multi-target tracking model includes: performing data augmentation on a sample image to obtain a plurality of sample augmented images, wherein the sample augmented image has at least one sample target, and the sample augmented image is annotated with actual position information and target sample information of the sample target, and the target sample information is used to describe the appearance characteristics of the sample target in the sample augmented image; for each sample augmented image, using the multi-target tracking model to perform feature extraction on the sample augmented image to obtain a predicted multi-scale sample feature map; using the multi-target tracking model to perform feature extraction on the sample text to obtain predicted text features, wherein the sample text is annotated with association information between the sample text and the sample target; fusing the predicted multi-scale sample feature map and the predicted text features to obtain predicted fusion features; performing prediction based on the predicted fusion features to obtain predicted position information and predicted target text similarity, wherein the predicted target text similarity represents the association between the target corresponding to the predicted position information and the sample text; determining predicted appearance features based on the predicted multi-scale sample feature map; and optimizing the network parameters of the multi-target tracking model based on the difference between the predicted position information and the actual position information, the difference between the predicted target text similarity and the association information of the sample text, and the difference between the predicted appearance features and the target sample information.

[0017] Among them, the step of obtaining the difference between the predicted appearance feature and the target sample information includes: obtaining a second intersection-over-union ratio (IoU) of the area corresponding to the predicted appearance feature and the area corresponding to the target sample information; in response to the second IoU being greater than a first threshold, taking the sample target corresponding to the target sample information as a positive sample of the sample target corresponding to the predicted appearance feature; in response to the second IoU being less than a second threshold, taking the sample target corresponding to the target sample information as a negative sample of the sample target corresponding to the predicted appearance feature; based on the positive sample data and negative sample data corresponding to the predicted appearance feature, obtaining a sample instance loss value, wherein the sample instance loss value represents the difference between the predicted appearance feature and the target sample information.

[0018] The method further includes any of the following: updating the target detection information of the current target for which a matching historical target is found into a trajectory library; creating a new trajectory for the current target for which a matching historical target is not found, and adding the new trajectory to the trajectory library; treating the current target for which a matching historical target is not found as a pending target, and temporarily storing the target detection information of the pending target in the trajectory library; and deleting the target detection information of the pending target temporarily stored in the trajectory library in response to the pending target still not finding a matching historical target within a preset time.

[0019] A second aspect of the present application provides an electronic device, comprising a memory and a processor coupled to each other, wherein the processor is configured to execute program instructions stored in the memory to implement the multi-target tracking method in the first aspect.

[0020] A third aspect of the present application provides a computer-readable storage medium having program instructions stored thereon, which, when executed by a processor, implements the multi-target tracking method in the first aspect.

[0021] The above scheme obtains the current tracking image and target text, performs target detection on the current tracking image based on the target text by using a multi-target tracking model, determines several current targets and target detection information corresponding to the current targets, and for each current target, uses the target detection information of the current target and the historical detection information corresponding to the historical targets to associate the selected historical targets that match the current target. Through the prompt of the target text and combined with the multi-target tracking model to perform multi-target detection, open vocabulary multi-target tracking can be realized, avoiding the need for real-time re-encoding of prompt words, reducing computational overhead, and improving the efficiency of open vocabulary multi-target tracking.

[0022] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and, together with the specification, are used to illustrate the technical solutions of the present application.

[0024] Figure 1 This is a flowchart of an embodiment of the multi-target tracking method of the present application;

[0025] Figure 2 This is a schematic diagram of the framework of an embodiment of the multi-target tracking model of the present application;

[0026] Figure 3 This is a schematic diagram of the structure of an embodiment of the vision-language aggregation module of the present application;

[0027] Figure 4 This is a schematic diagram of the structure of an embodiment of the cross-stage module of the present application;

[0028] Figure 5 This is a schematic diagram of the structure of an embodiment of the matching tracking module of the present application;

[0029] Figure 6 This is a structural diagram of an embodiment of a scale feature pyramid module of the present application;

[0030] Figure 7 This is a flowchart of an embodiment of a training method for a multi-target tracking model of the present application;

[0031] Figure 8 This is a flowchart of another embodiment of the multi-target tracking method of the present application;

[0032] Figure 9This is a schematic diagram of the framework of an embodiment of the electronic device of the present application;

[0033] Figure 10 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0034] The following describes the embodiments of the present application in detail with reference to the accompanying drawings.

[0035] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0036] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the previous and next associated objects are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0037] See also Figure 1 , Figure 1 It is a flowchart of an embodiment of the multi-target tracking method of the present application.

[0038] Specifically, the following steps may be included:

[0039] Step S110: obtaining the current tracking image and target text, where the target text is used to describe information about the target to be tracked.

[0040] This solution is primarily targeted at dynamic visual analysis applications such as intelligent security monitoring, autonomous driving perception, and human-computer interaction systems. It addresses key challenges such as target occlusion, identity transitions, and cross-modal feature matching during continuous multi-target tracking in complex scenarios. Furthermore, through a "cue-then-detect" multi-target tracking model, it enables open-vocabulary multi-target tracking, eliminating the need for real-time re-encoding of cue words, reducing computational overhead, and improving the efficiency of open-vocabulary multi-target tracking.

[0041] This solution innovatively integrates the spatiotemporal features of image sequences with semantic text descriptions of targets. By constructing a collaborative analysis framework for heterogeneous data, it achieves cross-modal target identity consistency. This can effectively improve trajectory continuity and identity recognition accuracy in dense target scenarios, providing a robust multi-target tracking solution for scenarios such as smart city management, traffic flow analysis, and industrial automation detection.

[0042] In some embodiments, the current tracking image may be obtained through a shooting device or may be an image obtained from a video stream.

[0043] Step S120: Utilizing the multi-target tracking model to perform target detection on the current tracking image based on the target text, and determining a number of current targets and target detection information corresponding to the current targets.

[0044] See also Figure 2 , Figure 2 This is a schematic diagram of the framework of an embodiment of the multi-target tracking model of the present application. The multi-target tracking model 200 includes a backbone network module 210, a text encoding module 220, a visual-language aggregation module 230, a prediction module 240, and a matching tracking module 250. Among them, the backbone network module 210 is used to extract features from the current tracking image, the text encoding module 220 is used to extract features from the target text, the visual-language aggregation module 230 is used to fuse the multi-scale feature map and the text features, the prediction module 240 is used to predict the first multi-scale fused feature map, and the matching tracking module 250 is used to perform deformable fusion processing on the multi-scale feature map. Among them, the target detection information also includes at least one of the current target's position information, type information, and appearance feature information.

[0045] In some embodiments, the multi-target tracking model 200 is a pre-trained model. It uses the target text input by the operator to perform target detection on the corresponding targets in the current tracking image to determine the current target in the current tracking image and the target detection information corresponding to the current target. For example, if the target text is "car," the multi-target tracking model 200 performs target detection on the current tracking image to determine the vehicle in the current tracking image and extracts the corresponding target detection information. The target detection information includes the vehicle's location information in the current tracking image, the vehicle type (e.g., sedan, SUV, truck, bus, etc.), and the vehicle's appearance characteristics (e.g., vehicle color, vehicle brand, license plate, etc.). For another example, if the target text is "human," the multi-target tracking model 200 performs target detection on the current tracking image to determine the presence of a pedestrian in the current tracking image and extracts the corresponding target detection information. The target detection information includes the pedestrian's location information in the current tracking image, gender (male or female), and appearance characteristics (e.g., long or short hair, clothing color, etc.). Furthermore, descriptive terms such as "white car" or "adult male" may be added to the target text. Therefore, no specific limitation is made here regarding the target text.

[0046] In a specific embodiment, the multi-target tracking model 200 is used to perform target detection on the current tracking image, as shown in steps S121 to S125 .

[0047] Step S121: extract features from the current tracking image to obtain a multi-scale feature map.

[0048] In some embodiments, the multi-target tracking model 200 can be used to perform a preset number of feature extraction stages on the current tracking image to obtain a plurality of feature maps. The feature maps obtained in the final stage are then pooled to obtain a pooled feature map. The at least one feature map is combined with the pooled feature map to obtain a multi-scale feature map.

[0049] Specifically, the backbone network module 210 can be used to perform feature extraction on the current tracking image to obtain a multi-scale feature map. In this embodiment, the backbone network module 210 can be the backbone network of YOLOv8-L, and its core modules are the C2f module and the SPPF module. The C2f module is used to perform feature extraction processing on the current tracking image for a preset number of stages to obtain several feature maps, wherein the C2f module includes 2 1x1 convolution layers and 2 3x3 convolution layers, thereby performing feature extraction processing on the current tracking image. The SPPF module is used to perform pooling processing on the feature map obtained in the final stage to obtain a pooled feature map, wherein the SPPF module divides the original current tracking image into multiple non-overlapping small areas, and selects the maximum value in each small area as the sampling point to reduce the width and height of the feature map. In addition, the preset number of stages in this embodiment can be set to 5. For the structural parameters of the YOLOv8-L backbone network, please refer to Table 1:

[0050] Table 1

[0051]

[0052] In stage 1 of the backbone network module 210, the input 640*640 current tracking image is smoothed using a Gaussian function, where the stride is 2 and the Gaussian kernel size KSize is 3. The smoothed image is then convolved using Conv. In stages Stage 2-Stage 4, after the smoothed image is convolved using Conv, it is then convolved using the C2f module. In stage Stage 5, after the smoothed image is convolved using Conv, it is then convolved using the C2f module, and the image processed by the C2f module is input into the SPPF module for pooling to obtain a pooled feature map.

[0053] Finally, the feature maps F1 and F2 output by stages Stage3 and Stage4 in the backbone network module 210 are combined with the pooled feature map F3 output by stage Stage5 to obtain a multi-scale feature map.

[0054] It is understandable that, in addition to the above-mentioned backbone network YOLOv8-L, the backbone network module 210 may also adopt a convolutional neural network, a Swin Transformer-Base backbone network, a CSPDarkNet-XX backbone network, etc., without specific limitation here.

[0055] Step S122: extracting features from the target text to obtain text features.

[0056] In some embodiments, the text encoding module 220 can be used to extract features from the target text to obtain text features. The text encoding module 220 uses a Transformer text encoder pre-trained by CLIP (Contrastive Language-Image Pretraining) to extract corresponding text features. Specifically, the text encoder is used to extract features from a given target text T, and the corresponding text features W obtained can be expressed as:

[0057] W=TextEncoder(T)∈R C∪D (1).

[0058] Where C is the number of nouns and D is the embedding dimension.

[0059] Step S123: Fusing the multi-scale feature map and the text feature to obtain a first multi-scale fused feature map.

[0060] In some embodiments, the multi-scale feature map includes a first resolution feature map, a second resolution feature map, and a third resolution feature map, and the resolutions of the first resolution feature map, the second resolution feature map, and the third resolution feature map are sequentially reduced. The multi-scale feature map and the text feature can be fused using the visual-language aggregation module 230 to obtain a first multi-scale fused feature map. In this embodiment, the visual-language aggregation module 230 follows the path aggregation network concept. Therefore, a multi-scale feature pyramid structure can be established in the visual-language aggregation module 230. The specific structure is as follows: Figure 3 At the same time, the vision-language aggregation module 230 also includes a cross-stage module (T-guided Cross Stage PartialLayer, T-CSPLayer), which integrates text features and can enhance the feature interaction between images and text, thereby improving open vocabulary capabilities.

[0061] Please refer to Figure 3, the steps of fusing the multi-scale feature map and the text feature using the visual-language aggregation module 230 to obtain the first multi-scale fused feature map are as follows: inputting the first resolution feature map F1, the second resolution feature map F2, the third resolution feature map F3 and the text feature into the visual-language aggregation module 230, the visual-language aggregation module 230 first upsamples the third resolution feature map to obtain a third upsampled feature map; using the first cross-stage module 231 to fuse the third upsampled feature map, the second resolution feature map and the text feature to obtain a first fused feature map; upsample the first fused feature map to obtain a fourth upsampled feature map; using the second cross-stage module 232 to fuse the fourth upsampled feature map The first downsampled feature map, the first resolution feature map and the text feature are fused to obtain a second fused feature map P1; the second fused feature map is downsampled to obtain a third downsampled feature map; the third downsampled feature map, the first fused feature map and the text feature are fused using the third cross-stage module 233 to obtain a third fused feature map P2; the third fused feature map is downsampled to obtain a fourth downsampled feature map; the fourth cross-stage module 234 is used to fuse the fourth downsampled feature map, the third resolution feature map and the text feature to obtain a fourth fused feature map P3; the second fused feature map P1, the third fused feature map P2 and the fourth fused feature map are fused P3 to obtain a first multi-scale fused feature map.

[0062] Please refer to Figure 4 , for text features W and multi-scale feature maps F l ∈R H×W×D (l∈{1, 2, 3}, where H represents the length of the feature map and W represents the width of the feature map. After several convolutional layers, the T-CSPLayer module uses Max-Sigmoid attention, as shown in formula (2), to aggregate text features into image features:

[0063]

[0064] Among them, X l It is the feature map of the lth multi-scale feature map after being processed by the T-CSPLayer module.

[0065] Furthermore, the updated X′ l Concatenate with other cross-stage features, extract features through a 1*1 convolution layer, and output a fused feature map.

[0066] It is understandable that in addition to using the vision-language aggregation module 230, the multi-target tracking model 200 can also use cross-modal projection methods, late fusion methods, etc. to fuse multi-scale feature maps and text features, and no specific limitations are made here.

[0067] Step S124: predicting the first multi-scale fusion feature map to obtain at least one of the position information and category information of the current target.

[0068] In some embodiments, the position information represents the position of the current bounding box corresponding to the current target. The prediction module 240 can be used to predict the first multi-scale fusion feature map to obtain at least one of the position information and category information of the current target. Specifically, the prediction module 240 includes a text comparison branch and a bounding box regression branch. Each branch of the prediction module 240 includes two 3*3 convolutional layers for feature extraction. The bounding box regression branch is used to perform bounding box regression on the first multi-scale fusion feature map to obtain the position of the current bounding box. Where K represents the number of objects. The bounding box regression branch predicts the probability distribution of the bounding box coordinates. In the bounding box regression branch, reg_max is the parameter used to calculate the integral target bounding box coordinates and is set to 16.

[0069] The text comparison branch is used to perform text normalization on the first multi-scale fusion feature map to obtain the category information of the current target. The contrastive layer in the text comparison branch can obtain the target-text similarity through formula (3):

[0070] s k，j =α*L2norm(e k )*L2norm(w j ) T +β (3).

[0071] Among them, L2norm is L2 norm normalization, w j ∈W is the jth text feature, e k represents the k-th target, α and β are the learnable scaling and offset factors of the affine transformation.

[0072] It is understandable that the first multi-scale fusion feature map can also be predicted by methods such as Anchor-Based (based on anchor boxes) and full convolutional networks, and no specific limitation is made here.

[0073] Step S125: Perform deformable fusion processing on the multi-scale feature map to obtain appearance feature information of the current target.

[0074] In some embodiments, the matching tracking module 250 can be used to perform deformable fusion processing on the multi-scale feature map to obtain the appearance feature information of the current target. Figure 5 The matching and tracking module 250 includes a scale feature pyramid module 251 , a deformable feature fusion module 252 and a tracking module 253 .

[0075] The multi-scale feature maps are fused using the scale feature pyramid module 251 to obtain a second multi-scale fused feature map. The second multi-scale fused feature map is fused using the deformable feature fusion module 252 to obtain a third multi-scale fused feature map. Finally, the tracking module 253 performs feature extraction on the third multi-scale fused feature map to obtain appearance feature information.

[0076] Among them, the scale feature pyramid module 251 follows the path aggregation network idea and includes four C2f modules. The specific structure is as follows: Figure 6 As shown. The multi-scale feature map includes a first resolution feature map F1, a second resolution feature map F2 and a third resolution feature map F3, and the resolutions of the first resolution feature map F1, the second resolution feature map F2 and the third resolution feature map F3 decrease in sequence. Therefore, the steps of using the scale feature pyramid module 251 to perform deformable fusion processing on the multi-scale feature map to obtain the appearance feature information of the current target include: after inputting the first resolution feature map F1, the second resolution feature map F2 and the third resolution feature map F3 into the scale feature pyramid module 251, first upsampling the third resolution feature map to obtain a first upsampling feature map; using the first C2f module 2511 to fuse the first upsampling feature map and the second resolution feature map to obtain a first sampling fusion feature map; upsampling the first sampling fusion feature map to obtain a second upsampling feature map; using the second C2f module 2512 to fuse the second upsampling feature map and the third resolution feature map to obtain a first sampling fusion feature map. The first sampling fusion feature map is fused with the first resolution feature map to obtain a second sampling fusion feature map P4; the second sampling fusion feature map is downsampled to obtain a first downsampled feature map; the first downsampled feature map and the first sampling fusion feature map are fused using the third C2f module 2513 to obtain a third sampling fusion feature map P5; the third sampling fusion feature map is downsampled to obtain a second downsampled feature map; the second downsampled feature map and the third resolution feature map are fused using the fourth C2f module 2514 to obtain a fourth sampling fusion feature map P6; the second sampling fusion feature map, the third sampling fusion feature map and the fourth sampling fusion feature map are fused to obtain a second multi-scale fusion feature map.

[0077] After obtaining the second multi-scale fused feature map, the deformable feature fusion module 252 uses deformable convolution to aggregate information between spatial dimensions and feature levels, learns the discriminative features between different instances, and outputs the third multi-scale fused feature map F(p), as shown in formula (4):

[0078]

[0079] Among them, L represents the total number of feature levels, K is the number of sampling positions of the convolution kernel, and w k represents the weight of the kth position, p krepresents the predefined offset of the k-th position, F j represents the feature map corresponding to the jth feature level, and and is the learnable offset and modulation factor for the kth position on the jth feature level.

[0080] Tracking module 250 includes a RoI feature extractor and a quasi-dense target feature extractor. The RoI feature extractor consists of three RoI Align layers; the quasi-dense target feature extractor includes four convolutional layers and one fully connected layer. Each convolutional layer is then normalized using GroupNorm and activated by the ReLU function.

[0081] Therefore, the tracking module 250 uses the RoI (Region of Interest) feature extractor to perform region recognition on the third multi-scale fusion feature map to obtain a region of interest. Then, the quasi-dense target feature extractor is used to perform quasi-dense feature extraction on the region of interest to obtain appearance feature information embedding.

[0082] Step S130: For each current target, the target detection information of the current target and the historical detection information corresponding to the historical targets are used to select the historical targets that match the current target and associate them.

[0083] The target detection information also includes the location information of the current target and the location confidence level corresponding to the location information.

[0084] In some embodiments, in order to improve the accuracy of target detection, the current target may be screened before step S130. Specifically, the current target whose confidence does not meet the preset confidence requirement is deleted. The preset confidence requirement may be that the position confidence of the current target is greater than or equal to the confidence threshold. In this embodiment, the confidence threshold may be set to 0.5. If the position confidence of the current target is less than the confidence threshold, the current target is deleted; if the position confidence of the current target is greater than or equal to the confidence threshold, the current target is retained and regarded as the current remaining current target. It can be understood that no specific limitation is made here regarding the setting of the confidence threshold.

[0085] In addition, the current targets that are greater than the current remaining ones can be further filtered. Specifically, among the current targets that are currently remaining, the current target with the highest position confidence is used as the first target, and the remaining current targets are used as the second targets; based on the position information, the first intersection-and-union ratio between the current target and each second target is obtained; the second target whose first intersection-and-union ratio is greater than the preset filtering value is deleted; and the current target with the highest confidence among the remaining current targets is used as the first target, and the remaining current targets are used as the second targets, and subsequent steps thereof are repeated until only one current target is left as the first target or all current targets are used as the first target. In this embodiment, the preset filtering value can be set to 0.3. It can be understood that no specific limitation is made here regarding the setting of the preset filtering value.

[0086] In some embodiments, after obtaining the remaining current targets, the remaining current targets can be matched with historical targets to construct the target trajectory. Specifically, see steps S131 to S135. The target detection information includes the current target's location information and current appearance feature information, and the historical detection information includes historical location information and historical appearance feature information.

[0087] Step S131: Calculate similarity between the current appearance feature information and each piece of historical appearance feature information to obtain an appearance similarity score.

[0088] In some embodiments, the bidirectional similarity between the current appearance feature information and the historical appearance feature information can be fused to obtain a first similarity score. Simultaneously, a second similarity score is obtained based on the difference between the cosine similarity between the current appearance feature information and the historical appearance feature information and a preset value. The central tendency statistics of the first similarity score and the second similarity score are obtained to obtain an appearance similarity score. For the calculation method of the first similarity score and the second similarity score, please refer to formulas (5)-(7).

[0089]

[0090] d z =(d bisoftmax (i, j)+d cosine (i, j)) / 2 (8).

[0091] in, represents the updated appearance features after the k-th frame trajectory is successfully matched, represents the appearance features of the detection box target matched by the k-th frame trajectory, μ is the weight hyperparameter, and d bisoftmax (i, j) is the first similarity score, d cosine (i, j) is the second similarity score, d z is the appearance similarity score.

[0092] Step S132: Calculate the distance using the location information and the historical location information to obtain a location score.

[0093] The position information represents the position of the current bounding box corresponding to the current target, and the historical position information represents the position of the historical bounding boxes corresponding to the historical targets.

[0094] In some embodiments, the historical bounding box that is closest to the current bounding box can be used as the nearest bounding box. The distance between the preset position point in the nearest bounding box and the preset position point in the current bounding box is used to obtain the inter-box distance value. Among them, the calculation method such as Euclidean and Manhattan methods can be used, and no specific limitation is made here. In response to the inter-frame distance value being less than the target distance threshold, a position score is obtained based on the difference between the inter-frame distance value and the target distance threshold, wherein the target distance threshold is obtained based on a scaling factor and a preset distance threshold, and the scaling factor is related to the frame difference data; or, in response to the inter-frame distance value being greater than or equal to the target distance threshold, the position score is a preset value. The calculation formula for the position score is:

[0095]

[0096] θ=exp(abs(diff(frame)))*5 (10).

[0097] Among them, d pos (i, j) is the position score, Represents a preset position point in the current bounding box, represents the preset position point in the nearest bounding box, τ c is the preset distance threshold, θ is the scaling factor, and diff(frame) is the frame difference data obtained by the inter-frame difference method.

[0098] Step S133: constructing a cost matrix based on each appearance similarity score and each position score.

[0099] In some embodiments, after obtaining each appearance similarity score and each position score, the appearance similarity score and position score corresponding to the same historical target can be fused to obtain a fused score corresponding to the historical target. The fused scores corresponding to different historical targets are then used as different elements in the cost matrix. Specifically, please refer to Formula 11:

[0100] CostMatrix=d pos (i,j)*(d bisoftmax (i, j)+d cosine (i, j)) / 2 (11).

[0101] Among them, CostMatrix is the cost matrix.

[0102] Step S134: Based on the cost matrix, determine the historical goals that match the current goal.

[0103] In some embodiments, according to the cost matrix, MaxCost matching is performed based on a greedy strategy, and the fusion scores of the current target and each historical target are obtained in descending order, and the largest fusion score is found, and the fusion score is compared with the matching threshold. If it is greater than the matching threshold, the historical target corresponding to the fusion score is successfully matched with the current target.

[0104] Step S135: Associating the matched current target with the historical target.

[0105] In some embodiments, the successfully matched current target is associated with the historical target, and the target detection information of the current target for which the matching historical target is found is updated to the trajectory library.

[0106] In addition, a new trajectory is created for the current target for which no matching historical target is found, and the new trajectory is added to the trajectory library; or, the current target for which no matching historical target is found is used as a pending target, and the target detection information of the pending target is temporarily stored in the trajectory library. In response to the pending target still not finding a matching historical target within a preset time, the target detection information of the pending target temporarily stored in the trajectory library is deleted.

[0107] See also Figure 7 , Figure 7 This is a flow chart of an embodiment of a training method for a multi-target tracking model of the present application. Specifically, the method may include the following steps:

[0108] Step S710: performing data enhancement on the sample image to obtain a plurality of sample enhanced images, wherein each sample enhanced image has at least one sample target.

[0109] The sample enhanced image is annotated with actual position information of the sample target and target sample information, and the target sample information is used to describe the appearance features of the sample target in the sample enhanced image.

[0110] In some embodiments, the sample data may be rotated to obtain images of different views, that is, to obtain several sample enhanced images.

[0111] Step S720: For each sample enhanced image, use the multi-target tracking model to perform feature extraction on the sample enhanced image to obtain a predicted multi-scale sample feature map.

[0112] In some embodiments, the backbone network module 210 in the multi-target tracking model 200 can be used to extract features from each sample enhanced image to obtain a predicted multi-scale sample feature map. Specifically, data enhancement is performed on image I to obtain corresponding images of two different views. and

[0113] Step S730: extract features from the sample text using the multi-target tracking model to obtain predicted text features.

[0114] The sample text is annotated with association information between the sample text and the sample target.

[0115] In some embodiments, the text encoding module 220 in the multi-target tracking model 200 can be used to extract features from the sample text to obtain predicted text features. In addition, a predicted online vocabulary can be constructed using the text encoder CLIP in the text encoding module 220.

[0116] Step S740: Fusing the predicted multi-scale sample feature map and the predicted text feature to obtain a predicted fusion feature.

[0117] In some embodiments, the vision-language aggregation module 230 in the multi-target tracking model 200 may be used to fuse the predicted multi-scale sample feature map and the predicted text feature to obtain a predicted fusion feature.

[0118] Step S750: Prediction is performed based on the prediction fusion features to obtain the predicted position information and the similarity of the predicted target text. The predicted target text similarity represents the correlation between the target corresponding to the predicted position information and the sample text.

[0119] In some embodiments, the prediction module 240 in the multi-target tracking model 200 can be used to perform gridding processing on the corresponding images of different views based on the feature map size to obtain K corresponding predicted position information and target text similarity.

[0120] Step S760: Determine the predicted appearance features based on the predicted multi-scale sample feature map.

[0121] In some embodiments, the SAM model can be used to perform detailed object segmentation on the image I to generate a set Q. Data enhancement is applied to the set Q. and Obtain corresponding target sets for two different views.

[0122] Step S770: Optimize the network parameters of the multi-target tracking model based on the difference between the predicted position information and the actual position information, the difference between the predicted target text similarity and the associated information of the sample text, and the difference between the predicted appearance features and the target sample information.

[0123] In some embodiments, in order to obtain the difference between the predicted position information and the actual position information, the predicted position information obtained by enhancing the images of different samples can be matched with the actual position information, and a text index is assigned to each predicted position information as a classification label. Based on task alignment learning, the classification score and bounding box quality are unified, and positive and negative samples are assigned. Then, based on the positive and negative samples, the bounding box regression loss is calculated simultaneously using the CIoU (Complete-IoU) loss function and the DFL (Distribution FocalLoss) loss function to obtain the loss Loss. iou and Loss dfl .

[0124] In some embodiments, in order to obtain the difference between the similarity of the predicted target text and the associated information of the sample text, the similarity of the predicted target text can be used to determine a text ID from the predicted online vocabulary, and then the cross entropy between the text ID and the associated information of the sample text is used to obtain the regional text contrast loss Loss. con .

[0125] In some embodiments, the step of obtaining the difference between the predicted appearance feature and the target sample information includes: obtaining a second intersection-and-union ratio of the area corresponding to the predicted appearance feature and the area corresponding to the target sample information. In response to the second intersection-and-union ratio being greater than the first threshold, the sample target corresponding to the target sample information is used as a positive sample of the sample target corresponding to the predicted appearance feature. In this embodiment, the first threshold can be set to 0.7. In response to the second intersection-and-union ratio being less than the second threshold, the sample target corresponding to the target sample information is used as a negative sample of the sample target corresponding to the predicted appearance feature. Based on the positive sample data and negative sample data corresponding to the predicted appearance feature, a sample instance loss value is obtained, wherein the sample instance loss value characterizes the difference between the predicted appearance feature and the target sample information. Specifically, please refer to Formula 12:

[0126]

[0127] Among them, q- is a negative sample, q+ is a positive sample, and Q- is a negative instance in the set Q.

[0128] In the above way, the total training loss can be obtained:

[0129] Loss total =Loss con +Loss iou+Loss dfl +Loss mask (13).

[0130] Use the stochastic gradient descent method to update the weight parameters of each network node in the multi-target tracking model until the loss value Loss_total converges, and save the multi-target tracking model. Momentum optimization, second-order optimization methods, etc. can also be used, and are not specifically limited here.

[0131] This method adopts a text-guided cross-stage module (T-CSPLayer) based on the CSP idea and Max-Sigmoid attention design to strengthen the feature interaction between text and image; uses a large vision-language model to encode online vocabulary for training, and adopts a "prompt first, then detect" strategy to create offline vocabulary in the inference stage, eliminating the need to re-encode prompt words in real time, significantly reducing computational overhead and improving inference speed.

[0132] This method adopts a feature matching tracking module and uses deformable convolution to aggregate information between spatial dimensions and feature levels. During training, it performs quasi-dense matching of target appearance features based on contrastive learning, considering matching between possible candidate targets in information-rich areas. During inference, it combines bidirectional softmax similarity and cosine similarity to improve the accuracy of association matching between targets and enhance the robustness of the matching association capability of the open vocabulary multi-target tracking method.

[0133] This method leverages the segmentation capabilities of the SAM large model to perform detailed instance segmentation on sample-enhanced images and applies different data augmentation methods, thereby providing more negative samples and supporting better generalization of this method in different fields.

[0134] In addition, this method is based on contrastive learning, which learns the distinguishing features between different targets, pushing features belonging to the same target closer together, while pulling features from different targets further apart, helping to learn better target representations for association and improving the performance of open vocabulary multi-target tracking methods.

[0135] See also Figure 8 , Figure 8 This is a flow chart of another embodiment of the multi-target tracking method of the present application. Specifically, it may include the following steps:

[0136] Step S810: Acquire the current tracking image and target text.

[0137] This step is the same as the above step 110 and will not be described again here.

[0138] Step S820: Target detection is performed on the current tracking image using a multi-target tracking model to determine a number of current targets and target detection information corresponding to the current targets. The target detection information includes the location information of the current targets.

[0139] This step is the same as the above step 120, so it will not be described again here.

[0140] Step S830: Filter the current target based on its location information to obtain a target to be matched.

[0141] In some embodiments, the location information includes location confidence. The location confidence of the current target is compared with the confidence threshold. If the location confidence of the current target is less than the confidence threshold, the current target is deleted; if the location confidence of the current target is greater than or equal to the confidence threshold, the current target is retained as the remaining current target. Among the remaining current targets, the current target with the highest location confidence is used as the first target, and the remaining current targets are used as second targets. Through the location information of the first target and the second target, the first intersection-and-union ratio between the current target and each second target is obtained. The second target whose first intersection-and-union ratio is greater than the preset screening value is deleted. Repeat the steps of using the current target with the highest confidence as the first target and the remaining current targets as the second targets among the remaining current targets, and the subsequent steps until only one current target is left as the first target or all current targets are used as the first target, and the last remaining current target is used as the matching target.

[0142] Step S840: Matching the target detection information of the target to be matched with the historical detection information corresponding to the historical target, and associating the successfully matched historical target with the target to be matched.

[0143] This step is the same as the above step 130 and will not be described again here.

[0144] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0145] See also Figure 9 , Figure 9 is a schematic diagram of an embodiment of an electronic device 90 of the present application. Electronic device 90 includes a memory 91 and a processor 92 coupled to each other. Processor 92 is configured to execute program instructions stored in memory 91 to implement the steps of any of the aforementioned multi-target tracking method embodiments. In a specific implementation scenario, electronic device 90 may include, but is not limited to, a microcomputer and a server. Furthermore, electronic device 90 may also include mobile devices such as laptops and tablet computers, without limitation herein.

[0146] Specifically, the processor 92 is used to control itself and the memory 91 to implement the steps in any of the above-mentioned multi-target tracking method embodiments. The processor 92 can also be called a CPU (Central Processing Unit). The processor 92 may be an integrated circuit chip with signal processing capabilities. The processor 92 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 92 can be implemented by an integrated circuit chip.

[0147] See also Figure 10 , Figure 10 1 is a schematic diagram of a framework of an embodiment of a computer-readable storage medium 10 of the present application. The computer-readable storage medium 10 stores program instructions 101 that can be executed by a processor, and the program instructions 101 are used to implement the steps of any of the above-mentioned multi-target tracking method embodiments.

[0148] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0149] The above description of the various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced with each other and will not be repeated herein for the sake of brevity.

[0150] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation methods described above are only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. For example, units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0151] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0152] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

Claims

1. A multi-target tracking method, characterized in that: include: Obtaining the current tracking image and target text, wherein the target text is used to describe information about the target to be tracked; Performing target detection on the current tracking image based on the target text using a multi-target tracking model to determine a number of current targets and target detection information corresponding to the current targets; For each current target, the target detection information of the current target and the historical detection information corresponding to the historical targets are used to select a historical target that matches the current target and associate it.

2. The method according to claim 1, characterized in that The target detection information further includes at least one of the position information, type information and appearance feature information of the current target; The method of performing target detection on the current tracking image based on the target text using the multi-target tracking model to determine a number of current targets and target detection information corresponding to the current targets includes: Performing feature extraction on the current tracking image to obtain a multi-scale feature map; Performing feature extraction on the target text to obtain text features; Fusing the multi-scale feature map and the text feature to obtain a first multi-scale fused feature map; Predicting the first multi-scale fusion feature map to obtain at least one of the position information and category information of the current target; and / or, Performing deformable fusion processing on the multi-scale feature map to obtain appearance feature information of the current target.

3. The method according to claim 2, characterized in that The performing deformable fusion processing on the multi-scale feature map to obtain appearance feature information of the current target includes: Fusing the multi-scale feature maps to obtain a second multi-scale fused feature map; Using variable convolution, the second multi-scale fusion feature map is fused to obtain a third multi-scale fusion feature map; Feature extraction is performed on the third multi-scale fusion feature map to obtain the appearance feature information.

4. The method according to claim 3, characterized in that The multi-scale feature map includes a first resolution feature map, a second resolution feature map, and a third resolution feature map, wherein the resolutions of the first resolution feature map, the second resolution feature map, and the third resolution feature map decrease in sequence; and fusing the multi-scale feature maps to obtain a second multi-scale fused feature map, comprising: Upsampling the third resolution feature map to obtain a first upsampling feature map; fusing the first upsampling feature map with the second resolution feature map to obtain a first sampling fusion feature map; Performing upsampling processing on the first sampled fusion feature map to obtain a second upsampling feature map; Fusing the second upsampled feature map with the first resolution feature map to obtain a second sampled fused feature map; Downsampling the second sampled fusion feature map to obtain a first downsampling feature map; Fusing the first down-sampling feature map and the first sampling fusion feature map to obtain a third sampling fusion feature map; Downsampling the third sampled fusion feature map to obtain a second downsampling feature map; Fusing the second downsampled feature map and the third resolution feature map to obtain a fourth downsampled fused feature map; The second sampling fusion feature map, the third sampling fusion feature map and the fourth sampling fusion feature map are fused to obtain the second multi-scale fusion feature map.

5. The method according to claim 3, characterized in that The performing feature extraction on the third multi-scale fusion feature map to obtain the appearance feature information includes: Performing region recognition on the third multi-scale fusion feature map to obtain a feature region of interest; Quasi-dense feature extraction is performed on the feature region of interest to obtain appearance feature information.

6. The method according to claim 2, characterized in that The extracting features of the current tracking image to obtain a multi-scale feature map includes: Performing feature extraction processing for a preset number of stages on the current tracking image to obtain a plurality of feature maps; Perform pooling on the feature map obtained in the final stage to obtain a pooled feature map; Combining at least one of the feature maps and the pooled feature map to obtain the multi-scale feature map; And / or, the position information represents the position of the current bounding box corresponding to the current target; and the predicting the first multi-scale fusion feature map to obtain at least one of the position information and category information of the current target includes at least one of the following steps: Performing bounding box regression processing on the first multi-scale fusion feature map to obtain a position of the current bounding box; Perform text normalization processing on the first multi-scale fusion feature map to obtain category information of the current target.

7. The method according to claim 2, characterized in that The multi-scale feature map includes a first resolution feature map, a second resolution feature map, and a third resolution feature map, wherein the resolutions of the first resolution feature map, the second resolution feature map, and the third resolution feature map decrease in sequence; The fusing the multi-scale feature map and the text feature to obtain a first multi-scale fused feature map includes: Upsampling the third resolution feature map to obtain a third upsampled feature map; Fusing the third up-sampled feature map, the second resolution feature map, and the text feature to obtain a first fused feature map; Performing upsampling processing on the first fused feature map to obtain a fourth upsampled feature map; fusing the fourth upsampled feature map, the first resolution feature map, and the text feature to obtain a second fused feature map; Downsampling the second fused feature map to obtain a third downsampled feature map; fusing the third downsampled feature map, the first fused feature map, and the text feature to obtain a third fused feature map; Downsampling the third fused feature map to obtain a fourth downsampled feature map; Fusing the fourth down-sampling feature map, the third resolution feature map, and the text feature to obtain a fourth fused feature map; The second fused feature map, the third fused feature map, and the fourth fused feature map are fused to obtain the first multi-scale fused feature map.

8. The method according to claim 2, characterized in that The multi-target tracking model includes a backbone network module, a text encoding module, a vision-language aggregation module, a prediction module and a matching tracking module; wherein the backbone network module is used to extract features of the current tracking image, the text encoding module is used to extract features of the target text, the vision-language aggregation module is used to fuse the multi-scale feature map and the text features, the prediction module is used to predict the first multi-scale fused feature map, and the matching tracking module is used to perform deformable fusion processing on the multi-scale feature map.

9. The method according to claim 1, characterized in that The target detection information includes the position information and current appearance feature information of the current target, and the historical detection information includes historical position information and historical appearance feature information; The step of utilizing the target detection information of the current target and the historical detection information corresponding to the historical targets to select and associate the historical targets that match the current target includes: Calculating similarity between the current appearance feature information and each of the historical appearance feature information to obtain an appearance similarity score; Calculating the distance between the location information and the historical location information to obtain a location score; constructing a cost matrix based on the appearance similarity scores and the position scores; Determining, based on the cost matrix, a historical goal that matches the current goal; The matched current target is associated with the historical target.

10. The method according to claim 9, characterized in that The calculating the similarity between the current appearance feature information and each of the historical appearance feature information to obtain an appearance similarity score includes: fusing the current appearance feature information and the historical appearance feature information using a bidirectional similarity to obtain a first similarity score; and Obtaining a second similarity score based on a difference between a cosine similarity between the current appearance feature information and the historical appearance feature information and a preset value; Obtaining a central tendency statistic of the first similarity score and the second similarity score to obtain the appearance similarity score; And / or, the location information represents the location of a current bounding box corresponding to the current target, and the historical location information represents the location of a historical bounding box corresponding to the historical target; and the performing distance calculation using the location information and the historical location information to obtain a location score includes: The historical bounding box closest to the current bounding box is used as the nearest bounding box; Obtaining an inter-frame distance value based on a distance between a preset position point in the nearest bounding frame and a preset position point in the current bounding frame; In response to the inter-frame distance value being less than a target distance threshold, the position score is obtained based on a difference between the inter-frame distance value and the target distance threshold, wherein the target distance threshold is obtained based on a scaling factor and a preset distance threshold, and the scaling factor is related to the frame difference data; or, in response to the inter-frame distance value being greater than or equal to the target distance threshold, the position score is a preset value.

11. The method according to claim 9, characterized in that The constructing a cost matrix based on the appearance similarity scores and the position scores includes: Fusing the appearance similarity score and the position score corresponding to the same historical target to obtain a fusion score corresponding to the historical target; The fusion scores corresponding to different historical targets are respectively used as different elements in the cost matrix; And / or, determining a historical target that matches the current target based on the cost matrix includes: Select the largest element in the cost matrix as the target element; In response to the numerical value corresponding to the target element being greater than a preset matching value, it is determined that the historical target corresponding to the target element matches the current target.

12. The method according to claim 1, characterized in that The target detection information also includes the location information of the current target and the location confidence corresponding to the location information; before associating the selected historical target that matches the current target using the target detection information of the current target and the historical detection information corresponding to the historical target for each current target, the method includes: Deleting the current target whose confidence level does not meet the preset confidence level requirement; and / or, Among the currently remaining current targets, the current target with the highest confidence is used as the first target, and the remaining current targets are used as second targets; based on the position information, the first intersection-and-union ratio between the current target and each of the second targets is obtained; the second targets whose first intersection-and-union ratio is greater than a preset screening value are deleted; and the steps of using the current target with the highest confidence as the first target and the remaining current targets as the second targets and subsequent steps are repeated until only one current target is left as the first target or all current targets are used as the first target.

13. The method according to claim 1, wherein The training method of the multi-target tracking model includes: Performing data augmentation on the sample image to obtain a plurality of sample augmented images, wherein the sample augmented image has at least one sample target, and the sample augmented image is annotated with actual position information of the sample target and target sample information, wherein the target sample information is used to describe the appearance characteristics of the sample target in the sample augmented image; For each of the sample enhanced images, extract features of the sample enhanced image using the multi-target tracking model to obtain a predicted multi-scale sample feature map; Extracting features from a sample text using the multi-target tracking model to obtain predicted text features, wherein the sample text is annotated with association information between the sample text and the sample target; Fusing the predicted multi-scale sample feature map and the predicted text feature to obtain a predicted fusion feature; Perform prediction based on the prediction fusion feature to obtain the predicted position information and the similarity of the predicted target text, wherein the predicted target text similarity represents the correlation between the target corresponding to the predicted position information and the sample text; Determining predicted appearance features based on the predicted multi-scale sample feature map; Based on the difference between the predicted position information and the actual position information, the difference between the predicted target text similarity and the associated information of the sample text, and the difference between the predicted appearance features and the target sample information, the network parameters of the multi-target tracking model are optimized.

14. The method according to claim 13, characterized in that The step of obtaining the difference between the predicted appearance feature and the target sample information includes: Obtaining a second intersection-over-union ratio between the area corresponding to the predicted appearance feature and the area corresponding to the target sample information; In response to the second intersection-over-union ratio being greater than a first threshold, taking the sample target corresponding to the target sample information as a positive sample of the sample target corresponding to the predicted appearance feature; In response to the second intersection-over-union ratio being less than a second threshold, taking the sample target corresponding to the target sample information as a negative sample of the sample target corresponding to the predicted appearance feature; Based on the positive sample data and the negative sample data corresponding to the predicted appearance feature, a sample instance loss value is obtained, wherein the sample instance loss value represents the difference between the predicted appearance feature and the target sample information.

15. The method according to claim 1, wherein The method further includes any of the following: Update the target detection information of the current target of the historical target that matches the target into the trajectory library; Creating a new trajectory for the current target for which no matching historical target is found, and adding the new trajectory to a trajectory library; The current target for which no matching historical target is found is regarded as a pending target, and target detection information of the pending target is temporarily stored in the trajectory library. In response to the pending target still not finding a matching historical target within a preset time, the target detection information of the pending target temporarily stored in the trajectory library is deleted.

16. An electronic device, characterized in that: It comprises a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement the multi-target tracking method according to any one of claims 1 to 15.

17. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the multi-target tracking method according to any one of claims 1 to 15 is implemented.

Images