A general category multi-target tracking method integrating prototype learning mechanism

Abstract

The present invention relates to a general-category multi-target tracking method that integrates a prototype learning mechanism. The method comprises: preprocessing point annotations provided in a target count dataset to obtain rough bounding box annotation information for each target; constructing a prototype feature learning module that models the prototype features of the target of interest; constructing a general-category multi-target tracking network that integrates the prototype learning mechanism based on the obtained prototype feature learning module that models the prototype features of the target of interest; training a general-category multi-target tracking model that integrates the prototype learning mechanism; inputting an image into the trained general-category multi-target tracking model that integrates the prototype learning mechanism, outputting the corresponding target bounding box, and finally obtaining the trajectory of the target of interest using a target association algorithm.

Description

Technical Field

[0001] The present invention belongs to the field of image processing and computer vision technology, and in particular relates to a general category multi-target tracking method integrating a prototype learning mechanism. Background Art

[0002] Multi-object tracking is a long-standing research problem in computer vision due to its wide range of applications, such as robotics, video surveillance, autonomous driving, and cell tracking. Recent research in multi-object tracking has made significant progress, partly due to advances in key components such as object detection, single object tracking, and object association algorithms, and partly due to the increasing availability of public multi-object tracking datasets. Despite these significant achievements, previous multi-object tracking research has primarily focused on specific object classes of interest (e.g., pedestrians, cars, cells, etc.) and relied on models specific to these classes. For example, detectors for objects of interest are typically pre-trained offline and sometimes exploit the motion patterns of specific object classes. Existing multi-object tracking algorithms still have significant limitations in generalizing to unseen objects, thus limiting the expansion of multi-object tracking into new application domains, especially those with limited data.

[0003] In contrast, general-purpose multi-object tracking (GMMT) does not require prior knowledge of the objects being tracked and aims to address the existing challenges of multi-object tracking. Consequently, GMMT has applications in areas such as video editing, animal behavior analysis, and vision-based object counting. Despite its widespread application, GMMT remains underexplored, except for a few early studies, due to a lack of high-quality training data.

[0004] Therefore, it is extremely necessary to propose a general category multi-target tracking method with low training data requirements and good performance. Summary of the Invention

[0005] The purpose of the present invention is to fill the gaps and shortcomings of the existing technology and provide a general category multi-target tracking method that integrates a prototype learning mechanism, including: preprocessing the point annotations provided in the target counting data set to obtain rough bounding box annotation information of each target; and constructing a prototype feature learning module that models the prototype features of the target of interest; based on the obtained prototype feature learning module that models the prototype features of the target of interest, constructing a general category multi-target tracking network that integrates the prototype learning mechanism; training the general category multi-target tracking model that integrates the prototype learning mechanism; inputting the image into the trained general category multi-target tracking model that integrates the prototype learning mechanism, outputting the corresponding target bounding box, and finally using the target association algorithm to obtain the trajectory of the target of interest.

[0006] It uses the point landmarks provided by the target counting dataset and a small amount of bounding box annotations to generate rough bounding box annotations, and uses a progressive training strategy to optimize the bounding box quality, greatly reducing the cost of manual annotation. It also designs a prototype learning network to significantly improve the performance of the general category multi-target tracking model.

[0007] To achieve the above object, the technical solution of the present invention is: a general category multi-target tracking method integrating prototype learning mechanism, comprising the following steps:

[0008] Step S1: preprocess the point annotations provided in the target counting dataset to obtain the rough bounding box annotation information of each target and obtain a training set;

[0009] Step S2: constructing a prototype feature learning module for modeling the prototype features of the target of interest;

[0010] Step S3: constructing a general category multi-target tracking model integrating the prototype learning mechanism based on the prototype feature learning module for modeling the prototype features of the target of interest obtained in step S2;

[0011] Step S4: using the training set obtained in step S1 and the rough bounding box annotation to train a general category multi-target tracking model that integrates the prototype learning mechanism;

[0012] Step S5: Input the image into the general category multi-target tracking model of the trained fusion prototype learning mechanism, output the corresponding target bounding box, and finally use the target association algorithm to obtain the trajectory of the target of interest.

[0013] In one embodiment of the present invention, step S1 specifically includes the following steps:

[0014] Step S11: Calculate each target point annotation p according to the point annotation provided in the target counting dataset. i The area occupied by each target point is calculated from the distance to its K nearest neighbors, and a rectangular area with an aspect ratio of 1:1 is obtained as its initial bounding box. The calculation formula is as follows:

[0015]

[0016] in, It is p i The set of neighboring targets, d i,j is the target point annotation p i and target point annotation p j The Euclidean distance between

[0017] Step S12: Calculate the aspect ratio of the object in each image based on the Q bounding box annotations provided by the object counting dataset, that is:

[0018]

[0019] in, represents the average aspect ratio of a given bounding box annotation, w i and h i Represent the width and length of a given bounding box respectively;

[0020] Step S13: re-adjust the initial bounding box obtained in S11 according to the aspect ratio obtained in S12, calculate the rough bounding box of each target, and assign an initial confidence τ, that is:

[0021]

[0022]

[0023]

[0024]

[0025] Among them, w i and h i denote the width and length of the estimated rough bounding box, respectively.

[0026] In one embodiment of the present invention, step S2 specifically includes the following steps:

[0027] Step S21: Use the features from the previous moment's detection results as the input of the prototype feature learning module, that is, use the ROIAlign algorithm to align the N results obtained from the previous moment's detection. The visual feature F of the image with the dimension of C×h×w at the previous moment t-1 Perform feature extraction, the specific expression is:

[0028]

[0029] Among them, ROIAlign(·) means feature clipping operation, which is used to clip the features of the region of interest from the image visual features. Represents the visual features F from the image t-1 The visual features of each target of interest obtained by cropping are of dimension C×7×7.

[0030] Step S22: The visual features of the target of interest obtained in S21 are The stacked layers are input into a 1×1 convolutional layer to interact and transform the feature map information, and a N×C×7×7 feature map F' is obtained. det , the specific expression is:

[0031]

[0032] Among them, Concat(·) means that the features are concatenated in a new dimension, and Conv(·) represents the convolution operation;

[0033] Step S23: The feature map F' obtained in S22 det Flatten and input into the multi-head self-attention mechanism to obtain the most representative features of the detection results at the previous moment The specific expression is:

[0034] q=w1(Flatten(F′ det ))+b1,

[0035] k=w2(Flatten(F′ det ))+b2,

[0036] v=w3(Flatten(F′ det ))+b3,

[0037]

[0038] Among them, Flatten(·) represents the flattening operation, w i 、b i They correspond to the weight and bias of the i-th linear mapping, q, k, v represent the query, key, and value required to calculate the attention, respectively. Softmax(·) represents the softmax mapping. Indicates the most representative feature of the detection result at the previous moment, d k represents the dimension of key k;

[0039] Step S24: The most representative feature obtained in S23 The prototype features of the current moment are obtained by concatenating them with the prototype features of the previous moment and the initial moment, performing 1×1 convolution, and adding a residual structure. The calculation formula is:

[0040]

[0041] in Represent the prototype features of the previous moment, initial moment and current moment respectively.

[0042] In one embodiment of the present invention, step S3 specifically includes the following steps:

[0043] Step S31: Based on the Faster-RCNN target detection network, ResNet-50 is first used as the backbone feature extraction network to extract the visual features of the current input image.

[0044] Step S32: The visual features of the input image obtained in S31 are Input into the prototype feature learning module constructed by S2 to model the prototype features of the target of interest, and obtain the prototype features of the target of interest at the current moment

[0045] Step S33: The prototype features of the target of interest at the current moment obtained in S32 are Input 1×1 convolution layer to adjust the feature space, and then combine it with the input image visual features obtained in step S31 Multiply to get the enhanced features of the prototype The calculation formula is:

[0046]

[0047] Where ⊙ represents the element-by-element matrix multiplication, represents the features after prototype enhancement, Conv(·) represents the convolution operation;

[0048] Step S34: Enhance the features of the prototype obtained in S33 Input into the region proposal network in Faster-RCNN, obtain M region proposals, and use the ROIAlign algorithm from Extract the corresponding features of the region proposal Then the regional proposal features and prototype features Multiply to perform further enhancement:

[0049]

[0050] in, Represents the enhanced M region proposal features;

[0051] Step S35: The enhanced M region proposal features obtained in S34 are Input the RCNN network in Faster-RCNN to obtain the final detection result after refining the region proposal.

[0052] In one embodiment of the present invention, step S4 specifically includes the following steps:

[0053] The final detection result obtained in step S41 and step S35 will be compared with the rough bounding box in S1. When it is greater than the confidence τ of the rough bounding box at the current moment, the rough bounding box will be updated;

[0054] Step S42: Calculate the gradients of the parameters in the deep network of the general category multi-target tracking model of the fusion prototype learning mechanism using the back propagation method according to the loss function of the general category multi-target tracking model of the fusion prototype learning mechanism, and update the parameters using the stochastic gradient descent method;

[0055] Step S43, repeating steps S41 to S42 in batches until the loss value calculated in step S42 converges and stabilizes, completing the training process of the general category multi-target tracking model integrating the prototype learning mechanism.

[0056] In one embodiment of the present invention, in step S41, the updating formula is as follows:

[0057]

[0058]

[0059] Where b refers to the rough bounding box used to supervise model training, α is the weight update coefficient, Refers to the detection bounding box output by the model that matches b.

[0060] In one embodiment of the present invention, the loss function L is as follows:

[0061] L total =L CE +λL RE ,

[0062]

[0063]

[0064] Among them, L CE represents the classification cross entropy loss, L RE represents the bounding box regression loss, λ is the parameter for balancing the loss, and b i 、 They are the ground truth bounding box annotation and the network's predicted bounding box.

[0065] In one embodiment of the present invention, step S5 specifically includes the following steps:

[0066] Step S51: Input the image and corresponding text features in the test set into the general category multi-target tracking model of the trained fusion prototype learning mechanism, and output the detection bounding box set at the corresponding moment.

[0067] Step S52: Use the SORT target association algorithm to obtain the trajectory set T of the target of interest = {traj i} 1:D, where D represents the number of objects of interest present in the video.

[0068] The present invention also provides a general category multi-target tracking system that integrates a prototype learning mechanism, including a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0069] The present invention also provides a computer-readable storage medium on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0070] Compared with the existing technology, the present invention has the following beneficial effects: this method uses the point landmarks provided by the target counting dataset and a small amount of bounding box annotations to generate rough bounding box annotations, and uses a progressive training strategy to optimize the bounding box quality, which greatly reduces the cost of manual annotation, and designs a prototype learning network, which greatly improves the performance of the general category multi-target tracking model. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 This is a schematic diagram of the overall process of the method according to the embodiment of the present invention;

[0072] Figure 2 This is a schematic diagram of the network model structure of an embodiment of the present invention;

[0073] Figure 3 Schematic diagram of the prototype learning network structure of an embodiment of the present invention. DETAILED DESCRIPTION

[0074] To make the features and advantages of this patent more clearly understood, the following embodiments are specifically described in detail as follows:

[0075] It should be noted that the following detailed description is illustrative and is intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used in this specification have the same meaning as commonly understood by those skilled in the art to which this application belongs.

[0076] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0077] This embodiment provides a general category multi-target tracking method that integrates the prototype learning mechanism, such as Figure 1 、 Figure 2 、 Figure 3 As shown, the following steps are included:

[0078] Step S1: pre-process the point annotations provided in the target counting dataset to obtain the rough bounding box annotation information of each target;

[0079] Step S2: constructing a prototype feature learning module for modeling the prototype features of the target of interest;

[0080] Step S3: constructing a general category multi-target tracking network integrating the prototype learning mechanism based on the prototype feature learning module for modeling the prototype features of the target of interest obtained in step S2;

[0081] Step S4: using the training set obtained in step S1 and the rough bounding box annotation to train a general category multi-target tracking model that integrates the prototype learning mechanism;

[0082] Step S5: Input the image into the general category multi-target tracking model of the trained fusion prototype learning mechanism, output the corresponding target bounding box, and finally use the target association algorithm to obtain the trajectory of the target of interest.

[0083] In this embodiment, step S1 specifically includes the following steps:

[0084] Step S11: Calculate each target point annotation p according to the point annotations provided in the Few-shot Object Counting 147 target counting dataset. i The approximate area occupied by the object is calculated based on the distance to its K=3 nearest neighbors, and a rectangular area with an aspect ratio of 1:1 is obtained as its initial bounding box. The calculation formula is as follows:

[0085]

[0086] in, It is p i The set of neighboring targets, d i,j is the target p i With the target p j The Euclidean distance between

[0087] Step S12: Calculate the approximate aspect ratio of the object in each image based on the Q=3 bounding box annotations provided by the object counting dataset, i.e.:

[0088]

[0089] in, Represents the average aspect ratio of the given bounding box annotation, wi and hi represent the width and length of the given bounding box respectively;

[0090] Step S13: re-adjust the initial bounding box obtained in S11 according to the aspect ratio obtained in S12, calculate a rough bounding box for each target, and assign it an initial confidence level τ = 0.6, that is:

[0091]

[0092]

[0093]

[0094]

[0095] Among them, w i and h i denote the width and length of the estimated rough bounding box, respectively.

[0096] Furthermore, step S2 specifically includes the following steps:

[0097] Step S21: Use the features from the detection results of the previous moment as the input of the prototype learning module, that is, use the ROIAlign algorithm to align the n results obtained from the detection of the previous moment The image visual feature F of the previous moment with dimension C×H×W t-1 Perform feature extraction, the specific expression is:

[0098]

[0099] Among them, ROIAlign(·) means feature clipping operation, which is used to clip the features of the region of interest from the image visual features. Represents the visual features F from the image t-1 The visual features of each target of interest obtained by cropping are of dimension C×7×7.

[0100] Step S22: The visual features of the target of interest obtained in step S21 are The stacked layers are input into a 1×1 convolutional layer to interact and transform the feature map information, and a N×C×7×7 feature map F' is obtained. det , the specific expression is:

[0101]

[0102] Among them, Concat(·) means that the features are concatenated in a new dimension, and Conv(·) represents the convolution operation.

[0103] Step S23: The feature map F' obtained in step S22 detFlatten and input into the multi-head self-attention mechanism to obtain the most representative features of the detection results at the previous moment The specific expression is:

[0104] q=w1(Flatten(F′ det ))+b1,

[0105] k=w2(Flatten(F′ det ))+b2,

[0106] v=w3(Flatten(F′ det ))+b3,

[0107]

[0108] Among them, Flatten(·) represents the flattening operation, w i 、b i They correspond to the weight and bias of the i-th linear mapping, q, k, v represent the query, key, and value required to calculate the attention, respectively. Softmax(·) represents the softmax mapping. Indicates the most representative feature of the detection result at the previous moment, d k represents the dimension of key k;

[0109] Step S24: The most representative feature obtained in step S23 Compared with the prototype characteristics of the previous moment and the initial moment Perform splicing, 1×1 convolution, and add a residual structure to obtain the prototype features of the current moment The calculation formula is:

[0110]

[0111] in Represent the prototype features of the previous moment, initial moment and current moment respectively.

[0112] In this embodiment, step S3 specifically includes the following steps:

[0113] Step S31: Based on the Faster-RCNN target detection network, ResNet-50 is first used as the backbone feature extraction network to extract the visual features of the current input image.

[0114] Step S32: The visual features of the input image obtained in step S31 are Input into the prototype feature learning module constructed by S2 to model the prototype features of the target of interest, and obtain the prototype features of the target of interest at the current moment

[0115] Step S33: The prototype features of the target of interest at the current moment obtained in step S32 are Input 1×1 convolution layer to adjust the feature space, and then combine it with the input image visual features obtained in step S31 Multiply to get the enhanced features of the prototype The calculation formula is:

[0116]

[0117] Where ⊙ represents the element-by-element matrix multiplication, Indicates the enhanced features of the prototype;

[0118] Step S34: Enhance the features of the prototype obtained in step S33 Input into the region proposal network in Faster-RCNN, obtain M region proposals, and use the ROIAlign algorithm from Extract the corresponding features of the region proposal Then the regional proposal features and prototype features Multiply to perform further enhancement:

[0119]

[0120] in, Represents the enhanced M region proposal features;

[0121] Step S35: The enhanced M region proposal features obtained in step S34 are Input into the classification branch and regression branch in the detection head, and obtain 2M classification scores and 4M proposal box adjustment parameters. Then, the threshold method and non-maximum suppression algorithm are used to filter and obtain N final detection results of the target of interest.

[0122] In this embodiment, step S4 specifically includes the following steps:

[0123] The detection results obtained in step S41 and step S35 will be compared with the rough bounding box in S1. When it is greater than the confidence τ of the rough bounding box at the current moment, the rough bounding box will be updated. The update formula is as follows:

[0124]

[0125]

[0126] Where b refers to the rough bounding box used to supervise model training, α is the weight update coefficient, Refers to the detection bounding box output by the model that matches b.

[0127] Step S42: Based on the loss function of the general category multi-target tracking model based on prototype learning, the gradients of the parameters in the deep network of the general category multi-target tracking model are calculated using the back propagation method, and the parameters are updated using the stochastic gradient descent method; the loss function L is as follows:

[0128] L total =L CE +λI RE ,

[0129]

[0130]

[0131] Among them, L CE represents the classification cross entropy loss, L RE represents the bounding box regression loss, λ is the parameter for balancing the loss, and b i 、 They are the ground truth bounding box annotation and the network's predicted bounding box.

[0132] Step S43: Repeat steps S41 to S42 in batches until the loss value calculated in step S42 converges and stabilizes, save the network parameters, and complete the training process of the general category multi-target tracking model based on prototype learning.

[0133] In this embodiment, step S5 specifically includes the following steps:

[0134] Step S51: Input the image and corresponding text features in the test set into the trained general category multi-target tracking model based on prototype learning, and output the detection bounding box set at the corresponding moment.

[0135] Step S52: Use the SORT target association algorithm to obtain the trajectory set T of the target of interest = {traj i} 1:D , where D represents the number of objects of interest present in the video.

[0136] The above program design scheme provided in this embodiment can be stored in a computer-readable storage medium in a coded form and implemented in the form of a computer program, and the basic parameter information required for calculation is input through computer hardware, and the calculation results are output.

[0137] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0138] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0139] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0140] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0141] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other manner. Any person skilled in the art may utilize the above-disclosed technical content to modify or modify the present invention into equivalent embodiments. However, any simple modifications, equivalent variations, and modifications to the above embodiments that do not depart from the technical content of the present invention and are based on the technical essence of the present invention remain within the scope of protection of the present invention.

Claims

1. A general category multi-target tracking method integrating prototype learning mechanism, characterized by: The following steps are involved: Step S1: preprocess the point annotations provided in the target counting dataset to obtain the rough bounding box annotation information of each target and obtain a training set; Step S2: constructing a prototype feature learning module for modeling the prototype features of the target of interest; Step S3: constructing a general category multi-target tracking model integrating the prototype learning mechanism based on the prototype feature learning module for modeling the prototype features of the target of interest obtained in step S2; Step S4: using the training set obtained in step S1 and the rough bounding box annotation to train a general category multi-target tracking model that integrates the prototype learning mechanism; Step S5: input the image into the general category multi-target tracking model of the trained fusion prototype learning mechanism, output the corresponding target bounding box, and finally use the target association algorithm to obtain the trajectory of the target of interest; Step S3 specifically includes the following steps: Step S31: Based on the Faster-RCNN target detection network, ResNet-50 is first used as the backbone feature extraction network to extract the visual features of the current input image. Step S32: The visual features of the input image obtained in S31 are Input into the prototype feature learning module constructed by S2 to model the prototype features of the target of interest, and obtain the prototype features of the target of interest at the current moment Step S33: The prototype features of the target of interest at the current moment obtained in S32 are Input 1×1 convolution layer to adjust the feature space, and then combine it with the input image visual features obtained in step S31 Multiply to get the enhanced features of the prototype The calculation formula is: Where ⊙ represents the element-by-element matrix multiplication, represents the features after prototype enhancement, Conv(·) represents the convolution operation; Step S34: Enhance the features of the prototype obtained in S33 Input into the region proposal network in Faster-RCNN, obtain M region proposals, and use the ROIAlign algorithm from Extract the corresponding features of the region proposal Then the regional proposal features and prototype features Multiply to perform further enhancement: in, Represents the enhanced M region proposal features; Step S35: The enhanced M region proposal features obtained in S34 are Input the RCNN network in Faster-RCNN to obtain the final detection result after refining the region proposal; Step S4 specifically includes the following steps: The final detection result obtained in step S41 and step S35 will be compared with the rough bounding box in S1. When it is greater than the confidence τ of the rough bounding box at the current moment, the rough bounding box will be updated; Step S42: Calculate the gradients of the parameters in the deep network of the general category multi-target tracking model of the fusion prototype learning mechanism using the back propagation method according to the loss function of the general category multi-target tracking model of the fusion prototype learning mechanism, and update the parameters using the stochastic gradient descent method; Step S43, repeating steps S41 to S42 in batches until the loss value calculated in step S42 converges and stabilizes, completing the training process of the general category multi-target tracking model integrating the prototype learning mechanism.

2. A general category multi-target tracking method integrating prototype learning mechanism according to claim 1, characterized in that: Step S1 specifically includes the following steps: Step S11: Calculate each target point annotation p according to the point annotation provided in the target counting dataset. i The area occupied by each target point is calculated from the distance to its K nearest neighbors, and a rectangular area with an aspect ratio of 1:1 is obtained as its initial bounding box. The calculation formula is as follows: in, It is p i The set of neighboring targets, d i,j is the target point annotation p i and target point annotation p j The Euclidean distance between Step S12: Calculate the aspect ratio of the object in each image based on the Q bounding box annotations provided by the object counting dataset, that is: in, represents the average aspect ratio of a given bounding box annotation, w i and h i Represent the width and length of a given bounding box respectively; Step S13: re-adjust the initial bounding box obtained in S11 according to the aspect ratio obtained in S12, calculate the rough bounding box of each target, and assign an initial confidence τ, that is: Among them, w i and h i denote the width and length of the estimated rough bounding box, respectively.

3. The general category multi-target tracking method integrating prototype learning mechanism according to claim 1 is characterized in that: Step S2 specifically includes the following steps: Step S21: Use the features from the previous moment's detection results as the input of the prototype feature learning module, that is, use the ROIAlign algorithm to align the N results obtained from the previous moment's detection. The visual feature F of the image with the dimension of C×h×w at the previous moment t-1 Perform feature extraction, the specific expression is: Among them, ROIAlign(·) means feature clipping operation, which is used to clip the features of the region of interest from the image visual features. Represents the visual features F from the image t-1 The visual features of each target of interest obtained by cropping are of dimension C×7×7. Step S22: The visual features of the target of interest obtained in S21 are The stacked layers are input into a 1×1 convolutional layer to interact and transform the feature map information, and a N×C×7×7 feature map F' is obtained. det , the specific expression is: Among them, Concat(·) means that the features are concatenated in a new dimension, and Conv(·) represents the convolution operation; Step S23: The feature map F' obtained in S22 det Flatten and input into the multi-head self-attention mechanism to obtain the most representative features of the detection results at the previous moment The specific expression is: q=w1(Flatten(F′ det ))+b1, k=w2(Flatten(F′ det ))+b2, v=w3(Flatten(F′ det ))+b3, Among them, Flatten(·) represents the flattening operation, w i 、b i They correspond to the weight and bias of the i-th linear mapping, q, k, v represent the query, key, and value required to calculate the attention, respectively. Softmax(·) represents the softmax mapping. Indicates the most representative feature of the detection result at the previous moment, d k represents the dimension of key k; Step S24: The most representative feature obtained in S23 The prototype features of the current moment are obtained by concatenating them with the prototype features of the previous moment and the initial moment, performing 1×1 convolution, and adding a residual structure. The calculation formula is: in Represent the prototype features of the previous moment, initial moment and current moment respectively.

4. The general category multi-target tracking method integrating prototype learning mechanism according to claim 1 is characterized in that: In step S41, the update formula is as follows: Where b refers to the rough bounding box used to supervise model training, α is the weight update coefficient, Refers to the detection bounding box output by the model that matches b.

5. A general category multi-target tracking method integrating prototype learning mechanism according to claim 4, characterized in that: The loss function L is as follows: THE total =L CE +λL RE , Among them, L CE represents the classification cross entropy loss, L RE represents the bounding box regression loss, λ is the parameter for balancing the loss, and b i 、 They are the ground truth bounding box annotation and the network's predicted bounding box.

6. The general category multi-target tracking method integrating prototype learning mechanism according to claim 1 is characterized in that: Step S5 specifically includes the following steps: Step S51: Input the image and corresponding text features in the test set into the general category multi-target tracking model of the trained fusion prototype learning mechanism, and output the detection bounding box set at the corresponding moment. Step S52: Use the SORT target association algorithm to obtain the trajectory set T of the target of interest = {traj i } 1:D , where D represents the number of objects of interest present in the video.

7. A general category multi-target tracking system integrating prototype learning mechanism, characterized by: The method comprises a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method steps according to any one of claims 1 to 6 can be implemented.

8. A computer-readable storage medium storing computer program instructions that can be executed by a processor, wherein when the processor executes the computer program instructions, the method steps according to any one of claims 1 to 6 can be implemented.

Images