Multi-target tracking method, multi-target tracking device, electronic device and storage medium

Abstract

The present application discloses a multi-target tracking method, a multi-target tracking device, an electronic device, and a storage medium, and relates to the field of computer vision technology. The method comprises: obtaining an image feature vector and a text feature vector of a detection object in a first image frame, wherein the text feature vector is obtained by a text feature extraction module, and the text feature extraction module is trained based on a data set without text annotations; obtaining a first feature similarity between the image feature vector and an image feature template of a track to be matched, and obtaining a second feature similarity between the text feature vector and a text feature template of a track to be matched; performing tracking and matching based on the first feature similarity and the second feature similarity, and obtaining a track tracking result of the detection object in the first image frame and the track to be matched. This method can introduce semantic features into multi-target tracking while being free from the limitations of manually set text, thereby effectively improving the performance of multi-target tracking.

Description

Technical Field

[0001] The present application belongs to the field of computer vision technology, and in particular relates to a multi-target tracking method, a multi-target tracking device, an electronic device, and a storage medium. Background Art

[0002] Multiple Object Tracking (MOT) is a key technology in computer vision, widely used in autonomous driving, video surveillance, behavior recognition, and other areas. In multi-object tracking systems based on visual images, noise factors such as scale changes, deformation, rapid motion, and illumination variations can affect the tracking results. Furthermore, mutual influences between individuals of the same type (for example, similar clothing colors and hairstyles) can reduce tracking performance. Summary of the Invention

[0003] The present application provides a multi-target tracking method, a multi-target tracking device, an electronic device and a storage medium to at least solve the problems of low multi-target tracking performance in related technologies.

[0004] This application provides a multi-target tracking method, comprising the following steps:

[0005] Performing target detection on each image frame of the video image sequence to obtain a target detection result for each image frame of the video image sequence, wherein the target detection result includes a detection object of the image frame;

[0006] Obtaining an image feature vector and a text feature vector of the detected object in the first image frame, wherein the text feature vector is obtained by a text feature extraction module, and the text feature extraction module is trained based on a data set without text annotations;

[0007] Obtaining a first feature similarity between the image feature vector and the image feature template of the to-be-matched trajectory, and obtaining a second feature similarity between the text feature vector and the text feature template of the to-be-matched trajectory;

[0008] Tracking and matching are performed based on the first feature similarity and the second feature similarity to obtain a trajectory tracking result of the detected object in the first image frame and the trajectory to be matched.

[0009] The present application also provides a multi-target tracking device, comprising:

[0010] A first processing module is configured to perform target detection on each image frame of a video image sequence to obtain a target detection result for each image frame of the video image sequence, wherein the target detection result includes a detection object of the image frame;

[0011] A second processing module is configured to obtain an image feature vector and a text feature vector of the detected object in the first image frame, wherein the text feature vector is obtained by a text feature extraction module, and the text feature extraction module is trained based on a data set without text annotations;

[0012] a third processing module, configured to obtain a first feature similarity between the image feature vector and the image feature template of the to-be-matched trajectory, and to obtain a second feature similarity between the text feature vector and the text feature template of the to-be-matched trajectory;

[0013] The fourth processing module is configured to perform tracking and matching based on the first feature similarity and the second feature similarity to obtain a trajectory tracking result of the detected object in the first image frame and the trajectory to be matched.

[0014] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned multi-target tracking methods when executing the computer program.

[0015] The present application also provides a non-volatile computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned multi-target tracking methods are implemented.

[0016] Through this application, it is possible to extract the image feature vector and text feature vector of the detection object, calculate the feature similarity between the image feature vector and the text feature vector and the feature template, perform tracking and matching based on the feature similarity corresponding to the image feature and the text feature, and output the tracking results. While introducing semantic features in multi-target tracking, it is not restricted by manually set text. It integrates the image feature vector and the text feature vector to adapt to the influence of factors such as scale change, deformation, rapid movement, and illumination change of the tracked target, and better distinguish different target objects between different trajectories in time sequence, thereby achieving the technical effect of effectively improving the performance of multi-target tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0018] Figure 1 One of the flow charts of the multi-target tracking method provided in the embodiment of the present application;

[0019] Figure 2 The second flowchart of the multi-target tracking method provided in the embodiment of the present application;

[0020] Figure 3 The third flowchart of the multi-target tracking method provided in the embodiment of the present application;

[0021] Figure 4 A schematic diagram of data flow for an adapter module provided in an embodiment of the present application;

[0022] Figure 5 A schematic diagram of the structure of a multi-target tracking device provided in an embodiment of the present application;

[0023] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0025] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0026] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0027] The following describes in detail the multi-target tracking method, multi-target tracking device, electronic device, and readable storage medium provided by the embodiments of the present application through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0028] The multi-target tracking method provided in the embodiments of the present application may be executed by an electronic device or a functional module or functional entity in the electronic device that can implement the multi-target tracking method. The electronic devices mentioned in the embodiments of the present application include but are not limited to computers, servers, etc.

[0029] like Figure 1 As shown, the multi-target tracking method includes: step 110, step 120, step 130 and step 140.

[0030] Step 110: Perform target detection on each image frame of the video image sequence to obtain target detection results for each image frame of the video image sequence.

[0031] The target detection result includes the detected object of the image frame.

[0032] It can be understood that a video image sequence is a dynamic visual data stream consisting of a plurality of consecutive image frames arranged in time sequence.

[0033] In this step, a target detector is used to perform target detection on each image frame of the video image sequence, and each image frame of the video image sequence is input into the target detector frame by frame. The target detection results output by the target detector include one or more detection objects in each image frame. The target detection results can also include a detection box for locating the detection object in the image frame, a label corresponding to the detection object, and confidence level and other information.

[0034] In actual implementation, the target detector can be a detector in the Tracking-by-Detection (TBD) tracking mode, such as the YOLO series detector. According to the target tracking category, the detector is used to perform target detection; the target detector can also be a detector in other tracking modes, which is not limited in the embodiments of the present application.

[0035] Step 120: Obtain an image feature vector and a text feature vector of the detected object in the first image frame.

[0036] Among them, the text feature vector is obtained through the text feature extraction module, and the text feature extraction module is trained based on a data set without text annotations.

[0037] It can be understood that the first image frame may be the image frame currently being processed, or any image frame in a video image sequence.

[0038] In this embodiment, feature extraction is performed on the first image frame to obtain an image feature vector and a text feature vector of the detection object in the first image frame, wherein the image feature vector is a feature representation used to characterize visual characteristics such as color, texture, and shape of the detection object, and the text feature vector is a feature representation used to characterize semantic characteristics of the detection object.

[0039] In this step, the target detection result of the first image frame can be sent to the image feature extraction module and the text feature extraction module respectively for feature extraction to obtain the image feature vector and text feature vector of the detected object in the first image frame.

[0040] The image feature extraction module may be any Re-Identification (ReID) feature extraction module, such as using a fast-ReID model to extract appearance features.

[0041] It should be noted that the text feature extraction module is trained based on a dataset without text annotations. No text annotations means that the images in the dataset do not have corresponding manually annotated text labels. The text feature extraction module can learn the semantic feature representation of the input image through methods such as contrastive learning, without relying on manually provided text descriptions or labels.

[0042] Step 130 : Obtain a first feature similarity between the image feature vector and the image feature template of the trajectory to be matched, and obtain a second feature similarity between the text feature vector and the text feature template of the trajectory to be matched.

[0043] It can be understood that the image feature template of the trajectory to be matched is a set of visual features extracted from the video image sequence that can represent a specific target trajectory, and the text feature template is a set of semantic features extracted from the video image sequence that can represent the specific target trajectory.

[0044] In this embodiment, the first image frame is an image frame for trajectory matching, and the image feature template and text feature template of the trajectory to be matched can be extracted based on the image frame before the first image frame in the video image sequence.

[0045] In this step, feature similarity distances are calculated based on the image feature vector and the text feature vector respectively to obtain a first feature similarity between the image feature vector and the image feature template of the trajectory to be matched, and a second feature similarity between the text feature vector and the text feature template of the trajectory to be matched.

[0046] In actual implementation, the cosine distance can be used to calculate the feature similarity distance.

[0047] The calculation formula for the first feature similarity is as follows:

[0048]

[0049] in, Indicates the detection object i The image feature vector at frame t and the trajectory to be matched j Corresponding image feature template The feature similarity distance between them.

[0050] It is understandable that The larger the distance value, the closer the detection object is. i and the trajectory to be matchedj The more dissimilar they are, The smaller the distance value, the closer the detection object is. i and the trajectory to be matched j The more similar they are, Represents the image feature vector and image feature templates The greater the similarity, the smaller the image feature similarity distance.

[0051] The calculation formula for the second feature similarity is as follows:

[0052]

[0053] in, Indicates the detection object i The text feature vector at the tth frame and the trajectory to be matched j Corresponding text feature template The feature similarity distance between them.

[0054] It is understandable that The larger the distance value, the closer the detection object is. i and the trajectory to be matched j The more dissimilar they are, The smaller the distance value, the closer the detection object is. i and the trajectory to be matched j The more similar they are, Represents text feature vector and text feature templates The greater the similarity, the smaller the text feature similarity distance.

[0055] Step 140 : Perform tracking and matching based on the first feature similarity and the second feature similarity to obtain a trajectory tracking result of the detected object in the first image frame and the trajectory to be matched.

[0056] It should be noted that the image feature vector can be the basic visual attributes extracted directly from the original pixel data, that is, it belongs to the underlying features and has a clear geometric or physical meaning; the text feature vector is a semantic representation abstracted from the underlying features, and the similarity distance of the tracking target is calculated from the two feature levels of the underlying image and the high-level semantics.

[0057] In this embodiment, multi-target tracking learning is performed using multi-modality of images and texts, and tracking matching is performed by combining the first feature similarity and the second feature similarity. Compared with a single image feature, it can better adapt to the influence of noise such as scale change, deformation, and illumination change on the tracking target. The text feature extraction module for extracting text feature vectors is trained based on a data set without text annotations. While reducing the workload of manual annotation, it can also avoid errors caused by manual annotation and improve the performance of multi-target tracking.

[0058] In actual implementation, the calculation formula for tracking and matching based on the similarity of the first feature and the second feature is as follows:

[0059]

[0060] in, The distance is the detection object obtained by combining the image feature vector and the text feature vector i and the trajectory to be matched j The similarity distance between The smaller the distance value, the closer the detection object is. i and the trajectory to be matched j The higher the probability of belonging to the same trajectory, and is the weighted hyperparameter, .

[0061] In actual implementation, for the first image frame, the first feature similarity and the second feature similarity between all detected objects and each track to be matched can be traversed and calculated to perform tracking and matching, determine the association relationship between each detected object and each track to be matched in the first image frame, and obtain the track tracking result of the first image frame.

[0062] In related technologies, text features are used for target tracking by manually annotating text description information or text category labels. Although natural language text information is introduced, it needs to be manually set according to the application scenario, and there are experience biases, deployment and usage limitations, or it cannot be used for tracking multiple targets.

[0063] In an embodiment of the present application, in order to address the problem of reduced tracking performance due to noise such as scale change, deformation, and illumination change of the tracked target in multi-target tracking, image feature vectors and text feature vectors are used for tracking based on the multimodal idea. The extraction of text feature vectors does not require manual labeling. Tracking and matching are performed based on the feature vectors of each detected object and the feature templates of each track to be matched to complete multi-target tracking. While introducing text semantic features, it is not restricted by manually set text. The image feature vectors and text feature vectors are integrated to adapt to the influence of factors such as scale change, deformation, rapid motion, and illumination change of the tracked target, and different target objects between different tracks are better distinguished in time sequence, effectively improving the multi-target tracking performance.

[0064] According to the multi-target tracking method provided in the embodiment of the present application, by extracting the image feature vector and text feature vector of the detection object, calculating the feature similarity between the image feature vector and the text feature vector and the feature template, tracking and matching are performed according to the feature similarity corresponding to the image feature and the text feature, and outputting the tracking result, semantic features are introduced into multi-target tracking while not being restricted by manually set text. The image feature vector and the text feature vector are integrated to adapt to the influence of factors such as scale change, deformation, rapid movement, and illumination change of the tracked target, and different target objects between different trajectories are better distinguished in time sequence, thereby effectively improving the multi-target tracking performance.

[0065] In some embodiments, the text feature extraction module is a multimodal pre-training model that achieves alignment of image modality and text modality representations through contrastive learning.

[0066] Among them, the multimodal pre-training model is a model trained through a data set without text annotations, which learns the association representation and semantic alignment between different modal data. Contrastive learning is used in the pre-training stage to enable the model to have cross-modal (image modality and text modality) understanding capabilities.

[0067] In actual implementation, the multimodal pre-training model can be a contrastive language-image pre-training (CLIP) model, which maps images and texts into a shared vector space through contrastive learning to achieve semantic understanding between images and texts.

[0068] It should be noted that the use of multimodal pre-training models such as the CLIP model can achieve text feature extraction without manual text annotation. While introducing text feature information, it is not restricted by manual settings. At the same time, the multimodal pre-training model can reuse the existing conventional multi-target tracking database without the need for text annotation.

[0069] In some embodiments, the multimodal pre-trained model includes an image encoder, an adapter module, and a text encoder, and the text feature vector is obtained by the following steps:

[0070] Input the target area image corresponding to the detection object into the image encoder to obtain the image coding features output by the image encoder;

[0071] Input the image encoding feature into the adapter module to obtain the image feature offset vector output by the adapter module;

[0072] The image feature offset vector and the text description vector to be learned are input into the text encoder to obtain the text feature vector output by the text encoder.

[0073] In this embodiment, based on the detection frame (or target position annotation information) corresponding to the detection object, the target area image corresponding to the detection object can be captured from the first image frame and input into the image encoder of the multimodal pre-trained model to obtain the image coding features corresponding to the detection object.

[0074] Among them, the image encoder can convert the visual information in the image (such as color, texture, shape, etc.) into a structured numerical representation, and abstract the key characteristics of the image through mathematical description.

[0075] It should be noted that the image coding features are input into the adapter module to obtain the image feature offset vector output by the adapter module. The image feature offset vector is generated by guiding the coding features of the image instance (i.e., the image coding features), which facilitates the subsequent establishment of cross-modal semantic associations.

[0076] The text description vector to be learned refers to the text representation automatically optimized through the training or inference process. The text description vector is dynamically adjustable. The image feature offset vector and the text description vector to be learned are input into the text encoder. The text description vector can be adjusted accordingly based on the image feature offset vector.

[0077] A specific embodiment is described below.

[0078] like Figure 3 As shown, the video image sequence is input into the target detector frame by frame to obtain the target area image corresponding to the detection object, and then input into the image encoder of the multimodal pre-trained model to obtain the image coding features corresponding to the detection object.

[0079] The image coding features are input into the adapter module, and the adapter module outputs an image feature offset vector guided by the image instance, which is input into the text encoder together with the text description vector to be learned, and outputs a text feature vector guided by the instance.

[0080] In this embodiment, after obtaining the image feature offset vector guided by the image instance, the image feature offset vector and the text description vector to be learned are input into the text encoder of the multimodal pre-training model, and the text encoder generates a text feature vector guided by the instance. Compared with a single image feature, multi-target tracking that integrates the text feature vector guided by the instance can better adapt to the influence of noise such as scale change, deformation, and illumination change on the tracking target, thereby improving the multi-target tracking performance.

[0081] In some embodiments, the multimodal pre-trained model is trained in the following manner:

[0082] Input the dataset without text annotations into the model to be trained;

[0083] Obtain the target loss function of the model to be trained. The target loss function is the sum of the loss values ​​of each sample tracking trajectory in the sample tracking sequence of the data set without text annotations.

[0084] Update the parameters of the adapter module and the text description vector to be learned until the objective loss function of the model to be trained is minimized, and a trained multimodal pre-training model is obtained.

[0085] In actual implementation, the dataset used for multimodal pre-training model training can be a conventionally annotated sample tracking sequence, such as Crowd Human, MOT17, MOT20 and other video databases. It is only necessary to perform position and ID annotation on the video image targets. ID annotation means that the same target uses the same serial number. There is no need to provide a text description of the target trajectory sequence, which can achieve the reuse of existing video databases (without text annotations) and also reduce the workload and annotation errors of self-calibration database personnel.

[0086] It can be understood that the model to be trained includes an image encoder, an adapter module and a text encoder. Training is the process of updating the parameters of the adapter module and the text description vector to be learned. The initial text description vector can be assigned by random initialization.

[0087] In this embodiment, an area image of the target to be tracked is obtained by labeling information (position information), and the area image is input into an image encoder for image feature encoding. During the training phase, an image encoder of a visual modality pre-trained by the CLIP model can be used, that is, during the training process, the parameters of the image encoder do not need to be updated and learned.

[0088] It should be noted that the loss value of the sample tracking trajectory is a quantitative indicator that measures the difference between the predicted matching target trajectory and the true trajectory. The sum of the loss values ​​of each sample tracking trajectory in the sample tracking sequence is used as the target loss function. With the goal of minimizing the target loss function, the parameters of the adapter module and the text description vector to be learned are updated so that the extracted text feature vector can achieve trajectory tracking more accurately.

[0089] In some embodiments, the loss value of the sample tracking trajectory is determined based on the first text feature similarity and the second text feature similarity.

[0090] The first text feature similarity is the average similarity of text feature vectors of multiple consecutive frames of the sample tracking trajectory, and the second text feature similarity is the similarity between the text feature vector of the sample tracking trajectory and the text feature vectors of other tracking trajectories in the sample tracking sequence.

[0091] It should be noted that the first text feature similarity can be expressed as the similarity distance between different sample frames of the same track, and the second text feature similarity can be expressed as the similarity distance between different tracks in the same sample frame.

[0092] In some embodiments, minimizing the objective loss function of the model to be trained is achieved by maximizing the first text feature similarity and minimizing the second text feature similarity.

[0093] In the process of text feature learning, the first text feature similarity is maximized (i.e., the similarity distance between the same trajectory in different sample frames is minimized), and the second text feature similarity is minimized (the similarity distance between different trajectories in the same sample frame is maximized). Minimizing the target loss function is obtained by maximizing the similarity of the text feature vector of the same trajectory and minimizing the similarity of the text feature vector between different trajectories. The text feature vector learned in this way is more discriminative and can better adapt to multi-target tracking of different objects in the same category. At the same time, the text feature vector can ensure the stability of the same trajectory feature and be updated in real time according to the changes in the target sample. It is more timely and can more effectively cope with the noise influence of environmental factors such as scale and lighting.

[0094] In some embodiments, the objective loss function is calculated as follows:

[0095]

[0096] in, Represents the target loss function, that is, the sample tracking sequence M The sum of the loss values ​​of the sample tracking trajectories, Represents the sample tracking trajectory The loss value.

[0097] In some embodiments, the loss value of the sample tracking trajectory is calculated as follows:

[0098]

[0099] in, Represents the sample tracking trajectory The first text feature similarity, that is, the sample tracking trajectory The average similarity of text feature vectors of multiple consecutive frames, Represents the sample tracking trajectory The second text feature similarity, that is, the sample tracking trajectory The similarity between the text feature vector of and the text feature vectors of other tracking tracks in the sample tracking sequence.

[0100] In some embodiments, the first text feature similarity The calculation formula is as follows:

[0101]

[0102] in, Represents the sample tracking trajectory In the ( i - k ) frame’s text feature vector, Represents the sample tracking trajectory exist N The average value of the text feature vector of the frame.

[0103] First text feature similarity Represents the same tracking object (sample tracking trajectory The corresponding target object) in a section of length N In a video sequence, the similarity value between the text feature vector of each frame and the average text feature vector corresponding to the video sequence can be calculated using the cosine function.

[0104] It is understandable that The larger the similarity value, the smaller the change in the text feature vector of the same trajectory.

[0105] In some embodiments, .

[0106] In some embodiments, the second text feature similarity The calculation formula is as follows:

[0107]

[0108] in, Indicates the sample tracking trajectory in the sample tracking sequence The average value of the text feature vectors of all tracking tracks other than (each tracking track has a corresponding average value), Represents the sample tracking trajectory The average value of the text feature vector.

[0109] Second text feature similarity The representation of a length of N On the video sequence, M The similarity value between the average text feature vectors of different trajectories can be calculated using the cosine function.

[0110] It is understandable that The smaller the similarity value, the greater the difference in the text feature vectors between different tracking trajectories, and the better the different trajectories can be distinguished.

[0111] During the training process, minimizing the loss is achieved by maximizing the similarity of the text feature vectors of the same trajectory and minimizing the similarity of the text feature vectors between different trajectories. Finally, the parameters of the text feature extraction module (including the parameters of the adapter module and the parameters of the text description vector) are learned. This can not only bring the same trajectory closer, but also distinguish different trajectories, effectively improving tracking performance. Compared with the lower-level image features, the features are more robust.

[0112] In some embodiments, the adapter module includes two linear fully connected layers and an activation layer connected between the two linear fully connected layers.

[0113] In this embodiment, Figure 4 As shown in the figure, the adapter module includes two linear fully connected layers (FC layers) and an activation layer (ReLU layer) connected between the two linear fully connected layers, which is used to learn the image feature offset vectors between different image instance sequences to facilitate the subsequent establishment of cross-modal semantic associations.

[0114] In actual implementation, the calculation formula of the adapter module is as follows:

[0115]

[0116] in, represents the image feature offset vector output by the adapter module, 、 are the weight parameters that need to be trained for the linear fully connected layers FC1 and FC2, Represents the image encoding features output by the image encoder.

[0117] In this embodiment, the calculated instance-guided image feature offset vector and the text description vector to be learned are input into a text encoder to generate an instance-guided text feature vector.

[0118] In actual implementation, the text encoder is calculated as follows:

[0119]

[0120] in, is the text feature vector based on instance guidance, is the text description vector to be learned, is the target category, is the image feature offset vector guided by image instances.

[0121] In an embodiment of the present application, in order to address the problems existing in manually setting text labels or text description information, a multimodal pre-training model is used to perform text feature extraction and learning based on the principle that text vectors of samples of the same trajectory are similar and text vectors of samples of different trajectories are different. There is no need to manually set text labels or text description information, and the existing multi-target tracking database can be effectively reused without the need for text annotation. At the same time, during the learning process, the text feature vector can be fine-tuned according to the real-time status of the tracked target object, so that the text feature vector can better adapt to the influence of factors such as scale changes, deformation, rapid motion, and illumination changes of the tracked target, and better distinguish different target objects between different trajectories in terms of time sequence.

[0122] In some embodiments, after obtaining the trajectory tracking results of the detected objects and the trajectories to be matched in the first image frame, the multi-target tracking method may further include:

[0123] The image feature template and the text feature template are updated for tracking and matching of a second image frame, where the second image frame is a subsequent image frame adjacent to the first image frame in the video image sequence.

[0124] In this embodiment, after determining the trajectory tracking result of the first image frame, the image feature template and the text feature template are updated so as to dynamically adapt to the appearance changes and scene interference of the target when tracking and matching is performed in the second image frame. While retaining the key target features, the template integrates the effective information of the first image frame to ensure the accurate representation of the target during the tracking process and suppress the interference of background noise.

[0125] In some embodiments, the image feature template is updated based on image feature vectors of a preset number of consecutive image frames, where the preset number of consecutive image frames includes a first image frame and an image frame preceding the first image frame in a video image sequence.

[0126] In this embodiment, the average value of the image feature vectors in a plurality of consecutive frames is calculated as the latest image feature template.

[0127] In actual implementation, the preset number is K, and the image feature vectors of the first image frame and the K-1 image frames before the first image frame in the video image sequence can be obtained to update the image feature template to facilitate tracking and matching of the second image frame.

[0128] For example, the first image frame is the 5th image frame in the video image sequence, and the preset number is 4. Then, the image feature vectors of the image frames from the 2nd to the 5th frame in the video image sequence are obtained, and the image feature template is updated to facilitate tracking and matching of the 6th image frame.

[0129] In actual implementation, the image feature template update process is shown in the following formula:

[0130]

[0131] in, Represents trajectory j The image feature template, Represents trajectory j In the ( t - k ) frame’s image feature vector, K is the preset quantity, is the image feature vector of the current frame (i.e. the first image frame), and the image feature template is composed of the trajectory j The latest series K The feature vectors of the images are averaged.

[0132] It can be understood that updating the image feature template can reduce the impact of noise on the target during tracking and matching.

[0133] In some embodiments, the text feature template is updated based on the text feature vector of the first image frame.

[0134] In this embodiment, the text feature template is updated with the latest text feature vector of the current frame, that is, the text feature template is updated in real time. The text feature extraction module is based on real-time tracking of the target while ensuring the highest similarity with the text information of the same trajectory and the lowest similarity with the text information of different trajectories. That is, the latest text feature vector can not only reflect the overall semantic characteristics of the trajectory, but also reflect the real-time semantic change characteristics of the current frame, and can effectively deal with the influence of noise such as scale change, deformation, and illumination change within the local time range.

[0135] In actual implementation, the text feature template update process is shown in the following formula:

[0136]

[0137] in, Represents trajectory j The text feature template, Represents trajectory j In the t The text feature vector of the frame, that is, the text feature vector of the current frame (that is, the first image frame).

[0138] A specific embodiment is described below.

[0139] like Figure 2 As shown, the video image sequence is input into the target detector frame by frame to obtain the target detection result output by the target detector. The target detection result may include the detected object in the image frame.

[0140] The detection object is input into the image feature extraction module and the text feature extraction module for feature extraction respectively, and the image feature similarity and text feature similarity are calculated respectively. Tracking and matching are performed based on the historical image feature template and text feature template. After obtaining the tracking result, the tracking position of the trajectory is updated, and the image feature template and text feature template are updated.

[0141] An embodiment of the present application also provides a multi-target tracking device.

[0142] like Figure 5 As shown, the multi-target tracking device includes:

[0143] A first processing module 510 is configured to perform target detection on each image frame of the video image sequence to obtain a target detection result for each image frame of the video image sequence, wherein the target detection result includes a detection object of the image frame;

[0144] A second processing module 520 is configured to obtain an image feature vector and a text feature vector of the detected object in the first image frame, wherein the text feature vector is obtained by a text feature extraction module trained based on a data set without text annotations;

[0145] The third processing module 530 is used to obtain a first feature similarity between the image feature vector and the image feature template of the track to be matched, and obtain a second feature similarity between the text feature vector and the text feature template of the track to be matched;

[0146] The fourth processing module 540 is configured to perform tracking and matching based on the first feature similarity and the second feature similarity to obtain a trajectory tracking result of the detected object in the first image frame and the trajectory to be matched.

[0147] According to the multi-target tracking device provided in the embodiment of the present application, by extracting the image feature vector and text feature vector of the detection object, calculating the feature similarity between the image feature vector and the text feature vector and the feature template, tracking and matching are performed according to the feature similarity corresponding to the image feature and the text feature, and the tracking result is output. While introducing semantic features in multi-target tracking, it is not restricted by manually set text, and integrates the image feature vector and the text feature vector to adapt to the influence of factors such as scale change, deformation, rapid movement, and illumination change of the tracked target, so as to better distinguish different target objects between different trajectories in time sequence, and effectively improve the multi-target tracking performance.

[0148] In some embodiments, the text feature extraction module is a multimodal pre-training model that achieves alignment of image modality and text modality representations through contrastive learning.

[0149] In some embodiments, the multimodal pre-trained model includes an image encoder, an adapter module, and a text encoder, and the text feature vector is obtained by the following steps:

[0150] Input the target area image corresponding to the detection object into the image encoder to obtain the image coding features output by the image encoder;

[0151] Input the image encoding feature into the adapter module to obtain the image feature offset vector output by the adapter module;

[0152] The image feature offset vector and the text description vector to be learned are input into the text encoder to obtain the text feature vector output by the text encoder.

[0153] In some embodiments, the multimodal pre-trained model is trained in the following manner:

[0154] Input the dataset without text annotations into the model to be trained;

[0155] Obtain the target loss function of the model to be trained. The target loss function is the sum of the loss values ​​of each sample tracking trajectory in the sample tracking sequence of the data set without text annotations.

[0156] Update the parameters of the adapter module and the text description vector to be learned until the objective loss function of the model to be trained is minimized, and a trained multimodal pre-training model is obtained.

[0157] In some embodiments, the loss value of the sample tracking trajectory is determined based on a first text feature similarity and a second text feature similarity, where the first text feature similarity is the average similarity of the text feature vectors of multiple consecutive frames of the sample tracking trajectory, and the second text feature similarity is the similarity between the text feature vector of the sample tracking trajectory and the text feature vectors of other tracking trajectories in the sample tracking sequence.

[0158] In some embodiments, minimizing the objective loss function of the model to be trained is achieved by maximizing the first text feature similarity and minimizing the second text feature similarity.

[0159] In some embodiments, the adapter module includes two linear fully connected layers and an activation layer connected between the two linear fully connected layers.

[0160] In some embodiments, after obtaining the trajectory tracking results of the detected object and the trajectory to be matched in the first image frame, the fourth processing module 540 is also used to update the image feature template and the text feature template for tracking and matching of the second image frame, where the second image frame is the next image frame adjacent to the first image frame in the video image sequence.

[0161] In some embodiments, the image feature template is updated based on image feature vectors of a preset number of consecutive image frames, where the preset number of consecutive image frames includes a first image frame and an image frame preceding the first image frame in a video image sequence.

[0162] In some embodiments, the text feature template is updated based on the text feature vector of the first image frame.

[0163] like Figure 6 As shown, an embodiment of the present application also provides an electronic device 600, including a processor 601, a memory 602, and a computer program stored in the memory 602 and executable on the processor 601. When the program is executed by the processor 601, each process of the above-mentioned multi-target tracking method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0164] An embodiment of the present application further provides a non-volatile computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned multi-target tracking method embodiments when running.

[0165] In an exemplary embodiment, the non-volatile computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0166] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above-mentioned multi-target tracking method embodiments are implemented.

[0167] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps in any of the above-mentioned multi-target tracking method embodiments.

[0168] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0169] The above is a detailed introduction to a multi-target tracking method and program product provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core ideas of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A multi-target tracking method, characterized in that: The following steps are involved: Performing target detection on each image frame of the video image sequence to obtain a target detection result for each image frame of the video image sequence, wherein the target detection result includes a detection object of the image frame; Obtaining an image feature vector and a text feature vector of the detected object in the first image frame, wherein the text feature vector is obtained by a text feature extraction module, and the text feature extraction module is trained based on a data set without text annotations; Obtaining a first feature similarity between the image feature vector and the image feature template of the to-be-matched trajectory, and obtaining a second feature similarity between the text feature vector and the text feature template of the to-be-matched trajectory; Performing tracking and matching based on the first feature similarity and the second feature similarity to obtain a trajectory tracking result of the detected object in the first image frame and the trajectory to be matched; The text feature extraction module is a multimodal pre-training model that achieves alignment of image modality and text modality representation through contrastive learning; The multimodal pre-trained model includes an image encoder, an adapter module, and a text encoder, and the text feature vector is obtained by the following steps: Inputting the target area image corresponding to the detection object into the image encoder to obtain the image coding features output by the image encoder; Inputting the image coding feature into the adapter module to obtain the image feature offset vector output by the adapter module; The image feature offset vector and the text description vector to be learned are input into the text encoder to obtain the text feature vector output by the text encoder.

2. The multi-target tracking method according to claim 1, characterized in that: The multimodal pre-training model is trained in the following way: Inputting the data set without text annotation into the model to be trained; Obtaining a target loss function of the model to be trained, where the target loss function is the sum of loss values ​​of each sample tracking trajectory in the sample tracking sequence of the data set without text annotations; Update the parameters of the adapter module and the text description vector to be learned until the target loss function of the model to be trained is minimized, thereby obtaining the trained multimodal pre-training model.

3. The multi-target tracking method according to claim 2, characterized in that: The loss value of the sample tracking trajectory is determined based on a first text feature similarity and a second text feature similarity, wherein the first text feature similarity is the average similarity of the text feature vectors of multiple consecutive frames of the sample tracking trajectory, and the second text feature similarity is the similarity between the text feature vector of the sample tracking trajectory and the text feature vectors of other tracking trajectories in the sample tracking sequence.

4. The multi-target tracking method according to claim 3, characterized in that: Minimizing the target loss function of the model to be trained is achieved by maximizing the first text feature similarity and minimizing the second text feature similarity.

5. The multi-target tracking method according to claim 1, characterized in that: The adapter module includes two linear fully connected layers and an activation layer connected between the two linear fully connected layers.

6. The multi-target tracking method according to any one of claims 1 to 5, characterized in that: After obtaining the trajectory tracking result of the detected object and the trajectory to be matched in the first image frame, the method further includes: The image feature template and the text feature template are updated for tracking and matching with a second image frame, where the second image frame is a subsequent image frame adjacent to the first image frame in the video image sequence.

7. The multi-target tracking method according to claim 6, characterized in that: The image feature template is updated based on the image feature vectors of a preset number of consecutive image frames, where the preset number of consecutive image frames include the first image frame and an image frame preceding the first image frame in the video image sequence.

8. The multi-target tracking method according to claim 6, characterized in that: The text feature template is updated based on the text feature vector of the first image frame.

9. A multi-target tracking device, characterized in that: include: A first processing module is configured to perform target detection on each image frame of a video image sequence to obtain a target detection result for each image frame of the video image sequence, wherein the target detection result includes a detection object of the image frame; A second processing module is configured to obtain an image feature vector and a text feature vector of the detected object in the first image frame, wherein the text feature vector is obtained by a text feature extraction module, and the text feature extraction module is trained based on a data set without text annotations; a third processing module, configured to obtain a first feature similarity between the image feature vector and the image feature template of the to-be-matched trajectory, and to obtain a second feature similarity between the text feature vector and the text feature template of the to-be-matched trajectory; a fourth processing module, configured to perform tracking and matching based on the first feature similarity and the second feature similarity, to obtain a trajectory tracking result of the detected object in the first image frame and the trajectory to be matched; The text feature extraction module is a multimodal pre-training model that achieves alignment of image modality and text modality representation through contrastive learning; The multimodal pre-trained model includes an image encoder, an adapter module, and a text encoder, and the text feature vector is obtained by the following steps: Inputting the target area image corresponding to the detection object into the image encoder to obtain the image coding features output by the image encoder; Inputting the image coding feature into the adapter module to obtain the image feature offset vector output by the adapter module; The image feature offset vector and the text description vector to be learned are input into the text encoder to obtain the text feature vector output by the text encoder.

10. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multi-target tracking method according to any one of claims 1 to 8.

11. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the multi-target tracking method according to any one of claims 1 to 8.

Images