A video multi-target tracking method, device, equipment and storage medium

Abstract

The application discloses a video multi-target tracking method and device, equipment and storage medium, and relates to the technical field of computer vision, which comprises the following steps: performing target detection on image frames corresponding to a current video image sequence; inputting the target detection result into a trained image feature extraction model based on language text information guidance to extract image features of each target to be tracked to obtain current image features; based on the image features and position information of each target object in the last image frame corresponding to the current video image sequence, and the current image features and target position information of any target to be tracked in the current image frame, performing trajectory matching calculation on any target to be tracked in the video image sequence to obtain multiple trajectory matching results, and outputting a multi-target tracking trajectory according to the multiple trajectory matching results corresponding to each target to be tracked. The application can reduce the influence of image noise on video multi-target tracking and improve the tracking performance and efficiency of multi-target tracking.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a video multi-target tracking method, apparatus, device, and storage medium. Background Technology

[0002] Multiple object tracking (MOT), a key technology in computer vision, has been widely applied in autonomous driving, video surveillance, and behavior recognition. However, while current MOT technologies contain specific, informative, and flexible visual features, these features are susceptible to environmental or deformation factors. For example, the tracked target may introduce image noise due to scale changes, deformation, occlusion, rapid movement, and lighting variations, thus affecting the MOT results.

[0003] Currently, to improve the performance of multi-target tracking, multi-target tracking methods based on natural language text information have been proposed. Natural language text information, due to its high semantic representation ability, is different from visual images and is not easily affected by image noise such as target deformation and ambient lighting. However, current multi-target tracking methods based on natural language text information require manual settings according to the application scenario when introducing text information, which has limitations. Furthermore, for each frame of image, its text information needs to be obtained based on the image information. Therefore, when the image information is affected by noise, the text information will also be affected, thereby reducing the original robustness of the text information. In addition, the prior information used by this method is empirical knowledge, which is easily incomplete due to cognitive bias, thus introducing errors. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a video multi-target tracking method, apparatus, device, and storage medium, which can reduce the impact of image noise on video multi-target tracking, improve the tracking performance of multi-target tracking, and increase the efficiency of multi-target tracking. The specific solution is as follows:

[0005] Firstly, this application discloses a video multi-object tracking method, including:

[0006] When a video image sequence to be tracked for multiple targets is received, target detection is performed on the image frames corresponding to the current video image sequence to obtain a target detection result containing multiple targets to be tracked and their corresponding target position information.

[0007] The target detection results are input into the trained image feature extraction model to extract the image features of each target to be tracked in the current image frame, thereby obtaining the current image features corresponding to each target to be tracked. The trained image feature extraction model includes a first model and a second model. The first model is a model for performing image feature extraction, and the second model is a model obtained by training a neural network model guided by language text information using a tracking training dataset.

[0008] Based on the image features and position information of each target object in the previous frame of the current video image sequence, and the current image features and target position information of any target to be tracked in the current image frame, trajectory matching calculation is performed on any target to be tracked in the video image sequence to obtain multiple trajectory matching results. The tracking trajectory of multiple targets to be tracked in the video image sequence is output according to the multiple trajectory matching results corresponding to each target to be tracked.

[0009] Optionally, the video multi-target tracking method further includes:

[0010] A historical image sequence to be tracked for multi-target tracking is obtained to form a tracking training dataset; each image frame in the tracking training dataset is labeled with the text attribute information and location information of the target object;

[0011] The tracking training dataset is input into the initial image feature extraction model to extract the image features of the target object in each image frame of the tracking training dataset using the first model in the initial image feature extraction model, thereby obtaining the first image features;

[0012] Based on the text attribute information of the target object in each image frame of the tracking training dataset, generate text prompt information for the target object;

[0013] The neural network model for image feature adjustment is trained based on the first image features corresponding to the same target object in each image frame of the tracking training dataset and the text prompt information to obtain a second model. The first model and the trained second model are then used as the trained image feature extraction model guided by language text information.

[0014] Optionally, the step of training a neural network model for image feature adjustment based on the first image features corresponding to the same target object in each image frame of the tracking training dataset and the text prompt information to obtain a second model includes:

[0015] The text prompt information is converted into text encoded information by a text encoder, and features are extracted from the text encoded information to obtain text features;

[0016] The neural network model for image feature adjustment is trained based on the first image features and the text features corresponding to the same target object in each image frame of the tracking training dataset to obtain a second model guided by language text information.

[0017] Optionally, the training process of the initial image feature extraction model further includes:

[0018] Obtain the second image features output by the second model guided by language text information after feature adjustment of the first image features;

[0019] Calculate the similarity between the first image feature and the second image feature corresponding to the same target object to obtain the first similarity;

[0020] Calculate the cosine similarity between the second image feature and the corresponding text feature to obtain the second similarity;

[0021] The loss function of the neural network model is calculated based on the first similarity and the second similarity, and the model parameters of the neural network model are updated using the loss function.

[0022] Optionally, the step of performing trajectory matching calculation on any target in the video image sequence based on the image features and position information of each target object in the previous frame image corresponding to the current video image sequence, and the current image features and target position information corresponding to any target to be tracked in the current image frame, to obtain multiple trajectory matching results, including:

[0023] Obtain the image features of each target object in the previous frame image corresponding to the current video image sequence from a pre-created image feature sequence template;

[0024] Based on the target location information corresponding to the target to be tracked and the historical location information corresponding to each target object in the previous frame image, the positional distance between each target to be tracked in the current video image sequence and multiple target objects in the previous frame image is calculated respectively, so as to obtain multiple first distances corresponding to each target to be tracked;

[0025] Calculate the Euclidean distance between the current image features and the image features of multiple target objects in the previous frame image, respectively, to obtain multiple second distances corresponding to each target to be tracked;

[0026] The target distance is obtained by weighted calculation based on the first distance and the second distance, and the Hungarian algorithm is used to perform trajectory matching calculation on multiple targets to be tracked in the video image sequence based on the target distance to obtain multiple trajectory matching results corresponding to each target to be tracked.

[0027] Optionally, after outputting the tracking trajectory of multiple targets in the video image sequence based on the multiple trajectory matching results corresponding to each of the aforementioned targets, the method further includes:

[0028] The image feature sequence templates corresponding to each of the targets to be tracked are updated based on the current image features.

[0029] Optionally, the step of performing a weighted calculation based on the first distance and the second distance to obtain the target distance includes:

[0030] The first distance and the second distance of the same target object and the same target to be tracked are respectively weighted and calculated to obtain multiple weighted calculated distances;

[0031] The maximum value among the multiple weighted calculated distances is determined, and based on the maximum value, the weighted calculated distances corresponding to the same target object and the same target to be tracked are normalized to obtain the target distances corresponding to the same target object and the same target to be tracked.

[0032] Secondly, this application discloses a video multi-target tracking device, comprising:

[0033] The target detection module is used to perform target detection on the image frames corresponding to the current video image sequence when a video image sequence to be tracked is received, and to obtain a target detection result containing multiple targets to be tracked and their corresponding target position information;

[0034] The image feature extraction module is used to input the target detection result into the trained image feature extraction model to extract the image features of each target to be tracked in the current image frame, and obtain the current image features corresponding to each target to be tracked; the trained image feature extraction model includes a first model and a second model; the first model is a model for performing image feature extraction, and the second model is a model obtained by training a neural network model based on language text information using a tracking training dataset;

[0035] The multi-target tracking module is used to perform trajectory matching calculations on any target in the video image sequence based on the image features and position information of each target object in the previous frame image corresponding to the current video image sequence, as well as the current image features and target position information corresponding to any target to be tracked in the current image frame, to obtain multiple trajectory matching results, and output the tracking trajectory of multiple targets in the video image sequence according to the multiple trajectory matching results corresponding to each target to be tracked.

[0036] Thirdly, this application discloses an electronic device, including a processor and a memory; wherein, when the processor executes a computer program stored in the memory, it implements the aforementioned video multi-target tracking method.

[0037] Fourthly, this application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned video multi-target tracking method.

[0038] As can be seen, when this application receives a video image sequence for multi-target tracking, it performs target detection on the image frame corresponding to the current video image sequence to obtain a target detection result containing multiple targets to be tracked and their corresponding target position information; the target detection result is input into a trained image feature extraction model to extract the image features of each target to be tracked in the current image frame to obtain the current image features corresponding to each target to be tracked; the trained image feature extraction model includes a first model and a second model; the first model is a model for image feature extraction, and the second model is a model obtained by training a neural network model based on language text information using a tracking training dataset; based on the image features and position information of each target object in the previous frame image corresponding to the current video image sequence, and the current image features and target position information corresponding to any target to be tracked in the current image frame, trajectory matching calculation is performed on any target to be tracked in the video image sequence to obtain multiple trajectory matching results, and the tracking trajectory of multiple targets to be tracked in the video image sequence is output according to the multiple trajectory matching results corresponding to each target to be tracked. This application employs an image feature extraction model guided by natural language text information to extract image features of each target to be tracked in the image frames corresponding to the current video image sequence. Then, based on the image features and position information of each target object in the previous frame corresponding to the current video image sequence, and the current image features and target position information of any target to be tracked in the current image frame, trajectory matching calculation is performed on any target to be tracked in the video image sequence to obtain multiple trajectory matching results. Based on the multiple trajectory matching results corresponding to each target to be tracked, the target tracking trajectory of all targets to be tracked in the current image frame is output. By introducing natural language features and employing a multi-target image feature extraction model guided by natural language information, this application can extract more discriminative image features, thereby reducing the impact of image noise on video multi-target tracking, improving the tracking performance of multi-target tracking, and increasing the efficiency of multi-target tracking. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0040] Figure 1 This is a flowchart of a video multi-target tracking method disclosed in this application;

[0041] Figure 2This is a schematic diagram of a specific image feature adjustment network training process disclosed in this application;

[0042] Figure 3 This is a schematic diagram illustrating a specific text prompt message generation method disclosed in this application;

[0043] Figure 4 This is a schematic diagram of a specific neural network model structure disclosed in this application;

[0044] Figure 5 This application discloses a specific video multi-target tracking method flowchart;

[0045] Figure 6 This application discloses a specific block diagram of a video multi-target tracking system;

[0046] Figure 7 This is a schematic diagram of the structure of a video multi-target tracking device disclosed in this application;

[0047] Figure 8 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0048] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0049] This application discloses a video multi-target tracking method. See also... Figure 1 As shown, the method includes:

[0050] Step S11: When a video image sequence to be tracked for multiple targets is received, target detection is performed on the image frames corresponding to the current video image sequence to obtain a target detection result containing multiple targets to be tracked and their corresponding target position information.

[0051] In this embodiment, when a video image sequence to be tracked (i.e., a video sequence transmitted over time, consisting of multiple frames) is received, a target detection algorithm is used to detect targets in the image frames corresponding to the currently received video image sequence. This yields a target detection result containing multiple targets to be tracked and their corresponding location information. In other words, the image frames corresponding to the video image sequence contain multiple targets to be tracked, and the target detection result also includes the location information of each target, such as the location of the target marked by a bounding box. The target detection algorithm includes, but is not limited to, YOLO (You Only Look Once) and Fast R-CNN (Region-based Convolutional Network, a fast target detection method based on region convolutional networks).

[0052] Step S12: Input the target detection result into the trained image feature extraction model to extract the image features of each target to be tracked in the current image frame, and obtain the current image features corresponding to each target to be tracked; the trained image feature extraction model includes a first model and a second model; the first model is a model for performing image feature extraction, and the second model is a model obtained by training a neural network model based on language text information using a tracking training dataset.

[0053] In this embodiment, after target detection is performed on the image frames corresponding to the current video image sequence, the target detection results, which contain information on multiple targets to be tracked and their corresponding positions, are further input into a pre-created trained image feature extraction model. This allows the first model in the trained image feature extraction model to extract image features of each target to be tracked in the current image frame, obtaining initial image features corresponding to each target. Furthermore, to improve the accuracy of multi-target tracking in the video, the initial image features output by the first model are adjusted using a second model guided by language text information in the trained image feature extraction model, thereby obtaining more accurate and favorable image features for multi-target tracking, and thus obtaining the current image features corresponding to each target to be tracked. It should be noted that the trained image feature extraction model includes two models: a first model and a second model. The first model is the trained model used for image feature extraction, and the second model is the model obtained by training a neural network model guided by language text information using a tracking training dataset. The purpose of adjusting features using the second model based on language text information is to extract features from multiple initial image features that are more relevant to the text information of the current image frame.

[0054] Specifically, the first model includes, but is not limited to, target re-identification network models, such as the Fast-ReID (FastPerson re-identification) network model, which uses computer vision algorithms to perform cross-camera tracking and can determine whether a specific pedestrian exists in a video sequence under different cameras. Specifically, it can identify visual features including edges, contours, shapes, textures, and regions.

[0055] Specifically, the process of acquiring the trained image feature extraction model may include: acquiring a historical image sequence to be tracked by multiple targets, obtaining a tracking training dataset; each image frame in the tracking training dataset is labeled with text attribute information and location information of the target object; inputting the tracking training dataset into an initial image feature extraction model, so as to use a first model in the initial image feature extraction model to extract image features of the target object in each image frame of the tracking training dataset, obtaining a first image feature; generating text prompt information for the target object based on the text attribute information corresponding to the target object in each image frame of the tracking training dataset; training a neural network model for image feature adjustment based on the first image feature and the text prompt information corresponding to the same target object in each image frame of the tracking training dataset, obtaining a second model, and using the first model and the trained second model as the trained image feature extraction model guided by language text information. In this embodiment, see Figure 2 As shown, a historical image sequence for multi-object tracking can be acquired first and used as a training set to obtain a tracking training dataset. It should be noted that this dataset can be a sequence of images already collected for multi-object tracking, and each image frame in this dataset is labeled with text attribute information and location information of different target objects. Next, the acquired tracking training dataset is input into an initial image feature extraction model. The first model in this initial image feature extraction model (such as a target re-identification network) extracts image features of multiple target objects within each image frame of the tracking training dataset, obtaining the first image features corresponding to each target object. Then, based on the multiple text attribute information corresponding to the same target object within multiple image frames of the tracking training dataset, a text prompt message for that target object is generated. It should be noted that this text prompt message is completely unaffected by the video images; therefore, the model learned based on this text prompt message (i.e., language text information) has higher robustness. For example, see... Figure 2As shown, the target attributes (i.e., text attribute information) corresponding to the target object in the current image frame include gender (male), age (middle-aged), clothing (grey short-sleeved shirt, brown trousers), presence or absence of glasses (no glasses), and bag accessories (black handbag). Then, all target attributes are merged into a complete text prompt message. For details, see [link to relevant documentation]. Figure 3 As shown, to improve the effectiveness and efficiency of feature extraction from the target to be tracked, multiple separate attribute features (i.e., text attribute information) can be fused into a pre-created array, such as... Figure 3 In the illustrated prompt description architecture, when a corresponding attribute exists, the attribute feature (i.e., text attribute information) is added to the corresponding position. If no corresponding attribute feature exists, such as the attribute feature of wearing glasses, it can be deleted. Therefore, the integrated text prompt information is: a middle-aged man carrying a black bag, wearing a gray short-sleeved shirt and brown trousers. Finally, the first image features and text prompt information corresponding to the same target object in each image frame of the above tracking training dataset will be used to train a neural network model based on multi-head attention (i.e., ... Figure 2 The first model (the image feature adjustment network in the video sequence) is trained to obtain a second model. The first model and the trained second model are then used as a trained image feature extraction model guided by language and text information. This model is then used to track the trajectories of multiple targets in a video image sequence in real time. Specifically, historical image sequences are used as the tracking training dataset to train an initial image feature extraction model containing the first model (the trained feature extraction model) and the second model (a neural network model based on self-attention). This is achieved by extracting image features of target objects from each image frame in the video sequence and using generated text prompts to train the neural network model used for image feature adjustment. This results in a trained image feature extraction model guided by language and text information. This model can handle the correlation information between multiple input image sequences and obtain more representative image features based on language and text information.

[0056] Specifically, the structure of the second model (i.e., the image feature adjustment model) is shown below. Figure 4As shown, the model includes N computational units with self-attention mechanisms. Each computational unit contains an attention layer, a first summing layer, a first normalization layer, a forward propagation layer, a second summing layer, and a second normalization layer, all connected sequentially. It should be noted that the input to the second model is the image feature vector calculated by the first model, and the output of the second model is the adjusted image feature vector calculated by the second model. Furthermore, during the training of the neural network model, the idea of ​​model distillation is employed. Specifically, linguistic text information is used to guide the learning of video image feature information. Thus, when using the trained image feature extraction model for multi-target tracking, only image feature extraction based on image frames of the video sequence is required, without introducing linguistic text information, thereby improving the computational efficiency of multi-target tracking. During the training of the image feature extraction model, this application only adjusts the parameters of the second model, i.e., the neural network model, while the parameters of the first model (such as the target re-identification network) remain unchanged, i.e., the parameters of the target re-identification network are fixed. This facilitates the rapid convergence of the entire model parameters (i.e., the trained image feature extraction model).

[0057] Specifically, training a neural network model for image feature adjustment based on the first image features corresponding to the same target object in each image frame of the tracking training dataset and the text prompt information to obtain a second model may include: converting the text prompt information into text encoding information through a text encoder, and extracting features from the text encoding information to obtain text features; training a neural network model for image feature adjustment based on the first image features corresponding to the same target object in each image frame of the tracking training dataset and the text features to obtain a second model guided by language text information.

[0058] In this embodiment, see Figure 2 As shown, during model training, after obtaining text prompts, the generated text prompts can be converted into text-encoded information using a text encoder. For example, the text prompts can be converted into text-encoded information (i.e., text feature sequences) using the text encoder of the robust multimodal CLIP model (Contrastive Language-Image Pre-Training, a pre-trained neural network model for matching images and text). Then, the neural network model used for image feature adjustment is trained based on the first image features and text-encoded information corresponding to the same target object in each image frame of the tracking training dataset, thereby obtaining the trained second model guided by language text information.

[0059] Furthermore, the training process of the initial image feature extraction model specifically includes: obtaining the second image features output by the second model guided by language text information after feature adjustment of the first image features; calculating the similarity between the first image features and the second image features corresponding to the same target object to obtain a first similarity; calculating the cosine similarity between the second image features and the corresponding text features to obtain a second similarity; calculating the loss function of the neural network model based on the first similarity and the second similarity, and updating the model parameters of the neural network model using the loss function. In this embodiment, see... Figure 2 As shown, during model training, the first image features corresponding to the target object in each image frame output by the first model (target re-identification network) can be adjusted by a second model (i.e., image feature adjustment network) guided by language text information to obtain the second image features. Then, the similarity between the first image features and the second image features corresponding to the same target object is calculated to obtain the first similarity. Next, the cosine similarity between the second image features and text features corresponding to the same target object is calculated to obtain the second similarity. Finally, based on the first similarity Second similarity Calculate the loss function of this neural network model (i.e., the image feature adjustment network). Then use this loss function The model parameters of the neural network model are updated to obtain the trained second model. As can be seen, this embodiment, after obtaining the image and text features of the same target object, calculates the model's loss function by measuring the similarity between images before and after adjustment, as well as between images and text. The model parameters are then adjusted using this loss function. This enhances the discriminability of visual features, improves the model's feature extraction ability and robustness, and increases the correlation between the extracted image features and the current image frame, thereby improving the performance of multi-target tracking.

[0060] The formula for calculating the loss function is as follows:

[0061] ;

[0062] In the formula, Let the loss function be... For the first similarity, The second similarity is given by n, where n is the length of a set of historical image sequences. and This is a hyperparameter.

[0063] Specifically, the first similarity The calculation formula is:

[0064] ;

[0065] in, Let I and I' represent the image feature vector before and after adjustment (i.e., the first image feature and the second image feature), respectively. T is a hyperparameter, and d(I,I') represents the Euclidean distance between the two feature vectors. Additionally, in... During the calculation process, when the distance between two features is greater than T, the distance between the two feature vectors is set to T.

[0066] Second similarity The calculation formula is:

[0067] ;

[0068] in, The loss value represents the similarity between the adjusted image features and text features, where θ represents the angle between the image feature vector and the text feature vector. The smaller the angle, the higher the similarity. The smaller it is.

[0069] It should be noted that during model training, a self-attention mechanism can also be used to calculate the cohesion between image feature vectors. The number of self-attention modules used can be set according to the actual application scenario. For example, for pedestrian tracking applications, the number of self-attention modules N=2.

[0070] Step S13: Based on the image features and position information of each target object in the previous frame image corresponding to the current video image sequence, and the current image features and target position information corresponding to any target to be tracked in the current image frame, perform trajectory matching calculation on any target to be tracked in the video image sequence to obtain multiple trajectory matching results, and output the tracking trajectory of multiple targets to be tracked in the video image sequence according to the multiple trajectory matching results corresponding to each target to be tracked.

[0071] In this embodiment, after extracting the image features of each target to be tracked in the current image frame using the trained image feature extraction model to obtain the current image features corresponding to each target to be tracked, trajectory matching calculation can be performed on any target to be tracked in the above video image sequence based on the image features and position information of each target object in the previous frame image corresponding to the current video image sequence, as well as the current image features and target position information corresponding to any target to be tracked in the current image frame, thereby obtaining multiple trajectory matching results corresponding to any target to be tracked. That is, the relationship between a target to be tracked in the current frame and each target object in the previous frame image is calculated respectively. Then, the tracking trajectory of all targets to be tracked in the current video image sequence (i.e., multi-target tracking trajectory sequence) is output according to the multiple trajectory matching results corresponding to each target to be tracked.

[0072] As can be seen, in this embodiment, when a video image sequence to be tracked for multiple targets is received, target detection is performed on the image frame corresponding to the current video image sequence to obtain a target detection result containing multiple targets to be tracked and their corresponding position information. The target detection result is input into a trained image feature extraction model to extract the image features of each target to be tracked in the current image frame to obtain the current image features corresponding to each target to be tracked. The trained image feature extraction model includes a first model and a second model. The first model is a model for image feature extraction, and the second model is a model obtained by training a neural network model based on language text information using a tracking training dataset. Based on the image features and position information of each target object in the previous frame image corresponding to the current video image sequence, and the current image features and target position information corresponding to any target to be tracked in the current image frame, trajectory matching calculation is performed on any target to be tracked in the video image sequence to obtain multiple trajectory matching results. The tracking trajectory of multiple targets to be tracked in the video image sequence is output according to the multiple trajectory matching results corresponding to each target to be tracked. This application embodiment employs an image feature extraction model guided by natural language text information to extract image features of each target to be tracked in the image frames corresponding to the current video image sequence. Then, based on the image features and position information of each target object in the previous frame corresponding to the current video image sequence, and the current image features and target position information of any target to be tracked in the current image frame, trajectory matching calculation is performed on any target to be tracked in the video image sequence to obtain multiple trajectory matching results. Based on the multiple trajectory matching results corresponding to each target to be tracked, the target tracking trajectory of all targets to be tracked in the current image frame is output. By introducing natural language features and employing a multi-target image feature extraction model guided by natural language information, this application embodiment can extract more discriminative image features, thereby reducing the impact of image noise on video multi-target tracking, improving the tracking performance of multi-target tracking, and increasing the efficiency of multi-target tracking.

[0073] This application discloses a specific video multi-target tracking method. (See also...) Figure 5 As shown, the method includes:

[0074] Step S21: When a video image sequence to be tracked for multiple targets is received, target detection is performed on the image frames corresponding to the current video image sequence to obtain a target detection result containing multiple targets to be tracked and their corresponding target position information.

[0075] Step S22: Input the target detection result into the trained image feature extraction model to extract the image features of each target to be tracked in the current image frame, and obtain the current image features corresponding to each target to be tracked; the trained image feature extraction model includes a first model and a second model; the first model is a model for performing image feature extraction, and the second model is a model obtained by training a neural network model based on language text information using a tracking training dataset.

[0076] Step S23: Obtain the image features of each target object in the previous frame image corresponding to the current video image sequence from the pre-created image feature sequence template.

[0077] In this embodiment, after extracting image features of each target to be tracked in the current image frame using the trained image feature extraction model, see [link to previous section]. Figure 6 As shown, the image features of each target object in the previous frame image corresponding to the current video image sequence can be obtained from a pre-created image feature sequence template.

[0078] Step S24: Based on the target position information corresponding to the target to be tracked and the historical position information corresponding to each target object in the previous frame image, calculate the positional distance between each target to be tracked in the current video image sequence and multiple target objects in the previous frame image, and obtain multiple first distances corresponding to each target to be tracked.

[0079] Furthermore, from Figure 6 The historical position information of each target object in the previous frame image is obtained from the target tracking box in the video image sequence. Then, based on the target position information of the target to be tracked and the historical position information of each target object in the previous frame image, the positional distance between each target to be tracked in the current video image sequence and multiple target objects in the previous frame image is calculated (that is, the positional distance between a current target and multiple historical trajectory objects is calculated), thereby obtaining multiple first distances corresponding to each target to be tracked.

[0080] Specifically, multiple first distances corresponding to any target to be tracked can be obtained by calculating the intersection-union ratio (IUU) between the target bounding box of any target to be tracked in the current video image sequence and the target bounding boxes of multiple target objects in the previous frame.

[0081] Step S25: Calculate the Euclidean distance between the current image features and the image features of multiple target objects in the previous frame image, respectively, to obtain multiple second distances corresponding to each target to be tracked.

[0082] Next, the Euclidean distance between the current image feature corresponding to a target to be tracked in the current video image sequence and the image features of multiple target objects in the previous frame image is calculated to obtain multiple second distances corresponding to each target to be tracked.

[0083] Step S26: Perform a weighted calculation based on the first distance and the second distance to obtain the target distance, and use the Hungarian algorithm to perform trajectory matching calculation on multiple targets to be tracked in the video image sequence based on the target distance to obtain multiple trajectory matching results corresponding to each target to be tracked, and output the tracking trajectory of multiple targets to be tracked in the video image sequence according to the multiple trajectory matching results corresponding to each target to be tracked.

[0084] In this embodiment, a target distance is first calculated by weighting the first distance and the second distance. Then, the Hungarian algorithm is used to perform trajectory matching calculation on multiple tracking targets in the current video image sequence based on the target distance, thereby obtaining multiple trajectory matching results corresponding to each target to be tracked. Finally, the tracking trajectories of multiple targets to be tracked in the current video image sequence are output according to the multiple trajectory matching results corresponding to all targets to be tracked. Specifically, the tracking trajectory can be described by ID annotation. For example, the same tracking object in different image frames can be annotated with ID1.

[0085] In this embodiment, the weighted calculation based on the first distance and the second distance to obtain the target distance may specifically include: performing weighted calculations on the first distance and the second distance for the same target object and the same target to be tracked, respectively, to obtain multiple weighted calculated distances; determining the maximum value among the multiple weighted calculated distances, and based on the maximum value, normalizing the weighted calculated distances corresponding to the same target object and the same target to be tracked, to obtain the target distances corresponding to the same target object and the same target to be tracked. In this embodiment, after obtaining the first distance and the second distance, a judgment distance (i.e., the target distance) can be calculated using the following weighting formula, and then, based on this judgment distance (i.e., the target distance), the Hungarian algorithm is used to perform trajectory matching calculations on multiple targets to be tracked in the video image sequence, thereby obtaining multiple trajectory matching results corresponding to each target to be tracked. The calculation formula for the weighted calculated distance is:

[0086] ;

[0087] In the formula, This represents the IOU distance (i.e., the first distance) between the target A to be tracked and the historical trajectory object 1. The normalized value of the Euclidean distance between the image feature sequence template vector of the target A to be tracked and the historical trajectory object 1 (i.e., the value after the second distance normalization) represents the value of the Euclidean distance between them. n represents the number of all historical trajectory objects (i.e. the target objects in the historical image frames).

[0088] It should be noted that, in order to reduce the impact of image noise, the weighted calculated distance corresponding to any target to be tracked can be normalized based on the maximum distance between any target and all historical trajectory objects to obtain the final judgment distance (i.e., target distance). In a specific embodiment, the calculation formula for the judgment distance (i.e., target distance) is as follows:

[0089] .

[0090] Furthermore, after outputting the tracking trajectory of multiple targets in the video image sequence based on the trajectory matching results corresponding to each of the targets to be tracked, the method may further include: updating the image feature sequence template corresponding to each target to be tracked based on the current image features. In this embodiment, after tracking multiple targets in the current video image sequence, the historical information in the image feature sequence template is updated based on the second image features output by the current model. The specific update formula is:

[0091] ;

[0092] in, This represents the image feature sequence template of the i-th tracked target at time (t-1). This represents the image features of the i-th tracked target frame at time t, which are used to update the historical image feature template through weighted calculation. The updated template can track targets in the next frame of the image.

[0093] For more detailed processing procedures of steps S21 to S22, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0094] As can be seen, in this embodiment, after extracting the image features of each target to be tracked in the current image frame through a trained image feature extraction model guided by language text information, the image features of each target object in the previous frame corresponding to the current video image sequence are obtained from a pre-created image feature sequence template. Based on the current position information and historical position information of the target to be tracked, the positional distance between each target to be tracked and multiple target objects in the previous frame is calculated to obtain multiple first distances. Then, the Euclidean distance between the current image features and the image features of multiple target objects in the corresponding previous frame is calculated to obtain multiple second distances. Finally, a target distance is obtained by weighted calculation based on the first distance and the second distance. The Hungarian algorithm is used to perform trajectory matching calculation on multiple targets to be tracked in the current video image sequence based on the target distance to obtain the tracking trajectory of multiple targets to be tracked. This application addresses multi-target tracking by introducing natural language features in addition to image features. It employs an image feature extraction model guided by language and text information to extract more discriminative image features for multi-target tracking, thereby improving the tracking performance. Furthermore, in real-time multi-target tracking, image feature extraction is only required based on image frames of the video sequence, eliminating the need to introduce language and text information, thus improving the efficiency of target tracking.

[0095] Accordingly, this application also discloses a video multi-target tracking device, see [link to relevant documentation]. Figure 7 As shown, the device includes:

[0096] The target detection module 11 is used to perform target detection on the image frames corresponding to the current video image sequence when a video image sequence to be tracked is received, and to obtain a target detection result containing multiple targets to be tracked and their corresponding target position information;

[0097] The image feature extraction module 12 is used to input the target detection result into the trained image feature extraction model to extract the image features of each target to be tracked in the current image frame, and obtain the current image features corresponding to each target to be tracked; the trained image feature extraction model includes a first model and a second model; the first model is a model for performing image feature extraction, and the second model is a model obtained by training a neural network model based on language text information using a tracking training dataset;

[0098] The multi-target tracking module 13 is used to perform trajectory matching calculations on any target in the video image sequence based on the image features and position information of each target object in the previous frame image corresponding to the current video image sequence, and the current image features and target position information corresponding to any target to be tracked in the current image frame, to obtain multiple trajectory matching results, and output the tracking trajectory of multiple targets in the video image sequence according to the multiple trajectory matching results corresponding to each target to be tracked.

[0099] The specific workflow of each of the above modules can be found in the relevant content disclosed in the foregoing embodiments, and will not be repeated here.

[0100] As can be seen, in this embodiment, when a video image sequence to be tracked for multiple targets is received, target detection is performed on the image frame corresponding to the current video image sequence to obtain a target detection result containing multiple targets to be tracked and their corresponding position information; the target detection result is input into a trained image feature extraction model to extract the image features of each target to be tracked in the current image frame to obtain the current image features corresponding to each target to be tracked; the trained image feature extraction model includes a first model and a second model; the first model is a model for performing image feature extraction, and the second model is a model obtained by training a neural network model based on language text information using a tracking training dataset; based on the image features and position information of each target object in the previous frame image corresponding to the current video image sequence, and the current image features and target position information corresponding to any target to be tracked in the current image frame, trajectory matching calculation is performed on any target to be tracked in the video image sequence to obtain multiple trajectory matching results, and the tracking trajectory of multiple targets to be tracked in the video image sequence is output according to the multiple trajectory matching results corresponding to each target to be tracked. This application embodiment employs an image feature extraction model guided by natural language text information to extract image features of each target to be tracked in the image frames corresponding to the current video image sequence. Then, based on the image features and position information of each target object in the previous frame corresponding to the current video image sequence, and the current image features and target position information of any target to be tracked in the current image frame, trajectory matching calculation is performed on any target to be tracked in the video image sequence to obtain multiple trajectory matching results. Based on the multiple trajectory matching results corresponding to each target to be tracked, the target tracking trajectory of all targets to be tracked in the current image frame is output. By introducing natural language features and employing a multi-target image feature extraction model guided by natural language information, this application embodiment can extract more discriminative image features, thereby reducing the impact of image noise on video multi-target tracking, improving the tracking performance of multi-target tracking, and increasing the efficiency of multi-target tracking.

[0101] In some specific embodiments, the video multi-target tracking device may further include:

[0102] The sequence acquisition unit is used to acquire historical image sequences to be tracked by multiple targets, and obtain a tracking training dataset; each image frame in the tracking training dataset is labeled with text attribute information and location information of the target object;

[0103] The dataset input unit is used to input the tracking training dataset into the initial image feature extraction model, so as to use the first model in the initial image feature extraction model to extract the image features of the target object in each image frame of the tracking training dataset, and obtain the first image features;

[0104] The prompt information generation unit is used to generate text prompt information for the target object based on the text attribute information of the target object in each image frame of the tracking training dataset.

[0105] The first model training unit is used to train a neural network model for image feature adjustment based on the first image features corresponding to the same target object in each image frame of the tracking training dataset and the text prompt information, to obtain a second model, and to use the first model and the trained second model as the trained image feature extraction model guided by language text information.

[0106] In some specific embodiments, the first model training unit may specifically include:

[0107] An information conversion unit is used to convert the text prompt information into text encoded information using a text encoder;

[0108] A text feature extraction unit is used to extract features from the text encoding information to obtain text features;

[0109] The second model training unit is used to train a neural network model for image feature adjustment based on the first image features and the text features corresponding to the same target object in each image frame of the tracking training dataset, so as to obtain a second model guided by language text information.

[0110] In some specific embodiments, the video multi-target tracking device may further include:

[0111] The feature adjustment unit is used to obtain the second image features output by the second model after the second model, guided by language text information, performs feature adjustment on the first image features;

[0112] The first similarity calculation unit is used to calculate the similarity between the first image feature and the second image feature corresponding to the same target object, and obtain the first similarity.

[0113] The second similarity calculation unit is used to calculate the cosine similarity between the second image feature and the corresponding text feature to obtain the second similarity.

[0114] The loss function calculation unit is used to calculate the loss function of the neural network model based on the first similarity and the second similarity;

[0115] The model parameter update unit is used to update the model parameters of the neural network model using the loss function.

[0116] In some specific embodiments, the multi-target tracking module 13 may specifically include:

[0117] The image feature acquisition unit is used to acquire the image features of each target object in the previous frame image corresponding to the current video image sequence from a pre-created image feature sequence template;

[0118] The first distance calculation unit is used to calculate the positional distance between each target to be tracked in the current video image sequence and multiple target objects in the previous frame image based on the target position information corresponding to the target to be tracked and the historical position information corresponding to each target object in the previous frame image, so as to obtain multiple first distances corresponding to each target to be tracked.

[0119] The second distance calculation unit is used to calculate the Euclidean distance between the current image features and the image features of multiple target objects in the previous frame image, respectively, to obtain multiple second distances corresponding to each target to be tracked.

[0120] The first weighted calculation unit is used to perform a weighted calculation based on the first distance and the second distance to obtain the target distance;

[0121] The trajectory matching unit is used to perform trajectory matching calculations on multiple targets to be tracked in the video image sequence based on the target distance using the Hungarian algorithm, and obtain multiple trajectory matching results corresponding to each target to be tracked.

[0122] In some specific embodiments, after the multi-target tracking module 13, the system may further include:

[0123] The template update unit is used to update the image feature sequence template corresponding to each of the targets to be tracked based on the current image features.

[0124] In some specific embodiments, the first weighted calculation unit may specifically include:

[0125] The second weighted calculation unit is used to perform weighted calculations on the first distance and the second distance of the same target object and the same target to be tracked, respectively, to obtain multiple weighted calculated distances;

[0126] A maximum value determination unit is used to determine the maximum value among the multiple weighted calculated distances;

[0127] The normalization unit is used to normalize the weighted calculated distances corresponding to the same target object and the same target to be tracked based on the maximum value, so as to obtain the target distances corresponding to the same target object and the same target to be tracked.

[0128] Furthermore, embodiments of this application also disclose an electronic device, Figure 8 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0129] Figure 8 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the video multi-target tracking method disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0130] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0131] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0132] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the video multi-target tracking method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0133] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned disclosed video multi-target tracking method. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0134] Furthermore, embodiments of this application also disclose a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the video multi-target tracking method disclosed above.

[0135] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0136] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0137] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0138] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0139] The above provides a detailed description of a video multi-target tracking method, apparatus, device, and storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A video multi-target tracking method, characterized in that, include: When a video image sequence to be tracked for multiple targets is received, target detection is performed on the image frames corresponding to the current video image sequence to obtain a target detection result containing multiple targets to be tracked and their corresponding target position information. The target detection results are input into the trained image feature extraction model to extract the image features of each target to be tracked in the current image frame, so as to obtain the current image features corresponding to each target to be tracked. The trained image feature extraction model includes a first model and a second model; the first model is a model used for image feature extraction, and the second model is a model obtained by training a neural network model guided by language text information using a tracking training dataset; Based on the image features and position information of each target object in the previous frame of the current video image sequence, and the current image features and target position information of any target to be tracked in the current image frame, trajectory matching calculation is performed on any target to be tracked in the video image sequence to obtain multiple trajectory matching results, and the tracking trajectory of multiple targets to be tracked in the video image sequence is output according to the multiple trajectory matching results corresponding to each target to be tracked. The method further includes: acquiring a historical image sequence to be tracked by multiple targets to obtain a tracking training dataset; each image frame in the tracking training dataset is labeled with text attribute information and location information of the target object; inputting the tracking training dataset into an initial image feature extraction model to extract image features of the target object in each image frame of the tracking training dataset using a first model in the initial image feature extraction model to obtain a first image feature; generating text prompt information for the target object based on the text attribute information corresponding to the target object in each image frame of the tracking training dataset; training a neural network model for image feature adjustment based on the first image feature and the text prompt information corresponding to the same target object in each image frame of the tracking training dataset to obtain a second model, and using the first model and the trained second model as the trained image feature extraction model guided by language text information; The step of training a neural network model for image feature adjustment based on the first image features corresponding to the same target object in each image frame of the tracking training dataset and the text prompt information to obtain a second model includes: converting the text prompt information into text encoded information through a text encoder, and extracting features from the text encoded information to obtain text features; and training a neural network model for image feature adjustment based on the first image features corresponding to the same target object in each image frame of the tracking training dataset and the text features to obtain a second model guided by language text information. The training process of the initial image feature extraction model further includes: obtaining the second image features output by the second model after feature adjustment of the first image features based on language text information; calculating the similarity between the first image features and the second image features corresponding to the same target object to obtain a first similarity; calculating the cosine similarity between the second image features and the corresponding text features to obtain a second similarity; calculating the loss function of the neural network model based on the first similarity and the second similarity, and updating the model parameters of the neural network model using the loss function.

2. The video multi-target tracking method according to claim 1, characterized in that, The method involves calculating the trajectory of any target object in the video image sequence based on the image features and position information of each target object in the previous frame corresponding to the current video image sequence, and the current image features and target position information corresponding to any target object to be tracked in the current image frame, to obtain multiple trajectory matching results, including: Obtain the image features of each target object in the previous frame image corresponding to the current video image sequence from a pre-created image feature sequence template; Based on the target location information corresponding to the target to be tracked and the historical location information corresponding to each target object in the previous frame image, the positional distance between each target to be tracked in the current video image sequence and multiple target objects in the previous frame image is calculated respectively, so as to obtain multiple first distances corresponding to each target to be tracked; Calculate the Euclidean distance between the current image features and the image features of multiple target objects in the previous frame image, respectively, to obtain multiple second distances corresponding to each target to be tracked; The target distance is obtained by weighted calculation based on the first distance and the second distance, and the Hungarian algorithm is used to perform trajectory matching calculation on multiple targets to be tracked in the video image sequence based on the target distance to obtain multiple trajectory matching results corresponding to each target to be tracked.

3. The video multi-target tracking method according to claim 2, characterized in that, After outputting the tracking trajectory of multiple targets in the video image sequence based on the multiple trajectory matching results corresponding to each of the aforementioned targets, the method further includes: The image feature sequence templates corresponding to each of the targets to be tracked are updated based on the current image features.

4. The video multi-target tracking method according to claim 2, characterized in that, The weighted calculation based on the first distance and the second distance to obtain the target distance includes: The first distance and the second distance of the same target object and the same target to be tracked are respectively weighted and calculated to obtain multiple weighted calculated distances; The maximum value among the multiple weighted calculated distances is determined, and based on the maximum value, the weighted calculated distances corresponding to the same target object and the same target to be tracked are normalized to obtain the target distances corresponding to the same target object and the same target to be tracked.

5. A video multi-target tracking device, characterized in that, include: The target detection module is used to perform target detection on the image frames corresponding to the current video image sequence when a video image sequence to be tracked is received, and to obtain a target detection result containing multiple targets to be tracked and their corresponding target position information; The image feature extraction module is used to input the target detection results into the trained image feature extraction model to extract the image features of each target to be tracked in the current image frame, and obtain the current image features corresponding to each target to be tracked. The trained image feature extraction model includes a first model and a second model; the first model is a model used for image feature extraction, and the second model is a model obtained by training a neural network model guided by language text information using a tracking training dataset; The multi-target tracking module is used to perform trajectory matching calculations on any target in the video image sequence based on the image features and position information of each target object in the previous frame image corresponding to the current video image sequence, and the current image features and target position information corresponding to any target to be tracked in the current image frame, to obtain multiple trajectory matching results, and output the tracking trajectory of multiple targets to be tracked in the video image sequence according to the multiple trajectory matching results corresponding to each target to be tracked; The sequence acquisition unit is used to acquire historical image sequences to be tracked by multiple targets, thus obtaining a tracking training dataset. The image frames in the tracking training dataset are labeled with the text attribute information and location information of the target object; The dataset input unit is used to input the tracking training dataset into the initial image feature extraction model, so as to use the first model in the initial image feature extraction model to extract the image features of the target object in each image frame of the tracking training dataset, and obtain the first image features; The prompt information generation unit is used to generate text prompt information for the target object based on the text attribute information of the target object in each image frame of the tracking training dataset. The first model training unit is used to train a neural network model for image feature adjustment based on the first image features corresponding to the same target object in each image frame in the tracking training dataset and the text prompt information, to obtain a second model, and to use the first model and the trained second model as the trained image feature extraction model guided by language text information. The first model training unit specifically includes: An information conversion unit is used to convert the text prompt information into text encoded information using a text encoder; A text feature extraction unit is used to extract features from the text encoding information to obtain text features; The second model training unit is used to train a neural network model for image feature adjustment based on the first image features and the text features corresponding to the same target object in each image frame in the tracking training dataset, so as to obtain a second model guided by language text information. The video multi-target tracking device further includes: The feature adjustment unit is used to obtain the second image features output by the second model after the second model, guided by language text information, performs feature adjustment on the first image features; The first similarity calculation unit is used to calculate the similarity between the first image feature and the second image feature corresponding to the same target object, and obtain the first similarity. The second similarity calculation unit is used to calculate the cosine similarity between the second image feature and the corresponding text feature to obtain the second similarity. The loss function calculation unit is used to calculate the loss function of the neural network model based on the first similarity and the second similarity; The model parameter update unit is used to update the model parameters of the neural network model using the loss function.

6. An electronic device, characterized in that, It includes a processor and a memory; wherein, when the processor executes a computer program stored in the memory, it implements the video multi-target tracking method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, Used to store a computer program; wherein, when the computer program is executed by a processor, it implements the video multi-target tracking method as described in any one of claims 1 to 4.

Images