Fine-grained visual multi-target tracking method, device, electronic device and storage medium

Abstract

The present application provides a fine-grained visual multi-target tracking method, device, electronic device and storage medium. The method includes: selecting two adjacent frames from continuous video frames, and extracting deep features of each frame in the two adjacent frames using a deep convolutional neural network; encoding the deep features using a multi-scale pixel-level feature encoder to generate a fine-grained pixel-by-pixel feature representation; analyzing the target motion relationship between the two adjacent frames using a feature aggregator to generate pixel-level fusion features; processing the pixel-level fusion features using a target decoder, and inputting the processed features into a multi-layer perceptron to generate a classification feature vector, a center point feature vector and a size feature vector of the target; calculating the center heat map of the target, and contextually associating the target in the video frame based on the center heat map, so as to track the target. The present application improves the accuracy of target trajectory tracking and enhances the target tracking effect.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a fine-grained visual multi-target tracking method, device, electronic device, and storage medium. Background Art

[0002] Vision-based multi-object tracking (MOT) plays a crucial role in autonomous driving, video surveillance, behavior recognition, and other fields. This technology detects individual objects in a video, identifies them in subsequent video frames, and correctly associates them with previously detected objects, thereby achieving trajectory prediction and accurate search.

[0003] However, existing multi-object tracking methods primarily rely on bounding boxes to represent the tracked objects. This approach is simple and easy to implement. For example, Wojke et al. used a convolutional neural network to extract target appearance features and used the intersection over union (IoU) of target boxes to associate objects between video frames. Similarly, Yang et al., after detecting target boxes, used feature re-identification and a graph cut connection formula to achieve target tracking and association.

[0004] While existing technologies demonstrate a certain degree of efficiency and low computational complexity in rapidly locating and tracking targets, they exhibit significant shortcomings when handling small targets or tracking in complex, crowded scenes. Target box representations, due to their inclusion of a large amount of non-target background information, obscure the semantics of the target. This leads to even poorer performance when tracking small targets or in crowded scenes.

[0005] Therefore, there is an urgent technical need to develop a method that can achieve accurate multi-target tracking in complex scenes, especially a technical solution that can overcome the problems of background noise and semantic ambiguity introduced by target boxes to improve the accuracy and efficiency of tracking. Summary of the Invention

[0006] In view of this, the embodiments of the present application provide a fine-grained visual multi-target tracking method, device, electronic device and storage medium to solve the problems of low target trajectory tracking accuracy and poor target tracking effect in the prior art.

[0007] According to a first aspect of an embodiment of the present application, a fine-grained visual multi-target tracking method is provided, comprising: selecting two adjacent frames from continuous video frames, and extracting deep features of each of the two adjacent frames using a deep convolutional neural network; encoding the deep features using a preset multi-scale pixel-level feature encoder to generate a fine-grained pixel-by-pixel feature representation; using the pixel-by-pixel feature representation as input to a feature aggregator, and analyzing the target motion relationship between the two adjacent frames using the feature aggregator to generate pixel-level fusion features to model the motion continuity between targets; processing the pixel-level fusion features using a target decoder, and inputting the processed features into a multi-layer perceptron, and using the multi-layer perceptron to generate a classification feature vector, a center point feature vector, and a size feature vector of the target; calculating a center heat map of the target based on the classification feature vector, the center point feature vector, and the size feature vector, and contextually associating the target in the video frame based on the center heat map to track the target.

[0008] According to a second aspect of an embodiment of the present application, a fine-grained visual multi-target tracking device is provided, comprising: an extraction module configured to select two adjacent frames of images from continuous video frames, and use a deep convolutional neural network to extract deep features of each frame of the two adjacent frames; an encoding module configured to use a preset multi-scale pixel-level feature encoder to encode the deep features and generate a fine-grained pixel-by-pixel feature representation; an analysis module configured to use the pixel-by-pixel feature representation as input to a feature aggregator, use the feature aggregator to analyze the target motion relationship between the two adjacent frames of images, and generate pixel-level fusion features to model the motion continuity between the targets; a processing module configured to use a target decoder to process the pixel-level fusion features, and input the processed features into a multi-layer perceptron, and use the multi-layer perceptron to generate a classification feature vector, a center point feature vector, and a size feature vector of the target; a calculation module configured to calculate a center heat map of the target based on the classification feature vector, the center point feature vector, and the size feature vector, and perform context association on the target in the video frame based on the center heat map to track the target.

[0009] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the steps of the above method are implemented when the processor executes the computer program.

[0010] According to a fourth aspect of the embodiments of the present application, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps of the above method are implemented.

[0011] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects:

[0012] By selecting two adjacent frames from continuous video frames, a deep convolutional neural network is used to extract the deep features of each frame in the two adjacent frames; a preset multi-scale pixel-level feature encoder is used to encode the deep features to generate a fine-grained pixel-by-pixel feature representation; the pixel-by-pixel feature representation is used as the input of the feature aggregator, and the feature aggregator is used to analyze the target motion relationship between the two adjacent frames to generate pixel-level fusion features to model the motion continuity between targets; the pixel-level fusion features are processed by the target decoder, and the processed features are input into the multi-layer perceptron, and the multi-layer perceptron is used to generate the classification feature vector, center point feature vector and size feature vector of the target; based on the classification feature vector, center point feature vector and size feature vector, the center heat map of the target is calculated, and the target in the video frame is contextually associated based on the center heat map to track the target. This application improves the accuracy of target trajectory tracking and enhances the target tracking effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0014] Figure 1 Schematic diagram of the process of fine-grained visual multi-target tracking method provided by the embodiment of the present application;

[0015] Figure 2 Schematic diagram of the structure of a fine-grained visual multi-target tracking device provided in an embodiment of the present application;

[0016] Figure 3 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0017] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0018] Multi-Object Tracking (MOT) is an important technology in computer vision, widely used in autonomous driving, video surveillance, and behavior recognition. The main task of MOT is to detect, identify, and track multiple moving objects in a video sequence, enabling the prediction of their trajectories and precise positioning and search.

[0019] With the development of computer vision technology, video multi-target tracking has been widely used in various fields, such as identifying and tracking surrounding vehicles and pedestrians in autonomous driving systems, monitoring the behavior of specific groups of people in video surveillance, and analyzing human movements in behavior recognition technology. These applications require the system to accurately detect objects in the video, identify their identities, and track their position and movement across different frames in the video sequence.

[0020] Currently, most video multi-object tracking methods use bounding boxes to mark and track objects. For example, Wojke et al. proposed using a convolutional neural network (CNN) to detect the appearance features of objects and generate bounding boxes representing them. The association between objects is achieved by calculating the intersection over union (IoU) of the bounding boxes in consecutive video frames. Similarly, after detecting the target boxes, Yang et al. designed a feature re-identification mechanism and a multi-cut connection formula for the graph to achieve association between objects.

[0021] While these methods are computationally simple and enable rapid target localization and tracking, they have significant limitations. Because the target bounding box often contains background information in addition to the target itself, the information within the bounding box can be semantically ambiguous. This is particularly true when tracking small targets or in complex, crowded scenes, where cluttered information within the bounding box can severely impact tracking accuracy and effectiveness.

[0022] Based on the limitations of the above-mentioned prior art, the technical problems to be solved by the technical solution of this application mainly focus on how to improve the accuracy of video multi-target tracking, especially the performance in tracking small targets or in complex and crowded scenes. Specifically, the problems that need to be solved in this application include how to reduce the background noise in the target frame and improve the clarity of the target semantics, so as to more accurately identify and track multiple targets in the video. In general, although the existing technology has made certain progress in multi-target tracking, there is still a lot of room for improvement in the performance of handling small target tracking and complex scenes. Solving this technical problem is of great significance to improving the overall performance and application scope of video multi-target tracking technology.

[0023] In view of the problems existing in the above-mentioned prior art, the technical solution proposed in this application aims to solve the problem of low accuracy of prior art when tracking small targets or complex scenes through a fine-grained visual multi-target tracking method. This technical solution extracts target features from video frames through a series of modular processing steps, establishes motion relationships between targets, and performs high-precision target tracking. The technical solution of this application mainly includes the following contents:

[0024] 1) Feature Extraction: A deep convolutional neural network (such as ResNet-32, ResNet-50, VGG-50, etc.) with shared weights is used to extract deep features from the two frames of the input video. The purpose of this step is to capture the basic appearance and shape information of the target in the video frame.

[0025] 2) Multi-scale pixel-level feature encoding: A multi-layer feature encoder is used to upsample the extracted deep features to generate fine-grained pixel-by-pixel feature representations. This can be implemented using a CNN or an attention neural network. The attention mechanism's encoding process relies on the input feature vector and generates image pixel features by calculating query features, key features, and value features.

[0026] 3) Feature Aggregation: The feature aggregator analyzes the object motion relationship between adjacent frames and generates pixel-level fusion features. This process calculates the motion trend between two frames and fuses the features into the same feature space, providing a continuous representation of the object motion.

[0027] 4) Target Decoding and Tracking: The fused features are further processed using the target decoder, and the target classification feature vector, center point feature vector, and size feature vector are obtained through a multi-layer perceptron. These feature vectors are used to calculate the target's center heat map, and the target's context is associated with the center heat map to complete target tracking.

[0028] By implementing the above technical solutions, the present application can accurately track multiple targets in the video at a fine-grained level, especially when dealing with small targets or complex scenes, showing higher accuracy and efficiency than existing technologies.

[0029] The contents of the technical solution of this application are described in detail below with reference to the accompanying drawings and specific embodiments.

[0030] Figure 1 FIG is a flow chart of the fine-grained visual multi-target tracking method provided in the embodiment of the present application. Figure 1 As shown, the fine-grained visual multi-target tracking method may specifically include:

[0031] S101, selecting two adjacent frames from continuous video frames, and extracting deep features of each of the two adjacent frames using a deep convolutional neural network;

[0032] S102, encoding the deep features using a preset multi-scale pixel-level feature encoder to generate a fine-grained pixel-by-pixel feature representation;

[0033] S103, using the pixel-by-pixel feature representation as input to a feature aggregator, using the feature aggregator to analyze the object motion relationship between two adjacent frames of image, and generating pixel-level fusion features to model the motion continuity between the objects;

[0034] S104, using a target decoder to process the pixel-level fusion features, and input the processed features into a multi-layer perceptron, which generates a classification feature vector, a center point feature vector, and a size feature vector of the target;

[0035] S105 , calculating a center heat map of the target based on the classification feature vector, the center point feature vector, and the size feature vector, and performing context association on the target in the video frame based on the center heat map so as to track the target.

[0036] First, before introducing the specific embodiments of the present application in detail, the overall implementation process of the technical solution of the present application is summarized. The present application aims to provide a method for fine-grained visual multi-target tracking. The main technical solution of the present application includes: for continuous video frame images of video input, first use the feature extraction module to extract the deep features of two adjacent frames of images respectively; set a multi-scale pixel-level feature encoder to encode the deep features of two adjacent frames of images respectively, and generate multi-layer image pixel features; after obtaining the image pixel features of adjacent frames, use the feature aggregator to model the target motion relationship between adjacent frame images, and generate pixel-level fusion features; after obtaining the pixel-level fusion features, further process the fusion features through the target decoder, and generate the target classification feature vector, center point feature vector and size feature vector through the multi-layer perceptron; calculate the center heat map of the target based on the above feature vectors, and calculate the context association of the tracked target in the image based on the center heat map to complete the target tracking.

[0037] In some embodiments, extracting deep features of each of two adjacent image frames using a deep convolutional neural network includes:

[0038] Two adjacent frames of images are input into a deep convolutional neural network configured with shared weights, where the deep convolutional neural network is used to analyze the image content and extract deep features of the image;

[0039] A deep convolutional neural network is used to perform feature extraction operations on each frame of the image to obtain deep features, where the deep features contain low-resolution feature representations of the deep visual information of the image.

[0040] Specifically, this embodiment describes a method for extracting deep features from images in consecutive video frames using a deep convolutional neural network (CNN), aiming to achieve high efficiency and accuracy in fine-grained visual multi-target tracking. The following is a detailed implementation process of this embodiment:

[0041] First, two consecutive frames are extracted from the video input. These two frames serve as the basis for processing, aiming to capture the movement and changes of the target in a short period of time.

[0042] Next, a suitable deep convolutional neural network model is selected as the feature extraction module. In this embodiment, models such as ResNet-32, ResNet-50, and VGG-50 can be selected. These models are selected for their deep structure and excellent feature extraction capabilities. The shared weight mechanism ensures computational efficiency and parameter consistency when processing consecutive frames.

[0043] The two preprocessed images are then fed into a deep convolutional neural network configured with shared weights. The network analyzes the image content and extracts features representing the underlying visual information. These features, represented at low resolution, contain sufficient visual information for subsequent object detection and tracking.

[0044] Finally, after processing through a deep convolutional neural network, deep features are extracted from each frame. These deep features capture the key visual attributes of the target, such as shape and texture, and provide a strong foundation for accurate target tracking.

[0045] It should be noted that this embodiment is not limited to the use of ResNet or VGG models. Other types of deep convolutional neural network models, such as deeper networks or networks optimized for specific tasks, can also be used based on specific application scenarios and requirements. In addition, for different tracking targets and background environments, the feature extraction process can be optimized by adjusting network parameters or introducing pre-trained models to improve tracking accuracy and robustness.

[0046] This example describes how to effectively use deep convolutional neural networks to extract deep features in multi-target video tracking, laying the foundation for efficient and accurate multi-target tracking. By properly selecting the network and adjusting its parameters, the entire feature extraction process can be optimized to achieve optimal tracking results based on specific application requirements.

[0047] In some embodiments, a preset multi-scale pixel-level feature encoder is used to encode deep features to generate fine-grained pixel-by-pixel feature representations, including:

[0048] Using a multi-scale pixel-level feature encoder with shared weights, the deep features of each frame are upsampled to generate fine-grained pixel-by-pixel feature representations.

[0049] Among them, the multi-scale pixel-level feature encoder includes a feature encoder based on convolutional neural network and a feature encoder based on attention neural network.

[0050] Specifically, this embodiment also details a method for encoding deep features of video frame images using a preset multi-scale pixel-level feature encoder to generate fine-grained pixel-by-pixel feature representations. This method is particularly suitable for fine-grained visual multi-target tracking and can improve tracking accuracy and efficiency. The following is a detailed implementation description of this embodiment:

[0051] First, two adjacent frames are extracted from the continuous video frames (marked as time points t and t-1), and the deep convolutional neural network is used to extract the deep features F of these two frames. (t) With F (t-1) This step ensures that the basic visual information and dynamic changes of the target can be captured.

[0052] Furthermore, a multi-scale pixel-level feature encoder with shared weights is used to upsample the extracted deep features. This process aims to generate fine-grained pixel-by-pixel feature representations. and Where l represents the number of upsampling layers. This step provides richer and more accurate information for subsequent target detection and tracking by refining the feature representation.

[0053] Furthermore, the multi-scale pixel-level feature encoder can be based on a convolutional neural network (CNN) such as the multi-scale convolutional neural network (MTCNN) or the feature pyramid network (FPN), or based on an attention neural network such as the Deformable Self-Attention Encoder (DSE). This provides flexibility to select the most appropriate encoder type based on the requirements and characteristics of a specific task.

[0054] Furthermore, in the feature encoding process based on the attention neural network, the input image features are first flattened into a one-dimensional feature vector with a length of H×W×C. These feature vectors are then input into multiple consecutive attention calculation modules to generate image pixel features. This process takes advantage of the attention mechanism and can focus more on the key information in the image, thereby improving the efficiency and accuracy of feature encoding.

[0055] In one example, when a feature vector is input into multiple consecutive attention calculation modules to generate image pixel features, the following formula can be used:

[0056]

[0057] Q=W q ·P (t)

[0058] K=W k ·P (t)

[0059] V=W v ·P (t)

[0060] Among them, P (t) W represents the feature vector after the input image feature of the t-th frame is flattened into 1 dimension, q 、W k 、W v Represent three learnable parameter matrices respectively. The generated Q, K, and V represent query features, key features, and value features respectively. d represents the length of the generated key features. For the generated l-th layer image pixel features, softmax is used to calculate the normalized value of attention.

[0061] It should be noted that this embodiment can also optimize the feature encoding process by introducing a more advanced encoder model or adjusting the parameters of the existing model to adapt to tracking environments of different complexities and changes. Combining the advantages of convolutional neural networks and attention neural networks, by fusing these two encoding methods, the quality of pixel-by-pixel feature representation and target tracking performance can be further improved. For real-time video multi-target tracking applications, the encoder can be specially optimized to reduce the amount of calculation and increase the processing speed, ensuring efficient target tracking under real-time or near real-time conditions.

[0062] Through the above-mentioned method of this embodiment, deep features can be effectively extracted and encoded, and high-quality pixel-by-pixel feature representation can be generated, providing strong technical support for achieving fine-grained visual multi-target tracking.

[0063] In some embodiments, a feature aggregator is used to analyze the object motion relationship between two adjacent frames of images to generate pixel-level fusion features to model the motion continuity between the objects, including:

[0064] The feature aggregator is used to calculate the motion trend vector between the corresponding levels in two adjacent frames to evaluate the position change of the target between consecutive frames;

[0065] According to the motion trend vector, a predetermined algorithm is used to fuse the features of two adjacent frames of images into the same feature space to obtain pixel-level fusion features, wherein the pixel-level fusion features include a fusion feature representation of continuity information in the time dimension.

[0066] Specifically, the embodiment of this application also details how to use a feature aggregator to analyze the motion relationship between two adjacent frames of images and generate pixel-level fusion features containing continuity information in the temporal dimension. This process is crucial for achieving high-precision fine-grained visual multi-target tracking. The following is the specific implementation process of this embodiment:

[0067] First, two adjacent frames are extracted from the continuous video frames, and deep convolutional neural networks are used to extract the deep features of these two frames. These features serve as the input of the feature aggregator, laying the foundation for the next step of motion trend analysis.

[0068] The feature aggregator then calculates the motion trend vector between corresponding layers in two adjacent frames. This step compares the feature differences between the two frames to assess the positional changes of the target between consecutive frames, thereby capturing the target's motion information. To enhance the accuracy and adaptability of this process, machine learning algorithms, such as optical flow estimation, can be introduced to more accurately calculate motion trends.

[0069] Furthermore, based on the calculated motion trend vector, the feature aggregator uses a predefined algorithm to fuse the features of two adjacent frames into the same feature space, generating pixel-level fused features. These fused features not only contain the visual information of the target but also incorporate the motion continuity information between the targets, providing rich context for subsequent target detection and tracking.

[0070] In an example, this embodiment can be implemented by the following formula:

[0071]

[0072] in, represents the image features of the lth layer after fusion, Indicates the deformation features of the lth layer corresponding to the image features of the t-1 frame and the t-th frame, ω (t-1)→(t) Indicates the similarity between the corresponding features of the t-1 frame and the t-th frame image.

[0073] Among them, the motion vector between adjacent image features can be calculated by the following formula:

[0074]

[0075] Among them, Φ(D (t-1) ,D (t) ) represents the motion trend between adjacent frame features, and τ is the deformation feature calculator.

[0076] Furthermore, based on the generated pixel-level fused features, these features can be further optimized, for example through noise reduction and feature enhancement techniques, to improve feature usability and target tracking accuracy. Furthermore, these fused features can be directly applied to target detection, recognition, and tracking algorithms to achieve precise target positioning and tracking.

[0077] This embodiment uses a feature aggregator to analyze and fuse the motion information of two adjacent frames, generating pixel-level fusion features that reflect the continuity of target motion. This approach not only improves the accuracy and robustness of multi-target tracking but also effectively tracks small targets or multiple targets in crowded scenes in complex dynamic environments. Furthermore, the method of this embodiment is highly adaptable and can be adjusted and optimized according to different application scenarios and tracking requirements, possessing broad practical value and application prospects.

[0078] In some embodiments, the pixel-level fusion features are processed using a target decoder, and the processed features are input into a multi-layer perceptron, which is used to generate a classification feature vector, a center point feature vector, and a size feature vector of the target, including:

[0079] The pixel-level fusion features are decoded using a pre-configured target decoder. During the decoding process, the pixel-level fusion features at each scale are analyzed and converted using the target decoder.

[0080] The features processed by the target decoder are input into three multi-layer perceptrons, which are used to refine the features. The three multi-layer perceptrons are used to generate the classification feature vector, center point feature vector and size feature vector of the target respectively.

[0081] Among them, the classification feature vector is used to represent different types of targets, the center point feature vector is used to represent the center position of the target in the image, and the size feature vector is used to represent the size information of the target.

[0082] Specifically, this embodiment uses a target decoder to process pixel-level fusion features and further uses a multi-layer perceptron (MLP) to generate the target's classification feature vector, center point feature vector, and size feature vector, aiming to achieve fine-grained visual multi-target tracking. The following is the specific implementation process of this embodiment:

[0083] First, the pixel-level fusion features obtained from the feature aggregator are decoded using a pre-configured target decoder. The key to this step is to analyze and transform the pixel-level fusion features at each scale to extract effective information for target tracking.

[0084] The decoder can be based on traditional decoding algorithms, or adopt deep learning technology such as decoders based on convolutional neural networks (CNN) or self-attention networks (such as Transformer). The most appropriate decoding technology is selected according to actual needs.

[0085] Furthermore, the features processed by the object decoder are then input into three independently configured multi-layer perceptrons, which further refine the features to accurately represent the category, center location, and size information of the object.

[0086] Among them, the three multi-layer perceptrons are responsible for generating the following key feature vectors:

[0087] Categorical feature vector Used to distinguish different types of targets;

[0088] Center point eigenvector Used to identify the center position of the target in the image;

[0089] Size feature vector Used to describe the size information (width and height) of the target.

[0090] Furthermore, the performance of the multilayer perceptron can be optimized by adjusting the number of layers, activation function, or training strategy to more accurately generate the target's classification, center point, and size feature vectors. The collaboration between the decoder and the multilayer perceptron is crucial. By selecting the multilayer perceptron structure and parameters that match the decoder's output features, the efficiency of feature utilization and the accuracy of target tracking can be effectively improved.

[0091] Furthermore, for real-time video analysis and multi-target tracking applications, by optimizing the implementation of the target decoder and multi-layer perceptron, such as adopting more efficient algorithms and hardware acceleration technologies, the processing speed can be significantly improved to meet the needs of real-time applications.

[0092] This embodiment uses a carefully designed object decoder and a multi-layer perceptron to effectively extract key information for multi-target tracking, including target category, location, and size, from pixel-level fusion features. This approach not only improves the accuracy and efficiency of multi-target tracking but also enhances the system's ability to handle complex scenarios, promising broad application prospects.

[0093] In some embodiments, a central heat map of the target is calculated based on the classification feature vector, the center point feature vector, and the size feature vector, including calculating the central heat map using the following formula:

[0094]

[0095]

[0096] Among them, i∈[1,N] represents the i-th candidate target, j∈[1,2] corresponds to width and height respectively, k l [i,h,w] represents the probability that the position corresponding to the coordinates (h,w) in the l-th layer image feature is the center point of the i-th target, s l [i,h,w,j] represents the length and width of the target bounding box corresponding to the center point with coordinates (h,w) in the image features of the lth layer. Combining vector k with vector s yields the center heat map of the target.

[0097] Specifically, three multi-layer perceptrons are used to post-process the features to generate classification feature vectors Center point eigenvector and size feature vector Afterwards, the embodiment of the present application will be based on the pixel fusion feature The target center point coordinates and the target width and height are used to calculate the target center heat map and model size heat map distribution. For the specific calculation method, please refer to the center heat map calculation formula provided in the above embodiment, which will not be repeated here.

[0098] In some embodiments, contextualizing an object in a video frame based on a center heat map to track the object includes:

[0099] The center heat map is used to evaluate the center position of each target in the video frame, so as to identify the high response area on the center heat map and locate the center point of the target;

[0100] Using the preset target association method, the correlation between the predicted target and the real target in consecutive video frames is calculated based on the target center position information obtained from the center heat map;

[0101] The target association method is used to analyze and determine the position change and continuity of each target between the current frame and the next frame to achieve target tracking.

[0102] Specifically, the embodiment of this application also details a target context association method based on a center heat map for accurate tracking of targets in video frames. This method uses a generated center heat map to evaluate the target center position and uses an advanced target association algorithm to calculate the correlation between targets, thereby tracking the target in consecutive video frames. The following is the specific implementation process of this embodiment:

[0103] First, a center heat map representing the center position of each object in the video frame is generated by the multi-layer perceptron processing of the previous embodiment. Each high response area on the center heat map corresponds to the center point of an object.

[0104] These center heatmaps are then evaluated to identify high-response regions to localize the center position of each object. This step forms the basis for subsequent object association and tracking.

[0105] Next, using a pre-defined target association method (such as the Kalman filter, Hungarian algorithm, or deep Hungarian network), the correlation between the predicted target and the real target is calculated between consecutive video frames based on the target center position information obtained from the center heat map. These target association methods can analyze and determine the position change and continuity of each target between the current frame and the next frame, achieving effective target tracking.

[0106] It should be noted that, depending on the application scenario and specific tracking requirements, appropriate target association methods can be selected and their parameters adjusted and optimized to improve tracking accuracy and robustness. To meet the needs of real-time or near-real-time video surveillance, the speed of center heat map generation and target association calculation can be increased through technical means such as algorithm optimization and parallel computing, thereby enhancing the real-time processing capabilities of the entire system.

[0107] In addition, in complex environments or crowded scenes, occlusion and interaction between targets may affect the tracking effect. By introducing more advanced algorithms, such as deep learning models, the system's adaptability and accuracy to these challenges can be further improved.

[0108] This example achieves precise tracking of multiple targets within a video frame through precise center point positioning and advanced target association methods. This method is not only applicable to conventional surveillance scenarios but also addresses demanding applications such as autonomous driving and behavior recognition, demonstrating broad application potential and practical value.

[0109] The technical solution of this application, by introducing a deep learning model, finely models the motion relationship between two adjacent frames of images, and innovatively maps this motion relationship into the same feature space through feature fusion, generating a center heat map with highly fine-grained feature information. Compared with the tracking technology that relies on traditional target frames, this method significantly improves the accuracy of target trajectory tracking, especially when dealing with target tracking problems in crowded scenes, and significantly reduces the loss rate of target trajectories. According to the technical solution provided in the embodiment of this application, the technical solution of this application has at least the following advantages:

[0110] By analyzing the motion relationship between two adjacent frames using a deep learning model and performing feature fusion, this technology provides richer and more detailed target feature information. Compared to traditional target box methods, the center heat map more accurately reflects the target's position, enabling higher-accuracy target tracking in complex and dynamic environments.

[0111] In crowded scenes, occlusion and cross-movement between multiple targets can easily lead to tracking loss. This technical solution uses fine-grained center heatmaps to effectively distinguish closely adjacent or partially occluded targets, significantly reducing the tracking loss rate in such scenarios.

[0112] This technical solution not only improves tracking accuracy and reduces loss rates, but also enhances the robustness of target trajectory tracking through the feature fusion mechanism of the deep model, maintaining stable tracking performance even when the target moves quickly or lighting conditions change.

[0113] Due to the significant improvements in target tracking accuracy, loss rate reduction, and robustness, this technical solution has broad application potential, particularly in areas such as autonomous driving, video surveillance, and crowd analysis, and can effectively address the challenges of multi-target tracking in various complex scenarios.

[0114] Therefore, in summary, the technical solution of this application has achieved a major technological breakthrough in the field of multi-target tracking through the introduction of deep learning models and innovative feature fusion methods, and provides an effective solution to the limitations of traditional methods in accuracy, loss rate and robustness.

[0115] The following are device embodiments of the present application, which can be used to implement the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0116] Figure 2 Schematic diagram of the structure of the fine-grained visual multi-target tracking device provided in the embodiment of the present application. Figure 2 As shown, the fine-grained visual multi-target tracking device includes:

[0117] The extraction module 201 is configured to select two adjacent frames from continuous video frames and extract deep features of each of the two adjacent frames using a deep convolutional neural network;

[0118] The encoding module 202 is configured to encode the deep features using a preset multi-scale pixel-level feature encoder to generate a fine-grained pixel-by-pixel feature representation;

[0119] An analysis module 203 is configured to use the pixel-by-pixel feature representation as an input to a feature aggregator, analyze the object motion relationship between two adjacent frames of images using the feature aggregator, and generate pixel-level fusion features to model the motion continuity between the objects;

[0120] The processing module 204 is configured to process the pixel-level fusion features using the target decoder, and input the processed features into a multi-layer perceptron to generate a classification feature vector, a center point feature vector, and a size feature vector of the target using the multi-layer perceptron;

[0121] The calculation module 205 is configured to calculate the center heat map of the target based on the classification feature vector, the center point feature vector and the size feature vector, and perform context association on the target in the video frame based on the center heat map to track the target.

[0122] In some embodiments, Figure 2 The extraction module 201 inputs two adjacent frames of images into a deep convolutional neural network configured with shared weights, wherein the deep convolutional neural network is used to analyze the image content and extract the deep features of the image; the deep convolutional neural network is used to perform feature extraction operations on each frame of the image to obtain deep features, wherein the deep features include low-resolution feature representations of the deep visual information of the image.

[0123] In some embodiments, Figure 2 The encoding module 202 uses a multi-scale pixel-level feature encoder configured with shared weights to upsample the deep features of each frame image to generate a fine-grained pixel-by-pixel feature representation; wherein the multi-scale pixel-level feature encoder includes a feature encoder based on a convolutional neural network and a feature encoder based on an attention neural network.

[0124] In some embodiments, Figure 2 The analysis module 203 uses a feature aggregator to calculate the motion trend vector between the corresponding levels in two adjacent frames of images to evaluate the position change of the target between consecutive frames; according to the motion trend vector, a predetermined algorithm is used to fuse the features of the two adjacent frames of images into the same feature space to obtain pixel-level fusion features, wherein the pixel-level fusion features include a fusion feature representation of continuity information in the time dimension.

[0125] In some embodiments, Figure 2 The processing module 204 uses a pre-configured target decoder to decode the pixel-level fusion features. During the decoding process, the target decoder is used to analyze and convert the pixel-level fusion features at each scale; the features processed by the target decoder are input into three multi-layer perceptrons, and the multi-layer perceptrons are used to refine the features. The three multi-layer perceptrons are respectively used to generate the classification feature vector, center point feature vector and size feature vector of the target; among them, the classification feature vector is used to represent different types of targets, the center point feature vector is used to represent the center position of the target in the image, and the size feature vector is used to represent the size information of the target.

[0126] In some embodiments, Figure 2 The calculation module 205 calculates the center heat map using the following formula:

[0127]

[0128]

[0129] Among them, i∈[1,N] represents the i-th candidate target, j∈[1,2] corresponds to width and height respectively, k l [i,h,w] represents the probability that the position corresponding to the coordinates (h,w) in the l-th layer image feature is the center point of the i-th target, s l [i,h,w,j] represents the length and width of the target bounding box corresponding to the center point with coordinates (h,w) in the image features of the lth layer. Combining vector k with vector s yields the center heat map of the target.

[0130] In some embodiments, Figure 2 The calculation module 205 uses the center heat map to evaluate the center position of each target in the video frame, so as to identify the high response area on the center heat map to locate the center point of the target; uses a preset target association method to calculate the correlation between the predicted target and the real target in the continuous video frames based on the target center position information obtained from the center heat map; uses the target association method to analyze and determine the position change and continuity of each target in the current frame and the next frame to achieve target tracking.

[0131] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0132] Figure 3 Schematic diagram of the structure of the electronic device 3 provided in the embodiment of the present application. Figure 3 As shown, the electronic device 3 of this embodiment includes: a processor 301, a memory 302, and a computer program 303 stored in the memory 302 and executable on the processor 301. When the processor 301 executes the computer program 303, the steps of the above-mentioned method embodiments are implemented. Alternatively, when the processor 301 executes the computer program 303, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0133] For example, computer program 303 may be divided into one or more modules / units, which are stored in memory 302 and executed by processor 301 to implement the present application. One or more modules / units may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of computer program 303 in electronic device 3.

[0134] The electronic device 3 may be a desktop computer, a notebook, a PDA, a cloud server or other electronic device. The electronic device 3 may include but is not limited to a processor 301 and a memory 302. Those skilled in the art will understand that Figure 3 It is only an example of electronic device 3 and does not constitute a limitation of electronic device 3. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0135] The processor 301 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0136] The memory 302 can be an internal storage unit of the electronic device 3, such as a hard drive or memory of the electronic device 3. The memory 302 can also be an external storage device of the electronic device 3, such as a plug-in hard drive, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, etc. equipped on the electronic device 3. Furthermore, the memory 302 can include both an internal storage unit of the electronic device 3 and an external storage device. The memory 302 is used to store computer programs and other programs and data required by the electronic device. The memory 302 can also be used to temporarily store data that has been output or is about to be output.

[0137] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0138] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0139] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0140] In the embodiments provided in this application, it should be understood that the disclosed apparatus / computer equipment and methods can be implemented in other ways. For example, the apparatus / computer equipment embodiments described above are merely schematic. For example, the division of modules or units is merely a logical function division. In actual implementation, there may be other division methods. Multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection of the apparatus or unit, which may be electrical, mechanical or other forms.

[0141] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0142] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0143] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, which may be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0144] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A fine-grained visual multi-target tracking method, characterized in that: include: Selecting two adjacent frames from continuous video frames, and extracting deep features of each of the two adjacent frames using a deep convolutional neural network; Encoding the deep features using a preset multi-scale pixel-level feature encoder to generate a fine-grained pixel-by-pixel feature representation; The pixel-by-pixel feature representation is used as an input of a feature aggregator, and the feature aggregator is used to analyze the object motion relationship between the two adjacent frames of image to generate pixel-level fusion features to model the motion continuity between the objects; The pixel-level fusion features are processed by a target decoder, and the processed features are input into a multi-layer perceptron, and the multi-layer perceptron is used to generate a classification feature vector, a center point feature vector, and a size feature vector of the target; A central heat map of the target is calculated based on the classification feature vector, the center point feature vector, and the size feature vector, and context association is performed on the target in the video frame based on the central heat map so as to track the target.

2. The method according to claim 1, characterized in that The extracting deep features of each of the two adjacent frames of image using a deep convolutional neural network includes: Inputting the two adjacent frames of image into a deep convolutional neural network configured with shared weights, respectively, wherein the deep convolutional neural network is used to analyze image content and extract deep features of the image; The deep convolutional neural network is used to perform a feature extraction operation on each frame of the image to obtain the deep features, wherein the deep features include low-resolution feature representations of deep visual information of the image.

3. The method according to claim 1, characterized in that The method of encoding the deep features using a preset multi-scale pixel-level feature encoder to generate a fine-grained pixel-by-pixel feature representation includes: Upsampling the deep features of each frame of the image using a multi-scale pixel-level feature encoder configured with shared weights to generate a fine-grained pixel-by-pixel feature representation; Among them, the multi-scale pixel-level feature encoder includes a feature encoder based on a convolutional neural network and a feature encoder based on an attention neural network.

4. The method according to claim 1, wherein The step of analyzing the target motion relationship between the two adjacent frames of image using the feature aggregator to generate pixel-level fusion features to model the motion continuity between the targets includes: Utilizing the feature aggregator to calculate the motion trend vector between corresponding levels in the two adjacent frames of image to evaluate the position change of the target between consecutive frames; According to the motion trend vector, a predetermined algorithm is used to fuse the features of two adjacent frames of images into the same feature space to obtain the pixel-level fusion feature, wherein the pixel-level fusion feature includes a fusion feature representation of continuity information in the time dimension.

5. The method according to claim 1, characterized in that The method of processing the pixel-level fusion features by using a target decoder and inputting the processed features into a multi-layer perceptron, and generating a classification feature vector, a center point feature vector, and a size feature vector of the target by using the multi-layer perceptron, includes: Decoding the pixel-level fusion features using a pre-configured target decoder. During the decoding process, for each scale of the pixel-level fusion features, the target decoder is used to analyze and convert the pixel-level fusion features. Inputting the features processed by the target decoder into three multi-layer perceptrons, and using the multi-layer perceptrons to refine the features, the three multi-layer perceptrons are used to generate the classification feature vector, the center point feature vector and the size feature vector of the target respectively; The classification feature vector is used to characterize different types of targets, the center point feature vector is used to characterize the center position of the target in the image, and the size feature vector is used to characterize the size information of the target.

6. The method according to claim 1, characterized in that The calculating of the center heat map of the target according to the classification feature vector, the center point feature vector and the size feature vector includes calculating the center heat map using the following formula: Among them, i∈[1,N] represents the i-th candidate target, j∈[1,2] corresponds to width and height respectively, k l [i,h,w] represents the probability that the position corresponding to the coordinates (h,w) in the l-th layer image feature is the center point of the i-th target, s l [i,h,w,j] represents the length and width of the target bounding box corresponding to the center point of the corresponding coordinate (h,w) in the l-th layer image feature. Combining vector k with vector s obtains the center heat map of the target. represents the image features of the lth layer after fusion, represents the center point feature vector of the i-th target, Represents the size feature vector of the i-th target.

7. The method according to claim 1, characterized in that The performing context association on the target in the video frame according to the central heat map so as to track the target includes: Using the center heat map, the center position of each target in the video frame is evaluated, so as to identify the high response area on the center heat map to locate the center point of the target; Using a preset target association method, the association degree between the predicted target and the real target in consecutive video frames is calculated based on the target center position information obtained from the center heat map; The target association method is used to analyze and determine the position change and continuity of each target between the current frame and the next frame, so as to achieve tracking of the target.

8. A fine-grained visual multi-target tracking device, characterized in that: include: An extraction module is configured to select two adjacent frames of images from continuous video frames and extract deep features of each of the two adjacent frames of images using a deep convolutional neural network; An encoding module is configured to encode the deep features using a preset multi-scale pixel-level feature encoder to generate a fine-grained pixel-by-pixel feature representation; an analysis module configured to use the pixel-by-pixel feature representation as an input to a feature aggregator, analyze the object motion relationship between the two adjacent frames of image using the feature aggregator, and generate pixel-level fusion features to model the motion continuity between the objects; a processing module configured to process the pixel-level fusion features using a target decoder, input the processed features into a multi-layer perceptron, and use the multi-layer perceptron to generate a classification feature vector, a center point feature vector, and a size feature vector of the target; The calculation module is configured to calculate the center heat map of the target based on the classification feature vector, the center point feature vector and the size feature vector, and to perform context association on the target in the video frame based on the center heat map so as to track the target.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images