Fine-grained visual multi-target tracking method and device, electronic equipment and storage medium
Through deep convolutional neural network and multi-scale pixel-level feature encoder, fine-grained features are generated, combined with multi-layer perceptrons to relate the target, solving the multi-objective tracking accuracy problem in small and medium-sized goals and complex scenarios in the existing technology, and achieving more Efficient and accurate tracking effects.
Patent Information
- Application Number
- CN202410402884.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-03
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-04-03
AI Technical Summary
The existing multi-objective tracking methods are not very accurate when dealing with small targets or complex scenarios, and the representation method of the target box contains a large amount of background information, resulting in semantic ambiguity, affecting the tracking effect.
Deep convolutional neural network is used to extract the deep features of video frames, and a multi-scale pixel-level feature encoder is used to generate fine-grained pixel-by-pixel feature representations. The target motion relationship is analyzed through feature aggregators, and the classification, center point and Size eigenvectors, calculate the central heat map for target association to achieve tracking.
Improve the accuracy and efficiency of target trajectory tracking, especially tracking performance in small targets or complex scenarios, significantly reduce the impact of background noise and enhance the target semantic clarity.
Smart Images

Figure CN119006515A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer vision technology, and in particular to a fine-grained visual multi-target tracking method, device, electronic device and storage medium. Background Art
[0002] In the fields of autonomous driving, video surveillance, behavior recognition, etc., vision-based video multi-object tracking technology (MOT) plays a vital role. This technology is dedicated to detecting each target in the video, and by identifying these targets in subsequent video frames, it can correctly associate them with historical detection targets, thereby achieving the purpose of trajectory prediction and accurate search.
[0003] However, existing multi-target tracking methods mainly rely on bounding boxes to represent the tracked targets. This method is simple and easy to implement. For example, Wojke et al. extracted target appearance features through convolutional neural networks and used the intersection over union (IoU) of target boxes to associate targets between video frames. Similarly, after detecting the target boxes, Yang et al. used feature re-identification and graph cut connection formulas to achieve target tracking and association.
[0004] Although existing technologies have shown certain efficiency and low computational complexity in quickly locating and tracking targets, they are obviously insufficient when dealing with small targets or tracking tasks in complex and crowded scenes. The target box representation method contains a lot of non-target background information, which makes the semantic information of the target unclear, especially when tracking small targets or tracking targets in crowded scenes, the effect is even worse.
[0005] Therefore, there is an urgent technical need to develop a method that can achieve accurate multi-target tracking in complex scenes, especially a technical solution that can overcome the problems of background noise and semantic ambiguity introduced by the target box, so as to improve the accuracy and efficiency of tracking. Summary of the invention
[0006] In view of this, the embodiments of the present application provide a fine-grained visual multi-target tracking method, device, electronic device and storage medium to solve the problems of low accuracy and poor target tracking effect in the prior art.
[0007] According to a first aspect of an embodiment of the present application, a fine-grained visual multi-target tracking method is provided, including: selecting two adjacent frames of images from continuous video frames, and extracting deep features of each frame of the two adjacent frames of images using a deep convolutional neural network; encoding the deep features using a preset multi-scale pixel-level feature encoder to generate a fine-grained pixel-by-pixel feature representation; using the pixel-by-pixel feature representation as input to a feature aggregator, and using the feature aggregator to analyze the target motion relationship between the two adjacent frames of images to generate pixel-level fusion features to model the motion continuity between targets; using a target decoder to process the pixel-level fusion features, and inputting the processed features into a multi-layer perceptron, and using the multi-layer perceptron to generate a classification feature vector, a center point feature vector, and a size feature vector of the target; calculating a center heat map of the target based on the classification feature vector, the center point feature vector, and the size feature vector, and contextualizing the target in the video frame based on the center heat map to track the target.
[0008] According to a second aspect of an embodiment of the present application, a fine-grained visual multi-target tracking device is provided, comprising: an extraction module, configured to select two adjacent frames of images from continuous video frames, and use a deep convolutional neural network to extract deep features of each of the two adjacent frames of images; an encoding module, configured to use a preset multi-scale pixel-level feature encoder to encode the deep features and generate a fine-grained pixel-by-pixel feature representation; an analysis module, configured to use the pixel-by-pixel feature representation as an input to a feature aggregator, use the feature aggregator to analyze the target motion relationship between the two adjacent frames of images, and generate pixel-level fusion features to model the motion continuity between the targets; a processing module, configured to use a target decoder to process the pixel-level fusion features, and input the processed features into a multi-layer perceptron, and use the multi-layer perceptron to generate a classification feature vector, a center point feature vector, and a size feature vector of the target; a calculation module, configured to calculate a center heat map of the target based on the classification feature vector, the center point feature vector, and the size feature vector, and perform context association on the target in the video frame based on the center heat map so as to track the target.
[0009] According to a third aspect of an embodiment of the present application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the steps of the above method are implemented when the processor executes the computer program.
[0010] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.
[0011] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects:
[0012] By selecting two adjacent frames from continuous video frames, a deep convolutional neural network is used to extract the deep features of each frame in the two adjacent frames; the deep features are encoded using a preset multi-scale pixel-level feature encoder to generate a fine-grained pixel-by-pixel feature representation; the pixel-by-pixel feature representation is used as the input of the feature aggregator, and the feature aggregator is used to analyze the target motion relationship between two adjacent frames to generate pixel-level fusion features to model the motion continuity between targets; the pixel-level fusion features are processed using a target decoder, and the processed features are input into a multi-layer perceptron, and the classification feature vector, center point feature vector and size feature vector of the target are generated using the multi-layer perceptron; the center heat map of the target is calculated based on the classification feature vector, center point feature vector and size feature vector, and the target in the video frame is contextually associated based on the center heat map to track the target. This application improves the accuracy of target trajectory tracking and improves the target tracking effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0014] Figure 1 It is a flowchart of a fine-grained visual multi-target tracking method provided in an embodiment of the present application;
[0015] Figure 2 is a schematic diagram of the structure of a fine-grained visual multi-target tracking device provided in an embodiment of the present application;
[0016] Figure 3 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0017] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.
[0018] Video multi-object tracking (MOT) technology is an important technology in the field of computer vision and is widely used in autonomous driving, video surveillance, behavior recognition and other fields. The main task of video multi-object tracking technology is to detect, identify and track multiple moving targets in a video sequence so as to predict their trajectories, perform precise positioning and search, etc.
[0019] With the development of computer vision technology, video multi-target tracking technology has been widely used in many fields, such as the identification and tracking of surrounding vehicles and pedestrians in autonomous driving systems, the behavior monitoring of specific groups of people in video surveillance, and the analysis of human motion in behavior recognition technology. These applications require the system to accurately detect targets in the video, identify their identities, and track the position and movement of these targets in different frames of the video sequence.
[0020] At present, most video multi-target tracking methods use target boxes to mark and track targets. For example, Wojke et al. proposed using convolutional neural networks (CNN) to detect the appearance features of targets and generate bounding boxes representing targets; the association between targets is achieved by calculating the intersection over union (IoU) of target bounding boxes in consecutive video frames. Similarly, after detecting the target boxes, Yang et al. designed a feature re-identification mechanism and a multi-cut connection formula of the graph to achieve the association between targets.
[0021] Although these methods are computationally simple and can achieve fast target location and tracking, they have obvious limitations. Since the target box usually contains background information in addition to the target itself, the information in the target box has a large semantic ambiguity. This is particularly prominent when tracking small targets or in complex and crowded scenes. The cluttered information in the target box will seriously affect the accuracy and effect of tracking.
[0022] Based on the limitations of the above-mentioned prior art, the technical problems to be solved by the technical solution of this application mainly focus on how to improve the accuracy of video multi-target tracking, especially the performance in tracking small targets or in complex and crowded scenes. Specifically, the problems that need to be solved in this application include how to reduce the background noise in the target frame and improve the clarity of the target semantics, so as to more accurately identify and track multiple targets in the video. In general, although the prior art has made certain progress in multi-target tracking, there is still much room for improvement in the performance when handling small target tracking and complex scenes. Solving this technical problem is of great significance to improving the overall performance and application scope of video multi-target tracking technology.
[0023] In view of the problems existing in the above-mentioned prior art, the technical solution proposed in this application aims to solve the problem of low accuracy of prior art in tracking small targets or complex scenes through a fine-grained visual multi-target tracking method. This technical solution extracts target features from video frames, establishes motion relationships between targets, and performs high-precision target tracking through a series of modular processing steps. The technical solution of this application mainly includes the following contents:
[0024] 1) Feature extraction: Use a deep convolutional neural network (such as ResNet-32, ResNet-50, VGG-50, etc.) with shared weights to extract deep features of the two frames of images. The purpose of this step is to capture the basic appearance and shape information of the target in the video frame.
[0025] 2) Multi-scale pixel-level feature encoding: Use a multi-layer feature encoder to upsample the extracted deep features to generate fine-grained pixel-by-pixel feature representations. This can be implemented based on CNN or attention neural networks, where the encoding process of the attention mechanism depends on the input feature vector and generates image pixel features by calculating query features, key features, and value features.
[0026] 3) Feature aggregation: The feature aggregator analyzes the target motion relationship between adjacent frames and generates pixel-level fusion features. This process calculates the motion trend between two frames and fuses the features into the same feature space, providing a continuous representation of the target motion.
[0027] 4) Target decoding and tracking: The target decoder is used to further process the fused features, and the target classification feature vector, center point feature vector and size feature vector are obtained through the multi-layer perceptron. Using these feature vectors, the center heat map of the target is calculated, and the context of the target is associated based on the center heat map to complete target tracking.
[0028] By implementing the above technical solutions, the present application can accurately track multiple targets in the video at a fine-grained level, especially when dealing with small targets or complex scenes, showing higher accuracy and efficiency than the existing technology.
[0029] The contents of the technical solution of the present application are described in detail below with reference to the accompanying drawings and specific embodiments.
[0030] Figure 1 FIG. 1 is a flow chart of a fine-grained visual multi-target tracking method provided in an embodiment of the present application. Figure 1 As shown, the fine-grained visual multi-target tracking method may specifically include:
[0031] S101, selecting two adjacent frames of images from continuous video frames, and extracting deep features of each of the two adjacent frames of images using a deep convolutional neural network;
[0032] S102, encoding the deep features using a preset multi-scale pixel-level feature encoder to generate a fine-grained pixel-by-pixel feature representation;
[0033] S103, using the pixel-by-pixel feature representation as an input to a feature aggregator, using the feature aggregator to analyze the object motion relationship between two adjacent frame images, and generating pixel-level fusion features to model the motion continuity between the objects;
[0034] S104, using a target decoder to process the pixel-level fusion features, and inputting the processed features into a multi-layer perceptron, and using the multi-layer perceptron to generate a classification feature vector, a center point feature vector, and a size feature vector of the target;
[0035] S105, calculating a central heat map of the target according to the classification feature vector, the center point feature vector and the size feature vector, and performing context association on the target in the video frame according to the central heat map, so as to track the target.
[0036] First of all, before introducing the specific embodiments of the present application in detail, the overall implementation process of the technical solution of the present application is summarized. The present application aims to provide a method for fine-grained visual multi-target tracking. The main technical solution of the present application includes: for continuous video frame images of video input, first use the feature extraction module to extract the deep features of two adjacent frames of images respectively; set a multi-scale pixel-level feature encoder to encode the deep features of two adjacent frames of images respectively, and generate multi-layer image pixel features; after obtaining the image pixel features of adjacent frames, use the feature aggregator to model the target motion relationship between adjacent frame images, and generate pixel-level fusion features; after obtaining the pixel-level fusion features, further process the fusion features through the target decoder, and generate the target classification feature vector, center point feature vector and size feature vector through the multi-layer perceptron; calculate the center heat map of the target based on the above feature vector, and calculate the context association of the tracked target in the image based on the center heat map to complete the target tracking.
[0037] In some embodiments, deep convolutional neural networks are used to extract deep features of each of two adjacent frames of images, including:
[0038] Input two adjacent frames of images into a deep convolutional neural network configured with shared weights, where the deep convolutional neural network is used to analyze the image content and extract deep features of the image;
[0039] A deep convolutional neural network is used to perform feature extraction operations on each frame of the image to obtain deep features, where the deep features contain low-resolution feature representations of the deep visual information of the image.
[0040] Specifically, this embodiment describes a method for deep feature extraction of images in continuous video frames using a deep convolutional neural network (CNN), aiming to achieve high efficiency and accuracy of fine-grained visual multi-target tracking. The following is a detailed implementation process of this embodiment:
[0041] First, two consecutive frames are extracted from the video input. These two frames serve as the basis for processing, aiming to capture the movement and changes of the target in a short period of time.
[0042] Next, a suitable deep convolutional neural network model is selected as the feature extraction module. In this embodiment, models such as ResNet-32, ResNet-50, and VGG-50 can be selected. These models are selected for their deep structure and excellent feature extraction capabilities, and the shared weight mechanism ensures computational efficiency and parameter consistency when processing consecutive frames.
[0043] Then, the two preprocessed images are fed into a deep convolutional neural network configured with shared weights. The network analyzes the image content and extracts features that represent the deep visual information of the image. These features are represented in a low-resolution form and contain enough visual information for the subsequent object detection and tracking process.
[0044] Finally, after processing through a deep convolutional neural network, the deep features of each frame are extracted. These deep features capture the key visual attributes of the target, such as shape, texture, etc., providing a strong foundation for accurate target tracking.
[0045] It should be noted that this embodiment is not limited to the use of ResNet or VGG models, and other types of deep convolutional neural network models, such as deeper networks or networks optimized for specific tasks, may also be selected according to specific application scenarios and requirements. In addition, for different tracking targets and background environments, the feature extraction process can be optimized by adjusting network parameters or introducing pre-trained models to improve the accuracy and robustness of tracking.
[0046] This embodiment describes in detail how to effectively use deep convolutional neural networks to extract deep features in video multi-target tracking tasks, laying the foundation for efficient and accurate multi-target tracking. Through appropriate network selection and parameter adjustment, the entire feature extraction process can be optimized according to specific application requirements to achieve the best tracking effect.
[0047] In some embodiments, a preset multi-scale pixel-level feature encoder is used to encode the deep features to generate a fine-grained pixel-by-pixel feature representation, including:
[0048] Using a multi-scale pixel-level feature encoder configured with shared weights, the deep features of each frame are upsampled separately to generate fine-grained pixel-by-pixel feature representation;
[0049] Among them, the multi-scale pixel-level feature encoder includes a feature encoder based on a convolutional neural network and a feature encoder based on an attention neural network.
[0050] Specifically, this embodiment also introduces in detail a method for encoding deep features of a video frame image using a preset multi-scale pixel-level feature encoder to generate a fine-grained pixel-by-pixel feature representation. This method is particularly suitable for fine-grained visual multi-target tracking and can improve the accuracy and efficiency of tracking. The following is a specific implementation description of this embodiment:
[0051] First, two adjacent frames are extracted from the continuous video frames (marked as time points t and t-1), and the deep features F of these two frames are extracted using a deep convolutional neural network. (t) With F (t-1) This step ensures that the basic visual information and dynamic changes of the target can be captured.
[0052] Furthermore, a multi-scale pixel-level feature encoder with shared weights is used to upsample the extracted deep features. This process aims to generate fine-grained pixel-by-pixel feature representations. and Where l represents the number of upsampling layers. This step provides richer and more accurate information for subsequent target detection and tracking by refining the representation of features.
[0053] Furthermore, the multi-scale pixel-level feature encoder can be based on a convolutional neural network (CNN) such as a multi-scale convolutional neural network MTCNN, a feature pyramid network FPN, etc., or based on an attention neural network such as a deformable self-attention encoder Deformable Transformer Encoder, etc. This provides flexibility to select the most appropriate encoder type according to the needs and characteristics of a specific task.
[0054] Furthermore, for the feature encoding process based on the attention neural network, the input image features are first flattened into a one-dimensional feature vector with a length of H×W×C, and then these feature vectors are input into multiple consecutive attention calculation modules to generate image pixel features. This process takes advantage of the attention mechanism and can focus more on the key information in the image, thereby improving the efficiency and accuracy of feature encoding.
[0055] In one example, when a feature vector is input into multiple consecutive attention calculation modules to generate image pixel features, the following formula can be used:
[0056]
[0057] Q=W q ·P (t)
[0058] K=W k ·P (t)
[0059] V=W v ·P (t)
[0060] Among them, P (t) represents the feature vector after the input image feature of the tth frame is flattened into 1 dimension, W q , W k , W v They represent three learnable parameter matrices respectively. The generated Q, K, and V represent query features, key features, and value features respectively. d represents the length of the generated key feature. For the generated image pixel features of the lth layer, softmax is used to calculate the normalized value of attention.
[0061] It should be noted that the present embodiment can also optimize the feature encoding process by introducing a more advanced encoder model or adjusting the parameters of the existing model to adapt to tracking environments of different complexities and changes. Combining the advantages of convolutional neural networks and attention neural networks, by fusing these two encoding methods, the quality of pixel-by-pixel feature representation and target tracking performance can be further improved. For real-time video multi-target tracking applications, the encoder can be specially optimized to reduce the amount of calculation and increase the processing speed, ensuring efficient target tracking in real-time or near real-time conditions.
[0062] Through the method of the present embodiment described above, deep features can be effectively extracted and encoded, and high-quality pixel-by-pixel feature representation can be generated, providing strong technical support for achieving fine-grained visual multi-target tracking.
[0063] In some embodiments, a feature aggregator is used to analyze the object motion relationship between two adjacent frames of images to generate pixel-level fusion features to model the motion continuity between the objects, including:
[0064] The feature aggregator is used to calculate the motion trend vector between the corresponding levels in two adjacent frames to evaluate the position change of the target between consecutive frames;
[0065] According to the motion trend vector, a predetermined algorithm is used to fuse the features of two adjacent frames of images into the same feature space to obtain a pixel-level fusion feature, wherein the pixel-level fusion feature contains a fusion feature representation of continuity information in the time dimension.
[0066] Specifically, the embodiment of the present application also elaborates on how to use the feature aggregator to analyze the motion relationship of the target between two adjacent frames of images, and based on this, generate pixel-level fusion features containing continuity information in the time dimension. This process is crucial for achieving high-precision fine-grained visual multi-target tracking. The following is the specific implementation process of this embodiment:
[0067] First, two adjacent frames are extracted from continuous video frames, and deep convolutional neural networks are used to extract the deep features of these two frames. These features serve as the input of the feature aggregator, laying the foundation for the next step of motion trend analysis.
[0068] Furthermore, the feature aggregator then calculates the motion trend vector between the corresponding levels in two adjacent frames. This step captures the motion information of the target by comparing the feature differences between the two frames and evaluating the position change of the target between consecutive frames. In order to enhance the accuracy and adaptability of this process, machine learning algorithms such as optical flow estimation can be introduced to more accurately calculate the motion trend.
[0069] Furthermore, based on the calculated motion trend vector, the feature aggregator uses a predetermined algorithm to fuse the features of two adjacent frames into the same feature space to generate pixel-level fusion features. These fusion features not only contain the visual information of the target, but also incorporate the motion continuity information between targets, providing rich contextual information for subsequent target detection and tracking.
[0070] In an example, this embodiment can be implemented by the following formula:
[0071]
[0072] in, represents the image features of the lth layer after fusion, Indicates the deformation features of the lth layer corresponding to the image features of the t-1 frame and the tth frame, ω (t-1)→(t) Indicates the similarity between the corresponding features of the t-1 frame and the t-th frame image.
[0073] Among them, the motion vector between adjacent image features can be calculated by the following formula:
[0074]
[0075] Among them, Φ(D (t-1) ,D (t) ) represents the motion trend between adjacent frame features, and τ is the deformation feature calculator.
[0076] Furthermore, based on the generation of pixel-level fusion features, these features can be further optimized, for example, by using noise reduction, feature enhancement and other technologies to improve feature availability and target tracking accuracy. In addition, these fusion features can be directly applied to target detection, recognition and tracking algorithms to achieve accurate positioning and tracking of targets.
[0077] This embodiment uses a feature aggregator to analyze and fuse the motion information of two adjacent frames of images to generate pixel-level fusion features that reflect the continuity of target motion. This method can not only improve the accuracy and robustness of multi-target tracking, but also effectively track small targets or multiple targets in crowded scenes in complex dynamic environments. In addition, the method of this embodiment has high adaptability and can be adjusted and optimized according to different application scenarios and tracking requirements, and has broad practical value and application prospects.
[0078] In some embodiments, the pixel-level fusion features are processed by a target decoder, and the processed features are input into a multi-layer perceptron, and the classification feature vector, the center point feature vector and the size feature vector of the target are generated by the multi-layer perceptron, including:
[0079] The pixel-level fusion features are decoded using a pre-configured target decoder. During the decoding process, the pixel-level fusion features of each scale are analyzed and converted using the target decoder.
[0080] The features processed by the target decoder are input into three multi-layer perceptrons, and the features are refined by the multi-layer perceptrons. The three multi-layer perceptrons are used to generate the classification feature vector, the center point feature vector and the size feature vector of the target respectively;
[0081] Among them, the classification feature vector is used to characterize different types of targets, the center point feature vector is used to characterize the center position of the target in the image, and the size feature vector is used to characterize the size information of the target.
[0082] Specifically, this embodiment uses a target decoder to process pixel-level fusion features, and further uses a multi-layer perceptron (MLP) to generate a classification feature vector, a center point feature vector, and a size feature vector of the target, in order to achieve fine-grained visual multi-target tracking. The following is a specific implementation process of this embodiment:
[0083] First, the pixel-level fusion features obtained from the feature aggregator are decoded using a pre-configured target decoder. The key to this step is to analyze and transform the pixel-level fusion features at each scale to extract effective information for target tracking.
[0084] The decoder can be based on a traditional decoding algorithm, or can use deep learning technology such as a decoder based on a convolutional neural network (CNN) or a self-attention network (such as Transformer). The most appropriate decoding technology can be selected according to actual needs.
[0085] Furthermore, the features processed by the target decoder are then input into three independently configured multi-layer perceptrons, which further refine the features to accurately represent the category, center position, and size information of the target.
[0086] Among them, the three multi-layer perceptrons are responsible for generating the following key feature vectors:
[0087] Categorical feature vector Used to distinguish different types of targets;
[0088] Center point feature vector Used to identify the center position of the target in the image;
[0089] Size feature vector Used to describe the size information (width and height) of the object.
[0090] Furthermore, the performance of the multilayer perceptron is optimized by adjusting the number of layers, activation function or training strategy to more accurately generate the classification, center point and size feature vectors of the target. The collaboration between the decoder and the multilayer perceptron is crucial. By selecting the multilayer perceptron structure and parameters that match the decoder output features, the efficiency of feature utilization and the accuracy of target tracking can be effectively improved.
[0091] Furthermore, for real-time video analysis and multi-target tracking applications, by optimizing the implementation of the target decoder and multi-layer perceptron, such as using more efficient algorithms and hardware acceleration technology, the processing speed can be significantly improved to meet the needs of real-time applications.
[0092] This embodiment effectively extracts key information for multi-target tracking from pixel-level fusion features, including the category, position and size of the target, through the application of a carefully designed target decoder and a multi-layer perceptron. This method not only improves the accuracy and efficiency of multi-target tracking, but also enhances the system's ability to handle complex scenes, and has broad application prospects.
[0093] In some embodiments, the central heat map of the target is calculated based on the classification feature vector, the center point feature vector, and the size feature vector, including using the following formula to calculate the central heat map:
[0094]
[0095]
[0096] Among them, i∈[1,N] represents the i-th candidate target, j∈[1,2] corresponds to the width and height respectively, and k l [i,h,w] represents the probability that the position with the corresponding coordinates (h,w) in the image feature of the lth layer is the center point of the i-th target, s l [i,h,w,j] represents the length and width of the target bounding box corresponding to the center point with coordinates (h,w) in the image features of the lth layer. The center heat map of the target is obtained by combining the vector k with the vector s.
[0097] Specifically, three multi-layer perceptrons are used to post-process the features to generate classification feature vectors Center point feature vector and size feature vector Afterwards, the embodiment of the present application will be based on the pixel fusion feature The target center point coordinates and the target width and height are used to calculate the target center heat map and model size heat map distribution. For the specific calculation method, please refer to the center heat map calculation formula provided in the above embodiment, which will not be repeated here.
[0098] In some embodiments, contextualizing an object in a video frame according to a central heat map so as to track the object includes:
[0099] The center heat map is used to evaluate the center position of each target in the video frame, so as to identify the high response area on the center heat map to locate the center point of the target;
[0100] Using the preset target association method, the correlation between the predicted target and the real target in the continuous video frames is calculated according to the target center position information obtained from the center heat map;
[0101] The target association method is used to analyze and determine the position change and continuity of each target between the current frame and the next frame to achieve target tracking.
[0102] Specifically, the embodiment of the present application also introduces in detail a target context association method based on a center heat map for accurate tracking of targets in video frames. The method uses the generated center heat map to evaluate the target center position, and uses an advanced target association algorithm to calculate the correlation between targets, thereby tracking the target in continuous video frames. The following is the specific implementation process of this embodiment:
[0103] First, a center heat map representing the center position of each target in the video frame is generated by the multi-layer perceptron processing of the previous embodiment. Each high response area on the center heat map corresponds to the center point of a target.
[0104] These center heatmaps are then evaluated to identify high-response regions to localize the center position of each object. This step is the basis for subsequent object association and tracking.
[0105] Then, using the preset target association method (such as Kalman filter algorithm, Hungarian algorithm or deep Hungarian network, etc.), the correlation between the predicted target and the real target between consecutive video frames is calculated based on the target center position information obtained from the center heat map. These target association methods can analyze and determine the position change and continuity of each target between the current frame and the next frame, and realize effective tracking of the target.
[0106] It should be noted that, according to different application scenarios and specific tracking requirements, appropriate target association methods can be selected and their parameters can be adjusted and optimized to improve the accuracy and robustness of tracking. In order to meet the needs of real-time or near real-time video surveillance, the speed of central heat map generation and target association calculation can be improved through technical means such as algorithm optimization and parallel computing, thereby improving the real-time processing capability of the entire system.
[0107] In addition, in complex environments or crowded scenes, occlusion and interaction between targets may affect the tracking effect. By introducing more advanced algorithms, such as deep learning models, the system's adaptability and accuracy to these challenges can be further improved.
[0108] This embodiment achieves accurate tracking of multiple targets in a video frame through precise center point positioning and advanced target association methods. This method is not only applicable to conventional monitoring scenarios, but also can cope with high-demand application scenarios such as autonomous driving and behavior recognition, showing a wide range of application potential and practical value.
[0109] The technical solution of the present application introduces a deep learning model to finely model the motion relationship between two adjacent frames of images, and innovatively maps this motion relationship into the same feature space through feature fusion, generating a central heat map with highly fine-grained feature information. Compared with the tracking technology that relies on traditional target frames, this method significantly improves the accuracy of target trajectory tracking, especially when dealing with target tracking problems in crowded scenes, and significantly reduces the loss rate of target trajectories. According to the technical solution provided in the embodiments of the present application, the technical solution of the present application has at least the following advantages:
[0110] By analyzing the motion relationship between two adjacent frames of images through a deep learning model and performing feature fusion, this technical solution can provide richer and more detailed target feature information. Compared with the traditional target box method, the center heat map can more accurately reflect the position of the target, thereby achieving higher accuracy target tracking in complex dynamic environments.
[0111] In crowded scenes, occlusion and cross-movement between multiple targets can easily lead to tracking loss. This technical solution can effectively distinguish closely adjacent or partially occluded targets through fine-grained center heat maps, significantly reducing the tracking loss rate in such scenes.
[0112] This technical solution not only improves tracking accuracy and reduces loss rate, but also enhances the robustness of target trajectory tracking through the feature fusion mechanism of the deep model. It can maintain stable tracking performance even when the target moves quickly or the lighting conditions change.
[0113] Due to the significant improvements in target tracking accuracy, loss rate reduction and robustness, this technical solution has a wide range of application potential, especially in the fields of autonomous driving, video surveillance, crowd analysis, etc., and can effectively cope with the challenges of multi-target tracking in various complex scenarios.
[0114] Therefore, in summary, the technical solution of the present application has achieved a major technological breakthrough in the field of multi-target tracking through the introduction of deep learning models and innovative feature fusion methods, and has provided an effective solution to the limitations of traditional methods in accuracy, loss rate and robustness.
[0115] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.
[0116] Figure 2 Schematic diagram of the structure of a fine-grained visual multi-target tracking device provided in an embodiment of the present application. Figure 2 As shown, the fine-grained visual multi-target tracking device includes:
[0117] The extraction module 201 is configured to select two adjacent frames of images from continuous video frames, and extract deep features of each of the two adjacent frames of images using a deep convolutional neural network;
[0118] The encoding module 202 is configured to encode the deep features using a preset multi-scale pixel-level feature encoder to generate a fine-grained pixel-by-pixel feature representation;
[0119] The analysis module 203 is configured to use the pixel-by-pixel feature representation as an input to a feature aggregator, and use the feature aggregator to analyze the object motion relationship between two adjacent frames of images to generate pixel-level fusion features to model the motion continuity between the objects;
[0120] The processing module 204 is configured to process the pixel-level fusion features using the target decoder, and input the processed features into a multi-layer perceptron, and use the multi-layer perceptron to generate a classification feature vector, a center point feature vector, and a size feature vector of the target;
[0121] The calculation module 205 is configured to calculate the center heat map of the target according to the classification feature vector, the center point feature vector and the size feature vector, and to perform context association on the target in the video frame according to the center heat map so as to track the target.
[0122] In some embodiments, Figure 2 The extraction module 201 inputs two adjacent frames of images into a deep convolutional neural network configured with shared weights, wherein the deep convolutional neural network is used to analyze the image content and extract the deep features of the image; the deep convolutional neural network is used to perform feature extraction operations on each frame of the image to obtain deep features, wherein the deep features contain low-resolution feature representations of the deep visual information of the image.
[0123] In some embodiments, Figure 2 The encoding module 202 uses a multi-scale pixel-level feature encoder configured with shared weights to upsample the deep features of each frame image respectively to generate a fine-grained pixel-by-pixel feature representation; wherein the multi-scale pixel-level feature encoder includes a feature encoder based on a convolutional neural network and a feature encoder based on an attention neural network.
[0124] In some embodiments, Figure 2 The analysis module 203 uses a feature aggregator to calculate the motion trend vector between corresponding levels in two adjacent frame images to evaluate the position change of the target between consecutive frames; according to the motion trend vector, a predetermined algorithm is used to fuse the features of the two adjacent frame images into the same feature space to obtain a pixel-level fusion feature, wherein the pixel-level fusion feature contains a fusion feature representation of continuity information in the time dimension.
[0125] In some embodiments, Figure 2 The processing module 204 uses a pre-configured target decoder to decode the pixel-level fusion features. During the decoding process, for the pixel-level fusion features of each scale, the target decoder is used to analyze and convert the pixel-level fusion features; the features processed by the target decoder are input into three multi-layer perceptrons, and the multi-layer perceptrons are used to refine the features. The three multi-layer perceptrons are used to generate classification feature vectors, center point feature vectors and size feature vectors of the targets respectively; wherein the classification feature vectors are used to characterize different types of targets, the center point feature vectors are used to characterize the center position of the target in the image, and the size feature vectors are used to characterize the size information of the target.
[0126] In some embodiments, Figure 2 The calculation module 205 calculates the center heat map using the following formula:
[0127]
[0128]
[0129] Among them, i∈[1,N] represents the i-th candidate target, j∈[1,2] corresponds to the width and height respectively, and k l [i,h,w] represents the probability that the position with the corresponding coordinates (h,w) in the image feature of the lth layer is the center point of the i-th target, s l [i,h,w,j] represents the length and width of the target bounding box corresponding to the center point with coordinates (h,w) in the image features of the lth layer. The center heat map of the target is obtained by combining the vector k with the vector s.
[0130] In some embodiments, Figure 2 The calculation module 205 uses the central heat map to evaluate the central position of each target in the video frame, so as to identify the high response area on the central heat map to locate the center point of the target; uses a preset target association method to calculate the correlation between the predicted target and the real target in the continuous video frames according to the target center position information obtained from the central heat map; uses the target association method to analyze and determine the position change and continuity of each target in the current frame and the next frame to achieve tracking of the target.
[0131] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0132] Figure 3 Schematic diagram of the structure of the electronic device 3 provided in the embodiment of the present application. Figure 3 As shown, the electronic device 3 of this embodiment includes: a processor 301, a memory 302, and a computer program 303 stored in the memory 302 and executable on the processor 301. When the processor 301 executes the computer program 303, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 301 executes the computer program 303, the functions of the modules / units in the above-mentioned device embodiments are implemented.
[0133] Exemplarily, the computer program 303 may be divided into one or more modules / units, which are stored in the memory 302 and executed by the processor 301 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 303 in the electronic device 3.
[0134] The electronic device 3 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 3 may include, but is not limited to, a processor 301 and a memory 302. Those skilled in the art will appreciate that Figure 3 It is only an example of the electronic device 3 and does not constitute a limitation of the electronic device 3. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.
[0135] The processor 301 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.
[0136] The memory 302 may be an internal storage unit of the electronic device 3, for example, a hard disk or memory of the electronic device 3. The memory 302 may also be an external storage device of the electronic device 3, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 3. Further, the memory 302 may also include both an internal storage unit and an external storage device of the electronic device 3. The memory 302 is used to store computer programs and other programs and data required by the electronic device. The memory 302 may also be used to temporarily store data that has been output or is to be output.
[0137] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.
[0138] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0139] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0140] In the embodiments provided in the present application, it should be understood that the disclosed devices / computer equipment and methods can be implemented in other ways. For example, the device / computer equipment embodiments described above are only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. Multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0141] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0142] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0143] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, which may be in source code form, object code form, executable file or some intermediate form. Computer-readable media may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.
[0144] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. A fine-grained visual multi-target tracking method, It is characterized in that include: Selecting two adjacent frames of images from continuous video frames, and extracting deep features of each of the two adjacent frames of images using a deep convolutional neural network; Encoding the deep features using a preset multi-scale pixel-level feature encoder to generate a fine-grained pixel-by-pixel feature representation; The pixel-by-pixel feature representation is used as an input of a feature aggregator, and the feature aggregator is used to analyze the object motion relationship between the two adjacent frames of images to generate pixel-level fusion features to model the motion continuity between the objects; The pixel-level fusion features are processed by a target decoder, and the processed features are input into a multi-layer perceptron, and a classification feature vector, a center point feature vector and a size feature vector of the target are generated by the multi-layer perceptron; The central heat map of the target is calculated according to the classification feature vector, the center point feature vector and the size feature vector, and the target in the video frame is contextually associated according to the central heat map so as to track the target.
2. The method according to claim 1, It is characterized in that The extracting deep features of each frame of the two adjacent frames of images by using a deep convolutional neural network includes: Inputting the two adjacent frames of images into a deep convolutional neural network configured with shared weights respectively, wherein the deep convolutional neural network is used to analyze image content and extract deep features of the image; The deep convolutional neural network is used to perform a feature extraction operation on each frame of the image to obtain the deep features, wherein the deep features contain low-resolution feature representations of deep visual information of the image.
3. The method according to claim 1, It is characterized in that The method of encoding the deep features by using a preset multi-scale pixel-level feature encoder to generate a fine-grained pixel-by-pixel feature representation includes: Using a multi-scale pixel-level feature encoder configured with shared weights, upsampling the deep features of each frame of the image to generate a fine-grained pixel-by-pixel feature representation; Among them, the multi-scale pixel-level feature encoder includes a feature encoder based on a convolutional neural network and a feature encoder based on an attention neural network.
4. The method according to claim 1, It is characterized in that The using the feature aggregator to analyze the target motion relationship between the two adjacent frames of images to generate pixel-level fusion features to model the motion continuity between the targets includes: Using the feature aggregator to calculate the motion trend vector between the corresponding levels in the two adjacent frames of images to evaluate the position change of the target between consecutive frames; According to the motion trend vector, a predetermined algorithm is used to fuse the features of two adjacent frames of images into the same feature space to obtain the pixel-level fusion feature, wherein the pixel-level fusion feature includes a fusion feature representation of continuity information in the time dimension.
5. The method according to claim 1, It is characterized in that The method of processing the pixel-level fusion features by using the target decoder and inputting the processed features into a multi-layer perceptron, and using the multi-layer perceptron to generate a classification feature vector, a center point feature vector and a size feature vector of the target includes: Decoding the pixel-level fusion features using a pre-configured target decoder, and in the decoding process, analyzing and converting the pixel-level fusion features at each scale using the target decoder; Input the features processed by the target decoder into three multi-layer perceptrons, and use the multi-layer perceptrons to refine the features. The three multi-layer perceptrons are used to generate a classification feature vector, a center point feature vector, and a size feature vector of the target respectively; The classification feature vector is used to characterize different types of targets, the center point feature vector is used to characterize the center position of the target in the image, and the size feature vector is used to characterize the size information of the target.
6. The method according to claim 1, It is characterized in that The calculating of the central heat map of the target according to the classification feature vector, the center point feature vector and the size feature vector comprises calculating the central heat map using the following formula: Among them, i∈[1,N] represents the i-th candidate target, j∈[1,2] corresponds to the width and height respectively, and k l [i,h,w] represents the probability that the position with the corresponding coordinates (h,w) in the image feature of the lth layer is the center point of the i-th target, s l [i,h,w,j] represents the length and width of the target bounding box corresponding to the center point with coordinates (h,w) in the image features of the lth layer. The vector k is combined with the vector s to obtain the center heat map of the target.
7. The method according to claim 1, It is characterized in that The performing context association on the target in the video frame according to the central heat map so as to track the target includes: Using the center heat map, the center position of each target in the video frame is evaluated, so as to identify the high response area on the center heat map to locate the center point of the target; Using a preset target association method, the association degree between the predicted target and the real target in the continuous video frames is calculated according to the target center position information obtained from the center heat map; The target association method is used to analyze and determine the position change and continuity of each target between the current frame and the next frame, so as to achieve tracking of the target.
8. A fine-grained visual multi-target tracking device, It is characterized in that include: An extraction module is configured to select two adjacent frames of images from continuous video frames, and use a deep convolutional neural network to extract deep features of each of the two adjacent frames of images; An encoding module is configured to encode the deep features using a preset multi-scale pixel-level feature encoder to generate a fine-grained pixel-by-pixel feature representation; An analysis module is configured to use the pixel-by-pixel feature representation as an input of a feature aggregator, analyze the object motion relationship between the two adjacent frames of images using the feature aggregator, and generate pixel-level fusion features to model the motion continuity between the objects; A processing module is configured to process the pixel-level fusion features using a target decoder, input the processed features into a multi-layer perceptron, and use the multi-layer perceptron to generate a classification feature vector, a center point feature vector, and a size feature vector of the target; The calculation module is configured to calculate the central heat map of the target according to the classification feature vector, the center point feature vector and the size feature vector, and to perform context association on the target in the video frame according to the central heat map so as to track the target.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium storing a computer program. It is characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
Moving target detection method based on particle filtering visual attention model
CN104050685A
Multi-view indoor pedestrian tracking method based on movement behavior mode
CN104680559A
CNN and SIFT optical flow fused self-adaptive moving target detection method
CN109934283A
Joint segmentation tracking depth estimation model training method and use method
CN117115786A
Efficient track-before detect algorithm with minimal prior knowledge
US10445862B1
Cited By
Multi-target tracking method and device, equipment and storage medium
CN120147365A
