Attention-based visual target tracking methods and terminals
By merging target detection and tracking in the same network model, and utilizing the attention mechanism and Kalman filter in the visual target tracking method, the problems of high computational cost and long latency in the existing technology are solved, and more efficient and accurate multi-target tracking is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- UNIV OF ELECTRONICS SCI & TECH OF CHINA
- Filing Date
- 2023-10-31
- Publication Date
- 2026-07-31
AI Technical Summary
Existing two-stage multi-target tracking methods are computationally expensive, have long latency, and are highly complex, making them difficult to meet the needs of real-time applications.
A visual target tracking method based on attention mechanism is adopted, which combines target detection and tracking into the same network model. The joint loss function of target detection and tracking is optimized through joint training, and the Transformer model is used to extract multi-head attention mechanism and Kalman filter for feature fusion and target matching.
It reduces computational and inference complexity, improves the accuracy and real-time performance of object detection, and enhances the model's generalization ability and object tracking performance.
Smart Images
Figure CN117372721B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of target tracking technology, and in particular to a visual target tracking method and terminal based on an attention mechanism. Background Technology
[0002] Object tracking is an important research area in computer vision. With the continuous development of computer technology and image processing technology, the description of object states and motion behaviors in various application scenarios is gradually increasing. Object tracking algorithms can track and locate moving targets in the real world and are widely used in fields such as intelligent video surveillance, traffic detection and management, autonomous driving, and virtual reality games. In the transportation field, tracking vehicles, pedestrians, and other transportation tools can provide scientific data for road congestion and accident early warning. Therefore, in-depth research and solutions to the object tracking problem have significant practical application value and research significance.
[0003] The widely adopted two-stage multi-object tracking method relies on two independent models. First, an object detection model is used to locate objects in the image to determine their bounding boxes. Next, an association model extracts re-identification (Re-ID) features from each bounding box and uses specific metrics to associate these bounding boxes with already tracked objects. The main task of the object detection model is to detect all existing objects in the current frame, while Re-ID is used to associate objects in the current frame with objects in previous frames. This association process relies on distance comparisons of Re-ID feature vectors and the intersection-union ratio (IOU) between object bounding boxes. However, this two-stage algorithm has some drawbacks:
[0004] (1) High computational cost: Two-stage algorithms usually require two separate stages: target detection and target tracking, which may result in high computational cost, especially in large-scale target tracking or real-time applications.
[0005] (2) Delay: Since it is divided into two stages, the two-stage multi-target tracking algorithm may introduce a certain delay, which may not be suitable for applications that require real-time response, such as real-time multi-target tracking.
[0006] (3) Complexity: Two-stage algorithms usually require complex architectures and models, including independent detectors and trackers, and require more parameter tuning and optimization. Summary of the Invention
[0007] The purpose of this invention is to overcome the problems of the prior art and provide a visual target tracking method and terminal based on an attention mechanism.
[0008] The objective of this invention is achieved through the following technical solution: a visual target tracking method based on an attention mechanism, the method comprising the following steps:
[0009] The image is input into the feature extraction unit to collect feature information of the input image;
[0010] The feature information is input into the target detection model to perform target detection, and the target detection result including the location information is obtained.
[0011] The feature information is input into the Re-ID model to generate Re-ID features representing the identity of each target. The similarity of the Re-ID features is compared to re-identify the targets, and then the targets that have already appeared are associated.
[0012] The target detection results and Re-ID features are input into the Tracking module for target tracking and matching, including the following sub-steps:
[0013] Assign a unique ID to each detected target in the first frame image;
[0014] Predict the next location of each tracked target;
[0015] Target matching is performed based on distance and similarity, associating the detected target in the current frame with the tracked target;
[0016] Based on the target association results, update the status of the tracked target and the target re-identification information, and repeat the above target tracking and matching steps in consecutive frame images to achieve multi-target matching and tracking.
[0017] In one example, the feature information of the acquired input image includes:
[0018] The first feature extraction module collects the first feature information of the input image;
[0019] The second feature extraction module performs feature acquisition based on the input first feature information to obtain the second feature information;
[0020] The first and second feature information are fused to obtain fused features, which are then used as the output of the feature extraction unit.
[0021] In one example, the first feature extraction module is a ResNet-34 model, including several residual blocks; the second feature extraction module is a Transformer model, including a Transformer module, which includes a multi-head attention layer and a feedforward network layer; the feature fusion process then includes:
[0022] The output of the Transformer module is divided into several independent feature vectors;
[0023] The segmented feature vectors are fused with the output features of the residual block to obtain fused features that integrate the localization information in the first feature information and the positional and semantic information in the second feature information.
[0024] In one example, when the input image consists of multiple downsampled images of different resolutions, the feature fusion process also includes:
[0025] Several feature vectors obtained from input images of the same resolution are fused with the residual block output features of corresponding images of the same resolution to obtain several first intermediate fused features.
[0026] The adjacent first intermediate fusion features are fused to obtain the second intermediate fusion feature;
[0027] The second intermediate fusion feature is processed by full connection to obtain the final fusion feature.
[0028] In one example, the method further includes: performing multi-scale fusion processing on the target detection results output by the target detection model and the Re-ID features representing the identities of each target output by the Re-ID model, and using the resulting multi-scale fused features as input information for target tracking and matching.
[0029] In one example, the multi-scale fusion process includes:
[0030] Normalize the Re-ID features and target detection results;
[0031] The target detection results are upgraded in dimensionality, while the Re-ID features are reduced in dimensionality.
[0032] The target detection results and Re-ID features of different dimensions are weighted and fused to obtain multi-scale fusion features, which are used as input information for target tracking and matching.
[0033] In one example, during the target tracking and matching process, a Kalman filter is used to predict the next position of each tracked target.
[0034] In one example, during the target tracking and matching process, the Hungarian algorithm is used to associate the target detected in the current frame with the target being tracked.
[0035] It should be further noted that the technical features corresponding to the above examples can be combined or replaced to form new technical solutions.
[0036] The present invention also includes a storage medium storing computer instructions that, when executed, perform the steps of the attention-based visual target tracking method formed by any or more of the above examples.
[0037] The present invention also includes a terminal comprising a memory and a processor, the memory storing computer instructions executable on the processor, wherein the processor, when executing the computer instructions, performs the steps of the attention-based visual target tracking method formed by any or more of the above examples.
[0038] Compared with the prior art, the beneficial effects of the present invention are:
[0039] 1. In one example, this invention employs a tracking model jointly trained with the target detection model to execute the target tracking and matching algorithm. This merges target detection and tracking into the same network model, thereby reducing computational and inference complexity, as well as computational cost and latency. Furthermore, this invention optimizes the joint loss function for target detection and target tracking during joint training. Traditional two-stage tracking algorithms rely entirely on the accuracy of target detection for target matching, while in this invention, the input to the Re-ID part is complete features provided by the feature extraction module, operating independently and in parallel with the target detection module. Therefore, this approach can, to some extent, avoid false positives and false negatives caused by errors in the target detection module, thus improving overall detection performance.
[0040] 2. In one example, fusing the features extracted by two feature extraction models can capture richer feature information, thereby improving the expressive power of the features and facilitating the improvement of the reliability of subsequent object detection and tracking.
[0041] 3. In one example, the output of the Transformer module is divided into several independent feature vectors, and the feature information of each sub-head in the multi-head attention mechanism is extracted. Each of the independent feature vectors carries different output information of the Transformer module. They are then fused with the output of the residual block, thereby further enhancing the generalization ability of the model and obtaining rich features that integrate localization information, position information and semantic information, thus improving the feature extraction capability.
[0042] 4. In one example, the input image is downsampled, and then feature information from images at different resolutions is extracted. Feature information at the same resolution is then fused to obtain intermediate fused features. Adjacent first intermediate fused features are then fused together before feature concatenation, achieving the fusion of features at lower and higher resolution levels. Fusing adjacent first intermediate fused features ensures that more critical features in the middle of the pyramid structure provide more feature information than those at the top and bottom of the pyramid. This increases effective information compared to traditionally adding global information, while ignoring a small amount of unimportant information, thus increasing the model's generalization ability.
[0043] 5. In one example, the target detection results are fused with the Re-ID features at multiple scales to combine the location information provided by the target detection results with the target identity information provided by the Re-ID features. This achieves effective utilization of multi-scale information and improves target tracking performance. Attached Figure Description
[0044] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings, which are used to provide a further understanding of the present application and constitute a part of the present application. The same reference numerals are used in these drawings to denote the same or similar parts. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation of the present application.
[0045] Figure 1 This is a flowchart of a tracking method in an example of the present invention;
[0046] Figure 2 This is a model framework diagram corresponding to the tracking method in an example of the present invention;
[0047] Figure 3 This is a feature fusion corresponding model framework diagram in an example of the present invention. Detailed Implementation
[0048] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0049] In the description of this invention, it should be noted that the directions or positional relationships indicated by terms such as "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer" are based on the directions or positional relationships shown in the accompanying drawings and are only for the convenience of describing this invention and simplifying the description. They do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as limiting this invention. Furthermore, the use of ordinal numbers (e.g., "first and second," "first to fourth," etc.) is for distinguishing objects and is not limited to this order, and should not be construed as indicating or implying relative importance.
[0050] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "joining" should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.
[0051] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0052] In one example, such as Figure 1 As shown, the attention-based visual target tracking method includes the following steps:
[0053] S1: Input the image into the feature extraction unit and collect the feature information of the input image;
[0054] S2: Input the feature information into the target detection model to perform target detection, and obtain the target detection results, including location information and category information;
[0055] S3: Input the feature information into the Re-ID model to generate Re-ID features representing the identity of each target. By comparing the similarity of the Re-ID features, the targets are re-identified, and then the targets that have already appeared are associated.
[0056] S4: Input the target detection results and Re-ID features into the Tracking module for target tracking and matching, including the following sub-steps:
[0057] S41: Initialization: For each detected target, initialize a tracker and assign a unique ID. The state vector and covariance matrix of each tracked target are also initialized.
[0058] S42: Prediction and Update: At each time step, for each tracked target, state prediction (position and velocity, etc.) is performed, preferably using Kalman filtering. The covariance matrix of the Kalman filter is updated to reflect the uncertainty of the prediction. For each associated tracked target, a measurement update step using Kalman filtering is used to update its state vector and covariance matrix based on the detection results. For unassociated tracked targets, their predicted state is maintained so that they can be reassociated in the future.
[0059] S43: Data Association: Calculate the distance and similarity between each predicted tracking target and all detected targets in the current frame, and preferably use the Hungarian algorithm to associate the tracking target with the detected targets to minimize the total association cost; after the association is completed, each tracking target is associated with one or more detected targets.
[0060] S44: Create a new track: For a new detected target that is not associated with any existing track target, initialize it as a new track target and assign it a unique ID;
[0061] S45: End Tracking: For tracked targets that have not been associated for a long time, remove them or mark them as inactive. This loop repeats continuously in consecutive frames to track target motion and provide continuous trajectory information. Kalman filtering can be used to smooth the target's trajectory and provide a good estimate of the target's position in the next frame, while the Hungarian algorithm can associate the target in the current frame with the target whose predicted position best matches, thus achieving multi-target matching. The entire process ensures the accuracy and continuity of multi-target tracking. In this way, Kalman filtering and the Hungarian algorithm can work together, combining position information and Re-ID information to achieve target matching and updating in multi-target tracking, which helps improve the accuracy and stability of multi-target tracking.
[0062] In step S1, the feature extraction unit can be a terminal loaded with a neural network model. The neural network model includes, but is not limited to, convolutional neural networks, SiamRPN, HRNet, ResNet series networks, and Transformer models. In this example, the Transformer model is preferred.
[0063] Among them, SiamRPN is a network architecture specifically designed for target tracking. It uses Siamese networks and Region Candidate Networks (RPNs) for tracking and has very powerful performance in visual target tracking tasks.
[0064] HRNet is a network architecture focused on providing high-resolution feature representations, which is particularly suitable for handling target scale variations and detailed information in multi-target tracking, and has achieved excellent results in some multi-target tracking tasks.
[0065] ResNet family networks, such as ResNet-50 and ResNet-101, typically perform well in multi-object tracking tasks. They possess deep hierarchical structures and powerful feature extraction capabilities, making them suitable for various multi-object tracking scenarios. ResNet-34, a variant of the ResNet family, reduces computational complexity to some extent compared to deeper ResNet models while still providing good feature representation capabilities. Its advantage lies in its relatively shallow yet powerful feature representation backbone. Compared to deeper ResNet variants, ResNet-34 has lower computational complexity, is easier to train, and typically provides sufficient performance in multi-object tracking tasks. Furthermore, ResNet-34's performance has been thoroughly validated in many multi-object tracking tasks, making it a stable and reliable choice, especially suitable for resource-constrained scenarios or those requiring real-time performance.
[0066] The Transformer model comprises a self-attention mechanism and feedforward neural network layers. Self-attention allows for global, adaptive associations between features, capturing long-range dependencies, which is highly helpful for detecting complex structures and contextual information in images. It helps the model understand the relationships between different regions, thus better capturing image semantics. Feedforward neural network layers perform non-linear transformations at each feature location, aiding in the extraction of higher-level feature representations. These layers typically include multilayer perceptrons, which enhance the abstraction of features.
[0067] In step S2, the object detection model includes, but is not limited to, YOLO, Faster R-CNN, SSD, and CenterNet, and is loaded and run on the terminal. In this example, the CenterNet model is preferred. YOLO is a popular object detection method that treats object detection as a regression problem, predicting both the object's location and category. The YOLO series includes versions such as YOLOv1, YOLOv2, YOLOv3, and YOLOv4, which achieve a good balance between speed and accuracy.
[0068] Faster R-CNN is a classic anchor-based object detection method that uses a Region Proposal Network (RPN) to generate candidate regions, followed by object classification and bounding box regression. Although it employs an anchor-based approach, it can also serve as the basis for object detection in multi-object tracking.
[0069] SSD is another anchor-based object detection method that uses a multi-layer convolutional network to simultaneously predict multiple anchor boxes of different scales and aspect ratios, achieving a good balance between speed and accuracy.
[0070] CenterNet is a deep learning model for object detection. Its core idea is to detect objects by predicting the center point of each object in an image, and then inferring the bounding box or keypoint location of the object. Compared to traditional object detection methods, CenterNet does not require anchor boxes, resulting in higher detection performance and computational efficiency, making it suitable for real-time tasks. Compared to traditional methods such as YOLO, Faster R-CNN, and SSD, CenterNet offers advantages such as higher object localization accuracy, no need for predefined anchor boxes, reduced false detections, high computational efficiency, and strong multi-task scalability. By predicting the center point location (positional information) of objects, CenterNet provides a more accurate and reliable object detection foundation for multi-object tracking, especially excelling in handling dense objects and complex scenes.
[0071] More specifically, this example of object detection first directly inputs the feature information output from step S1 into the CenterNet network. The CenterNet network structure includes an intermediate feature extraction layer and a detection head. The intermediate feature extraction layer extracts target location information and feature maps from the input fused features. Then, the detection head, the core component of CenterNet, uses the output of the intermediate feature extraction layer to generate the object detection result. In the detection head, the feature maps are further processed to predict the target's location, confidence score, and class probability distribution. Finally, based on the output of the detection head, post-processing steps such as non-maximum suppression (NMS) are used to filter and merge the detection results to obtain the final object detection result, including the target's location and class information, which can be used for subsequent object tracking.
[0072] In step S3, the Re-ID model includes, but is not limited to, SiameseNetWork, Triplet Network, and Attention-Based, and is loaded and run on the terminal. In this example, the Attention-Based model is preferred. SiameseNetWork is a classic Re-ID method that performs well and is widely used in multi-object tracking tasks. This method trains by comparing the similarity of the feature representations of two object pairs. Triplet Network is also a commonly used Re-ID method; it learns feature embeddings through triplet loss to better distinguish objects with different identities. The Attention-Based method allows the model to focus on important parts of the target features, thereby improving Re-ID performance.
[0073] More specifically, this example Re-ID task first feeds the feature information output from step S1 into an Attention-Based model. This Re-ID model includes an attention mechanism that automatically weights and focuses on the fused input features to better capture the relationships between features and the similarities between targets. Inside the Attention-Based Re-ID model, the attention mechanism calculates the weights of different feature subsets to highlight the information most important for target recognition. This process helps the model better distinguish the similarities and differences between different targets. Finally, the model generates a feature representation or embedding vector for each target to represent its identity features. By comparing the similarity of these embedding vectors, target re-identification can be performed, recognizing and associating previously encountered targets.
[0074] In step S4, the model performance is improved by optimizing the joint loss function for object detection and object tracking. This joint training step is preferably performed before step S1. Specifically, the expression for the joint loss function is:
[0075]
[0076] Among them, L detection It is the loss for the detection task, including object detection and bounding box offset, size loss, and it measures the model's performance in detecting the center position of the object and accurately locating the bounding box. L indentity The loss is for the Re-ID task, used to learn the re-identification features of the target to ensure significant re-identification differences between different objects. w1 and w2 are learnable weight parameters used to balance the importance between the detection and re-identification tasks; they determine the relative weight of each task in the overall loss. By adjusting these parameters, the impact of different tasks on the model can be controlled. and This is the exponential function used to adjust the weight parameters w1 and w2. By applying the exponential function, we can ensure that the weight parameters are positive and make them easier to adjust to find a suitable balance during training. The total loss L... total This is used to comprehensively consider the loss of the detection and re-identification tasks, and adjusts their relative importance through weight parameters w1 and w2. This allows for balancing the two tasks during training to obtain optimal performance. Depending on the specific problem and dataset, w1 and w2 can be adjusted to meet different needs. After training is completed, the center point position of the object is obtained through step S2, and the Re-ID features are obtained through step S3. Then, the cost matrix is solved, and matching is performed.
[0077] In this example, a tracking model jointly trained with the object detection model is used to execute the object tracking and matching algorithm. Object detection and tracking are merged into the same network model, thereby reducing computational and inference complexity, as well as computational cost and latency. Furthermore, this invention optimizes the joint loss function for object detection and tracking during joint training, which to some extent avoids false positives and false negatives, improving overall detection performance.
[0078] In one example, the feature information collected from the input image includes:
[0079] S11: The first feature extraction module collects the first feature information of the input image; optionally, the first feature extraction module is any one of the SiamRPN, HRNet, and ResNet series networks, preferably ResNet-34.
[0080] S12: The second feature extraction module performs feature acquisition based on the input first feature information to obtain the second feature information; optionally, the first feature extraction module is a Transformer model.
[0081] S13: The first feature information and the second feature information are fused to obtain the fused feature, and the fused feature is used as the output of the feature extraction unit.
[0082] Specifically, such as Figure 2 As shown, the first feature extraction module is a ResNet-34 model containing multiple residual blocks, each with several convolutional layers. The network takes an image as input, which is passed through a 7x7 convolutional layer with a stride of 2 to reduce image size and extract basic features. Then, a series of residual blocks, each containing multiple convolutional layers, are passed, stacked together and normalized by batch normalization layers. Skip connections exist within each residual block, allowing gradients to propagate faster throughout the network, helping to mitigate the vanishing gradient problem. Between some residual blocks, pooling layers reduce the size of the feature maps, and finally, a global average pooling layer sums the average values of each channel of the feature maps into a fixed-size feature vector.
[0083] Specifically, such as Figure 2As shown, the second feature extraction module is a Transformer model. The Transformer model includes a Transformer module, which comprises a multi-head attention layer and a feedforward network layer. Specifically, it includes a multi-head attention layer, a first residual and normalization layer, a feedforward network layer, and another first residual and normalization layer connected sequentially. The input feature vector matrix is fed into the multi-head attention encoding layer and the first residual and normalization layer. The output of the first residual and normalization layer is connected to the second residual and normalization layer. The Transformer is used to further extract ResNet features. First, the output feature map of ResNet is transformed into a two-dimensional matrix, where each row represents a feature vector at a location. Then, this matrix is input into the Transformer model, which automatically learns the relationships between features and generates new feature representations. This process includes self-attention mechanisms and location encoding to effectively model and integrate features. Finally, the required feature representations are extracted from the output of the Transformer for use in subsequent tasks. Furthermore, the Transformer further optimizes and refines the features extracted by ResNet to adapt to subsequent Detection and Re-ID tasks.
[0084] At this point, after the ResNet-34 model and the Transformer model have extracted features (the Transformer model's feature extraction is based on the output features of the ResNet-34 model), the output features of the two models are input into the integration module, i.e., the feature fusion model, for feature fusion processing, including the following sub-steps:
[0085] S11”: Divide the output of the Transformer module into several independent feature vectors;
[0086] S12”: Several feature vectors are fused with the residual block output features respectively to obtain fused features that integrate the positioning information in the first feature information and the positional and semantic information in the second feature information.
[0087] In one example, when the input image consists of multiple downsampled images of different resolutions, after the ResNet-34 model and the Transformer model perform feature extraction respectively, the feature fusion process also includes:
[0088] S11”': Several feature vectors obtained based on the input image of the same resolution are fused with the residual block output features of the corresponding image of the same resolution to obtain several first intermediate fused features;
[0089] S12”': Perform fusion processing on adjacent first intermediate fusion features to obtain second intermediate fusion features;
[0090] S13”': Perform fully connected processing on the second intermediate fusion feature to obtain the final fusion feature.
[0091] In this example, features from ResNet-34 and Transformer features from the encoder are used as input. Positional and semantic information from the Transformer features and localization information from the ResNet-34 features are fused to achieve comprehensive and effective cross-domain feature alignment. Specifically, to ensure the two sets of features have the same dimensionality, a fully connected layer is used to map the feature vectors extracted by the Transformer to the same dimension as the ResNet-34 feature map. Feature fusion is achieved by appropriately designing the weight matrix of the fully connected layer. Then, a weighted fusion strategy assigns weights to the ResNet-34 and Transformer features respectively. These weights can be automatically learned from the training data or manually set based on the task and domain knowledge. The two sets of features are then fused according to their weights to obtain a new feature representation, which is then used for subsequent Detection and Re-ID tasks. By fusing information from different sources, a more expressive and robust feature representation is obtained, which helps improve model performance. This process needs to be optimized during training to find the optimal feature fusion strategy and weight allocation.
[0092] More specifically, the pyramid structure of the feature extraction network in this example provides a four-layer structure, and the feature fusion processing architecture is as follows: Figure 3 As shown, the resolutions include the original image resolution, downsampled to one-quarter of the original image resolution, downsampled to one-eighth of the original image resolution, and downsampled to one-sixteenth of the original image resolution. The corresponding output feature sizes are R(H x W x C), R(H / 4x W / 4x C), R(H / 8x W / 8x C), and R(H / 16x W / 16x C), respectively. Then, for each resolution, the Transformer module (Trans) contains a multi-head self-attention layer and a feedforward network layer. The expression for a single self-attention layer is as follows:
[0093]
[0094] Where Q, K, V are the feature input matrices; d k Let be the dimension of the vector. Here, Q and K have the same dimension, and Qi and K also have the same dimension. This example multi-head attention layer ensures that long sequences do not push the result to too small a gradient after passing through the softmax function, thus avoiding gradient vanishing.
[0095] The expression for multi-head attention is as follows:
[0096] headi = Attention(Q) i ,K,V)
[0097]
[0098] MultiHeadAttention(Q,K,V)=Concat(head1,head2...head n W
[0099] After obtaining MultiHeadAttention(Q,K,V), which is Trans, it is divided into two parts, PT1 and PT2. The features between different heads can be regarded as independent of each other. At this time, the feature information of each sub-head is:
[0100] PT1=Concat(head1,head2...head (n / 2) W
[0101] PT2 = Concat(head) (n / 2+1) ,head (n / 2+2) ...head (n) W
[0102] Then, the segmented Transformer features PT1 and PT2 are weighted and fused with the ResNet-34 extracted features D1 and D2 respectively:
[0103] D1 = w1 * PT1 + w2 * Res1
[0104] D2=w3*PT2+w2*Res1
[0105] PF1 = D1 + D2
[0106] Where w is the weight parameter. The same operation is performed on the downsampled data to obtain the hierarchical features PF1, PF2, PF3, and PF4 (first intermediate fusion features) under the pyramid structure. The features of the lower resolution level (corresponding to features extracted from low-resolution images) are fused with the features of the higher resolution level (corresponding to features extracted from high-resolution images). This allows the more critical features in the middle of the pyramid structure to provide more feature information than those at the top and bottom of the pyramid. Therefore, compared to traditionally adding global information, this increases more effective information while ignoring a small amount of unimportant information, thus increasing the model's generalization ability. The specific fusion expression is:
[0107] FF1 = (1-α) * low-resolution layer feature PF2 + α * high-resolution layer feature PF1
[0108] FF2 = (1-α) * low-resolution layer feature PF3 + α * high-resolution layer feature PF2
[0109] FF3 = (1-α) * low-resolution layer feature PF4 + α * high-resolution layer feature PF3
[0110] FF = FF1 + FF2 + FF3
[0111] Where α is the weight parameter, preferably 0.75; FF1, FF2, and FF3 are the second intermediate fusion features. The feature fusion method proposed in this example effectively fuses the information from ResNet-34 and Transformer to perform cross-domain feature alignment. The entire network is optimized through detection loss to achieve subsequent domain-adaptive object detection and Re-ID tasks.
[0112] In one example, the target detection results (target detection features) output by the target detection model are fused with the Re-ID features representing the identities of each target output by the Re-ID model at multiple scales. The resulting multi-scale fused features serve as input information for target tracking and matching. The target detection task differs from the Re-ID task, and therefore the target detection features and Re-ID features have different focuses. Re-ID features are typically based on human posture, appearance, clothing, vehicle color, and appearance to capture the physical features of a person or vehicle. Target detection features primarily focus on the shape, texture, edges, and background information of the target object to aid in target identification and localization. Furthermore, target detection features provide location information, while Re-ID feature extraction provides the target's identity information. Target tracking algorithms can utilize both of these aspects to achieve accurate target tracking. This stems from the effective use of multi-scale information to improve the tracking performance of objects or vehicles in images or videos. Therefore, this invention proposes a highly optimized fusion algorithm to fuse Re-ID features and target detection features at different scales.
[0113] In one example, multi-scale fusion processing involves complex technical operations such as weight allocation, multi-scale feature fusion, and feature normalization, aiming to maximize the accuracy of vehicle or pedestrian recognition and detection in images or videos, including:
[0114] S041: Normalize the Re-ID features and object detection features to ensure that the Re-ID features and object detection features have similar ranges and distributions;
[0115] S042: Dimensionality is increased for object detection features and decreased for Re-ID features. Specifically, since Re-ID features and object detection features have different scales (Re-ID features are more dimensional than object detection features), the lower-dimensional object detection features (e.g., 256-dimensional) are first increased to 512-dimensionality via linear mapping, and then further increased to 1024-dimensionality. This is achieved through matrix multiplication, using a 512x256 matrix A and a 1024x512 matrix B for linear mapping.
[0116] y = Ax
[0117] y = Bx
[0118] The higher-dimensional Re-ID features are reduced in two reverse directions (from 1024 dimensions to 512 dimensions, and then to 256 dimensions). Specifically, the dimensionality reduction is performed by calculating the weight values of adjacent dimensions. Here, we take the reduction of a 1024-dimensional feature vector to 512 dimensions as an example:
[0119]
[0120] Where y is the new feature vector after dimensionality reduction, with a dimension of 512; x is the original 1024-dimensional feature vector; w represents the weights of adjacent dimensions, typically ranging from 1 to 1024, where w i Let w represent the weight of the i-th dimension. This formula uses the weights of adjacent dimensions for dimensionality reduction, resulting in a new feature vector y, with the dimensionality reduced to 512. i Tables can be set up according to specific needs and data properties, typically representing the importance of corresponding dimensions. This dimensionality reduction method can be used to reduce feature dimensionality while preserving important information. Through the above dimensionality enhancement and reduction processes, features at three different scales under the pyramid structure were obtained.
[0121] S043: Weighted fusion of target detection features and Re-ID features from different dimensions is performed to obtain multi-scale fusion features, which serve as input information for target tracking and matching. Specifically, the multi-scale fusion expression is:
[0122] F_fused=α1*F_ReID_scale1+β1*F_Detection_scale1+α2*F_ReID_scale2+β2*
[0123] F_Detection_scale2+α3*F_ReID_scale3+β3*F_Detection_scale3
[0124] Where α1, α2, and α3 represent the weights of the 256-dimensional, 512-dimensional, and 1024-dimensional Re-ID features, respectively; β1, β2, and β3 represent the weights of the 256-dimensional, 512-dimensional, and 1024-dimensional object detection features, respectively; F_ReID_scale1 and F_Detection_scale1 are the ReID and detection features at the first scale of 256 dimensions; F_ReID_scale2 and F_Detection_scale2 are the ReID and detection features at the 512-dimensional scale; F_ReID_scale3 and F_Detection_scale3 are the ReID and detection features at the 1024-dimensional scale; and F_fused is the multi-scale fusion feature.
[0125] Existing Re-ID features are entirely based on the target detection bounding boxes determined by the previous target detection step. This means that the quality of Re-ID features depends entirely on the accuracy of target detection. In the network of this invention, they are no longer treated as a serial relationship but as a parallel relationship. The Re-ID features are no longer strongly correlated with target detection but are treated as an independent module, thereby improving the accuracy of target tracking.
[0126] Combining the above examples, we obtain a preferred example of the present invention, in which the method includes the following steps:
[0127] S10': Features are processed separately for the ResNet-34 model and the Transformer model;
[0128] S11': Several feature vectors obtained based on the input image of the same resolution are fused with the residual block output features of the corresponding image of the same resolution to obtain several first intermediate fused features;
[0129] S12': Perform fusion processing on adjacent first intermediate fusion features to obtain second intermediate fusion features;
[0130] S13': Perform fully connected processing on the second intermediate fusion feature to obtain the final fusion feature.
[0131] S2': The target detection model performs target detection based on the final fusion features to obtain the target detection result, including location information and category information;
[0132] S3': The Re-ID model generates Re-ID features representing the identities of each target based on the final fused features. It then re-identifies the targets by comparing the similarity of the Re-ID features, and further associates the targets that have already appeared.
[0133] S4': The target detection results (target detection features) output by the target detection model and the Re-ID features representing the identities of each target output by the Re-ID model are fused at multiple scales. The resulting multi-scale fused features are used as input information for target tracking and matching.
[0134] S5': Target tracking matching includes:
[0135] S51': Initialization: For each detected target, a tracker is initialized and assigned a unique ID. The state vector and covariance matrix of each tracked target are also initialized.
[0136] S52': Prediction and Update: At each time step, for each tracked target, state prediction is performed using a Kalman filter, and the covariance matrix of the Kalman filter is updated to reflect the uncertainty of the prediction; for each associated tracked target, a measurement update step using a Kalman filter is used to update its state vector and covariance matrix based on the detection results; for unassociated tracked targets, their predicted state is maintained so that they can be reassociated in the future.
[0137] S53': Data Association: Calculate the distance and similarity between each predicted tracking target and all detected targets in the current frame, and use the Hungarian algorithm to associate the tracking target with the detected targets. After the association is completed, each tracking target is associated with one or more detected targets.
[0138] S54': Create a new track: For a new detected target that is not associated with any existing track target, initialize it as a new track target and assign it a unique ID;
[0139] S55': End Tracking: For tracked targets that have not been associated for a long time, remove them or mark them as inactive. This loop repeats continuously in consecutive frames to achieve multi-target tracking and provide continuous trajectory information.
[0140] This invention transforms two-stage multi-target tracking into one-stage multi-target tracking, simplifying the algorithm architecture, reducing computational burden and processing latency, facilitating end-to-end optimization, making it suitable for online learning, and improving overall performance and efficiency. This transformation helps achieve simpler, more efficient, and more real-time multi-target tracking. Furthermore, this invention leverages the excellent modeling and sequence processing capabilities of the Transformer to solve the problem of long-range correlation and interaction modeling between targets, effectively capturing the spatiotemporal relationships of targets and handling complex situations such as target trajectory prediction, target overlap, and occlusion, thereby improving the accuracy and robustness of multi-target tracking and making it easier to track targets in complex scenes. Simultaneously, this invention proposes using a Transformer structure to enhance feature extraction in the backbone network and proposes a novel feature fusion method. This method combines the capabilities of convolutional neural networks to extract local features with the global relationship modeling capabilities of the Transformer, effectively improving feature expressiveness and allowing ResNet and Transformer to leverage their respective advantages, thus improving model performance and robustness.
[0141] This embodiment provides a storage medium that has the same inventive concept as the attention-based visual target tracking method formed by any or more of the above examples. It stores a feature extraction unit, a target detection model, a Re-ID model, and computer instructions. When the computer instructions are executed, they perform the steps of the attention-based visual target tracking method formed by any or more of the above examples.
[0142] Based on this understanding, the technical solution of this embodiment, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0143] This application also includes a terminal having the same inventive concept as any or more examples of the attention-based visual target tracking method described above, including a memory and a processor. The memory stores a feature extraction unit, a target detection model, a Re-ID model, and computer instructions executable on the processor. When the processor executes the computer instructions, it performs the steps of the attention-based visual target tracking method described above. The processor may be a single-core or multi-core central processing unit or a specific integrated circuit, or one or more integrated circuits configured to implement this invention.
[0144] In one example, the terminal, i.e., the electronic device, is manifested in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: at least one processing unit (processor) mentioned above, at least one storage unit mentioned above, and a bus connecting different system components (including storage units and processing units).
[0145] The storage unit stores program code that can be executed by the processing unit, causing the processing unit to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit can execute the aforementioned attention-based visual target tracking method.
[0146] The storage unit may include a readable medium in the form of a volatile storage unit, such as a random access memory (RAM) 3201 and / or a cache storage unit, and may further include a read-only memory (ROM).
[0147] The storage unit may also include a program / utility having a set (at least one) of program modules, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.
[0148] A bus can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus that uses any of the various bus structures.
[0149] The electronic device can also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the electronic device, and / or any device that enables the electronic device to communicate with one or more other computing devices (e.g., routers, modems, etc.). This communication can be performed via input / output (I / O) interfaces. Furthermore, the electronic device can communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter. The network adapter communicates with other modules of the electronic device via a bus. It should be understood that other hardware and / or software modules can be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0150] Through the above description, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to this exemplary embodiment can be embodied in the form of a software product, which can be stored on a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the method of the exemplary embodiment of this application.
[0151] The above detailed embodiments are a description of the present invention. It should not be considered that the specific embodiments of the present invention are limited to these descriptions. For those skilled in the art, several simple deductions and substitutions can be made without departing from the concept of the present invention, and all of these should be considered to fall within the protection scope of the present invention.
Claims
1. A visual object tracking method based on an attention mechanism, characterized in that: Includes the following steps: The image is input into the feature extraction unit to collect feature information of the input image; The feature information is input into the target detection model to perform target detection, and the target detection result including the location information is obtained. The feature information is input into the Re-ID model to generate Re-ID features representing the identity of each target. The similarity of the Re-ID features is compared to re-identify the targets, and then the targets that have already appeared are associated. The target detection results and Re-ID features are input into the Tracking module for target tracking and matching, including the following sub-steps: Assign a unique ID to each detected target in the first frame image; Predict the next location of each tracked target; Target matching is performed based on distance and similarity, associating the detected target in the current frame with the tracked target; Based on the target association results, update the status of the tracked target and the target re-identification information, and repeat the above target tracking and matching steps in consecutive frame images to achieve multi-target matching and tracking; The feature information of the input image collected includes: The first feature extraction module collects the first feature information of the input image; The second feature extraction module performs feature acquisition based on the input first feature information to obtain the second feature information; The first feature information and the second feature information are fused to obtain the fused feature, which is then used as the output of the feature extraction unit. The first feature extraction module is a ResNet-34 model, including several residual blocks; the second feature extraction module is a Transformer model, including a Transformer module, which comprises a multi-head attention layer and a feedforward network layer; the feature fusion process at this point includes: The output of the Transformer module is divided into several independent feature vectors; The segmented feature vectors are fused with the output features of the residual block to obtain fused features that integrate the localization information in the first feature information and the positional and semantic information in the second feature information.
2. The visual target tracking method based on the attention mechanism according to claim 1, characterized in that: When the input image consists of multiple downsampled images of different resolutions, the feature fusion process also includes: Several feature vectors obtained from input images of the same resolution are fused with the residual block output features of corresponding images of the same resolution to obtain several first intermediate fused features. The adjacent first intermediate fusion features are fused to obtain the second intermediate fusion feature; The second intermediate fusion feature is processed by full connection to obtain the final fusion feature.
3. The visual target tracking method based on the attention mechanism according to claim 1, characterized in that: The method further includes: performing multi-scale fusion processing on the target detection results output by the target detection model and the Re-ID features representing the identities of each target output by the Re-ID model, and using the resulting multi-scale fusion features as input information for target tracking and matching.
4. The visual target tracking method based on the attention mechanism according to claim 3, characterized in that: The multi-scale fusion process includes: Normalize the Re-ID features and target detection results; The target detection results are upgraded in dimensionality, while the Re-ID features are reduced in dimensionality. The target detection results and Re-ID features of different dimensions are weighted and fused to obtain multi-scale fusion features, which are used as input information for target tracking and matching.
5. The visual target tracking method based on the attention mechanism according to claim 1, characterized in that: In the target tracking and matching process, a Kalman filter is used to predict the next position of each tracked target.
6. The visual target tracking method based on the attention mechanism according to claim 1, characterized in that: During the target tracking and matching process, the Hungarian algorithm is used to associate the detected target in the current frame with the tracked target.
7. A terminal, comprising a memory and a processor, wherein the memory stores a feature extraction unit, a target detection model, a Re-ID model, and computer instructions capable of running on the processor, characterized in that: When the processor executes the computer instructions, it performs the steps of the visual target tracking method based on the attention mechanism according to any one of claims 1-6.