Target tracking method and device based on wavelet convolution heuristic multi-frequency graph feature fusion
By adopting wavelet convolution network and graph attention mechanism in the target tracking method, the problem of difficulty in capturing small-target feature in the prior art and insufficient correlation ability between template features and search features is solved, and higher target tracking accuracy and robustness are achieved.
Patent Information
- Application Number
- CN202411975970.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-31
- Publication Date
- 2025-05-02
AI Technical Summary
Existing target tracking methods are difficult to effectively capture the distinguishing characteristics of small targets, and the correlation between template features and search features is poor, resulting in a low success rate of target tracking.
A feature capture network based on wavelet convolution is adopted, combined with a convolutional neural network, the features of the input image are extracted, and the image features and template image features are fused through the graph attention mechanism to predict the target position.
The receptive field of the feature capture network is expanded through the wavelet convolution network, improve the ability to capture low-frequency information, and enhance the tracking accuracy and robustness of the target. The graph attention mechanism improves the correlation between template features and search features and improves the success rate of target tracking.
Smart Images
Figure CN119919444A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision, and in particular to a target tracking method and device based on wavelet convolution-inspired multi-frequency image feature fusion. Background Art
[0002] The limited target area and unclear visual structure of small objects result in less available information, which poses a great challenge to small object trackers.
[0003] The usual mitigation method is to increase the attention weight of the target area to pay more attention to the features of small targets. However, due to the small amount of information available on small targets, existing target tracking methods are difficult to capture enough distinguishing features. Even increasing the attention weight of the target area fails to significantly improve its feature quality, and only limited feature enhancement effects can be achieved. In addition, existing target tracking methods mostly use template sample images to train tracking models for search feature recognition. The correlation between template features and search features is poor. When matching template features and search features based on deep cross-correlation operations, semantic information will be lost and may fall into local optimality, affecting the target tracking success rate. Summary of the invention
[0004] The object of the present invention is to provide a target tracking method and device based on wavelet convolution-inspired multi-frequency image feature fusion to improve the success rate of target tracking in order to address all or part of the above-mentioned problems.
[0005] The technical solution adopted by the present invention is as follows:
[0006] A target tracking method based on wavelet convolution-inspired multi-frequency image feature fusion, comprising:
[0007] A feature capture network is used to extract search image features and template image features respectively, wherein the feature capture network is formed by cascading a wavelet convolutional network and a convolutional neural network;
[0008] Using a graph attention mechanism to fuse the search image features with the template image features;
[0009] Predict the target location based on the fused features.
[0010] Furthermore, the method for extracting input image features by the feature capture network includes:
[0011] Extracting a feature map of an input image using the wavelet convolutional network;
[0012] Extracting deep features in the feature map using the convolutional neural network;
[0013] The features extracted from some levels of the convolutional neural network are weighted and fused to obtain input image features.
[0014] Furthermore, the utilizing the graph attention mechanism to fuse the search image features and the template image features includes:
[0015] Calculating an attention matrix based on the similarity between the search image features and the template image features;
[0016] Aggregating the template image features and the search image features based on the attention matrix;
[0017] The aggregated features are fused to the search image features.
[0018] Further, calculating the attention matrix based on the similarity between the search image features and the template image features includes:
[0019] respectively calculating the similarity between each region of the same size between the search image feature and the template image feature;
[0020] Each of the similarities is normalized to obtain a corresponding attention score to form an attention matrix.
[0021] Further, respectively calculating the similarity between each region of the same size between the search image feature and the template image feature, including:
[0022] Dividing the search image features and the template image features according to the sizes to form a search sub-image set and a template sub-image set respectively;
[0023] For each subgraph in the search subgraph set and the template subgraph set, the similarity between each two is calculated respectively.
[0024] Furthermore, aggregating the template image features and the search image features based on the attention matrix includes:
[0025] For each subgraph in the search subgraph set, feature aggregation is performed on each subgraph in the target subgraph set according to the attention matrix.
[0026] Furthermore, the size is 1×1×C, where C is the number of feature channels.
[0027] Furthermore, predicting the target position based on the fusion feature includes:
[0028] Based on the fusion features, a target positioning model is used to predict the target center and size; the target positioning model includes a classification branch for predicting target category information and a regression branch for predicting target position information.
[0029] Furthermore, during the training phase, the classification branch uses a binary cross entropy loss function to calculate the classification loss, and the regression branch uses an intersection-over-union loss function to calculate the regression loss.
[0030] To solve the above problems, the present invention also provides a target tracking device, including a processor and a storage medium, wherein the storage medium stores a computer program, and when the processor runs the computer program, it executes the above-mentioned target tracking method based on wavelet convolution-inspired multi-frequency image feature fusion.
[0031] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:
[0032] The target tracking method based on wavelet convolution-inspired multi-frequency image feature fusion proposed in this application can effectively extract small target features by using a feature capture network constructed by combining a wavelet convolution network and a convolutional neural network. The input image is decomposed into multiple discrete wavelet sub-bands without losing information through a wavelet convolution network, and a cascaded wavelet decomposition can be performed, thereby expanding the receptive field of the feature capture network while only introducing a small number of trainable parameters, so that the feature capture network can more effectively capture low-frequency information in the input image, improve the responsiveness to the shape, promote the tracking accuracy of the target (especially small targets), and help the network learn and generalize more efficiently. Furthermore, the search image features and template image features are feature-fused using a graph attention mechanism, which improves the correlation between template features and search features, thereby improving the tracking model's ability to understand target features, and effectively improving the accuracy and robustness of target (including small targets) tracking. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] The present invention will now be described by way of example with reference to the accompanying drawings, in which:
[0034] Figure 1 It is a flow chart of a target tracking method based on wavelet convolution-inspired multi-frequency image feature fusion provided in an embodiment of the present application.
[0035] Figure 2 It is a data flow diagram of a target tracking method based on wavelet convolution-inspired multi-frequency image feature fusion provided in an embodiment of the present application.
[0036] Figure 3 It is a network structure diagram of the classification branch and the regression branch in the embodiment of the present application. DETAILED DESCRIPTION
[0037] All features disclosed in this specification, or steps in all methods or processes disclosed, except mutually exclusive features and / or steps, can be combined in any manner.
[0038] Any feature disclosed in this specification (including any additional claims and abstract), unless otherwise stated, may be replaced by other equivalent or alternative features having similar purposes. That is, unless otherwise stated, each feature is only an example of a series of equivalent or similar features.
[0039] In order to address the problems that it is difficult to capture sufficient distinguishing features for small targets during target tracking, and that the correlation between template features and search features is poor, resulting in poor target tracking (especially tracking of small targets), the embodiments of the present application provide a target tracking method and device based on wavelet convolution-inspired multi-frequency image feature fusion, aiming to improve target tracking, especially the accuracy and robustness of small target tracking.
[0040] In some embodiments, Figure 1 As shown in FIG, the target tracking method based on wavelet convolution-inspired multi-frequency image feature fusion includes the following steps:
[0041] S1. Using a feature capture network to extract search image features and template image features respectively, the feature capture network is formed by cascading a wavelet convolutional network and a convolutional neural network.
[0042] As an optional implementation, the method for extracting input image features by a feature capture network includes:
[0043] S1.1. Use wavelet convolutional network to extract feature maps of input images.
[0044] S1.2. Use convolutional neural network to extract deep features in feature maps.
[0045] S1.3. Perform weighted fusion on the features extracted from some levels of the convolutional neural network to obtain the input image features.
[0046] The embodiment of the present application uses the Resnet50 network as an example of a convolutional neural network. In some feasible implementations, for an input image, the input image is first processed using a wavelet convolutional network to obtain a feature map containing rich detail information, and then the Resnet50 network is used to extract deep features of the feature map. This process is characterized as follows in some specific embodiments:
[0047]
[0048] Where I is the input image, F X is the input image feature extracted by the feature capture network, F w is the feature map extracted by the wavelet convolutional network, F Resnet50is the feature map extracted by the Resnet50 network, F3, F4, and F5 are the features output by the third, fourth, and fifth stages of the Resnet50 network, respectively, a1, a2, and a3 are trainable parameters, WTConv() represents the operation of the wavelet convolution network, and Activate() represents the activation function. The feature capture network adaptively fuses the features of different depths extracted from Resnet50, thereby providing more reliable feature information for the tracking model, enabling the tracking model to better understand the structure and semantic information in the image, thereby improving the accuracy and robustness of target tracking. In a similar embodiment, F X The features output by other stages of the Resnet50 network can be obtained by weighting the corresponding trainable parameters, and are not limited to the features output by the third stage, the fourth stage, and the fifth stage in the above embodiment.
[0049] In some feasible implementations, wavelet convolution realizes multi-scale analysis of the image by decomposing the image into four (or other numbers) discrete wavelet sub-bands. This decomposition method not only retains all the information of the image, but also allows cascaded wavelet decomposition, thereby significantly expanding the receptive field of the network.
[0050] S2. Utilize the graph attention mechanism to fuse the search image features and the template image features.
[0051] As an optional implementation, step S3 includes the following sub-steps:
[0052] S3.1. Calculate the attention matrix based on the similarity between the search image features and the template image features.
[0053] The search image features and the template image features are both three-dimensional feature matrices. In some feasible implementations, the attention matrix is formed by respectively calculating the similarity between each region of the same size between the search image features and the template image features, and then normalizing each similarity to obtain the corresponding attention score.
[0054] The determination of the area in the search image features and the template image features can be performed by sequentially marking with a sliding window of a predetermined size, so as to perform similarity calculation on the area marked by the current sliding window in the search image features and the template image features.
[0055] As an optional implementation, the search image features and the template image features can be divided according to the predetermined size to form a search sub-graph set and a template sub-graph set respectively; then, for each sub-graph in the search sub-graph set and the template sub-graph set (i.e., the features of each divided area), the similarity between them is calculated respectively.
[0056] For example, with F Z and FT Represent the template image features and search image features respectively, and use the size of 1×1×C to map F Z and F T Divide F Z and F T They are divided into multiple sub-graphs, where C represents the number of feature channels. In this way, we get the template sub-graph set V Z and search subgraph set V T For the set V Z and V T For each subgraph in , we calculate the similarity between each pair to get the corresponding correlation score, and then normalize all similarities to get the attention score corresponding to each similarity. Specifically, for the search subgraph set V T Each subgraph in the template subgraph set V Z Each subgraph in has a similarity, so the set V T For each subgraph in , there is an attention submatrix, and the set of attention submatrices of each subgraph constitutes the search image feature F T and template image features F Z The attention matrix between .
[0057] S3.2. Aggregate template image features and search image features based on the attention matrix.
[0058] As an optional implementation, for each region of a predetermined size in the search image feature, feature aggregation is performed on each region of a predetermined size in the template feature image according to the attention matrix.
[0059] In some feasible implementations, the template image feature F Z and search image features F T The template subgraph set V obtained by partitioning Z and search subgraph set V T For example, for the set V T For each subgraph in the set V, we use the attention matrix Z Each subgraph in the set V is subjected to feature aggregation. T For each subgraph in the set V, according to its attention submatrix, Z Each subgraph in is weightedly fused to obtain a new set V ′ T .
[0060] S3.3. Fusion of aggregated features into search image features.
[0061] In step S3.2, the template image features and the search image features are aggregated, and this step fuses the aggregated features into the search image features.
[0062] As an optional implementation, the set V obtained in the previous text ′ T For example, in this step, the set V T and set V ′ T The sub-images at the same position in the image are subjected to feature fusion to obtain a feature response map representing the fusion feature. The fusion of the search image feature and the template image feature can be represented by the following formula:
[0063] F S =GraphAttension(F Z ,F T ),
[0064] In the formula, F S Represents the feature response graph, GraphAttension() represents the feature fusion operation based on the graph attention mechanism, abbreviated as GAM.
[0065] S3. Predict the target position based on the fused features.
[0066] The prediction of the target position includes the prediction of the target center and size. As an optional implementation, step S3 includes:
[0067] Based on the fused features, the target center and size are predicted using a target localization model, which includes a classification branch for predicting target category information and a regression branch for predicting target position information.
[0068] like Figure 2 As shown, the characteristic response diagram F is obtained S After that, the characteristic response graph F S Input to the classification branch and regression branch of the target localization model.
[0069] In some feasible implementations, in the classification branch, the feature response map F S The image is parsed and reduced in dimension, and the output is a category information graph F of size 2×H×W. cls , the category information map contains the category information and confidence of the predicted target.
[0070] In the regression branch, the characteristic response map F S The processing similar to the classification branch is performed, and the final output is a position information map F of size 4×H×W reg , the location information map contains all possible location information (coordinates) of the predicted target.
[0071] The classification branch and regression branch are based on the characteristic response map F S Get category information graph F cls and location information diagram F reg The expression is:
[0072]
[0073] Among them, FC cls () represents the fully connected network of the classification branch, FC reg () represents a fully connected network of the regression branch. In some specific embodiments, the network structure of the classification branch and the regression branch is as follows: Figure 3 shown.
[0074] In the embodiments of the present application, the trainable parameters involved in the feature capture network and the target positioning model are obtained by training using a sample data set.
[0075] As an optional implementation, the present application decouples the classification task and the regression task, and designs corresponding loss functions for the requirements of the classification branch and the regression branch respectively, while the overall training task is to minimize the overall loss as the optimization goal. The overall loss Loss is expressed as:
[0076] Loss=Weight cls ×loss cls +Weight loc ×loss loc ,
[0077] In the formula, loss cls is the classification loss of the classification branch, loss loc is the regression loss of the regression branch, Weight cls is the weight of the classification loss, which can be set to 1.2. loc is the weight of the regression loss and can be set to 3.
[0078] As an optional implementation, the classification loss of the classification branch is cls The binary cross entropy loss function is used for calculation, expressed as:
[0079]
[0080] Where N is the number of predicted samples, label i is the label of the predicted target, pred i is the probability of predicting the target.
[0081] As an optional implementation of the regression branch, its regression loss loss locThe intersection-over-union loss function is used for calculation, which is expressed as:
[0082]
[0083] Where A represents the area of the predicted box, B represents the area of the real box, A∩B represents the intersection of the predicted box and the real box, and A∪B represents the union of the predicted box and the real box.
[0084] It can be seen from the embodiments of the present application that the target tracking method based on wavelet convolution-inspired multi-frequency image feature fusion proposed in the present application is end-to-end from the feature capture network to the target positioning model. It can be trained directly on the established network without the need to use a complex training method such as knowledge distillation that requires building both a teacher network and a student network, and the training efficiency is higher.
[0085] In some embodiments, the target tracking device provided by the present application includes a processor and a storage medium, wherein the storage medium stores a computer program, and when the processor runs the computer program, it executes the target tracking method based on wavelet convolution-inspired multi-frequency image feature fusion of the above embodiment.
[0086] The present invention is not limited to the above-mentioned specific embodiments, but extends to any new features or any new combination disclosed in this specification, as well as any new method or process steps or any new combination disclosed.
Claims
1. A target tracking method based on wavelet convolution-inspired multi-frequency image feature fusion, characterized in that: include: A feature capture network is used to extract search image features and template image features respectively, wherein the feature capture network is formed by cascading a wavelet convolutional network and a convolutional neural network; Using a graph attention mechanism to fuse the search image features with the template image features; Predict the target location based on the fused features.
2. The target tracking method based on wavelet convolution-inspired multi-frequency image feature fusion as claimed in claim 1, characterized in that: The method for extracting input image features through the feature capture network includes: Extracting a feature map of an input image using the wavelet convolutional network; Extracting deep features in the feature map using the convolutional neural network; The features extracted from some levels of the convolutional neural network are weighted and fused to obtain input image features.
3. The target tracking method based on wavelet convolution-inspired multi-frequency image feature fusion as claimed in claim 1, characterized in that: The utilizing the graph attention mechanism to perform feature fusion of the search image features and the template image features includes: Calculating an attention matrix based on the similarity between the search image features and the template image features; Aggregating the template image features and the search image features based on the attention matrix; The aggregated features are fused to the search image features.
4. The target tracking method based on wavelet convolution-inspired multi-frequency image feature fusion as claimed in claim 3, characterized in that: Calculating an attention matrix based on the similarity between the search image features and the template image features includes: respectively calculating the similarity between each region of the same size between the search image feature and the template image feature; Each of the similarities is normalized to obtain a corresponding attention score to form an attention matrix.
5. The target tracking method based on wavelet convolution-inspired multi-frequency image feature fusion as claimed in claim 4, characterized in that: Respectively calculating the similarity between each region of the same size between the search image feature and the template image feature, including: Dividing the search image features and the template image features according to the sizes to form a search sub-image set and a template sub-image set respectively; For each subgraph in the search subgraph set and the template subgraph set, the similarity between each two is calculated respectively.
6. The target tracking method based on wavelet convolution-inspired multi-frequency image feature fusion as claimed in claim 5, characterized in that: Aggregating the template image features and the search image features based on the attention matrix includes: For each subgraph in the search subgraph set, feature aggregation is performed on each subgraph in the target subgraph set according to the attention matrix.
7. The target tracking method based on wavelet convolution-inspired multi-frequency image feature fusion as described in any one of claims 4 to 6, characterized in that: The size is 1×1×C, where C is the number of feature channels.
8. The target tracking method based on wavelet convolution-inspired multi-frequency image feature fusion as claimed in claim 1, characterized in that: The predicting of the target position based on the fusion feature includes: Based on the fusion features, a target positioning model is used to predict the target center and size; the target positioning model includes a classification branch for predicting target category information and a regression branch for predicting target position information.
9. The target tracking method based on wavelet convolution-inspired multi-frequency image feature fusion as claimed in claim 8, characterized in that: During the training phase, the classification branch uses a binary cross entropy loss function to calculate the classification loss, and the regression branch uses an intersection-over-union loss function to calculate the regression loss.
10. A target tracking device, comprising a processor and a storage medium, wherein the storage medium stores a computer program, characterized in that: When the processor runs the computer program, it executes the target tracking method based on wavelet convolution-inspired multi-frequency image feature fusion as described in any one of claims 1-9.