Multimodal image fusion object tracking method
By employing a multimodal feature fusion method combining Siamese convolutional neural networks and Transformer models, the accuracy and robustness issues of multimodal image fusion target tracking in complex scenarios are addressed, achieving more efficient feature fusion and target localization.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHANGCHUN INST OF OPTICS FINE MECHANICS & PHYSICS CHINESE ACAD OF SCI
- Filing Date
- 2023-07-11
- Publication Date
- 2026-07-21
AI Technical Summary
Existing multimodal image fusion target tracking methods lack sufficient tracking accuracy, robustness, and stability in complex scenes, especially under the influence of factors such as occlusion, illumination changes, interference from similar targets, and scale changes, resulting in performance degradation.
A Siamese convolutional neural network is used for multimodal feature extraction. The sharing ratio is determined by calculating the similarity between block layers. Asymmetric convolutional layers and a multimodal feature fusion network based on the Transformer model are combined to adaptively evaluate feature weights and reliability. Finally, training is completed by iteratively optimizing the loss function.
It improves tracking accuracy, robustness, and stability in complex scenarios, enhances network performance and computational efficiency, and strengthens the reliability and robustness of features.
Smart Images
Figure CN116740480B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of target tracking technology, and specifically provides a multimodal image fusion target tracking method. Background Technology
[0002] In traditional multimodal image fusion target tracking methods, mid-range feature-level fusion methods often employ sparse representation-based and graph-based methods for feature extraction and fusion of multimodal image data. While these traditional methods can achieve good tracking results in some simple scenes, their tracking performance tends to decline significantly when dealing with interference factors such as occlusion, lighting variations, similar target interference, and scale changes in complex scenes.
[0003] In recent years, with the development of deep learning, feature-level fusion methods based on deep features have been successfully introduced into visual target tracking. This method mainly uses deep neural networks to extract features from multimodal image data. The extracted deep features are more effective and robust than manually designed shallow features, making it easier to improve the overall performance of visual target tracking in complex scenes. Zhang et al. used convolutional neural networks to extract features from visible light and infrared images in a parallel structure, then directly concatenated the extracted features and fed them into a specific layer for classification and localization. Li and Wu et al. used dual-stream convolutional neural networks to extract features from visible light and infrared images respectively, and proposed a fusion network for adaptive feature fusion. Although the above two methods can utilize multimodal deep features from different perspectives to fuse and improve tracking performance, they do not analyze and address the fusion weights and reliability of each modality's data during the fusion process. Zhang and Ye et al. introduced Siamese convolutional neural networks to solve the problem of multimodal image data fusion based on existing methods. By using the same basic network in each branch to achieve independent feature extraction, and performing cross-correlation calculations after fusion, they achieved target discrimination and localization. Because Siamese convolutional neural network (SiN) structures play a crucial role in few-shot learning and can be trained offline, this type of method offers high real-time performance while maintaining accuracy and robustness. However, this method still processes multimodal data independently, failing to fully consider the common characteristics among the data, resulting in high feature redundancy and increased computational burden. Furthermore, the fusion strategy lacks adaptability when calculating weights and reliability among different modalities, leading to poor robustness of the fused features. To address this, CLLi and C.Li et al. employed a partially shared convolutional layer model in Siamese networks to simultaneously extract features from multimodal image data, effectively capturing common features. However, this shared model was not integrated with the network's layer depth, resulting in poor network performance and computational efficiency, and the adaptability of the fusion strategy was not further improved.
[0004] In summary, although the multimodal image fusion target tracking method using mid-level feature fusion has greatly improved tracking performance after the introduction of Siamese convolutional neural networks, the tracking accuracy, robustness and stability of this method in complex scenarios still need to be improved. Summary of the Invention
[0005] To address the aforementioned problems, this invention provides a multimodal image fusion target tracking method, which effectively improves tracking accuracy, robustness, and stability in complex scenarios.
[0006] The multimodal image fusion target tracking method provided by this invention includes the following steps:
[0007] S1. Use a twin convolutional neural network to extract features from multimodal image data to obtain multimodal features;
[0008] S2. Calculate the similarity between different modal features of the Block layers in the two branches of the Siamese convolutional neural network, and determine the sharing ratio between the Block layers;
[0009] S3. Input the multimodal features into the multimodal feature fusion network, perform feature fusion on the multimodal features, and obtain multimodal fused features;
[0010] S4. Input the multimodal fusion features into the asymmetric convolutional layer for calculation to obtain the final features;
[0011] S5. Classify and regress the final features, and use them as data input for the loss function of the Siamese convolutional neural network. Optimize the loss function iteratively to complete the training of the Siamese convolutional neural network.
[0012] S6. Input the multimodal image data of the target to be tracked into the trained Siamese convolutional neural network to obtain the localization information of the target to be tracked.
[0013] Preferably, the Siamese convolutional neural network includes two structurally identical branch networks: a target template branch network and a search region branch network. Both branch networks employ an improved ResNet-18 network, with the following topology:
[0014] The first convolutional layer, Conv1, has a kernel size of 7×7, a kernel count of 64, a stride of 2, and padding of 1.
[0015] Maxpool is a max pooling layer with a pooling window size of 3×3, a stride of 2, and padding of 1.
[0016] Block1, the convolution kernel is Step size is 2, padding size is 1;
[0017] Block2, the convolution kernel is Step size is 2, padding size is 1;
[0018] Block3, the convolution kernel is Step size is 2, padding size is 1;
[0019] Block4, the convolution kernel is The step size is 2, and the padding size is 1.
[0020] Preferably, the formula for calculating the similarity between different modal features of Block2, Block3, and Block4 levels in the branch network is as follows:
[0021]
[0022] in, For the improved ResNet-18 network, b is a two-dimensional bias matrix. and This represents the feature tensor between different modalities at the same level depth in the same branch network;
[0023] The cross-correlation calculation results between levels are used as the sharing ratio in Block2, Block3, and Block4.
[0024] Preferably, the multimodal feature fusion network includes a context enhancement module and an adaptive feature fusion module built based on the encoder-decoder structure in the Transformer model. The adaptive feature fusion module uses the same decoder as the context enhancement module.
[0025] Preferably, the encoder includes a multi-head self-attention mechanism and instance normalization computation. The multi-head self-attention mechanism is defined as follows:
[0026] MultiHead(Q,K,V)=Concat(H1,...,H n W O (2);
[0027]
[0028]
[0029] Where Q, K, and V are the input vector matrices, and W is the input vector matrix. O W n Q W n K W n V Let d be a parameter matrix. k Let K be the dimension of matrix K;
[0030] The output and input of the multi-head self-attention mechanism are calculated using instance-normalized computation, as shown in the following formula:
[0031] Y encoder =Ins.Norm(X+Multihead(Q,K,V))(5);
[0032] Where X is the encoder input, Ins.Norm represents instance normalization computation, and Y... encoderThis is the feature tensor that is the final output of the encoder.
[0033] Preferably, the decoder processes the input feature sequence X. Q and X KV Spatial location encoding is performed, and the input feature sequences are fused using a multi-head cross-attention mechanism in the form of residuals. Instance normalization is then performed on the fusion result, and a fully connected feedforward neural network is used to enhance the fitting ability of this result. Finally, the multimodal fused features of the decoder's final output are calculated using the following formula:
[0034]
[0035]
[0036] Among them, P Q and P KV They correspond to X respectively Q and X KV Spatial location encoding, MultiHead-Cross represents a multi-head cross-attention mechanism. Y represents the feature tensor in the decoder after multi-head cross-attention mechanism and instance normalization processing. decoder Let be the feature tensor of the final output of the decoder, and FFN represent a fully connected feedforward neural network.
[0037] Preferably, the multimodal fusion features of the two branch networks after processing by the multimodal feature fusion network are combined using asymmetric convolutional layers and broadcast addition in feature tensor calculation. The feature mapping calculation process of the asymmetric convolutional layer for the sub-window i of the multimodal fusion features in the search region branch network is as follows:
[0038]
[0039] in, For multimodal fusion features in the target template branch network, To search for multimodal fusion features in the branch network of the search region The i-th child window, ω z ω x respectively applied to and The asymmetric convolutional layer, where n is the number of sub-windows representing the multimodal fusion features in the search region branch network, + b This refers to broadcast addition in the computation of the characteristic tensor.
[0040] Preferably, in step S6, the multimodal image data of the target to be tracked is input into a trained Siamese convolutional neural network for feature extraction. The trained multimodal feature fusion network is used to fuse multimodal features between different modalities. The trained asymmetric convolutional layer is used to combine the multimodal fusion features of different branch networks. The combined final features are then classified and regressed to obtain the location information of the target to be tracked.
[0041] Compared with the prior art, the present invention can achieve the following beneficial effects:
[0042] This invention fully combines the network's layer depth with the sharing mode of convolutional layers, and gradually changes the sharing ratio of convolutional layers as the network layers deepen, effectively improving network performance and computational efficiency. It also designs a multimodal feature fusion network based on an attention mechanism, which can adaptively and accurately evaluate the weights and reliability of multimodal features, thereby making the fused features more reliable and robust, and achieving higher accuracy in target tracking. Attached Figure Description
[0043] Figure 1 This is a flowchart of a multimodal image fusion target tracking method provided according to an embodiment of the present invention;
[0044] Figure 2 This is a schematic diagram of the tracking process of the multimodal image fusion target tracking method provided in an embodiment of the present invention;
[0045] Figure 3 This is a schematic diagram of cross-correlation calculation between multimodal features at the Block2 to Block4 level depths provided by an embodiment of the present invention;
[0046] Figure 4 This is a schematic diagram of the encoder-decoder principle in a multimodal fusion network provided according to an embodiment of the present invention;
[0047] Figure 5 This is a schematic diagram of the asymmetric convolutional layer and the multimodal fusion feature calculation of each branch according to an embodiment of the present invention. Detailed Implementation
[0048] In the following description, embodiments of the invention will be described with reference to the accompanying drawings. In the description below, the same modules are denoted by the same reference numerals. Where the same reference numerals are used, their names and functions are also the same. Therefore, their detailed description will not be repeated.
[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not constitute a limitation thereof.
[0050] Figure 1 The flowchart of a multimodal image fusion target tracking method provided according to an embodiment of the present invention is shown.
[0051] Figure 2 The tracking process of the multimodal image fusion target tracking method provided in the present invention is illustrated.
[0052] like Figure 1 and Figure 2 As shown, the multimodal image fusion target tracking method provided in this embodiment of the invention mainly includes two stages: network training and target tracking. Specifically:
[0053] Network training phase:
[0054] S1. Feature extraction is performed on multimodal image data (template branch image and search region branch network image) using two branches of a Siamese convolutional neural network to obtain multimodal features. The Siamese convolutional neural network consists of two structurally identical branch networks: a target template branch network and a search region branch network. These two branch networks share the same weights and network structure, allowing them to process two different inputs in parallel and output feature representations for each input separately. Both branch networks use a self-improved ResNet-18 network to extract features from images of different modalities. The improved network topology is as follows:
[0055] The first convolutional layer, Conv1, has a kernel size of 7×7, a kernel count of 64, a stride of 2, and padding of 1.
[0056] Maxpool is a max pooling layer with a pooling window size of 3×3, a stride of 2, and padding of 1.
[0057] Block1, the convolution kernel is Step size is 2, padding size is 1;
[0058] Block2, the convolution kernel is Step size is 2, padding size is 1;
[0059] Block3, the convolution kernel is Step size is 2, padding size is 1;
[0060] Block4, the convolution kernel is The step size is 2, and the padding size is 1.
[0061] The specific network structure is shown in Table 1:
[0062] Table 1 Network Structure
[0063]
[0064] The improved ResNet-18 network has been pre-trained, mainly by adjusting the stride in Block 3 and Block 4 to 1, so that the convolutional features output by the three convolutional blocks Block 2, Block 3, and Block 4 have a uniform size.
[0065] Figure 3 The process of calculating the cross-correlation between multimodal features at the Block 2 to Block 4 level depths provided by an embodiment of the present invention is illustrated.
[0066] like Figure 3 As shown, after the multimodal features of S2 are extracted, the similarity between different modal features at Block 2 to Block 4 levels under the target template branch network and the search region branch network is calculated using formula (1). The similarity calculation formula is as follows:
[0067]
[0068] in, For the improved ResNet-18 network, b is a two-dimensional bias matrix. and This represents the feature tensor between different modalities at the same depth in the same branch network.
[0069] The cross-correlation calculation results between layers are used as the sharing ratio in Block 2, Block 3, and Block 4, and are recorded and stored accordingly. After training, the statistical analysis of the recorded similarity data is performed to determine the final sharing ratio between Block 2 to Block 4 layers. In this embodiment, the cross-correlation calculation results between different Block layers are 0.25, 0.5, and 0.75, respectively. Therefore, the sharing ratio of convolutional layers between different Block layers is 25%, 50%, and 75%. By controlling the sharing ratio of convolutional layers between different Block layers, the performance and computational efficiency of the multimodal feature extraction network can be effectively improved.
[0070] S3. To improve the reliability and robustness of multimodal fusion features, a multimodal feature fusion network based on an attention mechanism is proposed. This network can adaptively and accurately evaluate the weights and reliability of multimodal features. The multimodal feature fusion network includes a context enhancement module and an adaptive feature fusion module. The context enhancement module is built by optimizing the encoder-decoder structure in the Transformer model. The adaptive feature fusion module uses the same decoder as the context enhancement module.
[0071] Figure 4 The principle of the encoder-decoder in the multimodal fusion network provided according to an embodiment of the present invention is illustrated.
[0072] like Figure 4 As shown, multimodal features are input into a multimodal feature fusion network to perform feature fusion. First, the features enter the context enhancement module. The features of different modal data under the template branch and search region branch networks will pass through the encoder respectively.
[0073] The encoder includes a multi-head self-attention mechanism and instance normalization computation. The multi-head self-attention mechanism allows the network model to consider different attention distributions and to pay attention to data information from multiple perspectives. Its definition is as follows:
[0074] MultiHead(Q,K,V)=Concat(H1,...,H n W O (2);
[0075]
[0076]
[0077] Where Q, K, and V are the input vector matrices, and W is the input vector matrix. O W n Q W n K W n V Let d be a parameter matrix. k Let H be the dimension of matrix K. n This indicates the number of attention heads. An attention head is a parallel operating unit in a self-attention mechanism, where each attention head learns and focuses on different features and contextual information.
[0078] After processing by the multi-head self-attention mechanism, the output and input of the multi-head self-attention mechanism are normalized for instance, and the calculation result is output as data. The calculation formula for instance normalization is as follows:
[0079] Y encoder =Ins.Norm(X+Multihead(Q,K,V))(5);
[0080] Where X is the encoder input, Ins.Norm represents instance normalization computation, and Y... encoder This is the feature tensor that is the final output of the encoder.
[0081] The output data of encoders from different modalities within the same branch are interleaved and distributed to the corresponding decoders. First, the decoder processes the input feature sequence X. Q and X KV Spatial location encoding is performed to enable the decoder to effectively distinguish the location information in the input feature sequence. The decoder uses a multi-head cross-attention mechanism to fuse the input feature sequence in the form of residuals, and performs instance normalization calculation on the fusion result. A fully connected feedforward neural network is used on the calculation result to enhance the fitting ability of the network model. Finally, the multimodal fusion features of the decoder's final output are calculated through equations (6) and (7):
[0082]
[0083]
[0084] Among them, P Q and P KV They correspond to X respectively Q and X KV Spatial location encoding, MultiHead-Cross represents a multi-head cross-attention mechanism. Y represents the feature tensor in the decoder after multi-head cross-attention mechanism and instance normalization processing. decoder Let be the feature tensor of the final output of the decoder, and FFN represent a fully connected feedforward neural network.
[0085] Figure 5 The process of calculating the multimodal fusion features of the asymmetric convolutional layer and each branch according to an embodiment of the present invention is illustrated.
[0086] like Figure 5 As shown in Figure S4, the multimodal fusion features output from the two branches of the multimodal feature fusion network after processing are combined using asymmetric convolutional layers and broadcast addition in feature tensor calculation, thereby replacing the traditional cross-correlation calculation. This allows the feature combination calculation process to benefit from end-to-end learning. When the asymmetric convolutional layer calculates the multimodal fusion features in each branch, the asymmetric convolutional kernel uses the same length and width dimensions as the fusion features in the target template branch network. The feature mapping calculation process of the asymmetric convolutional layer for the sub-window i of the multimodal fusion features in the search region branch network is as follows:
[0087]
[0088] in, For multimodal fusion features in the target template branch network, To search for multimodal fusion features in the branch network of the search region The i-th child window, ω z ω xrespectively applied to and The asymmetric convolutional layer, where n is the number of sub-windows representing the multimodal fusion features in the search region branch network, + b This refers to broadcast addition in the computation of the characteristic tensor.
[0089] S5. After obtaining the final features by formula (8), classify and regress the final features and use them as the data input of the loss function of the Siamese convolutional neural network. Also, use the true values of the labels in the training dataset as the data input of the loss function. Optimize the loss function and adjust the network parameters through iterative methods to complete the training of the Siamese convolutional neural network.
[0090] Target tracking phase:
[0091] S6. The infrared and visible light images of the target to be tracked are collected in real time and input into the trained twin convolutional neural network for feature extraction. The sharing ratio between layers has been fixed after the training process. The trained multimodal feature fusion network is used to fuse multimodal features between different modes. Finally, the trained asymmetric convolutional layer is used to combine the multimodal fusion features of different branches of the network. The combined final features are classified and regressed to obtain the localization information of the target to be tracked.
[0092] Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.
[0093] The specific embodiments of the present invention described above do not constitute a limitation on the scope of protection of the present invention. Any other corresponding changes and modifications made in accordance with the technical concept of the present invention should be included within the scope of protection of the claims of the present invention.
Claims
1. A multimodal image fusion target tracking method, characterized in that, Includes the following steps: S1. Feature extraction is performed on multimodal image data using a Siamese convolutional neural network to obtain multimodal features. The Siamese convolutional neural network includes two structurally identical branch networks: a target template branch network and a search region branch network. Both branch networks use an improved ResNet-18 network, and their topology is as follows: The first convolutional layer, Conv1, has a kernel size of 7×7, a kernel count of 64, a stride of 2, and padding of 1. Maxpool is a max pooling layer with a pooling window size of 3×3, a stride of 2, and padding of 1. Block1, convolution kernel is The step size is 2, and the fill size is 1. Block2, the convolution kernel is The step size is 2, and the fill size is 1. Block3, the convolution kernel is The step size is 2, and the fill size is 1. Block4, the convolution kernel is The step size is 2, and the fill size is 1. S2. Calculate the similarity of Block levels in the two branches of the Siamese convolutional neural network between different modal features, and determine the sharing ratio between Block levels; the formulas for calculating the similarity of Block2, Block3, and Block4 levels in the branch network between different modal features are as follows: （1）； in, For the improved ResNet-18 network, b It is a two-dimensional bias matrix. and This represents the feature tensor between different modalities at the same level depth in the same branch network; The cross-correlation calculation results between levels are used as the sharing ratio in Block2, Block3, and Block4; S3. Input the multimodal features into the multimodal feature fusion network, perform feature fusion on the multimodal features, and obtain multimodal fused features; S4. Input the multimodal fusion features into the asymmetric convolutional layer for calculation to obtain the final features; use the asymmetric convolutional layer and the broadcast addition in the feature tensor calculation to combine the multimodal fusion features of the two branch networks after processing by the multimodal feature fusion network. The asymmetric convolutional layer performs sub-windowing of the multimodal fusion features in the branch network of the search region. i The feature mapping calculation process is as follows: ； in, For multimodal fusion features in the target template branch network, To search for multimodal fusion features in the branch network of the search region The i Each sub-window respectively applied to and Asymmetric convolutional layers n The number of sub-windows representing the multimodal fusion features in the search region branch network, + b This refers to broadcast addition in the computation of the characteristic tensor. S5. Classify and regress the final features, and use them as data input for the loss function of the Siamese convolutional neural network. Optimize the loss function iteratively to complete the training of the Siamese convolutional neural network. S6. Input the multimodal image data of the target to be tracked into the trained Siamese convolutional neural network to obtain the localization information of the target to be tracked.
2. The multimodal image fusion target tracking method as described in claim 1, characterized in that, The multimodal feature fusion network includes a context enhancement module and an adaptive feature fusion module, which are built based on the encoder-decoder structure in the Transformer model. The adaptive feature fusion module uses the same decoder as the context enhancement module.
3. The multimodal image fusion target tracking method as described in claim 2, characterized in that, The encoder includes a multi-head self-attention mechanism and instance normalization computation. The multi-head self-attention mechanism is defined as follows: （2）； （3）； （4）； in, Q , K , V These are the input vector matrices, W O , W n Q , W n K , W n V For the parameter matrix, d k For matrix K The dimension; The output and input of the multi-head self-attention mechanism are calculated using instance-normalized computation, as shown in the following formula: （5）； in, X For the encoder input, Ins.Norm Indicates instance normalization calculation, Y encoder This is the feature tensor that is the final output of the encoder.
4. The multimodal image fusion target tracking method as described in claim 3, characterized in that, The decoder processes the input feature sequence X Q and X KV Spatial location encoding is performed, and the input feature sequences are fused using a multi-head cross-attention mechanism in the form of residuals. Instance normalization is then performed on the fusion result, and a fully connected feedforward neural network is used to enhance the fitting ability of this result. Finally, the multimodal fused features of the decoder's final output are calculated using the following formula: （6）； （7）； in, P Q and P KV They correspond to as X Q and X KV Spatial location encoding, MultiHead-Cross This indicates a multi-head cross-attention mechanism. This represents the feature tensor in the decoder after multi-head cross-attention mechanism and instance normalization processing. The feature tensor is the final output of the decoder. FFN This represents a fully connected feedforward neural network.
5. The multimodal image fusion target tracking method as described in claim 1, characterized in that, In S6, the multimodal image data of the target to be tracked is input into the trained Siamese convolutional neural network for feature extraction. The trained multimodal feature fusion network is used to fuse multimodal features between different modalities. The trained asymmetric convolutional layer is used to combine the multimodal fusion features of different branch networks. The combined final features are classified and regressed to obtain the localization information of the target to be tracked.