Attention-guided matching-association hyperspectral and RGB video fusion tracking method
Through the attention mechanism-guided hyperspectral and RGB video fusion tracking method, the fusion problem of unregistered hyperspectral and RGB modal video data is solved, the target tracking performance is improved, and the effective hyperspectral and RGB modal data is achieved. Fusion and target position prediction.
Patent Information
- Application Number
- CN202311753368.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-20
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2043-12-20
AI Technical Summary
The prior art is difficult to effectively integrate under-registered and aligned hyperspectral and RGB modal video data, resulting in poor target tracking performance.
Using attention mechanism to guide the fusion tracking method of matching the associated hyperspectral and RGB videos, the attention of Transformer is utilized by establishing a fusion tracking model, including hyperspectral and RGB modal branches, fusion modules, division modules, hyperspectral and RGB prediction modules. The mechanism adaptively captures the relationship between the two modal information, realizing the effective fusion of multimodal data and target position prediction.
Effective multimodal information fusion of unregistered hyperspectral and RGB modal video data is achieved, which alleviates the tracking difficulties caused by modal spatial differences and improves the target tracking performance.
Smart Images

Figure CN117689692B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of computer vision and relates to a video tracking method, in particular to a fusion tracking method based on hyperspectral video and RGB video. Background Art
[0002] Object tracking is an important task in the field of computer vision, which requires predicting the position of the target in the video based on its appearance features. At present, most object tracking algorithms are developed based on RGB modal data, but this modal data may not provide enough information to distinguish different targets in some complex scenarios. Hyperspectral images are a type of data that can simultaneously record the spatial position and spectral information of an object. It can provide more material information when the target appearance is similar, thereby enhancing the performance of object tracking. However, the spatial resolution of hyperspectral images is usually lower than that of RGB images. Therefore, how to effectively fuse hyperspectral information with RGB information and take advantage of the advantages of both modal data is the current research focus.
[0003] Although the existing multimodal fusion tracking methods can fuse video information of different modalities, such algorithms require strict matching and alignment of video data of different modalities. However, the two modal data in the actual hyperspectral-RGB multimodal video dataset are not completely aligned, and there are certain spatial differences. This requires a lot of preliminary work such as multimodal video data registration to achieve effective fusion of hyperspectral-RGB multimodal video data. In addition, how to make full use of the complementary information of hyperspectral and RGB modal data to improve the performance of target tracking is also an urgent problem to be solved in fusion tracking work. Summary of the invention
[0004] The present invention proposes a hyperspectral and RGB video fusion tracking method guided by attention mechanism matching association, the purposes of which are: 1. To solve the problem that incompletely registered and aligned hyperspectral and RGB modal video data cannot be effectively fused with multimodal information; 2. To solve the problem that the complementary information of hyperspectral and RGB modal data cannot be fully utilized, resulting in poor target tracking performance.
[0005] The technical solution of the present invention is as follows:
[0006] A hyperspectral and RGB video fusion tracking method guided by an attention mechanism for matching association, comprising the following steps:
[0007] Step 1, establishing a fusion tracking model, the fusion tracking model includes a hyperspectral modality branch, an RGB modality branch, a fusion module, a partitioning module, a hyperspectral prediction module and an RGB prediction module;
[0008] The hyperspectral modality branch is used to obtain a hyperspectral modality-specific representation based on a hyperspectral image, and the RGB modality branch is used to obtain an RGB modality-specific representation based on an RGB image; the fusion module is based on Transformer, and is used to fuse the hyperspectral modality-specific representation and the RGB modality-specific representation to obtain a multimodal fusion representation, and the division module is used to divide the multimodal fusion representation into a hyperspectral sub-representation and an RGB sub-representation; the hyperspectral prediction module is used to obtain a hyperspectral target result, i.e., a target position in a currently input hyperspectral image, based on the hyperspectral sub-representation, and the RGB prediction module is used to obtain an RGB target result, i.e., a target position in a currently input RGB image, based on the RGB sub-representation;
[0009] Step 2, obtaining the hyperspectral modal video data a and the target position Pa1 in the first frame of the hyperspectral image in the hyperspectral modal video data a, and obtaining the hyperspectral template patch a′1 in the first frame of the hyperspectral image and the hyperspectral search area a2 in the second frame of the hyperspectral image based on the target position Pa1; using the hyperspectral template patch a′1 as the first input of the hyperspectral modal branch, and using the hyperspectral search area a2 as the second input of the hyperspectral modal branch;
[0010] At the same time, the RGB modality video data b and the target position Pb1 in the first frame RGB image of the RGB modality video data b are obtained, and based on the target position Pb1, the RGB template patch b′1 in the first frame RGB image and the RGB search area b2 in the second frame RGB image are obtained; the RGB template patch b′1 is used as the first input of the RGB modality branch, and the RGB search area b2 is used as the second input of the RGB modality branch;
[0011] Step 3, set the count value i=2;
[0012] Step 4: Input the first input and the second input of the current hyperspectral modality branch into the hyperspectral modality branch, and input the first input and the second input of the current RGB modality branch into the RGB modality branch to obtain the target position Pa in the i-th frame hyperspectral image output by the fusion tracking model. i and the target position Pb in the i-th frame RGB image i ;
[0013] Step 5, let i=i+1: if i is greater than the total number of frames of the hyperspectral modality video data a or i is greater than the total number of frames of the RGB modality video data b, then execute step 8; otherwise, obtain the next frame, i.e., the hyperspectral image of the i-th frame and the RGB image of the i-th frame, and then execute step 6;
[0014] Step 6: For the i-th frame of hyperspectral image, based on the target position Pa in the i-1-th frame of hyperspectral image i-1Get the search area a of the i-th frame hyperspectral image i Then the hyperspectral template patch a′1 is used as the first input of the hyperspectral modality branch, and the search area a of the i-th frame hyperspectral image is i As the second input of the current hyperspectral modality branch;
[0015] For the i-th RGB image, the search area b of the i-th RGB image is obtained based on the target position in the i-1-th RGB image. i , and then the RGB template patch b′1 is used as the first input of the RGB modality branch, and the search area b of the i-th frame RGB image is i As the second input of the current RGB modality branch;
[0016] Step 7: Return to step 4.
[0017] Step 8: Obtain the tracking result based on the target position corresponding to each frame of the RGB image.
[0018] As a further improvement of the hyperspectral and RGB video fusion tracking method guided by the attention mechanism for matching association: for the first frame of hyperspectral image, the hyperspectral template patch is obtained by expanding the target position in the hyperspectral image of the frame as the center to obtain a region with a side length twice that of the target position, and then shaping to obtain the hyperspectral template patch;
[0019] For a certain frame of hyperspectral image, the method of obtaining its hyperspectral search area is: taking the target position of the previous frame of hyperspectral image as the center, expanding to obtain an area with a side length four times that of the target position, and then shaping to obtain the hyperspectral search area;
[0020] For the first frame of RGB image, the method of obtaining its RGB template patch is as follows: taking the target position in the frame of RGB image as the center, expanding to obtain a region with a side length twice that of the target position, and then reshaping to obtain the RGB template patch;
[0021] For a certain frame of RGB image, the method of obtaining its RGB search area is: taking the target position of the previous frame of RGB image as the center, expanding to obtain an area with a side length four times that of the target position, and then reshaping to obtain the RGB search area.
[0022] As a further improvement of the hyperspectral and RGB video fusion tracking method guided by the attention mechanism matching association: the hyperspectral modality branch contains a hyperspectral template sub-branch, a hyperspectral search sub-branch and a cross-correlation operator;
[0023] The hyperspectral template sub-branch and the hyperspectral search sub-branch have the same structure and share weights; the hyperspectral template sub-branch is used to process the first input a1 of the hyperspectral modal branch, the hyperspectral search sub-branch is used to process the second input a2 of the hyperspectral modal branch, and the cross-correlation operator is based on the output F of the hyperspectral template sub-branch. a1 and the output F of the hyperspectral search subbranch a2 Perform convolution operation to get the result Indicates that F a1 As the convolution kernel and F a2 Perform convolution operation.
[0024] As a further improvement of the hyperspectral and RGB video fusion tracking method guided by the attention mechanism to match the association, the hyperspectral modality branch also includes a feature self-enhancement module, which uses the multi-head self-attention mechanism of Transformer to suppress the redundant noise of the hyperspectral modality specific representation and enhance the hyperspectral modality specific representation information to obtain the enhanced hyperspectral modality specific representation R Ea .
[0025] As a further improvement of the hyperspectral and RGB video fusion tracking method guided by the attention mechanism matching association: the RGB modality branch contains an RGB template sub-branch, an RGB search sub-branch and a cross-correlation operator;
[0026] The RGB template sub-branch and the RGB search sub-branch have the same structure and share weights; the RGB template sub-branch is used to process the first input b1 of the RGB modality branch, the RGB search sub-branch is used to process the second input b2 of the RGB modality branch, and the cross-correlation operator is based on the output F of the RGB template sub-branch. b1 and the output F of the RGB search sub-branch b2 Perform convolution operation to get the result Indicates that F b1 As the convolution kernel and F b2 Perform convolution operation.
[0027] As a further improvement of the attention mechanism guided matching-associated hyperspectral and RGB video fusion tracking method: the RGB modality branch also includes a feature self-enhancement module, which uses the Transformer's multi-head self-attention mechanism to suppress the redundant noise of the RGB modality-specific representation and enhance the RGB modality-specific representation information to obtain the enhanced RGB modality-specific representation R Eb .
[0028] As a further improvement of the attention mechanism guided matching-associated hyperspectral and RGB video fusion tracking method, the processing process of the fusion module is as follows: the hyperspectral modality specific representation REa and RGB modality-specific representation R Eb The spatial dimensions are deconstructed into semantic units, and then spliced and fused to obtain the multimodal fusion representation R. ab .
[0029] As a further improvement of the attention mechanism guided matching-associated hyperspectral and RGB video fusion tracking method, the processing process of the split module is: the multimodal fusion representation R output by the fusion module ab Divided into two parts: corresponding to R Ea Get hyperspectral sub-characterization r aba , corresponding to R Eb Get RGB sub-representation R abb .
[0030] As a further improvement of the attention mechanism guided matching associated hyperspectral and RGB video fusion tracking method:
[0031] Both the hyperspectral prediction module and the RGB prediction module contain a classification head and a regression head;
[0032] The hyperspectral prediction module is based on the input hyperspectral sub-representation, and obtains the hyperspectral classification response map ClR through the classification head and regression head respectively. a and hyperspectral regression response map ReR a , then based on the hyperspectral classification response map ClR a and hyperspectral regression response map ReR a Get the target position of the current hyperspectral image;
[0033] The RGB prediction module is based on the input RGB sub-representation, and obtains the RGB classification response map ClR through the classification head and regression head respectively. b and RGB regression response map ReR b , then based on the RGB classification response map ClR b and RGB regression response map ReR b Get the target position of the current RGB image.
[0034] Compared with the prior art, the present invention has the following beneficial effects:
[0035] 1. In this method, a hyperspectral modal branch and an RGB modal branch are set up. By updating the search area of the second input in the two modal branches respectively, the modality-specific representation of the same target in the spatially mismatched multimodal data is obtained, and the effective multimodal information fusion of incompletely registered and aligned hyperspectral and RGB modal video data is realized, which alleviates the problem of difficulty in multimodal information tracking caused by the spatial difference between the two modalities.
[0036] 2. This method uses the attention mechanism in Transformer to adaptively capture the relationship between the two modal information. The hyperspectral modality-specific representation and RGB modality-specific representation obtained by the hyperspectral modality branch and the RGB modality branch are respectively input into two feature self-enhancement modules to enhance the modality-specific semantic information, and then deconstruct them into semantic units in the direction of the spatial dimension. Finally, the Transformer-based attention mechanism is used to globally match and associate all semantic units of the two modal data, thereby realizing the adaptive aggregation of the two modal information and improving the tracking performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 It is a schematic diagram of the overall structure of the fusion tracking model in the present invention;
[0038] Figure 2 It is a structural diagram of the feature self-enhancement module;
[0039] Figure 3 It is a structural diagram of the Transformer-based fusion module. DETAILED DESCRIPTION
[0040] The technical solution of the present invention is described in detail below with reference to the accompanying drawings:
[0041] like Figure 1 , a hyperspectral and RGB video fusion tracking method guided by attention mechanism matching association, the steps include:
[0042] Step 1: Establish a fusion tracking model, which includes a hyperspectral modality branch, an RGB modality branch, a fusion module, a partitioning module, a hyperspectral prediction module and an RGB prediction module.
[0043] (i) The hyperspectral modality branch is used to obtain a specific representation of the hyperspectral modality based on the hyperspectral image.
[0044] Specifically, the hyperspectral modality branch contains a hyperspectral template sub-branch, a hyperspectral search sub-branch and a cross-correlation operator.
[0045] The hyperspectral template sub-branch and the hyperspectral search sub-branch have the same structure and shared weights.
[0046] The hyperspectral template sub-branch f a1 (·) is used to process the first input a1 of the hyperspectral modality branch, the hyperspectral search sub-branch f a2 (·) is used to process the second input a2 of the hyperspectral modality branch. The cross-correlation operator is based on the output F of the hyperspectral template sub-branch. a1 =f a1 (a1) and the output F of the hyperspectral search subbranch a2=f a2 (a2) Perform convolution operation to obtain the result Indicates that F a1 As the convolution kernel and F a2 Perform convolution operation.
[0047] Furthermore, the hyperspectral modality branch also includes a feature self-enhancement module FSE(·). Figure 2 The feature self-enhancement module uses the multi-head self-attention mechanism of Transformer to suppress the redundant noise of the hyperspectral modality-specific representation and enhance the hyperspectral modality-specific representation information to obtain the enhanced hyperspectral modality-specific representation R Ea =FSE(R a ).
[0048] (ii) The RGB modality branch is used to obtain an RGB modality specific table based on an RGB image.
[0049] Specifically, the RGB modality branch contains an RGB template sub-branch, an RGB search sub-branch, and a cross-correlation operator.
[0050] The RGB template sub-branch and the RGB search sub-branch have the same structure and share weights.
[0051] The RGB template sub-branch f b1 (·) is used to process the first input b1 of the RGB modality branch, the RGB search sub-branch f b2 (·) is used to process the second input b2 of the RGB modality branch. The cross-correlation operator is based on the output F of the RGB template sub-branch. b1 =f b1 (b1) and the output F of the RGB search sub-branch b2 =f b2 (b2) Perform convolution operation to obtain the result Indicates that F b1 As the convolution kernel and F b2 Perform convolution operation.
[0052] Furthermore, the RGB modality branch also includes a feature self-enhancement module FSE(·). Figure 2 As above, the feature self-enhancement module uses the multi-head self-attention mechanism of Transformer to suppress the redundant noise of the RGB modality-specific representation and enhance the RGB modality-specific representation information to obtain the enhanced RGB modality-specific representation R Eb =FSE(R b ).
[0053] (III) The fusion module is based on Transformer, and is used to fuse the hyperspectral modality-specific representation and the RGB modality-specific representation to obtain a multimodal fusion representation.
[0054] Specifically, Figure 3 The processing process of the fusion module is: the hyperspectral modality specific representation R Ea and RGB modality-specific representation R Eb The spatial dimensions are deconstructed into semantic units, and then spliced and fused to obtain the multimodal fusion representation R. ab The process can be expressed as R ab =TrFM(R Ea ,R Eb ).
[0055] (iv) The division module is used to characterize the multimodal fusion into a hyperspectral sub-characterization and an RGB sub-characterization.
[0056] Specifically, the processing process of the division module is: the multimodal fusion representation R output by the fusion module ab Divided into two parts: corresponding to R Ea Get the hyperspectral sub-characterization R aba , corresponding to R Eb Get RGB sub-representation R abb .
[0057] (V) Both the hyperspectral prediction module and the RGB prediction module contain a classification head and a regression head.
[0058] The hyperspectral prediction module is used to obtain a hyperspectral target result, that is, a target position in a currently input hyperspectral image, according to the hyperspectral sub-characterization.
[0059] The hyperspectral prediction module is based on the input hyperspectral sub-representation, and obtains the hyperspectral classification response map ClR through the classification head and regression head respectively. a and hyperspectral regression response map ReR a , then based on the hyperspectral classification response map ClR a and hyperspectral regression response map ReR a Get the target position of the current hyperspectral image.
[0060] Similarly, the RGB prediction module is used to obtain an RGB target result, that is, a target position in a currently input RGB image, according to the RGB sub-representation.
[0061] The RGB prediction module is based on the input RGB sub-representation, and obtains the RGB classification response map ClR through the classification head and regression head respectively. b and RGB regression response map ReR b, then based on the RGB classification response map ClR b and RGB regression response map ReR b Get the target position of the current RGB image.
[0062] Step 2: Obtain the hyperspectral modal video data a and the target position Pa1 in the first frame of the hyperspectral image in the hyperspectral modal video data a, and obtain the hyperspectral template patch a′1 in the first frame of the hyperspectral image and the hyperspectral search area a2 in the second frame of the hyperspectral image based on the target position Pa1. The target position Pa1 can be expressed as (x a ,y a ,w a ,h a ), where (x a ,y a ) is the coordinate of the upper left corner of the target box in the first frame of the hyperspectral modality image, w a and h a are respectively the width and height of the target box in the first frame of the hyperspectral modality image.
[0063] Specifically, for the first frame of hyperspectral image, the method of obtaining its hyperspectral template patch a′1 is as follows: expanding with the target position in the frame of hyperspectral image as the center to obtain an area with a side length twice that of the target position, and then reshaping it to a size of 128×128 to obtain the hyperspectral template patch.
[0064] For a certain frame of hyperspectral image, the method of obtaining its hyperspectral search area is: expanding the target position of the previous frame of hyperspectral image as the center to obtain an area with a side length four times that of the target position, and then reshaping it to a size of 256×256 to obtain the hyperspectral search area.
[0065] The hyperspectral template patch a′1 is taken as the first input of the hyperspectral modality branch, and the hyperspectral search area a2 is taken as the second input of the hyperspectral modality branch.
[0066] At the same time, the RGB modality video data b and the target position Pb1 in the first frame RGB image of the RGB modality video data b are obtained, and based on the target position Pb1, the RGB template patch b′1 in the first frame RGB image and the RGB search area b2 in the second frame RGB image are obtained. The target position Pb1 can be expressed as (x b ,y b ,w b ,h b ), where (x b ,y b ) is the coordinate of the upper left corner of the target box in the first frame of the RGB modality image, w b and h b They are respectively the width and height of the target box in the first frame of the RGB modality image.
[0067] Similarly, for the first frame of RGB image, the RGB template patch b′1 is obtained by expanding the target position in the frame of RGB image as the center to obtain an area with a side length twice that of the target position, and then reshaping it to a size of 128×128 to obtain the RGB template patch.
[0068] For a certain frame of RGB image, the method of obtaining its RGB search area is: expanding with the target position of the previous frame of RGB image as the center to obtain an area with a side length four times that of the target position, and then reshaping it to a size of 256×256 to obtain the RGB search area.
[0069] The RGB template patch b′1 is taken as the first input of the RGB modality branch, and the RGB search area b2 is taken as the second input of the RGB modality branch.
[0070] Step 3: Set the count value i=2.
[0071] Step 4: Input the first input and the second input of the current hyperspectral modality branch into the hyperspectral modality branch, and input the first input and the second input of the current RGB modality branch into the RGB modality branch to obtain the target position Pa in the i-th frame hyperspectral image output by the fusion tracking model. i and the target position Pb in the i-th frame RGB image i .
[0072] Step 5, let i=i+1: If i is greater than the total number of frames of the hyperspectral modality video data a or i is greater than the total number of frames of the RGB modality video data b, then execute step 8. Otherwise, obtain the next frame, i.e., the hyperspectral image of the i-th frame and the RGB image of the i-th frame, and then execute step 6.
[0073] Step 6: For the i-th frame of hyperspectral image, based on the target position Pa in the i-1-th frame of hyperspectral image i-1 Get the search area a of the i-th frame hyperspectral image i Then the hyperspectral template patch a′1 is used as the first input of the hyperspectral modality branch, and the search area a of the i-th frame hyperspectral image is i As the second input of the current hyperspectral modality branch.
[0074] For the i-th RGB image, the search area b of the u-th RGB image is obtained based on the target position in the i-1-th RGB image. i , and then the RGB template patch b′1 is used as the first input of the RGB modality branch, and the search area b of the i-th frame RGB image is i As the second input of the current RGB modality branch.
[0075] Step 7: Return to step 4.
[0076] Step 8: Obtain the tracking result based on the target position corresponding to each frame of the RGB image.
Claims
1. A hyperspectral and RGB video fusion tracking method guided by attention mechanism matching association, characterized by the steps include: Step 1, establishing a fusion tracking model, the fusion tracking model includes a hyperspectral modality branch, an RGB modality branch, a fusion module, a partitioning module, a hyperspectral prediction module and an RGB prediction module; The hyperspectral modality branch is used to obtain a hyperspectral modality-specific representation based on a hyperspectral image, and the RGB modality branch is used to obtain an RGB modality-specific representation based on an RGB image; The fusion module is based on Transformer and is used to fuse the hyperspectral modality-specific representation and the RGB modality-specific representation to obtain a multimodal fusion representation. The processing process of the fusion module is: Ea and RGB modality-specific representation R Eb The spatial dimensions are deconstructed into semantic units, and then spliced and fused to obtain the multimodal fusion representation R. ab The division module is used to divide the multimodal fusion representation into a hyperspectral sub-representation and an RGB sub-representation. The processing process of the division module is: the multimodal fusion representation R output by the fusion module ab Divided into two parts: corresponding to the hyperspectral modality specific representation R Ea Get the hyperspectral sub-characterization R aba , corresponding to the RGB modality-specific representation R Eb Get RGB sub-representation R abb ; The hyperspectral prediction module is used to obtain a hyperspectral target result, i.e., a target position in a currently input hyperspectral image, according to the hyperspectral sub-characterization, and the RGB prediction module is used to obtain an RGB target result, i.e., a target position in a currently input RGB image, according to the RGB sub-characterization; Step 2, obtaining the hyperspectral modal video data a and the target position Pa1 in the first frame of the hyperspectral image in the hyperspectral modal video data a, and obtaining the hyperspectral template patch a′1 in the first frame of the hyperspectral image and the hyperspectral search area a2 in the second frame of the hyperspectral image based on the target position Pa1; using the hyperspectral template patch a′1 as the first input of the hyperspectral modal branch, and using the hyperspectral search area a2 as the second input of the hyperspectral modal branch; At the same time, the RGB modality video data b and the target position Pb1 in the first frame RGB image of the RGB modality video data b are obtained, and based on the target position Pb1, the RGB template patch b′1 in the first frame RGB image and the RGB search area b2 in the second frame RGB image are obtained; the RGB template patch b′1 is used as the first input of the RGB modality branch, and the RGB search area b2 is used as the second input of the RGB modality branch; Step 3, set the count value i=2; Step 4: Input the first input and the second input of the current hyperspectral modality branch into the hyperspectral modality branch, and input the first input and the second input of the current RGB modality branch into the RGB modality branch to obtain the target position Pa in the i-th frame hyperspectral image output by the fusion tracking model. i and the target position Pb in the i-th frame RGB image i ; Step 5, let i=i+1: if i is greater than the total number of frames of the hyperspectral modality video data a or i is greater than the total number of frames of the RGB modality video data b, then execute step 8; otherwise, obtain the next frame, i.e., the hyperspectral image of the i-th frame and the RGB image of the i-th frame, and then execute step 6; Step 6: For the i-th frame of hyperspectral image, based on the target position Pa in the i-1-th frame of hyperspectral image i-1 Get the search area a of the i-th frame hyperspectral image i Then the hyperspectral template patch a′1 is used as the first input of the hyperspectral modality branch, and the search area a of the i-th frame hyperspectral image is i As the second input of the current hyperspectral modality branch; For the i-th RGB image, the search area b of the i-th RGB image is obtained based on the target position in the i-1-th RGB image. i , and then the RGB template patch b′1 is used as the first input of the RGB modality branch, and the search area b of the i-th frame RGB image is i As the second input of the current RGB modality branch; Step 7: Return to step 4. Step 8: Obtain the tracking result based on the target position corresponding to each frame of the RGB image; The hyperspectral modality branch contains a hyperspectral template sub-branch, a hyperspectral search sub-branch and a cross-correlation operator; The hyperspectral template sub-branch and the hyperspectral search sub-branch have the same structure and share weights; the hyperspectral template sub-branch is used to process the first input a1 of the hyperspectral modal branch, the hyperspectral search sub-branch is used to process the second input a2 of the hyperspectral modal branch, and the cross-correlation operator is based on the output F of the hyperspectral template sub-branch. a1 and the output F of the hyperspectral search subbranch a2 Perform convolution operation to get the result Indicates that F a1 As the convolution kernel and F a2 Perform convolution operation; The hyperspectral modality branch also includes a feature self-enhancement module, which uses the multi-head self-attention mechanism of Transformer to suppress the redundant noise of the hyperspectral modality-specific representation and enhance the hyperspectral modality-specific representation information to obtain the enhanced hyperspectral modality-specific representation R Ea ; The RGB modality branch contains an RGB template sub-branch, an RGB search sub-branch, and a cross-correlation operator; The RGB template sub-branch and the RGB search sub-branch have the same structure and share weights; the RGB template sub-branch is used to process the first input b1 of the RGB modality branch, the RGB search sub-branch is used to process the second input b2 of the RGB modality branch, and the cross-correlation operator is based on the output F of the RGB template sub-branch. b1 and the output F of the RGB search sub-branch b2 Perform convolution operation to get the result Indicates that F b1 As the convolution kernel and F b2 Perform convolution operation; The RGB modality branch also includes a feature self-enhancement module, which uses the multi-head self-attention mechanism of Transformer to suppress the redundant noise of the RGB modality-specific representation and enhance the RGB modality-specific representation information to obtain the enhanced RGB modality-specific representation R Eb .
2. The hyperspectral and RGB video fusion tracking method guided by the attention mechanism matching association as claimed in claim 1, characterized in that: For the first frame of hyperspectral image, the method of obtaining its hyperspectral template patch is as follows: taking the target position in the frame of hyperspectral image as the center, expanding to obtain a region with a side length twice that of the target position, and then shaping to obtain the hyperspectral template patch; For a certain frame of hyperspectral image, the method of obtaining its hyperspectral search area is: taking the target position of the previous frame of hyperspectral image as the center, expanding to obtain an area with a side length four times that of the target position, and then shaping to obtain the hyperspectral search area; For the first frame of RGB image, the method of obtaining its RGB template patch is as follows: taking the target position in the frame of RGB image as the center, expanding to obtain a region with a side length twice that of the target position, and then reshaping to obtain the RGB template patch; For a certain frame of RGB image, the method of obtaining its RGB search area is: taking the target position of the previous frame of RGB image as the center, expanding to obtain an area with a side length four times that of the target position, and then reshaping to obtain the RGB search area.
3. The hyperspectral and RGB video fusion tracking method guided by the attention mechanism matching association as claimed in claim 1, characterized in that: The hyperspectral prediction module and the RGB prediction module both contain a classification head and a regression head. The hyperspectral prediction module obtains the hyperspectral classification response map ClR through the classification head and regression head based on the input hyperspectral sub-representation. a and hyperspectral regression response map ReR a , then based on the hyperspectral classification response map ClR a and hyperspectral regression response map ReR a Get the target position of the current hyperspectral image; The RGB prediction module is based on the input RGB sub-representation, and obtains the RGB classification response map ClR through the classification head and regression head respectively. b and RGB regression response map ReR b , then based on the RGB classification response map ClR b and RGB regression response map ReR b Get the target position of the current RGB image.
