Small sample rgb-t segmentation method and system based on hierarchical matching guided feature enhancement network
The Hierarchical Matching Guided Feature Enhancement Network (HMFENet) addresses the issues of insufficient generalization ability and inadequate information utilization in the RGB-T semantic segmentation method under conditions with few samples, achieving high-precision segmentation in harsh environments.
Patent Information
- Application Number
- CN202510197022.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-21
- Publication Date
- 2025-11-11
- Estimated Expiration
- 2045-02-21
AI Technical Summary
Existing RGB-T semantic segmentation methods have insufficient generalization ability under limited sample conditions, cannot effectively utilize multimodal information, and are difficult to calculate the correspondence between different scale pixel sets between query images and support images, leading to overfitting and information loss.
The Hierarchical Matching Guided Feature Enhancement Network (HMFENet) is adopted. Through multi-scale and multi-layer feature extraction, multi-layer relation vector establishment, mixer module fusion and constraint loss function training, fine-grained correspondence is constructed to enhance the complementarity of RGB and TIR features.
It improves the segmentation accuracy of RGB-T semantic segmentation under adverse weather and complex lighting conditions, avoids coarse segmentation granularity and overfitting problems, and demonstrates better segmentation results.
Smart Images
Figure CN120088482B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image segmentation technology, and in particular to a semantic segmentation method that combines visible light images and thermal infrared images. Background Technology
[0002] Semantic segmentation is a key task in scene understanding, aiming to assign a category label to each pixel in an image, and has wide applications in fields such as autonomous driving, medical imaging, and the military. However, the environmental adaptability and robustness of current single visible light image segmentation methods are still relatively limited, leading to the gradual introduction of dual-modal semantic segmentation methods using visible light and thermal infrared (RGB-T) images. Thermal infrared (TIR) images can highlight targets but perform poorly in capturing details. Visible light (RGB) images have rich texture details but are susceptible to scene brightness. By integrating information from these two modalities, more reliable semantic segmentation can be achieved even in adverse environments.
[0003] Currently, supervised learning-based semantic segmentation techniques are limited by the need for large amounts of labeled data, especially RGB-T image datasets. Furthermore, these methods lack generalization ability when faced with limited training data. Therefore, few-shot learning (FSL) is introduced to complete semantic segmentation tasks, known as few-shot segmentation (FSS). FSS algorithms can be divided into two main approaches: prototype-based and matching-based. However, the prototype structure was originally proposed for image classification tasks. When applied to pixel-level prediction tasks like semantic segmentation, the averaging operation leads to significant information loss and overfitting. Therefore, matching-based methods can use convolution as a metric. These methods typically establish dense pixel-to-pixel correspondences. Given the fine-grained nature of segmentation tasks, they avoid information compression in prototypes and often achieve better results. However, for pixel-level prediction tasks, calculating the correspondences between pixel sets at different scales between the query image and support images remains a significant challenge. Most recent bimodal FSS studies have neither utilized feature representations at different levels from early to late convolutional stages nor constructed pairwise feature associations at different scales, both of which hinder the effectiveness of capturing dense pixel correspondences. In view of this, the present invention proposes a Hierarchical Matching Guided Mask Fusion Network (HMMFNet). Summary of the Invention
[0004] RGB-T semantic segmentation provides reliable support for autonomous driving tasks under adverse weather and complex lighting conditions. Considering the difficulties in collecting and annotating RGB-T data, this invention introduces few-shot semantic segmentation (FSS). Compared to ordinary FSS, RGB-T FSS faces the dual challenges of modality fusion and the inherent characteristics of TIR modalities, challenges that are further exacerbated under limited data conditions. Simultaneously, existing methods often fail to fully utilize multimodal information, neglecting the information redundancy problem in multimodal learning and weakening the complementarity of cross-modal features.
[0005] To address the shortcomings of existing technologies, this invention provides a few-sample RGB-T segmentation method based on a hierarchical matching-guided feature enhancement network, comprising the following steps:
[0006] Step 1: Create a dataset by dividing the dataset into a training set and a test set, each containing a query set and a support set. The support set contains support images and corresponding support masks for different modalities, while the query set contains query images and corresponding query masks for different modalities.
[0007] Step 2, construct a hierarchical matching-guided feature enhancement network, including the following sub-steps:
[0008] Step 2.1: Extract multi-scale, multi-layer features from the supporting image and the query image;
[0009] Step 2.2: Perform multi-scale support mask extraction on the support mask;
[0010] Step 2.3: Establish multi-level relationship vectors between supporting features and query features under different modalities;
[0011] Step 2.4: Convert the multi-layer relation vectors into intermediate masks respectively;
[0012] Step 2.5: Fuse the intermediate masks to generate the prediction mask, which is the segmentation result;
[0013] Step 2.6: Construct the loss function for training the hierarchical matching guided feature enhancement network;
[0014] Step 3: Select several groups of images from the training set and input them into the hierarchical matching guided feature enhancement network for indiscriminate iterative training;
[0015] Step 4: Select K sets of images from the training set and input them into the trained network model. Perform K forward propagation cycles to obtain K predicted masks. Finally, vote on each pixel position based on all K predicted masks to obtain the segmentation result.
[0016] Furthermore, in step 1, multiple pairs of thermal infrared images I with the same scene resolution of H×W are obtained under N target categories and 3N basic categories respectively. T Visible light image I R With the corresponding mask M, the datasets for the basic category and the target category are respectively divided into training sets D. tr and test set D te D te Used to construct a requirements partitioning task, D tr This is used to construct a series of simulated segmentation tasks, each containing a support set. and a query set in and M s These represent the supported images and corresponding support masks in RGB and TIR modes, respectively. and M q These represent the query images and corresponding query masks in RGB and TIR modes, respectively.
[0017] Furthermore, the specific implementation method of step 2.1 is as follows;
[0018] Establish a feature extraction model, and and The data is fed into the pre-trained feature extractor Swin_Transformer to obtain a set of multi-scale, multi-layer features. in, These represent the query images in RGB and TIR modes, respectively. Let i ∈ {1 / 8, 1 / 6, 1 / 32} and l ∈ {1, ..., L} represent the supported images in RGB and TIR modes, respectively. i}, L i It is the number of layers at scale i. This represents the query feature of the l-th layer at the i-th size in the RGB modality. This represents the query feature of the l-th layer at the i-th size in the TIR mode. This represents the support features of the l-th layer at the i-th size in the RGB modality. The support feature of the l-th layer at the i-th size in the TIR mode is represented by the following formula:
[0019]
[0020] Swin_Transformer is a Transformer architecture that efficiently extracts multi-scale features with its hierarchical structure and sliding window self-attention mechanism.
[0021] Furthermore, in step 2.3, based on the relevant blocks of the attention mechanism, query features are established for the RGB modality respectively. and supporting features Multi-level relation vectors between and query features under TIR modality and supporting features Multi-level relation vector The specific implementation method is as follows:
[0022] Step 2.3.1, Establishing multi-layer relation vectors in RGB mode: In the multi-layer features corresponding to scale i, for a given layer l, first, the two-dimensional input... and All elements are flattened, and then, based on the original Transformer architecture, positional encoding and linear projection are added to further flatten the structure. and Generate Q and K matrices, and apply support masks at different scales i. Flattening is performed to construct the V matrix. The formulas for calculating Q, K, and V are as follows:
[0023]
[0024] Among them, W q W k W v ∈R d×d is a learnable parameter, d is the dimension of the query and support vectors, and i∈{1 / 8,1 / 6,1 / 32};
[0025] Step 2.3.2: Perform dot product attention calculation on Q and K, use the Softmax function to measure the similarity between the query pixel and all supporting pixels, and then multiply the result by the similarity-based weight V to obtain the attention-based F. Att The calculation formula is:
[0026]
[0027] Where d represents the dimension of K;
[0028] Step 2.3.3, for scale i and layer l, F Att Remodeling to 2D to generate relational features The calculation formula is:
[0029]
[0030] Where Reshape represents the reshaping operation;
[0031] Step 2.3.4: Analyze the relational features of all layers. Summing by positions yields the corresponding relational features. The calculation formula is:
[0032]
[0033] Step 2.3.5: Use bilinear interpolation to match the relational features at scales of 1 / 32 and 1 / 16. Upsampling is performed to reshape the feature to the same size as the 1 / 8 scale relational feature. Then, the three features are added element-wise to produce a multi-scale aggregated feature. The calculation formula is:
[0034]
[0035] Upsample represents the upsampling operation. This indicates bitwise addition;
[0036] Step 2.3.6, will The final multi-layer relation vector is generated by skipping the last layer of features at a 1 / 8 scale. The calculation formula is:
[0037]
[0038] Here, Concat represents the connection operation;
[0039] Step 2.3.7: Obtain the multi-layer relation vector under the RGB modality. Subsequently, the same method was used to extract the multilevel relation vectors under the TIR modes.
[0040] Furthermore, in step 2.4, the mixer module is used to convert the multi-layer relation vectors into intermediate masks. The mixer module consists of a series of 5-layer networks. The first, third, and fifth layers are convolutional blocks with the same structure, which is a combination of convolution and ReLU operations repeated twice. The second and fourth layers are upsampling blocks, and the output of the fifth layer is the intermediate mask.
[0041] Furthermore, in step 2.5, the intermediate mask is fused using the fusion module, and the specific implementation method is as follows:
[0042] Step 2.5.1, convert the two intermediate masks and The synthesized mask is obtained by adding each bit sequentially, and then the channel attention model is used to apply the mask to each bit. and The processing is performed, and the results are multiplied bit-by-bit by the synthesized mask to obtain RGB enhancement features and TIR enhancement features. Then, the two enhancement masks are concatenated to generate a shallow fusion mask F. shallow F shallow The calculation formula is as follows:
[0043]
[0044] Where CA stands for Channel Attention Model. This is a bitwise addition operation. This is a bitwise multiplication operation;
[0045] Step 2.5.2, F shallow Feeding in a convolutional block consisting of 1×1 convolutions, 3×3 convolutions, BatchNorm, and ReLU, a deep fusion mask F is generated. deep F deep The calculation formula is as follows:
[0046] F deep =RELU(BN(Conv3×3(Conv1×1(F shallow ))))
[0047] Where ReLU is the linear rectification function and BN is the batch normalization function;
[0048] Step 2.5.3, apply the shallow fusion mask F shallow and the deep fusion mask F deep The concatenation, through a dilated convolutional block, generates the final prediction mask M. pre That is, the segmentation result, M pre The calculation formula is:
[0049] M pre =DCBR(F shallow +F deep )
[0050] DCBR is an expanded convolutional block, consisting of 3×3 convolutions, BatchNorm, and ReLU.
[0051] Furthermore, in step 2.6, the hierarchical matching guided feature enhancement network is trained using a method that incorporates cross-entropy loss, support modality difference loss, and query modality difference loss. The specific implementation is as follows:
[0052] Step 2.6.1: Prepare the last layer features of the support and query images at a 1 / 32 scale for both RGB and TIR modalities, as well as the corresponding support and query masks at that scale:
[0053] Step 2.6.2, mutual information loss I(v) R ,v T The calculation process of ) is as follows: Let The calculation formula is as follows:
[0054]
[0055] in, Represents entropy, and Represents the marginal entropy. Represents joint entropy;
[0056] Step 2.6.3, calculate v R ,v T The Kullback-Leibler divergence (KL divergence) between KL(v R ||v T ) and KL(v T ||v R The calculation formula is as follows:
[0057]
[0058] in, It is the cross-entropy, where p represents v. * The position of a certain pixel; v * For v R or v T ;
[0059] Step 2.6.4, calculate the support mode difference loss L. SMI ;
[0060]
[0061] Step 2.6.5, let The query modal difference loss L was calculated in the same way. QMI ;
[0062] Step 2.6.6, Cross-entropy loss L C The calculation formula is:
[0063]
[0064] in, Indicates query mask M q The pixel value at position (x,y). Indicates the prediction mask M pre For the pixel value at position, h and w represent w. * Height and width;
[0065] Step 2.6.7, the total training loss L is L SMI ,L QMI and L C The weighted sum is calculated using the following formula:
[0066] L=αL C +β(lSMI +L QMI )
[0067] Where α and β are learnable loss weights.
[0068] Furthermore, in step 3, from the training set D tr Two sets of images containing the same target category are randomly selected. A set of construction Another set of constructions Will The input is fed into a hierarchical matching-guided feature enhancement network, and the output is a predicted mask M. pre and with M q As the ground truth, supervised training is performed and the network parameters are updated; a series of simulated tasks are constructed in the above manner, and undifferentiated iterative training is performed on these tasks.
[0069] Furthermore, in the testing phase of step 4, from the test set D... te Select K groups of images containing the same target category to construct and another structure Inputting a network model with fixed parameters and performing K forward propagations yields a set of prediction masks. Each input is: and Finally, a vote is taken at each pixel position based on all K predicted masks to obtain the segmentation result.
[0070] This invention also provides a few-sample RGB-T segmentation system based on a hierarchical matching-guided feature enhancement network, comprising:
[0071] The processor and memory are used to store program instructions, and the processor is used to call the stored instructions in the memory to execute the few-sample RGB-T segmentation method based on hierarchical matching guided feature enhancement network as described in the above technical solution.
[0072] Compared with the prior art, the advantages and beneficial effects of the present invention are as follows: (1) An FSL framework HMFENet for RGB-Thermal segmentation is proposed, which comprehensively analyzes feature information at different levels and establishes fine-grained correspondence; (2) A new constraint function is introduced, which enhances the complementarity of RGB and TIR features by reducing redundancy and selecting key complementary multimodal features; (3) HMFENet is superior to the existing FSS network, showing state-of-the-art results and avoiding problems such as coarse segmentation granularity and overfitting. Attached Figure Description
[0073] Figure 1 This is a diagram of the overall architecture of the present invention;
[0074] Figure 2 This is a schematic diagram of the relevant block network structure based on the attention mechanism of the present invention;
[0075] Figure 3 This is a schematic diagram of the mixer network structure of the present invention;
[0076] Figure 4 This is a schematic diagram of the fusion module network structure of the present invention;
[0077] Figure 5 The figure shows the experimental results of this invention. Detailed Implementation
[0078] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.
[0079] This invention proposes a hierarchical matching guided feature enhancement network (HMFENet) for RGB-T FSS, the overall network block diagram of which is shown below. Figure 1 As shown, the main steps include:
[0080] Step 1: Establish a dataset by acquiring multiple pairs of thermal infrared (TIR) images of the same scene with a resolution of H×W under N target categories and 3N basic categories. T Visible light (RGB) image I R And the corresponding mask M. This invention divides the datasets for the basic category and the target category into training sets D respectively. tr and test set D te D te Used to construct a requirements partitioning task, D tr This is used to construct a series of simulated segmentation tasks. Each task contains a support set. and a query set in and M s These represent the supported images and corresponding support masks in RGB and TIR modes, respectively. and M q These represent the query images and corresponding query masks in RGB and TIR modes, respectively.
[0081] Step 2, construct a hierarchical matching-guided feature enhancement network, including the following sub-steps:
[0082] Step 2.1, Multi-scale and multi-layer feature extraction: Establish a feature extraction model, and... and The data is fed into the pre-trained feature extractor Swin_Transformer to obtain a set of multi-scale, multi-layer features. Where i∈{1 / 8,1 / 6,1 / 32}, l∈{1,...,L} i}, L i It is the number of layers at scale i. This represents the query feature of the l-th layer at the i-th size in the RGB modality. This represents the query feature of the l-th layer at the i-th size in the TIR mode. This represents the support features of the l-th layer at the i-th size in the RGB modality. The support feature of the l-th layer at the i-th size in the TIR mode is represented by the following formula:
[0083]
[0084] Swin_Transformer is a Transformer architecture that efficiently extracts multi-scale features with its hierarchical structure and sliding window self-attention mechanism.
[0085] Step 2.2, Multi-scale support mask extraction: Extract the support mask M s Downsampling is performed to obtain support masks at different scales i. The calculation formula is:
[0086]
[0087] Downsample represents the downsampling operation.
[0088] Step 2.3, Multi-level Relation Vector Construction: Design relevant blocks based on attention mechanisms to build query features under RGB modalities respectively. and supporting features Multi-level relation vectors between and query features under TIR modality and supporting features Multi-level relation vector
[0089] Furthermore, the specific implementation of step 2.3 includes the following sub-steps:
[0090] Step 2.3.1, considering the consistent operating procedures of the two modes, this invention uses the RGB mode to illustrate the specific process. In the multi-layer features corresponding to scale i, for a given layer l, the two-dimensional input is first... and All elements are flattened, and then, based on the original Transformer architecture, positional encoding and linear projection are added to further flatten the structure. and Generate Q and K matrices and apply support masks. Flattening is performed to construct the V matrix. The formulas for calculating Q, K, and V are as follows:
[0091]
[0092] Among them, W q W k W v ∈R d×d is a learnable parameter, and d is the dimension of the query and support vectors.
[0093] Step 2.3.2: Perform dot product attention calculation on Q and K, use the Softmax function to measure the similarity between the query pixel and all supporting pixels, and then multiply the result by the similarity-based weight V to obtain the attention-based F. Att The calculation formula is:
[0094]
[0095] Where d represents the dimension of K.
[0096] Step 2.3.3: For scale i and layer l, the model will be reshaped into 2D to generate relational features. The calculation formula is:
[0097]
[0098] Reshape represents the reshaping operation.
[0099] Step 2.3.4, for the relational features of all layers. Summing by positions yields the corresponding relational features. The calculation formula is:
[0100]
[0101] Step 2.3.5: Use bilinear interpolation to match the relational features at scales of 1 / 32 and 1 / 16. Upsampling is performed to reshape the feature to the same size as the 1 / 8 scale relational feature. Then, the three features are added element-wise to produce a multi-scale aggregated feature. The calculation formula is:
[0102]
[0103] Upsample represents the upsampling operation. This indicates addition by place value.
[0104] Step 2.3.6, will The final multi-layer relation vector is generated by skipping the last layer of features at a 1 / 8 scale. The calculation formula is:
[0105]
[0106] Concat is a connection operation.
[0107] Step 2.3.7: Obtain the multi-layer relation vector under the RGB modality. Subsequently, this invention employs the same method to extract multilevel relation vectors under TIR modes.
[0108] Step 2.4, Intermediate Mask Generation: Establish a mixer module to generate the multi-layer relation vectors respectively. and Convert to intermediate mask and The mixer consists of five cascaded layers. The first, third, and fifth layers are convolutional blocks with identical structures, consisting of two repetitions of a combination of convolution and ReLU operations. The second and fourth layers are upsampling blocks, and the output of the fifth layer serves as an intermediate mask. and
[0109] Step 2.5: Intermediate Mask Fusion: Establish a fusion module to fuse the intermediate mask. and Generate prediction mask M pre That is, the segmentation result.
[0110] Furthermore, the specific implementation of step 2.5 includes the following sub-steps:
[0111] Step 2.5.1: Combine the two intermediate masks and The synthesized mask is obtained by adding each bit sequentially, and then the channel attention model is used to apply the mask to each bit. and The processing results are multiplied bit-by-bit by the synthesized mask to obtain RGB enhancement features and TIR enhancement features. Then, the two enhancement masks are concatenated to generate a shallow fusion mask F. shallow F shallow The calculation formula is as follows:
[0112]
[0113] Where CA stands for Channel Attention Model. This is a bitwise addition operation. This is a bitwise multiplication operation.
[0114] Step 2.5.2: F shallowFeeding in a convolutional block consisting of 1×1 convolutions, 3×3 convolutions, BatchNorm, and ReLU, a deep fusion mask F is generated. deep F deep The calculation formula is as follows:
[0115] F deep =RELU(BN(Conv3×3(Conv1×1(F shallow ))))
[0116] Where ReLU is the linear rectification function and BN is the batch normalization function.
[0117] Step 2.5.3: Apply the shallow fusion mask F shallow and the deep fusion mask F deep The concatenation, through a dilated convolutional block, generates the final prediction mask M. pre This refers to the segmentation result. M pre The calculation formula is:
[0118] M pre =DCBR(F shallow +F deep )
[0119] DCBR is an expanded convolutional block, consisting of 3×3 convolutions, BatchNorm, and ReLU.
[0120] Step 2.6: Backpropagation: The calculation of model loss includes cross-entropy loss, support mode difference loss, and query mode difference loss.
[0121] Furthermore, the specific implementation of step 2.6 includes the following sub-steps:
[0122] Step 2.6.1: Prepare the last layer (l=3) features of the support and query images at a scale of 1 / 32 (i=1 / 32) for both RGB and TIR modes, as well as the corresponding support and query masks at that scale:
[0123] Step 2.6.2: Support modal difference loss L SMI For example, let's introduce the mutual information loss I(v) R ,v T The calculation process of ) is as follows: The calculation formula is as follows:
[0124]
[0125] in, Represents entropy, and Represents the marginal entropy. This represents the joint entropy.
[0126] Step 2.6.3: Calculate the v R ,v T The Kullback-Leibler divergence (KL divergence) between KL(v R ||v T ) and KL(v T ||v R The calculation formula is as follows:
[0127]
[0128] in, It is the cross-entropy, where p represents v. * The position of a certain pixel, v * For v R or v T .
[0129] Step 2.6.4: Calculate the support mode difference loss L SMI .
[0130]
[0131] Step 2.6.5: Let The query modal difference loss L was calculated in the same way. QMI .
[0132] Step 2.6.6: Cross-entropy loss L C The calculation formula is:
[0133]
[0134] in, Indicates query mask M q The pixel value at position (x,y). Indicates the prediction mask M pre For the pixel value at position, h and w represent v. * Height and width.
[0135] Step 2.6.7: The total training loss L is L SMI ,L QMI and L C The weighted sum is calculated using the following formula:
[0136] L=αL C +β(L SMI +L QMI )
[0137] Where α and β are learnable loss weights.
[0138] Step 3, Model Training. Construct a simulated task: from D tr Two sets of images containing the same target category are randomly selected. A set of construction Another set of constructions Will Input into the model, output prediction mask M pre and with M q As ground truth, supervised training is performed and network parameters are updated. A series of simulated tasks are constructed in the above manner, and undifferentiated iterative training is performed on these tasks.
[0139] Step 4, Testing. For the testing phase under K-shot, from D... te Select K groups of images containing the same target category to construct and another structure A model with fixed parameters is used, and a set of prediction masks is obtained by performing K forward propagations. Each input is: and Finally, a vote is taken at each pixel position based on all K predicted masks to obtain the segmentation result.
[0140] Figure 5 The segmentation results of HMFENET and other state-of-the-art methods in a 1-shot setting are visualized. The figures clearly show that the proposed HMFENET achieves the best segmentation results, especially in challenging scenarios such as overexposed, dark, or complex backgrounds. For example, in the Car-Night class, the ASGNet method fails due to extremely harsh environmental conditions, but HMFENET maintains stable segmentation performance. Furthermore, in relatively well-lit environments, such as the Car-Day class, HMFENET can also produce more refined segmentation results.
[0141] On the other hand, embodiments of the present invention also provide a few-sample RGB-T segmentation system based on a hierarchical matching guided feature enhancement network, comprising:
[0142] The processor and memory are used to store program instructions, and the processor is used to call the stored instructions in the memory to execute the few-sample RGB-T segmentation method based on hierarchical matching guided feature enhancement network as described in the above technical solution.
[0143] The specific embodiments described above are merely illustrative examples illustrating the spirit of the present invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.
Claims
1. A few-sample RGB-T segmentation method based on a hierarchical matching-guided feature enhancement network, characterized in that, Includes the following steps: Step 1: Create a dataset by dividing the dataset into a training set and a test set, each containing a query set and a support set. The support set contains support images and corresponding support masks for different modalities, while the query set contains query images and corresponding query masks for different modalities. Step 2, construct a hierarchical matching-guided feature enhancement network, including the following sub-steps: Step 2.1: Extract multi-scale, multi-layer features from the supporting image and the query image; Step 2.2: Perform multi-scale support mask extraction on the support mask; Step 2.3: Based on the relevant blocks of the attention mechanism, establish multi-layer relation vectors under different modalities. The specific implementation method is as follows: Step 2.3.1, Establishing multi-layer relation vectors in RGB mode: (corresponding to scale) In the multi-layer features, for a given layer First, input the two-dimensional data. and All elements are flattened, and then, based on the original Transformer architecture, positional encoding and linear projection are added to further flatten the structure. and Generate Q and K matrices and apply them to different scales. Support mask below To construct the V matrix, flatten the matrix. The calculation formula is: in, These are learnable parameters. It is the dimension of the query and support vectors. ; Step 2.3.2: Perform dot product attention calculation on Q and K, use the Softmax function to measure the similarity between the query pixel and all supporting pixels, and then combine the result with similarity-based weights. Features of attention mechanism obtained through multiplication ; Step 2.3.3, will Remodeling to 2D to generate relational features ,in, As a scale, For layers; Step 2.3.4: Analyze the relational features of all layers. Sum of digits; Step 2.3.5: Use bilinear interpolation to match the relational features at scales of 1 / 32 and 1 / 16. Upsampling is performed to reshape the feature to the same size as the 1 / 8 scale relational feature. Then, the three features are added element-wise to produce a multi-scale aggregated feature. ; Step 2.3.6, will The final multi-layer relation vector is generated by skipping the last layer of features at a 1 / 8 scale. ; Step 2.3.7: Extract the multilevel relation vector under the TIR mode using the same method. ; Step 2.4: Convert the multi-layer relation vectors into intermediate masks respectively; Step 2.5: Fuse the intermediate masks to generate the prediction mask, which is the segmentation result; Step 2.6: Construct the loss function for training the hierarchical matching guided feature enhancement network; Step 3: Select several groups of images from the training set and input them into the hierarchical matching guided feature enhancement network for indiscriminate iterative training; Step 4: Select K sets of images from the training set and input them into the trained network model. Perform K forward propagation operations to obtain K prediction masks. Finally, based on all... Each predicted mask is voted on at each pixel position to obtain the segmentation result.
2. The few-sample RGB-T segmentation method based on hierarchical matching guided feature enhancement network as described in claim 1, characterized in that: In step 1, multiple pairs of thermal infrared images with the same scene resolution of H×W are acquired under N target categories and 3N basic categories respectively. Visible light images and the corresponding mask The datasets for the basic category and the target category are divided into training sets and training sets respectively. and test set , Used to build a requirements splitting task. This is used to construct a series of simulated segmentation tasks, each containing a support set. and a query set ,in , These represent the supported images and corresponding support masks in RGB and TIR modes, respectively. , and These represent the query images and corresponding query masks in RGB and TIR modes, respectively.
3. The few-sample RGB-T segmentation method based on hierarchical matching guided feature enhancement network as described in claim 1, characterized in that: The specific implementation method of step 2.1 is as follows; Establish a feature extraction model, and , , and Feed into the pre-trained feature extractor Obtain a set of multi-scale, multi-layer features. ;in, , These represent the query images in RGB and TIR modes, respectively. , These represent the supported images in RGB and TIR modes, respectively. , , yes Number of layers at the scale Indicates the RGB mode. Size below Query characteristics of the layer Indicates the first TIR mode Size below Query characteristics of the layer Indicates the RGB mode. Size below Layer support features, Indicates the first TIR mode Size below The supporting features of a layer are calculated using the following formula: in It is a kind of The architecture, with its hierarchical structure and sliding window self-attention mechanism, efficiently extracts multi-scale features.
4. The few-sample RGB-T segmentation method based on hierarchical matching guided feature enhancement network as described in claim 1, characterized in that: In step 2.3.2, the characteristics of the attention mechanism are discussed. The calculation formula is: in, express Dimension size; In step 2.3.3, The calculation formula is: in Indicates a reshaping operation; In step 2.3.4, The calculation formula is: In step 2.3.5, The calculation formula is: in Indicates an upsampling operation. This indicates bitwise addition; In step 2.3.6, The calculation formula is: in, For connection operations.
5. The few-sample RGB-T segmentation method based on hierarchical matching guided feature enhancement network as described in claim 1, characterized in that: In step 2.4, the mixer module is used to convert the multi-layer relation vectors into intermediate masks. The mixer module consists of a series of 5 layers. The first, third and fifth layers are convolutional blocks with the same structure, which is a combination of convolution and ReLU operations repeated twice. The second and fourth layers are upsampling blocks, and the output of the fifth layer is the intermediate mask.
6. The few-sample RGB-T segmentation method based on hierarchical matching guided feature enhancement network as described in claim 1, characterized in that: In step 2.5, the intermediate mask is fused using the fusion module. The specific implementation method is as follows: Step 2.5.1, convert the two intermediate masks and The synthesized mask is obtained by adding each bit sequentially, and then the channel attention model is used to apply the mask to each bit. and The processing results are multiplied bit-by-bit by the synthesized mask to obtain RGB enhancement features and TIR enhancement features. Then, the two enhancement masks are concatenated to generate a shallow fusion mask. , The calculation formula is as follows: in, For channel attention model, This is a bitwise addition operation. This is a bitwise multiplication operation; Step 2.5.2, will Sent by convolution, convolution, and The convolutional blocks are composed of elements that generate a deep fusion mask. , The calculation formula is as follows: Where ReLU is the linear rectification function and BN is the batch normalization function; Step 2.5.3, apply the shallow fusion mask. and the deep fusion mask The concatenation, through a dilated convolutional block, generates the final prediction mask. That is, the segmentation result. The calculation formula is: in It is an extended convolutional block, composed of convolution, and composition.
7. The few-sample RGB-T segmentation method based on hierarchical matching guided feature enhancement network as described in claim 1, characterized in that: In step 2.6, the hierarchical matching guided feature enhancement network is trained using a method that incorporates cross-entropy loss, support modality difference loss, and query modality difference loss. The specific implementation is as follows: Step 2.6.1: Prepare the last layer features of the support and query images at a 1 / 32 scale for both RGB and TIR modalities, as well as the corresponding support and query masks at that scale: ; Step 2.6.2, Mutual Information Loss The calculation process: Let , The calculation formula is as follows: in, Represents entropy, and Represents the marginal entropy. Represents joint entropy; Step 2.6.3, Calculation Kullback-Leibler divergence (KL divergence) and The calculation formula is as follows: in, It's cross-entropy. express The position of a certain pixel in the image; for or ; Step 2.6.4, calculate the support mode difference loss. ; Step 2.6.5, let , The query modal difference loss was calculated in the same way. ; Step 2.6.6, Cross-entropy Loss The calculation formula is: in, Indicates the query mask exist The pixel value of the location, Indicates the prediction mask For the pixel value at the location express Height and width; Step 2.6.7, Total Training Loss yes and The weighted sum is calculated using the following formula: in , These are learnable loss weights.
8. The few-sample RGB-T segmentation method based on hierarchical matching guided feature enhancement network as described in claim 2, characterized in that: In step 3, from the training set Two sets of images containing the same target category are randomly selected. A set used for construction Another set of constructions ,Will The input is fed into a hierarchical matching-guided feature enhancement network, and the output is a predicted mask. and with As the ground truth, supervised training is performed and the network parameters are updated; a series of simulated tasks are constructed in the above manner, and undifferentiated iterative training is performed on these tasks.
9. The few-sample RGB-T segmentation method based on hierarchical matching guided feature enhancement network as described in claim 2, characterized in that: In the testing phase of step 4, from the test set Selecting those containing the same target category Group Image Construction and another structure The input is fed into a network model with fixed parameters, and a set of prediction masks is obtained by performing K forward propagations. Each input is and Finally, based on all Each predicted mask is voted on at each pixel position to obtain the segmentation result.
10. A few-sample RGB-T segmentation system based on a hierarchical matching-guided feature enhancement network, characterized in that, include: The processor and memory, wherein the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute the few-sample RGB-T segmentation method based on a hierarchical matching guided feature enhancement network as described in any one of claims 1-9.