Multi-modal few-sample semantic segmentation-oriented cross-scale feature matching method

Abstract

The invention relates to a cross-scale feature matching method oriented to multi-modal few-sample semantic segmentation. According to the cross-scale feature matching method, selective cross-modal feature fusion is realized in combination with different modal characteristics; according to the method and the system, the feature matching of support query objects of different scales is realized by using a proposed pixel-to-patch cross-attention matching module (pixel-to-patch cross-attention matching module, so that accurate pixel-level segmentation is carried out on new-class images without labels.

Description

Technical Field

[0001] The present invention relates to a cross-scale feature matching method for multi-modal few-sample semantic segmentation, and belongs to the technical field of artificial intelligence pattern recognition. Background Art

[0002] As a basic research problem in scene understanding, the goal of semantic segmentation is to divide an image into several regions according to the categories of objects in it and achieve dense prediction at the pixel level. Although semantic segmentation models have made great progress in the past decade, under complex conditions such as low light and complex background, single-modal semantic segmentation tasks based on visible light images are difficult to meet the needs of stable detection.

[0003] In recent years, scholars at home and abroad have begun to shift their research focus to multi-modal technology, in which depth modality (depth) images provide additional spatial geographic information, and infrared modality (thermal) images provide surface temperature information of objects. Sending both as supplementary modalities into the model can effectively improve the semantic segmentation performance in complex scenes. The inventions with patent publication numbers CN118552738A and CN115661455A respectively realize semantic segmentation by combining visible light with depth images and infrared images. In order to obtain more cross-modal information, the present invention simultaneously utilizes depth and infrared image information and proposes a semantic segmentation task based on visible light-depth-infrared (RGB-DT or VDT). However, the quality of supplementary modality data is not stable, and there are inherent barriers between different modalities. An effective method is needed to achieve efficient fusion of multimodal data by the model.

[0004] In addition, pixel-level annotation is time-consuming and labor-intensive, and the lack of high-quality multimodal semantic segmentation datasets in the field has greatly restricted the development of the model. Therefore, scholars proposed multi-modal few-shot semantic segmentation (multi-modal FSS), which aims to learn a general model that uses extremely small labeled samples to predict unlabeled samples for new categories of data. Existing methods can be divided into two categories: prototype-based and metric learning (matching-based). Compared with the former, which uses mask average pooling to compress features and cause information loss, metric learning uses modules such as mutual attention to achieve global matching of feature pairs, bringing performance improvement.

[0005] The invention with patent publication number CN116468895A discloses a similarity matrix-guided few-sample semantic segmentation method and system. Based on the prototype learning strategy, the method calculates the similarity matrix of the query image and the support image multiplied by the true value mask, and further obtains a rough binary semantic segmentation result. As a priori map, it guides the generation of the prototype and outputs a refined binary semantic segmentation result. The invention improves the accuracy of small-sample semantic segmentation.

[0006] However, conventional feature matching modules are difficult to support the situation where the size difference of the query objects is large, and an effective method is needed to achieve feature matching of objects of different scales, that is, to achieve pixel-to-patch matching. Based on this, the present invention provides a cross-scale feature matching method for multimodal few-sample semantic segmentation. Summary of the invention

[0007] In order to overcome the defects in the prior art that unstable multimodal features in complex scenes are difficult to efficiently fuse and that feature pairs of support query objects of different scales are difficult to model and match, the present invention provides a cross-scale feature matching method for multimodal few-shot semantic segmentation, which combines the characteristics of different modalities to achieve selective cross-modal feature fusion; the proposed pixel-to-patch cross attention matching module (PTPCA) is used to achieve feature matching of support query objects of different scales, thereby making accurate pixel-level segmentation of unlabeled new class images.

[0008] A cross-scale feature matching method for multimodal few-shot semantic segmentation includes the following steps:

[0009] Step 1: Send the support and query images of the visible light modality and infrared modality to the feature extraction module to extract the query features F of the visible light modality respectively. RGB,Q , the query feature F of infrared mode T,Q , the support feature F of the visible light modality RGB,S and the infrared modality support feature F T,S ;

[0010] Step 2: The extracted infrared and visible light features are grouped according to the source of the set and sent to the spatial fusion module built based on the convolutional neural network to generate the fusion feature F E,Q and F E,S ;

[0011] Step 3: Group the 2nd, 3rd, and 4th layer features of the fusion feature into the pixel-image block mutual attention matching module to obtain the three-layer preliminary output and

[0012] Step 4: Downsample the query image of the depth modality according to the size of the preliminary output obtained in step 3 to obtain The initial output corresponding to the size is sent to the prior fusion module constructed by Transformer, and the output represents the depth information.

[0013] Step 5: Different levels of Fusion is performed to obtain a rough mask M guided by the cross-scale mutual attention mechanism. c , and combine it with the fusion query features of layer 1 and layer 2 The concatenation is performed in the channel dimension and fed into the decoder to obtain the prediction P of the query image.

[0014] The feature extraction module in step 1 is ResNet50 pre-trained with the ImageNet dataset, which contains 5 ResNet blocks. More specifically, the feature extraction module parameters used by the query and support branches are completely shared and frozen, so that the model can focus more on the generalization ability of detecting unknown class objects during the training phase.

[0015] The step 2 of grouping the extracted infrared and visible light features according to the source of the set specifically includes: dividing the extracted infrared and visible light features into two groups according to the source of the set, namely the infrared and visible light features extracted by the query branch and the infrared and visible light branches extracted by the support branch.

[0016] The step 2 spatial fusion module specifically includes:

[0017] The infrared and visible light sent to the spatial fusion module are grouped according to the level, and the splicing operation of the channel dimension is performed separately; the spliced ​​features of each layer are sent to the feedforward neural network, the required features are selectively enhanced, and the fusion features are generated.

[0018] The pixel-image block mutual attention matching module in step 3 specifically includes:

[0019] The input support and query fusion features and After straightening the spatial dimensions, they are linearly mapped and used as query elements and key elements for matrix calculation, and the output is a large score matrix with a shape of head×hw×hw;

[0020] The last two dimensions of the obtained large score matrix are expanded in turn, and the corresponding matrix changes are performed to obtain independent small score matrices M related to the query component and the support component respectively. attn,Q and M attn,S ;

[0021] M attn,Qand M attn,S Independently send it to the pixel-image block pooling module to obtain a new small score matrix M' that integrates multi-scale features attn,Q and M' attn,S ;

[0022] The new small score matrix M' associated with the query component and the support component respectively attn,Q and M' attn,S After the corresponding matrix transformation, the outputs are added to obtain the score matrix M' that can achieve modeling of objects of different scales, and then sent to the normalized exponential function Softmax to compare with the true value mask of the straightened support image. Perform matrix multiplication and finally output the preliminary output described in step 2

[0023]

[0024] The matrix changes performed after the large score matrix is ​​expanded specifically include: generating an independent small score matrix M related to the query component: attn,Q When generating independent small score matrices related to support components, the hw of the third dimension of the large score matrix is ​​split, and finally the independent small score matrix with the shape of head×hw×h×w is obtained.

[0025] The pixel-image block pooling module is composed of two serial average pooling layers including residual connections. Through the pixel-image block pooling module, object feature modeling of different scales in the score matrix can be achieved.

[0026] The matrix transformation of the new small score matrix specifically includes: operating the new small score matrix M' associated with the query component attn,Q For example, after concatenating the 3rd and 4th dimensions to generate a matrix of shape head×hw×hw, the 2nd and 3rd dimensions are swapped to achieve the operation of the new small score matrix related to the support component. It is only necessary to concatenate the 3rd and 4th dimensions.

[0027] The prior fusion module in step 4 is constructed by Transformer and specifically includes: and After straightening, they are fed into the mutual attention module as query elements and key / value elements respectively to generate outputs representing the depth information

[0028] The output fusion of different levels is specifically to send the output to the convolution block in order from high to low levels, and then splice it with the output of the previous level after upsampling.

[0029] The decoder in step 5 is constructed by a convolutional neural network, and the output of the last layer should be 1×h×w, where 1 represents the number of channels required for the prediction mask, and the value range is 0 or 1, indicating whether the prediction class is included or not, respectively. h is the height of the input image, and w is the width of the input image.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] The present invention relates to a cross-scale feature matching method for multi-modal few-sample semantic segmentation. For images of visible light modality, depth modality and infrared modality acquired from different sensors, different cross-modal fusion strategies are implemented according to the characteristics of each modality; at the same time, objects of different sizes in the support query feature are modeled to achieve efficient prediction of new class images for learning with few samples. In this method, a spatial fusion module and a priori fusion module are used for infrared modality images and depth modality images respectively to achieve feature supplementation and enhancement with the visible light modality; the pixel-image mutual attention matching module implemented based on the average pooling module can achieve cross-scale support-query object pair feature modeling and enhance the model's detection ability for new classes. Combined with the modules, this method can efficiently obtain complementary beneficial features between modalities, capture the modeling relationship between support elements and query elements, achieve high-quality multi-modal few-sample semantic segmentation prediction, and thus ensure the stability of the visual algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0033] Figure 1 This is a model network structure diagram of a cross-scale feature matching method for multimodal few-sample semantic segmentation in the present invention, which mainly includes five structures: feature extraction module, spatial fusion module SF, prior fusion module PF, pixel-image block mutual attention matching module and decoder;

[0034] Figure 2 This is a specific structural diagram of a pixel-image block mutual attention matching module of a cross-scale feature matching method for multimodal few-sample semantic segmentation in the present invention. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] Combination Figure 1 , a cross-scale feature matching method for multimodal few-shot semantic segmentation, comprising the following steps:

[0037] Step 1: Send the support and query images of the visible light modality and infrared modality to the feature extraction module to extract the query features F of the visible light modality respectively. RGB,Q , the query feature F of infrared mode T,Q , the support feature F of the visible light modality RGB,S and the infrared modality support feature F T,S ;

[0038] Step 2: The extracted infrared and visible light features are grouped according to the source of the collection and sent to the spatial fusion module (SF) built based on the convolutional neural network to generate the fusion feature F E,Q and F E,S ; The extracted infrared and visible light features are grouped according to the source of the collection, specifically divided into two groups, namely the infrared and visible light features extracted by the query branch and the infrared and visible light branches extracted by the support branch.

[0039] Specifically, the specific processing process of the spatial fusion module is as follows:

[0040] 1) The infrared and visible light sent to the spatial fusion module are grouped according to the level, and the splicing operation of the channel dimension is performed respectively;

[0041] 2) Send the concatenated features of each layer into the feedforward neural network, selectively enhance the required features, and generate fused features.

[0042] Step 3: Group the 2nd, 3rd, and 4th layer features of the fusion feature into the pixel-image block mutual attention matching module to obtain the three-layer preliminary output and

[0043] Specifically, the pixel-image block mutual attention matching module is as follows:

[0044] 1) Fusion features of input support and query and After straightening the spatial dimensions, they are linearly mapped and used as query elements and key elements for matrix calculation, and the output is a large score matrix with a shape of head×hw×hw;

[0045] 2) Expand the last two dimensions of the obtained large score matrix in turn and perform corresponding matrix changes to obtain independent small score matrices M related to the query component and the support component respectively. attn,Q and M attn,S ;

[0046] 3) M attn,Q and M attn,S Independently send it to the pixel-image block pooling module to obtain a new small score matrix M' that integrates multi-scale features attn,Q and M' attn,S ;

[0047] 4) The new small score matrix M' associated with the query component and the support component attn,Q and M' attn,S After the corresponding matrix transformation, the outputs are added to obtain the score matrix M' that can achieve modeling of objects of different scales, and then sent to the normalized exponential function Softmax to compare with the true value mask of the straightened support image. Perform matrix multiplication and finally output the preliminary output described in step 2

[0048] The matrix transformation performed after the large score matrix is ​​expanded is specifically to generate an independent small score matrix M related to the query component. attn,Q When generating the independent small score matrix related to the support component, the third dimension is split, and finally the independent small score matrix of the shape of head×hw×h×w is obtained.

[0049] The pixel-image block pooling module is composed of two serial 3×3 average pooling layers including residual connections. Through the pixel-image block pooling module, object feature modeling of different scales in the score matrix can be achieved.

[0050] The matrix transformation of the new small score matrix is ​​specifically to operate the new small score matrix M' associated with the query component. attn,Q For example, concatenate the 3rd and 4th dimensions to generate a matrix of shape head×hw×hw, and then swap the 2nd and 3rd dimensions. Relatively speaking, to operate the new small score matrix related to the support component, only the 3rd and 4th dimensions need to be concatenated.

[0051] Step 4: Downsample the query image of the depth modality according to the size of the preliminary output obtained in step 3 to obtain The initial output corresponding to the size is sent to the Prior Fusion Module (PF) constructed by Transformer, and the output represents the output of the depth information. The prior fusion module is constructed by Transformer. and After straightening, they are fed into the mutual attention module as query elements and key / value elements respectively to generate outputs representing the depth information

[0052] Step 5: Different levels of Fusion is performed to obtain a rough mask M guided by the cross-scale mutual attention mechanism. c , and combine it with the fusion query features of layer 1 and layer 2 The concatenation is performed in the channel dimension and sent to the decoder to obtain the prediction P of the query image. The output fusion of the different levels is specifically to send the output to the convolution block in order from high to low levels, and then concatenate it with the output of the previous level after upsampling. The decoder module is constructed by a convolutional neural network, and its last layer output should be 1×h×w, where 1 represents the number of channels required for the prediction mask, the value range is 1 or 0, indicating whether the prediction class is included or not, respectively, h is the height of the input image, and w is the width of the input image.

[0053] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions and variations of these embodiments are made without departing from the principles and spirit of the present invention, and still fall within the scope of protection of the present invention.

Claims

1. A cross-scale feature matching method for multimodal few-shot semantic segmentation, characterized by: The following steps are involved: Step 1: Send the support and query images of the visible light modality and infrared modality to the feature extraction module to extract the query features F of the visible light modality respectively. RGB,Q , the query feature F of infrared mode T,Q , the support feature F of the visible light modality RGB,S and the infrared modality support feature F T,S ; Step 2: The extracted infrared and visible light features are grouped according to the source of the set and sent to the spatial fusion module built based on the convolutional neural network to generate the fusion feature F E,Q and F E,S ; Step 3: Group the 2nd, 3rd, and 4th layer features of the fusion feature into the pixel-image block mutual attention matching module to obtain the three-layer preliminary output and Step 4: Downsample the query image of the depth modality according to the size of the preliminary output obtained in step 3 to obtain The initial output corresponding to the size is sent to the prior fusion module constructed by Transformer to output the output representing the depth information Step 5: Different levels of Fusion is performed to obtain a rough mask M guided by the cross-scale mutual attention mechanism. c , and combine it with the fusion query features of layer 1 and layer 2 The concatenation is performed in the channel dimension and fed into the decoder to obtain the prediction P of the query image.

2. The cross-scale feature matching method for multimodal few-sample semantic segmentation according to claim 1, characterized in that: The feature extraction module in step 1 is ResNet50 pre-trained with the ImageNet dataset, and includes 5 ResNet blocks.

3. The cross-scale feature matching method for multimodal few-sample semantic segmentation according to claim 1, characterized in that: The step 2 of grouping the extracted infrared and visible light features according to the source of the set specifically includes: dividing the extracted infrared and visible light features into two groups according to the source of the set, namely the infrared and visible light features extracted by the query branch and the infrared and visible light branches extracted by the support branch.

4. The cross-scale feature matching method for multimodal few-sample semantic segmentation according to claim 2, characterized in that: The step 2 spatial fusion module specifically includes: The infrared and visible light sent to the spatial fusion module are grouped according to the levels, and the splicing operation of the channel dimension is performed respectively; the spliced ​​features of each layer are sent to the feedforward neural network, the required features are selectively enhanced, and the fusion features are generated.

5. The cross-scale feature matching method for multimodal few-sample semantic segmentation according to claim 1, characterized in that: The pixel-image block mutual attention matching module in step 3 specifically includes: The input support and query fusion features and After straightening the spatial dimensions, they are linearly mapped and used as query elements and key elements for matrix calculation, and the output is a large score matrix with a shape of head×hw×hw; The last two dimensions of the obtained large score matrix are expanded in turn, and the corresponding matrix changes are performed to obtain independent small score matrices M related to the query component and the support component respectively. attn,Q and M attn,S ; M attn,Q and M attn,S Independently send it to the pixel-image block pooling module to obtain a new small score matrix M' that integrates multi-scale features attn,Q and M' attn,S ; The new small score matrix M' associated with the query component and the support component respectively attn,Q and M' attn,S After the corresponding matrix transformation, the outputs are added to obtain the score matrix M' that can achieve modeling of objects of different scales, and then sent to the normalized exponential function Softmax to compare with the true value mask of the straightened support image. Perform matrix multiplication and finally output the preliminary output described in step 2 6. The cross-scale feature matching method for multimodal few-sample semantic segmentation according to claim 5, characterized in that: The matrix changes performed after the large score matrix is ​​expanded specifically include: generating an independent small score matrix M related to the query component: attn,Q When generating independent small score matrices related to support components, the hw of the third dimension of the large score matrix is ​​split, and finally the independent small score matrix with the shape of head×hw×h×w is obtained.

7. The cross-scale feature matching method for multimodal few-sample semantic segmentation according to claim 5, characterized in that: The pixel-image block pooling module is composed of two serial average pooling layers including residual connections.

8. The cross-scale feature matching method for multimodal few-sample semantic segmentation according to claim 5, characterized in that: The matrix transformation of the new small score matrix specifically includes: operating the new small score matrix M' associated with the query component attn,Q For example, after concatenating the 3rd and 4th dimensions to generate a matrix of shape head×hw×hw, the 2nd and 3rd dimensions are swapped to achieve the operation of the new small score matrix related to the support component. It is only necessary to concatenate the 3rd and 4th dimensions.

9. The cross-scale feature matching method for multimodal few-sample semantic segmentation according to claim 1, characterized in that: The prior fusion module in step 4 is constructed by Transformer and specifically includes: and After straightening, they are fed into the mutual attention module as query elements and key / value elements respectively to generate outputs representing the depth information 10. The cross-scale feature matching method for multimodal few-sample semantic segmentation according to claim 1, characterized in that: The decoder in step 5 is constructed by a convolutional neural network, and the output of the last layer should be 1×h×w, where 1 represents the number of channels required for the prediction mask, and the value range is 1 or 0, indicating whether the prediction class is included or not, respectively. h is the height of the input image, and w is the width of the input image.

Images