A cross-scale feature matching method for multi-modal few-shot semantic segmentation

Abstract

The application relates to a cross-scale feature matching method for multi-modal few-shot semantic segmentation, which combines different modal characteristics to realize selective cross-modal feature fusion; pixel-to-patch cross attention (PTPCA) is used to realize the matching of support query object features of different scales, and then accurate pixel-level segmentation is performed on unlabeled new class images.

Description

TECHNICAL FIELD

[0001] The application relates to a cross-scale feature matching method for multi-modal few-shot semantic segmentation, and belongs to the technical field of artificial intelligence pattern recognition. BACKGROUND

[0002] As a basic research problem in scene understanding, the purpose of semantic segmentation is to segment an image into several regions according to the categories of objects in the image, so as to realize dense prediction at the pixel level. Although great progress has been made in semantic segmentation models in the past decade, it is difficult for single-modal semantic segmentation tasks based on visible light images to meet the needs of stable detection under complex conditions such as low light and complex background.

[0003] In recent years, domestic and foreign scholars have begun to shift their research focus to multi-modal technology, in which depth images provide additional spatial geographic information, and thermal images provide object surface temperature information. By feeding both as supplementary modalities into the model, the semantic segmentation performance in complex scenes can be effectively improved. The inventions with patent publication numbers CN118552738A and CN115661455A respectively realize semantic segmentation by combining visible light with depth images and infrared images. In order to obtain more cross-modal information, the present application simultaneously utilizes depth and infrared image information, and proposes a semantic segmentation task based on visible light-depth-thermal (RGB-D-T or VDT). However, the quality of supplementary modal data is not stable, and there is an inherent barrier between different modalities, so an effective method is needed to realize the efficient fusion of multi-modal data by the model.

[0004] In addition, pixel-level labeling is time-consuming and labor-intensive, and there is a lack of high-quality multi-modal semantic segmentation dataset in the field, which to a large extent limits the development of the model. Therefore, scholars have proposed multi-modal few-shot semantic segmentation (multi-modal FSS), which aims to learn a general model that can use very small labeled samples to predict unlabeled samples for new category data. Existing methods can be divided into two categories: prototype-based and matching-based. Compared with the former, which uses mask average pooling to compress features and causes information loss, the latter uses modules such as mutual attention to realize global matching of feature pairs, bringing performance improvement.

[0005] Patent publication number CN116468895A discloses a similarity matrix-guided few-shot semantic segmentation method and system. Based on a prototype learning strategy, this method calculates a similarity matrix between a query image and a support image multiplied by a ground-truth mask. This matrix then generates a rough binary semantic segmentation result. This matrix serves as a priori map to guide prototype generation and output a refined binary semantic segmentation result. This invention improves the accuracy of few-shot semantic segmentation.

[0006] However, conventional feature matching modules struggle to support large size differences between query objects. An effective method is needed to match features of objects of different scales, i.e., pixel-to-patch matching. To address this issue, the present invention provides a cross-scale feature matching method for multimodal few-shot semantic segmentation. Summary of the Invention

[0007] In order to overcome the shortcomings of the existing technology, such as the difficulty in efficiently fusing unstable multimodal features in complex scenes and the difficulty in modeling and matching feature pairs of support query objects of different scales, the present invention provides a cross-scale feature matching method for multimodal few-shot semantic segmentation. By combining the characteristics of different modalities, selective cross-modal feature fusion is achieved. The proposed pixel-to-patch cross attention matching module (PTPCA) is used to match support query object features of different scales, thereby making accurate pixel-level segmentation of unlabeled new class images.

[0008] A cross-scale feature matching method for multimodal few-shot semantic segmentation includes the following steps:

[0009] Step 1: Send the support and query images of the visible light modality and infrared modality to the feature extraction module to extract the query features F of the visible light modality respectively. RGB,Q , infrared modality query feature F T,Q , support features F of visible light modality RGB,S and the infrared modality support feature F T,S ;

[0010] Step 2: The extracted infrared and visible light features are grouped according to the source of the collection and sent to the spatial fusion module built based on the convolutional neural network to generate the fusion feature F E,Q and F E,S ;

[0011] Step 3: Group the 2nd, 3rd, and 4th layer features of the fusion feature into the pixel-image block mutual attention matching module to obtain the three-layer preliminary output and

[0012] Step four: down-sampling the query image of the depth modality according to the size of the preliminary output obtained in step three, to obtain and the preliminary output corresponding to the size is sent into the prior fusion module constructed by the Transformer, and an output representing the depth information is output

[0013] Step five: fusing the fusion query features of different levels to obtain a rough mask M guided by the cross-scale mutual attention mechanism c , and the fusion query features of the first layer and the second layer are spliced in the channel dimension and sent into the decoder to obtain the prediction P of the query image.

[0014] The feature extraction module in step one is ResNet50 pre-trained on the ImageNet dataset, including 5 ResNet blocks. More specifically, the parameters of the feature extraction modules used by the query and support branches are completely shared and frozen, so that the model pays more attention to the generalization ability of unknown class object detection in the training stage.

[0015] In step two, the extracted infrared and visible light features are grouped according to the source of the set, which specifically includes grouping the extracted infrared and visible light features into two groups according to the source of the set, which are the infrared and visible light features extracted by the query branch and the infrared and visible light features extracted by the support branch.

[0016] The spatial fusion module in step two specifically includes:

[0017] The infrared and visible light sent into the spatial fusion module are grouped according to the level, and channel dimension splicing operation is performed respectively; the spliced features of each layer are sent into the feedforward neural network to selectively enhance the required features and generate fusion features.

[0018] The pixel-image block mutual attention matching module in step three specifically includes:

[0019] The support and query fusion features sent in and are respectively straightened in the spatial dimension, then linearly mapped, and used as query elements and key elements respectively for matrix calculation, outputting a large score matrix with a shape of head×hw×hw;

[0020] The last two dimensions of the obtained large score matrix are unfolded in turn, and the corresponding matrix changes are performed to obtain independent small score matrices related to the query component and the support component, respectively attn,Q and attn,S ;

[0021] M attn,Qand M attn,S Independently input into the pixel-image block pooling module to obtain a new small score matrix M' that integrates multi-scale features attn,Q and M' attn,S ;

[0022] The new small score matrix M' associated with the query component and the support component respectively attn,Q and M' attn,S After the corresponding matrix transformation, the outputs are added to obtain the score matrix M' that can achieve modeling of objects of different scales, and it is sent to the normalized exponential function Softmax and compared with the true value mask of the straightened support image. Perform matrix multiplication and finally output the preliminary output described in step 2

[0023]

[0024] The matrix changes after the large score matrix is ​​expanded specifically include generating independent small score matrices M related to the query components: attn,Q When generating the independent small score matrix related to the support component, the hw of the third dimension of the large score matrix is ​​split, and finally the independent small score matrix with the shape of head×hw×h×w is obtained.

[0025] The pixel-to-image-patch pooling module is composed of two serial average pooling layers with residual connections. Through the pixel-to-image-patch pooling module, object feature modeling of different scales in the score matrix can be achieved.

[0026] The matrix transformation of the new small score matrix specifically includes: operating the new small score matrix M' related to the query component attn,Q For example, after concatenating the 3rd and 4th dimensions to generate a matrix of shape head×hw×hw, the 2nd and 3rd dimensions are swapped to operate the new small score matrix related to the support component, and only the 3rd and 4th dimensions need to be concatenated.

[0027] The prior fusion module in step 4 is constructed by Transformer, specifically including: and After straightening, they are fed into the mutual attention module as query elements and key / value elements respectively to generate output representing the depth information

[0028] The output fusion of different levels is specifically to send the output into the convolution block in order from high to low levels, and then splice it with the output of the previous level after upsampling.

[0029] The decoder in the fifth step is constructed by a convolutional neural network, and the last layer output thereof should be 1xh x w, wherein 1 represents 1 channel number required by the prediction mask, the value range is 0 or 1, and 0 or 1 respectively represents containing a prediction class and not containing a prediction class, h is the height of the input image, and w is the width of the input image.

[0030] Compared with the prior art, the present application has the beneficial effects that:

[0031] The cross-scale feature matching method for multi-modal few-shot semantic segmentation provided by the present application is directed to images in visible light modalities, depth modalities and infrared modalities acquired from different sensors, different cross-modal fusion strategies are implemented according to the characteristics of each modality, and meanwhile, objects of different sizes in support query features are modeled to realize efficient prediction of new class pictures with few-shot labeling. In the method, a spatial fusion module and a prior fusion module are respectively used for infrared modality images and depth modality images to realize feature complementation and enhancement with the visible light modalities; a pixel-image mutual attention matching module based on an average pooling module can realize cross-scale support-query object pair feature modeling to enhance the detection capability of the model for new classes. In combination with the modules, the present method can efficiently acquire beneficial features complementary between modalities, capture the modeling relationship between support elements and query elements, realize high-quality multi-modal few-shot semantic segmentation prediction, and further ensure the stability of the visual algorithm. BRIEF DESCRIPTION OF DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0033] Figure 1 The model network structure diagram of the cross-scale feature matching method for multi-modal few-shot semantic segmentation provided by the present application mainly comprises five structural components of a feature extraction module, a spatial fusion module SF, a prior fusion module PF, a pixel-image block mutual attention matching module and a decoder.

[0034] Figure 2 The specific structure diagram of the pixel-image block mutual attention matching module of the cross-scale feature matching method for multi-modal few-shot semantic segmentation provided by the present application. DETAILED DESCRIPTION

[0035] With reference to the accompanying drawings: clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of the present application.

[0036] In combination Figure 1 A cross-scale feature matching method for multi-modal few-shot semantic segmentation includes the following steps:

[0037] Step one: send the visible light modality and infrared modality support and query images into the feature extraction module, and extract the visible light modality query feature F RGB,Q , the infrared modality query feature F T,Q , the visible light modality support feature F RGB,S and the infrared modality support feature F T,S ;

[0038] Step two: group the extracted infrared and visible light features according to the set source and send them into the spatial fusion module (SF) constructed based on the convolutional neural network, to generate fusion features F E,Q and F E,S ; group the extracted infrared and visible light features according to the set source, specifically, into two groups, which are the infrared and visible light features extracted by the query branch and the infrared and visible light branches extracted by the support branch.

[0039] Specifically, the specific processing process of the spatial fusion module is as follows:

[0040] 1) Group the infrared and visible light sent into the spatial fusion module according to the level, and perform channel dimension splicing operation respectively;

[0041] 2) send the spliced features of each layer into the feedforward neural network, selectively enhance the required features, and generate fusion features.

[0042] Step three: group the second, third and fourth layer features of the fusion features into the pixel-image block mutual attention matching module, and obtain three layer preliminary outputs and

[0043] Specifically, the pixel-image block mutual attention matching module contains the following:

[0044] 1) After the support and query fusion features and are straightened in the spatial dimension, linear mapping is performed, and matrix calculation is performed as query elements and key elements respectively, outputting a large score matrix with the shape of head×hw×hw;

[0045] 2) sequentially unfolding the last two dimensions of the obtained large score matrix, and performing corresponding matrix transformation, to obtain independent small score matrices M attn,Q and M attn,S ;

[0046] 3) independently sending M attn,Q and M attn,S into the pixel-image patch pooling module to obtain new small score matrices M' attn,Q and M' attn,S ;

[0047] 4) performing corresponding matrix transformation on the new small score matrices M' attn,Q and M' attn,S related to the query component and the support component respectively, adding the outputs to obtain a score matrix M' that can realize object modeling of different scales, and sending it into the normalization exponential function Softmax, and performing matrix multiplication with the true value mask of the flattened support image to finally output the preliminary output described in step 2

[0048] The matrix transformation performed after unfolding the large score matrix is specifically that, when generating the independent small score matrix M attn,Q related to the query component, the second dimension hw of the large score matrix is disassembled to obtain a matrix with a shape of head x h x w x hw, and then the fourth dimension is inserted into the second dimension to obtain head x hw x h x w. Conversely, when generating the independent small score matrix related to the support component, the third dimension is disassembled, and finally the same independent small score matrix with a shape of head x hw x h x w is obtained.

[0049] The pixel-image patch pooling module is composed of two serial 3x3 average pooling layers containing residual connections. After passing through the pixel-image patch pooling module, object feature modeling of different scales in the score matrix can be realized.

[0050] The matrix transformation performed on the new small score matrix is specifically that, taking the operation on the new small score matrix M' attn,Q related to the query component as an example, the third and fourth dimensions are spliced to generate a matrix with a shape of head x hw x hw, and then the second and third dimensions are exchanged. Conversely, to operate the new small score matrix related to the support component, only the third and fourth dimensions need to be spliced.

[0051] Step four: down-sampling the query image of the depth modality according to the size of the preliminary output obtained in step three to obtain The initial output corresponding to the size is sent to the prior fusion module (PF) constructed by Transformer, and the output represents the output of the depth information. The prior fusion module is constructed by Transformer. Specifically, and After straightening, they are fed into the mutual attention module as query elements and key / value elements respectively to generate output representing the depth information

[0052] Step 5: Different levels of Fusion is performed to obtain a coarse mask M guided by the cross-scale mutual attention mechanism output c , and combine it with the fusion query features of layer 1 and layer 2 The prediction P of the query image is obtained by splicing in the channel dimension and feeding it into the decoder. The outputs of different levels are fused by feeding them into the convolutional blocks in descending order of hierarchy, and then splicing them with the output of the previous level after upsampling. The decoder module is constructed using a convolutional neural network, and the output of its last layer should be 1×h×w, where 1 represents the number of channels required for the prediction mask, and the value range is 1 or 0, indicating whether the predicted class is included or not, respectively. h is the height of the input image, and w is the width of the input image.

[0053] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. It is apparent to those skilled in the art that various changes, modifications, substitutions, and variations to these embodiments may be made without departing from the principles and spirit of the present invention, and these changes and modifications still fall within the scope of protection of the present invention.

Claims

1. A cross-scale feature matching method for multimodal few-shot semantic segmentation, characterized by: The following steps are involved: Step 1: Send the support and query images of the visible light modality and infrared modality to the feature extraction module to extract the query features F of the visible light modality respectively. RGB,Q , infrared modality query feature F T,Q , support features F of visible light modality RGB,S and the infrared modality support feature F T,S ; Step 2: The extracted infrared and visible light features are grouped according to the source of the collection and sent to the spatial fusion module built based on the convolutional neural network to generate the fusion feature F E,Q and F E,S ; Step 3: Group the 2nd, 3rd, and 4th layer features of the fusion feature into the pixel-image block mutual attention matching module to obtain the three-layer preliminary output and Step 4: Downsample the query image of the depth modality according to the size of the preliminary output obtained in step 3 to obtain The initial output corresponding to the size is sent to the prior fusion module constructed by Transformer to output the output representing the depth information Step 5: Different levels of Fusion is performed to obtain a coarse mask M guided by the cross-scale mutual attention mechanism output c , and combine it with the fusion query features of layer 1 and layer 2 Splice in the channel dimension and send it to the decoder to obtain the prediction P of the query image; The spatial fusion module in step 2 specifically includes: The infrared and visible light sent to the spatial fusion module are grouped according to the level and spliced ​​in the channel dimension respectively; the spliced ​​features of each layer are sent to the feedforward neural network to selectively enhance the required features and generate fusion features; The pixel-image block mutual attention matching module in step 3 specifically includes: The input support and query fusion features and After straightening the spatial dimensions, perform linear mapping and use them as query elements and key elements for matrix calculation, outputting a large score matrix with a shape of head×hw×hw; Expand the last two dimensions of the obtained large score matrix in turn and perform corresponding matrix changes to obtain independent small score matrices M related to the query component and support component respectively. attn,Q and M attn,S ; M attn,Q and M attn,S Independently input into the pixel-image block pooling module to obtain a new small score matrix M' that integrates multi-scale features attn,Q and M' attn,S ; The new small score matrix M' associated with the query component and the support component respectively attn,Q and M' attn,S After the corresponding matrix transformation, the outputs are added to obtain the score matrix M' that can achieve modeling of objects of different scales, and it is sent to the normalized exponential function Softmax and compared with the true value mask of the straightened support image. Perform matrix multiplication and finally output the preliminary output described in step 2 The prior fusion module in step 4 is constructed by Transformer and specifically includes: and After straightening, they are fed into the mutual attention module as query elements and key / value elements respectively to generate output representing the depth information 2. The cross-scale feature matching method for multimodal few-shot semantic segmentation according to claim 1, characterized in that: The feature extraction module in step 1 is ResNet50 pre-trained with the ImageNet dataset, and includes 5 ResNet blocks.

3. The cross-scale feature matching method for multimodal few-shot semantic segmentation according to claim 1, characterized in that: The step 2 of grouping the extracted infrared and visible light features according to the source of the collection specifically includes: dividing the extracted infrared and visible light features into two groups according to the source of the collection, namely the infrared and visible light features extracted by the query branch and the infrared and visible light branches extracted by the support branch.

4. The cross-scale feature matching method for multimodal few-shot semantic segmentation according to claim 1, characterized in that: The matrix changes after the large score matrix is ​​expanded specifically include generating independent small score matrices M related to the query components: attn,Q When generating the independent small score matrix related to the support component, the hw of the third dimension of the large score matrix is ​​split, and finally the independent small score matrix with the shape of head×hw×h×w is obtained.

5. The cross-scale feature matching method for multimodal few-shot semantic segmentation according to claim 1, characterized in that: The pixel-image block pooling module is composed of two serial average pooling layers with residual connections.

6. The cross-scale feature matching method for multimodal few-shot semantic segmentation according to claim 1, characterized in that: The matrix transformation of the new small score matrix specifically includes: operating the new small score matrix M' related to the query component attn,Q For example, after concatenating the 3rd and 4th dimensions to generate a matrix of shape head×hw×hw, the 2nd and 3rd dimensions are swapped to operate the new small score matrix related to the support component, and only the 3rd and 4th dimensions need to be concatenated.

7. The cross-scale feature matching method for multimodal few-shot semantic segmentation according to claim 1, characterized in that: The decoder in step 5 is constructed by a convolutional neural network, and its last layer output should be 1×h×w, where 1 represents the number of channels required for predicting the mask, and the value range is 1 or 0, indicating whether the predicted class is included or not, respectively. h is the height of the input image, and w is the width of the input image.

Images