A small sample semantic segmentation method based on visual large model multi-scale prompt

Abstract

The application discloses a small sample semantic segmentation method based on a visual large model and multi-scale prompts, and steps are as follows: constructing the same kind of support query sample pair; acquiring support fusion features, query fusion features and visual priori of the query image; modeling and down-sampling the support fusion features and the query fusion features, and extracting multi-scale support features and query features; generating multi-scale prompt encoding and target prompt encoding by using the multi-scale support features and the query features; inputting the target prompt encoding and the query image into a prompt type visual large model to acquire a segmentation result of the query image; inputting the support query sample pair in the small sample data into a trained prediction model to segment the query image, and calculating a segmentation accuracy according to a prediction result. Compared with the prior art, the application can fully utilize semantic information under multi-scale, the generated prompt information is more accurate, and the precision of the small sample semantic segmentation task can be further improved in various scenes.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision semantic segmentation, and particularly relates to a small sample semantic segmentation method based on visual large model multi-scale prompting. BACKGROUND

[0002] The neural network model based on deep learning has achieved remarkable results in many fields, but its success is based on a large amount of labeled data. For those classes that are rarely included or not included in the training data, the effect of the deep neural network is often greatly reduced, and the generalization ability for new classes is often not very strong. For downstream tasks, it is usually necessary to collect data again, and the original model is retrained or fine-tuned using the collected related data. However, collecting a large amount of labeled data is very costly, especially for dense prediction tasks such as semantic segmentation, the cost is even higher, and a lot of time is also needed, which greatly limits the application of deep learning models. In order to alleviate the dependence of deep learning network on data, small sample learning aims to train a network that can be well generalized to new classes with scarce labeled training data. Only a small amount of labeled samples can complete the specified task, in other words, it is urgent to train a network with learning ability, which can learn a new concept through a small amount of learning materials, in order to alleviate the problem of over-reliance on training data of the current deep learning network.

[0003] The main method of small sample semantic segmentation model is to complete the matching of pixel position features through prior knowledge. The current small sample semantic segmentation method mainly obtains the general ability for all classes based on the pre-trained backbone network, so the prior for the downstream segmentation task is not necessarily optimal. Therefore, by introducing a visual large model for the segmentation task, the segmentation accuracy can be further improved.

[0004] There are mainly two methods for improving the small sample semantic segmentation accuracy using a visual large model. One is to use the preliminary segmentation result of the target image as prior knowledge and introduce it into the model for use. The other is to generate a visual prompt for the current target and input it into the visual large model to complete the segmentation task. SUMMARY

[0005] The main purpose of the application is to improve the segmentation accuracy of target objects by using the general ability of the visual large model, and to provide a small sample semantic segmentation method based on visual large model multi-scale prompting.

[0006] In order to achieve the above purpose, the technical scheme adopted by the application is as follows:

[0007] A small sample semantic segmentation method based on visual large model multi-scale prompting, the small sample semantic segmentation method comprising the following steps:

[0008] S1, construct a same-class support query sample pair using a semantic segmentation public dataset, the same-class support query sample pair comprising a query image , a support image , and a support mask ;

[0009] S2, obtain support fusion features and query fusion features respectively by fusing a feature extraction module , and obtain visual priors of the query image, wherein the fusion feature extraction module comprises a backbone network, a pair of parallel feature fusion convolution layers, and a pair of parallel feature activation convolution layers connected in sequence;

[0010] S3, a multi-scale feature extraction module models and down-samples the support fusion features and the query fusion features respectively, and extracts multi-scale support features and query features respectively;

[0011] S4, a multi-scale hint generation module generates multi-scale hint encodings using the multi-scale support features and the query features, fuses the multi-scale hint encodings from bottom to top, and generates a target hint encoding; wherein the step S4 is as follows:

[0012] S41, input the query features and the support features of the current scale, obtain a similarity map of the query features and the support features using cosine similarity, adjust the similarity map using an intermediate result of the next scale, aggregate a learnable hint vector input using the adjusted similarity map, and generate a similarity hint encoding of the current scale;

[0013] S42, fuse the similarity hint encoding of the current scale with the hint encoding of the next scale and the intermediate result, generate a hint encoding of the current scale, and output the hint encoding and the intermediate result generated therefrom to the previous scale for continuous iteration until the target hint encoding is generated;

[0014] S5, input the target hint encoding and the query image into a hint-based visual large model SAM to obtain a segmentation result of the query image, wherein the visual large model SAM comprises an encoder and a decoder, the query image is input into the encoder to obtain a query image encoding, and the target hint encoding and the query image encoding are input into the decoder to obtain the segmentation result; construct a prediction model comprising a fusion feature extraction module , a multi-scale feature extraction module , a multi-scale hint generation module , and the visual large model SAM in sequence, use an intermediate result and a final result to form a total loss function, train the prediction model, and keep the parameters of the visual large model SAM in a frozen state during the training process;

[0015] S6, input the support query sample pair in the small sample data into the trained prediction model, segment the query image, and calculate the segmentation accuracy according to the prediction result.

[0016] Further, the process of constructing the same type of support query image sample pair in step S1 is as follows:

[0017] The public data set of semantic segmentation is divided into four groups according to the category, of which three groups of training set are used for training, and the remaining one group of test set is used for testing. In each group of data, the training set and the test set respectively contain the support set and the query set. The support set is composed of support images and their corresponding dense annotation masks, and the query set is composed of single images of the same type and their masks. The dense annotation mask of the support image in the support set is used as prior knowledge. In the training process, the dense annotation mask of the query image is used to supervise the learning of the prediction network to optimize the network parameters. In the test stage, the dense annotation mask of the query image is used as a standard to evaluate the performance of the model.

[0018] Further, the process of obtaining the support fusion feature and the query fusion feature and the query visual prior in step S2 is as follows:

[0019] S21, use a pre-trained ResNet-50 as a backbone network. ResNet is a deep residual network, which is a deep learning algorithm that uses cross-layer connection to obtain residual error. ResNet-50 is a specific implementation of deep residual network, which has been widely used in image classification and segmentation. For specific details, please refer to the paper K.M. He, X.Y. Zhang, S.Q. Ren, SQ, and J. Sun, Deep Residual Learning for Image Recognition [C], in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2016, pp. 770-778. Select the output of the 2nd and 3rd layers in ResNet-50 as the intermediate layer feature, and the output of the 4th and 5th layers as the high layer feature. Because ResNet-50 is pre-trained based on the classification task, the size of the intermediate layer feature is larger and contains more semantic information, while the size of the high layer feature is smaller and is beneficial for classification tasks. For dense segmentation tasks, directly using high layer features cannot improve performance.

[0020] Respectively input the query image and the support image into the backbone network to obtain the feature:

[0021]

[0022]

[0023] wherein, denotes a pre-trained backbone network, , denote query feature and support feature of the i-th intermediate layer output respectively, denote the channel number of the i-th intermediate layer feature, denote the height of the i-th intermediate layer feature, denote the width of the i-th intermediate layer feature; respectively input the query intermediate layer feature and the support intermediate layer feature to the respective feature fusion convolutional layer for dimension reduction and feature fusion, to obtain the query intermediate feature and the support intermediate feature

[0024] ; generate visual prior using high-level features and , first adjust the size of the support mask to be consistent with the support intermediate feature

[0025] using the bilinear interpolation method, and remove irrelevant background features in the high-level support feature using the Hadamard product method: wherein, is the binary mask of the support image, and “⊙” denotes the Hadamard product; the bilinear interpolation is a mathematical method that considers the four pixel values closest to the interpolation point, assigns different weights to each point according to the distance between the interpolation point, and then performs linear interpolation;

[0026] wherein, is the binary mask of the support image, and “⊙” denotes the Hadamard product; the bilinear interpolation is a mathematical method that considers the four pixel values closest to the interpolation point, assigns different weights to each point according to the distance between the interpolation point, and then performs linear interpolation;

[0027] use the method of calculating the cosine similarity of each pixel position of the query feature and the support feature to explore the association between pixels :

[0028] wherein “ ” denotes the 3-dimensional vector inner product operation at all positions of the feature, represents the L2 norm;

[0029] for each position of the query feature, take the maximum similarity value as the prior value of the current position, and normalize the value to 0 to 1 using the min-max value normalization method to obtain the visual prior ;

[0030] ​​Visual prior is essentially a rough segmentation result of the target object in the query image. Using visual prior can make each module pay more attention to the region where the target object may exist, thereby improving the segmentation accuracy.

[0031] S22. Use bilinear interpolation to apply the support mask. Size adjustment to support intermediate features Consistent, the prototype vector of the current category is obtained using masked average pooling:

[0032]

[0033] in, This is the prototype vector of the current category. Indicates support for intermediate features In position eigenvalues, To support the mask In position The labeled value, It is a conditional function; if the value in parentheses is true, it takes the value 1; otherwise, it takes the value 0.

[0034] Query intermediate features prototype vector and visual prior The data is concatenated along the channel dimension and then passed through a query feature activation convolutional layer to obtain the query fusion feature. It will support intermediate features. prototype vector Concatenate along the channel dimension, then pass through a support feature activation convolutional layer to obtain support fusion features. .

[0035] Furthermore, the process by which the multi-scale feature extraction module extracts multi-scale features using fused features in step S3 is as follows:

[0036] The multi-scale feature extraction module includes Layered attention blocks, with each layer of attention blocks connected sequentially, and the output of each attention block being feature information at a single scale. The outputs of the layer attention blocks are combined to form a multi-scale feature pyramid; each attention block includes a local attention block LocalAttnBlock and a self-attention block SelfAttnBlock connected in sequence. The formal representation of a single layer is as follows:

[0037] in, Indicates the current floor number. It is the first Features of layer output, , is the first layer output feature, and has , , , represents the number of channels of the first layer pyramid feature, represents the height of the first layer pyramid feature, represents the width of the first layer pyramid feature;

[0038] The local attention block is good at local modeling, and the global modeling ability is weak; the advantage of the self-attention block is global modeling, but the local modeling ability is weak; by combining the local attention block and the self-attention block, the advantages can be complementary, so that the multi-scale feature extraction module is excellent in local modeling and global modeling;

[0039] Each local attention block is essentially a multi-branch convolution layer, which obtains feature information under different receptive fields through different size convolution kernels; the basic operation of each self-attention block is attention mechanism :

[0040]

[0041] wherein, is a query vector, is a key vector, is a value vector, is a query feature, is a key feature, is a value feature, is a corresponding projection matrix, is a value of feature dimension; if , , are the same feature, it is called self-attention, and if they are different, it is mutual attention; The operation is applied to each row of the matrix, which is used to convert the attention score into a probability distribution, , is an input matrix, is a matrix whose value at position ;

[0042] The query fusion feature and the support fusion feature are respectively input into the multi-scale feature extraction module to obtain the multi-scale query feature pyramid and the multi-scale support feature pyramid , wherein, represents the query feature pyramid characteristics of layers, representing a support feature pyramid characteristics of layers;

[0043] The characteristics of each layer in the feature pyramid are different in size, so the characteristics at each scale save different semantic information. Specifically, the characteristics of the lower layer are larger in size and can save more local detail information, and the characteristics of the higher layer are smaller in size and contain more abstract semantic information. The rich semantic information at multiple scales can be used to construct more accurate prompt information.

[0044] Further, the multi-scale prompt generation module in the step S4 comprises a layer-wise attention prompt encoding generation block, the output of each layer being a prompt encoding at the current scale, and each attention prompt encoding generation block first mines similar information between the characteristics at the current scale according to cosine similarity, and then aggregates the prompt encoding at the next scale and the intermediate result to generate the prompt encoding at the current scale;

[0045] The attention prompt encoding generation blocks at different layers input feature information at different scales, so the similar prompt encodings generated by the attention prompt encoding generation blocks at different layers focus on different aspects. Specifically, the similar prompt encodings generated by using the characteristics with a larger size focus more on local details and are directed to regions with more distinguishability in the target object rather than the overall region, and the granularity of the prompt is smaller. The similar prompt encodings generated by using the characteristics with a smaller size focus more on abstract information and generate more rough prompt information, but some details may be missing, and the granularity of the prompt is larger. Using the top-down optimization method can make the focus of the generated prompt encoding gradually transition from the overall to the local and gradually deepen from the abstract to the details.

[0046] Further, the process of generating the similar prompt encoding at the current scale in the step S41 is as follows:

[0047] For the first characteristics of layers and , the shapes of the characteristics and are first reshaped, the shape of is reshaped into , and the reshaped characteristics and are obtained, respectively; because the prompt encoding should only be related to the similarity between the query characteristics and the support characteristics, and not directly related to a single characteristic, and the data distribution of the prompt encoding is relatively independent of the data distribution of the query characteristics and the support characteristics, a set of learnable prompt vectors is introduced to decouple the data distribution between the prompt encoding and the characteristics, and similarly, the prompt vectors Shape reshaping Calculate the query vector in the attention mechanism respectively Key vector Value vector , obtain the query vector Key vector Sum value vector , where the query vector From query features key vector From supporting features value vector Learnable cue vectors derived from a set of inputs The formal representation is as follows:

[0048] in, , , They are respectively , , The projection matrix;

[0049] To obtain the similarity relationship between each pixel location, calculate and The cosine similarity between them yields a similarity graph matrix. :

[0050] in," " indicates the vector dot product operation. The L2 norm of a vector. For temperature hyperparameters, the following can be taken: ;

[0051] Along The first dimension The operation yields a normalized similarity graph matrix. :

[0052]

[0053] in, For matrix In position The value, For matrix In position The value;

[0054] The support mask is obtained by using bilinear interpolation. Adjust to the corresponding size and reshape to obtain the desired result. ,use By removing the similarity values ​​at irrelevant background locations, we obtain the similarity matrix after masking the background. ;

[0055] Under multi-scale conditions, the mask scores output by the next layer are used as a similarity prior to adjust the attention map and progressively optimize the segmentation results. The formal representation is as follows:

[0056] in, For the first The layer similarity prior is obtained by adjusting the size of the segmentation mask logits score output by the next layer through bilinear interpolation. A logits score greater than 0 indicates foreground and less than 0 indicates background. The larger the absolute value, the greater the probability. It is a combination of matrix flattening and dimension expansion copying operations, first compressing the vector into a one-dimensional shape. Then, it is copied along the second dimension until its shape matches... Consistent;

[0057] Using the adjusted similarity matrix Aggregate the learnable cue vectors of the input to obtain the first... Layer similarity hint encoding ;

[0058] By proactively introducing similarity priors from the next scale to adjust the similarity matrix, the generation of similarity cue encoding incorporates key information relevant to the next scale, paying closer attention to key regions that may contain the target object; the addition operation embodies the similarity matrix... and similar priors The equal relationship is similar to a "voting" mechanism, which judges the final similarity by combining the results of the similarity matrix and similarity priors, thereby generating a more accurate similarity hint code.

[0059] Furthermore, the process of generating the cue code at the current scale and continuing to iterate until the target cue code is generated in step S42 is as follows:

[0060] S421. If the current layer is the last layer, that is... Then, the similarity cue encoding at the current scale will be used directly. As a prompt code under the current scale ,Right now = The prompt code After being resized using bilinear interpolation, the data is directly input into the decoder of the large visual model (SAM) to obtain the intermediate segmentation score of the last layer. , As a similarity prior input, it is used to input the first In the layer cross-attention prompt encoding generation block, the prompt encoding is also input to the first layer for iterative optimization.

[0061] S422, if the current layer is not the last layer, i.e. the prompt encoding of the next layer needs to be aggregated, and the optimization is performed step by step from bottom to top, and first the prompt encoding of the first layer is obtained. The prompt encoding is up-sampled to a shape consistent with the scale of the current layer by using bilinear interpolation, and the similar prompt encoding of the current layer is obtained. The similar prompt encoding is concatenated together and input to a fusion layer for information fusion to obtain the prompt encoding at the current scale. After resizing by bilinear interpolation, the prompt encoding is input to the decoder of the visual large model SAM to obtain the segmentation intermediate result score of the current layer. The segmentation intermediate result score is input to the first layer cross-attention prompt encoding generation block as similar prior, and the prompt encoding is obtained. The prompt encoding is also input to the first layer for iterative optimization.

[0062] The fusion layer is essentially a multi-layer perception MLP, which is two linear layers connected in sequence:

[0063]

[0064] wherein, is a nonlinear activation function, which can be selected from activation functions, , and are constants. and are weight parameters of the MLP.

[0065] The step S422 is repeated, and after iterative optimization of the layer, the prompt encoding output by the first layer is obtained.The prompt encoding output by the first layer is obtained by aggregating the prompt encoding at multiple scales and iterative optimization through the intermediate results of the subsequent layers, fully utilizing the semantic information of different granularities at multiple scales as the target prompt encoding.

[0066] Further, the process of inputting the query picture and the final prompt encoding into the prompt-based visual large model SAM to obtain the segmentation result, constructing a prediction model and training in step S5 is as follows:

[0067] S51, input the query image into the encoder of the visual large model SAM to obtain the query image code, and then output the segmentation result of the query image according to the prompt code and the query image code; input the multi-scale prompt code generated in step S4 and the query image code into the decoder of the visual large model SAM to generate a multi-scale segmentation result ; select the last prediction output result , and the remaining intermediate results are input into the multi-scale prompt generation module ;

[0068] SAM is a "segment everything model" that can provide general image segmentation capability, and the core goal is to be able to segment objects in any type of image without the need for specific training data sets or task adjustments; SAM mainly includes a picture encoder and a decoder, and uses a new prompt method to accurately segment target objects in images, but SAM itself does not have specific semantic information and only has general segmentation capability, so specific prompt information needs to be provided for downstream tasks such as small sample semantic segmentation; SAM is widely used in many segmentation or target detection tasks, and specific details can be referred to in the paper A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, et al., "Segment anything," in Proceedings of the IEEE / CVF International Conference on Computer Vision, 2023, pp. 4015-4026; the present application uses a specific implementation SAMv2 of SAM as a basic model;

[0069] S52, the fusion feature extraction module , the multi-scale feature extraction module , the multi-scale prompt generation module and the visual large model SAM are used to build a prediction model; the final prediction result and the intermediate result are used to build a total loss function , the prediction model is trained with the loss function as the optimization target, and the loss function is calculated as follows:

[0070] wherein, The total cross-entropy loss is used to evaluate the classification accuracy of the foreground and background in the output results, and is used to continuously optimize intermediate and final prediction results. , To query the mask of the image, the cross-entropy loss function , It's a real label. It is the predicted probability; To improve the similarity matrix distillation loss between different layers of the multi-scale cue generation module, the similarity relationships between high-level abstract features are made as close as possible to the similarity relationships between low-level details. KL divergence loss function , Is it the true distribution? right The probability, It is to estimate the distribution. right The probability, For the first The similarity vector is obtained by performing masked average pooling and normalization operations along the first dimension on the similarity matrix of the layer; The DICE loss for the final prediction result is related to a generalized definition of intersection-union ratio (IU), which can more directly optimize the segmentation result. , pixel position The probability value of the predicted outcome can be taken as . , , The actual value of the pixel location label; , , These are the weight hyperparameters for each loss;

[0071] Through total cross-entropy loss Distillation loss and DICE losses The combination of these methods allows the output target cue encoding to fully utilize semantic information at different granularities across multiple scales, resulting in more accurate cue information.

[0072] Furthermore, the process in step S6 is as follows:

[0073] S61. Based on the trained prediction model, the segmentation result of the query image is obtained through the forward propagation of the network; if the final segmentation mask logits score is greater than 0, it is considered as foreground, otherwise it is predicted as background.

[0074] S62. Calculate the segmentation accuracy based on the prediction results, and use the average intersection-union ratio (mIoU) as the evaluation index of segmentation accuracy.

[0075] mIoU is an evaluation index widely used in semantic segmentation tasks, especially in multi-class segmentation, because it balances the performance of each class regardless of class frequency; mIoU is the average of the intersection over union (IoU) of all classes, which can comprehensively reflect the segmentation performance of the model on known classes; the calculation formula is as follows:

[0076]

[0077]

[0078] wherein, is the number of true positives of class , that is, the correctly segmented pixels; is the number of false positives of class , that is, the pixels misclassified as class is the number of false negatives of class , that is, the pixels that are actually class is the total number of classes.

[0079] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0080] (1) The present application proposes a method of constructing a multi-scale feature extraction module by combining a local attention block with a self-attention block, which realizes complementary advantages and makes the multi-scale feature extraction module perform well in both local modeling and global modeling, and the generated multi-scale feature pyramid can retain more rich and accurate semantic information.

[0081] (2) The multi-scale prompt generation module designed by the present application is a bottom-up iterative network structure, which can use the intermediate prediction results of the next layer as similar priors to optimize the results of the current layer, and can fuse the prompt codes from coarse granularity to fine granularity, gradually transition the focus of the prompt codes from the whole to the local, and gradually deepen from abstraction to details, improving the accuracy of the target prompt code.

[0082] (3) The method proposed by the present application combines the visual large model by generating target prompt codes, fully utilizes the prior knowledge and general ability of the visual large model, improves the generalization ability to new classes not seen in the training process, and can effectively alleviate the problem of degradation of deep learning generalization performance; compared with other prompt generation methods, the target prompt code generated by the present application has better explainability, and the information of each position of the target prompt code can be one-to-one corresponding to the prediction result, which is more convenient for optimizing the result.

[0083] (4) The method for multi-scale prompting based on a visual large model can continuously utilize intermediate results for iterative optimization in a small sample semantic segmentation task, can achieve better performance with less iteration training times, and improves the accuracy of small sample semantic segmentation. BRIEF DESCRIPTION OF DRAWINGS

[0084] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0085] Figure 1 The flow chart of the small sample semantic segmentation method based on the multi-scale prompting of the visual large model disclosed in the present application is shown in

[0086] Figure 2 The structure of the multi-scale feature extraction module and the multi-scale prompting generation module in the embodiment of the present application and the connection structure therebetween are shown in

[0087] Figure 3 The training flow chart in the embodiment of the present application is shown in DETAILED DESCRIPTION

[0088] In order to make the person skilled in the art better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0089] In the present application, "embodiment" means that the specific features, structures or characteristics described in conjunction with the embodiment can be included in at least one embodiment of the present application. The appearance of this phrase in the specification does not necessarily mean the same embodiment, nor is it an independent or alternative embodiment to other embodiments. Those skilled in the art explicitly and implicitly understand that the embodiments described in the present application can be combined with other embodiments.

[0090] Embodiment 1

[0091] Figure 1 The flow chart of the small sample semantic segmentation method based on the multi-scale prompting of the visual large model disclosed in the present application is shown in Figure 1 The method comprises the following steps:

[0092] S1, construct a same-class support query sample pair using a semantic segmentation public dataset, the same-class support query sample pair comprising a query image , a support image , and a support mask ;

[0093] S2, obtain support fusion features and query fusion features respectively by fusing a feature extraction module , and obtain visual priors of the query image, wherein the feature extraction module comprises a backbone network, a pair of parallel feature fusion convolution layers, and a pair of parallel feature activation convolution layers connected in sequence;

[0094] S3, a multi-scale feature extraction module models and down-samples the support fusion features and the query fusion features respectively, and extracts multi-scale support features and query features respectively;

[0095] S4, a multi-scale hint generation module generates multi-scale hint encodings using the multi-scale support features and query features, fuses the multi-scale hint encodings from bottom to top, and generates a target hint encoding; wherein the process of step S4 is as follows:

[0096] S41, input the query features and the support features of the current scale, obtain a similarity map of the query features and the support features using cosine similarity, adjust the similarity map using an intermediate result of the next scale, aggregate a learnable hint vector input using the adjusted similarity map, and generate a similarity hint encoding of the current scale;

[0097] S42, fuse the similarity hint encoding of the current scale with the hint encoding of the next scale and the intermediate result, generate a hint encoding of the current scale, and output the hint encoding and the intermediate result generated using the hint encoding to the previous scale for continuous iteration until the target hint encoding is generated;

[0098] S5, input the target hint encoding and the query image into a hint-based visual large model SAM to obtain a segmentation result of the query image, wherein the visual large model SAM comprises an encoder and a decoder, the query image is input into the encoder to obtain a query image encoding, and the target hint encoding and the query image encoding are input into the decoder to obtain the segmentation result; construct a prediction model comprising the feature extraction module , the multi-scale feature extraction module , the multi-scale hint generation module , and the visual large model SAM in sequence, use an intermediate result and a final result to form a total loss function, train the prediction model, and keep the parameters of the visual large model SAM in a frozen state during the training process;

[0099] S6, input the support query sample pairs in the small sample data into the trained prediction model, segment the query image, and calculate the segmentation accuracy according to the prediction result.

[0100] In step S1 of this embodiment, the semantic segmentation public data set is evenly divided into 4 groups according to the category, 3 groups are used as the training set, and the remaining 1 group is used as the test set, and a total of 4 different combinations of training set and test set can be established; for the PASCAL data set, the training set contains 15 categories, and the test set contains 5 categories, and the test set contains 1000 support query sample pairs; in the 1-shot small sample setting, each support query sample pair contains 1 query image , 1 support image and 1 corresponding support mask ; the size of each query image and support image is ;

[0101] In step S2 of this embodiment, the process of obtaining support fusion features and query fusion features and query visual prior is as follows:

[0102] S21, use the pre-trained ResNet-50 as the backbone network; select the output of the 2nd and 3rd layers in ResNet-50 as the intermediate layer features, and the output of the 4th and 5th layers as the high layer features, respectively input the query image and the support image into the backbone network to obtain the features:

[0103]

[0104]

[0105] wherein, denotes the pre-trained backbone network, , denote the query features and support features of the layer output, respectively, denotes the channel number of the layer feature, denotes the height of the layer feature, denotes the width of the layer feature;

[0106] respectively input the query intermediate layer feature and the support intermediate layer feature to the respective feature fusion convolutional layer for dimension reduction and feature fusion, to obtain the query intermediate feature and the support intermediate feature This pair of parallel feature fusion convolutional layers has the same structure, and the kernel size is the same. The number of input channels is 1536 and the number of output channels is 256.

[0107] Utilizing high-level features and To generate visual priors, a support mask is first generated using bilinear interpolation. Size adjustment to support intermediate features Consistent, using Hadamard product to remove high-level support features. Unrelated background features:

[0108] in, To support binary masks for images, "⊙" represents the Hadamard product;

[0109] The method of calculating the cosine similarity between the query features and the supporting features at each pixel position is used to discover the correlation between pixels. :

[0110] in" "" indicates the inner product operation of 3D vectors at all positions of the feature. Represents the L2 norm;

[0111] For each location's query feature, the maximum similarity value is taken as the prior value for the current location. A minimum-maximum normalization method is then used to normalize this value to between 0 and 1, yielding the visual prior. ;

[0112] S22. Use bilinear interpolation to apply the support mask. Size adjustment to support intermediate features Consistent, the prototype vector of the current category is obtained using masked average pooling:

[0113] in, This is the prototype vector of the current category. Indicates support for intermediate features In position eigenvalues, To support the mask In position The labeled value, It is a conditional function; if the value in parentheses is true, it takes the value 1; otherwise, it takes the value 0.

[0114] Query intermediate features prototype vector and visual prior The data is concatenated along the channel dimension and then passed through a query feature activation convolutional layer to obtain the query fusion feature. The kernel size of the convolutional layer activated by the query feature is The input channel count is 514, and the output channel count is 64; it will support intermediate features. prototype vector Concatenate along the channel dimension, then pass through a support feature activation convolutional layer to obtain support fusion features. The kernel size of the convolutional layer that supports feature activation is [size missing]. The number of input channels is 512, and the number of output channels is 64.

[0115] In step S3 of this embodiment, the process by which the multi-scale feature extraction module extracts multi-scale features using fused features is as follows:

[0116] like Figure 2 As shown, the multi-scale feature extraction module contains three attention blocks, each connected sequentially. The output of each attention block is the feature information at a single scale. The outputs of the three attention blocks are combined to form a multi-scale feature pyramid. Each attention block includes one locally-attention block (LocalAttnBlock) and one self-attention block (SelfAttnBlock) connected sequentially. The formal representation of a single layer is as follows:

[0117] in, Indicates the current floor number. , It is the first Features of layer output, , It is the first Features of layer output, And there are , , , Indicates the first The number of channels characteristic of a multi-tiered pyramid. , Indicates the first The high height of the pyramid features , Indicates the first The width of the pyramid features ;

[0118] Each local attention block is essentially a multi-branch convolutional layer that uses convolutional kernels of different sizes to acquire feature information from different receptive fields; the basic operation of each self-attention block is the attention mechanism. :

[0119]

[0120] wherein, is a query vector, is a key vector, is a value vector, is a query feature, is a key feature, is a value feature, is a corresponding projection matrix, is a value of a feature dimension; if , , is the same feature, it is called self-attention, and if it is different, it is mutual attention; the operation is applied to each row of the matrix to convert the attention score into a probability distribution, , is an input matrix, is a matrix is the value at position .

[0121] respectively input the query fusion feature and the support fusion feature into the multi-scale feature extraction module to obtain the multi-scale query feature pyramid and the multi-scale support feature pyramid wherein, denotes the feature of the layer of the query feature pyramid, denotes the feature of the layer of the support feature pyramid.

[0122] The multi-scale prompt generation module in step S4 of the embodiment contains 3 layers of mutual attention prompt encoding generation blocks, as shown in Figure 2 , the output of each layer is the prompt encoding at the current scale, and each mutual attention prompt encoding generation block first mines the similarity information between the features at the current scale according to the cosine similarity, then aggregates the prompt encoding at the next scale and the intermediate result to generate the prompt encoding at the current scale; specifically including the following steps:

[0123] Step S41, for the features and at the layer, first reshape the shapes of the features and , reshape the shape of to , respectively to obtain the reshaped features and ; introduce a set of learnable prompt vectors , similarly reshape the prompt vectors to obtain , respectively, to obtain query vector , key vector , and value vector , where query vector comes from query feature , key vector comes from support feature , and value vector comes from a set of input learnable prompt vectors , which is formalized as follows:

[0124]

[0125] where , , are projection matrices of , , respectively;

[0126] The cosine similarity between and is calculated to obtain a similarity graph matrix :

[0127] where “ ” represents the inner product operation of vectors, represents the L2 norm of the vector, is a temperature hyperparameter, and ;

[0128] The first dimension of is operated along the direction to obtain the normalized similarity graph matrix :

[0129]

[0130] where is the value of matrix at position , and is the value of matrix at position ;

[0131] The support mask is adjusted to the corresponding size by using the bilinear interpolation method, reshaped to obtain , and the irrelevant background position similarity value is removed by using to obtain the similarity matrix after masking the background​​​​ ;

[0132] In the condition of multi-scale, the mask score output by the next layer is used as a similar prior to adjust the attention map, and the segmentation result is gradually optimized, which is formalized as follows:

[0133] Wherein, is the similar prior of the i-th layer, which is obtained by adjusting the size of the segmentation mask logits score output by the next layer through bilinear interpolation, and the logits score greater than 0 is foreground and less than 0 is background, and the greater the absolute value is, the greater the probability is; is the combination operation of matrix flattening operation and dimension expansion replication operation, which first compresses the vector into one-dimensional shape , and then expands and replicates along the second dimension until its shape is consistent with ;

[0134] The adjusted similar matrix is used to aggregate the input learnable prompt vector to obtain the similar prompt encoding of the i-th layer ; ;

[0135] Step S42, generate the prompt encoding of the current scale and continue iteration to generate the target prompt encoding, the specific process is as follows:

[0136] S421, if the current layer is the last layer, i.e. , then directly use the similar prompt encoding of the current scale as the prompt encoding of the current scale , that is = ; the prompt encoding is directly input into the decoder of the visual large model SAM after being adjusted in size by bilinear interpolation, to obtain the segmentation intermediate result score of the last layer as a similar prior input into the 2nd layer of mutual attention prompt encoding generation block, and the prompt encoding is also input into the i-th layer for iterative optimization;

[0137] S422, if the current layer is not the last layer, i.e. , then the prompt encoding of the next layer needs to be aggregated to optimize step by step from bottom to top, first, the prompt encoding of the i-th layer is up-sampled to the shape consistent with the scale of the current layer by using the bilinear interpolation method, and the prompt encoding of the current i-th layer is also input into the i-th layer for iterative optimization; ​​​Concatenate together and input into a fusion layer for information fusion to obtain prompt encoding at the current scale ; After resizing by bilinear interpolation, input into the decoder of the visual large model SAM to obtain the segmentation intermediate result score of the current layer , input it into the layer cross-attention prompt encoding generation block as similar prior, and the prompt encoding is also input into the layer for iterative optimization

[0138] The fusion layer is essentially a multi-layer perceptron (MLP), which is two linear layers connected in sequence:

[0139]

[0140] wherein, is a nonlinear activation function, and the activation function , and are constants and are weight parameters of the MLP

[0141] Repeat step S422, after 3 iterations, the prompt encoding output by the first layer is the target prompt encoding output.

[0142] In step S5 of the embodiment, the query image and the final prompt encoding are input into the prompt-based visual large model SAM to obtain the segmentation result, and the process of constructing and training the prediction model is as follows:

[0143] S51, select a specific implementation SAMv2 of SAM as the base model, as shown in Figure 1 , input the query image into the encoder of the visual large model SAM to obtain the query image encoding, and then output the segmentation result of the query image according to the prompt encoding and the query image encoding; input the multi-scale prompt encoding generated in step S4 into the decoder of the visual large model SAM together with the query image encoding to generate multi-scale segmentation results ; select as the final prediction output result , and the remaining intermediate results are input into the multi-scale prompt generation module as similar prior

[0144] S52, as shown in Figure 3 , the fusion feature extraction module , the multi-scale feature extraction module , the multi-scale prompt generation module and the visual large model SAM to construct a prediction model; according to the final prediction result and the intermediate result to construct a total loss function :

[0145] wherein, is the total cross-entropy loss, , is the mask of the query image, the cross-entropy loss function , is the real label, is the predicted probability; is the distillation loss, , the KL divergence loss function , is the real distribution probability of , is the estimated distribution probability of , is the similarity vector obtained by mask average pooling and normalization operation of the similarity matrix of the i-th layer along the first dimension; is the DICE loss of the final prediction result, , is the probability value predicted by the pixel position , , , , is the pixel position label real value; , , are the weight hyperparameters of each loss respectively, and are taken as , , ;

[0146] The prediction model constructed by taking the loss function as the optimization objective is trained, the number of training rounds is 60, and during the training process, the parameters of the visual large model SAM are kept frozen, and only the parameters in the fusion feature extraction module , the multi-scale feature extraction module and the multi-scale prompt generation module are updated, the training algorithm adopts the AdamW algorithm, the learning rate is set to , and the weight decay hyperparameter is set to .

[0147] In step S6 of the embodiment, the specific process is as follows:

[0148] S61, obtaining the segmentation result of the query image through forward propagation of the network based on the trained prediction model; if the final segmentation mask logits score is greater than 0, it is considered as foreground, otherwise it is predicted as background;

[0149] S62, calculating the segmentation accuracy according to the prediction result, using the mean intersection over union (mIoU) as the evaluation index of segmentation accuracy; mIoU is the average value of the intersection over union (IoU) of all classes, and the calculation formula is as follows:

[0150]

[0151]

[0152] wherein, is the number of true positives of class , that is, the correctly segmented pixels; is the number of false positives of class , that is, the pixels misclassified as class is the number of false negatives of class , that is, the pixels that are actually class is the total number of classes.

[0153] In this embodiment, the data set is the public data set PASCAL, and the images of a total of 20 classes are evenly divided into 4 parts, 3 of which are used as the training set, and the remaining 1 part is used as the test set. The small sample scenario is set to 1-shot, and each support query sample pair contains 1 query image , 1 support image and 1 support mask ; the proposed method is compared with various small sample semantic segmentation methods, and the compared methods are HSNet, BAM, HDMNet and SAM-RSP respectively; all methods use ResNet-50 network as the backbone network, and the segmentation results are shown in Table 1:

[0154] Table 1. Comparison table of experimental data (1-shot setting)

[0155]

[0156] Taking mIoU as the evaluation index, under the 1-shot setting, the average segmentation accuracy in various scenarios is 3.5% higher than that of the previous best method, and the performance is improved under each different data division.

[0157] Embodiment 2

[0158] The overall steps of this embodiment are consistent with the overall steps in embodiment 1.

[0159] In step S1 of this embodiment, the same type of support query sample pairs will be constructed using the semantic segmentation public data set. In the 5-shot setting, each support query sample pair contains 1 query image , 5 support images and 5 corresponding support masks . Other settings remain the same as in embodiment 1.

[0160] In step S2 of this embodiment, the support fusion features and query fusion features are obtained by the fusion feature extraction module . In the process of obtaining the visual prior of the query image, compared with embodiment 1, the number of visual priors and support fusion features obtained in this embodiment is 5 times that in embodiment 1. For the visual prior , the average of the 5 visual priors is directly taken as the final visual prior. For the support fusion features , no other processing is performed. Other processes remain the same as in step S2 of embodiment 1.

[0161] In step S3 of this embodiment, the multi-scale feature extraction module extracts multi-scale features from the fusion features. The process is referred to step S3 in embodiment 1. The number of multi-scale support features obtained is 5 times that in embodiment 1.

[0162] In step S4 of this embodiment, the multi-scale hint generation module contains 3 layers of mutual attention hint encoding generation blocks. The output of each layer is the hint encoding at the current scale. Each mutual attention hint encoding generation block first mines the similarity information between the current scale features according to the cosine similarity, then aggregates the hint encoding of the next scale and the intermediate result to generate the hint encoding at the current scale. The specific steps include the following:

[0163] S41, the specific steps remain the same as in step S41 of embodiment 1. The difference is that, since the number of multi-scale support features obtained in this embodiment is 5 times that in embodiment 1, the number of similarity hint encodings generated at a single scale is also 5 times that in embodiment 1. The average of the 5 similarity hint encodings is taken as the final similarity hint encoding at the current scale ;

[0164] S42, refer to step S42 in embodiment 1;

[0165] The query image and the final prompt code are input into the prompt type visual large model SAM in step S5 of the embodiment to obtain a segmentation result, and specific steps refer to step S5 in embodiment 1.

[0166] In step S6 of the embodiment, the support query sample pair in the small sample data is input into the trained prediction model to segment the query image, and the segmentation accuracy is calculated according to the prediction result, and specific steps refer to step S6 in embodiment 1.

[0167] In the embodiment, the data set is a public data set PASCAL, and a total of 20 classes of images are evenly divided into 4 parts, 3 of which are used as a training set, and the remaining 1 part is used as a test set, and the small sample scene is set to 5-shot, each support query sample pair contains 1 query image , 5 support images and 5 corresponding support masks ; the proposed method is compared with various small sample semantic segmentation methods, and the methods used for comparison are HSNet, BAM, HDMNet and SAM-RSP; all methods use ResNet-50 network as the backbone network, and the segmentation results of the present application and other methods mIoU under 5-shot setting are shown in the following table 2:

[0168] Table 2. Comparison table of experimental data (under 5-shot setting)

[0169]

[0170] Taking mIoU as an evaluation index, under 5-shot setting, the average segmentation accuracy in various scenes is 2.0% higher than the previous best method, and the performance is improved under each different data division.

[0171] The results of embodiment 1 and embodiment 2 show that the present application can be used in a small sample semantic segmentation visual task system, under the condition of only having a small amount of labeled support samples, the new class objects not seen in the model training process can be segmented, which can greatly reduce the labeling cost of new class target objects. The present application can further mine rich semantic information from multi-scale feature information under various scenes and different task settings, and the target prompt code generated can more accurately point to the class object to be segmented in the current scene, and the final segmentation result can also achieve higher average accuracy, so the target prompt code generated by the present application by using multi-scale information for continuous iteration and optimization is more robust.

[0172] It should be noted that, for the foregoing method embodiments, for the sake of brief description, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited to the action sequence described, because according to the present application, certain steps can be performed in other sequences or simultaneously.

[0173] The technical features of the above embodiments can be combined in any manner. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described, but as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present disclosure.

[0174] The above embodiments are the preferred embodiments of the present application, but the embodiments of the present application are not limited to the above embodiments, and any changes, modifications, substitutions, combinations, simplifications made without departing from the spirit and principles of the present application shall be equivalent replacement methods and shall be within the scope of the present application.

Claims

1. A few-sample semantic segmentation method based on multi-scale cues from a large visual model, characterized in that, The few-sample semantic segmentation method includes the following steps: S1. Construct similar support query pairs using a publicly available semantic segmentation dataset. These similar support query pairs include query images. Supports images and support mask ; S2, through the fusion feature extraction module The module acquires supporting fusion features and query fusion features separately, and simultaneously acquires the visual prior of the query image. It consists of a backbone network connected in sequence, a pair of parallel feature fusion convolutional layers, and a pair of parallel feature activation convolutional layers; S3, Multi-scale Feature Extraction Module The support fusion features and query fusion features are modeled and downsampled separately, and multi-scale support features and query features are extracted respectively. S4, Multi-scale prompt generation module Multi-scale cue codes are generated using multi-scale support features and query features. These multi-scale cue codes are then fused from the bottom up to generate the target cue code. Step S4 is as follows: S41. Input the query features and supporting features at the current scale, use cosine similarity to obtain the similarity map of the query features and supporting features, use the intermediate results of the next scale to adjust the similarity map, use the adjusted similarity map to aggregate the input learnable hint vector, and generate the similarity hint code at the current scale. S42. Fuse the similarity cue code at the current scale with the cue code at the next scale and the intermediate results to generate the cue code at the current scale. Output the cue code and the intermediate results generated by it to the previous scale to continue iterating until the target cue code is generated. S5. Input the target cue encoding and the query image into the cue-based visual large-scale model (SAM) to obtain the segmentation result of the query image. The visual large-scale model (SAM) includes an encoder and a decoder. The query image is input into the encoder to obtain the query image encoding, and the target cue encoding and the query image encoding are input into the decoder to obtain the segmentation result. Construct a prediction model, which includes sequentially connected fusion feature extraction modules. Multi-scale feature extraction module Multi-scale prompt generation module Together with the visual large model SAM, the intermediate and final results are used to form the total loss function to train the prediction model. During the training process, the parameters of the visual large model SAM are kept frozen. S6. Input the support samples from the small sample data into the trained prediction model to segment the query image, and calculate the segmentation accuracy based on the prediction results.

2. The few-sample semantic segmentation method based on multi-scale visual model cues according to claim 1, characterized in that, The process of constructing similar support query sample pairs in step S1 is as follows: The public dataset for semantic segmentation is divided into four groups on average according to the categories. Three groups are used for training, and the remaining group is used for testing. In each group, the training set and the test set contain a support set and a query set, respectively. The support set consists of support images and their corresponding densely labeled masks, while the query set consists of single images of the same category and their masks. The densely labeled masks of the support images in the support set are used as prior knowledge. During training, the prediction network is supervised by the densely labeled masks of the query set images to optimize the network parameters. During the testing phase, a densely labeled mask of the query image was used as a standard to evaluate model performance.

3. The few-sample semantic segmentation method based on multi-scale visual model cues according to claim 1, characterized in that, The process of obtaining supporting fusion features, query fusion features, and query visual priors in step S2 is as follows: S21. Using a pre-trained ResNet-50 as the backbone network, select the outputs of layers 2 and 3 of ResNet-50 as intermediate layer features, and the outputs of layers 4 and 5 as high-level features, and then apply these features to the query image. and supporting images Input, retrieve features: in, This represents the pre-trained backbone network. , They represent the first The query features and supporting features output by the layer. Indicates the first The number of channels in the layer feature. Indicates the first The height of layer features, Indicates the first The width of the layer feature; Query intermediate layer features respectively and support for intermediate layer features The inputs are fed into their respective feature fusion convolutional layers for dimensionality reduction and feature fusion to obtain intermediate query features. and support intermediate features ; Utilizing high-level features and Generate visual priors using support masks Remove high-level support features To uncover the relationships between pixels, we use a method that calculates the cosine similarity between the query feature and the supporting features at each pixel location, without considering irrelevant background features. : in" "" indicates the inner product operation of 3D vectors at all positions of the feature. Represents the L2 norm; For each location's query feature, the maximum similarity value is taken as the prior value for the current location. A minimum-maximum normalization method is then used to normalize this value to between 0 and 1, yielding the visual prior. ; S22. Use bilinear interpolation to apply the support mask. Size adjustment to support intermediate features Consistent, the prototype vector of the current category is obtained using masked average pooling: in, This is the prototype vector of the current category. Indicates support for intermediate features In position eigenvalues, To support the mask In position The labeled value, It is a conditional function; if the value in parentheses is true, it takes the value 1; otherwise, it takes the value 0. Query intermediate features prototype vector and visual prior The data is concatenated along the channel dimension and then passed through a query feature activation convolutional layer to obtain the query fusion feature. It will support intermediate features. prototype vector Concatenate along the channel dimension, then pass through a support feature activation convolutional layer to obtain support fusion features. .

4. The few-sample semantic segmentation method based on multi-scale visual model cues according to claim 1, characterized in that, The process by which the multi-scale feature extraction module extracts multi-scale features using fused features in step S3 is as follows: The multi-scale feature extraction module includes Layered attention blocks, with each layer of attention blocks connected sequentially, and the output of each attention block being feature information at a single scale. The outputs of the layer attention blocks are combined to form a multi-scale feature pyramid; each attention block includes a local attention block LocalAttnBlock and a self-attention block SelfAttnBlock connected in sequence. The formal representation of a single layer is as follows: in, Indicates the current floor number. It is the first Features of layer output, , It is the first Features of layer output, And there are , , , Indicates the first The number of channels characteristic of a multi-tiered pyramid. Indicates the first The high height of the pyramid features Indicates the first The width of the pyramidal features; Each local attention block is essentially a multi-branch convolutional layer that uses convolutional kernels of different sizes to acquire feature information from different receptive fields; the basic operation of each self-attention block is the attention mechanism. : in, For query vector, For key vectors, For value vectors, To query features, Key features, For value characteristics, For the corresponding projection matrix, The value of the feature dimension; if , , If they have the same feature, it is called self-attention; if they have different features, it is called mutual attention. The operation is applied to each row of the matrix to transform the attention scores into a probability distribution. , For the input matrix, For matrix In position The value; The query fusion features and supporting fusion features are respectively input into the multi-scale feature extraction module to obtain the multi-scale query feature pyramid. and multi-scale support feature pyramid ,in, Indicates the query feature pyramid level Features of the layer Indicates the support feature pyramid number 1 Characteristics of the layer.

5. A few-sample semantic segmentation method based on multi-scale visual model cues according to claim 4, characterized in that, The multi-scale prompt generation module in step S4 Include The layer mutual attention cue coding generation block outputs the cue coding at the current scale. Each mutual attention cue coding generation block first mines the similarity information between features at the current scale based on cosine similarity, and then aggregates the cue coding of the next scale and intermediate results to generate the cue coding at the current scale.

6. The few-sample semantic segmentation method based on multi-scale visual model cues according to claim 5, characterized in that, The process of generating the similarity hint code at the current scale in step S41 is as follows: For the Features of the layer and First, reshape the features and The shape will Shape reshaping The reshaped features were obtained respectively. and Input learnable cue vectors Similarly, the cue vector Shape reshaping Calculate the query vector in the attention mechanism respectively Key vector Value vector , obtain the query vector Key vector Sum value vector , where the query vector From query features key vector From supporting features value vector Learnable cue vectors derived from a set of inputs The formal representation is as follows: in, , , They are respectively , , The projection matrix; To obtain the similarity relationship between each pixel location, calculate and The cosine similarity between them yields a similarity graph matrix. : in," " indicates the vector dot product operation. The L2 norm of a vector. This refers to temperature hyperparameters. Along The first dimension The operation yields a normalized similarity graph matrix. : in, For matrix In position The value, For matrix In position The value; The support mask is obtained by using bilinear interpolation. Adjust to the corresponding size and reshape to obtain the desired result. ,use By removing the similarity values ​​at irrelevant background locations, we obtain the similarity matrix after masking the background. ; Under multi-scale conditions, the mask scores output by the next layer are used as a similarity prior to adjust the attention map and progressively optimize the segmentation results. The formal representation is as follows: in, For the first The layer similarity prior is obtained by adjusting the size of the segmentation mask logits score output by the next layer through bilinear interpolation. A logits score greater than 0 indicates foreground and less than 0 indicates background. The larger the absolute value, the greater the probability. It is a combination of matrix flattening and dimension expansion copying operations, first compressing the vector into a one-dimensional shape. Then, extend the copy along the second dimension until the shape matches... Consistent; Using the adjusted similarity matrix Aggregate the learnable cue vectors of the input to obtain the first... Layer similarity hint encoding .

7. A few-sample semantic segmentation method based on multi-scale visual model cues according to claim 6, characterized in that, The process of generating the current scale cue code and continuing to iterate until the target cue code is generated in step S42 is as follows: S421. If the current layer is the last layer, that is... Then, the similarity cue encoding at the current scale will be used directly. As a prompt code under the current scale ,Right now = The prompt code After being resized using bilinear interpolation, the data is directly input into the decoder of the large visual model (SAM) to obtain the intermediate segmentation score of the last layer. , As a similarity prior input, it is used to input the first In the layer mutual attention cue coding generation block, cue coding Also enter the number Iterative optimization is performed within the layer; S422. If the current layer is not the last layer, i.e. Then it is necessary to aggregate the prompt codes of the next layer and optimize step by step from bottom to top. First, the prompt codes of the first layer need to be aggregated. Layer prompt encoding The shape is upsampled using bilinear interpolation to match the scale of the current layer, and the shape is upsampled to match the scale of the current layer. Layer similarity hint encoding The data is concatenated together and input into a fusion layer for information fusion to obtain the prompt code at the current scale. ; After being resized using bilinear interpolation, the data is input into the decoder of the large visual model (SAM) to obtain the current [number of elements]. Layer segmentation intermediate result score Enter it into the first Layer mutual attention hints are used as similarity priors in the generated blocks, and the hints are encoded. Also enter the number Iterative optimization is performed within the layer; Repeat step S422, after After the layer iterations, the prompt encoding output by the first layer As the target prompt code for the output.

8. A few-sample semantic segmentation method based on multi-scale visual model cues according to claim 7, characterized in that, In step S5, the query image and the final prompt code are input into the prompt-based visual large model (SAM) to obtain the segmentation result. The process of constructing and training the prediction model is as follows: S51. Input the query image into the encoder of the visual large model SAM to obtain the query image encoding, and then output the segmentation result of the query image according to the prompt encoding and the query image encoding. The multi-scale cue code generated in step S4, together with the query image code, is input into the decoder of the Visual Large Model (SAM) to generate multi-scale segmentation results. ; Select As the final prediction output Other intermediate results As a similar prior input to the multi-scale cue generation module middle; S52, sequentially connected fusion feature extraction modules Multi-scale feature extraction module Multi-scale prompt generation module A prediction model is constructed using the visual large model SAM; based on the final prediction results and intermediate results Construct the overall loss function loss function To optimize the prediction model built for target training, the loss function The calculation method is as follows: in, For the total cross-entropy loss, , To find the mask of the query image, the cross-entropy loss function , It's a real label. It is the predicted probability; To address the distillation loss of the similarity matrix between different layers of the multi-scale cue generation module, KL divergence loss function , Is it the true distribution? right The probability, It is to estimate the distribution. right The probability, For the first The similarity vector is obtained by performing masked average pooling and normalization operations along the first dimension on the similarity matrix of the layer; The DICE loss for the final prediction result, , pixel position The probability value of the predicted outcome. The actual value of the pixel location label; , , These are the weight hyperparameters for each loss.

9. A few-sample semantic segmentation method based on multi-scale visual model cues according to claim 1, characterized in that, The process in step S6 is as follows: S61. Based on the trained prediction model, the segmentation result of the query image is obtained through the forward propagation of the network; if the final segmentation mask logits score is greater than 0, it is considered as foreground, otherwise it is predicted as background. S62. Calculate the segmentation accuracy based on the prediction results, and use the average intersection-union ratio (mIoU) as the evaluation index for segmentation accuracy.

Images