Small sample image segmentation method and system based on prototype affinity hybrid network

Abstract

The invention discloses a small sample image segmentation method and system based on a prototype affinity hybrid network, and the method comprises the following steps: carrying out the feature extraction of a support query sample pair, and obtaining a support feature corresponding to a support image and a query feature corresponding to a query image; calculating the similarity between the adaptive query prototype and the query features to obtain a conservative query prediction mask; guiding feature enhancement by using a prototype to obtain an enhanced support feature and an enhanced query feature; generating a final query prediction mask by using the attention-processing-based enhanced support feature and the attention-processing-based enhanced query feature output by the last attention block; training the initial segmentation model by using a loss function to obtain a target segmentation model; according to the method, complementary advantages of global semantic guidance and local detail matching can be realized, the segmentation precision and robustness of the model in a complex scene are improved, and the requirement of high-precision pixel-level analysis is met.

Description

Technical Field

[0001] The present invention relates to the field of machine learning and computer vision technology, and in particular to a small sample image segmentation method and system based on a prototype affinity hybrid network. Background Art

[0002] With the continuous development of neural networks, semantic segmentation, a fundamental task in computer vision, aims to assign a specific semantic category to each pixel in an image, thereby achieving pixel-level analysis of the image. The development of deep learning has promoted the practical application of semantic segmentation in tasks such as scene understanding and object recognition. However, to achieve high performance, traditional segmentation methods often rely on training datasets with large amounts of pixel-level annotations. However, collecting and annotating sufficient data requires significant resources, and in some scenarios, collecting sufficient data itself is a challenge, which imposes significant time and labor costs on practical applications. To overcome the dependence on the number of training samples, small-sample image segmentation methods have emerged and are gradually demonstrating their unique advantages. Small-sample image segmentation methods can segment similar objects in a query image under the guidance of only a small number of pixel-level annotated support images, providing new ideas and solutions to address the problems of sample scarcity and high annotation costs.

[0003] Currently, there are two main approaches to small-sample image segmentation: First, prototype learning-based methods compress the pixel features of the target object in the support image into a single or several prototype feature vectors, then find the pixel locations of similar features in the query image to segment the desired object. Second, affinity learning-based methods use target object pixels in the support image to match query image pixels, generating a query image mask. The learned features are then used to predict the affinities of cross-image pixels and perform feature matching.

[0004] Both of the above methods have their limitations. The prototype learning-based method uses a limited number of prototype feature vectors to represent the target category. Although this method can suppress the misjudgment of background pixels, it is difficult to cover the internal diversity of the target category, resulting in some foreground pixels being missed due to insufficient matching with the prototype. In particular, the segmentation accuracy is significantly reduced at the edge of the target or in areas with complex textures. The affinity learning-based method transfers target class information by establishing pixel-level matching between the support and query images. Although this method can improve the coverage of the target area, it lacks effective constraints on the foreground-background semantic relationship and is easily affected by local interference such as similar colors and textures, which can misclassify irrelevant background pixels as targets.

[0005] In order to avoid the limitations of a single method that may lead to incomplete target area segmentation or incorrect background area segmentation, technical personnel in this field have been seeking a small sample image segmentation method based on a prototype affinity hybrid network, so as to achieve the complementary advantages of global semantic guidance and local detail matching, improve the segmentation accuracy and robustness of the model in complex scenes, and meet the needs of high-precision pixel-level analysis in practical applications. Summary of the Invention

[0006] The purpose of the present invention is to provide a small sample image segmentation method and system based on prototype affinity hybrid network to improve the segmentation accuracy of the target area.

[0007] The small sample image segmentation method based on the prototype affinity hybrid network designed by the present invention to achieve one of the above purposes is special in that it includes the following steps:

[0008] Step 1: Obtain a support-query sample pair comprising a support set and a query set, wherein the support set comprises a support image and a support mask corresponding to the support image, and the query set comprises a query image; perform feature extraction on the support-query sample pair to obtain support features corresponding to the support image and query features corresponding to the query image;

[0009] Step 2: Perform masked average pooling on the supporting features to obtain a supporting prototype, calculate the similarity between the supporting prototype and the query feature to obtain a prototype-related query prediction mask, perform masked average pooling on the prototype-related query prediction mask and the query feature, and mix them with the supporting prototype to obtain an adaptive query prototype, calculate the similarity between the adaptive query prototype and the query feature, and obtain a conservative query prediction mask;

[0010] Step 3: Input the support features and the query features into a sequence of multiple attention blocks for iterative processing; before inputting each attention block, use the prototype to guide feature enhancement to obtain enhanced support features and enhanced query features;

[0011] Step 4: In each attention block, an attention mechanism is applied to the enhanced support features and the enhanced query features, and the attention scores in the attention mechanism are calibrated. The calibrated attention scores are used to perform weighted aggregation on the enhanced support features and the enhanced query features to obtain enhanced support features after attention processing and enhanced query features after attention processing. The final query prediction mask is generated using the enhanced support features after attention processing and the enhanced query features after attention processing output by the last attention block;

[0012] Step 5: Obtain the query true mask corresponding to the query image, construct a loss function based on the query true mask and the final query predicted mask, and use the loss function to train the initial image segmentation model to obtain the target image segmentation model.

[0013] Furthermore, step 1 specifically includes:

[0014] Step 1.1: performing center cropping, resizing, and normalization processing on the support image and the query image to obtain a preprocessed support image and a preprocessed query image;

[0015] Step 1.2: Using ResNet50 as a shared backbone network to extract features from the preprocessed support image and the preprocessed query image. During training, the shared backbone network only performs feature extraction and is frozen during training.

[0016] Step 1.3: Perform feature extraction on the pre-processed support image and the pre-processed query image respectively through a shared backbone network to obtain support features corresponding to the support image and query features corresponding to the query image.

[0017] Furthermore, step 2 specifically includes:

[0018] Step 2.1: Multiply the support features pixel by pixel using the support mask to set the background region features of the support image to zero, thereby retaining only the foreground region features of the support image, calculate the average vector of the foreground region features of the support image, and average the calculated average vector results of the foreground region features of the support image to obtain a support prototype;

[0019] Step 2.2: Calculate the cosine similarity between the support prototype and the feature vector of each pixel in the query feature to obtain a query prediction mask related to the prototype;

[0020] Step 2.3: Multiplying the query features pixel by pixel using the query prediction mask associated with the prototype to set the background region features of the query image to zero, thereby retaining only the foreground region features of the query image, calculating the average vector of the foreground region features of the query image, and averaging the calculated average vector of the foreground region features of the query image with the support prototype to obtain an adaptive query prototype;

[0021] Step 2.4: Calculate the cosine similarity between the adaptive query prototype and the feature vector of each pixel in the query feature to obtain a conservative query prediction mask.

[0022] Furthermore, step 3 specifically includes:

[0023] Step 3.1: Perform mask average pooling on the support mask to extract the support foreground prototype and the support background prototype from the support features. This process is described as follows:

[0024]

[0025]

[0026] in, Indicates the The layer's supporting foreground prototype, Indicates the The layer's supporting background prototype, Indicates the The supported features of the layer, Indicates support for mask, Indicates the row index and column index of the support image corresponding to the support feature;

[0027] Step 3.2: Generate an affinity query prediction mask based on the supporting foreground prototype and the supporting background prototype. This process is described in the following formula:

[0028]

[0029] in, Indicates the affinity query prediction mask for the layer, Indicates the The support features corresponding to the support image of the layer, represents the temperature parameter;

[0030] Step 3.3: Extract the affinity query foreground prototype based on the affinity query prediction mask. This process is described as follows:

[0031]

[0032] in, Indicates the Affinity query foreground prototype of the layer;

[0033] The support foreground prototype and the query foreground prototype are fused through adaptive weights to generate an affinity-based prototype. This process is described in the following formula:

[0034]

[0035]

[0036] in, represents the adaptive weight of affinity, Indicates the affinity-based prototyping of layers;

[0037] Step 3.4: Extract the conservative query foreground prototype based on the conservative query prediction mask. This process is described as follows:

[0038]

[0039] in, Indicates the Conservative query foreground prototype of the layer, represents the conservative query prediction mask;

[0040] The supporting foreground prototype and the query foreground prototype are fused through adaptive weights to generate a prototype-based prototype. This process is described in the following formula:

[0041]

[0042]

[0043] in, represents conservative adaptive weights, Indicates the Prototype-based prototyping of layers;

[0044] Step 3.5: The affinity-based prototype and the prototype-based prototype are respectively spliced with the support features, and the affinity-based prototype and the prototype-based prototype are respectively spliced with the query features. This process is described as follows:

[0045]

[0046]

[0047] in, Indicates affinity enhancement characteristics, represents the prototype enhancement feature, represents channel dimensionality reduction convolution, Indicates the Layer The features of an image, Indicates the category of the image, Indicates support for images, represents the query image;

[0048] Step 3.6: Concatenate the affinity-enhanced support features and the prototype-enhanced support features, and output the enhanced support features through residual connection. Concatenate the affinity-enhanced query features and the prototype-enhanced query features, and output the enhanced query features through residual connection. This process is described as follows:

[0049]

[0050] in, Indicates enhanced support features and enhanced query features.

[0051] Furthermore, step 4 specifically includes:

[0052] Step 4.1: Flatten the conservative query prediction mask and the support mask into vector form and generate a reweighting matrix using matrix multiplication. This process is described as follows:

[0053]

[0054] in, Indicates the The reweighting matrix of the layer, for Indicates that the linear projection operator maps the features to dimensional space, represents the space flattening function, represents the matrix transpose, Indicates the spatial dimension to which the matrix belongs;

[0055] Step 4.2: Perform three-value division on the prototype-based prototype to generate a confidence mask of the query image. This process is described as follows:

[0056]

[0057] in, represents the confidence mask of the query image, represents the foreground threshold, represents the background threshold;

[0058] Step 4.3: Construct an attention mask using the confidence mask of the query image and the support mask to identify foreground and background mismatches between the query image and the support image. This process is described as follows:

[0059]

[0060] in, Indicates the Attention mask of the layer;

[0061] Step 4.4: Calibrate the attention score by the attention mask. This process is described as follows:

[0062]

[0063] in, Indicates the The calibrated attention scores of the layers, represents the query embedding in the attention mechanism, represents the key embedding in the attention mechanism, Represents the embedding space dimension in the attention mechanism;

[0064] Step 4.5: Use the calibrated attention scores to aggregate the contextual information of the support features and update the query features. This process is described as follows:

[0065]

[0066] in, Represents the linear projection of the support feature.

[0067] Furthermore, in step 5, a loss function is constructed based on the query real mask and the final query prediction mask, and the expression of the loss function is:

[0068]

[0069] in, represents the loss function, represents the main loss term, represents the auxiliary loss term, represents the loss function of the similarity coefficient, represents the final query prediction mask, Indicates the query real mask, represents the weight coefficient, represents the number of attention blocks, represents the binary cross entropy loss function.

[0070] The small sample image segmentation system based on the prototype and affinity hybrid network designed by the present invention to achieve the second purpose above is special in that it includes:

[0071] a feature extraction module configured to obtain a support-query sample pair comprising a support set and a query set, wherein the support set comprises a support image and a support mask corresponding to the support image, and the query set comprises a query image; perform feature extraction on the support-query sample pair to obtain support features corresponding to the support image and query features corresponding to the query image;

[0072] A conservative query mask prediction module is configured to perform masked average pooling on the supporting features to obtain a supporting prototype, calculate the similarity between the supporting prototype and the query features to obtain a prototype-related query prediction mask, perform masked average pooling on the prototype-related query prediction mask and the query features, and mix the masked average pooling with the supporting prototype to obtain an adaptive query prototype, calculate the similarity between the adaptive query prototype and the query features, and obtain a conservative query prediction mask;

[0073] a feature enhancement module, configured to input the support features and the query features into a sequence comprising a plurality of attention blocks for iterative processing; before inputting each attention block, utilizing a prototype to guide feature enhancement to obtain enhanced support features and enhanced query features;

[0074] an attention calibration module, configured to apply an attention mechanism to the enhanced support features and the enhanced query features in each attention block, calibrate the attention scores in the attention mechanism, perform weighted aggregation on the enhanced support features and the enhanced query features using the calibrated attention scores, obtain enhanced support features after attention processing and enhanced query features after attention processing, and generate a final query prediction mask based on the enhanced support features after attention processing and the enhanced query features after attention processing output by the last attention block;

[0075] A model training module is used to obtain a true query mask corresponding to the query image, construct a loss function based on the true query mask and the final query prediction mask, and use the loss function to train the initial image segmentation model to obtain a target image segmentation model.

[0076] To achieve the third objective of the present invention, an image segmentation device is designed, characterized in that the image segmentation device comprises:

[0077] An image acquisition module to be segmented, used for acquiring the image to be segmented;

[0078] The image segmentation processing module is used to input the image to be segmented into the image segmentation model and output the segmentation result of the image to be segmented; wherein, the segmentation model is trained according to the above-mentioned small sample image segmentation method based on the prototype affinity hybrid network.

[0079] In order to achieve the fourth purpose of the present invention, a computer device is designed, including a memory, a processor, and a computer program stored in the memory and run on the processor, characterized in that when the processor executes the computer program, it implements the above-mentioned small sample image segmentation method based on prototype affinity hybrid network.

[0080] In order to achieve the fifth purpose of the present invention, a computer program product is designed, which includes computer instructions, and the computer instructions are used to enable a computer to execute the above-mentioned small sample image segmentation method based on prototype affinity hybrid network.

[0081] The present invention has the following beneficial effects:

[0082] (1) The present invention generates a preliminary query prediction by using a support prototype, and then uses the prediction and query features to perform average mask pooling to obtain a preliminary query prototype, which is then fused with the support prototype to obtain an adaptive query prototype, thereby generating a more reliable conservative query prediction mask. Before entering the attention module, the features are enhanced by fusing the affinity prototype calculated based on the support foreground / background prototype and the prototype calculated based on the conservative query prediction mask, and then fused using adaptive weights. This enhancement allows the features input to the attention module to contain richer, task-related semantic information, which helps the subsequent attention mechanism to more accurately capture the correspondence between support and query features. The conservative query prediction mask and the support mask are used to generate a reweighted matrix, and an attention score confidence mask is designed to hard-filter potential mismatched areas, so that the attention calculation is more focused on the foreground-foreground related areas, reducing interference, and effectively improving the model's ability to distinguish between targets and backgrounds when there are only a small number of labeled samples, thereby improving segmentation accuracy and robustness to changes in different samples.

[0083] (2) The present invention explicitly extracts and utilizes the global representation information of the target category in the support set by constructing multiple prototypes such as support prototypes, adaptive query prototypes, and affinity prototypes, and effectively transfers it to the processing process of the query features; by calculating the cosine similarity between the support prototypes and the query-derived prototypes, the adaptive weights are determined to dynamically balance the importance of the prior information from the support set and the prediction information of the query set itself, making the prototype fusion more targeted; the reweighted matrix and attention mask in the attention mechanism directly utilize the prediction mask and support mask generated from the prototype path, so that the model can more efficiently utilize the support set information to guide the pixel-level segmentation of the query set, thereby improving the efficiency of information utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Figure 1 A flow chart showing an embodiment of a small sample image segmentation method based on a prototype affinity hybrid network of the present invention is shown.

[0085] Figure 2 A schematic diagram of the network architecture of an embodiment of a small sample image segmentation method based on a prototype affinity hybrid network of the present invention is shown.

[0086] Figure 3 A schematic diagram of a prototype-guided feature enhancement architecture is shown in an embodiment of a small sample image segmentation method based on a prototype affinity hybrid network of the present invention.

[0087] Figure 4 A schematic diagram of the attention score calibration architecture of an embodiment of a small sample image segmentation method based on a prototype affinity hybrid network of the present invention is shown.

[0088] Figure 5A schematic diagram showing a comparison between an embodiment of a small sample image segmentation method based on a prototype affinity hybrid network of the present invention and an existing method.

[0089] Figure 6 A module diagram showing an embodiment of a small sample image segmentation system based on a prototype affinity hybrid network of the present invention.

[0090] Figure 7 A schematic structural diagram of an image segmentation device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0091] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0092] like Figures 1 to 7 As shown, the embodiment of the present invention discloses a small sample image segmentation method and system based on a prototype affinity hybrid network, which can improve the segmentation accuracy of the target area.

[0093] Example 1

[0094] This embodiment discloses a small sample image segmentation method based on a prototype affinity hybrid network, which includes the following steps:

[0095] Step 1: Obtain a support-query sample pair comprising a support set and a query set, wherein the support set comprises a support image and a support mask corresponding to the support image, and the query set comprises a query image; perform feature extraction on the support-query sample pair to obtain support features corresponding to the support image and query features corresponding to the query image;

[0096] Step 2: Perform masked average pooling on the supporting features to obtain a supporting prototype, calculate the similarity between the supporting prototype and the query feature to obtain a prototype-related query prediction mask, perform masked average pooling on the prototype-related query prediction mask and the query feature, and mix them with the supporting prototype to obtain an adaptive query prototype, calculate the similarity between the adaptive query prototype and the query feature, and obtain a conservative query prediction mask;

[0097] Step 3: Input the support features and the query features into a sequence of multiple attention blocks for iterative processing; before inputting each attention block, use the prototype to guide feature enhancement to obtain enhanced support features and enhanced query features;

[0098] Step 4: In each attention block, an attention mechanism is applied to the enhanced support features and the enhanced query features, and the attention scores in the attention mechanism are calibrated. The calibrated attention scores are used to perform weighted aggregation on the enhanced support features and the enhanced query features to obtain enhanced support features after attention processing and enhanced query features after attention processing. The final query prediction mask is generated using the enhanced support features after attention processing and the enhanced query features after attention processing output by the last attention block;

[0099] Step 5: Obtain the query true mask corresponding to the query image, construct a loss function based on the query true mask and the final query predicted mask, and use the loss function to train the initial image segmentation model to obtain the target image segmentation model.

[0100] Wherein, step 1 specifically includes:

[0101] Step 1.1: performing center cropping, resizing, and normalization processing on the support image and the query image to obtain a preprocessed support image and a preprocessed query image;

[0102] For example, the image is uniformly scaled to 473×473 resolution, the pixel values are normalized to the range of [0,1], and random horizontal flipping, random cropping and color jittering are performed;

[0103] Step 1.2: Using ResNet50 as a shared backbone network to extract features from the preprocessed support image and the preprocessed query image. During training, the shared backbone network only performs feature extraction and is frozen during training.

[0104] Step 1.3: Perform feature extraction on the pre-processed support image and the pre-processed query image respectively through a shared backbone network to obtain support features corresponding to the support image and query features corresponding to the query image.

[0105] Among them, step 2 specifically includes:

[0106] Step 2.1: Multiply the support features pixel by pixel using the support mask to set the background region features of the support image to zero, thereby retaining only the foreground region features of the support image, calculate the average vector of the foreground region features of the support image, and average the calculated average vector results of the foreground region features of the support image to obtain a support prototype;

[0107] Step 2.2: Calculate the cosine similarity between the support prototype and the feature vector of each pixel in the query feature to obtain a query prediction mask related to the prototype;

[0108] Step 2.3: Multiplying the query features pixel by pixel using the query prediction mask associated with the prototype to set the background region features of the query image to zero, thereby retaining only the foreground region features of the query image, calculating the average vector of the foreground region features of the query image, and averaging the calculated average vector of the foreground region features of the query image with the support prototype to obtain an adaptive query prototype;

[0109] Step 2.4: Calculate the cosine similarity between the adaptive query prototype and the feature vector of each pixel in the query feature to obtain a conservative query prediction mask.

[0110] Among them, step 3 specifically includes:

[0111] Step 3.1: Perform mask average pooling on the support mask to extract the support foreground prototype and the support background prototype from the support features. This process is described as follows:

[0112]

[0113]

[0114] in, Indicates the The layer's supporting foreground prototype, Indicates the The layer's supporting background prototype, Indicates the The supported features of the layer, Indicates support for mask, Indicates the row index and column index of the support image corresponding to the support feature;

[0115] Step 3.2: Generate an affinity query prediction mask based on the supporting foreground prototype and the supporting background prototype. This process is described in the following formula:

[0116]

[0117] in, Indicates the affinity query prediction mask for the layer, Indicates the The support features corresponding to the support image of the layer, represents the temperature parameter;

[0118] Step 3.3: Extract the affinity query foreground prototype based on the affinity query prediction mask. This process is described as follows:

[0119]

[0120] in, Indicates the Affinity query foreground prototype of the layer;

[0121] The support foreground prototype and the query foreground prototype are fused through adaptive weights to generate an affinity-based prototype. This process is described in the following formula:

[0122]

[0123]

[0124] in, represents the adaptive weight of affinity, Indicates the affinity-based prototyping of layers;

[0125] Step 3.4: Extract the conservative query foreground prototype based on the conservative query prediction mask. This process is described as follows:

[0126]

[0127] in, Indicates the Conservative query foreground prototype of the layer, represents the conservative query prediction mask;

[0128] The supporting foreground prototype and the query foreground prototype are fused through adaptive weights to generate a prototype-based prototype. This process is described in the following formula:

[0129]

[0130]

[0131] in, represents conservative adaptive weights, Indicates the Prototype-based prototyping of layers;

[0132] Step 3.5: The affinity-based prototype and the prototype-based prototype are respectively spliced with the support features, and the affinity-based prototype and the prototype-based prototype are respectively spliced with the query features. This process is described as follows:

[0133]

[0134]

[0135] in, Indicates affinity enhancement characteristics, represents the prototype enhancement feature, represents channel dimensionality reduction convolution, Indicates the Layer The features of an image, Indicates the category of the image, Indicates support for images, Represents the query image; thus, the original feature information is retained through the residual structure to avoid gradient disappearance. The dual prototype can guide the enhancement of the foreground area and suppress background noise;

[0136] Step 3.6: Concatenate the affinity-enhanced support features and the prototype-enhanced support features, and output the enhanced support features through residual connections. Concatenate the affinity-enhanced query features and the prototype-enhanced query features, and output the enhanced query features through residual connections, so that the features are more focused on the target area and the aggressive tendency of affinity learning is alleviated. This process is described as follows:

[0137]

[0138] in, Indicates enhanced support features and enhanced query features.

[0139] Among them, step 4 specifically includes:

[0140] Step 4.1: Flatten the conservative query prediction mask and the support mask into vector form and generate a reweighting matrix using matrix multiplication. This process is described as follows:

[0141]

[0142] in, Indicates the The reweighting matrix of the layer, for Indicates that the linear projection operator maps the features to dimensional space, represents the space flattening function, represents the matrix transpose, Indicates the spatial dimension to which the matrix belongs. When and When both are foreground or background, A larger value enhances the correlation, otherwise the (FG-BG) value is smaller, suppressing the false association;

[0143] Step 4.2: Perform three-value division on the prototype-based prototype to generate a confidence mask of the query image. This process is described as follows:

[0144]

[0145] in, represents the confidence mask of the query image, represents the foreground threshold, represents the background threshold;

[0146] It should be noted that the foreground threshold is set to 0.7 and the background threshold is set to 0.3;

[0147] Step 4.3: Construct an attention mask using the confidence mask of the query image and the support mask to identify foreground and background mismatches between the query image and the support image. This process is described as follows:

[0148]

[0149] in, Indicates the Attention mask of the layer;

[0150] It should be noted that when the support pixel is foreground and the query pixel is high-confidence background (or vice versa), it is determined to be a deterministic mismatch, and its weight in the attention matrix is set to negative infinity and returned to zero after Softmax to completely block the transmission of erroneous information;

[0151] Step 4.4: Calibrate the attention scores using the attention mask. This dual calibration of soft reweighting and hard masking significantly improves the segmentation accuracy with almost no increase in computational load. This process is described as follows:

[0152]

[0153] in, Indicates the The calibrated attention scores of the layers, represents the query embedding in the attention mechanism, represents the key embedding in the attention mechanism, Represents the embedding space dimension in the attention mechanism;

[0154] Step 4.5: Use the calibrated attention scores to aggregate the contextual information of the support features and update the query features. This process is described as follows:

[0155]

[0156] in, Represents the linear projection of the support feature.

[0157] In step 5, a loss function is constructed based on the query true mask and the final query prediction mask. The expression of the loss function is:

[0158]

[0159] in, represents the loss function, represents the main loss term, represents the auxiliary loss term, represents the loss function of the similarity coefficient, represents the final query prediction mask, Indicates the query real mask, represents the weight coefficient, represents the number of attention blocks, represents the binary cross entropy loss function.

[0160] It should be noted that the prototype-affinity hybrid network in this embodiment achieves state-of-the-art performance on the PASCAL-5i and COCO-20i datasets. In the 1-shot and 5-shot settings, the prototype-affinity hybrid network achieves mean Intersection Over Union (MIOU) of 71.6% and 72.3%, respectively, representing improvements of 4.3% and 3.9% over the baseline model HDMNet, and surpassing the leading method AENet by over 2.1%. By dynamically synergizing the conservative predictions of the prototype predictor with the aggressive predictions of the affinity learner, it precisely balances segmentation coverage and accuracy. By fusing the complementary prototypes of the prototype predictor and affinity learner, foreground semantic representation is enhanced. Using the hybrid network alone, it improves mIoU by 1.9%. The ASC module suppresses mismatches through soft reweighting and hard masking, achieving a 2.6% improvement. When the two modules work together, mIoU further increases to 71.6% (+3.2%), validating the complementary nature of the two modules. The features of the prototype-affinity hybrid network focus more closely on the target area, while the baseline model is susceptible to background interference.

[0161] Example 2

[0162] This embodiment discloses a small sample image segmentation system based on a prototype affinity hybrid network, the system comprising:

[0163] a feature extraction module configured to obtain a support-query sample pair comprising a support set and a query set, wherein the support set comprises a support image and a support mask corresponding to the support image, and the query set comprises a query image; perform feature extraction on the support-query sample pair to obtain support features corresponding to the support image and query features corresponding to the query image;

[0164] A conservative query mask prediction module is configured to perform masked average pooling on the supporting features to obtain a supporting prototype, calculate the similarity between the supporting prototype and the query features to obtain a prototype-related query prediction mask, perform masked average pooling on the prototype-related query prediction mask and the query features, and mix the masked average pooling with the supporting prototype to obtain an adaptive query prototype, calculate the similarity between the adaptive query prototype and the query features, and obtain a conservative query prediction mask;

[0165] a feature enhancement module, configured to input the support features and the query features into a sequence comprising a plurality of attention blocks for iterative processing; before inputting each attention block, utilizing a prototype to guide feature enhancement to obtain enhanced support features and enhanced query features;

[0166] an attention calibration module, configured to apply an attention mechanism to the enhanced support features and the enhanced query features in each attention block, calibrate the attention scores in the attention mechanism, perform weighted aggregation on the enhanced support features and the enhanced query features using the calibrated attention scores, obtain enhanced support features after attention processing and enhanced query features after attention processing, and generate a final query prediction mask based on the enhanced support features after attention processing and the enhanced query features after attention processing output by the last attention block;

[0167] A model training module is used to obtain a true query mask corresponding to the query image, construct a loss function based on the true query mask and the final query prediction mask, and use the loss function to train the initial image segmentation model to obtain a target image segmentation model.

[0168] Example 3

[0169] This embodiment discloses an image segmentation device, including:

[0170] An image acquisition module to be segmented, used for acquiring the image to be segmented;

[0171] The image segmentation processing module is used to input the image to be segmented into the image segmentation model and output the segmentation result of the image to be segmented; wherein, the segmentation model is trained according to the small sample image segmentation method based on the prototype affinity hybrid network.

[0172] The specific definition of the image segmentation device can be found in the definition of the small sample image segmentation method based on the prototype affinity hybrid network above, and will not be repeated here. The various modules in the above-mentioned image segmentation device can be implemented in whole or in part by software, hardware, or a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of the above-mentioned modules.

[0173] Example 4

[0174] This embodiment discloses a computer device. The computer device includes a memory, a processor, and a network interface that are interconnected through a system bus. It should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application specific integrated circuit (ASIC), a programmable gate array (FPGA), a digital signal processor (DSP), an embedded device, etc.

[0175] The computer device may be a desktop computer, notebook computer, PDA, cloud server, etc. The computer device may interact with the user via a keyboard, mouse, remote control, touchpad, or voice control device.

[0176] Example 5

[0177] The present embodiment discloses a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of computer program instructions in computer-readable media includes but is not limited to source files, executable files, installation package files, etc., and accordingly, the way in which computer program instructions are executed by a computer includes but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to the computer. Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the program and module described above can refer to the corresponding process description in the aforementioned method embodiment, and will not be repeated here.

[0178] The algorithm and display provided herein are not inherently related to any particular computer, virtual system or other device. Various general-purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious that the structure required for constructing this type of system. In addition, the present invention is not directed to any specific programming language. It should be understood that various programming languages can be utilized to realize the content of the present invention described herein, and the above description of specific languages is for the purpose of disclosing the best mode of the present invention.

[0179] In the description provided herein, numerous specific details are described. However, it is understood that embodiments of the present invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques are not shown in detail so as not to obscure the understanding of this description.

[0180] Similarly, it should be understood that in order to streamline the present disclosure and aid understanding of one or more of the various inventive aspects, in the above description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected in the claims, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Accordingly, the claims that follow the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of the invention.

Claims

1. A small sample image segmentation method based on prototype affinity hybrid network, characterized in that: The steps include: Step 1: Obtain a support-query sample pair comprising a support set and a query set, wherein the support set comprises a support image and a support mask corresponding to the support image, and the query set comprises a query image; perform feature extraction on the support-query sample pair to obtain support features corresponding to the support image and query features corresponding to the query image; Step 2: Perform masked average pooling on the supporting features to obtain a supporting prototype, calculate the similarity between the supporting prototype and the query feature to obtain a prototype-related query prediction mask, perform masked average pooling on the prototype-related query prediction mask and the query feature, and mix them with the supporting prototype to obtain an adaptive query prototype, calculate the similarity between the adaptive query prototype and the query feature, and obtain a conservative query prediction mask; Step 3: Input the support features and the query features into a sequence of multiple attention blocks for iterative processing; before inputting each attention block, use the prototype to guide feature enhancement to obtain enhanced support features and enhanced query features; Step 4: In each attention block, an attention mechanism is applied to the enhanced support features and the enhanced query features, and the attention scores in the attention mechanism are calibrated. The calibrated attention scores are used to perform weighted aggregation on the enhanced support features and the enhanced query features to obtain enhanced support features after attention processing and enhanced query features after attention processing. The final query prediction mask is generated using the enhanced support features after attention processing and the enhanced query features after attention processing output by the last attention block; Step 5: Obtain the query true mask corresponding to the query image, construct a loss function based on the query true mask and the final query predicted mask, and use the loss function to train the initial image segmentation model to obtain the target image segmentation model.

2. The small sample image segmentation method based on prototype affinity hybrid network according to claim 1 is characterized in that: Step 1 specifically includes: Step 1.1: performing center cropping, resizing, and normalization processing on the support image and the query image to obtain a preprocessed support image and a preprocessed query image; Step 1.2: Using ResNet50 as a shared backbone network to extract features from the preprocessed support image and the preprocessed query image. During training, the shared backbone network only performs feature extraction and is frozen during training. Step 1.3: Perform feature extraction on the pre-processed support image and the pre-processed query image respectively through a shared backbone network to obtain support features corresponding to the support image and query features corresponding to the query image.

3. The small sample image segmentation method based on prototype affinity hybrid network according to claim 1 is characterized in that: Step 2 specifically includes: Step 2.1: Multiply the support features pixel by pixel using the support mask to set the background region features of the support image to zero, thereby retaining only the foreground region features of the support image, calculate the average vector of the foreground region features of the support image, and average the calculated average vector results of the foreground region features of the support image to obtain a support prototype; Step 2.2: Calculate the cosine similarity between the support prototype and the feature vector of each pixel in the query feature to obtain a query prediction mask related to the prototype; Step 2.3: Multiplying the query features pixel by pixel using the query prediction mask associated with the prototype to set the background region features of the query image to zero, thereby retaining only the foreground region features of the query image, calculating the average vector of the foreground region features of the query image, and averaging the calculated average vector of the foreground region features of the query image with the support prototype to obtain an adaptive query prototype; Step 2.4: Calculate the cosine similarity between the adaptive query prototype and the feature vector of each pixel in the query feature to obtain a conservative query prediction mask.

4. The small sample image segmentation method based on prototype affinity hybrid network according to claim 1 is characterized in that: Step 3 specifically includes: Step 3.1: Perform mask average pooling on the support mask to extract the support foreground prototype and the support background prototype from the support features. This process is described as follows: ； ； in, Indicates the The layer's supporting foreground prototype, Indicates the Layer's supporting background prototype, Indicates the The supported features of the layer, Indicates support for mask, Indicates the row index and column index of the support image corresponding to the support feature; Step 3.2: Generate an affinity query prediction mask based on the supporting foreground prototype and the supporting background prototype. This process is described in the following formula: ； in, Indicates the affinity query prediction mask for the layer, Indicates the The support features corresponding to the support image of the layer, represents the temperature parameter; Step 3.3: Extract the affinity query foreground prototype based on the affinity query prediction mask. This process is described as follows: ； in, Indicates the Affinity query foreground prototype of the layer; The support foreground prototype and the query foreground prototype are fused through adaptive weights to generate an affinity-based prototype. This process is described in the following formula: ； ； in, represents the adaptive weight of affinity, Indicates the affinity-based prototyping of layers; Step 3.4: Extract the conservative query foreground prototype based on the conservative query prediction mask. This process is described as follows: ； in, Indicates the Conservative query foreground prototype of the layer, represents the conservative query prediction mask; The supporting foreground prototype and the query foreground prototype are fused through adaptive weights to generate a prototype-based prototype. This process is described in the following formula: ； ； in, represents conservative adaptive weights, Indicates the Prototype-based prototyping of layers; Step 3.5: The affinity-based prototype and the prototype-based prototype are respectively spliced with the support features, and the affinity-based prototype and the prototype-based prototype are respectively spliced with the query features. This process is described as follows: ； ； in, Indicates affinity enhancement characteristics, represents the prototype enhancement feature, represents channel dimensionality reduction convolution, Indicates the Layer The features of an image, Indicates the category of the image, Indicates support for images, represents the query image; Step 3.6: Concatenate the affinity-enhanced support features and the prototype-enhanced support features, and output the enhanced support features through residual connection. Concatenate the affinity-enhanced query features and the prototype-enhanced query features, and output the enhanced query features through residual connection. This process is described as follows: ； in, Indicates enhanced support features and enhanced query features.

5. The small sample image segmentation method based on prototype affinity hybrid network according to claim 1 is characterized in that: Step 4 specifically includes: Step 4.1: Flatten the conservative query prediction mask and the support mask into vector form and generate a reweighting matrix using matrix multiplication. This process is described in the following formula: ； in, Indicates the The reweighting matrix of the layer, for Indicates that the linear projection operator maps the features to dimensional space, represents the space flattening function, represents the matrix transpose, Indicates the spatial dimension to which the matrix belongs; Step 4.2: Perform three-value division on the prototype-based prototype to generate a confidence mask of the query image. This process is described as follows: ； in, represents the confidence mask of the query image, represents the foreground threshold, represents the background threshold; Step 4.3: Construct an attention mask using the confidence mask of the query image and the support mask to identify foreground and background mismatches between the query image and the support image. This process is described as follows: ； in, Indicates the Attention mask of the layer; Step 4.4: Calibrate the attention score by the attention mask. This process is described as follows: ； in, Indicates the The calibrated attention scores of the layers, represents the query embedding in the attention mechanism, represents the key embedding in the attention mechanism, Represents the embedding space dimension in the attention mechanism; Step 4.5: Use the calibrated attention scores to aggregate the contextual information of the support features and update the query features. This process is described as follows: ； in, Represents the linear projection of the support feature.

6. The small sample image segmentation method based on prototype affinity hybrid network according to claim 1, characterized in that: In step 5, a loss function is constructed based on the query real mask and the final query prediction mask. The expression of the loss function is: ； in, represents the loss function, represents the main loss term, represents the auxiliary loss term, represents the loss function of the similarity coefficient, represents the final query prediction mask, Indicates the query real mask, represents the weight coefficient, represents the number of attention blocks, represents the binary cross entropy loss function.

7. A small sample image segmentation system based on prototype affinity hybrid network, characterized by: include: a feature extraction module configured to obtain a support-query sample pair comprising a support set and a query set, wherein the support set comprises a support image and a support mask corresponding to the support image, and the query set comprises a query image; perform feature extraction on the support-query sample pair to obtain support features corresponding to the support image and query features corresponding to the query image; A conservative query mask prediction module is configured to perform masked average pooling on the supporting features to obtain a supporting prototype, calculate the similarity between the supporting prototype and the query features to obtain a prototype-related query prediction mask, perform masked average pooling on the prototype-related query prediction mask and the query features, and mix the masked average pooling with the supporting prototype to obtain an adaptive query prototype, calculate the similarity between the adaptive query prototype and the query features, and obtain a conservative query prediction mask; a feature enhancement module, configured to input the support features and the query features into a sequence comprising a plurality of attention blocks for iterative processing; Before inputting each attention block, prototype-guided feature enhancement is used to obtain enhanced support features and enhanced query features; an attention calibration module, configured to apply an attention mechanism to the enhanced support features and the enhanced query features in each attention block, calibrate the attention scores in the attention mechanism, perform weighted aggregation on the enhanced support features and the enhanced query features using the calibrated attention scores, obtain enhanced support features after attention processing and enhanced query features after attention processing, and generate a final query prediction mask based on the enhanced support features after attention processing and the enhanced query features after attention processing output by the last attention block; A model training module is used to obtain a true query mask corresponding to the query image, construct a loss function based on the true query mask and the final query prediction mask, and use the loss function to train the initial image segmentation model to obtain a target image segmentation model.

8. An image segmentation device, characterized in that: The image segmentation device comprises: An image acquisition module to be segmented, used for acquiring the image to be segmented; An image segmentation processing module is used to input the image to be segmented into an image segmentation model and output the segmentation result of the image to be segmented; wherein, the segmentation model is trained according to the small sample image segmentation method based on the prototype affinity hybrid network according to any one of claims 1 to 6.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, it implements the small sample image segmentation method based on the prototype affinity hybrid network according to any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the small sample image segmentation method based on a prototype affinity hybrid network according to any one of claims 1 to 6.

Images