A generalized small sample image segmentation method and system

Abstract

The application provides a generalized small sample image segmentation method and system, mainly through feature diffusion noise adding and denoising of a feature map, low-level texture interference is inhibited and expression of high-level semantic features is enhanced; a learnable group label is introduced, a personalized group label related to a context of the feature map after denoising is generated by dynamically adjusting a relationship between features and labels, modeling capability for complex relationships between classes is enhanced, and each channel in the feature map is weighted by using the personalized group label, a class of each pixel position is predicted, and segmentation of the image is realized. The application not only improves overall performance of a model, but also provides an efficient and robust solution for semantic segmentation in a complex scene and a cross-domain task, and has significant technical advantages and application value.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of machine learning, in particular to a generalized few-shot image segmentation method and system. BACKGROUND

[0002] Semantic segmentation, as an important task in computer vision, aims to assign semantic labels to each pixel in an image. With the rapid development of deep neural networks, especially convolutional neural networks (CNN) and vision Transformers, semantic segmentation has made significant progress in performance. Classic methods such as fully convolutional networks (FCN) have successfully promoted the development of semantic segmentation technology by introducing multi-scale context information and attention mechanisms, achieving remarkable results. However, these methods are highly dependent on large-scale pixel-level labeled data, so it is difficult to directly extend them to new classes, usually requiring a large amount of labeled data, resulting in increased time and cost.

[0003] To address this problem, few-shot semantic segmentation (FSS) has emerged. The goal of FSS is to perform semantic segmentation with only a small amount of labeled samples, usually by first training on base classes and then extending to new classes using a small amount of labeled samples. However, traditional FSS methods often result in a decline in performance on base classes when extending to new classes. Moreover, the FSS assumption is that the support image and query image contain the same new class, as shown in Figure 1 To overcome this limitation, generalized few-shot semantic segmentation (GFSS) is proposed.

[0004] The goal of generalized few-shot semantic segmentation is to not require paired support samples at test time, but to simultaneously segment base and new classes through a single inference, thereby better simulating real-world application scenarios. Generalized few-shot segmentation is a semantic segmentation task that aims to accurately classify target pixels in an image while distinguishing between base and new classes. Specifically, this task provides a large amount of labeled data for base classes and only a small amount of labeled samples for new classes, with a background class representing pixels that do not belong to any target class. Generalized few-shot segmentation requires the model to have strong learning ability on base classes and to be able to quickly expand knowledge through limited new class samples, thereby achieving differentiation of all target classes and backgrounds.

[0005] Generalized few-shot semantic segmentation typically employs a two-stage training framework. In the first stage, the model is fully trained on data from the base and background categories to learn the semantic features of the base categories. In the second stage, the model updates the segmentation head using a small amount of support data from the new categories, enabling it to simultaneously predict pixel classifications for the base, new categories, and background. This approach aims to improve the model's segmentation performance across all categories by combining foundational and new knowledge.

[0006] Currently, a variety of technical solutions have emerged in the field of generalized small-sample segmentation. For example, the Context-Aware Prototype Learning (CAPL) method enriches the feature representation of new categories by leveraging contextual information in support samples and query samples, thereby improving segmentation performance. However, CAPL fails to fully consider the relationship between new classes and base classes, which may lead to damage to the base class prototypes, thereby affecting the segmentation performance of the base class. Another approach is the Projection onto Orthogonal Prototypes (POP) framework. POP learns a series of orthogonal prototypes, each corresponding to a specific semantic category, thereby avoiding the influence of base class features when learning new classes. POP freezes the base class prototypes and ensures orthogonality between prototypes when learning new classes, thereby reducing the interference of new class learning on the base class. In addition, POP represents the background by measuring the residual of feature projection, effectively solving the "semantic drift" problem in the new class learning process.

[0007] However, although existing small-shot segmentation and generalized small-shot segmentation technologies have made some progress in performance improvement, they still have many shortcomings in handling complex scenes and maintaining a balance between basic and new class segmentation performance:

[0008] Firstly, the prior art is deficient in performance trade-off between base classes and new classes. For example, the CAPL method enriches the feature representation of new classes through context information, but easily destroys the prototype structure of base classes, leading to the decline of base class segmentation performance. On the other hand, POP alleviates the interference of new class learning on base classes through orthogonality constraint, but its strategy of fixing base class prototypes lacks flexibility in diversified scenarios and is difficult to dynamically adapt to complex semantic relationships between classes. Secondly, the prior art has limitations in modeling inter-class relationships. CAPL and POP fail to fully capture the complex similarity relationship between base classes and new classes, especially in the context of semantically similar classes, which is prone to class confusion. This deficiency makes it difficult for the model to effectively distinguish between semantically similar classes in multi-class segmentation tasks. In addition, the prior art lacks adaptability in complex background and rich detail scenarios. In images containing high-frequency features such as edges and detailed information, as well as low-frequency features such as texture and uniform regions, context modeling methods such as CAPL are easily disturbed by background noise, while POP's strategy of modeling background based on residuals may ignore important fine-grained semantic information in the background, resulting in reduced segmentation accuracy. Finally, existing methods have limited dynamic feature adjustment and generalization ability. Methods such as CAPL and POP rely on fixed feature processing strategies and lack dynamic adjustment capabilities, making it difficult to dynamically optimize feature distribution according to different scenarios. In addition, existing technologies lack scalability in new class learning, making it difficult to efficiently adapt to the semantic features of new classes with a small amount of support data.

[0009] In summary, the prior art has significant deficiencies in performance balance between base classes and new classes, adaptability to complex scenarios, class relationship modeling, and dynamic feature adjustment capabilities. SUMMARY

[0010] In order to overcome the deficiencies in the prior art described above, the purpose of the present application is to provide a generalized small sample image segmentation method and system.

[0011] In order to achieve the above-mentioned purpose of the present application, the present application provides a generalized small sample image segmentation method, comprising the following steps:

[0012] Obtaining an image sample;

[0013] Constructing a generalized small sample semantic segmentation model, and training the generalized small sample semantic segmentation model using the image sample:

[0014] Extracting a feature map from the image sample in the generalized small sample semantic segmentation model, and performing semantic steady-state diffusion on the feature map after extracting the feature: diffusing the feature map with noise, rearranging the diffused feature, and removing the added noise to obtain a denoised feature map;

[0015] Perform context-aware semantic tagging on the denoised feature map: Global average pooling is performed on the denoised feature map to generate a channel-level global personalized vector, which is then input into the perception layer to generate context-aware channel weights. Learnable group tags are generated for the denoised feature map, where each tag corresponds to a semantic category. The channel weights are applied to the learnable group tags to generate personalized group tags that are context-sensitive to the current input denoised feature map.

[0016] The personalized group labels are used to dynamically weight each channel in the feature map, predict the category of each pixel position, and achieve image segmentation.

[0017] In this generalized small-sample image segmentation method, semantic steady-state diffusion is used to enable the model to retain the high-level semantic information of the image, while the introduction of diversity enhances its adaptability to complex scenes. The denoising process further enhances the robustness and adaptability of the model, making it perform better in small-sample and generalized small-sample segmentation tasks. At the same time, the feature expression ability is enhanced through environmental perception semantic labeling, and the intra-class compactness and inter-class separation of features are improved. Finally, by weighting each channel in the feature map, the model can focus on features related to the current category or pixel, thereby improving the category discrimination ability.

[0018] Optionally, the steps of performing feature diffusion noise addition on the feature map are: calculating the change or difference degree of the feature values ​​between local adjacent positions to estimate the gradient strength, quantifying the change trend of the position of each feature point in the feature map, and generating dynamic noise according to the gradient amplitude of each feature position, and superimposing the generated dynamic noise on the original feature at each feature position.

[0019] This optional solution performs better in texture suppression because isotropic Gaussian blurring may blur edges and other high-level semantic information. Therefore, in this embodiment, anisotropic noise addition is used to dynamically adjust the feature map and perform feature noise diffusion.

[0020] Optionally, when generating dynamic noise, the dynamic noise coefficient is inversely proportional to the gradient amplitude.

[0021] This optional approach reduces noise in high-gradient regions, preventing damage to key structures, while increasing noise in low-gradient regions, increasing feature diversity. Introducing randomness into smooth regions helps reduce the model's overreliance on low-level textures. By reducing the influence of low-level textures, the model can focus more on category-relevant, high-level semantic features, improving its generalization capabilities for complex scenes.

[0022] Optionally, the denoising and denoising processes are supervised using reconstruction loss and feature diversity loss;

[0023] The reconstruction loss uses mean square error as a metric to measure the difference between the original feature map and the denoised feature map;

[0024] The feature diversity loss measures the diversity of features by calculating the variance of the denoised feature map.

[0025] By minimizing the reconstruction loss, the constrained model ensures that the high-level semantic information of the image is preserved while avoiding the loss of key information during the denoising process. By introducing feature diversity loss, the model can focus more on high-level semantic information, improving its generalization ability for complex scenes.

[0026] Optionally, the steps for generating learnable group labels are:

[0027] Initialize a set of learnable group semantic labels, each label corresponds to a latent semantic category;

[0028] Dynamically adjust the similarity between each pixel feature and each tag in the group tag, generate the degree of belonging of each pixel feature to each group tag, perform weighted aggregation on the pixel features assigned to each tag, and update the group tag.

[0029] Optionally, when generating the degree of belonging of each pixel feature to each group label, one pixel feature is allowed to be assigned to multiple group labels simultaneously, but with different weights or probabilities.

[0030] Optionally, a contrastive loss is used to promote discrimination between labels and optimize the feature representation of each pixel by maximizing the similarity of each pixel feature with the positive sample label while minimizing the similarity with the negative sample label.

[0031] Optionally, a dynamic weighting mechanism based on MMoE is designed, where each personalized group tag is used as a "Gate" to capture the features of a specific semantic category and dynamically weight each channel in the feature map through the Gate;

[0032] Perform convolution on the weighted feature map to map the channel dimension to the number of categories;

[0033] Calculate the category probability distribution of each pixel position, and take the category with the highest probability at each pixel position as the prediction result of the pixel.

[0034] Optionally, the training is divided into a first training phase and a second training phase;

[0035] In the first training phase, the generalized small-shot semantic segmentation model is trained on the base category data and background categories to learn the semantic features of the base categories;

[0036] In the second training stage, the new category is used as a data set to update the new category in the generalized small sample semantic segmentation model after the first training stage, and the base category and the new category are classified.

[0037] The application also provides a generalized small sample image segmentation system, comprising a data acquisition module for acquiring image samples, a processing module and a storage module;

[0038] The data acquisition module is connected with the processing module, and sends the image samples to the processing module, the processing module and the storage module are in communication connection, the storage module is used for storing at least one executable instruction, and the executable instruction makes the processing module execute the operation corresponding to the generalized small sample image segmentation method as described above according to the image samples, and the image is segmented.

[0039] The application has the following beneficial effects:

[0040] The application significantly improves the feature expression ability and generalization ability of the model in the image segmentation task by introducing the environment perception semantic labeling technology and the semantic steady diffusion technology, and overcomes a plurality of limitations in the prior art. The beneficial technical effects brought by the application are analyzed from multiple aspects as follows:

[0041] 1. Enhancing the generalization ability of the model to complex scenes

[0042] Through the semantic steady diffusion module, the anisotropic noise technology is used to suppress low-level texture features, while enhancing the expression of high-level semantic features, which significantly improves the adaptability of the model under different data domains such as illumination conditions and sensor resolution. Compared with the traditional method, the application can more effectively capture the semantic difference between categories, thereby improving the segmentation performance of the model in complex background or similar category scenes.

[0043] 2. Improving the class discrimination ability

[0044] By introducing the group semantic labeling contrast learning method, the application can significantly improve the compactness of the intra-class features and the separability of the inter-class features. The dynamically generated group labels effectively capture the semantic structure of different categories in the image, thereby enhancing the ability of the model to capture fine-grained semantic differences. Compared with the traditional fixed weight distribution method, the dynamic weighting mechanism of the application makes the model perform better in different category segmentation tasks.

[0045] 3. Efficiently using a small amount of new category data

[0046] In the second training stage, the model is fine-tuned through a small amount of new category support data, which can quickly adapt to the semantic distribution of the new category while maintaining the recognition ability of the original category. This efficient transfer learning mechanism makes the model perform better in dealing with small sample learning and expanding new categories.

[0047] 4. Improve feature diversity and expression ability

[0048] Anisotropic diffusion technology effectively enhances the diversity of features by dynamically adjusting the noise intensity. In high gradient areas, key structures such as edges are protected, and randomness is introduced in smooth areas to reduce the over-reliance of the model on low-level textures. This method preserves high-level semantic information and significantly improves the expressive power of the model in complex backgrounds.

[0049] 5. Solve the defects of the prior art

[0050] The present application can flexibly adjust the feature weight according to the context information, and strengthen the capture ability of semantic related features, so as to more efficiently adapt to the changes of different categories and scenes. In addition, through the anisotropic diffusion module, the low-level texture interference is suppressed, and the expression ability of high-level semantic features is improved, further enhancing the generalization performance of the model in complex backgrounds and cross-domain tasks, effectively solving the shortcomings of traditional methods.

[0051] In summary, through the optimization and innovative design of key modules in the image segmentation task, the present application not only improves the overall performance of the model, but also provides an efficient and robust solution for semantic segmentation in complex scenes and cross-domain tasks, with significant technical advantages and application value.

[0052] Additional aspects and advantages of the application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0053] The above and / or additional aspects and advantages of the present application will become apparent and be readily appreciated from the following description, including the accompanying drawings, in which:

[0054] Figure 1 is a flowchart of the prior small sample segmentation method;

[0055] Figure 2 is a network flow schematic diagram of the model of the present application. DETAILED DESCRIPTION

[0056] Embodiments of the present application are described in detail below, examples of which are shown in the accompanying drawings, in which the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below are exemplary and are intended only to explain the present application, and cannot be understood as limiting the present application.

[0057] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal communication between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0058] like Figure 2 As shown, the present invention provides a generalized small-sample image segmentation method. It primarily designs a semantic steady-state diffusion module that dynamically adjusts the intensity of anisotropic noise to suppress low-level texture interference and enhance the expression of high-level semantic features. Reducing noise intensity in high-frequency feature regions protects edge structure, while enhancing randomness in low-frequency feature regions increases feature diversity, thereby improving the model's robustness and generalization in complex backgrounds. A feature denoising strategy is also designed. A denoising network optimizes the diffusion-processed features, removing redundant information while preserving high-level semantic features. Secondly, a set of learnable group labels is introduced to address the limitations of existing techniques in modeling the relationship between base and new classes. By dynamically adjusting the relationship between features and labels, the model enhances its ability to model complex relationships between classes. A Multi-Gated Channel Mixture-of-Experts (MGC-MoE) output mechanism is also proposed to further improve the model's performance in multi-class segmentation tasks. By context-awarely adjusting feature weights, each group label acts as a dynamic "gate," focusing on the feature distribution associated with the current class or pixel. Through MGC-MoE, the model can flexibly handle the semantic associations between base classes and new classes, significantly improving multi-category segmentation performance.

[0059] The specific steps include:

[0060] Get image samples and use them as datasets;

[0061] Build a generalized small-sample semantic segmentation model and train it. The training is divided into two stages: the first training stage and the second training stage.

[0062] In the first training phase, the model is fully trained on the basic category data and background categories, and most of the categories in the dataset are used for the first training phase. The details are as follows:

[0063] Extract features:

[0064] Given an input image Use a pre-trained backbone network (such as a frozen version of ResNet50 or VGG16) to extract feature representations Where H and W are the height and width of the image, H×W and C represent the spatial dimension and channel dimension respectively. θ Mapping these features into a low-dimensional embedding space to obtain a set of compact features Where D represents the feature dimension after dimensionality reduction.

[0065] After extracting features, the semantic steady-state diffusion module is used to dynamically adjust the feature diffusion and noise of the feature map to enhance feature diversity, suppress low-level texture interference, and highlight high-level semantic category structure. Specifically,

[0066] This embodiment estimates the gradient strength by the change or difference in the eigenvalues ​​between local adjacent positions, which is used to quantify the change trend of the position of each feature point in the feature map, that is, the change in the eigenvalue of the feature map at a certain position and its neighborhood. The calculation formula of the gradient strength is as follows:

[0067] grad x =h i+1,j -h i,j ,grad y =h i,j+1 -h i,j

[0068]

[0069] Among them, h i,j is the eigenvalue at i,j, grad x is the rate of change of the eigenvalue in the horizontal direction (x direction), which represents the difference between the eigenvalues ​​of the current pixel and its right pixel. y It is the rate of change of the eigenvalue in the vertical direction (y direction), indicating the difference between the eigenvalues ​​of the current pixel and the pixel above it. It is the gradient strength, which quantifies the changing trend of the eigenvalue at the position (x, y). The larger the value, the more drastic the change at that point (such as the edge or significant feature area), and the smaller the value, the more gradual the change (such as the flat area).

[0070] By calculating the gradient, the local change intensity of each position in the feature map can be captured (such as strong local changes at the edge and weak changes in texture details). These changes are used to guide the subsequent noise addition strategy.

[0071] Noise is generated at each feature location. Specifically, a dynamic noise coefficient is generated based on the gradient amplitude at each feature location, dynamically adjusting the noise intensity based on local changes. When generating dynamic noise, the dynamic noise coefficient is inversely proportional to the gradient amplitude. In areas with large gradient amplitudes (such as edges), the noise coefficient c(x,y) is small, reducing noise interference and protecting key image structures. In areas with small gradient amplitudes (such as textured or uniform areas), the noise coefficient c(x,y) is large, enhancing feature diversity.

[0072] In this embodiment, the square of the gradient amplitude is used to calculate the noise coefficient c(x,y):

[0073]

[0074] Where c(x,y) is the noise coefficient, which is used to control the dynamic intensity of the noise. K is the smoothing parameter of the gradient amplitude, which is a constant and is used to control the influence of the gradient intensity on c(x,y).

[0075] Gradient strength can prevent high-frequency features (such as edge information) from being blurred, retaining high-level semantic information of the image, while allowing strong randomness to be applied to low-frequency areas, thereby improving the model's generalization ability for complex scenes.

[0076] Superimpose the generated dynamic noise onto the extracted feature map at each feature location:

[0077] h' (x,y) =h (x,y) +c(x,y)·noise

[0078] Among them, h' (x,y) is the feature after adding noise, h (x,y) is the extracted feature, and noise is random noise with a standard deviation of 1.

[0079] The diffused feature h' (x,y) Rearrange and pass it to the denoising network to remove the added noise, and finally obtain the optimized feature representation.

[0080] The denoising process is:

[0081] h=Relu(Conv2D2(Relu(Conv2D1(h')))

[0082] Conv2D is a two-dimensional convolution operation used to extract local features. The subscripts of Conv2D1() and Conv2D2() are to indicate that the two convolutions are different. Relu is an activation function that introduces nonlinearity to improve the expressiveness of the model. h is the feature map after denoising.

[0083] This embodiment introduces reconstruction loss and feature diversity loss to supervise the denoising and denoising processes to supervise the learning effect of the semantic steady-state diffusion module and help the model better focus on high-level semantic information and reduce excessive reliance on low-level textures.

[0084] Among them, the introduction of reconstruction loss is intended to ensure that the features after noise perturbation processing can still effectively restore the semantic structure and key details of the image. In the process of feature diffusion and noise addition, noise may damage certain structures of the image, especially when processing high-frequency features. Therefore, reconstruction loss is used to constrain the denoising network to restore the high-level information of the original image or feature map, while ensuring that key details are not lost during the denoising process. In this scheme, L2 loss (mean square error) is used as the metric for reconstruction loss to measure the difference between the original feature map and the denoised feature map. Specifically, the reconstruction loss is calculated as follows:

[0085]

[0086] Among them, hi is the original eigenvalue of the i-th feature map, h i is the eigenvalue of the feature map recovered after denoising, and H*W is the total number of elements in the feature map. By minimizing the reconstruction loss, the constrained model ensures that the high-level semantic information of the image is preserved while avoiding the loss of key information during the denoising process.

[0087] The introduction of feature diversity loss is to enable the model to better adapt to complex scenes and to encourage the model to introduce more randomness in low-gradient areas (such as textured areas or smooth areas). The purpose of this is to enhance the diversity of feature maps and avoid the model's over-reliance on low-level texture features. By introducing diversity, the model can focus more on high-level semantic information and improve its generalization ability for complex scenes. Feature diversity loss measures the diversity of features by calculating the variance of feature maps:

[0088]

[0089] where h i represents the eigenvalue in the feature map restored after denoising, μ f is the mean eigenvalue of the feature map restored after denoising, and H*W is the total number of elements in the feature map.

[0090] Next, the denoised feature map is input into the classification module for classification. In this embodiment, the denoised feature map is first group-labeled.

[0091] Intra-class feature compactness indicates the proximity between features with the same semantics, while inter-class feature separability indicates the distance between features with different semantics. To enhance feature representation, a group semantic label contrastive learning method is used to improve both intra-class compactness and inter-class separability of feature embeddings. This method uses contrastive learning to leverage learned group labels to bring intra-class semantic features closer together while separating inter-class semantic features. Each component is described in detail below.

[0092] Group semantic tag contrastive learning attempts to cluster similar semantic features through a set of learnable group tags, where each group tag represents a unique semantic object concept. In this embodiment, a set of learnable group semantic tags P = {p1, p2, ..., p k}, each tag p k Corresponding to a potential semantic category. For each pixel feature h ij , by calculating the pixel feature and group label p k The similarity of the pixels is obtained to obtain the degree of belonging to each group of labels, so that all pixel features are assigned to the labels. This is called soft assignment in this embodiment. During the assignment, a pixel feature is allowed to be assigned to multiple group labels at the same time, but with different weights or probabilities. The obtained soft assignment is A = {a ij,k} i＝1,…W,j＝1,…H,k＝1,...K , to enhance the compactness of intra-class features and the separability of inter-class features:

[0093]

[0094] Among them, a ij,k is the similarity between the noise-removed feature at pixel (i, j) and the kth tag, sim(·) is a similarity function, such as cosine similarity, τ is a temperature parameter used to control the sensitivity of the assignment probability, and h ij is the eigenvalue after removing noise at pixel point (i, j), and p k The current a ij,k The corresponding semantic tag, p k' The semantic labels for other categories are updated by weighted aggregation of pixel features assigned to each label:

[0095]

[0096] Among them, (a ij,k ·h ij ) refers to the similarity between the feature after removing noise at pixel point (i, j) and the kth mark and the pixel feature h ij Repolymerization.

[0097] Updated group label P = {p'1, p'2, ..., p' k} better reflects the semantic group structure in the image, as each group label represents a unique semantic object concept, thereby aiming to achieve intra-class compactness and inter-class feature separability.

[0098] Here, contrastive loss is used to further promote the distinction between labels, and the feature representation and group label of each pixel are optimized by maximizing the similarity of each pixel feature with the positive sample label while minimizing the similarity with the negative sample label. This example uses the following contrastive loss function to optimize the feature representation and group label of all pixels in a small batch:

[0099]

[0100] By minimizing the contrastive loss, the model is encouraged to bring intra-class semantic features closer together while separating inter-class semantic features. The module is able to assign similar features to similar group labels, thereby promoting semantic consistency within objects.

[0101] Images of the same category may have perspective or semantic differences. To enable the tag to perceive the environment and provide personalized semantic tags, global average pooling is performed on the feature map h after noise removal to generate a channel-level global personalized vector:

[0102]

[0103] h d,i,j is the eigenvalue of the pixel point (i, j) in channel d after noise removal, z d is the global personalized response of channel d, and the global description vector Z is input into a two-layer perceptron (MLP) to generate context-aware channel weights:

[0104] s=σ(W2·δ(W1·z))W1,W2 are the weight matrices of the first and second layer perceptrons, δ is the nonlinear activation function Relu, which introduces nonlinear characteristics to enhance the expressiveness of the model, and σ is the activation function of the output layer. s is normalized to [0,1] to indicate the relative importance of channel weights. The context weight s is dynamically applied to the aforementioned group labels P={p'1,p'2,...,p' k}, (where K represents the number of group labels and D represents the embedding dimension), generates personalized group labels P′ related to the current context of the feature map h after noise removal, and dynamically adapts to the semantic feature distribution of different scenes, thereby improving the unified segmentation performance of new classes and basic classes:

[0105]

[0106] Where ⊙ represents the element-wise channel weighting. The adjusted mark It is more consistent with the current semantic distribution and can match pixel features more accurately.

[0107] When performing image segmentation, this embodiment designs a Multi-Gated Channel Mixture-of-Experts (MGC-MoE) mechanism based on MMoE, which is a dynamic weighting mechanism that can dynamically adjust the weight of each channel according to the context of the feature map. This dynamic weighting method enables the model to more flexibly utilize feature information in segmentation tasks at different pixel positions and different categories. Each group label can be regarded as a "Gate", which is responsible for capturing the features of a specific semantic category. By dynamically generating weights through the Gate, the model can focus on features related to the current category or pixel, thereby improving the ability to distinguish categories.

[0108] Specifically, in MGC-MoE, each group tag is regarded as an independent Gate, which is used to capture the features of a specific semantic category. After the input feature map is processed by group tag, the personalized group tag is used to weight each channel in the feature map to obtain C refers to the number of categories. Through this dynamic weighting approach, the model can focus more on semantically relevant features of the current category or pixel position while suppressing interference from irrelevant features. The formula for this process is as follows:

[0109]

[0110] where ⊙ is the Hadamard product, Represents the features of channel d in the feature map after removing noise, It is a semantically personalized group tag used to dynamically adjust the weight.

[0111] In order to map channel-level feature information into category-level semantic distribution This embodiment introduces a 1×1 convolution operation on the weighted feature map. Each category has a total of D channels. The D channels are combined into a single value using weights to represent the probability of category C. This process can be expressed as:

[0112]

[0113] Where W d is the weight matrix that maps the D channel to 1, is the feature of the pixel at (i, j), O c (i, j) is the value of the pixel at (i, j), and the category probability distribution of each pixel position is obtained through softmax:

[0114]

[0115] in, the prediction result of the pixel position (i, j) in the category c.

[0116] Take the category with the highest probability at each pixel position as the prediction result of the pixel:

[0117]

[0118] The cross-entropy loss is used as the loss function of segmentation, which can effectively measure the difference between the predicted category probability distribution of the model and the real label:

[0119]

[0120] Therefore, the total loss of the generalized small sample semantic segmentation model in the embodiment is:

[0121]

[0122] where λ recon , λ div , λ CE , and λ contrastive are the weights of each loss term, controlling the contribution of each loss to the total loss.

[0123] In the second training stage, the model uses a small amount of new categories (i.e. the remaining categories in the dataset except for the categories involved in the first training stage) as the dataset to fine-tune the generalized small sample semantic segmentation model after the first training stage, and adapt to the semantic features of the new categories by updating the model structure and group labels. The goal of this stage is to quickly improve the model's performance on new categories using limited data while maintaining the ability to recognize original categories. The number of new categories in the second stage is small, usually only one / five images per new category. One image scenario is 1-shot generalized small sample segmentation, and five images scenario is 5-shot generalized small sample segmentation.

[0124] In the verification stage, the model no longer performs feature diffusion and noise enhancement operations, but directly applies the optimized feature extraction and classification module for prediction. The purpose of this stage is to evaluate the model's generalization ability and classification performance in a multi-category scenario.

[0125] Finally, the trained generalized small sample image segmentation model can be used to classify images that need to be segmented.

[0126] Embodiment Two

[0127] The present invention also provides a generalized small sample image segmentation system, including a data acquisition module, a processing module and a storage module for acquiring image samples; the data acquisition module is connected to the processing module and sends image samples to the processing module, the processing module and the storage module are communicatively connected, and the storage module is used to store at least one executable instruction, and the executable instruction enables the processing module to perform operations corresponding to the generalized small sample image segmentation method as described in Example 1 according to the image sample to segment the image.

[0128] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0129] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

Claims

1. A generalized small sample image segmentation method, characterized in that: The following steps are involved: Get image samples; Construct a generalized small-sample semantic segmentation model and train the generalized small-sample semantic segmentation model using the image samples: Extract feature maps from image samples in a generalized small-sample semantic segmentation model. After feature extraction, perform semantic steady-state diffusion on the feature maps: perform feature diffusion and noise on the feature maps. The steps are: calculate the change or difference of the feature values ​​between local adjacent positions to estimate the gradient strength, quantify the change trend of each feature point position in the feature map, and generate dynamic noise based on the gradient amplitude of each feature position. When generating dynamic noise, use the square of the gradient amplitude to calculate the dynamic noise coefficient, which is inversely proportional to the gradient amplitude; superimpose the generated dynamic noise on the original feature at each feature position: h' (x,y) =h (x,y) +c(x,y)·noise Among them, h' (x,y) is the feature after adding noise, h (x,y) is the extracted feature, noise is random noise with a standard deviation of 1; Rearrange the features after diffusion processing and remove the added noise to obtain the denoised feature map; Perform context-aware semantic tagging on the denoised feature map: Global average pooling is performed on the denoised feature map to generate a channel-level global personalized vector, which is then input into the perception layer to generate context-aware channel weights. Learnable group tags are generated for the denoised feature map, where each tag corresponds to a semantic category. The channel weights are applied to the learnable group tags to generate personalized group tags that are context-sensitive to the current input denoised feature map. The personalized group labels are used to dynamically weight each channel in the feature map, in the following steps: A dynamic weighting mechanism based on MMoE is designed, which takes each personalized group tag as a "Gate" to capture the features of a specific semantic category; each channel in the feature map is dynamically weighted by the Gate: after the input feature map is processed by the group tag, the personalized group tag is used to weight each channel in the feature map to obtain C refers to the number of categories, and the formula is as follows: where ⊙ is the Hadamard product, Represents the features of channel d in the feature map after removing noise, It is a semantically personalized group tag used to dynamically adjust the weight; The category of each pixel position is predicted to achieve image segmentation.

2. The generalized small sample image segmentation method according to claim 1, characterized in that: The denoising and denoising processes are supervised using reconstruction loss and feature diversity loss; The reconstruction loss uses mean square error as a metric to measure the difference between the original feature map and the denoised feature map; The feature diversity loss measures the diversity of features by calculating the variance of the denoised feature map.

3. The generalized small sample image segmentation method according to claim 1, characterized in that: The steps to generate learnable group labels are: Initialize a set of learnable group semantic labels, each label corresponds to a latent semantic category; Dynamically adjust the similarity between each pixel feature and each tag in the group tag, generate the degree of belonging of each pixel feature to each group tag, perform weighted aggregation on the pixel features assigned to each tag, and update the group tag.

4. The generalized small sample image segmentation method according to claim 3, characterized in that: When generating the degree of belonging of each pixel feature to each group label, a pixel feature is allowed to be assigned to multiple group labels at the same time, but with different weights or probabilities.

5. The generalized small sample image segmentation method according to claim 1, characterized in that: Contrastive loss is used to promote the discrimination between labels and optimize the feature representation of each pixel by maximizing the similarity of each pixel feature with the positive sample label while minimizing the similarity with the negative sample label.

6. The generalized small sample image segmentation method according to claim 1, characterized in that: Perform convolution on the weighted feature map to map the channel dimension to the number of categories; Calculate the category probability distribution of each pixel position, and take the category with the highest probability at each pixel position as the prediction result of the pixel.

7. The generalized small sample image segmentation method according to claim 1, characterized in that: The training is divided into the first training phase and the second training phase; In the first training phase, the generalized small-shot semantic segmentation model is trained on the base category data and background categories to learn the semantic features of the base categories; In the second training phase, the new categories are used as data sets to update the new categories in the generalized small-sample semantic segmentation model after the first training phase, and the basic categories and new categories are classified at the same time.

8. A generalized small sample image segmentation system, characterized by: It includes a data acquisition module, a processing module and a storage module for acquiring image samples; The data acquisition module is connected to the processing module and sends image samples to the processing module. The processing module is communicatively connected to the storage module. The storage module is used to store at least one executable instruction. The executable instruction enables the processing module to perform an operation corresponding to the generalized small sample image segmentation method according to any one of claims 1 to 7 according to the image sample to segment the image.

Images