Generalized small sample image segmentation method and system

Abstract

The invention provides a generalized small sample image segmentation method and system, and the method mainly suppresses low-level texture interference and enhances the expression of high-level semantic features through carrying out the feature diffusion noise adding and denoising of a feature map. A learnable group mark is introduced, a personalized group mark related to the context of the de-noised feature map is generated by dynamically adjusting the relationship between the features and the mark, the modeling ability for the complex relationship between categories is enhanced, and each channel in the feature map is dynamically weighted by using the personalized group mark, so that the complexity of the feature map is improved. And predicting the category of each pixel position to realize image segmentation. According to the method, the overall performance of the model is improved, an efficient and robust solution is provided for semantic segmentation in complex scenes and cross-domain tasks, and the method has remarkable technical advantages and application value.

Description

Technical Field

[0001] The present invention relates to the field of machine learning, and in particular to a generalized small sample image segmentation method and system. Background Art

[0002] As an important task in computer vision, semantic segmentation aims to assign semantic labels to each pixel in an image. With the rapid development of deep neural networks, especially convolutional neural networks (CNNs) and visual transformers, semantic segmentation has made significant progress in performance. Classic methods such as fully convolutional networks (FCNs) have successfully promoted the development of semantic segmentation technology by introducing multi-scale contextual information and attention mechanisms, and have achieved remarkable results. However, these methods are highly dependent on large-scale pixel-level annotated data, so they are difficult to directly expand when facing new categories. They usually require a large amount of annotated data, which increases time and cost.

[0003] In order to deal with this problem, Few-Shot Segmentation (FSS) came into being. The goal of FSS is to perform semantic segmentation when there are only a few labeled samples. It is usually done by first fully training on the basic categories and then generalizing to new categories using a small number of labeled samples. However, traditional FSS methods often lead to a decrease in the performance of the basic categories when generalizing to new categories. And the assumption of FSS is that the support image and the query image contain the same new category, such as Figure 1 As shown in Figure 1, this requires a lot of manpower to process the dataset, and each time only focuses on the segmentation of a new category in the query image, which does not conform to the application scenarios in the real world. To overcome this limitation, Generalized Few-Shot Segmentation (GFSS) was proposed.

[0004] The goal of generalized few-shot semantic segmentation is to segment both base classes and novel classes simultaneously through a single inference without the need for paired support samples during testing, thereby better simulating actual application scenarios. Generalized few-shot segmentation is a semantic segmentation task that aims to accurately classify target pixels in an image while distinguishing between base classes and novel classes. Specifically, the task provides a large amount of labeled data for base classes and a small number of labeled samples for novel classes, as well as a background class that represents pixels that do not belong to any target class. Generalized few-shot segmentation requires the model to have strong learning capabilities on base classes, while being able to quickly expand knowledge through limited new class samples, thereby distinguishing all target classes and backgrounds.

[0005] Generalized small-shot semantic segmentation usually adopts a two-stage training framework. In the first stage, the model is fully trained on the base category data and background categories to learn the semantic features of the base category. In the second stage, the model updates the segmentation head with a small amount of support data for the new category, so that it can simultaneously predict the pixel classification of the base category, the new category, and the background. This method aims to improve the segmentation performance of the model on all categories by combining basic knowledge and new knowledge.

[0006] At present, a variety of technical solutions have emerged in the field of generalized small sample segmentation. For example, the Context-Aware Prototype Learning (CAPL) method enriches the feature representation of new categories by utilizing the contextual information in the support samples and query samples, thereby improving the segmentation performance. However, CAPL fails to fully consider the relationship between the new class and the base class, which may lead to the destruction of the base class prototype, thereby affecting the segmentation performance of the base class. Another method is the Projection onto Orthogonal Prototypes (POP) framework. POP avoids the influence of the base class features when learning new classes by learning a series of orthogonal prototypes, each of which corresponds to a specific semantic category. POP reduces the interference of new class learning on the base class by freezing the base class prototype and ensuring the orthogonality between prototypes when learning new classes. In addition, POP represents the background by measuring the residual of feature projection, thereby effectively solving the "semantic drift" problem in the new class learning process.

[0007] However, although the existing small sample segmentation and generalized small sample segmentation technologies have made some progress in performance improvement, there are still many deficiencies in dealing with complex scenes and maintaining the balance between basic and new class segmentation performance:

[0008] First, the existing technologies are insufficient in terms of the performance trade-off between base classes and new classes. For example, the CAPL method enriches the feature representation of new classes through contextual information, but it is easy to destroy the prototype structure of the base class, resulting in a decrease in the segmentation performance of the base class. On the other hand, POP reduces the interference of new class learning on the base class through orthogonality constraints, but its strategy of fixing the prototype of the base class lacks flexibility in diversified scenarios and is difficult to dynamically adapt to the complex semantic associations between categories. Secondly, the existing technologies have limitations in modeling the relationship between categories. Both CAPL and POP fail to fully capture the complex similarity relationship between the base class and the new class, especially in the scene of semantically similar categories, where category confusion is prone to occur. This deficiency makes it difficult for the model to effectively distinguish semantically similar categories in multi-category segmentation tasks. In addition, the existing technologies show insufficient adaptability in complex backgrounds and scenes with rich details. In images containing high-frequency features (such as edges and detail information) and low-frequency features (such as textures and uniform areas), context modeling methods such as CAPL are easily disturbed by background noise, while POP's strategy based on residual modeling background may ignore important fine-grained semantic information in the background, resulting in reduced segmentation accuracy. Finally, existing methods have limited dynamic feature adjustment and generalization capabilities. Methods such as CAPL and POP rely on fixed feature processing strategies and lack dynamic adjustment capabilities, making it difficult to dynamically optimize feature distribution according to different scenarios. In addition, in new class learning, existing technologies have insufficient expansion capabilities and are difficult to efficiently adapt to the semantic features of new classes with a small amount of supporting data.

[0009] Overall, the existing technologies have significant deficiencies in terms of performance balance between basic classes and new classes, adaptability to complex scenarios, category relationship modeling, and dynamic feature adjustment capabilities. Summary of the invention

[0010] In order to overcome the defects in the above-mentioned prior art, an object of the present invention is to provide a generalized small sample image segmentation method and system.

[0011] In order to achieve the above-mentioned object of the present invention, the present invention provides a generalized small sample image segmentation method, comprising the following steps:

[0012] Get image samples;

[0013] Construct a generalized small sample semantic segmentation model, and use the image samples to train the generalized small sample semantic segmentation model:

[0014] Extract feature maps from image samples in the generalized small sample semantic segmentation model, and perform semantic steady-state diffusion on the feature maps after feature extraction: perform feature diffusion and noise addition on the feature maps, rearrange the features after diffusion processing, and remove the added noise to obtain the denoised feature maps;

[0015] Perform context-aware semantic tagging on the denoised feature map: perform global average pooling on the denoised feature map to generate a global personalized vector at the channel level, and input it into the perception layer to generate context-aware channel weights; generate a learnable group tag for the denoised feature map, each tag in the group tag corresponds to a semantic category, and apply the channel weight to the learnable group tag to generate a personalized group tag related to the context of the denoised feature map currently input;

[0016] The personalized group label is used to dynamically weight each channel in the feature map, predict the category of each pixel position, and achieve image segmentation.

[0017] In this generalized small sample image segmentation method, semantic steady-state diffusion is used to enable the model to retain high-level semantic information of the image, while diversity is introduced to enhance the adaptability to complex scenes. The denoising process further enhances the robustness and adaptability of the model, making it perform better in small sample and generalized small sample segmentation tasks. At the same time, the feature expression ability is enhanced through environmental perception semantic labeling, and the intra-class compactness and inter-class separation of features are improved. Finally, by weighting each channel in the feature map, the model can focus on features related to the current category or pixel, thereby improving the category distinction ability.

[0018] Optionally, the steps of performing feature diffusion noise addition on the feature map are: calculating the change or difference degree of the feature values ​​between local adjacent positions to estimate the gradient strength, quantifying the change trend of each feature point position in the feature map, and generating dynamic noise according to the gradient amplitude of each feature position, and superimposing the generated dynamic noise on the original feature at each feature position.

[0019] This optional solution performs better in texture suppression because isotropic Gaussian blur may blur edges and other high-level semantic information. Therefore, in this embodiment, anisotropic noise addition is used to dynamically adjust the feature map to perform feature noise diffusion.

[0020] Optionally, when generating dynamic noise, the dynamic noise coefficient is inversely proportional to the gradient amplitude.

[0021] This option makes the noise in high-gradient areas smaller, avoiding damage to key structures; the noise in low-gradient areas is larger, increasing feature diversity. Introducing randomness in smooth areas helps reduce the model's over-reliance on low-level textures. By reducing the influence of low-level textures, the model can focus more on category-related high-level semantic features, improving the model's generalization ability for complex scenes.

[0022] Optionally, the denoising and denoising processes are supervised using reconstruction loss and feature diversity loss;

[0023] The reconstruction loss uses mean square error as a metric to measure the difference between the original feature map and the denoised feature map;

[0024] The feature diversity loss measures the diversity of features by calculating the variance of the denoised feature map.

[0025] By minimizing the reconstruction loss, the constrained model can ensure that the high-level semantic information of the image is preserved while avoiding the loss of key information during the denoising process. By introducing feature diversity loss, the model can focus more on high-level semantic information and improve the generalization ability of complex scenes.

[0026] Optionally, the steps to generate learnable group labels are:

[0027] Initialize a set of learnable group semantic tags, each tag corresponds to a latent semantic category;

[0028] Dynamically adjust the similarity between each pixel feature and each tag in the group tag, generate the degree of belonging of each pixel feature to each group tag, perform weighted aggregation on the pixel features assigned to each tag, and update the group tag.

[0029] Optionally, when generating the degree of belonging of each pixel feature to each group label, one pixel feature is allowed to be simultaneously assigned to multiple group labels but with different weights or probabilities.

[0030] Optionally, a contrastive loss is used to promote discrimination between labels and optimize the feature representation of each pixel by maximizing the similarity of each pixel feature with the positive sample label while minimizing the similarity with the negative sample label.

[0031] Optionally, a dynamic weighting mechanism based on MMoE is designed to treat each personalized group tag as a "Gate" to capture the features of a specific semantic category and dynamically weight each channel in the feature map through the Gate;

[0032] Perform convolution on the weighted feature map to map the channel dimension to the number of categories;

[0033] Calculate the category probability distribution of each pixel position, and take the category with the highest probability at each pixel position as the prediction result of the pixel.

[0034] Optionally, the training is divided into a first training phase and a second training phase;

[0035] In the first training phase, the generalized small-shot semantic segmentation model is trained on the base category data and background categories to learn the semantic features of the base category to learn the base category;

[0036] In the second training stage, the new categories are used as data sets to update the new categories in the generalized small-sample semantic segmentation model after the first training stage, and the basic categories and new categories are classified at the same time.

[0037] The present application also proposes a generalized small sample image segmentation system, including a data acquisition module, a processing module and a storage module for acquiring image samples;

[0038] The data acquisition module is connected to the processing module and sends image samples to the processing module. The processing module is communicatively connected to the storage module. The storage module is used to store at least one executable instruction. The executable instruction enables the processing module to perform operations corresponding to the above-mentioned generalized small sample image segmentation method according to the image sample to segment the image.

[0039] The beneficial effects of the present invention are:

[0040] The present invention significantly improves the feature expression and generalization capabilities of the model in image segmentation tasks by introducing environment-aware semantic labeling technology and semantic steady-state diffusion technology, overcoming many limitations of the prior art. The following analyzes the beneficial technical effects brought by the present invention from multiple aspects:

[0041] 1. Enhance the model’s generalization ability for complex scenarios

[0042] Through the semantic steady-state diffusion module, anisotropic noise addition technology is used to suppress low-level texture features, while enhancing the expression of high-level semantic features, significantly improving the model's adaptability under different data domains such as lighting conditions and sensor resolution. Compared with traditional methods, this invention can more effectively capture the semantic differences between categories, thereby improving the segmentation performance of the model in complex backgrounds or scenes of similar categories.

[0043] 2. Improve the ability to distinguish categories

[0044] By introducing the group semantic labeling contrast learning method, the present invention can significantly improve the compactness of intra-class features and the separability of inter-class features. The dynamically generated group labels effectively capture the semantic structure of different categories in the image, thereby enhancing the model's ability to capture fine-grained semantic differences. Compared with the traditional fixed weight allocation method, the dynamic weighting mechanism of the present invention enables the model to perform better in segmentation tasks of different categories.

[0045] 3. Efficiently utilize small amounts of new category data

[0046] In the second training phase, the model is fine-tuned with a small amount of new category support data, which allows it to quickly adapt to the semantic distribution of new categories while maintaining the ability to recognize the original categories. This efficient transfer learning mechanism enables the model to perform better when dealing with few-sample learning and expanding new categories.

[0047] 4. Improve feature diversity and expression capabilities

[0048] Anisotropic diffusion technology effectively enhances the diversity of features by dynamically adjusting the noise intensity. It protects key structures such as edges in high-gradient areas and introduces randomness in smooth areas, reducing the model's over-reliance on low-level textures. This approach retains high-level semantic information and significantly improves the model's expressiveness in complex backgrounds.

[0049] 5. Solve the defects of existing technology

[0050] The present invention can flexibly adjust feature weights according to contextual information and enhance the ability to capture semantically relevant features, thereby more efficiently adapting to changes in different categories and scenes. In addition, the anisotropic diffusion module suppresses low-level texture interference and improves the expressiveness of high-level semantic features, further enhancing the generalization performance of the model in complex backgrounds and cross-domain tasks, effectively solving the shortcomings of traditional methods.

[0051] In summary, the present invention not only improves the overall performance of the model through optimization and innovative design of key modules in image segmentation tasks, but also provides an efficient and robust solution for semantic segmentation in complex scenes and cross-domain tasks, which has significant technical advantages and application value.

[0052] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0054] Figure 1 It is a flow chart of the existing small sample segmentation method;

[0055] Figure 2 It is a network flow diagram of the model of the present invention. DETAILED DESCRIPTION

[0056] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0057] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal connection between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0058] like Figure 2 As shown, the present invention provides a generalized small sample image segmentation method, which mainly designs a semantic steady-state diffusion module, suppresses low-level texture interference and enhances the expression of high-level semantic features by dynamically adjusting the intensity of anisotropic noise. Reduce the noise intensity in the high-frequency feature area to protect the edge structure, enhance the randomness in the low-frequency feature area to increase the feature diversity, thereby improving the robustness and generalization ability of the model in complex backgrounds, and design a feature denoising strategy, optimize the features after diffusion processing through the denoising network, remove redundant information and retain high-level semantic features. Secondly, a set of learnable group tags is introduced to solve the limitation problem of the existing technology in modeling the relationship between the basic class and the new class. By dynamically adjusting the relationship between the feature and the tag, the model can enhance the modeling ability of the complex relationship between the categories. A multi-gated channel expert mixture output mechanism (Multi-Gated Channel Mixture-of-Experts, MGC-MoE) is also proposed to further improve the performance of the model in multi-category segmentation tasks. By adjusting the feature weights through context awareness, each group tag acts as a dynamic "Gate" to focus on the feature distribution related to the current category or pixel. Through MGC-MoE, the model can flexibly handle the semantic associations between base classes and new classes, significantly improving multi-category segmentation performance.

[0059] The specific steps include:

[0060] Get image samples and use them as data sets;

[0061] A generalized small sample semantic segmentation model is constructed and trained. The training is divided into two stages: the first training stage and the second training stage.

[0062] In the first training phase, the model is fully trained on the basic category data and background categories, and most of the categories in the dataset are used in the first training phase. The details are as follows:

[0063] Extract features:

[0064] Given an input image Use a pre-trained backbone network (such as a frozen version of ResNet50 or VGG16) to extract feature representations Where H and W are the height and width of the image, H×W and C represent the spatial dimension and channel dimension respectively. Then, through the projector h θ Mapping these features into a low-dimensional embedding space results in a compact set of features Where D represents the feature dimension after dimensionality reduction.

[0065] After extracting features, the semantic steady-state diffusion module is used to dynamically adjust the feature diffusion and noise of the feature map to enhance feature diversity, suppress low-level texture interference, and highlight high-level semantic category structure. Specifically,

[0066] This embodiment estimates the gradient strength by the change or difference of the eigenvalues ​​between local adjacent positions, so as to quantify the change trend of the position of each feature point in the feature map, that is, the change of the eigenvalues ​​of the feature map at a certain position and its neighborhood. The calculation formula of the gradient strength is as follows:

[0067] grad x =h i+1,j -h i,j ,grad y =h i,j+1 -h i,j

[0068]

[0069] Among them, h i,j is the eigenvalue at i,j, grad x is the rate of change of the eigenvalue in the horizontal direction (x direction), indicating the difference between the eigenvalues ​​of the current pixel and its right pixel, grad y It is the rate of change of the eigenvalue in the vertical direction (y direction), indicating the difference between the eigenvalue of the current pixel and the pixel above it. It is the gradient strength, which quantifies the changing trend of the eigenvalue at the position (x, y). The larger the value, the more drastic the change at that point (such as the edge or significant feature area), and the smaller the value, the more gradual the change (such as the flat area).

[0070] Through gradient calculation, the local change intensity of each position in the feature map can be captured (such as strong local changes at the edge, weak changes in texture details, etc.), and these changes are used to guide the subsequent noise addition strategy.

[0071] Generate noise at each feature position. Specifically, a dynamic noise coefficient is generated based on the gradient amplitude of each feature position to dynamically adjust the noise intensity according to local changes. When generating dynamic noise, the dynamic noise coefficient is inversely proportional to the gradient amplitude. In areas with large gradient amplitudes (such as edges), the noise coefficient c(x,y) is small, reducing noise interference and protecting key image structures. In areas with small gradient amplitudes (such as textures or uniform areas), the noise coefficient c(x,y) is large, enhancing the diversity of features.

[0072] In this embodiment, the square of the gradient amplitude is used to calculate the noise coefficient c(x, y):

[0073]

[0074] Where c(x,y) is the noise coefficient, which is used to control the dynamic intensity of the noise, and K is the smoothing parameter of the gradient amplitude, which is a constant and is used to control the influence of the gradient intensity on c(x,y).

[0075] The gradient strength can prevent high-frequency features (such as edge information) from being blurred, retaining high-level semantic information of the image, while allowing strong randomness to be applied to low-frequency areas, thereby improving the model's generalization ability for complex scenes.

[0076] Superimpose the generated dynamic noise onto the extracted feature map at each feature location:

[0077] h' (x,y) =h (x,y) +c(x,y)·noise

[0078] Among them, h' (x,y) is the feature after adding noise, h (x,y) is the extracted feature, and noise is a random noise with a standard deviation of 1.

[0079] The diffused feature h' (x,y) Rearrange and pass it to the denoising network to remove the added noise, and finally obtain the optimized feature representation.

[0080] The denoising process is:

[0081]

[0082] Conv2D is a two-dimensional convolution operation used to extract local features. The subscripts of Conv2D1() and Conv2D2() are to indicate that the two convolutions are different. Relu is an activation function to introduce nonlinearity to improve the model's expressiveness. It is the feature map after denoising.

[0083] This embodiment introduces reconstruction loss and feature diversity loss to supervise the denoising and denoising processes to supervise the learning effect of the semantic steady-state diffusion module and help the model better focus on high-level semantic information and reduce excessive reliance on low-level textures.

[0084] Among them, the introduction of reconstruction loss is intended to ensure that the features after noise perturbation processing can still effectively restore the semantic structure and key details of the image. In the process of feature diffusion and noise addition, noise may damage certain structures of the image, especially when processing high-frequency features. Therefore, reconstruction loss is used to constrain the denoising network to restore the high-level information of the original image or feature map, while ensuring that key details are not lost during the denoising process. In this scheme, L2 loss (mean square error) is used as the metric of reconstruction loss to measure the difference between the original feature map and the denoised feature map. Specifically, the reconstruction loss is calculated as follows:

[0085]

[0086] Among them, hi is the original eigenvalue of the i-th feature map, is the eigenvalue of the feature map restored after denoising, and H*W is the total number of elements in the feature map. By minimizing the reconstruction loss, the constrained model can ensure that the high-level semantic information of the image is preserved while avoiding the loss of key information during the denoising process.

[0087] The introduction of feature diversity loss is to enable the model to better adapt to complex scenes, so as to encourage the model to introduce more randomness in low-gradient areas (such as textured areas or smooth areas). The purpose of this is to enhance the diversity of feature maps and avoid the model's over-reliance on low-level texture features. By introducing diversity, the model can focus more on high-level semantic information and improve its generalization ability for complex scenes. Feature diversity loss measures the diversity of features by calculating the variance of feature maps:

[0088]

[0089] in represents the eigenvalue in the feature map restored after denoising, μ f is the mean eigenvalue of the feature map restored after denoising, and H*W is the total number of elements in the feature map.

[0090] Next, the denoised feature map is input into a classification module for classification. In this embodiment, the denoised feature map is first group-labeled.

[0091] Intra-class feature compactness indicates the closeness between features with the same semantics, while inter-class feature separability indicates the distance between features with different semantics. In order to enhance the feature expression capability, a group semantic label contrastive learning method is used to improve the intra-class compactness and inter-class separability of feature embedding. This method uses contrastive learning to bring the intra-class semantic features closer and separate the inter-class semantic features using the learned group labels. Each component will be described in detail below.

[0092] Group semantic tag contrastive learning attempts to group similar semantic features together through a set of learnable group tags, where each group tag represents a unique semantic object concept. In this embodiment, a set of learnable group semantic tags P = {p1, p2, ..., p k}, each tag p k Corresponding to a potential semantic category. For each pixel feature h ij , by calculating the pixel feature and the group label p k The similarity of the pixels is obtained to obtain the degree of belonging to each group of labels, so that all pixel features are assigned to the labels. This is called soft assignment in this embodiment. During the assignment, a pixel feature is allowed to be assigned to multiple group labels at the same time, but with different weights or probabilities. The obtained soft assignment is A = {a ij,k} i＝1,"W,j＝1,"H,k＝1,…K , to enhance the compactness of intra-class features and the separability of inter-class features:

[0093]

[0094] Among them, a ij,k is the similarity between the noise-removed feature at pixel (i, j) and the kth tag, sim(·) is a similarity function, such as cosine similarity, and τ is a temperature parameter used to control the sensitivity of the assignment probability. is the eigenvalue after removing noise at pixel point (i, j), and p k The current a ij,k The corresponding semantic tag, p k' The group label is updated by weighted aggregation of pixel features assigned to each label:

[0095]

[0096] Among them, (a ij,k ·h ij ) refers to the similarity between the feature after removing noise at the pixel point (i, j) and the kth mark and the pixel feature h ij Repolymerization.

[0097] Updated group label P = {p'1, p'2, ..., p' k} better reflects the semantic group structure in the image, as each group label represents a unique semantic object concept, thereby aiming to achieve intra-class compactness and inter-class feature separability.

[0098] Here, contrast loss is used to further promote the distinction between labels, and the feature representation and group labeling of each pixel are optimized by maximizing the similarity of each pixel feature with the positive sample label and minimizing the similarity with the negative sample label. This embodiment uses the following contrast loss function to optimize the feature representation and group labeling of all pixels in a small batch:

[0099]

[0100] By minimizing the above contrastive loss, the model is encouraged to bring intra-class semantic features closer together while separating inter-class semantic features. The module is able to assign similar features to similar group labels, thereby promoting semantic consistency within objects.

[0101] There may be differences in perspective or semantics between images of the same category. In order to make the tag aware of the environment and provide personalized semantic tags, the feature map after noise removal is Perform global average pooling to generate a channel-level global personalized vector:

[0102]

[0103] is the eigenvalue of the pixel (i, j) in channel d after noise removal, z d is the global personalized response of channel d. The global description vector Z is input into a two-layer perceptron (MLP) to generate context-aware channel weights:

[0104] s=σ(W2·δ(W1·z))

[0105] W1, W2 are the weight matrices of the first and second layer perceptrons, δ is the nonlinear activation function Relu, which introduces nonlinear characteristics to enhance the model's expressiveness, and σ is the activation function of the output layer. Normalize s to [0, 1] to indicate the relative importance of channel weights. Dynamically apply context weight s to the aforementioned group marker P = {p'1, p'2, ..., p' k}, (where K represents the number of group tags and D represents the embedding dimension) to generate a feature map similar to the current one after noise removal. The context-dependent personalized group label P′ dynamically adapts to the semantic feature distribution of different scenarios, thereby improving the unified segmentation performance of new and basic classes:

[0106]

[0107] Where ⊙ represents the element-wise channel weighting. The adjusted mark It is more consistent with the current semantic distribution and can match pixel features more accurately.

[0108] When performing image segmentation, this embodiment designs a multi-gated channel expert mixture mechanism (Multi-Gated Channel Mixture-of-Experts, MGC-MoE) based on MMoE, which is a dynamic weighting mechanism that can dynamically adjust the weight of each channel according to the context of the feature map. This dynamic weighting method enables the model to more flexibly utilize feature information in segmentation tasks at different pixel locations and different categories. Each group label can be regarded as a "Gate", which is responsible for capturing the features of a specific semantic category. By dynamically generating weights through the Gate, the model can focus on features related to the current category or pixel, thereby improving the ability to distinguish categories.

[0109] Specifically, in MGC-MoE, each group tag is regarded as an independent Gate to capture the features of a specific semantic category. After the input feature map is processed by the group tag, the personalized group tag is used to weight each channel in the feature map to obtain C refers to the number of categories. Through this dynamic weighting method, the model can focus more on the semantically relevant features of the current category or pixel position, while suppressing the interference of irrelevant features. The formula for this process is as follows:

[0110]

[0111] Where: is the Hadamard product, It represents the characteristics of channel d in the feature map after removing noise, It is a semantically personalized group tag used to dynamically adjust the weight.

[0112] In order to map channel-level feature information to category-level semantic distribution This embodiment introduces a 1×1 convolution operation on the weighted feature map. Each category has a total of D channels. The D channels are combined into one value through weights to represent the probability of category C. This process can be expressed as:

[0113]

[0114] Where W d is the weight matrix that maps the D channel to 1, is the feature of the pixel at (i, j), O c (i, j) is the value of the pixel at (i, j), and the category probability distribution of each pixel position is obtained through softmax:

[0115]

[0116] in, is the prediction result of pixel position (i, j) in category c.

[0117] Take the category with the highest probability at each pixel position as the prediction result of the pixel:

[0118]

[0119] Using cross entropy loss as the loss function for segmentation, it can effectively measure the difference between the category probability distribution predicted by the model and the actual label:

[0120]

[0121] In summary, the total loss of the generalized small sample semantic segmentation model in this embodiment is:

[0122]

[0123] Among them, λ recon , div , CE , contrastive are the weights of each loss item, controlling the contribution of each loss to the total loss.

[0124] In the second training phase, the model uses a small number of new categories (i.e., the remaining categories in the dataset except those involved in the first training phase) as the dataset to fine-tune the generalized few-shot semantic segmentation model that has undergone the first training phase, and adapts to the semantic features of the new categories by updating the model structure and group labels. The goal of this phase is to use limited data to quickly improve the model's performance on new categories while maintaining the ability to recognize the original categories. The number of new categories in the second phase is very small, usually each new category has only one / five pictures, one scene is 1-shot generalized few-shot segmentation, and five scenes are 5-shot generalized few-shot segmentation.

[0125] In the validation phase, the model no longer performs enhancement operations such as feature diffusion and noise addition, but directly applies the optimized feature extraction and classification modules for prediction. The purpose of this phase is to evaluate the generalization ability and classification performance of the model in multi-category scenarios.

[0126] Finally, the trained generalized small sample image segmentation model can be used to classify the images that need to be segmented.

[0127] Embodiment 2

[0128] The present invention also provides a generalized small sample image segmentation system, comprising a data acquisition module, a processing module and a storage module for acquiring image samples; the data acquisition module is connected to the processing module, and sends image samples to the processing module, the processing module and the storage module are communicatively connected, and the storage module is used to store at least one executable instruction, and the executable instruction enables the processing module to perform operations corresponding to the generalized small sample image segmentation method as recorded in Example 1 according to the image sample to segment the image.

[0129] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0130] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A generalized small sample image segmentation method, characterized in that: The following steps are involved: Get image samples; Construct a generalized small sample semantic segmentation model, and use the image samples to train the generalized small sample semantic segmentation model: Extract feature maps from image samples in the generalized small sample semantic segmentation model, and perform semantic steady-state diffusion on the feature maps after feature extraction: perform feature diffusion and noise addition on the feature maps, rearrange the features after diffusion processing, and remove the added noise to obtain the denoised feature maps; Perform context-aware semantic tagging on the denoised feature map: perform global average pooling on the denoised feature map to generate a global personalized vector at the channel level, and input it into the perception layer to generate context-aware channel weights; generate a learnable group tag for the denoised feature map, each tag in the group tag corresponds to a semantic category, and apply the channel weight to the learnable group tag to generate a personalized group tag related to the context of the denoised feature map currently input; The personalized group label is used to dynamically weight each channel in the feature map, predict the category of each pixel position, and achieve image segmentation.

2. The generalized small sample image segmentation method according to claim 1, characterized in that: The steps of feature diffusion noise addition to the feature map are as follows: calculate the change or difference degree of the feature value between local adjacent positions to estimate the gradient strength, quantify the change trend of each feature point position in the feature map, generate dynamic noise according to the gradient amplitude of each feature position, and superimpose the generated dynamic noise on the original feature at each feature position.

3. The generalized small sample image segmentation method according to claim 1, characterized in that: When generating dynamic noise, the dynamic noise coefficient is inversely proportional to the gradient amplitude.

4. The generalized small sample image segmentation method according to claim 1, characterized in that: The reconstruction loss and feature diversity loss are used to supervise the denoising and de-noising processes; The reconstruction loss uses mean square error as a metric to measure the difference between the original feature map and the denoised feature map; The feature diversity loss measures the diversity of features by calculating the variance of the denoised feature map.

5. The generalized small sample image segmentation method according to claim 1, characterized in that: The steps to generate learnable group labels are: Initialize a set of learnable group semantic tags, each tag corresponds to a latent semantic category; Dynamically adjust the similarity between each pixel feature and each tag in the group tag, generate the degree of belonging of each pixel feature to each group tag, perform weighted aggregation on the pixel features assigned to each tag, and update the group tag.

6. The generalized small sample image segmentation method according to claim 5, characterized in that: When generating the degree of belonging of each pixel feature to each group label, one pixel feature is allowed to be assigned to multiple group labels at the same time, but with different weights or probabilities.

7. The generalized small sample image segmentation method according to claim 1, characterized in that: Contrastive loss is used to promote the discrimination between labels and optimize the feature representation of each pixel by maximizing the similarity of each pixel feature with the positive sample label while minimizing the similarity with the negative sample label.

8. The generalized small sample image segmentation method according to claim 1, characterized in that: Design a dynamic weighting mechanism based on MMoE, treat each personalized group tag as a "Gate" to capture the features of a specific semantic category, and dynamically weight each channel in the feature map through the Gate; Perform convolution on the weighted feature map to map the channel dimension to the number of categories; Calculate the category probability distribution of each pixel position, and take the category with the highest probability at each pixel position as the prediction result of the pixel.

9. The generalized small sample image segmentation method according to claim 1, characterized in that: The training is divided into the first training phase and the second training phase; In the first training phase, the generalized small-shot semantic segmentation model is trained on the base category data and background categories to learn the semantic features of the base category to learn the base category; In the second training stage, the new categories are used as data sets to update the new categories in the generalized small-sample semantic segmentation model after the first training stage, and the basic categories and new categories are classified at the same time.

10. A generalized small sample image segmentation system, characterized in that: It includes a data acquisition module, a processing module and a storage module for acquiring image samples; The data acquisition module is connected to the processing module and sends image samples to the processing module. The processing module is communicatively connected to the storage module. The storage module is used to store at least one executable instruction. The executable instruction enables the processing module to perform operations corresponding to the generalized small sample image segmentation method as described in any one of claims 1 to 9 according to the image sample to segment the image.

Images