A generalized small sample partition method and system for class relation mining

Abstract

The disclosure provides a generalized small sample segmentation method and system for class relationship mining, relating to the technical field of image segmentation, which performs generalized small sample segmentation on remote sensing images through a two-stage trained segmentation model; in the two-stage training, class relationship mining is performed using background consistency modeling technology and inter-class relationship mining technology; in the first stage of base class training based on a large number of labeled samples, the background consistency modeling technology is used to optimize the feature distribution between the background and the new class, and between the new classes; in the second stage of new class training based on a small number of labeled new class samples, the inter-class relationship mining technology is used to enhance the new class prototype using the base class prototype, and based on the base class prototype and the new class prototype, base class query features, new class query features and background features are generated for segmenting objects of the corresponding classes; the present application quickly adapts to new classes using a small number of samples, and efficiently completes the semantic segmentation task of remote sensing images by dealing with the problems of new classes and new scenes that may occur in remote sensing scenes.

Description

Technical Field

[0001] This disclosure relates to the field of image segmentation technology, specifically to a generalized few-sample segmentation method and system for class relationship mining. Background Technology

[0002] Semantic segmentation of remote sensing images is a technique that applies ordinary semantic segmentation tasks to remote sensing images. It aims to classify and label each pixel in a remote sensing image, accurately classifying different regions into predefined land cover categories. Due to the large viewing area and varying land cover scales of remote sensing images, segmenting objects of different sizes within different categories becomes more challenging. Furthermore, target objects in remote sensing images exhibit complex and variable characteristics. In different scene changes, due to environmental differences (such as geographical location and seasonal variations), the appearance of target objects may change significantly, and new categories of objects that were not previously labeled or defined in the scene may even be introduced.

[0003] When faced with new scenes and new object categories, traditional remote sensing semantic segmentation methods often struggle to quickly adapt to and segment these new scenes and object categories using only a small number of labeled samples. In this context, generalized few-sample segmentation of remote sensing images offers an effective solution, enabling the segmentation model to quickly adapt to the segmentation of new object categories using only a small number of new remote sensing images, while maintaining the model's ability to segment known categories.

[0004] The training process for generalized few-shot semantic segmentation tasks generally consists of two stages. The first stage trains the model based on a large amount of labeled data (known categories), enabling it to accurately segment base class objects while reserving space for future new categories (unknown categories). In the second stage, the model receives a small number of new category samples (unknown categories from the first stage) to learn semantic segmentation for these new categories, while simultaneously retaining its segmentation ability for the base classes. Generalized few-shot segmentation, through this two-stage optimization strategy, allows the model to quickly adapt to new categories, making it particularly suitable for scenarios involving changes in the geographical environment, such as the emergence of new buildings, road expansion, or changes in land features caused by natural disasters. However, remote sensing generalized few-shot semantic segmentation tasks still face many challenges:

[0005] 1) It is difficult to distinguish between multiple categories and scales of objects in remote sensing images.

[0006] Due to the unique shooting angles and scenes, remote sensing images differ significantly from natural images, and their object distribution is inconsistent with ordinary samples. Furthermore, the scale differences between different categories of objects, and even within the same category, are significant, requiring segmentation models to capture detailed information while adapting to complex scale variations. Although many remote sensing segmentation models incorporate multi-scale feature decoders to enhance their performance, balancing multi-scale feature fusion with effective identification of remote sensing objects remains a key challenge in practical applications.

[0007] 2) The semantic offset problem of background features.

[0008] Because the new object class lacks effective label support during the first stage of training, its features are labeled as background features, causing a bias in the model's processing of new class features and true background features (i.e., semantic offset). In the second stage, some pixels originally defined as background are labeled as new object classes, leading to confusion between the two stages regarding the definition of some pixels. This confusion makes the model prone to failing to handle the distribution differences between new classes and between new classes and the background in the first stage. In other words, the model primarily learns the semantic distribution of the base class in the first stage, while the feature information of the new class is not sufficiently trained and separated, especially when the semantic features of the new class are similar to those of the background. Therefore, accurately representing background features in the first stage while effectively learning new class features and avoiding confusion with background information becomes a key issue in generalized few-shot semantic segmentation.

[0009] 3) Inter-class object relationships are ignored.

[0010] Existing methods neglect the correlation between base classes and new classes. Exploring these inter-class object relationships can benefit the segmentation of new classes. Furthermore, exploring the correlation between features often aids the model's learning process. Remote sensing images have complex backgrounds, and objects of different categories often share similar structural and semantic information. Base class features can provide a gain for learning new class features. However, traditional generalized few-shot segmentation methods typically ignore the visual and semantic relationships between base and new classes, assuming that the new and old classes are completely independent and share no information. In reality, base class knowledge can significantly help in learning and segmenting new class features. Therefore, modeling the correlation between base and new classes becomes a key problem that needs to be solved in generalized few-shot segmentation of remote sensing.

[0011] Therefore, existing generalized few-sample segmentation techniques for remote sensing images suffer from problems such as difficulty in distinguishing multi-class and multi-scale targets in remote sensing images, background semantic shift, and neglect of the correlation between base class and new class, which affect the overall performance of remote sensing image semantic segmentation tasks. Summary of the Invention

[0012] To address the aforementioned issues, this disclosure proposes a generalized few-sample segmentation method and system for class relation mining. This method utilizes a small number of samples to quickly adapt to new classes, addressing potential new classes and scenarios in remote sensing scenes, and efficiently completing semantic segmentation tasks for remote sensing images.

[0013] According to some embodiments, the present disclosure adopts the following technical solutions:

[0014] A generalized small-sample segmentation method for class relation mining includes:

[0015] Acquire the remote sensing image to be segmented;

[0016] The segmentation model trained in two stages is used to perform generalized small sample segmentation of the remote sensing image to obtain the image segmentation result map corresponding to the remote sensing image.

[0017] In the two-stage training, background consistency modeling and inter-class relationship mining techniques are used to mine class relationships. In the first stage of base class training based on a large number of labeled samples, background consistency modeling is used to optimize the feature distribution between the background and the new class, and between the new classes, by utilizing background knowledge distillation. In the second stage of new class training based on a small number of labeled new class samples, inter-class relationship mining techniques are used to model the relationship between the base class prototype and the new class prototype, and the base class prototype is used to enhance the new class prototype. Based on the base class prototype and the enhanced new class prototype, base class query features, new class query features, and background features are generated for segmenting objects of the corresponding categories.

[0018] According to some embodiments, the present disclosure adopts the following technical solutions:

[0019] A generalized small-sample segmentation system for class relation mining includes:

[0020] The image acquisition module is configured to acquire the remote sensing image to be segmented;

[0021] The image segmentation module is configured to perform generalized small sample segmentation of the remote sensing image using a segmentation model trained in two stages, and obtain the image segmentation result map corresponding to the remote sensing image.

[0022] In the two-stage training, background consistency modeling and inter-class relationship mining techniques are used to mine class relationships. In the first stage of base class training based on a large number of labeled samples, background consistency modeling is used to optimize the feature distribution between the background and the new class, and between the new classes, by utilizing background knowledge distillation. In the second stage of new class training based on a small number of labeled new class samples, inter-class relationship mining techniques are used to model the relationship between the base class prototype and the new class prototype, and the base class prototype is used to enhance the new class prototype. Based on the base class prototype and the enhanced new class prototype, base class query features, new class query features, and background features are generated for segmenting objects of the corresponding categories.

[0023] According to some embodiments, the present disclosure adopts the following technical solutions:

[0024] A computer program product includes a computer program that, when executed by a processor, implements the generalized few-sample segmentation method for class relation mining.

[0025] According to some embodiments, the present disclosure adopts the following technical solutions:

[0026] A non-transitory computer-readable storage medium is provided for storing computer instructions, which, when executed by a processor, implement the generalized small sample segmentation method for class relation mining.

[0027] According to some embodiments, the present disclosure adopts the following technical solutions:

[0028] An electronic device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to perform a generalized few-sample segmentation method for class relation mining.

[0029] Compared with the prior art, the beneficial effects of this disclosure are as follows:

[0030] This disclosure provides a generalized few-sample segmentation method based on class relation mining. It captures information about objects at different scales in an image through a multi-scale feature decoder, improving the segmentation capability for targets at different scales. It employs background consistency modeling technology and utilizes background knowledge distillation to enhance the generalization of potential new classes in background features. Furthermore, it leverages inter-class relation mining technology to enhance new class prototypes by using information from base class prototypes, thereby improving the segmentation capability for new classes. The method demonstrates significant effectiveness in generalized few-sample semantic segmentation in remote sensing, providing new research ideas and directions for further exploration and application of deep learning technology in the field of remote sensing. Attached Figure Description

[0031] The accompanying drawings, which form part of this disclosure, are used to provide a further understanding of this disclosure. The illustrative embodiments of this disclosure and their descriptions are used to explain this disclosure and do not constitute an undue limitation of this disclosure.

[0032] Figure 1 This is a network framework diagram of the segmentation model in Example 1.

[0033] Figure 2 The flowchart for background consistency modeling in Example 1.

[0034] Figure 3 This is a flowchart of the inter-class relationship mining module in Example 1.

[0035] Figure 4 The image shows the evaluation results for Example 1. Detailed Implementation

[0036] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0037] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of this disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0038] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0039] Example 1

[0040] One embodiment of this disclosure provides a generalized few-sample segmentation method for class relation mining. This method utilizes a small number of samples to quickly adapt to new classes, addressing potential new class and scene issues in remote sensing scenarios, and efficiently completing semantic segmentation tasks for remote sensing images. The steps include:

[0041] Step S1: Obtain the remote sensing image to be segmented;

[0042] Step S2: Using the segmentation model trained in two stages, perform generalized small sample segmentation on the remote sensing image to obtain the image segmentation result map corresponding to the remote sensing image;

[0043] In the two-stage training, background consistency modeling and inter-class relationship mining techniques are used to mine class relationships. In the first stage of base class training based on a large number of labeled samples, background consistency modeling is used to optimize the feature distribution between the background and the new class, and between the new classes, by utilizing background knowledge distillation. In the second stage of new class training based on a small number of labeled new class samples, inter-class relationship mining techniques are used to model the relationship between the base class prototype and the new class prototype, and the base class prototype is used to enhance the new class prototype. Based on the base class prototype and the enhanced new class prototype, base class query features, new class query features, and background features are generated for segmenting objects of the corresponding categories.

[0044] The class relationship mining here includes background consistency modeling and inter-class relationship mining, which are used to optimize the feature distribution between background and new class, between new class and new class, and between base class prototype and new class prototype, respectively, ultimately improving the accuracy and performance of segmentation.

[0045] As one example, the specific implementation process is described below from the perspective of the structure and training of the segmentation model:

[0046] I. Problem Description

[0047] The Generalized Few-Shot Segmentation of Remote Sensing Images Based on Class Relation Mining (CRM) model employs a two-stage training strategy similar to the generalized few-shot semantic segmentation model. During training, the training dataset D is divided into two groups with no overlap in object categories; one group contains only the base class C. b The sample set D b and a set containing only the new class C n The sample set D n Base class C b It contains k categories with rich image data and corresponding labels, and a new class C. n It contains j categories, each with a small amount of image data and corresponding labels.

[0048] The first stage is the base class training stage. During this stage, the remote sensing generalized few-shot semantic segmentation method is consistent with the ordinary few-shot semantic segmentation method. The model trains on the base class sample set D. b The model is trained on the above platform, with the input being samples (i.e., remote sensing images) of the corresponding category and their labels. Where N is the total number of samples, at this stage, the model extracts base class semantic information from a large amount of data, aiming to learn good feature representation capabilities.

[0049] The second stage is the new class training stage, where the model uses the new class sample set D. n This involves registering the new class information into the model from the previous stage, forming a new classifier, and enabling the model to adapt to the new class C. n The segmentation task. At this point, each new class has only M pairs of image samples and their annotations. In this stage, the model uses a limited number of labeled samples to register new semantic segmentation knowledge into the model.

[0050] Through two-stage training, the model not only achieves efficient processing of base classes, but also flexibly extends to new classes, demonstrating strong adaptability under small sample conditions.

[0051] During the testing phase, the model is evaluated on remote sensing images in the test set. Unlike ordinary few-shot segmentation tasks that only focus on the performance of new classes, generalized few-shot segmentation tasks require the model to predict the labels of all base class and new class objects simultaneously, and to comprehensively evaluate the performance of base class and new class objects. During testing, the remote sensing images input to the model may contain objects of the new class, base class, or both, and the model cannot predict whether the input image will contain objects of the new class, base class, or both. The model needs to be able to predict C... b ∪C nThe ability to label all categories of objects follows the same input-output paradigm as semantic segmentation.

[0052] II. Structure and Training of the Segmentation Model

[0053] Network architecture of segmentation model as follows Figure 1 As shown, the overall network architecture includes a feature extractor, a multi-scale feature decoder (MFD), a consistent background modeling (CBM) module, an inter-class relation mining (IRM) module, a feature projection module, a base class classifier, and a new class classifier.

[0054] The system comprises the following components: a feature extractor that extracts multi-scale features from the input remote sensing image; a multi-scale feature fusion decoder that decodes and fuses the extracted multi-scale features to capture detailed information and semantic features of targets at different scales, generating decoded features; a background consistency modeling module that optimizes the feature distribution between background and new classes, and between new classes, using background knowledge distillation; an inter-class relationship mining module that models the relationship between base class prototypes and new class prototypes using inter-class relationship mining techniques, and uses base class prototypes to enhance new class prototypes; a feature projection module that generates base class query features, new class query features, and background features based on base class prototypes and enhanced new class prototypes; a base class classifier that performs object segmentation on the base class query features and background features; and a new class classifier that performs object segmentation on the new class query features and background features.

[0055] The CRM model employs a two-stage training strategy:

[0056] (1) Base class training process

[0057] First, the base class image x b After feature extractor θ e Then, encoded features are generated, i.e., multi-scale features are extracted.

[0058] Then, the multi-scale feature decoder θ, which includes a pyramid pooling module (PPM), is used. d The process of generating decoding features f involves the following steps: first, the multi-scale features enter the pyramid pooling module to generate segmentation features p, and then the multi-scale decoder θ... dThe segmentation feature p is refined by introducing features from the first three layers of the feature extractor layer by layer. Each layer of features is processed by the Conv-Bn-ReLU layer and then fused with the segmentation feature p. The low-resolution features are beneficial for learning large objects and their localization information, while the high-resolution features gradually supplemented by the multi-scale decoder are beneficial for learning small objects and their detailed information, thus obtaining a representation that is beneficial for multi-scale object segmentation.

[0059] Specifically, the model uses background consistency modeling to process the segmentation features p output by the pyramid pooling module. This technique copies a pre-trained encoder θ′ at the start of training. e , by feature extractor θ e and pre-trained encoder θ′ e Process the base class image x separately b The pyramid pooling feature p is then blurred, and p is constrained to θ′. e The background pixels of the output features are similar.

[0060] Secondly, after obtaining the decoded feature f, the model generates the base class prototype u through orthogonal initialization. base And project the decoded feature f onto the base class prototype u base Direction to obtain base class query features f base , where u base It is the base class prototype, obtained through training using orthogonal constraints.

[0061] Finally, the base class query feature f base The weakened background features f′ are input into the base class classifier to predict the base class objects. The weakened background features are obtained by subtracting the base class query features f′ from the decoded features f′. base get.

[0062] After the first stage of base class training, the feature extractor θ is obtained. e and multi-scale feature decoder θ d And the trained base class prototype u base This will be used for training new classes in the next stage.

[0063] (2) Training process for new classes

[0064] The model uses a small number of sample base class images x b and new class image x n Make fine adjustments, specifically:

[0065] First, the model inputs the samples into the frozen feature extractor θ. e and multi-scale feature decoder θ d To generate the decoding feature f, consistent with the initialization method of the base class prototype, the model uses orthogonal initialization to generate the new class prototype u. novelDuring training, it is constrained to remain orthogonal to prototypes of other classes, and u is used. base and u novel Projecting the decoded features f onto each feature yields the base class query features f. base And new category query features f novel Decode feature f minus base class query feature f base And new class query features f novel The weakened background feature f′ is obtained.

[0066] Subsequently, the model employs an inter-class relationship mining strategy, introducing a randomly initialized bias prototype u. biaa For the base class prototype u base Generate a perturbation to break its connection with the new class prototype u novel The orthogonal relationship is established to establish u base with u novel The connection, and through this relationship, strengthen u novel Generate the enhanced prototype of the new class u′ novel .

[0067] Finally, the model projects the decoded features f onto the base class prototype u. base and the enhanced new prototype u′ novel Direction, generating base class query features f base And new class query features f′ novel The base class classifier queries the base class feature f. base As input, the new class classifier uses the new class query feature f′ novel The background feature f′ is taken as input, and predictions are made separately. The prediction results of the two are combined to generate the final prediction.

[0068] The background consistency modeling module, feature projection module, inter-class relationship mining module, and loss function will be explained separately below.

[0069] 1. Background Consistency Modeling

[0070] In remote sensing generalized few-shot segmentation tasks, training samples typically contain both base class and new class objects. However, because the first stage of training only includes labels for base class objects, the locations of new class objects are labeled as background. This leads to a bias in the model's understanding of background features when it uses the same training samples again in the second stage to learn new class object segmentation, making the learning of new class features more difficult. This is known as the background semantic shift problem. To alleviate the semantic shift problem and uncover potential new classes in background features, this embodiment utilizes pre-trained encoder parameters to extract original background features and distills the corresponding background feature knowledge into the currently trainable features, achieving consistent modeling and enhancement of background features. This preserves generalization ability for subsequent learning of new class features.

[0071] The model uses the parameters of the pre-trained model as the initialization parameters for the feature extractor. The generated features exhibit strong discriminative power and generalization ability. However, in the first training stage, without constraints on the features, the model may mistakenly identify background features similar to those of the base class as base class features, leading to overfitting. Therefore, in the first training stage, the model is constrained to learn only the foreground features of the base class and their segmentation methods, while maintaining the consistency between the background features and the original background feature distribution. During training, the model maintains the correlation between pixels in the background features and uses the original background features to enhance the current background features, further improving its generalization ability. The model also improves the processing capability of new classes and background features by distilling the correlation of each pixel in the background from the pre-trained model into the current model. The process is as follows: Figure 2 As shown, specifically:

[0072] (1) Generate the current background feature p1 and the original background feature p2.

[0073] Before the first phase of training begins, CRM copies an initial pre-trained encoder θ′. b In background consistency modeling, because the pre-trained encoder is pre-trained using a classification task, its feature representation is relatively blurry for object contours and edges. Therefore, the pyramid pooling feature p undergoes downsampling and upsampling to minimize the impact on the spatial information of the pyramid pooling feature p. The pyramid pooling feature is mapped to a feature space with the same dimension as p" for consistency modeling. To reduce the impact on foreground object features, the model only processes background features, using image x b Using the corresponding label index, we extract the clean background features to obtain the current background feature p1 with dimension N×c, where N is the number of background pixels; similarly, we extract the original background feature p2 from the encoded features p” generated by the pre-trained encoder using the background index.

[0074] (2) Calculate the similarity matrix between the current background feature p1 and the original background feature p2.

[0075] After obtaining the current background feature p1 and the original background feature p2, calculate their similarity distributions to obtain similarity matrices sim1 and sim2, as shown in Formula 1:

[0076]

[0077] To enhance the model's ability to distinguish background features, the similarity matrix sim2 of the original background features is contrast-enhanced. During contrast enhancement, the mean u of the last dimension of sim2 is first calculated, as shown in Formula 2:

[0078]

[0079] Then, the mean of sim2 is subtracted and the cosine similarity is calculated to obtain the enhanced sim′2, as shown in Formula 3:

[0080]

[0081] (3) Minimize the distribution distance between the two similarity matrices by using mean square error constraint.

[0082] After obtaining sim1 and sim′2, in order to ensure the consistency of background features, i.e., to constrain the difference between p1 and p2 to be small, a mean squared error constraint is adopted to construct the background feature distillation loss. bg To constrain the minimization of the distribution distance between the similarity matrices sim1 and sim′2, as shown in Equation 4:

[0083]

[0084] Using the above method, relation distillation, a technique in knowledge distillation, is employed. In the first stage of training, the model not only focuses on the output of individual background pixels but also considers the relationships and similarities between pixels. Relationship distillation helps to better preserve the generalization of background pixels and improves the model's adaptability in background processing by focusing on the relationships between pixels. The background consistency modeling technique utilizes the generalization ability of the pre-trained model and uses pixel relationship distillation between background pixels to preserve the generalization of the trained model and alleviate the background semantic offset problem between new classes and background. By using upsampling and downsampling operations to blur the features of the trained model, the relation distillation of the pre-trained model is made to minimize the destruction of the segmentation ability of the trained model. At the same time, its relation constraints can enhance the discriminability and adaptability between new classes and between new classes and background. In contrast, in classical methods, the optimization of feature distribution between background and new classes is ignored.

[0085] 2. Feature projection module

[0086] To learn categorical features, the model learns categorical prototype representations using learnable parameters. The model generates base class prototypes u using orthogonal initialization. base and new class u novel Meanwhile, the model uses a multi-scale decoder to meet the needs of multi-scale features in remote sensing images, so as to better adapt to targets of different scales and obtain decoded features f.

[0087] To enable the prototype to acquire categorical features, CRM employs a prototype projection technique, projecting the feature vector f along the prototype vector direction to generate a new vector, as shown in Equation 5:

[0088]

[0089] Where u represents the prototype of all base classes and new classes.

[0090] After obtaining the decoding feature f and projecting it onto the prototype of each category, projection features similar to the corresponding category prototype are obtained. These features can be used by the classifier to segment objects of the corresponding category.

[0091] Subtracting the projected features of all categories from the original feature vector yields the corresponding background features, as shown in Formula 6:

[0092]

[0093] Where f′ represents the background feature, n is the number of categories, and f is the decoding feature.

[0094] 3. Inter-class relationship mining module

[0095] After retrieving all base class and background features, the model uses a classifier to classify these query features. This method effectively trains the base class prototype u. base However, if the new class prototype u is handled in the same way novel , then it is believed that u base and u novel Orthogonal. But if the two are completely orthogonal, it means u base with u novel There is no overlap in the feature space; the two classes only differ and do not establish any association. Although orthogonal constraints enhance the distinguishability between the base class and the new class, they ignore the potential commonalities and connections between them. To fully utilize the base class information and improve the segmentation capability of the new class, this embodiment maintains u base Orthogonal, and designed as follows Figure 3 The inter-class relationship mining module shown is used to model u novel with u base The relationships between them.

[0096] Specifically, CRM assumes that each new class has a different similarity to different base class prototypes, and that the new class prototype can be supplemented and enhanced with information from the base class prototype. To explore the relationship between the two, the model introduces a similarity to the base class prototype u. base Bias prototype u with the same dimension and random initialization bias The bias prototype u bias Add base class prototype u base In this process, the orientation of the base class prototype is changed, thus obtaining the enhanced base class prototype u′. base , making u′ base With the new class prototype u novel The orthogonality is no longer present, which provides the possibility of exploring the correlation between the new class and the base class prototype, as shown in Equation 7:

[0097] u′ base =u base +ubias (7)

[0098] To model the relationship between the base class and the prototype of the new class, the model calculates the prototype u of the new class. novel and the enhanced base class prototype u′ base The similarity matrix is ​​used to obtain the similarity relationship between each new class prototype and the base class prototype, as shown in Formula 8:

[0099]

[0100] Where affin represents the similarity relationship between the base class and the prototype of the new class. The larger the value, the higher the correlation between the corresponding base class and the prototype of the new class. T is the matrix transpose.

[0101] To make u novel From u base The corresponding category features are obtained and the prototype representation is enhanced. After obtaining the affin, the affin matrix is ​​used to evaluate u. novel Enhancements are made, as shown in Formula 9:

[0102] u′ novel =W(u) novel +affin×u base )+B (9)

[0103] Where W represents the weight parameters of the linear layer, and B represents the bias matrix.

[0104] The model is enhanced with a new class prototype u′ novel Then, the decoded feature f′ is projected onto the corresponding prototype direction to generate the enhanced new class query feature f′. novel As shown in Formula 10:

[0105]

[0106] Finally, the model will incorporate the new class query features F n ′ ovel The data is then combined with background features f′ and input into a new classifier for segmentation prediction.

[0107] Base class features contain information beneficial to the learning of new classes, while classical methods ignore the correlation between base class features and new class features. Inter-class relationship mining, through the base class prototype u base Introducing the bias prototype u biasBy modeling the relationship between the new class and its base classes, a relationship matrix between the prototypes of the new class and the base classes is effectively established. Information beneficial for segmenting the new class is dynamically extracted from the base class prototypes. This ensures that the enhanced new class prototype not only contains unique information about the new class but also beneficial information from other base classes, thus significantly improving the segmentation ability of the new class. This inter-class relationship mining module enriches the model's expressive power and enhances its ability to process new classes.

[0108] 4. Loss Function

[0109] In the first stage of training, the model uses the pixel-wise cross-entropy loss function as the segmentation loss function. In the semantic segmentation task, this loss function constrains the model's optimization process by calculating the difference between the predicted class and the true class of each pixel. The expression is as follows:

[0110]

[0111] Where h is the height of the image, w is the width of the image, and p is the predicted class probability at the corresponding location.

[0112] In addition, the model uses background features to distill the loss. bg The model ensures consistency of background features. Simultaneously, it introduces prototype orthogonality loss to ensure that feature prototypes of different categories remain orthogonal, thereby improving the model's ability to distinguish new classes. The expression is as follows:

[0113]

[0114] Where u represents the set of orthogonal prototypes, and r is the total number of orthogonal prototypes.

[0115] Combining the commonly used segmentation loss function in segmentation models seg The model achieves good base class segmentation performance and generalization ability to new classes during the basic training phase, as shown in the following expression:

[0116] loss = loss seg +loss bg +loss orth (13)

[0117] In the second training phase, since the background consistency was effectively modeled in the first phase, ensuring the generalizability of background features, the model can efficiently learn new class features in the background. At this point, the background no longer contains objects of unknown class, so consistency constraints are no longer necessary. Therefore, the model is further fine-tuned by combining pixel-wise segmentation loss and orthogonal loss. This process ensures that the model can still maintain accurate semantic segmentation capabilities when adding new classes. The loss function expression is as follows:

[0118] loss = loss seg+loss orth (14)

[0119] This embodiment compares and evaluates the segmentation effect of this method, and the results are as follows: Figure 4 As shown:

[0120] As shown in the figure, this method demonstrates high accuracy and robustness in remote sensing image segmentation tasks, effectively distinguishing and identifying major land cover categories such as buildings, roads, water bodies, and vegetation, while preserving edge details and spatial structure information well. Even in complex scenes, this method maintains good segmentation consistency and class discriminability, indicating its good generalization ability across different land cover types. This provides more reliable technical support for the refined analysis of generalized small-sample remote sensing images.

[0121] Example 2

[0122] One embodiment of this disclosure provides a generalized small-sample segmentation system for class relation mining, comprising:

[0123] The image acquisition module is configured to acquire the remote sensing image to be segmented;

[0124] The image segmentation module is configured to perform generalized small sample segmentation of the remote sensing image using a segmentation model trained in two stages, and obtain the image segmentation result map corresponding to the remote sensing image.

[0125] In the two-stage training, background consistency modeling and inter-class relationship mining techniques are used to mine class relationships. In the first stage of base class training based on a large number of labeled samples, background consistency modeling is used to optimize the feature distribution between the background and the new class, and between the new classes, by utilizing background knowledge distillation. In the second stage of new class training based on a small number of labeled new class samples, inter-class relationship mining techniques are used to model the relationship between the base class prototype and the new class prototype, and the base class prototype is used to enhance the new class prototype. Based on the base class prototype and the enhanced new class prototype, base class query features, new class query features, and background features are generated for segmenting objects of the corresponding categories.

[0126] Example 3

[0127] One embodiment of this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the generalized few-sample segmentation method for class relation mining.

[0128] Example 4

[0129] One embodiment of this disclosure provides a non-transitory computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the generalized small sample segmentation method for class relationship mining.

[0130] Example 5

[0131] One embodiment of this disclosure provides an electronic device, including a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to execute a generalized small sample segmentation method for class relationship mining.

[0132] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0133] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0134] While the specific embodiments of this disclosure have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of this disclosure. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of this disclosure are still within the scope of protection of this disclosure.

Claims

1. A generalized small sample partitioning method for class relationship mining, characterized in that, The method comprises: acquiring a remote sensing image to be segmented; performing remote sensing image generalized small sample segmentation on the remote sensing image through a two-stage trained segmentation model to obtain an image segmentation result image corresponding to the remote sensing image; wherein, in the two-stage training, a background consistency modeling technology and an inter-class relationship mining technology are used for class relationship mining, in a first-stage base class training based on a large number of labeled samples, the background consistency modeling technology is used, background knowledge distillation is used, and the feature distribution of the background and new classes and the feature distribution between new classes are optimized, in a second-stage new class training based on a small number of labeled new class samples, the inter-class relationship mining technology is used, the relationship between the base class prototype and the new class prototype is modeled, and the base class prototype is used to enhance the new class prototype, based on the base class prototype and the enhanced new class prototype, base class query features, new class query features and background features are generated for segmenting objects of corresponding classes; the segmentation model comprises a feature extractor, a multi-scale feature fusion decoder, a background consistency modeling module, an inter-class relationship mining module, a feature projection module, a base class classifier and a new class classifier; the feature extractor extracts multi-scale features of the input remote sensing image; the multi-scale feature fusion decoder decodes and fuses the extracted multi-scale features, captures the detail information and semantic features of targets of different scales, and generates decoding features; the background consistency modeling module uses background knowledge distillation to optimize the feature distribution of the background and new classes and the feature distribution between new classes; the inter-class relationship mining module uses the inter-class relationship mining technology to model the relationship between the base class prototype and the new class prototype, and uses the base class prototype to enhance the new class prototype; the feature projection module generates base class query features, new class query features and background features for the decoding features based on the base class prototype and the enhanced new class prototype; the base class classifier performs object segmentation on the base class query features and the background features; the new class classifier performs object segmentation on the new class query features and the background features.

2. The generalized small sample partitioning method for category relationship mining of claim 1, wherein, The background consistency modeling module uses a pre-trained encoder to extract original background features, extracts current background features from the segmentation features generated by the multi-scale feature fusion decoder, calculates a similarity matrix of the original background features and a similarity matrix of the current background features, adopts background feature distillation loss to constrain the distribution distance of the two similarity matrices to be minimized, and optimizes the feature distribution of the background and new classes and the feature distribution between new classes.

3. The generalized small sample partitioning method for category relationship mining of claim 1, wherein, The inter-class relationship mining module calculates the similarity matrix of the base class prototype and the new class prototype, obtains the similarity relationship between each new class prototype and the base class prototype, enhances the new class prototype using the similarity matrix, and obtains the enhanced new class prototype.

4. The generalized small sample partitioning method for category relationship mining of claim 1, wherein, The base class query features, the new class query features and the background features generated for the decoding features are specifically: projecting the decoding features to the base class prototype direction to obtain the base class query features; projecting the decoding features to the enhanced new class prototype direction to obtain the new class query features; subtracting the base class query features of all classes from the decoding features to obtain the corresponding background features.

5. The generalized small sample size partitioning method for category relationship mining of claim 1, wherein, The loss function of the base class training comprises a pixel-wise cross-entropy loss, a prototype orthogonality loss and a background feature distillation loss. The loss function of the new class training includes a pixel-wise cross-entropy loss and a prototype orthogonality loss.

6. A generalized small sample partitioning system for class relationship mining, comprising: The generalized small sample segmentation method based on class relationship mining according to any one of claims 1-5 comprises: An image acquisition module configured to acquire a remote sensing image to be segmented; An image segmentation module configured to perform remote sensing image generalized small sample segmentation on the remote sensing image by using the two-stage trained segmentation model to obtain an image segmentation result image corresponding to the remote sensing image. In the two-stage training, the class relationship mining is performed using a background consistency modeling technique and a class relationship mining technique. In the first-stage base class training based on a large number of labeled samples, the background consistency modeling technique is used to optimize the feature distribution between the background and the new class and between the new classes by using background knowledge distillation. In the second-stage new class training based on a small number of labeled new class samples, the class relationship mining technique is used to model the relationship between the base class prototype and the new class prototype, and the base class prototype is used to enhance the new class prototype. Based on the base class prototype and the enhanced new class prototype, base class query features, new class query features, and background features are generated for segmenting objects of corresponding classes.

7. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the generalized small sample segmentation method based on class relationship mining according to any one of claims 1-5.

8. A non-transitory computer-readable storage medium, comprising: The non-transitory computer readable storage medium is used to store computer instructions, which are executed by the processor to implement the generalized small sample segmentation method based on class relationship mining according to any one of claims 1-5.

9. An electronic device, comprising: The computer program is executed by the processor to implement the generalized small sample segmentation method based on class relationship mining according to any one of claims 1-5. The computer program is executed by the processor to implement the generalized small sample segmentation method based on class relationship mining according to any one of claims 1-5. The computer program is executed by the processor to implement the generalized small sample segmentation method based on class relationship mining according to any one of claims 1-5.

Images