Generalized small sample image segmentation method, system, medium and equipment

Abstract

The invention discloses a generalized small sample image segmentation method and system, a medium and equipment. The generalized small sample image segmentation method comprises a base class learning stage and a new class learning stage, the base class learning stage comprises the following steps: extracting multi-scale image features of a remote sensing image based on a pyramid visual deformer backbone network; performing super-pixel segmentation on the remote sensing image to generate a shape feature map; fusing the multi-scale image features with the shape feature map to obtain fused features; supervision is carried out based on a self-reconstruction constraint mechanism, and an overall network, namely a base class learning network, at the stage is supervised; the new class learning stage comprises the steps of performing mask initialization on the fusion features based on a base class knowledge mining mechanism to obtain a base class prototype and a new class prototype, updating an affinity matrix based on the base class prototype and the new class prototype, and updating the new class prototype based on the updated affinity matrix and the base class prototype; and reasoning based on the base prototype and the updated new prototype to obtain a segmented image of the remote sensing image.

Description

Technical Field

[0001] The present invention belongs to the technical field of image segmentation, and in particular relates to a generalized small sample image segmentation method, system, medium and equipment. Background Art

[0002] The statements herein merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] As carriers of observational data, remote sensing images play a key role in numerous applications. Semantic segmentation aims to distinguish the geospatial categories of all locations in a given image. However, training an ideal neural network is labor-intensive and resource-intensive, and the resulting model suffers from fragile generalization. Specifically, the labeling cost of the tens of thousands of densely labeled training samples required for segmentation is extremely high; trained models can only output known categories and are difficult to extend to prediction tasks involving unseen categories. These unfavorable factors have hindered the further development of remote sensing semantic segmentation.

[0004] A simple approach to alleviate this problem is to leverage few-shot semantic segmentation (FSS), adapting the model using a small amount of support annotations for unseen / novel categories. However, Tian et al., in their work "Generalized few-shot semantic segmentation" presented at CVPR 2022, pointed out a significant flaw that hinders the practical application of these methods: existing FSS research assumes that support images contain categories present in the query image, which can lead to expensive manual screening. In parallel, Tian et al. proposed a new task called generalized few-shot semantic segmentation (GFSS). Generalized few-shot semantic segmentation relaxes the strong assumption that support images are of the same category as the query image. Unlike traditional few-shot segmentation, which only supports image-defined categories, GFSS employs a support set that includes multiple novel categories, enabling the model to simultaneously segment both base and novel categories in the query image. This means that under this new paradigm, there is no need to provide support images with the same target category as the query image. Overall, the field of generalized few-shot semantic segmentation has made remarkable research progress in recent years. Despite this, most existing work only deals with the analysis and processing of natural images and has not been further extended to the more challenging field of remote sensing images.

[0005] Traditional generalized small-sample semantic segmentation methods have difficulty achieving ideal results in remote sensing segmentation due to the differences between remote sensing images and natural images. By analyzing the special properties of remote sensing images, it is found that there are three main reasons why traditional generalized small-sample semantic segmentation networks perform poorly in this field: first, compared with natural images, the higher resolution of remote sensing images poses challenges, and previous networks were not designed for high-resolution data, resulting in poor results; second, the particularity of remote sensing content exacerbates the difficulty. Specifically, compared with natural images, the differences between categories such as grassland and wasteland, rivers and roads in remote sensing images are smaller. When such similar categories are classified as base classes and new classes respectively, the model has difficulty distinguishing them, resulting in blurred boundaries between new classes and base classes.

[0006] The inventors discovered that the existing generalized few-shot segmentation (GFSS) method has the following technical bottlenecks in remote sensing image processing:

[0007] (1) Traditional network architectures are not optimized for the high-resolution characteristics of remote sensing images, resulting in the prominent problem of missed detection of small-scale targets (such as vehicles and roads);

[0008] (2) Due to the fixed shooting angle of remote sensing images, similar land feature categories (such as grassland and wasteland, river and road) have weak differences in spectral and texture characteristics, making it difficult for existing models to effectively distinguish them;

[0009] (3) During the base class training phase, all new categories are uniformly labeled as background, which leads to confusion in learning new class features and limits the generalization ability of the model. Summary of the Invention

[0010] The purpose of the present invention is to overcome the deficiencies in the above-mentioned prior art and to provide a generalized small sample image segmentation method, system, medium and equipment, which can effectively process large-scale remote sensing images, have better recognition capabilities for small-scale targets, enhance the network's perception of remote sensing targets, and at the same time prevent confusion of new category features, improve inter-class distinction, and strengthen the network's discrimination ability, making the segmented image more accurate and effective and the class boundaries clearer.

[0011] In order to achieve the above object, the present invention is implemented through the following technical solutions:

[0012] On the one hand, the technical solution of the present invention provides a generalized small sample image segmentation method, comprising: a base class learning stage and a new class learning stage;

[0013] The base class learning stage includes: extracting multi-scale image features of remote sensing images based on a pyramid visual deformer backbone network; performing superpixel segmentation on the remote sensing images to generate a shape feature map; fusing the multi-scale image features with the shape feature map to obtain a fused feature; and performing supervision based on a self-reconstruction constraint mechanism, and updating the overall network of this stage, i.e., the base class learning network, based on the supervision.

[0014] The new class learning stage includes: based on the base class knowledge mining mechanism, masking the fusion features to initialize the base class prototype and the new class prototype, updating the affinity matrix based on the base class prototype and the new class prototype, and updating the new class prototype based on the updated affinity matrix and the base class prototype; and reasoning based on the base class prototype and the updated new class prototype to obtain a segmented image of the remote sensing image.

[0015] In at least one embodiment, the self-reconstruction constraint mechanism is used for supervision, and the overall network of this stage, i.e., the base class learning network, is updated based on supervision, including: after obtaining the fusion features, obtaining a restored image from the fusion features based on the self-reconstruction constraint, and supervising the restored image and the original input image; segmenting the fusion features to obtain a segmented image, and supervising the segmented image and the labeled image; and updating the base class learning network based on the above two types of supervision.

[0016] In at least one embodiment, the architecture of the pyramid visual deformer includes four stages, each stage consisting of a block embedding and encoding layer; the input features of each stage are divided into blocks of a set size, and after being processed by the encoder, the stage features are output to obtain multi-scale image features.

[0017] In at least one embodiment, performing superpixel segmentation on the remote sensing image to generate a shape feature map specifically includes:

[0018] The remote sensing image is segmented into superpixels based on the SLIC algorithm; the contour boundary of each superpixel is extracted to obtain a set of boundary points; the curvature of each boundary point is calculated and the curvature value sequence is used as the superpixel shape feature to generate a shape feature map.

[0019] In at least one embodiment, the curvature is calculated as follows:

[0020]

[0021] Where Cur represents the curvature estimation function, P ij is the superpixel boundary point, N ij P ij The neighborhood of P ij For the neighborhood point (x ij ,y ij ) coordinate difference product, and ε is a numerical stability constant.

[0022] In at least one embodiment, the affinity matrix A is calculated as follows:

[0023]

[0024] Where <·,·> represents the dot product between prototype vectors, and the dimension of affinity matrix A is [n novel ,n base ], whose element A ij Is a scalar, representing the prototype of the i-th new class With the j-th base class prototype The affinity score between and are all vectors of dimension C.

[0025] In at least one embodiment, the updated new class prototype is obtained by element-by-element matrix multiplication and addition operations:

[0026]

[0027] Where, P Ni Initialize the prototype of the new class obtained by a small number of labeled new class samples, P B For the base class prototype, A ij is the affinity matrix.

[0028] On the other hand, the technical solution of the present invention further provides a generalized small sample image segmentation system, comprising:

[0029] The base class learning module is configured to: extract multi-scale image features of remote sensing images based on a pyramid visual deformer backbone network; perform superpixel segmentation on the remote sensing images to generate a shape feature map; fuse the multi-scale image features with the shape feature map to obtain a fused feature; perform supervision based on a self-reconstruction constraint mechanism, and update the overall network at this stage, i.e., the base class learning network, based on the supervision;

[0030] The new class learning module is configured as follows: based on the base class knowledge mining mechanism, the fusion features are masked and initialized to obtain the base class prototype and the new class prototype; based on the base class prototype and the new class prototype, the affinity matrix is updated; based on the updated affinity matrix and the base class prototype, the new class prototype is updated; based on the base class prototype and the updated new class prototype, reasoning is performed to obtain the segmented image of the remote sensing image.

[0031] The beneficial effects of the technical solution of the present invention are as follows:

[0032] 1) The present invention improves remote sensing segmentation performance through a two-stage design: (a) In the base class learning stage, shape representation embedding is introduced to enhance the network's perception of remote sensing targets. At the same time, self-reconstruction constraints are used to prevent confusion of new class features and improve inter-class discrimination, solving the problem that the existing model is difficult to distinguish between similar land object categories; (b) In the new class learning stage, a base class knowledge mining mechanism is designed, and the base class prototype is used to optimize the new class prototype representation, thereby strengthening the network's discrimination ability, solving the problems of new class feature learning confusion and limited model generalization ability in the existing technology.

[0033] 2) The present invention uses PVT as the backbone network to extract multi-scale image features, which can process high-resolution images and obtain multi-layer effective features, better cope with the segmentation task of emphasizing the high-resolution characteristics of remote sensing images, and avoid the problem that traditional network architecture is not optimized for the high-resolution characteristics of remote sensing images, resulting in prominent problems of missed detection of small-scale targets (such as vehicles and roads).

[0034] 3) Compared with existing methods, the proposed method can effectively process large-scale remote sensing images, has better recognition capabilities for small-sized targets, and is more advantageous in distinguishing easily confused foreground areas from prototypes. These characteristics are beneficial to improving the recall rate of segmented targets by avoiding target omission (misidentifying new classes as background). BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0036] Figure 1 This is a schematic diagram of the overall network architecture of a generalized small sample image segmentation method of the present invention;

[0037] Figure 2 It is a schematic diagram of the semantic segmentation results of remote sensing images using a generalized small sample image segmentation method of the present invention. DETAILED DESCRIPTION

[0038] It should be noted that the following detailed description is illustrative and is intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0039] Explanation of terms:

[0040] Pyramid Vision Transformer (PVT): The Pyramid Vision Transformer is a visual transformer architecture used for computer vision tasks such as image recognition and segmentation. The core idea of PVT is to introduce a pyramid structure into the Transformer model, which can extract feature representations of different resolutions from images.

[0041] DIaM: DIaM (Distilled Information Maximization) is a baseline method for the Generalized Few-Shot Semantic Segmentation (GFSS) task, published in CVPR 2023. The core idea of DIaM is to use the theoretical framework of mutual information to dynamically integrate information from new and old categories.

[0042] Patch embedding: Patch embedding refers to the Transformer structure compressing image blocks into feature vectors through convolution and linear mapping when processing image data.

[0043] SLIC algorithm: SLIC is a superpixel segmentation algorithm that can segment an image into a series of small areas with similar local features and coherent boundaries, called "superpixels".

[0044] mIoU: mIoU is a commonly used performance evaluation metric in image segmentation tasks. It is calculated by calculating the intersection over union (IoU) ratio (IoU) of the predicted region and the ground-truth region for each category. The mIoU is then averaged across all categories. In layman's terms, mIoU measures the degree of spatial overlap between the predicted image and the ground-truth annotations; higher values indicate better segmentation performance.

[0045] As introduced in the background technology, the purpose of the present invention is to overcome the shortcomings of the above-mentioned existing technologies and provide a generalized small sample image segmentation method, system, medium and equipment, which can effectively process large-scale remote sensing images, have better recognition capabilities for small-scale targets, enhance the network's perception of remote sensing targets, and at the same time prevent confusion of new category features, improve inter-class distinction, and strengthen the network's discrimination ability, making the segmented image more accurate and effective and the class boundaries clearer.

[0046] Example 1

[0047] In a typical implementation of the present invention, this embodiment discloses a generalized small sample image segmentation method, comprising: a base class learning phase and a new class learning phase;

[0048] The base class learning stage includes: extracting multi-scale image features of remote sensing images based on the pyramid visual deformer backbone network; performing superpixel segmentation on the remote sensing images to generate shape feature maps; fusing the multi-scale image features with the shape feature maps to obtain fused features; performing supervision based on the self-reconstruction constraint mechanism and updating the network based on the supervision;

[0049] The new class learning stage includes: based on the base class knowledge mining mechanism, masking the fusion features to obtain the base class prototype and the new class prototype, updating the affinity matrix based on the base class prototype and the new class prototype, and updating the new class prototype based on the updated affinity matrix and the base class prototype; reasoning based on the base class prototype and the updated new class prototype to obtain the segmented image of the remote sensing image.

[0050] The generalized small sample image segmentation method is described in detail below with reference to specific embodiments.

[0051] like Figure 1 As shown, the training paradigm of this embodiment follows the DIaM format and consists of two main phases: base class learning and new class learning. In the base class learning phase, shape representation embedding and self-reconstruction constraints are introduced to enhance the network's ability to perceive remote sensing targets. In the new class learning phase, a base class knowledge mining mechanism is designed to utilize the discriminative knowledge accumulated during the base class training phase to optimize the prototype representation of new classes.

[0052] Specifically, in the base class training stage, for remote sensing image I rs First, the pyramid visual deformer (hereinafter referred to as PVT) is used as the feature extraction backbone network for image feature extraction. The PVT architecture consists of four stages, each of which consists of a block embedding and encoding layer. The input feature F i Divided into P i After being processed by the encoder, the i-th stage feature X is output. i , ultimately obtaining multi-scale features {X1, X2, X3, X4}. Using PVT as the backbone network to extract multi-scale image features can process high-resolution images and obtain multi-layer effective features. This makes it better suited for segmentation tasks that prioritize the high-resolution characteristics of remote sensing images. This avoids the problem of traditional network architectures not being optimized for the high-resolution characteristics of remote sensing images, which can lead to the over-detection of small-scale objects (such as vehicles and roads).

[0053] In remote sensing image analysis, target shape features serve as strong discriminant indicators because the viewing angle of remote sensing images is relatively fixed, and the shape information of different categories is more stable than that of natural images (for example, rivers often appear as long, narrow strips, and ships typically have pointed features to distinguish them from vehicles). This characteristic is particularly important in generalized small-sample learning, significantly reducing the model's dependence on the number of labeled samples. By mapping the shape features of different target categories into shape space, the model's ability to recognize new categories can be effectively enhanced.

[0054] Therefore, in this embodiment, the remote sensing image I is firstly processed using the SLIC algorithm. rs Perform superpixel segmentation:

[0055] SLIC(I rs )→{S1,S2,θ,S N} (1)

[0056] Among them, each S i Represents an independent superpixel. After segmentation, each superpixel S i The contour boundary B i , obtain l boundary point sets {P i1 ,P i2 ,θ,P il}. Define P ij The neighborhood of ij , calculate the curvature K of each point by the following formula ij :

[0057] K ij =Cur(P ij ,N ij ) (2)

[0058] The curvature calculation function is specifically defined as:

[0059]

[0060] Among them, Cur represents the curvature estimation function, and the numerator contains P ij For the neighborhood point (x ij ,y ij ) coordinate difference product, ε is the numerical stability constant. i1 ,K i2 ,θ,K il} as superpixel shape features to generate shape feature map F shape ∈R H×W×l , where H and W represent the height and width of the feature in spatial resolution, and each pixel position encodes the curvature feature vector of the corresponding superpixel (x, y). By introducing the shape representation embedding mechanism, a multi-dimensional shape feature map F is obtained. shapeThis significantly enhances the model's ability to discriminate remote sensing targets based on shape details, thereby improving recognition accuracy in complex remote sensing scenarios. Shape features are fused with multi-scale image features to generate fused features, which enhance the representation of features using prior information such as shape.

[0061] In generalized small-sample segmentation of remote sensing images, labeling all new classes as background during base class training leads to confusion during feature learning, preventing the network from distinguishing potentially diverse new classes of objects within the background. To address this issue, this embodiment introduces self-reconstruction constraints as a training strategy to enhance the model's representational discriminability for new classes of objects. In this way, the model not only learns to distinguish between base classes and new classes, but also discriminates between different new classes of objects at the feature level.

[0062] Specifically, during the base class training phase, the supervision labels only contain n base Base class information, n novel Each new class is labeled as background, and the reconstruction constraint requires the model to recover the complete image from the extracted features, which forces the model to capture more detailed image features and learn the unique visual properties of each class (including new classes). Figure 1 As shown, after obtaining the fusion features, the fusion features are segmented by the segmentation head to obtain the segmentation image S base , and segment the image S base Compare with the annotated images from the public dataset for supervision; at the same time, the fusion features are reconstructed to obtain the restored image I rec , the process can be formally expressed as:

[0063]

[0064] Among them, F seg is the decoding feature of the segmentation head. The restored image I is obtained rec After that, the image I will be restored rec With the original input remote sensing image I rs Self-supervision is achieved by calculating the mean squared error (MSE) pixel by pixel, using it as the loss to perform gradient propagation on the network to update network parameters. This two-step supervision updates the entire network, creating the base class learning network. By introducing a self-reconstruction constraint mechanism during the base class learning phase, without requiring additional labeling, the model is forced to distinguish between new class representations by treating each new class as an independent label rather than a single background class. This also forces the network to focus on the high-resolution, fine-grained texture features required for reconstruction. This effectively prevents confusion with unseen new classes, improves the differentiation of representations between classes, and enables better and more efficient recognition of small objects in remote sensing imagery.

[0065] The base class training phase relies on a large amount of labeled data, enabling the model to generate highly discriminative representations for different base classes. Based on this, when initializing new class prototypes, this embodiment introduces a base class knowledge mining mechanism, dedicated to maximizing the extraction of useful information from existing base class knowledge.

[0066] Specifically, after obtaining the fusion features through the base class learning network, the base class prototype P is first obtained by Masked Average Pooling (MAP) using the limited support prototype mask initialization. B and the new class prototype P N , base class prototype P B The dimension is [n novel ,C], new class prototype P N The dimension is [n novel ,C]. To further enhance the new class prototype, in this embodiment, an attention mechanism is used to measure the similarity between the new class and the base class prototype, and the new class prototype is adjusted through a linear combination of the base class prototype. This method enables the new class prototype to inherit the characteristics of the base class while enhancing the response to the unique characteristics of the new class.

[0067] Specifically, based on the base class prototype P B and the new class prototype P N Update the affinity matrix A. The specific calculation method is as follows:

[0068]

[0069] Where <·,·> represents the dot product between prototype vectors, and the dimension of affinity matrix A is [n novel ,n base ], whose element A ij Is a scalar, representing the prototype of the i-th new class With the j-th base class prototype The affinity score between and are all vectors of dimension C. The updated affinity matrix and base class prototype are obtained by element-by-element matrix multiplication and addition to obtain the updated new class prototype P′ Ni , the specific calculation is as follows:

[0070]

[0071] Where, P Ni Initialize the prototype of the new class obtained by a small number of labeled new class samples, P B For the base class prototype, A ij is the affinity matrix. The DIaM method is used to B And the updated new class prototype P′ Ni Perform inference to obtain the segmented image S of the remote sensing image all .

[0072] In this example, by introducing a base class knowledge mining mechanism during the new class learning phase, the affinity scores of each new class prototype and all base class prototypes are first calculated. Then, a weighted sum of the base class prototypes is taken based on the scores, thereby optimizing and enhancing the new class prototypes and, in turn, strengthening the network's discriminative capabilities. This knowledge transfer and feature enhancement strategy provides an effective approach to addressing the problem of class generalization in small-sample learning.

[0073] like Figure 2 As shown in the figure, in challenging scenarios with many small targets and low image contrast, this method can accurately segment categories such as houses and vehicles in a given remote sensing image, and the targets all have clear segmentation boundaries.

[0074] This embodiment proposes a generalized small sample image segmentation method that improves remote sensing segmentation performance through a two-stage innovative design:

[0075] (a) In the base class learning stage, shape representation embedding is introduced to enhance the network's perception of remote sensing targets. At the same time, self-reconstruction constraints are used to prevent confusion of new category features and improve inter-class discrimination.

[0076] (b) In the new class learning stage, a base class knowledge mining mechanism is designed to use the base class prototype to optimize the new class prototype representation and enhance the network's discrimination ability.

[0077] Compared with the current best methods in the field of generalized small sample segmentation, the following conclusions can be drawn:

[0078] a) Compared with other methods, the method of the present invention is significantly superior to the existing state-of-the-art methods. Although methods such as CAPL and HBNC have achieved commendable results in the generalized small sample segmentation task of natural images, these methods lack the ability to perceive the shape of remote sensing targets. In addition, during the base class training process, category confusion often occurs between different new classes, resulting in poor performance on remote sensing images. The method of this embodiment can improve performance indicators, and in remote sensing applications, the effectiveness of the base class and new class is nearly doubled (mIoU score). Table 1 compares the performance of current mainstream methods (CAPL, HBNC, DlaM) and this method in the base class, new class and overall W-mIoU (%) indicators. As can be seen from the results, this method has achieved improvements in all three dimensions, reaching 41.57%, 11.01% and 23.23% in the base class, new class and overall, respectively. Compared with the existing method DlaM, this paper improves the overall W-mIoU by 9.53 percentage points, which fully demonstrates that this method has stronger generalization ability and overall robustness for new categories while maintaining good recognition performance for learned categories.

[0079] Table 1

[0080] method source Base Class New Class W-mIoU (%) CAPL CVPR'22 21.57 5.38 11.86 HBNC IJCV'23 21.25 3.65 9.772 DlaM CVPR'23 29.48 3.18 13.70 This method — 41.57 11.01 23.23

[0081] like Figure 2 As shown in Figure 3, for road classes (targets with significant linear / strip-shaped geometric features), this method exhibits excellent discrimination ability.

[0082] b) Compared with the baseline method (DIaM), the mIoU of the model in this embodiment on the base class and the new class is improved by 41.0% and 246.2% respectively. This is mainly attributed to the designed shape representation embedding module, self-reconstruction constraint mechanism and base class knowledge mining module. Figure 2 As shown in the figure, the class boundaries segmented by the model of the present invention are clearer, which further proves the above-mentioned design advantages.

[0083] Example 2

[0084] In a typical implementation of the present invention, this embodiment discloses a generalized small sample image segmentation system, comprising:

[0085] The base class learning module is configured to: extract multi-scale image features of remote sensing images based on the pyramid visual deformer backbone network; perform superpixel segmentation on the remote sensing images to generate shape feature maps; fuse the multi-scale image features with the shape feature maps to obtain fused features; perform supervision based on the self-reconstruction constraint mechanism and update the network based on the supervision;

[0086] The new class learning module is configured as follows: based on the base class knowledge mining mechanism, the fusion features are masked and initialized to obtain the base class prototype and the new class prototype; based on the base class prototype and the new class prototype, the affinity matrix is updated; based on the updated affinity matrix and the base class prototype, the new class prototype is updated; based on the base class prototype and the updated new class prototype, reasoning is performed to obtain the segmented image of the remote sensing image.

[0087] Example 3

[0088] In a typical embodiment of the present invention, this embodiment provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the steps of the generalized small sample image segmentation method described in Example 1 are implemented, and the steps include: a base class learning phase and a new class learning phase;

[0089] The base class learning stage includes: extracting multi-scale image features of remote sensing images based on the pyramid visual deformer backbone network; performing superpixel segmentation on the remote sensing images to generate shape feature maps; fusing the multi-scale image features with the shape feature maps to obtain fused features; performing supervision based on the self-reconstruction constraint mechanism and updating the network based on the supervision;

[0090] The new class learning stage includes: based on the base class knowledge mining mechanism, masking the fusion features to obtain the base class prototype and the new class prototype, updating the affinity matrix based on the base class prototype and the new class prototype, and updating the new class prototype based on the updated affinity matrix and the base class prototype; reasoning based on the base class prototype and the updated new class prototype to obtain the segmented image of the remote sensing image.

[0091] Example 4

[0092] In a typical embodiment of the present invention, this embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the generalized small sample image segmentation method described in Example 1 are implemented. The steps include: a base class learning phase and a new class learning phase;

[0093] The base class learning stage includes: extracting multi-scale image features of remote sensing images based on the pyramid visual deformer backbone network; performing superpixel segmentation on the remote sensing images to generate shape feature maps; fusing the multi-scale image features with the shape feature maps to obtain fused features; performing supervision based on the self-reconstruction constraint mechanism and updating the network based on the supervision;

[0094] The new class learning stage includes: based on the base class knowledge mining mechanism, masking the fusion features to obtain the base class prototype and the new class prototype, updating the affinity matrix based on the base class prototype and the new class prototype, and updating the new class prototype based on the updated affinity matrix and the base class prototype; reasoning based on the base class prototype and the updated new class prototype to obtain the segmented image of the remote sensing image.

[0095] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A generalized small sample image segmentation method, characterized in that: include: Base class learning stage and new class learning stage; The base class learning stage includes: extracting multi-scale image features of remote sensing images based on a pyramid visual deformer backbone network; performing superpixel segmentation on the remote sensing images to generate a shape feature map; fusing the multi-scale image features with the shape feature map to obtain a fused feature; and performing supervision based on a self-reconstruction constraint mechanism, and updating the overall network of this stage, i.e., the base class learning network, based on the supervision. The new class learning stage includes: based on the base class knowledge mining mechanism, masking the fusion features to initialize the base class prototype and the new class prototype, updating the affinity matrix based on the base class prototype and the new class prototype, and updating the new class prototype based on the updated affinity matrix and the base class prototype; and reasoning based on the base class prototype and the updated new class prototype to obtain a segmented image of the remote sensing image.

2. The generalized small sample image segmentation method according to claim 1, characterized in that: The self-reconstruction constraint mechanism is used for supervision, and the overall network of this stage, i.e., the base class learning network, is updated based on supervision. The method includes: after obtaining the fused features, obtaining a restored image from the fused features based on the self-reconstruction constraint, and supervising the restored image and the original input image; segmenting the fused features to obtain a segmented image, and supervising the segmented image and the labeled image; and updating the base class learning network based on the above two types of supervision.

3. The generalized small sample image segmentation method according to claim 1, wherein: The architecture of the pyramid visual deformer consists of four stages, each of which consists of a block embedding and encoding layer. The input features of each stage are divided into blocks of a set size, and after processing by the encoder, the stage features are output to obtain multi-scale image features.

4. The generalized small sample image segmentation method according to claim 1, wherein: The superpixel segmentation of the remote sensing image to generate the shape feature map specifically includes: The remote sensing image is segmented into superpixels based on the SLIC algorithm; the contour boundary of each superpixel is extracted to obtain a set of boundary points; the curvature of each boundary point is calculated and the curvature value sequence is used as the superpixel shape feature to generate a shape feature map.

5. The generalized small sample image segmentation method according to claim 4, characterized in that: The curvature is calculated as follows: Where Cur represents the curvature estimation function, P ij is the superpixel boundary point, N ij P ij The neighborhood of P ij For the neighborhood point (x ij ,y ij ) coordinate difference product, and ε is a numerical stability constant.

6. The generalized small sample image segmentation method according to claim 1, wherein: The affinity matrix A is calculated as follows: Where <·,·> represents the dot product between prototype vectors, and the dimension of affinity matrix A is [n novel ,n base ], whose element A ij Is a scalar, representing the prototype of the i-th new class With the j-th base class prototype The affinity score between and are all vectors of dimension C.

7. The generalized small sample image segmentation method according to claim 1, wherein: The updated new class prototype is obtained through element-by-element matrix multiplication and addition operations: Where, P Ni Initialize the prototype of the new class obtained by a small number of labeled new class samples, P B For the base class prototype, A ij is the affinity matrix.

8. A generalized small sample image segmentation system, characterized by: include: The base class learning module is configured to: extract multi-scale image features of remote sensing images based on the pyramid visual deformer backbone network; Performing superpixel segmentation on the remote sensing image to generate a shape feature map; fusing the multi-scale image features with the shape feature map to obtain a fused feature; Supervision is performed based on the self-reconstruction constraint mechanism, and the overall network at this stage is updated based on supervision, that is, the base class learning network; The new class learning module is configured as follows: based on the base class knowledge mining mechanism, the fusion features are masked and initialized to obtain the base class prototype and the new class prototype; based on the base class prototype and the new class prototype, the affinity matrix is updated; based on the updated affinity matrix and the base class prototype, the new class prototype is updated; based on the base class prototype and the updated new class prototype, reasoning is performed to obtain the segmented image of the remote sensing image.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that When the program is executed by a processor, the steps of a generalized small sample image segmentation method as described in any one of claims 1 to 7 are implemented.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that When the processor executes the program, the steps of a generalized small sample image segmentation method as described in any one of claims 1 to 7 are implemented.

Images