A generalized small sample image segmentation method, system, medium and device

Abstract

The application discloses a generalized small sample image segmentation method, system, medium and equipment, including: a base class learning stage and a new class learning stage; the base class learning stage includes: extracting multi-scale image features of a remote sensing image based on a pyramid visual transformer backbone network; performing superpixel segmentation on the remote sensing image to generate a shape feature map; fusing the multi-scale image features and the shape feature map to obtain fused features; performing supervision based on a self-reconstruction constraint mechanism, and based on the overall network of this stage, namely a base class learning network; the new class learning stage includes: performing mask initialization on the fused features based on a base class knowledge mining mechanism to obtain a base class prototype and a new class prototype, updating an affinity matrix based on the base class prototype and the new class prototype, and updating the new class prototype based on the updated affinity matrix and the base class prototype; and performing reasoning based on the base class prototype and the updated new class prototype to obtain a segmented image of the remote sensing image.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image segmentation, and particularly relates to a generalized few-shot image segmentation method, system, medium and device. BACKGROUND

[0002] The statements herein are provided only to enhance understanding of the present application and are not necessarily intended to constitute the prior art.

[0003] As an observation data carrier, remote sensing images play a key role in numerous applications. The semantic segmentation task aims to distinguish the geospatial class at all locations in a given image. However, training an ideal neural network is costly in terms of manpower and material resources, and the generalization of the resulting model is very fragile. Specifically, the tens of thousands of dense annotation training samples required for the segmentation task are extremely costly to annotate; the trained model can only output known classes, making it difficult to extend to prediction tasks for unseen classes. These unfavorable factors hinder the further development of remote sensing semantic segmentation.

[0004] A simple way to alleviate this problem is to use few-shot semantic segmentation (FSS) to adapt the model through a small number of support annotations of unseen / new classes. However, Tian et al. pointed out in their CVPR 2022 work "Generalized few-shot semantic segmentation" that there are important defects that hinder the practical application of these methods: existing FSS research assumes that the support samples contain the classes present in the query image, which can lead to an expensive manual screening process. At the same time, Tian et al. proposed a new task called generalized few-shot semantic segmentation (GFSS). Generalized few-shot semantic segmentation relaxes the strong assumption that the support image and the query image have the same class. Unlike traditional few-shot segmentation, which only supports the classes defined by the support image, GFSS uses a support set containing multiple new classes, enabling the model to simultaneously segment the base classes and new classes in the query image. This means that under this new paradigm, there is no need to provide support images with the same target classes as the query image. Overall, the field of generalized few-shot semantic segmentation has made remarkable research progress in recent years. However, most of the existing work only deals with the analysis and processing of natural images and has not been further expanded to the more challenging field of remote sensing images.

[0005] Traditional generalized few-shot semantic segmentation methods are difficult to achieve ideal results in remote sensing segmentation due to the differences between remote sensing images and natural images. Through the analysis of the special properties of remote sensing images, there are three main reasons why traditional generalized few-shot semantic segmentation networks perform poorly in this field: first, compared with natural images, the higher resolution of remote sensing images brings challenges, and previous networks are not designed for high-resolution data, resulting in poor performance; second, the special nature of remote sensing content increases the difficulty. Specifically, compared with natural images, the differences between grassland and wasteland, rivers and roads, and other categories in remote sensing images are smaller. When similar categories are divided into base classes and new classes, respectively, the model is difficult to distinguish, resulting in blurred boundaries between new classes and base classes.

[0006] The inventors found that existing generalized few-shot segmentation (GFSS) methods have the following technical bottlenecks in remote sensing image processing:

[0007] (1) The traditional network architecture is not optimized for the high-resolution characteristics of remote sensing images, resulting in prominent small-scale target (such as vehicles, roads) missing detection problems;

[0008] (2) Due to the fixed viewing angle of remote sensing images, similar ground object categories (such as grassland and wasteland, rivers and roads) have weak differences in spectral and texture features, and existing models are difficult to effectively distinguish;

[0009] (3) During the base class training stage, all new classes are uniformly labeled as background, causing confusion in new class feature learning, and limiting the model's generalization ability. SUMMARY

[0010] The purpose of the present application is to overcome the deficiencies in the prior art, and to provide a generalized few-shot image segmentation method, system, medium and device, which can effectively process large-size remote sensing images, has better recognition ability for small-size targets, enhances the network's perception ability for remote sensing targets, prevents confusion of new class features, improves the distinction between classes, strengthens the network's discrimination ability, and makes the segmented image more accurate and effective, and the class boundary clearer.

[0011] In order to achieve the above purpose, the present application is realized by the following technical solutions:

[0012] On the one hand, the technical scheme of the present application provides a generalized few-shot image segmentation method, comprising: a base class learning stage and a new class learning stage;

[0013] The base class learning stage includes: extracting multi-scale image features of the remote sensing image based on a pyramid visual transformer backbone network; performing superpixel segmentation on the remote sensing image to generate a shape feature map; fusing the multi-scale image features and the shape feature map to obtain fused features; supervising based on a self-reconstruction constraint mechanism, and updating the overall network of this stage, i.e., the base class learning network, based on the supervision;

[0014] The new class learning stage includes: obtaining base class prototypes and new class prototypes by performing mask initialization on the fused features based on a base class knowledge mining mechanism, updating an affinity matrix based on the base class prototypes and the new class prototypes, and updating the new class prototypes based on the updated affinity matrix and the base class prototypes; and performing reasoning based on the base class prototypes and the updated new class prototypes to obtain a segmented image of the remote sensing image.

[0015] In at least one embodiment, the supervision based on the self-reconstruction constraint mechanism and the updating of the overall network of this stage, i.e., the base class learning network, based on the supervision include: after obtaining the fused features, obtaining a restored picture from the fused features based on the self-reconstruction constraint, and supervising the restored picture and the original input picture; performing segmentation on the fused features to obtain a segmented picture, and supervising the segmented picture and a labeled image; and updating the base class learning network based on the above two kinds of supervision.

[0016] In at least one embodiment, the architecture of the pyramid visual transformer includes four stages, each stage being composed of a block embedding and an encoding layer; the input features of each stage are divided into blocks of a set size, and after being processed by the encoder, stage features are output, and multi-scale image features are obtained.

[0017] In at least one embodiment, the superpixel segmentation on the remote sensing image to generate a shape feature map specifically includes:

[0018] Performing superpixel segmentation on the remote sensing image based on the SLIC algorithm; extracting the contour boundary of each superpixel to obtain a set of boundary points; calculating the curvature of each boundary point and taking the curvature value sequence as the shape feature of the superpixel to generate a shape feature map.

[0019] In at least one embodiment, the curvature is calculated in the following manner:

[0020]

[0021] In the formula, Cur represents a curvature estimation function, P ij is a superpixel boundary point, N ij is a neighborhood of P ij , the numerator contains P ij is a neighborhood point (x ij , y ij ) coordinate difference product, and ε is a numerical stability constant.

[0022] In at least one embodiment, the affinity matrix A is calculated as follows:

[0023]

[0024] where <·,·> denotes the dot product between prototype vectors, the affinity matrix A has dimension [n novel ,n base ], and its elements A ij are scalars representing the affinity score between the i-th new class prototype and the j-th base class prototype , and both P and P are vectors of dimension C.

[0025] In at least one embodiment, the updated new class prototypes are obtained by element-wise matrix addition as follows:

[0026]

[0027] where P Ni is the new class initialization prototype obtained by a small number of labeled new class samples, P B is the base class prototype, and A ij is the affinity matrix.

[0028] In another aspect, the technical scheme of the present application also provides a generalized small sample image segmentation system, comprising:

[0029] a base class learning module configured to extract multi-scale image features of a remote sensing image based on a pyramid visual transformer backbone network, perform superpixel segmentation on the remote sensing image to generate a shape feature map, fuse the multi-scale image features and the shape feature map to obtain fused features, and supervise based on a self-reconstruction constraint mechanism and update the overall network of this stage, i.e., the base class learning network, based on the supervision;

[0030] a new class learning module configured to mask initialize the fused features based on a base class knowledge mining mechanism to obtain base class prototypes and new class prototypes, update an affinity matrix based on the base class prototypes and the new class prototypes, update the new class prototypes based on the updated affinity matrix and the base class prototypes, and perform inference based on the base class prototypes and the updated new class prototypes to obtain a segmentation image of the remote sensing image.

[0031] The technical scheme of the present application has the following advantages:

[0032] 1) The present application improves the performance of remote sensing segmentation through a two-stage design: (a) In the base class learning stage, the shape representation embedding enhancement network is introduced to enhance the perception ability of remote sensing targets, and the self-reconstruction constraint is adopted to prevent the confusion of new class features and improve the inter-class discrimination, which solves the problem that the existing model is difficult to distinguish similar ground object classes; (b) In the new class learning stage, a base class knowledge mining mechanism is designed to optimize the new class prototype representation using the base class prototype, and the network discrimination ability is strengthened, which solves the problem of feature learning confusion of new classes and limited model generalization ability in the prior art.

[0033] 2) The present application extracts multi-scale image features using PVT as the backbone network, which can process high-resolution images and obtain multi-layer effective features, better cope with the high-resolution characteristics of remote sensing images in the segmentation task, and avoid the problem that the traditional network architecture is not optimized for the high-resolution characteristics of remote sensing images, resulting in the problem of small-scale target (such as vehicle, road) detection problem.

[0034] 3) Compared with the prior art, the present application can effectively process large-size remote sensing images, has better recognition ability for small-size targets, and has more advantages in distinguishing easily confused foreground regions and prototypes, which are all conducive to avoiding the omission of targets (mistakenly identifying new classes as background) and thus improving the recall rate of segmented targets. BRIEF DESCRIPTION OF DRAWINGS

[0035] The drawings accompanying the specification of the present application form a part thereof and serve to provide further understanding of the present application, the illustrative embodiments of the present application and its description serve to explain the present application, and do not constitute an improper limitation of the present application.

[0036] Figure 1 is the overall network architecture schematic diagram of a generalized small sample image segmentation method of the present application;

[0037] Figure 2 is a schematic diagram of the semantic segmentation result of a generalized small sample image segmentation method of the present application on a remote sensing image. DETAILED DESCRIPTION

[0038] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used in the present application have the same meaning as generally understood by those skilled in the art to which the present application belongs.

[0039] Term explanation:

[0040] Pyramid Vision Transformer (PVT): Pyramid Vision Transformer is a visual Transformer architecture for computer vision tasks such as image recognition and segmentation. The core idea of PVT is to introduce a pyramid structure into the Transformer model, which can extract feature representations of different resolutions from images.

[0041] DIaM: DIaM (Distilled Information Maximization) is a baseline method for Generalized Few-Shot Semantic Segmentation (GFSS) tasks, published in CVPR 2023. The core idea of DIaM is to dynamically integrate the information of new and old classes using the theoretical framework of Mutual Information.

[0042] Patch embedding: Patch embedding refers to the process of compressing image patches into feature vectors through convolution and linear mapping, etc. in the Transformer structure when processing image data.

[0043] SLIC algorithm: SLIC is a superpixel segmentation algorithm that can divide an image into a series of small regions called "superpixels" that are locally similar and have coherent boundaries.

[0044] mIoU: mIoU is a commonly used performance evaluation indicator in image segmentation tasks. The calculation method is as follows: for each class, calculate the intersection and union ratio (IoU) of the predicted region and the true region. Then take the average of the IoU of all classes, which is mIoU. In simple terms, mIoU measures the degree of overlap between the predicted image and the true label in space, and the higher the value, the better the segmentation performance.

[0045] As introduced in the background art, the purpose of the present application is to overcome the shortcomings of the prior art, and to provide a generalized small sample image segmentation method, system, medium and device, which can effectively process large size remote sensing images, has better recognition ability for small size targets, enhances the network's perception ability of remote sensing targets, prevents new class feature confusion, improves the inter-class discrimination, strengthens the network's discrimination ability, makes the segmented image more accurate and effective, and the class boundary clearer.

[0046] Embodiment 1

[0047] In a typical embodiment of the present application, the embodiment discloses a generalized small sample image segmentation method, comprising: a base class learning stage and a new class learning stage.

[0048] The base class learning stage includes: extracting multi-scale image features of the remote sensing image based on a pyramid visual transformer backbone network; performing superpixel segmentation on the remote sensing image to generate a shape feature map; fusing the multi-scale image features and the shape feature map to obtain fused features; performing supervision based on a self-reconstruction constraint mechanism, and updating the network based on the supervision;

[0049] The new class learning stage includes: based on a base class knowledge mining mechanism, performing mask initialization on the fused features to obtain a base class prototype and a new class prototype, updating an affinity matrix based on the base class prototype and the new class prototype, and updating the new class prototype based on the updated affinity matrix and the base class prototype; performing reasoning based on the base class prototype and the updated new class prototype to obtain a segmented image of the remote sensing image.

[0050] The above generalized small sample image segmentation method will be described in detail below with reference to specific embodiments.

[0051] As shown in Figure 1 , the training paradigm of the embodiment follows the DIaM form, and mainly has two stages of base class learning and new class learning. In the base class learning stage, shape representation embedding and self-reconstruction constraint are introduced to enhance the perception ability of the network to remote sensing targets; in the new class learning stage, a base class knowledge mining mechanism is designed to optimize the representation of the new class prototype by using the discriminative knowledge accumulated in the base class training stage.

[0052] Specifically, in the base class training stage, for a remote sensing image I rs , first, a pyramid visual transformer (hereinafter referred to as PVT) is used as a feature extraction backbone network to extract image features. The PVT architecture includes four stages, each stage is composed of a block embedding and an encoding layer, and the input feature F i of each stage is divided into P i size blocks, and the i-th stage feature X i is output after the encoder is processed, and finally the multi-scale feature {X1, X2, X3, X4} is obtained. Using PVT as the backbone network to extract multi-scale image features can process high-resolution images and obtain multi-layer effective features, better cope with the high-resolution characteristics of the remote sensing image segmentation task, and avoid the problem that the traditional network architecture is not optimized for the high-resolution characteristics of the remote sensing image, thereby highlighting the problem of missing small-scale targets (such as vehicles and roads).

[0053] In remote sensing image analysis, the shooting angle of remote sensing images is relatively fixed, and the shape information of different categories is more stable than natural images (for example, rivers often appear as long strips, and ships usually have sharp features to distinguish from vehicles). Therefore, the shape features of the target become strong discriminative indicators. This characteristic is particularly important in generalized small sample learning, which can significantly reduce the dependence of the model on the amount of labeled samples. By mapping the shape features of different categories of targets to the shape space, the recognition ability of the model for new categories can be effectively enhanced.

[0054] Therefore, in the embodiment, first, the SLIC algorithm is used to perform superpixel segmentation on the remote sensing image I rs to obtain a superpixel set S rs .

[0055] SLIC(I N )→{S1,S2,θ,S i} (1)

[0056] where each S i represents an independent superpixel. The contour boundary B i of each superpixel S i1 is extracted, and l boundary point sets {P i2 ,P il ,θ,P ij} are obtained by equidistant sampling. The neighborhood of P ij is defined as N ij , and the curvature K ij of each point is calculated by the following formula:

[0057] K ij =Cur(P ij ,N ij ) (2)

[0058] The curvature calculation function is specifically defined as:

[0059]

[0060] where Cur represents the curvature estimation function, the numerator contains the coordinate difference product of P ij as the neighborhood point (x ij ,y i1 ), and ε is a numerical stability constant. The curvature value sequence {K i2 ,K il ,θ,K shape} is used as the shape feature of the superpixel, and a shape feature map F H×W×l is generated, where H and W represent the height and width of the feature in the spatial resolution, and each pixel position encoding (x, y) corresponds to the curvature feature vector of the superpixel. By introducing a shape representation embedding mechanism, a multi-dimensional shape feature map F shape, which significantly enhances the model's ability to distinguish remote sensing targets based on shape details, thereby improving the recognition accuracy in complex remote sensing scenarios. The shape feature is fused with the multi-scale image feature to obtain a fused feature, so as to enhance the representation ability of the feature through shape prior information.

[0061] In the generalized small sample segmentation of remote sensing images, labeling all new classes as background in the base class training stage can cause confusion in the feature learning process, so the network cannot distinguish the potentially diverse new class targets in the background. To solve this problem, the self-reconstruction constraint is introduced as a training strategy to enhance the model's representation and discrimination of new class targets. In this way, the model not only learns to distinguish between base classes and new classes, but also needs to identify different new class targets at the feature level.

[0062] Specifically, in the base class training stage, the supervision label only contains n base base class information, and n novel new classes are labeled as background. The reconstruction constraint requires the model to recover the complete image from the extracted features, which forces the model to capture more detailed image features and learn the unique visual attributes of each class (including new classes). As Figure 1 shown, after obtaining the fused feature, the fused feature is segmented by the segmentation head to obtain a segmented image S base , and the segmented image S base is compared with the labeled image from the public dataset for supervision; at the same time, the fused feature is obtained through the reconstruction head to obtain a restored image I rec . This process can be formally represented as:

[0063]

[0064] where F seg is the decoding feature of the segmentation head. After obtaining the restored image I rec , the mean square error is calculated between the restored image I rec and the original input remote sensing image I rs pixel by pixel, and the loss is used as the gradient backpropagation of the network to update the network parameters, so as to realize self-supervision. The entire network is updated through the above two supervisions, i.e. the base class learning network. By introducing the self-reconstruction constraint mechanism in the base class learning stage, without additional annotation cost, by regarding different new classes as independent labels rather than a single background class, the model is forced to distinguish new class representations, and the network is forced to pay attention to high-resolution fine-grained textures required for reconstruction, which can effectively prevent confusion of unseen new classes, improve the representation and discrimination of classes, and better and more effectively identify small targets in remote sensing images.

[0065] The base class training phase relies on a large amount of labeled data, enabling the model to generate highly discriminative representations for different base classes. Based on this, this embodiment introduces a base class knowledge mining mechanism when initializing new class prototypes, aiming to maximize the extraction of useful information from existing base class knowledge.

[0066] Specifically, after obtaining the fused features through the base class learning network, the base class prototype P is first obtained by initializing with a finite support prototype mask using Masked Average Pooling (MAP). B and the new class prototype P N Base class prototype P B Dimension is [n novel [,C], new class prototype P N Dimension is [n novel [C]. To further enhance the new class prototype, this embodiment employs an attention mechanism to measure the similarity between the new class and the base class prototype, and adjusts the new class prototype through a linear combination of the base class prototypes. This method enables the new class prototype to both inherit the characteristics of the base class and enhance its response to the unique characteristics of the new class.

[0067] Specifically, based on the base class prototype P B and the new class prototype P N The affinity matrix A is updated, and the specific calculation method is as follows:

[0068]

[0069] In the formula, <·,·> represent the dot product between prototype vectors, and the affinity matrix A has dimensions [n]. novel ,n base ], its element A ij Let be a scalar, representing the prototype of the i-th new class. With the prototype of the j-th base class Affinity score between them and Both are vectors of dimension C. The updated affinity matrix and the base class prototype are used to obtain the updated new class prototype P′ through element-wise matrix multiplication and addition operations. Ni The specific calculations are as follows:

[0070]

[0071] In the formula, P Ni To initialize the prototype of a new class obtained from a small number of labeled new class samples, P B As the base class prototype, A ij This is the affinity matrix. The DIAM method is used to perform affinity analysis on the base class prototype P. B and the updated new class prototype P′ Ni By performing reasoning, a segmented image S of the remote sensing image is obtained. all .

[0072] In this embodiment, by introducing a base class knowledge mining mechanism in the new class learning stage, the affinity score of each new class prototype and all base class prototypes is first calculated, and then the base class prototypes are weighted and summed according to the score, so as to optimize and enhance the new class prototype, and thus the network discrimination ability is enhanced. This knowledge transfer and feature enhancement strategy provides an effective way to solve the class generalization problem in small sample learning.

[0073] As shown in Figure 2 In the challenge scene of many small targets and low-contrast images, the method can accurately segment the categories such as houses and vehicles in the given remote sensing image, and the target has a clear segmentation boundary.

[0074] The generalized small sample image segmentation method proposed in this embodiment improves the remote sensing segmentation performance through two-stage innovative design:

[0075] (a) In the base class learning stage, the shape representation embedding enhancement network is introduced to improve the perception ability of remote sensing targets, and the self-reconstruction constraint is used to prevent new class feature confusion and improve the inter-class discrimination;

[0076] (b) In the new class learning stage, a base class knowledge mining mechanism is designed to optimize the new class prototype representation using the base class prototype, and to strengthen the network discrimination ability.

[0077] Compared with the current optimal method in the field of generalized small sample segmentation, the following conclusions can be drawn:

[0078] a) Compared with other methods, the method of the present application is significantly better than the existing most advanced method. Although the methods such as CAPL and HBNC have achieved commendable results in the generalized small sample segmentation task of natural images, these methods have insufficient shape perception ability for remote sensing targets. In addition, during the base class training process, class confusion often occurs between different new classes, which leads to poor performance on remote sensing images. The method of the present embodiment can improve the performance indicators, and make the effectiveness of the base class and the new class in remote sensing applications increase by nearly two times (mIoU score). Table 1 compares the performance of the current mainstream methods (CAPL, HBNC, DlaM) and the method of the present application in the base class, new class and overall W-mIoU (%) indicators. From the results, it can be seen that the method of the present application has improved in three dimensions, reaching 41.57%, 11.01% and 23.23% in the base class, new class and overall, respectively. Compared with the existing method DlaM, the overall W-mIoU of the present application is improved by 9.53 percentage points, which fully embodies the method of the present application in maintaining good recognition performance for learned classes, while having stronger new class generalization ability and overall robustness.

[0079] Table 1

[0080] Method Source Base class New class W-mIoU (%) CAPL CVPR'22 21.57 5.38 11.86 HBNC IJCV'23 21.25 3.65 9.772 DlaM CVPR'23 29.48 3.18 13.70 The method — 41.57 11.01 23.23

[0081] As Figure 2 shown, for road class (target with significant linear / band-shaped geometric features), the proposed method shows excellent discrimination ability.

[0082] b) Compared with the baseline method (DIaM), the mIoU of the proposed model on base class and new class is improved by 41.0% and 246.2% respectively. This is mainly due to the design of the shape representation embedding module, the self-reconstruction constraint mechanism and the base class knowledge mining module. As Figure 2 shown, the class boundary segmented by the proposed model is clearer, which further proves the above design advantages.

[0083] Embodiment 2

[0084] In a typical embodiment of the present application, the embodiment discloses a generalized small sample image segmentation system, comprising:

[0085] The base class learning module is configured to extract multi-scale image features of the remote sensing image based on the pyramid visual transformer backbone network; perform superpixel segmentation on the remote sensing image to generate a shape feature map; fuse the multi-scale image features and the shape feature map to obtain a fused feature; supervise based on a self-reconstruction constraint mechanism, and update the network based on the supervision;

[0086] The new class learning module is configured to initialize the fused feature based on a base class knowledge mining mechanism to obtain a base class prototype and a new class prototype; update the affinity matrix based on the base class prototype and the new class prototype; update the new class prototype based on the updated affinity matrix and the base class prototype; and infer based on the base class prototype and the updated new class prototype to obtain a segmented image of the remote sensing image.

[0087] Embodiment 3

[0088] In a typical embodiment of the present application, the embodiment provides a computer readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a generalized small sample image segmentation method as introduced in embodiment 1, the steps comprising: a base class learning stage and a new class learning stage;

[0089] The base class learning stage comprises: extracting multi-scale image features of the remote sensing image based on the pyramid visual transformer backbone network; performing superpixel segmentation on the remote sensing image to generate a shape feature map; fusing the multi-scale image features and the shape feature map to obtain a fused feature; supervising based on a self-reconstruction constraint mechanism, and updating the network based on the supervision;

[0090] The new class learning stage comprises: based on a base class knowledge mining mechanism, the fusion features are initialized to obtain a base class prototype and a new class prototype, an affinity matrix is updated based on the base class prototype and the new class prototype, and the new class prototype is updated based on the updated affinity matrix and the base class prototype; and inference is performed based on the base class prototype and the updated new class prototype to obtain a segmentation image of the remote sensing image.

[0091] Embodiment 4

[0092] In a typical embodiment of the present application, the embodiment provides a computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the steps of a generalized small sample image segmentation method as introduced in Embodiment 1 when executing the program, and the steps comprise a base class learning stage and a new class learning stage.

[0093] The base class learning stage comprises: based on a pyramid visual transformer backbone network, multi-scale image features of the remote sensing image are extracted; the remote sensing image is subjected to superpixel segmentation to generate a shape feature map; the multi-scale image features are fused with the shape feature map to obtain fusion features; supervision is performed based on a self-reconstruction constraint mechanism, and the network is updated based on the supervision.

[0094] The new class learning stage comprises: based on a base class knowledge mining mechanism, the fusion features are initialized to obtain a base class prototype and a new class prototype, an affinity matrix is updated based on the base class prototype and the new class prototype, and the new class prototype is updated based on the updated affinity matrix and the base class prototype; and inference is performed based on the base class prototype and the updated new class prototype to obtain a segmentation image of the remote sensing image.

[0095] The above is only a preferred embodiment of the present application and is not used to limit the present application, and for those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A generalized small sample image segmentation method, characterized in that, The method comprises: a base class learning stage and a new class learning stage; the base class learning stage comprises: extracting multi-scale image features of a remote sensing image based on a pyramid visual transformer backbone network; performing superpixel segmentation on the remote sensing image to generate a shape feature map; fusing the multi-scale image features and the shape feature map to obtain fused features; performing supervision based on a self-reconstruction constraint mechanism, and updating an overall network of this stage, i.e., a base class learning network, based on the supervision; the new class learning stage comprises: obtaining base class prototypes and new class prototypes by mask initialization of the fused features based on a base class knowledge mining mechanism, updating an affinity matrix based on the base class prototypes and the new class prototypes, updating the new class prototypes based on the updated affinity matrix and the base class prototypes, and performing inference based on the base class prototypes and the updated new class prototypes to obtain a segmentation image of the remote sensing image; the supervision based on the self-reconstruction constraint mechanism and the updating of the overall network of this stage, i.e., the base class learning network, based on the supervision, comprises: after obtaining the fused features, obtaining a restored picture from the fused features based on the self-reconstruction constraint, and performing supervision on the restored picture and the original input picture; performing segmentation on the fused features to obtain a segmented picture, and performing supervision on the segmented picture and a labeled image; and updating the base class learning network based on the above two kinds of supervision.

2. The generalized small sample image segmentation method of claim 1, wherein, The architecture of the pyramid visual transformer comprises four stages, each stage being composed of a block embedding and an encoding layer; input features of each stage are divided into blocks of a set size, and stage features are output after being processed by an encoder, thereby obtaining multi-scale image features.

3. The generalized small sample image segmentation method of claim 1, wherein, The superpixel segmentation on the remote sensing image to generate the shape feature map specifically comprises: performing superpixel segmentation on the remote sensing image based on an SLIC algorithm; extracting the contour boundary of each superpixel to obtain a set of boundary points; calculating the curvature of each boundary point, and taking the curvature value sequence as the shape feature of the superpixel to generate the shape feature map.

4. The generalized small sample image segmentation method of claim 3, wherein, The calculation method of the curvature specifically comprises: wherein denotes a curvature estimation function, is a superpixel boundary point, is a neighborhood of the numerator contains is a neighborhood point a coordinate difference product, is a numerical stability constant.

5. The generalized small sample image segmentation method of claim 1, wherein, The affinity matrix A The calculation is: wherein, denotes the dot product between prototype vectors, the affinity matrix has dimension , whose elements are scalars, denote the affinity score between the th new class prototype and the th base class prototype , and are vectors of dimension , is the number of new class instances labeled as background in the base class training phase supervision labels, is the number of base class information in the base class training phase supervision labels.

6. The generalized small sample image segmentation method of claim 1, wherein, the updated new class prototypes are obtained through element-by-element matrix multiplication and addition operations: wherein is the initialization prototype of the new class obtained by a small amount of labeled new class samples, is the prototype of the base class, is the affinity matrix.

7. A generalized small sample image segmentation system, characterized by, The method comprises: a base class learning module configured to extract multi-scale image features of a remote sensing image based on a pyramid visual transformer backbone network; perform superpixel segmentation on the remote sensing image to generate a shape feature map; fuse the multi-scale image features and the shape feature map to obtain fused features; perform supervision based on a self-reconstruction constraint mechanism, and update an overall network of this stage, i.e., a base class learning network, based on the supervision; a new class learning module configured to obtain base class prototypes and new class prototypes by mask initialization of the fused features based on a base class knowledge mining mechanism; update an affinity matrix based on the base class prototypes and the new class prototypes; update the new class prototypes based on the updated affinity matrix and the base class prototypes; and perform inference based on the base class prototypes and the updated new class prototypes to obtain a segmentation image of the remote sensing image; the supervision based on the self-reconstruction constraint mechanism and the updating of the overall network of this stage, i.e., the base class learning network, based on the supervision, comprises: after obtaining the fused features, obtaining a restored picture from the fused features based on the self-reconstruction constraint, and performing supervision on the restored picture and the original input picture; perform segmentation on the fused features to obtain a segmented picture, and perform supervision on the segmented picture and a labeled image; and update the base class learning network based on the above two kinds of supervision.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program, when executed by the processor, implements the steps of a generalized small sample image segmentation method as claimed in any one of claims 1-6.

9. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the steps of a generalized small sample image segmentation method as claimed in any one of claims 1-6 when executing the program.

Images