Remote sensing small sample segmentation method and system for multi-view context learning
By employing a multi-view context learning approach and utilizing rotation data augmentation and sparse cue feature embedding, the problems of domain dissimilarity and rotation invariance in remote sensing image segmentation are solved, achieving efficient and accurate segmentation under conditions of few samples.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HUNAN UNIV
- Filing Date
- 2026-01-12
- Publication Date
- 2026-04-17
AI Technical Summary
Traditional remote sensing image segmentation methods rely on fully supervised deep learning, which requires massive amounts of labeled data. They cannot quickly respond to new categories, and there are domain differences and target rotation invariance issues in remote sensing images, resulting in poor segmentation performance.
A multi-view context learning approach is adopted, which generates multi-view reference images through rotation data augmentation. Combined with sparse cue feature embedding and a pre-trained image encoder, rotation-invariant feature modeling is achieved using a LoRA adapter to generate the final segmentation mask.
Achieving accurate segmentation of targets in any direction in complex backgrounds with very few reference images breaks down the domain barriers between natural images and remote sensing scenes, improving segmentation efficiency and accuracy.
Smart Images
Figure CN121883504A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of remote sensing image processing technology, and in particular to a remote sensing few-sample segmentation method and system based on multi-view context learning. Background Technology
[0002] With the rapid development of aerospace and Earth observation technologies, the amount of high-resolution remote sensing image data acquired has increased exponentially. Remote sensing image interpretation plays an irreplaceable role in fields such as urban planning, precision agriculture, disaster assessment, and military reconnaissance. Among these, semantic segmentation is one of the core tasks of remote sensing interpretation, aiming to classify each pixel in the image.
[0003] Traditional remote sensing image segmentation methods mostly rely on fully supervised deep learning algorithms, which typically require massive amounts of pixel-level labeled data for training. However, in real-world remote sensing applications, they face the following significant challenges: a1. High annotation cost: Remote sensing images are huge and have dense ground features, making pixel-level annotation extremely time-consuming and labor-intensive, and requiring the participation of professional personnel.
[0004] a2. Emerging New Categories: The real world is dynamic and constantly changing, with new types of terrain features (such as specific types of aircraft or temporary buildings) constantly appearing that are not included in the training set. Traditional fully supervised models typically need to collect new data and retrain when faced with these invisible categories, which cannot meet the requirements for rapid response.
[0005] Although existing adaptation methods based on the Segment Anything model (SAM) have made some progress, they still have significant technical limitations when processing remote sensing images, as follows: b1. Significant Domain Differences: SAM is primarily trained on natural scene images (mainly side-view shots with centered objects). However, remote sensing images are typically viewed from above and contain complex textured backgrounds. Directly using features extracted from a frozen SAM encoder often fails to accurately represent the semantic information of remotely sensed objects.
[0006] b2. Target Orientation Diversity: Unlike natural images, remote sensing images are taken from a top-down perspective, and ground features (such as airplanes, ships, and oil tanks) can rotate in any direction. Traditional methods based on a single reference image often fail to achieve good feature matching results when the target orientation in the reference image differs significantly from that in the image under test.
[0007] Therefore, there is an urgent need to develop a remote sensing few-sample segmentation method that integrates efficient domain adaptation and rotation-invariant feature modeling, so as to break down the domain barrier between natural images and remote sensing scenes and achieve accurate segmentation of targets in any direction in complex backgrounds with very few reference samples. Summary of the Invention
[0008] This invention provides a remote sensing few-sample segmentation method and system based on multi-view context learning to solve the technical problems mentioned in the background art.
[0009] To achieve the above objectives, the technical solution of the present invention is implemented as follows: This invention provides a remote sensing few-shot segmentation method based on multi-view context learning, comprising the following steps: S1. Rotate the reference image to perform data augmentation, generate multiple reference images from different perspectives, and stitch the multiple reference images from different perspectives with the image to be tested to obtain a joint input image. S2. Generate uniformly distributed positive and negative cue points within each reference image region in the joint input image. The foreground region is marked as a positive cue point and the background region is marked as a negative cue point. Input the coordinate information of the positive and negative cue points into the cue encoder to obtain the sparse cue feature embedding. S3. Input the joint input image into a pre-trained image encoder to extract features and generate joint features; S4. The sparse cue feature embedding and the joint feature are input into the mask decoder to generate the final segmentation mask.
[0010] In another aspect, the present invention provides a remote sensing small sample segmentation system configured to perform the above-described remote sensing small sample segmentation method.
[0011] The beneficial effects of this invention are: This invention discloses a remote sensing few-sample segmentation method based on multi-view context learning. It uses rotation-invariant features for modeling, breaking down the domain barrier between natural images and remote sensing scenes. It can achieve accurate segmentation of targets in any direction in complex backgrounds with very few reference images. Attached Figure Description
[0012] Figure 1 This is a flowchart of the present invention; Figure 2 This refers to the combined input image obtained by stitching together in an embodiment of the present invention; Figure 3 This is a block diagram illustrating the operation of the present invention; Figure 4 This is a structural block diagram of the image encoder in this invention. Detailed Implementation
[0013] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Preferred embodiments of the invention are shown in the drawings. However, the invention can be implemented in many other different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a thorough and complete understanding of the disclosure of the invention.
[0014] Reference Figure 1 and Figure 3 This application provides a remote sensing few-sample segmentation method based on multi-view context learning, including the following steps: S1. Rotate the reference image to perform data augmentation, generate multiple reference images from different perspectives, and stitch the multiple reference images from different perspectives with the image to be tested to obtain a joint input image. S2. Generate uniformly distributed positive and negative cue points within each reference image region in the joint input image. The foreground region is marked as a positive cue point, and the background region is marked as a negative cue point. Input the coordinate information of the positive and negative cue points into the cue encoder to obtain sparse cue feature embedding. This step aims to replace manual clicking and automatically generate dense point cue using a binary mask with known ground truth values to guide the model to focus on the correct ground features. S3. Input the joint input image into a pre-trained image encoder to extract features. The image encoder integrates a LoRA adapter (i.e., a low-rank adaptive fine-tuning module), which generates joint features by freezing the backbone parameters and using a self-attention mechanism to achieve feature interaction between the support image and the query image. S4. The sparse cue feature embedding and the joint feature are input into the mask decoder to generate the final segmentation mask.
[0015] In some embodiments, S1 specifically includes the following steps: S11. First, the image to be tested is prepared while maintaining the aspect ratio. and reference image Perform data processing to make the image under test and reference image The width and height are both D, and the processed image to be tested is obtained. and the processed reference image ; S12, Create a 2D resolution A 2D canvas, divided into four different quadrants: top left, bottom left, top right, and bottom right. S13. Process the reference image Rotate the image at three different angles to generate three reference images from different perspectives. The angle of rotation is θ=k 90°, of which k =0, 1, 2, 3; S14. Use three reference images from different perspectives. and the processed image to be tested The image is filled into four different quadrants of the canvas to obtain the joint input image. This layout not only preserves the image under test... For completeness, reference images will also be made. The rotational changes of the target are explicitly encoded into the input, making subsequent feature extraction rotationally invariant.
[0016] The following example illustrates the process of stitching together input images: First, create a 2D resolution. A 2D canvas is used, which is then divided into four quadrants: top left, bottom left, top right, and bottom right. The image to be tested is then displayed. Place the original view of the reference image (i.e., rotated 0 degrees) in the upper right quadrant; place the original view of the reference image in the upper left quadrant; place the view of the reference image rotated 90 degrees clockwise in the lower left quadrant; place the view of the reference image rotated 180 degrees in the lower right quadrant; see an example of combining the input images. Figure 2 As shown. The layout aims to extract rotation-invariant feature representations, i.e., joint features, through a multi-head self-attention layer of the image encoder.
[0017] In some embodiments, S2 specifically includes the following steps: S21. Generate a uniformly distributed [image] within each reference image region of the joint input image. n n (like ) grid; S22. Calculate the coordinates of the geometric center of each grid cell; S24. Check the value of the geometric center coordinates of each network on the corresponding mask map of the reference image. If the value at this position is 1, it represents the foreground area, so mark the point as a positive cue point and corresponding to a positive label; if the value at this point is 0, it represents the background area, so mark the point as a negative cue point and corresponding to a negative label. S25. Combine the geometric center coordinates of each network with the positive and negative labels to form a sequence, and input it into the cue encoder to generate a sparse cue feature embedding. .
[0018] In some embodiments, the formula for calculating the geometric center coordinates of the network in S22 is: ; ; in, 、 Representing the current geometric center of the network x, y Axis coordinates; 、 These represent the starting coordinates of the current reference image in the joint input image; Indicates the column number; Indicates the row number; 、 Let represent the horizontal step size and the vertical step size, respectively. The formulas for calculating the horizontal step size and the vertical step size are as follows: ; ; in, These represent the width and height of the current reference image, respectively; (symbol) This indicates the floor division method.
[0019] In some embodiments, the process between S22 and S24 further includes: S23. Determine whether the geometric center coordinates of each grid are within a set distance from the target contour. If so, discard the geometric center coordinates of that grid. This step prevents misjudgment caused by edge blurring, retaining only core area points with high confidence and pure background points.
[0020] In some embodiments, refer to Figure 3 and Figure 4 The image encoder is based on the Vision Transformer (ViT) architecture and includes multiple Transformer blocks connected in series. Each Transformer block includes a first normalization layer, a multi-head self-attention layer, a second normalization layer, and a multilayer perceptron connected in sequence. The input of the first normalization layer is divided into two branches, namely the first branch and the second branch. The first branch is connected to the first normalization layer, and the second branch is connected to the second normalization layer through a residual. The input of the second normalization layer is divided into two branches, namely the third branch and the fourth branch. The third branch is connected to the second normalization layer, and the fourth branch is connected to the output of the multilayer perceptron through a residual. Query matrix in multi-head self-attention layer Sum matrix The corresponding linear transformation layers all embed LoRA adapters; during training, the multi-head self-attention layer keeps its original parameters frozen, only updating and adjusting the trainable parameters within the LoRA adapter, thereby enabling the query matrix to be updated. Sum matrix Feature reconstruction; the multi-head self-attention layer achieves implicit style alignment: the test image and the reference image are automatically aligned in terms of depth interaction, lighting, tone and texture distribution; A multilayer perceptron (MLP) consists of sequentially connected dimensionality-upgrading and dimensionality-reducing layers. The dimensionality-upgrading layer is used for feature dimensionality upgrading, and the dimensionality-reducing layer is used for feature dimensionality reduction. Both the dimensionality-upgrading and dimensionality-reducing layers embed LoRA adapters. During training, the original parameters within the MLP are kept frozen, and only the trainable parameters within the LoRA adapters are updated and adjusted, thereby reconstructing the output features of the dimensionality-upgrading and dimensionality-reducing layers. The LoRA adapter requires very few training parameters to capture the high-frequency texture features unique to remote sensing images, achieving efficient domain transfer. At the same time, the LoRA adapter avoids the overfitting risk and huge computational cost associated with full fine-tuning.
[0021] Furthermore, through the LoRA adapter, the image encoder avoids direct updates to its large parameter set during fine-tuning. Taking a single multilayer perceptron as an example, the original parameter set is: , This is a channel-level approach, while the LoRA adapter only has the following parameters: .because And channel dimension It is usually 256 or greater, where, Indicates the rank; the LoRA adapter introduces fewer additional parameters than the original number of parameters. This design retains the general visual representations learned by pre-trained image encoders on large-scale datasets, while also giving the model the ability to segment remote sensing images at extremely low cost.
[0022] In some embodiments, refer to Figure 3 and Figure 4 S3 specifically includes the following steps: S31. Input the joint input image into the first Transformer block within the pre-trained image encoder; firstly, normalize the joint input image using the first normalization layer in the first Transformer block to obtain the normalization result. ,in Represents the set of real numbers. The number of elements in the sequence. For channel dimensions; S32. Normalize the results The input is fed into a multi-head self-attention layer, and the query matrix is obtained by projection through three linear transformation layers. Key matrix Sum matrix ; S33. Then the multi-head self-attention layer is based on the query matrix. Key matrix Sum matrix Calculate the self-attention feature, and add it to the input feature of the first Transformer block to output the intermediate feature. ; S34. Utilize the second normalization layer to process intermediate features. Normalization is performed, and the result is input into the upscaling layer of the multilayer perceptron to obtain the activated features. ; S35, activate the features The input is fed into the dimensionality reduction layer of a multilayer perceptron to obtain the perceptron output features. The perceptron outputs features With the intermediate features Add them together to obtain the output features of the first Transformer block. ; S36, Output features The input is fed into the remaining Transformer block to obtain the joint features.
[0023] In some embodiments, the query matrix in S32 The formula for calculation is: ; in, The weights represent the pre-trained query matrix. This represents the scaling factor used to adjust the branch weights of the LoRA adapter; and For trainable low-rank matrices for the query matrix, Let represent the rank of the low-dimensional latent space, and satisfy rank... Much smaller than channel dimension ; The bond matrix in S32 The formula for calculation is: ; in, Represents the weights of the pre-trained key matrix; The median matrix of S32 The formula for calculation is: ; in, The weights represent the values in the pre-trained matrix. 、 For trainable low-rank matrices of value matrices; The intermediate features in S33 The formula for calculation is: ; in, This represents the input features of the first Transformer block; This represents the normalized exponential function; Indicates the scaling factor; Key matrix The transpose of the matrix; This represents the weights of the pre-trained multi-head self-attention layer; Indicates transpose; The activated features in S34 The formula for calculation is: ; in, Indicates the activation function; This represents the weights of the pre-trained up-dimensional layer; This represents a trainable low-rank matrix for the up-dimensional layer; The sensor output features in S35 The formula for calculation is: ; in, These are the weights of the pre-trained dimensionality reduction layer; This represents a trainable low-rank matrix for the dimension reduction layer; The output feature in S35 The formula for calculation is: .
[0024] In some embodiments, refer to Figure 1 and Figure 3 S4 specifically includes the following steps: S41, Combine feature maps Embedding with sparse cue features Both are input into the mask decoder; S42, The mask decoder uses its internal cross-attention mechanism to embed sparse cue features. As a query matrix Query Query the image to be tested Information in the middle; S43, mask decoder combined with joint feature map Using the information retrieved from S42, output the final prediction mask; S44. The final predicted mask is upsampled to restore the original resolution, resulting in the final binarized segmentation result, which is the final segmentation mask.
[0025] This invention discloses a remote sensing few-sample segmentation method based on multi-view context learning. It uses rotation-invariant features for modeling, breaking down the domain barrier between natural images and remote sensing scenes. It can achieve accurate segmentation of targets in any direction in complex backgrounds with very few reference images.
[0026] In another aspect, the present invention provides a remote sensing small sample segmentation system configured to perform the above-described remote sensing small sample segmentation method.
[0027] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Furthermore, the technical solutions of the various embodiments of the present invention can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A remote sensing few-sample segmentation method based on multi-view context learning, characterized in that, Includes the following steps: S1. Perform rotational data augmentation on the reference image to generate multiple reference images from different perspectives. Then, stitch the multiple reference images from different perspectives with the image to be tested to obtain a joint input image. S2. Generate uniformly distributed positive and negative cue points within each reference image region in the joint input image. The foreground region is marked as a positive cue point, and the background region is marked as a negative cue point. Input the coordinate information of the positive and negative cue points into the cue encoder to obtain the sparse cue feature embedding. S3. Input the joint input image into a pre-trained image encoder to extract features and generate joint features; S4. The sparse cue feature embedding and the joint feature are input into the mask decoder to generate the final segmentation mask.
2. The remote sensing few-sample segmentation method based on multi-view context learning according to claim 1, characterized in that, S1 specifically includes the following steps: S11. First, the image to be tested is prepared while maintaining the aspect ratio. and reference image Perform data processing to make the image under test and reference image The width and height are both D, and the processed image to be tested is obtained. and the processed reference image ; S12, Create a 2D resolution A 2D canvas, divided into four different quadrants: top left, bottom left, top right, and bottom right. S13. Process the reference image Rotate the image at three different angles to generate three reference images from different perspectives. The angle of rotation is θ=k 90° ,in k =0, 1, 2, 3; S14. Use three reference images from different perspectives. and the processed image to be tested Fill the four different quadrants of the canvas to obtain the joint input image.
3. The remote sensing few-sample segmentation method based on multi-view context learning according to claim 1, characterized in that, S2 specifically includes the following steps: S21. Generate uniformly distributed n within each reference image region of the joint input image. n grids; S22. Calculate the coordinates of the geometric center of each grid cell; S24. Check the value of the geometric center coordinates of each network on the corresponding mask map of the reference image. If the value at this position is 1, it represents the foreground area. Then mark this point as a positive prompt point and corresponding to a positive label. If the value of this point is 0, it represents the background area, and this point is marked as a negative prompt point, corresponding to a negative label; S25. Combine the geometric center coordinates of each network with the positive and negative labels to form a sequence, and input it into the cue encoder to generate a sparse cue feature embedding. .
4. The remote sensing few-sample segmentation method based on multi-view context learning according to claim 3, characterized in that, The formula for calculating the geometric center coordinates of the network in S22 is: ； ； in, 、 Representing the current geometric center of the network x, y Axis coordinates; 、 These represent the starting coordinates of the current reference image in the joint input image; Indicates the column number; Indicates the row number; 、 Let represent the horizontal step size and the vertical step size, respectively. The formulas for calculating the horizontal step size and the vertical step size are as follows: ； ； in, and These represent the width and height of the current reference image, respectively; (symbol) This indicates the floor division method.
5. The remote sensing few-sample segmentation method based on multi-view context learning according to claim 4, characterized in that, Between S22 and S24, there is also: S23. Determine whether the geometric center coordinates of each grid are within the set distance from the target contour. If so, discard the geometric center coordinates of that grid.
6. The remote sensing few-sample segmentation method based on multi-view context learning according to claim 4, characterized in that, The image encoder includes multiple Transformer blocks connected in series. Each Transformer block includes a first normalization layer, a multi-head self-attention layer, a second normalization layer, and a multilayer perceptron connected in sequence. The input of the first normalization layer is divided into two branches, namely the first branch and the second branch. The first branch is connected to the first normalization layer, and the second branch is connected to the second normalization layer through a residual. The input of the second normalization layer is divided into two branches, namely the third branch and the fourth branch. The third branch is connected to the second normalization layer, and the fourth branch is connected to the output of the multilayer perceptron through a residual. Query matrix in multi-head self-attention layer Sum matrix The corresponding linear transformation layers all embed LoRA adapters; during training, the multi-head self-attention layer keeps its original parameters frozen, only updating and adjusting the trainable parameters within the LoRA adapter, thereby enabling the query matrix to be updated. Sum matrix Feature reconstruction; A multilayer perceptron consists of a dimensionality-upgrading layer and a dimensionality-reducing layer connected in sequence, and both the dimensionality-upgrading and dimensionality-reducing layers are embedded with LoRA adapters. During the training process, the original parameters within the multilayer perceptron are kept frozen and unchanged. Only the trainable parameters inside the LoRA adapters in the multilayer perceptron are updated and adjusted, thereby realizing the reconstruction of the output features of the dimensionality-upgrading and dimensionality-reducing layers.
7. The remote sensing few-sample segmentation method based on multi-view context learning according to claim 6, characterized in that, S3 specifically includes the following steps: S31. Input the joint input image into the first Transformer block within the pre-trained image encoder; firstly, normalize the joint input image using the first normalization layer in the first Transformer block to obtain the normalization result. ,in Represents the set of real numbers. The number of elements in the sequence. For channel dimensions; S32. Normalize the results The input is fed into a multi-head self-attention layer, and the query matrix is obtained by projection through three linear transformation layers. Key matrix Sum matrix ; S33. Then the multi-head self-attention layer is based on the query matrix. Key matrix Sum matrix Calculate the self-attention feature, and add it to the input feature of the first Transformer block to output the intermediate feature. ; S34. Utilize the second normalization layer to process intermediate features. Normalization is performed, and the result is input into the upscaling layer of the multilayer perceptron to obtain the activated features. ; S35, activate the features The input is fed into the dimensionality reduction layer of a multilayer perceptron to obtain the perceptron output features. and output features from the perceptron With the intermediate features Add them together to obtain the output features of the first Transformer block. ; S36, Output features The input is fed into the remaining Transformer block to obtain the joint features.
8. The remote sensing few-sample segmentation method based on multi-view context learning according to claim 7, characterized in that, The query matrix in S32 The formula for calculation is: ； in, The weights represent the pre-trained query matrix. This represents the scaling factor used to adjust the branch weights of the LoRA adapter; and For trainable low-rank matrices for the query matrix, Let represent the rank of the low-dimensional latent space, and satisfy rank... Much smaller than channel dimension ; The bond matrix in S32 The formula for calculation is: ； in, Represents the weights of the pre-trained key matrix; The median matrix of S32 The formula for calculation is: ； in, The weights represent the values in the pre-trained value matrix; 、 For trainable low-rank matrices of value matrices; The intermediate features in S33 The formula for calculation is: ； in, This represents the input features of the first Transformer block; This represents the normalized exponential function; Indicates the scaling factor; Key matrix The transpose of the matrix; This represents the weights of the pre-trained multi-head self-attention layer; Indicates transpose; The activated features in S34 The formula for calculation is: ； in, Indicates the activation function; This represents the weights of the pre-trained up-dimensional layer; This represents a trainable low-rank matrix for the up-dimensional layer; The sensor output features in S35 The formula for calculation is: ； in, These are the weights of the pre-trained dimensionality reduction layer; This represents a trainable low-rank matrix for the dimension reduction layer; The output feature in S35 The formula for calculation is: 。 9. A remote sensing few-sample segmentation method based on multi-view context learning according to claim 8, characterized in that, S4 specifically includes the following steps: S41, Combine feature maps Embedding with sparse cue features Both are input into the mask decoder; S42, The mask decoder embeds sparse cue features through its internal cross-attention mechanism. As a query matrix Query Query the image to be tested Information in the middle; S43, mask decoder combined with joint feature map Using the information retrieved from S42, output the final prediction mask; S44. The final predicted mask is upsampled to restore the original resolution, resulting in the final binarized segmentation result, which is the final segmentation mask.
10. A remote sensing small sample segmentation system, characterized in that, The remote sensing small sample segmentation method according to any one of claims 1 to 9 is configured or executed.