A method and apparatus for semantic segmentation of remote sensing images based on point annotation extended networks

Abstract

A method and apparatus for semantic segmentation of remote sensing images based on point annotation extended networks are presented. The method utilizes dynamic label extension guided by high-dimensional feature similarity, fusing SAM and pseudo-generation branches to achieve accurate extraction of semantic segments from remote sensing images. The network model includes an encoder, decoder, SAM branch, pseudo-generation branch, and EMA module. The process involves: collecting remote sensing datasets and corresponding point annotation label data; training a constructed PENet model using the dataset to obtain a trained PENet model; inputting the remote sensing image data to be processed; and analyzing and calculating the semantic segmentation result using the trained PENet model. This invention has good universality, effectively utilizing a small amount of point annotation data to generate high-quality pseudo-labels, alleviating the problem of missing size and contour details, and improving the accuracy and robustness of semantic segmentation.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and more specifically, relates to a method and apparatus for semantic segmentation of remote sensing images based on point annotation extended networks. Background Technology

[0002] Semantic segmentation of remote sensing images is a crucial task in remote sensing image analysis, widely applied in fields such as environmental monitoring, urban planning, and agricultural management. Existing semantic segmentation methods primarily focus on fully supervised learning models, relying on large amounts of pixel-level labeled data. However, acquiring pixel-level labeled data is time-consuming and expensive, limiting the widespread adoption of these methods in practical applications. Therefore, there is an urgent need for an effective semi-supervised or weakly supervised method to reduce the requirement for labeled data. Summary of the Invention

[0003] The present invention aims to overcome the above-mentioned shortcomings of the prior art and provides a method and apparatus for semantic segmentation of remote sensing images based on point annotation extended networks.

[0004] The Point Annotation Extension Network (PENet) proposed in this invention achieves effective utilization of point annotation data and generates high-quality pseudo-labels through dynamic label expansion guided by high-dimensional feature similarity, thereby addressing the problem of missing size and contour details. PENet comprises two main branches: the SAM branch and the pseudo-generation branch. The SAM branch generates additional pseudo-labels by adding to the point annotations, while the pseudo-generation branch expands the pseudo-labels through feature space annotation similarity, thus improving segmentation accuracy.

[0005] To achieve the above objectives, according to one aspect of the present invention, a remote sensing image semantic segmentation method based on a point annotation extension network generates additional point annotations through a SAM branch, utilizes dynamic label expansion guided by high-dimensional feature similarity to address the problem of missing size and contour details, and then calculates pseudo-labels through a pseudo-generation branch to expand feature space annotation similarity; including:

[0006] Collect remote sensing image datasets and prepare corresponding point annotation label data. Train the constructed PENet model using the dataset to obtain the trained PENet model.

[0007] Input remote sensing image data to be processed, and analyze and calculate it through the trained PENet model to obtain the semantic segmentation mask image of the remote sensing image.

[0008] In some alternative implementations, the preprocessing of the collected remote sensing image data and point annotation label data includes:

[0009] The collected remote sensing images and mask label data were cropped to a size of 256×256. Data augmentation was then performed to expand the dataset, including horizontal and vertical flipping, rotation, and blurring. The training, validation, and test sets were then divided in a 3:1:1 ratio.

[0010] In some alternative implementations, the constructed PENet model includes:

[0011] Based on the ResNet18-based FPN network architecture, the SAM branch is used to generate supplementary point annotations to compensate for the lack of object size and contour information. The pseudo-generation branch expands the pseudo-labels by calculating the similarity of the annotations in the feature space. Deep features are used by the EMA module to enhance the capture of spatial information across channels.

[0012] In some alternative implementations, the SAM branch includes:

[0013] Initial SAM pseudo-labels are generated using point annotations. Points from individual objects are used as interactive cues, while points from other objects are input as negative samples to improve the accuracy of the pseudo-labels.

[0014] In some alternative implementations, the pseudo-generated branch includes:

[0015] The feature space distance between labeled and unlabeled pixels is calculated, pseudo-generated labels are constructed by modeling, and dynamically adjusted during training to enhance the supervision information.

[0016] In some alternative implementations, the EMA module includes:

[0017] To balance spatial and semantic features, a portion of the channel dimension is reshaped into a batch dimension and broken down into multiple sub-features. An activation mechanism generates a similarity matrix through the dot product between the input data and parameters, while a modulation mechanism adjusts the weights based on this similarity. The final weighted representation enhances the model's feature representation capabilities.

[0018] In some alternative implementations, the trained PENet model includes:

[0019] Input the training data into the PENet model, set the optimizer to Adam, and the initial learning rate to 0.001. Calculate the loss and update the network parameters using partial cross-entropy loss for supervision; select the best training weights from the validation set and save the weight file.

[0020] In some alternative implementations, the analysis and computation performed using the trained PENet model includes:

[0021] Load the saved weight file, perform semantic segmentation on the test set remote sensing images, obtain the final semantic segmentation mask, and perform qualitative and quantitative analysis.

[0022] A second aspect of the invention relates to a computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the remote sensing image semantic segmentation method based on point annotation extended networks of the present invention.

[0023] A third aspect of the present invention relates to a remote sensing image semantic segmentation apparatus based on a point annotation extended network, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the remote sensing image semantic segmentation method based on a point annotation extended network of the present invention.

[0024] This invention is based on Point Annotation Extension Network (PENet). On the basis of traditional encoder-decoder architecture, it integrates dynamic label extension guided by high-dimensional feature similarity, which has high universality and is applicable to semantic segmentation tasks of most point-labeled remote sensing image datasets.

[0025] This invention uses SAM branch and pseudo-generation branch to solve the problem of missing size and contour details in remote sensing images by generating high-quality pseudo labels, which significantly improves the accuracy and robustness of semantic segmentation.

[0026] This invention combines the EMA module to dynamically adjust the labels, effectively enhancing the performance of self-supervised learning, improving the ability to capture spatial information across channels, and improving the accuracy of segmentation results.

[0027] The advantages of this invention are: good universality, it can effectively utilize a small amount of point annotation data to generate high-quality pseudo-labels, alleviate the problem of missing size and contour details, and improve semantic segmentation accuracy and robustness. Attached Figure Description

[0028] Figure 1 This is a flowchart of the remote sensing image semantic segmentation method based on point annotation extended network of the present invention.

[0029] Figure 2 This is a model diagram of the remote sensing image semantic segmentation method based on point annotation extended network of the present invention.

[0030] Figure 3 This is a diagram of the SAM branch module of the present invention.

[0031] Figure 4 This is a diagram of the pseudo-branch module of the present invention.

[0032] Figure 5 This is a diagram of the EMA module of the present invention.

[0033] Figure 6The image shows the result of the remote sensing image semantic segmentation method based on point annotation extended network of the present invention. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0035] Example 1

[0036] The method for extracting buildings from remote sensing images based on a dual-path skip attention mechanism of the present invention is implemented as follows: Figure 1 As shown, it mainly includes the following steps:

[0037] Step 1: Collect remote sensing image datasets and prepare corresponding point annotation label data. Train the constructed PENet model using the dataset to obtain the trained PENet model.

[0038] Step 2: Input the remote sensing image data to be processed, and analyze and calculate it through the trained PENet model to obtain the semantic segmentation mask image of the remote sensing image.

[0039] Figure 2 This is a model diagram of a remote sensing image semantic segmentation method based on a point annotation extended network. Specifically, it can be divided into an encoding / decoding architecture, a SAM branch, and a pseudo-generation branch process:

[0040] Encoding is performed using a ResNet18 pre-trained network, followed by 1×1 convolutional operations to obtain intermediate features with the same number of channels at different scales. For deep features, an EMA module is used to enhance the capture of spatial information across channels. Upsampled features are used to calculate channel sums, and finally, a class mask is output through a segmentation head.

[0041] The supervision information is obtained by processing point labels in the SAM branch, supplemented by dynamic expansion of the feature space before the segmentation head through a pseudo-generated branch. The loss uses partial cross-entropy loss.

[0042] Figure 3 Here is the SAM branch module diagram:

[0043] To improve the accuracy of pseudo-labels, points from individual objects are used as interactive prompts, while points from other objects are input as negative samples. The point positions in the label map are traversed one by one, with positive and negative samples input to obtain preliminary segmentation results for each position. Finally, semantic labels are assigned to the point positions and integrated to obtain pseudo-labels.

[0044] Figure 4 This is a pseudo-branch generation module diagram:

[0045] The initial supervision information is obtained by using the SAM branch, and the input pixels are divided into labeled pixels x. l and unlabeled pixels x u The feature space distance between labeled and unlabeled pixels is calculated. Pseudo-generated labels are constructed through modeling and dynamically adjusted during training to enhance supervision. The specific implementation steps are as follows:

[0046] For a given input image with annotation categories, labeled pixels x are constructed by calculating the average features of these pixels. l The prototype:

[0047]

[0048] After obtaining the average feature prototype, calculate the distance and variance:

[0049] d i =f(x)-μ i (2)

[0050]

[0051] Based on the distance and variance calculated above, a model is constructed for the feature distribution between labeled and unlabeled samples:

[0052]

[0053] The result of G is a value between 0 and 1, representing the probability of each pixel belonging to each class as a soft score. These scores are self-supervised using the segmentation prediction results.

[0054] Figure 5 This is the EMA module diagram:

[0055] The input is X, and it is divided into several X's. ′ Group operations are performed. The activation mechanism generates a similarity matrix by calculating the dot product between the input data and the parameters, where each element of the matrix represents the similarity between a portion of the input data and the parameters. The higher the similarity, the more important that portion is to the current task. Then, the modulation mechanism uses the similarity matrix derived from the activation mechanism to adjust the weights of each portion. A weight vector is obtained by applying softmax normalization to each row of the similarity matrix, which reflects the importance of each portion to the current task. Finally, the input data is weighted and summed using this weight vector to obtain the network's weighted representation of the input data. The mathematical operations are as follows:

[0056] M1(X ′ )=f 1×1 ([XAvgPool(X ′ ),YAvgPool(X ′ )])⊙X ′(5)

[0057] M2(X ′ )=σ2(f 3×3 (X ′ (6)

[0058] M3(X ′ =AvgPool(f 3×3 (X ′ (7)

[0059]

[0060] Where σ1, σ2, ⊙, These represent the sigmoid function, softmax function, channel convolution operation, and element-wise multiplication operation, respectively.

[0061] Figure 6 This is a result image of the remote sensing image semantic segmentation method based on point annotation extended network, used to demonstrate the remote sensing semantic segmentation effect based on the method of this invention.

[0062] Example 2

[0063] This embodiment relates to a computer-readable storage medium storing a program that, when executed by a processor, implements the remote sensing image semantic segmentation method based on point annotation extended networks as described in Embodiment 1.

[0064] Example 3

[0065] This embodiment relates to a remote sensing image semantic segmentation device based on point annotation extended networks, including a memory and a processor. The memory stores executable code, and when the processor executes the executable code, it implements the remote sensing image semantic segmentation method based on point annotation extended networks of Embodiment 1.

[0066] The above-described embodiments are merely one implementation of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A semantic segmentation method for remote sensing images based on point annotation extended networks, characterized in that: Includes the following steps: Collect remote sensing image datasets and prepare corresponding point annotation label data. Train the constructed PENet model using the dataset to obtain the trained PENet model. Input remote sensing image data to be processed, analyze and calculate through the trained PENet model to obtain a semantic segmentation mask image of the remote sensing image; The constructed PENet model includes: a ResNet18-based FPN network architecture, a SAM branch for generating supplementary point annotations to compensate for the lack of object size and contour information, a pseudo-generation branch for expanding pseudo-labels by calculating the similarity of annotations in feature space, and deep features for enhancing cross-channel spatial information capture through the EMA module. The SAM branch includes: generating initial SAM pseudo-labels using point annotations; and inputting points of a single object as interactive prompts while inputting points of other objects as negative samples to improve the accuracy of the pseudo-labels. The pseudo-generation branch includes: calculating the feature space distance between labeled pixels and unlabeled pixels, constructing pseudo-generated labels through modeling, and dynamically adjusting them during training to enhance supervision information; The EMA module includes: reshaping a portion of the channel dimension into a batch dimension and splitting it into multiple sub-features to balance spatial and semantic features; an activation mechanism generates a similarity matrix through the dot product between input data and parameters, while a modulation mechanism adjusts the weights based on this similarity; and the final weighted representation enhances the model's feature representation capability.

2. The method according to claim 1, characterized in that, The collection of remote sensing image datasets and the preparation of corresponding point annotation label data include: The collected remote sensing images and mask label data were cropped to a size of 256×256; data augmentation was performed to expand the dataset, including horizontal and vertical flipping, rotation, and blurring, and the training set, validation set, and test set were divided in a 3:1:1 ratio.

3. The method according to claim 1, characterized in that, The analysis and computation performed using the trained PENet model include: Input the training data into the PENet model, set the optimizer to Adam, and the initial learning rate to 0.001; use partial cross-entropy loss for supervision to calculate the loss and update the network parameters; select the best training weights through the validation set and save the weight file.

4. The method according to claim 1, characterized in that, The trained PENet model includes: Load the saved weight file, perform semantic segmentation on the test set remote sensing images, obtain the final semantic segmentation mask, and perform qualitative and quantitative analysis.

5. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements the remote sensing image semantic segmentation method based on point annotation extended networks as described in any one of claims 1-4.

6. A remote sensing image semantic segmentation device based on a point annotation extended network, comprising a memory and a processor, wherein, The memory stores executable code, and when the processor executes the executable code, it implements the remote sensing image semantic segmentation method based on point annotation extended network as described in any one of claims 1-4.

Images