Remote sensing image semantic segmentation method and device based on point annotation expansion network

By guiding dynamic label expansion based on the high-dimensional feature similarity of the point annotation extension network, high-quality pseudo-labels are generated, which solves the problem of missing size and contour details in remote sensing images, significantly improves semantic segmentation accuracy and robustness, and is suitable for a small number of Point annotation dataset.

CN118968046AActive Publication Date: 2024-11-15ZHEJIANG UNIV OF TECH

Patent Information

Application Number
CN202410876596.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-02
Publication Date
2024-11-15
Estimated Expiration
2044-07-02

AI Technical Summary

Technical Problem

The existing remote sensing image semantic segmentation method relies on a large number of pixel-level labeled data, and obtaining labeled data is time-consuming and expensive, limiting the promotion of the method in practical applications, and urgently requiring semi-supervised or weakly supervised methods to reduce the demand for labeled data.

Method used

It is proposed to use point annotation extension network (PENet), dynamic tag expansion guided by high-dimensional feature similarity to generate high-quality pseudo-labels, and use SAM branches and pseudo-generated branches to improve segmentation accuracy, solving the problem of missing size and contour details.

Benefits of technology

Effectively utilize a small amount of point annotation data to generate high-quality pseudo-labels, significantly improving semantic segmentation accuracy and robustness, and is suitable for most point label remote sensing image data sets, alleviating the problem of missing size and contour details.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118968046A_ABST
    Figure CN118968046A_ABST
Patent Text Reader

Abstract

The invention discloses a remote sensing image semantic segmentation method and device based on a point annotation expansion network, and the method comprises the steps: employing dynamic label expansion guided by high-dimensional feature similarity, and achieving the precise extraction of remote sensing image semantic segmentation through the fusion of an SAM branch and a pseudo generation branch; a network model comprises an encoder, a decoder, an SAM branch, a pseudo generation branch and an EMA module. Comprising the following steps: collecting a remote sensing data set and corresponding point annotation label data, and training a constructed PENet model through the data set to obtain a trained PENet model; inputting remote sensing image data to be processed, and performing analysis and calculation through the trained PENet model to obtain a semantic segmentation result. The method is good in universality, a small amount of point annotation data can be effectively utilized to generate a high-quality pseudo label, the problem of size and contour detail missing is relieved, and semantic segmentation precision and robustness are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and more specifically, relates to a remote sensing image semantic segmentation method and device based on a point annotation extension network. Background Art

[0002] Semantic segmentation of remote sensing images is one of the important tasks in remote sensing image analysis and is widely used in environmental monitoring, urban planning, agricultural management and other fields. Existing semantic segmentation methods mainly focus on fully supervised learning models and rely on a large amount of pixel-level annotated data. However, obtaining pixel-level annotated data is time-consuming and expensive, which limits the promotion of these methods in practical applications. Therefore, an effective semi-supervised or weakly supervised method is urgently needed to reduce the demand for annotated data. Summary of the invention

[0003] The present invention aims to overcome the above-mentioned shortcomings of the prior art and provides a remote sensing image semantic segmentation method and device based on a point annotation extension network.

[0004] The Point Annotation Extension Network (PENet) proposed in this paper achieves effective utilization of point annotation data and generates high-quality pseudo labels to solve the problem of missing size and contour details through dynamic label expansion guided by high-dimensional feature similarity. PENet consists of two main branches: SAM branch and pseudo generation branch. The SAM branch generates additional pseudo labels by annotating points, and the pseudo generation branch expands pseudo labels by feature space annotation similarity, thereby improving segmentation accuracy.

[0005] To achieve the above object, according to one aspect of the present invention, a remote sensing image semantic segmentation method based on a point annotation extension network generates additional point annotations through a SAM branch, uses dynamic label extension guided by high-dimensional feature similarity to solve the problem of missing size and contour details, and then calculates feature space annotation similarity through a pseudo generation branch to expand pseudo labels; comprising:

[0006] Collect remote sensing image data sets and prepare corresponding point annotation label data, train the constructed PENet model through the data sets, and obtain a trained PENet model;

[0007] Input the remote sensing image data to be processed, analyze and calculate it through the trained PENet model, and obtain the remote sensing image semantic segmentation mask image.

[0008] In some optional embodiments, the preprocessing of the collected remote sensing image data and point annotation label data includes:

[0009] The collected remote sensing images and mask label data are cropped to a size of 256 × 256. Data augmentation is performed to expand the dataset, including horizontal and vertical flipping, rotation, blurring, and dividing the training set, validation set, and test set into a 3:1:1 ratio.

[0010] In some optional implementation schemes, the constructed PENet model includes:

[0011] Based on the FPN network architecture of ResNet18, the SAM branch is used to generate supplementary point annotations to make up for the lack of object size and contour information. The pseudo generation branch expands the pseudo labels by calculating the similarity of the annotation positions in the feature space. The deep features are used to enhance the capture of spatial information across channels through the EMA module.

[0012] In some optional embodiments, the SAM branch comprises:

[0013] The point annotations are used to generate the initial SAM pseudo-labels. The points of a single object are input as interactive prompts, while the points of other objects are input together as negative samples to improve the accuracy of the pseudo-labels.

[0014] In some optional embodiments, the pseudo generation branch includes:

[0015] The feature space distance between labeled pixels and unlabeled pixels is calculated, pseudo-generated labels are constructed through modeling, and dynamically adjusted during training to enhance supervision information.

[0016] In some optional embodiments, the EMA module comprises:

[0017] The spatial and semantic features are balanced by reshaping part of the channel dimension into the batch dimension and splitting it into multiple sub-features. The excitation mechanism generates a similarity matrix through the dot product between the input data and the parameters, while the modulation mechanism adjusts the weights according to this similarity. The final weighted representation enhances the feature representation ability of the model.

[0018] In some optional implementation schemes, the trained PENet model includes:

[0019] Input the training data into the PENet model, set the optimizer to Adam, and the initial learning rate to 0.001. Use partial cross-entropy loss supervision to calculate the loss and update the network parameters; select the best training weights through the validation set and save the weight file.

[0020] In some optional embodiments, the analysis and calculation by the trained PENet model includes:

[0021] Load the saved weight file, perform semantic segmentation on the test set remote sensing images, obtain the final semantic segmentation mask, and perform qualitative and quantitative analysis.

[0022] A second aspect of the present invention relates to a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the remote sensing image semantic segmentation method based on a point annotation extension network of the present invention.

[0023] The third aspect of the present invention relates to a remote sensing image semantic segmentation device based on a point annotation extended network, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the remote sensing image semantic segmentation method based on a point annotation extended network of the present invention is implemented.

[0024] The present invention is based on the Point Annotation Extension Network (PENet). On the basis of the traditional encoding and decoding architecture, it integrates dynamic label expansion guided by high-dimensional feature similarity. It has high universality and is suitable for semantic segmentation tasks of most point-label remote sensing image datasets.

[0025] The present invention uses the SAM branch and the pseudo generation branch to generate high-quality pseudo labels, thereby solving the problem of missing size and contour details in remote sensing images and significantly improving the accuracy and robustness of semantic segmentation.

[0026] The present invention combines the EMA module to dynamically adjust the labels, effectively enhancing the performance of self-supervised learning, improving the ability to capture spatial information across channels, and improving the accuracy of segmentation results.

[0027] The advantages of the present invention are: good universality, the ability to effectively use a small amount of point annotation data to generate high-quality pseudo-labels, alleviate the problem of missing size and contour details, and improve the accuracy and robustness of semantic segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a flow chart of the remote sensing image semantic segmentation method based on the point annotation extension network of the present invention.

[0029] Figure 2 This is a model diagram of the remote sensing image semantic segmentation method based on the point annotation extension network of the present invention.

[0030] Figure 3 This is a SAM branch module diagram of the present invention.

[0031] Figure 4 This is a pseudo-branch module diagram of the present invention.

[0032] Figure 5 This is the EMA module diagram of the present invention.

[0033] Figure 6This is a result diagram of the remote sensing image semantic segmentation method based on the point annotation extension network of the present invention. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0035] Example 1

[0036] The remote sensing image building extraction method based on the dual-path jump attention mechanism of the present invention has the following implementation process: Figure 1 As shown, it mainly includes the following steps:

[0037] Step 1: Collect remote sensing image datasets and prepare corresponding point annotation label data, train the constructed PENet model through the dataset, and obtain a trained PENet model.

[0038] Step 2: Input the remote sensing image data to be processed, analyze and calculate it through the trained PENet model, and obtain the remote sensing image semantic segmentation mask image.

[0039] Figure 2 This is a model diagram of the remote sensing image semantic segmentation method based on the point annotation extension network. It can be specifically divided into the encoding and decoding architecture, SAM branch and pseudo generation branch process:

[0040] The encoding is done through the ResNet18 pre-trained network, and then the intermediate features of different scales and the same number of channels are obtained after the 1×1 convolution operation. For deep features, the EMA module is used to enhance the spatial information capture across channels. The upsampled features are used to calculate the channel sum, and finally the segmentation head is used to output the category mask.

[0041] The supervision information is obtained by processing point labels through the SAM branch, supplemented by the dynamic expansion of the feature space before the segmentation head through the pseudo generation branch. The Loss uses partial cross entropy loss.

[0042] Figure 3 This is the SAM branch module diagram:

[0043] When inputting the points of a single object as interactive prompts, other object points are input as negative samples to improve the accuracy of pseudo labels. The point positions in the label map are traversed one by one, and the preliminary segmentation results of each position are obtained according to the positive and negative sample inputs. Finally, the semantic labels assigned to the point positions are integrated to obtain pseudo labels.

[0044] Figure 4 This is a pseudo-generation branch module diagram:

[0045] Use the SAM branch to obtain the initial supervision information and divide the input pixels into labeled pixels x l and unlabeled pixels x u . Calculate the feature space distance between labeled pixels and unlabeled pixels, construct pseudo-generated labels through modeling, and dynamically adjust them during training to enhance supervision information. The specific implementation steps are as follows:

[0046] For a given input image with annotated categories, the labeled pixels x are constructed by computing the average features of these pixels l Prototype:

[0047]

[0048] After obtaining the average feature prototype, calculate the distance and variance:

[0049] d i =f(x)-μ i (2)

[0050]

[0051] Model the distribution of features between labeled and unlabeled based on the distance and variance calculated above:

[0052]

[0053] The result of G is a value between 0 and 1 representing the probability of each pixel belonging to each class as a soft score. These scores are self-supervised using the segmentation prediction results.

[0054] Figure 5 This is the EMA module diagram:

[0055] Input is X, divided into several X ′ Perform group operations. The incentive mechanism generates a similarity matrix by calculating the dot product between the input data and the parameters, where each element of the matrix represents the similarity between part of the input data and the parameters. The higher the similarity, the more important the part is to the current task. The modulation mechanism then uses the similarity matrix derived from the incentive mechanism to adjust the weights of each part. The weight vector is obtained by applying softmax normalization to each row of the similarity matrix, which reflects the importance of each part to the current task. Finally, the weight vector is used to perform weighted summation on the input data to obtain the network's weighted representation of the input data. The mathematical operation is as follows:

[0056] M1(X ′ )=f 1×1 ([XAvgPool(X ′ ),YAvgPool(X ′ )])⊙X ′, (5)

[0057] M2(X ′ )=σ2(f 3×3 (X ′ )), (6)

[0058] M3(X ′ )=AvgPool(f 3×3 (X ′ )), (7)

[0059]

[0060] Among them, σ1, σ2, ⊙, They represent the sigmoid function, softmax function, channel convolution operation, and element multiplication operation respectively.

[0061] Figure 6 This is a result diagram of the remote sensing image semantic segmentation method based on the point annotation extension network, which is used to demonstrate the remote sensing semantic segmentation effect based on the method of the present invention.

[0062] Example 2

[0063] This embodiment relates to a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the remote sensing image semantic segmentation method based on a point annotation extension network of embodiment 1 is implemented.

[0064] Example 3

[0065] The present embodiment relates to a remote sensing image semantic segmentation device based on a point annotation extension network, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the remote sensing image semantic segmentation method based on a point annotation extension network of Example 1 is implemented.

[0066] The above-mentioned embodiment only expresses one implementation mode of the present invention, and its description is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A remote sensing image semantic segmentation method based on a point annotation extension network, characterized by: The following steps are involved: Collect remote sensing image data sets and prepare corresponding point annotation label data, train the constructed PENet model through the data sets, and obtain a trained PENet model; Input the remote sensing image data to be processed, analyze and calculate it through the trained PENet model, and obtain the remote sensing image semantic segmentation mask image.

2. The method according to claim 1, characterized in that The method of collecting remote sensing image data sets and preparing corresponding point annotation label data includes: The collected remote sensing images and mask label data are cropped to a size of 256 × 256. Data augmentation is performed to expand the dataset, including horizontal and vertical flipping, rotation, blurring, and dividing the training set, validation set, and test set into a 3:1:1 ratio.

3. The method according to claim 1, characterized in that: The constructed PENet model includes: Based on the FPN network architecture of ResNet18, the SAM branch is used to generate supplementary point annotations to make up for the lack of object size and contour information. The pseudo generation branch expands the pseudo labels by calculating the similarity of the annotation positions in the feature space. The deep features are used to enhance the capture of spatial information across channels through the EMA module.

4. The method according to claim 3, characterized in that The SAM branch includes: Initial SAM pseudo-labels are generated using point annotations; points of a single object are input as interaction cues while other object points are input together as negative samples to improve the accuracy of pseudo-labels.

5. The method according to claim 3, characterized in that: The pseudo generation branch includes: The feature space distance between labeled pixels and unlabeled pixels is calculated, pseudo-generated labels are constructed through modeling, and dynamically adjusted during training to enhance supervision information.

6. The method according to claim 3, characterized in that The EMA module includes: The spatial and semantic features are balanced by reshaping part of the channel dimension into the batch dimension and splitting it into multiple sub-features. The excitation mechanism generates a similarity matrix through the dot product between the input data and the parameters, while the modulation mechanism adjusts the weights according to this similarity; the final weighted representation enhances the feature representation ability of the model.

7. The method according to claim 1, characterized in that The analysis and calculation by the trained PENet model include: Input the training data into the PENet model, set the optimizer to Adam, and the initial learning rate to 0.

001. Use partial cross-entropy loss supervision to calculate the loss and update the network parameters; select the best training weights through the validation set and save the weight file.

8. The method according to claim 1, characterized in that The trained PENet model includes: Load the saved weight file, perform semantic segmentation on the test set remote sensing images, obtain the final semantic segmentation mask, and perform qualitative and quantitative analysis.

9. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, the remote sensing image semantic segmentation method based on a point annotation extension network described in any one of claims 1 to 8 is implemented.

10. A remote sensing image semantic segmentation device based on a point annotation extension network, comprising a memory and a processor, wherein: The memory stores executable codes, and when the processor executes the executable codes, the remote sensing image semantic segmentation method based on point annotation extension network described in any one of claims 1 to 8 is implemented.

Citation Information

Patent Citations

  • Cross consistency self-training remote sensing image semantic segmentation network training method and device

    CN114283285A

  • Weak supervision deep learning classification method based on remote sensing image point semantic tag

    CN117746130A

  • Method and system of annotation densification for semantic segmentation

    US20200211200A1

Cited By

  • Remote sensing image building semantic segmentation system based on visual language model

    CN120388178A

  • Camouflage target detection method for pseudo label optimization based on point supervision

    CN120876840A

  • A Pseudo-Label Optimization Method for Camouflaged Target Detection Based on Point Supervision

    CN120876840B