Remote sensing image small sample segmentation method based on depth mutual information and variational encoder

Through the combination of deep mutual information maximization and independent variation encoder, the problem of intrinsic connection unmined and prototype confusion in the small sample semantic segmentation method is solved, and the performance and accuracy of the semantic segmentation algorithm are significantly improved.

CN118941792APending Publication Date: 2024-11-12NORTHWESTERN POLYTECHNICAL UNIV
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202410991321.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-23
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The existing small sample semantic segmentation method fails to fully explore the inherent connection between supporting images and querying images, resulting in the inability to accurately activate the homologous object area, and there are prototype confusion problems such as foreground background mixing.

Method used

The deep mutual information maximization method is used to maximize the mutual information between the support set and the query set, and the initial prototype is decoded into a foreground prototype and a background prototype through an independent variable encoder to solve the problem of prototype confusion.

Benefits of technology

The performance and accuracy of semantic segmentation algorithm are improved, the robustness and accuracy of the system are enhanced, and the accuracy of the segmentation results are significantly improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941792A_ABST
    Figure CN118941792A_ABST
Patent Text Reader

Abstract

The invention discloses a remote sensing image small sample segmentation method based on depth mutual information and a variational encoder, and the method comprises the steps: maximizing the mutual information between a support set and a query set through a depth mutual information maximization method, thereby fully mining the internal relation between a support image and a query image; and implicit information in the prototype is obtained through the characterization capability of the self-variation encoder, and the initial prototype is decoded into a foreground prototype and a background prototype, so that the problem of prototype confusion is solved. According to the method, the limitation of an existing small sample semantic segmentation method is solved, and the speed of a semantic segmentation algorithm is further improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to a remote sensing image small sample segmentation method based on deep mutual information and variational encoder. Background Art

[0002] Semantic segmentation is a fundamental computer vision task that aims to classify each pixel of an image into a set of predefined semantic classes. Current methods are mainly based on fully supervised segmentation methods using fully convolutional networks. Although traditional fully supervised methods have achieved good performance, they are very dependent on datasets with a large number of pixel-level annotations, which brings a lot of time and manpower costs to practical applications. In addition to the requirements for training data, these models also have poor generalization capabilities for agnostic categories. In order to cope with the above problems, few-shot semantic segmentation methods have been proposed. Given a set of images to be segmented (support set), few-shot semantic segmentation methods use information from several densely annotated samples (query set) to identify target regions containing predefined semantic categories.

[0003] Existing few-shot segmentation methods usually learn from a few support images, and then input the learned knowledge into the parameter module for segmentation query. Specifically, C. Zhang et al. introduced a small-shot segmentation method CANet using masked average pooling in the paper "CANet: Class-agnostics segmentation networks with iterative refinement and attentive few-shot learning," in Proc. IEEE / CVF Conf. Comput. Vis. Pattern Recognit., Jun. 2019, pp. 5212–5221. This method is a new segmentation network that uses a dual-branch dense comparison module and an iterative optimization module for multi-level feature comparison and refinement and uses a masked average pooling method to collect guidance information that specifically considers pixels belonging to support images. M. Siam et al. proposed an adaptive mask proxy strategy in the paper "Adaptive masked weight imprinting for few-shot segmentation," 2019, arXiv:1902.11123. This strategy uses multi-resolution average pooling of mask embeddings to construct the final segmentation layer weights for new classes.

[0004] These methods all have their limitations. First, they do not fully exploit the intrinsic relationship between the support image and the query image, and cannot represent and activate the homologous object regions. Second, they do not correct the prototype confusion caused by the foreground-background mixing phenomenon brought about by mask average pooling. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, the present invention provides a remote sensing image small sample segmentation method based on deep mutual information and variational encoder. First, the mutual information between the support set and the query set is maximized through the deep mutual information maximization method, so as to fully explore the intrinsic connection between the support image and the query image; then the implicit information in the prototype is obtained through the representation ability of the self-variational encoder, and the initial prototype is decoded into a foreground prototype and a background prototype, so as to solve the prototype confusion problem. The present invention solves the limitations of the existing small sample semantic segmentation method and further improves the speed of the semantic segmentation algorithm.

[0006] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0007] Step 1: Data division and preprocessing;

[0008] The training images are divided into support set images s and query set images q according to the small sample learning problem setting, and then the input images are center cropped, resized, and normalized, and the processed images are used as training samples X of the feature learning algorithm;

[0009] Step 2: Use the pre-trained ResNet101 network to extract features from the support set images and query set images respectively;

[0010] First, the support set images and query set images are input into the pre-trained ResNet101 network. Through the multi-layer convolution and pooling operations of the network, the middle-level features of the images are extracted. and high-level features F s 、F q ;

[0011] Step 3: Targeting the acquired high-level features F s 、F q , use the mutual information maximization method to calculate F s 、F q The mutual information between s ,F q ), the calculation method is as follows:

[0012]

[0013] Where: I(F s ; F q ): represents the high-level feature F sand F q The mutual information between

[0014] x∈F s : Indicates that in the feature set F s A feature x in

[0015] y∈F q : Indicates that in the feature set F q A feature y in

[0016] p(x,y): represents the joint probability distribution of features x and y appearing simultaneously;

[0017] p(x): represents the probability distribution of feature x;

[0018] p(y): represents the probability distribution of feature y;

[0019] A discriminator D is introduced to estimate and maximize the mutual information between the global features of the support set images and the query set images;

[0020] Use InfoNCE loss to construct the loss function;

[0021] Step 4: Output the middle-level features of step 2 Through a prior generator, which uses mid-level features and high-level features to generate a prior mask V of the query image q , the prior mask contains preliminary information about the target area in the query image; specifically, first calculate The cosine similarity of each pixel is calculated, and then the maximum similarity among all supporting pixels is taken as the response value, and finally normalized;

[0022]

[0023] in: Indicates support for image features Image features The cosine similarity between ;

[0024] Supports mid-level features of images;

[0025] Check the mid-level features of the image;

[0026] T: represents the transpose operation;

[0027] ∥·∥: represents the norm of a vector;

[0028] Y Q : represents the prior mask of the query image;

[0029] max is generated by taking the response value of the support pixel with the maximum similarity to the query pixel;

[0030] Normalization: Prior mask Y Q Normalize it so that its value is between 0 and 1;

[0031] min(Y Q ) and max(Y Q ) represent Y Q The minimum and maximum values ​​in ;

[0032] ∈ is a constant used to avoid the situation where the denominator is zero;

[0033] Step 5: Use the deep mutual information maximization module to extract the high-level features F of the output support image s Generate an initial prototype V of the support image through mask average pooling s ; Specifically, mask average pooling performs weighted averaging of high-level features according to the mask information in the support image, thereby obtaining an initial prototype that represents the overall features of the support image;

[0034]

[0035] M q : indicates the mask of the support image;

[0036] A pool (·): indicates pooling operation;

[0037] Represents the high-level features of the supporting image;

[0038] V s : represents the feature vector after pooling;

[0039] represents the normalized feature vector;

[0040] Step 6: Targeting Initial Prototype V s Perform prototype decoupling operations;

[0041] By changing V s Input the autoencoder structure, extract the implicit variable z of the prototype and decode it into the background prototype V through regularization constraints. b With the foreground prototype V f , only select the foreground prototype V f As output combined with F q Perform semantic segmentation operation:

[0042]

[0043] z=μ φ(x)+σ φ (x)⊙∈

[0044]

[0045] z: represents hidden variables;

[0046] μ φ (x): represents the mean function of input x;

[0047] σ φ (x): represents the standard deviation function of input x;

[0048] ∈: represents the noise sampled from the standard normal distribution;

[0049] ⊙: represents element-by-element multiplication operation;

[0050] represents normal distribution;

[0051] q φ (z|x): represents the posterior distribution of the latent variable z given x;

[0052] D(·): represents the discriminator;

[0053] Indicates expected value;

[0054] G(·): represents the generator.

[0055] Preferably, the loss function constructed using InfoNCE loss is:

[0056]

[0057] N: represents the number of samples;

[0058] K: represents the number of negative samples;

[0059] Represents the feature vector of the positive sample that matches the i-th sample;

[0060] Represents the feature vector of negative samples that do not match the i-th sample;

[0061] z i : represents the feature vector of the i-th sample.

[0062] The beneficial effects of the present invention are as follows:

[0063] 1. The system is more robust. Using the mutual information maximization method, the mutual information between the support and query global features is estimated and maximized through the discriminator D. This method enhances the correlation between global features, helps to classify and segment images more accurately, and improves the performance of the overall algorithm.

[0064] 2. The system accuracy is higher. s Perform prototype decoupling operation, extract the implicit variables of the prototype through the autoencoder structure, and decode it into the background prototype V through regularization constraints. b With the foreground prototype V f This decoupling operation can effectively separate foreground and background features, thereby improving the accuracy of target region segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 It is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0066] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0067] The present invention discloses a novel remote sensing image small sample segmentation method based on deep mutual information and variational encoder, which mainly solves the limitations of existing small sample semantic segmentation methods and further improves the speed of semantic segmentation algorithms. Specifically, the purpose of the present invention is to improve the following aspects.

[0068] 1. Through the deep mutual information maximization method, the mutual information between the support set and the query set is maximized, so as to fully explore the intrinsic connection between the support image and the query image.

[0069] 2. The implicit information in the prototype is obtained through the representation capability of the autovariation encoder, and the initial prototype is decoded into foreground prototype and background prototype, thereby solving the prototype confusion problem.

[0070] The technical solution of the present invention includes two modules: a deep mutual information maximization module and a prototype decoupling and segmentation module. Figure 1 The left side is the flowchart of the deep mutual information maximization module, and the right side is the flowchart of the prototype decoupling and segmentation module.

[0071] Deep Mutual Information Maximization Module:

[0072] 1. First, the present invention processes the segmented images in the small sample segmentation training set to improve the training effect of the feature learning algorithm. Specifically, the segmented images are firstly center cropped to reduce unnecessary edge information. Then, the cropped images are resized to meet the input requirements of the algorithm and ensure the consistency of the image size. Subsequently, the resized images are normalized and preprocessed using data enhancement technology. The diversity of the training samples is increased by operations such as rotation, flipping, and scaling, and the processed images are used as training samples X of the feature learning algorithm.

[0073] 2. For training samples, a pre-trained ResNet101 network is used to extract features from the support set images and query set images. Specifically, the support set images and query set images are first input into the pre-trained ResNet101 network, and the mid-level features of these images are extracted through the multi-layer convolution and pooling operations of the network. and high-level features F s 、F q , obtain the feature representation of each image in the support set and query set at different network levels.

[0074] 3. For the acquired high-level features F s 、F q Use the mutual information maximization method to calculate F s 、F q The mutual information between I(F5,F q ). A discriminator D is introduced to estimate and maximize the mutual information between support and query global features. The discriminator attempts to distinguish feature pairs obtained from the same input data from feature pairs obtained from different input data. The InfoNCE loss is used to construct the loss function to enhance the correlation between global features.

[0075] Prototype decoupling and split modules:

[0076] 1. Use the deep mutual information maximization module to extract the output mid-level features and high-level features F s 、F q Through a prior generator, which uses these features to generate a prior mask V of the query image q This prior mask contains preliminary information about the target region in the query image.

[0077] 2. Use the deep mutual information maximization module to extract the high-level features F of the output support image s Generate an initial prototype V of the support image through mask average pooling sSpecifically, mask average pooling performs weighted averaging of high-level features according to the mask information in the support image, thereby obtaining an initial prototype that represents the overall features of the support image.

[0078] 3. For the initial prototype V s Prototype decoupling is performed, specifically by changing V s Input the autoencoder structure, extract the implicit variables of the prototype, and through the regularization constraint, decode it into the background prototype V b With the foreground prototype V f , only select the foreground prototype V f As output combined with F q Perform semantic segmentation operations.

[0079] Example:

[0080] Step 1: First, data division and preprocessing. The training images are divided into support set s and query set q according to the small sample learning problem setting. Then the input images are center cropped, resized, and normalized, and the processed images are used as training samples X for the feature learning algorithm.

[0081] Step 2: Use the pre-trained ResNet101 network to extract features from the support set images and query set images. First, input the support set images and query set images into the pre-trained ResNet101 network, and extract the mid-level features of these images through the network's multi-layer convolution and pooling operations. and high-level features F s 、F q .

[0082] Step 3: For the obtained high-level features F s 、F q Use the mutual information maximization method to calculate F s 、F q The mutual information between s ,F q ), the calculation method is as follows:

[0083]

[0084] A discriminator D is introduced to estimate and maximize the mutual information between support and query global features. InfoNCE loss is used to construct the loss function to enhance the correlation between global features.

[0085]

[0086] Step 4: Use the deep mutual information maximization module to extract the output mid-level features and high-level features F s 、Fq Through a prior generator, which uses these features to generate a prior mask V of the query image q This prior mask contains preliminary information about the target region in the query image.

[0087] Step 5: Use the deep mutual information maximization module to extract the high-level features F of the output support image s Generate an initial prototype V of the support image through mask average pooling s Specifically, mask average pooling performs weighted averaging of high-level features according to the mask information in the support image, thereby obtaining an initial prototype that represents the overall features of the support image.

[0088]

[0089] Step 6: For the initial prototype V s Prototype decoupling is performed, specifically by changing V s Input the autoencoder structure, extract the implicit variable z of the prototype and pass the regularization constraint to make it decoded into the background prototype V b With the foreground prototype V f , only select the foreground prototype V f As output combined with F q Perform semantic segmentation operations.

[0090]

[0091] z=μ φ (z)+σ φ (z)⊙∈

[0092]

[0093] The effect of the present invention can be further illustrated by the following simulation experiment.

[0094] 1. Simulation conditions

[0095] The present invention conducts simulation experiments on a central processing unit Intel(R) Core(TM) i7-10700F CPU, 16G memory, a graphics processor NVIDIA GeForce RTX 1660SUPER GPU, and a Windows 10 operating system using Python software and a PyTorch deep learning framework. The data set used in the simulation is a public data set.

[0096] 2. Simulation content

[0097] First, the training set is used to perform feature extraction and mutual information maximization according to the part of the deep mutual information maximization module in the specific implementation method. Then, the method in prototype decoupling and classification is combined with the accuracy evaluation standard of image semantic segmentation to perform accuracy evaluation.

[0098] In order to demonstrate the effectiveness of the algorithm, the category-free segmentation network (CAnet) with iterative refinement and focused few-shot learning, the pyramid graph network (PGnet) with connected attention, and the prior-guided feature enhancement network (PFEnet) were selected for algorithm comparison.

[0099] The CAnet method is introduced in detail in the paper "C.Zhang, G.Lin, F.Liu, R.Yao, and C.Shen.Canet:Class-agnostic segmentation networks with iterative refinement and attentive few-shot learning.In CVPR, 2019."; the PGnet method is introduced in detail in the paper "C.Zhang, G.Lin, F.Liu, J.Guo, Q.Wu, and R.Yao.Pyramid graph networks with connection attentionsfor region-based oneshot semantic segmentation.In ICCV, 2019."; the PFEnet method is introduced in detail in the paper "Z.Tian, ​​H.Zhao, M.Shu, Z.Yang, R.Li, and J.Jia,"Prior guided feature enrichment network for few-shot segmentation."IEEE Transactions on Pattern Analysis and Machine Intelligence, p.1050–1065, Aug 2020." The comparison results on the PASCAL-5i dataset are shown in Table 1.

[0100] Table 1

[0101] Method CAnet PGnet PFEnet Ours Accuracy 55.4% 56.0% 60.8% 65.6%

[0102] As can be seen from Table 1, the segmentation accuracy of the present invention is significantly higher than that of the traditional small sample segmentation algorithm. The present invention extends the efficient deep mutual information maximization and self-encoder method to improve the practicality and robustness of the algorithm; the present invention cleverly mines the mutual information between the query image and the support image, making the method more targeted; Table 2 shows the comparison results of the algorithm in the coco dataset.

[0103] Table 2

[0104] Method CAnet PGnet PFEnet Ours Accuracy 52.0% 59.2% 63.3% 67.8%

[0105] It can be seen from Table 2 that the segmentation accuracy of the method of the present invention is also higher than that of the traditional method. The effectiveness of the present invention can be verified through the above simulation experiments.

Claims

1. A remote sensing image small sample segmentation method based on deep mutual information and variational encoder, characterized in that: The steps include: Step 1: Data division and preprocessing; The training images are divided into support set images s and query set images q according to the small sample learning problem setting, and then the input images are center cropped, resized, and normalized, and the processed images are used as training samples X of the feature learning algorithm; Step 2: Use the pre-trained ResNet101 network to extract features from the support set images and query set images respectively; First, the support set images and query set images are input into the pre-trained ResNet101 network. Through the multi-layer convolution and pooling operations of the network, the middle-level features of the images are extracted. and high-level features F s 、F q ; Step 3: Targeting the acquired high-level features F s 、F q , use the mutual information maximization method to calculate F s 、F q The mutual information between s ,F q ), the calculation method is as follows: Where: I(F s ; F q ): represents the high-level feature F s and F q The mutual information between x∈F s : Indicates that in the feature set F s A feature x in y∈F q : Indicates that in the feature set F q A feature y in p(x,y): represents the joint probability distribution of features x and y appearing simultaneously; p(x): represents the probability distribution of feature x; p(y): represents the probability distribution of feature y; A discriminator D is introduced to estimate and maximize the mutual information between the global features of the support set images and the query set images; Use InfoNCE loss to construct the loss function; Step 4: Output the middle-level features of step 2 Through a prior generator, which uses mid-level features and high-level features to generate a prior mask V of the query image q , the prior mask contains preliminary information about the target area in the query image; specifically, first calculate The cosine similarity of each pixel is calculated, and then the maximum similarity among all supporting pixels is taken as the response value, and finally normalized; in: Indicates support for image features Image features The cosine similarity between ; Supports mid-level features of images; Check the mid-level features of the image; T: represents the transpose operation; ||·||: represents the norm of a vector; Y Q : represents the prior mask of the query image; max is generated by taking the response value of the support pixel with the maximum similarity to the query pixel; Normalization: Prior mask Y Q Normalize it so that its value is between 0 and 1; min(Y Q ) and max(Y Q ) represent Y Q The minimum and maximum values ​​in ; ∈ is a constant used to avoid the situation where the denominator is zero; Step 5: Use the deep mutual information maximization module to extract the high-level features F of the output support image s Generate an initial prototype V of the support image through mask average pooling s ; Specifically, mask average pooling performs weighted averaging of high-level features according to the mask information in the support image, thereby obtaining an initial prototype that represents the overall features of the support image; M q : indicates the mask of the support image; A pool (·): indicates pooling operation; Represents the high-level features of the supporting image; V s : represents the feature vector after pooling; represents the normalized feature vector; Step 6: Targeting Initial Prototype V s Perform prototype decoupling operations; By changing V s Input the autoencoder structure, extract the implicit variable z of the prototype and decode it into the background prototype V through regularization constraints. b With the foreground prototype V f , only select the foreground prototype V f As output combined with F q Perform semantic segmentation operation: z=μ φ (x)+σ φ (x)⊙∈ z: represents hidden variables; μ φ (x): represents the mean function of input x; σ φ (x): represents the standard deviation function of input x; ∈: represents the noise sampled from the standard normal distribution; ⊙: represents element-by-element multiplication operation; represents normal distribution; q φ (z|x): represents the posterior distribution of the latent variable z given x; D(·): represents the discriminator; Indicates expected value; G(·): represents the generator.

2. The remote sensing image small sample segmentation method based on deep mutual information and variational encoder according to claim 1, characterized in that: The loss function constructed using InfoNCE loss is: N: represents the number of samples; K: represents the number of negative samples; Represents the feature vector of the positive sample that matches the i-th sample; Represents the feature vector of negative samples that do not match the i-th sample; z i : represents the feature vector of the i-th sample.

Citation Information

Cited By

  • Few-sample defect detection method based on prototype prompt fine-tuning visual basic model SAM

    CN119624927A

  • A few-shot defect detection method based on prototype-cued fine-tuning of visual anchor model (SAM)

    CN119624927B