A method, system, device and medium for cross-resolution semantic segmentation of SAR images based on contrastive self-supervised learning

Abstract

A method, system, device and medium for cross-resolution semantic segmentation of SAR images based on contrastive self-supervised learning; the method comprises: collecting SAR images of two resolutions in the same area to form a training set; cropping images of different resolutions according to relative positions so that each obtained sub-image corresponds to the same position in the same area, and using them as a group of contrast pairs; clustering the contrast pairs in the same group and their enhanced data to the same cluster center using a clustering method, and clustering different groups of contrast pairs to different cluster centers; freezing the feature extractor in the clustering stage for feature extraction of the segmentation task and training downstream classifiers; performing semantic segmentation using the trained model; merging the semantic segmentation results to obtain the final segmented image; the system, device and medium are used to implement the method; the present invention has the advantages of enhancing the model's sensitivity to changes at different scales, improving robustness and accuracy, and reducing dependence on a large amount of manually labeled data.

Description

Technical Field

[0001] The present invention belongs to the technical field of synthetic aperture radar (SAR) image interpretation, and specifically relates to a SAR image cross-resolution semantic segmentation method, system, device and medium based on comparative self-supervised learning. Background Art

[0002] Synthetic Aperture Radar (SAR) imagery object classification is a key technology in remote sensing, playing an important role in a variety of fields, including environmental monitoring, resource surveys, and urban planning. SAR imagery offers unique advantages, including all-day, all-weather observation capabilities and high-resolution imaging that is unrestricted by lighting and weather conditions. SAR imagery object classification can be categorized into traditional methods and deep learning approaches, depending on how features are acquired.

[0003] Traditional methods primarily include hand-crafted feature extraction and classification methods based on texture, statistical features, and classifiers. These methods are often based on expertise accumulated over years of research and have a strong theoretical foundation. These methods often perform well in specific scenarios, are simple and robust, and are highly capable of processing high-dimensional feature spaces. However, they require a high level of expertise and experience. Deep learning techniques, particularly convolutional neural networks (CNNs), have made significant progress in SAR object classification, automatically learning feature representations and achieving more accurate classification results while reducing reliance on specialized knowledge. However, this reliance is limited to feature selection, where specialized knowledge and experience are not required. Supervised deep learning algorithms still require large amounts of labeled data for training, resulting in high data acquisition costs. Furthermore, the generalization capabilities of some deep learning methods cannot be guaranteed in diverse environments.

[0004] Furthermore, in cross-resolution semantic segmentation of SAR imagery, single-resolution image data is often used as input for training models. However, single-resolution processing methods primarily rely on a uniform image resolution for feature extraction and analysis, which may not fully exploit the diverse features of SAR imagery at different resolutions. Due to the lack of capture of features at different scales, single-resolution methods can lead to poor model generalization, especially in complex scenes or with diverse object types, limiting classifier performance.

[0005] Patent application publication number CN118097148A discloses a method, system, device, and medium for weakly supervised water semantic segmentation in SAR images based on a color matching CAM model. The core of this method lies in adaptive weakly supervised learning using an adaptive resolution feature learning model, enabling the semantic segmentation model to learn features of objects in SAR images across resolutions even with only partial labels. However, this method still requires labeling all images at a given resolution, which is time-consuming and labor-intensive. Summary of the Invention

[0006] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a cross-resolution semantic segmentation method, system, device and medium for SAR imagery based on contrastive self-supervised learning. Through an innovative method based on multi-resolution analysis and contrastive learning, it is possible to simultaneously capture the local details and overall structure of the ground object, and perform effective feature learning with a small amount of labeled data. It has the advantages of enhancing the model's sensitivity to changes at different scales, improving robustness and accuracy, and reducing dependence on a large amount of manually labeled data.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is:

[0008] A cross-resolution semantic segmentation method for SAR images based on contrastive self-supervised learning includes the following steps:

[0009] Step 1: Acquire two SAR images with different resolutions, wherein the two resolutions include A resolution and B resolution;

[0010] Step 2: Crop the SAR images of each resolution in step 1 according to their relative sizes so that each image obtained after cropping at resolution A and each image obtained after cropping at resolution B correspond to the same location in the same area, and use them as a set of comparison pairs, which are used as positive samples. Each image obtained after cropping at resolution A and each image obtained after cropping at resolution B correspond to different locations in the same area, forming a comparison pair, which are used as negative samples. Then, only a part of the comparison pairs of the positive sample are annotated to obtain partially labeled data, and the rest are unlabeled data.

[0011] Step 3: Select a feature extractor and design a loss function. Use the features extracted by the feature extractor to cluster all the unlabeled data obtained in step 2. This includes clustering images of similar features at different resolutions in the unlabeled data to form clusters. By distinguishing the clusters of each type, we learn the essential representation of the features of each type of feature and dynamically update the feature extractor by minimizing the loss function.

[0012] Step 4: Freeze the feature extractor that has been dynamically updated in step 3, perform feature extraction on the partially labeled data obtained in step 2, and obtain high-dimensional features; set downstream tasks, input the high-dimensional features into the classifier for training, and obtain a segmentation model;

[0013] Step 5: Use the segmentation model obtained in step 4 to perform semantic segmentation and obtain the semantic segmentation result;

[0014] Step 6: Merge the semantic segmentation results of step 5 to obtain the final segmented image.

[0015] The specific method of step 2 is:

[0016] Step 2.1: Determine the cropping block size; select and analyze the purity of the dataset after cropping at different sizes, that is, the proportion of the main ground object categories in each image block, and select the cropping size with the highest purity as the final cropping size;

[0017] Step 2.2, data cropping: For the selected SAR image, crop it according to the cropping block size determined in step 2.1 to form image blocks; use an overlapping cropping strategy, moving a certain step size each time to ensure that there are overlapping areas between image blocks;

[0018] Step 2.3: Construct cross-resolution SAR image comparison pairs; group the two resolution images corresponding to the same feature into a comparison pair;

[0019] In step 2.4, the cross-resolution SAR image comparison pairs constructed in step 2.3 are partially annotated to obtain partially labeled data and unlabeled data.

[0020] The specific method of step 3 is:

[0021] Step 3.1, select feature extractor;

[0022] Step 3.2, Feature Extraction and Encoding: Use data augmentation techniques to process all unlabeled data obtained in step 2 to obtain multiple enhanced samples. These enhanced samples are input to the feature extractor in step 3.1 to extract low-level spatial features of the enhanced samples. The projection head module further extracts high-level semantic features. The projection head module consists of convolution kernels of different scales and pooling layers.

[0023] Step 3.3, construct and match prototype set: construct a prototype set module, which contains K prototype vectors, denoted as {c1, c2, c3, c4...c k}, as the cluster center of the clustering task, match the high-level semantic features extracted in step 3.2 with the prototype set, and determine a cluster code z for each feature vector. These codes will serve as representative representations of the features;

[0024] Step 3.4, construct loss function and exchange prediction: define the loss function of the exchange prediction problem, which contains two items: from feature z t Predictive coding q s and from feature z s Predictive coding q t , the loss function is:

[0025]

[0026] The above formula contains the characteristic z tPredictive coding q s And from the feature z s Predictive coding q t Each term represents the cross entropy loss between the encoding and the probability, where the probability is determined by the feature z i The dot product of all prototypes in the prototype set is obtained; It is used to measure the distance between feature z and encoding q. The specific form is as follows:

[0027]

[0028] in,

[0029]

[0030] Among them, τ represents the temperature parameter, c k represents the original cluster center, c k' represents the updated cluster center;

[0031] The above represents the case of a pair of contrast pairs in the training sample set; let τ represent the set of multi-view sample transformations, x nt Represents image sample x n The t-th viewing angle sample, the corresponding image feature vector and encoding are z nt and q nt , x ns Represents image sample x n For the s-th viewing angle sample, the corresponding image feature vector and encoding are z ns and q ns , Indicates that the image feature z nt Mapping to get c k The encoding of , T is the transposed symbol; applying the above formula to N images and paired data expansion, the overall loss function of the exchange prediction problem is as follows:

[0032]

[0033] Step 3.5, network training and optimization; train the feature extractor and clustering task network by minimizing the overall loss function obtained in step 3.4. The overall loss function includes reconstruction loss, classification loss, and domain adaptation loss; use the backpropagation algorithm to update the network parameters, including the parameters of the prototype set module and the feature extractor, to optimize the consistency of feature representation and cluster center. Dynamically update the prototype set module during training, use cross entropy loss to calculate the consistency between the feature vector and the cluster encoding, and obtain the probability distribution through the dot product of the feature vector and the prototype set.

[0034] The specific steps of freezing the feature extractor in the clustering stage in step 4 for feature extraction of the segmentation task and training the downstream classifier are as follows:

[0035] In step 4.1, after the partially labeled data obtained in step 2 passes through the feature extractor in step 3, the extracted features are passed through the adaptive average pooling layer, which resizes the input feature map to H×W. The mathematical expression of the adaptive average pooling layer is:

[0036]

[0037] Among them, P av (F in ) is the feature vector after pooling;

[0038] Step 4.2: The feature vector after pooling in step 4.1 then enters the fully connected layer. Each fully connected layer f c Expressed as a linear transformation followed by a nonlinear activation function, the fully connected operation of the lth layer is expressed as:

[0039] Z l =W l X l +b l

[0040] X l =Z l-1

[0041] Among them, W l is the weight matrix of layer l, b l is the bias vector, X l is the output of the l-1th layer. For the first layer, X l It is the P after pooling av (F in ), Z l is the linear output of the lth layer;

[0042] In step 4.3, the linear output of the fully connected layer in step 4.2 is converted into a nonlinear output using the ReLU activation function, which is defined as:

[0043] A l =ReLu(Z l )=max(0,Z l )

[0044] This means that if Z l The elements in A are less than 0, then l This element will be set to 0;

[0045] Step 4.4, the last fully connected layer Fc outputs the image sample X i The feature representation Zk , Z k The mathematical expression is:

[0046] Z k =W z Z k-1 +b z

[0047] Among them, k means that k fully connected layers are designed, Z k Represents the probability of mapping to each category;

[0048] Step 4.4, the output Z of the last fully connected layer obtained in step 4.3 k Solve and get the category number pred predicted by the SAR image block after passing through the feature extractor and classifier. The calculation formula is:

[0049] pred=argmax(Z k );

[0050] In step 4.5, the category number pred obtained in step 4.4 is compared with the labeled data obtained in step 2 to obtain the loss, and the parameters of the segmentation model are updated through backpropagation of the loss to obtain the final segmentation model.

[0051] The specific method of step 5 is:

[0052] The SAR image to be segmented is cropped in a non-overlapping manner, and the rows and columns of the cropped image are labeled. The cropping method is the same as step 2. For a single image, only the cropping method of the corresponding resolution needs to be used. There is only one type of ground object in the cropped image block. Then, the feature extractor trained in step 3 is used to extract features from the input SAR image block to be segmented to obtain a high-dimensional feature representation. Then, in the downstream task, the segmentation model obtained in step 4 is used to classify the high-dimensional features, and the image position where the ground object corresponding to the result is located is set to the color representing the ground object to obtain the semantic segmentation result.

[0053] The specific method of step 6 is:

[0054] According to the semantic segmentation results obtained in step 5, the segmented images are reassembled according to the row and column label information of the initial cropping to obtain the final semantic segmentation result image.

[0055] The present invention also provides a SAR image cross-resolution semantic segmentation system based on contrastive self-supervised learning, comprising:

[0056] A SAR image acquisition module is used to acquire SAR images with two different resolutions, wherein the two resolutions include A resolution and B resolution;

[0057] The SAR image processing module is used to crop the SAR images of each resolution according to their relative sizes, so that each image obtained after cropping at resolution A and each image obtained after cropping at resolution B correspond to the same location in the same area, and use them as a set of comparison pairs, which are used as positive samples. Each image obtained after cropping at resolution A and each image obtained after cropping at resolution B correspond to different locations in the same area, forming a comparison pair, which are used as negative samples. Then, only a part of the comparison pairs of the positive sample are annotated to obtain partially labeled data, and the rest are unlabeled data.

[0058] The feature extractor update module is used to select a feature extractor and design a loss function. It uses the features extracted by the feature extractor to cluster all unlabeled data, including clustering images of similar features at different resolutions in the unlabeled data to form clusters. By distinguishing between different clusters, it learns the essential representation of the features of each type of feature and dynamically updates the feature extractor by minimizing the loss function.

[0059] The segmentation model acquisition module is used to freeze the dynamically updated feature extractor, perform feature extraction on some labeled data, and obtain high-dimensional features; and set downstream tasks to input the high-dimensional features into the classifier for training to obtain the segmentation model;

[0060] The semantic segmentation result acquisition module is used to perform semantic segmentation using the segmentation model to obtain semantic segmentation results;

[0061] The final segmented image output module is used to merge the semantic segmentation results and obtain the final segmented image.

[0062] The present invention also provides a SAR image cross-resolution semantic segmentation device based on contrastive self-supervised learning, comprising:

[0063] Memory: a computer-readable device storing a computer program for the aforementioned method for cross-resolution semantic segmentation of SAR images based on contrastive self-supervised learning;

[0064] Processor: used to implement the method of cross-resolution semantic segmentation of SAR images based on contrastive self-supervised learning when executing the computer program.

[0065] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the cross-resolution semantic segmentation method of SAR images based on contrastive self-supervised learning.

[0066] Compared with the prior art, the present invention has the following beneficial effects:

[0067] First, the present invention is based on a cross-resolution learning framework. Cross-resolution learning allows the model to process images of different resolutions simultaneously, which helps the model learn more robust feature representations that are adaptable to SAR imagery under varying imaging conditions. Combining images at different resolutions provides richer contextual information, and fusing features from different sources provides a more comprehensive image representation, helping to improve the model's accuracy in SAR object classification.

[0068] Second, the present invention designs a self-supervised learning framework, which uses contrastive learning to learn to distinguish feature representations of different categories by comparing positive and negative sample pairs. This method enables the model to focus more on distinguishing different categories of objects, thereby improving the accuracy of classification. The proxy task under the contrastive learning framework encourages the network to learn to distinguish the feature representations of different images, while ensuring that different views of the same image are close to each other in the feature space, which helps the model capture more refined features. In addition, self-supervised learning reduces the dependence on large amounts of labeled data and learns useful feature representations by designing proxy tasks, which is very valuable for application scenarios such as SAR images where it is expensive to obtain labeled data. Features pre-trained through self-supervised learning can be transferred in downstream tasks, achieving high-performance classification or segmentation even with only a small amount of labeled data.

[0069] Third, the present invention designs multi-resolution samples and performs clustering tasks, i.e., multi-resolution analysis. This allows the model to extract features of objects at different scales, which helps capture both local details and the overall structure of the objects. Furthermore, clustering learning can help the model learn the inherent structure of the data. By clustering similar samples together, the model's consistent representation of object features can be enhanced. Through online clustering and exchange prediction mechanisms, the model can capture the essential features of SAR imagery without relying on explicit label information, which is particularly effective when labeled data is scarce.

[0070] In summary, the present invention, through an innovative method based on multi-resolution analysis and contrastive learning, can simultaneously capture the local details and overall structure of the ground object, and perform effective feature learning with a small amount of labeled data. It has the advantages of enhancing the model's sensitivity to changes at different scales, improving robustness and accuracy, and reducing dependence on large amounts of manually labeled data. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 This is a flow chart of an implementation method of an embodiment of the present invention.

[0072] Figure 2 This is the original 1m resolution SAR image of Hanzhong area in Shaanxi Province.

[0073] Figure 3This is the original 3m resolution SAR image of Hanzhong area in Shaanxi Province.

[0074] Figure 4 This is a result diagram of segmenting a 1m resolution SAR image of Hanzhong area in Shaanxi Province using the method of the present invention.

[0075] Figure 5 This is a result diagram of segmenting a 3m resolution SAR image of Hanzhong area in Shaanxi Province using the method of the present invention.

[0076] Figure 6 This is a comparison chart of local results of segmenting SAR images in Hanzhong area of ​​Shaanxi Province using the existing supervision method and the method of the present invention. DETAILED DESCRIPTION

[0077] The embodiments and effects of the present invention are further described below with reference to the accompanying drawings.

[0078] The present invention proposes a method based on cross-resolution contrastive learning. Through multi-resolution analysis, the model can extract features of land objects at different scales, which helps to capture the local details and overall structure of the land objects. Multi-resolution processing not only enriches the hierarchy of features, but also enables the model to focus on the microscopic features and macroscopic patterns of land objects at the same time. This method increases the sensitivity of the model to changes at different scales, thereby improving the robustness, accuracy and generalization ability of the model. By introducing contrastive learning, the model can perform effective feature learning with a small amount of labeled data. The core of contrastive learning is to learn feature representations that distinguish different categories by comparing positive and negative sample pairs. This method reduces the reliance on a large amount of manually labeled data, thereby solving the problem of labor costs, and provides a solution for improving the generalization ability of the model by learning more robust feature representations.

[0079] Reference Figure 1 A cross-resolution semantic segmentation method for SAR images based on contrastive self-supervised learning can be used to accurately extract multi-scale ground features from cross-resolution SAR images, including the following steps:

[0080] Step 1: Acquire two SAR images with different resolutions; the SAR images include 1m resolution and 3m resolution;

[0081] Step 2: Crop the SAR images of each resolution in step 1 according to their relative sizes so that each image obtained after cropping at resolution A and each image obtained after cropping at resolution B correspond to the same location in the same area, and use them as a set of comparison pairs, which are used as positive samples. Each image obtained after cropping at resolution A and each image obtained after cropping at resolution B correspond to different locations in the same area, forming a comparison pair, which are used as negative samples. Then, only a part of the comparison pairs of the positive sample are annotated to obtain partially labeled data, and the rest are unlabeled data. Specifically:

[0082] Step 2.1, determine the cropping block size; investigate commonly used data sets and summarize their data volume and size to determine the appropriate cropping size. Select different cropping sizes (such as 32×32, 64×64, etc.) as alternatives. Analyze the purity of the dataset after cropping at different sizes, that is, the proportion of the main ground object categories in each image block. After verification, the present invention chooses to use an image block of 126*126 pixels on a 1m resolution image and an image block of 42*42 pixels on a 3m resolution image;

[0083] Step 2.2, data cropping; for the selected SAR image, crop it according to the cropping size determined in step 2.1 to form image blocks; adopt an overlapping cropping strategy, move a certain step size each time, the step size needs to be manually set, and the default is 1 / 2 of the image length to ensure that there is an overlapping area between the image blocks; since the length and width of the 1m resolution image are three times that of the 3m resolution image, in order to crop the image blocks of the two resolutions to the same area, the step size when cropping the 1m image needs to be set to three times that of the 3m image. After verification, the present invention chooses to use a step size of 42 pixels for the 1m resolution image and a step size of 14 pixels for the 3m resolution image;

[0084] Step 2.3: Construct cross-resolution SAR image comparison pairs: Starting from different resolutions, group the two resolution images corresponding to the same feature into a comparison pair. Place the cropped images at resolutions A and B in two separate folders. Name them A_1.jpg, A_2.jpg, B_1.jpg, B_2.jpg, etc. in the order of cropping and resolution. At this point, A_1.jpg and B_1.jpg, and A_2.jpg and B_2.jpg, form a comparison pair of two images each. This allows for more accurate semantic information.

[0085] In step 2.4, the cross-resolution SAR image comparison pairs constructed in step 2.3 are partially annotated to obtain partially labeled data and unlabeled data.

[0086] Step 3: Select a feature extractor and design a loss function. Use the features extracted by the feature extractor to cluster all the unlabeled data obtained in step 2. This includes clustering images of similar features at different resolutions to form clusters. By distinguishing between clusters, the essential representations of each feature type are learned, and the feature extractor is dynamically updated by minimizing the loss function. This framework helps learn useful semantic information about features from SAR images in the absence of explicit labels.

[0087] The specific method of step 3 is:

[0088] Step 3.1, select a feature extractor, such as ResNet50;

[0089] The feature extractor is selected from currently popular feature extractors, such as Resnet, VGG, and GoogleNet series networks.

[0090] Step 3.2, Feature Extraction and Encoding: All unlabeled data obtained in step 2 are processed using data augmentation techniques to obtain multiple augmented samples. These augmented samples are input into the feature extractor in step 3.1 to extract low-level spatial features of the augmented samples. High-level semantic features are further extracted using a projection head module, which consists of convolution kernels of different scales and pooling layers.

[0091] Step 3.2, construct and match prototype set: Construct a prototype set module, which contains K prototype vectors, denoted as {c1, c2, c3, c4...c k}, as the cluster center of the clustering task. Match the high-level semantic features extracted in step 3.2 with the prototype set, and determine a cluster code z for each feature vector. These codes will serve as representative representations of the features;

[0092] Step 3.4, construct loss function and exchange prediction: define the loss function of the exchange prediction problem, which contains two items: from feature z t Predictive coding q s and from feature z s Predictive coding q t , the loss function is:

[0093]

[0094] The above formula contains the characteristic z t Predictive coding q s And from the feature z s Predictive coding q t Each term represents the cross entropy loss between the encoding and the probability, where the probability is determined by the feature z i The dot product of all prototypes in the prototype set is obtained; It is used to measure the distance between feature z and encoding q. The specific form is as follows:

[0095]

[0096] in,

[0097]

[0098] Among them, τ represents the temperature parameter, c k represents the original cluster center, c k' represents the updated cluster center;

[0099] The above represents the case of a pair of contrast pairs in the training sample set; let τ represent the set of multi-view sample transformations, x nt Represents image sample x n The t-th viewing angle sample, the corresponding image feature vector and encoding are z nt and q nt , x ns Represents image sample x n For the s-th viewing angle sample, the corresponding image feature vector and encoding are z ns and q ns , Indicates that the image feature z nt Mapping to get c k The encoding of , T is the transposed symbol; applying the above formula to N images and paired data expansion, the overall loss function of the exchange prediction problem is as follows:

[0100]

[0101] Step 3.5, network training and optimization; train the feature extractor and clustering task network by minimizing the overall loss function obtained in step 3.4. The overall loss function includes reconstruction loss, classification loss and domain adaptation loss; use the backpropagation algorithm to update the network parameters, including the parameters of the prototype set module and the feature extractor, to optimize the consistency of feature representation and cluster center; dynamically update the prototype set module during training to more accurately capture the data distribution and enhance the model's ability to extract and distinguish SAR image features; use cross entropy loss to calculate the consistency between the feature vector and the cluster code, and obtain the probability distribution through the dot product of the feature vector and the prototype set.

[0102] Step 4: Freeze the feature extractor that has been dynamically updated in step 3, perform feature extraction on the partially labeled data obtained in step 2, and obtain high-dimensional features; set downstream tasks, input the high-dimensional features into the classifier for training, and obtain a segmentation model;

[0103] The specific steps of freezing the feature extractor in the clustering stage in step 4 for feature extraction of the segmentation task and training the downstream classifier are as follows:

[0104] In step 4.1, after the partially labeled data obtained in step 2 passes through the feature extractor in step 3, the extracted features are passed through the adaptive average pooling layer. This layer resizes the input feature map to H×W to facilitate processing by the subsequent fully connected layer. The mathematical expression of the adaptive average pooling layer is:

[0105]

[0106] Among them, P av (F in ) is the feature vector after pooling.

[0107] Step 4.2: The feature vector after pooling in step 4.1 then enters the fully connected layer. Each fully connected layer f c Expressed as a linear transformation followed by a nonlinear activation function, the fully connected operation of the lth layer is expressed as:

[0108] Z l =W l X l +b l

[0109] X l =Z l-1

[0110] Among them, W l is the weight matrix of layer l, b l is the bias vector, X l is the output of the l-1th layer. For the first layer, X l It is the P after pooling av (F in ), Z l is the linear output of the lth layer;

[0111] In step 4.3, the linear output of the fully connected layer in step 4.2 is converted into a nonlinear output using the ReLU activation function, which is defined as:

[0112] A l =ReLU(Z l )=max(0,Z l )

[0113] This means that if Z l The elements in A are less than 0, then l This element will be set to 0;

[0114] Step 4.4, the last fully connected layer Fc outputs the image sample Xi The feature representation Z k , Z k The mathematical expression is:

[0115] Z k =W z Z k-1 +b z

[0116] Among them, k means that k fully connected layers are designed, Z k Represents the probability of mapping to each category;

[0117] Step 4.4, the output Z of the last fully connected layer obtained in step 4.3 k Solve and obtain the predicted category number pred of the SAR image block after passing through the feature extractor and classifier. The calculation formula is:

[0118] pred=argmax(Z k );

[0119] In step 4.5, the category number pred obtained in step 4.4 is compared with the labeled data obtained in step 2 to obtain the loss. The parameters of the segmentation model are updated through backpropagation of the loss to obtain the final segmentation model.

[0120] Step 5: Use the segmentation model obtained in step 4 to perform semantic segmentation and obtain the semantic segmentation result;

[0121] The specific method of step 5 is:

[0122] The SAR image to be segmented is cropped in a non-overlapping manner. This process requires labeling the rows and columns of the cropped image. The cropping method is the same as in step 2. For a single image, only the cropping method corresponding to the resolution is used. No comparison pairs are required at this stage. The prior assumption for this stage is that there is only one type of ground object within the cropped image block. The feature extractor trained in step 3 is then used to extract features from the input SAR image block to be segmented, obtaining a high-dimensional feature representation. The segmentation model obtained in step 4 is then used to classify the high-dimensional features in the downstream task. The image location corresponding to the ground object is set to the color representing the ground object, resulting in the semantic segmentation result.

[0123] Step 6: Merge the semantic segmentation results of step 5 to obtain the final segmented image.

[0124] Merge the semantic segmentation results and obtain the final segmented image. Reassemble the images that have been segmented in step 5 according to the row and column label information of the initial cropping to obtain the final semantic segmentation result image.

[0125] The effects of the present invention will be further described below in conjunction with simulation experiments.

[0126] 1. Simulation experiment conditions:

[0127] The hardware test platform for the simulation experiment of the present invention is: CPU 48 cores 192G; graphics card is Ascend 910PremiumA 32G

[0128] The software platform for the simulation experiment of the present invention is: operating system: EulerOS2.0 (SP8); online platform: ModelArts; notebook integrated development environment, Python 3.7.6; hardware platform: Ascend 910

[0129] Experimental method: There is an existing supervised CNN method and the present invention's cross-resolution semantic segmentation method for SAR images based on contrastive self-supervised learning.

[0130] 2. Simulation content and result analysis:

[0131] Figure 2 This is the original 1m resolution SAR image of Hanzhong area in Shaanxi Province;

[0132] Figure 3 This is the original 3m resolution SAR image of Hanzhong area in Shaanxi Province;

[0133] Figure 4 This is the result of segmenting a 1m resolution SAR image of Hanzhong, Shaanxi Province using the method of the present invention;

[0134] Figure 5 This is the result of segmenting a 3m resolution SAR image of Hanzhong, Shaanxi Province using the method of the present invention;

[0135] Figure 6 This is a comparison chart of local results of segmenting SAR images in Hanzhong area of ​​Shaanxi Province using the existing supervised method and the method of the present invention.

[0136] The simulation experiment of the present invention is to mark and train on 50% of the data in the constructed comparison pair, and test on the remaining 50% of the data. The detailed parameters of the data used in the present invention are shown in Table 1.

[0137] Table 1 Detailed parameters of the data used in the present invention

[0138] area size Resolution Polarization Stripe Mode Hanzhong area 10011*7710 1m HH UFS Hanzhong area 3337*2570 3m HH UFS

[0139] Table 2 shows the comparison of quantitative evaluation results of the simulation experiment of the present invention for 1m and 3m resolution data in Hanzhong area of ​​Shaanxi Province.

[0140] Table 2 Quantitative evaluation results of the segmentation results of the present invention on two resolution data in Hanzhong, Shaanxi Province

[0141] Model PA MPA MOIU FMIOU Kappa Unet 0.593 0.248 0.162 0.443 0.283 The present invention 0.795 0.674 0.540 0.670 0.631

[0142] According to Table 2, the present invention uses only half of the labeled data in the data of Hanzhong area of ​​Shaanxi Province with a resolution of 1m and 3m in the same area photographed by the GF3 satellite. The average intersection-over-union ratio (MIOU) of ground object segmentation is improved by 3.78% compared with traditional segmentation network technologies such as Unet, and the pixel accuracy (PA) is improved by 2.02%. The visual effect of segmentation is significantly improved, and it is significantly better than the supervised model in all segmentation performance indicators, which shows the effectiveness of this method in ground object classification tasks. In addition, from a qualitative perspective, Figure 6 This shows that the village identification by the method of the present invention is more precise and accurate than the segmentation by the traditional supervised method such as Unet.

[0143] The present invention also provides a SAR image cross-resolution semantic segmentation system based on contrastive self-supervised learning, comprising:

[0144] A SAR image acquisition module, configured to acquire two SAR images with different resolutions in step 1, wherein the two resolutions include A resolution and B resolution;

[0145] The SAR image processing module is used to implement step 2 by cropping the SAR images of each resolution in step 1 according to relative sizes, so that each image obtained after cropping with resolution A and each image obtained after cropping with resolution B correspond to the same location in the same area, and use them as a set of comparison pairs, which are used as positive samples. Each image obtained after cropping with resolution A and each image obtained after cropping with resolution B correspond to different locations in the same area, forming a comparison pair, which are used as negative samples. Then, only a part of the comparison pairs of the positive sample are annotated to obtain partially labeled data, and the rest are unlabeled data.

[0146] The feature extractor update module is used to implement the feature extractor selection in step 3 and design the loss function. The feature extractor is used to cluster all the unlabeled data obtained in step 2, including clustering images of different resolutions of similar objects in the unlabeled data to form clusters. At the same time, by distinguishing the clusters of each type, the essential representation of the characteristics of each type of object is learned, and the feature extractor is dynamically updated by minimizing the loss function.

[0147] The segmentation model acquisition module is used to freeze the feature extractor dynamically updated in step 3 in step 4, extract features from the partially labeled data obtained in step 2 to obtain high-dimensional features, and set downstream tasks to input the high-dimensional features into the classifier for training to obtain the segmentation model;

[0148] The semantic segmentation result acquisition module is used to implement semantic segmentation in step 5 using the segmentation model obtained in step 4 to obtain the semantic segmentation result;

[0149] The final segmented image output module is used to merge the semantic segmentation results of step 5 in step 6 and obtain the final segmented image.

[0150] The present invention also provides a SAR image cross-resolution semantic segmentation device based on contrastive self-supervised learning, comprising:

[0151] Memory: a computer-readable device storing a computer program for the aforementioned method for cross-resolution semantic segmentation of SAR images based on contrastive self-supervised learning;

[0152] Processor: used to implement the method of cross-resolution semantic segmentation of SAR images based on contrastive self-supervised learning when executing the computer program.

[0153] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the cross-resolution semantic segmentation method of SAR images based on contrastive self-supervised learning.

Claims

1. A cross-resolution semantic segmentation method for SAR images based on contrastive self-supervised learning, characterized by: The following steps are involved: Step 1: Acquire two SAR images with different resolutions, wherein the two resolutions include A resolution and B resolution; Step 2: Crop the SAR images of each resolution in step 1 according to their relative sizes so that each image obtained after cropping at resolution A and each image obtained after cropping at resolution B correspond to the same location in the same area, and use them as a set of comparison pairs, which are used as positive samples. Each image obtained after cropping at resolution A and each image obtained after cropping at resolution B correspond to different locations in the same area, forming a comparison pair, which are used as negative samples. Then, only a part of the comparison pairs of the positive sample are annotated to obtain partially labeled data, and the rest are unlabeled data. Step 3: Select a feature extractor and design a loss function. Use the features extracted by the feature extractor to cluster all the unlabeled data obtained in step 2. This includes clustering images of similar features at different resolutions in the unlabeled data to form clusters. By distinguishing the clusters of each type, we learn the essential representation of the features of each type of feature and dynamically update the feature extractor by minimizing the loss function. Step 4: Freeze the feature extractor that has been dynamically updated in step 3, perform feature extraction on the partially labeled data obtained in step 2, and obtain high-dimensional features; set downstream tasks, input the high-dimensional features into the classifier for training, and obtain a segmentation model; Step 5: Perform semantic segmentation using the segmentation model obtained in step 4 to obtain the semantic segmentation result; Step 6: Merge the semantic segmentation results of step 5 to obtain the final segmented image.

2. The method for cross-resolution semantic segmentation of SAR images based on contrastive self-supervised learning according to claim 1, characterized in that: The specific method of step 2 is: Step 2.1: Determine the cropping block size; select and analyze the purity of the dataset after cropping at different sizes, that is, the proportion of the main ground object categories in each image block, and select the cropping size with the highest purity as the final cropping size; Step 2.2, data cropping: For the selected SAR image, crop it according to the cropping block size determined in step 2.1 to form image blocks; use an overlapping cropping strategy, moving a certain step size each time to ensure that there are overlapping areas between image blocks; Step 2.3: Construct cross-resolution SAR image comparison pairs; group the two resolution images corresponding to the same feature into a comparison pair; In step 2.4, the cross-resolution SAR image comparison pairs constructed in step 2.3 are partially annotated to obtain partially labeled data and unlabeled data.

3. The method for cross-resolution semantic segmentation of SAR images based on contrastive self-supervised learning according to claim 1, characterized in that: The specific method of step 3 is: Step 3.1, select feature extractor; Step 3.2, Feature Extraction and Encoding: Use data augmentation techniques to process all unlabeled data obtained in step 2 to obtain multiple enhanced samples. These enhanced samples are input to the feature extractor in step 3.1 to extract low-level spatial features of the enhanced samples. The projection head module further extracts high-level semantic features. The projection head module consists of convolution kernels of different scales and pooling layers. Step 3.3, construct and match prototype set: construct a prototype set module, which contains K prototype vectors, denoted as {c1, c2, c3, c4...c k }, as the cluster center of the clustering task, match the high-level semantic features extracted in step 3.2 with the prototype set, and determine a cluster code z for each feature vector. These codes will serve as representative representations of the features; Step 3.4, construct loss function and exchange prediction: define the loss function of the exchange prediction problem, which contains two items: from feature z t Predictive coding q s and from feature z s Predictive coding q t , the loss function is: The above formula contains the characteristic z t Predictive coding q s And from the feature z s Predictive coding q t Each term represents the cross entropy loss between the encoding and the probability, where the probability is determined by the feature z i The dot product of all prototypes in the prototype set is obtained; It is used to measure the distance between feature z and encoding q. The specific form is as follows: in, Among them, τ represents the temperature parameter, c k represents the original cluster center, c k' represents the updated cluster center; The above represents the case of a pair of contrast pairs in the training sample set; let Ω represent the set of multi-view sample transformations, x nt Represents image sample x n The t-th viewing angle sample, the corresponding image feature vector and encoding are z nt and q nt , x ns Represents image sample x n For the s-th viewing angle sample, the corresponding image feature vector and encoding are z ns and q ns , Indicates that the image feature z ns Mapping to get c k The encoding of , T is the transposed symbol; applying the above formula to N images and paired data expansion, we get the following overall loss function for the exchange prediction problem: Step 3.5, network training and optimization; train the feature extractor and clustering task network by minimizing the overall loss function obtained in step 3.

4. The overall loss function includes reconstruction loss, classification loss, and domain adaptation loss; use the backpropagation algorithm to update the network parameters, including the parameters of the prototype set module and the feature extractor, to optimize the consistency of feature representation and cluster center. Dynamically update the prototype set module during training, use cross entropy loss to calculate the consistency between the feature vector and the cluster encoding, and obtain the probability distribution through the dot product of the feature vector and the prototype set.

4. The method for cross-resolution semantic segmentation of SAR images based on contrastive self-supervised learning according to claim 1, characterized in that: The specific steps of freezing the feature extractor in the clustering stage in step 4 for feature extraction of the segmentation task and training the downstream classifier are as follows: In step 4.1, after the partially labeled data obtained in step 2 passes through the feature extractor in step 3, the extracted features are passed through the adaptive average pooling layer, which resizes the input feature map to H×W. The mathematical expression of the adaptive average pooling layer is: Among them, P av (F in ) is the feature vector after pooling; Step 4.2: The feature vector after pooling in step 4.1 then enters the fully connected layer. Each fully connected layer f c Expressed as a linear transformation followed by a nonlinear activation function, the fully connected operation of the lth layer is expressed as: Z l =W l X l +b l X l =Z l-1 Among them, W l is the weight matrix of layer l, b l is the bias vector, X l is the output of the l-1th layer. For the first layer, X l It is the P after pooling av (F in ), Z l is the linear output of the lth layer; In step 4.3, the linear output of the fully connected layer in step 4.2 is converted into a nonlinear output using the ReLU activation function, which is defined as: AND l =ReLU(Z l )=max(0,Z l ) This means that if Z l The elements in A are less than 0, then l This element will be set to 0; Step 4.4, the last fully connected layer Fc outputs the image sample X i The feature representation Z k , Z k The mathematical expression is: WITH k =In z WITH k-1 +b z Among them, k means that k fully connected layers are designed, Z k Represents the probability of mapping to each category; Step 4.4, the output Z of the last fully connected layer obtained in step 4.3 k Solve and get the category number pred predicted by the SAR image block after passing through the feature extractor and classifier. The calculation formula is: before=argmax(Z k ); In step 4.5, the category number pred obtained in step 4.4 is compared with the labeled data obtained in step 2 to obtain the loss, and the parameters of the segmentation model are updated through backpropagation of the loss to obtain the final segmentation model.

5. The method for cross-resolution semantic segmentation of SAR images based on contrastive self-supervised learning according to claim 1, characterized in that: The specific method of step 5 is: The SAR image to be segmented is cropped in a non-overlapping manner, and the rows and columns of the cropped image are labeled. The cropping method is the same as step 2. For a single image, only the cropping method of the corresponding resolution needs to be used. There is only one type of ground object in the cropped image block. Then, the feature extractor trained in step 3 is used to extract features from the input SAR image block to be segmented to obtain a high-dimensional feature representation. Then, in the downstream task, the segmentation model obtained in step 4 is used to classify the high-dimensional features, and the image position where the ground object corresponding to the result is located is set to the color representing the ground object to obtain the semantic segmentation result.

6. The method for cross-resolution semantic segmentation of SAR images based on contrastive self-supervised learning according to claim 1, characterized in that: The specific method of step 6 is: According to the semantic segmentation results obtained in step 5, the segmented images are reassembled according to the row and column label information of the initial cropping to obtain the final semantic segmentation result image.

7. A SAR image cross-resolution semantic segmentation system based on contrastive self-supervised learning based on the method according to any one of claims 1 to 6, characterized in that: include: A SAR image acquisition module is used to acquire SAR images with two different resolutions, wherein the two resolutions include A resolution and B resolution; The SAR image processing module is used to crop the SAR images of each resolution according to their relative sizes, so that each image obtained after cropping at resolution A and each image obtained after cropping at resolution B correspond to the same location in the same area, and use them as a set of comparison pairs, which are used as positive samples. Each image obtained after cropping at resolution A and each image obtained after cropping at resolution B correspond to different locations in the same area, forming a comparison pair, which are used as negative samples. Then, only a part of the comparison pairs of the positive sample are annotated to obtain partially labeled data, and the rest are unlabeled data. The feature extractor update module is used to select a feature extractor and design a loss function. It uses the features extracted by the feature extractor to cluster all unlabeled data, including clustering images of similar features at different resolutions in the unlabeled data to form clusters. By distinguishing between different clusters, it learns the essential representation of the features of each type of feature and dynamically updates the feature extractor by minimizing the loss function. The segmentation model acquisition module is used to freeze the dynamically updated feature extractor, perform feature extraction on some labeled data, and obtain high-dimensional features; and set downstream tasks to input the high-dimensional features into the classifier for training to obtain the segmentation model; The semantic segmentation result acquisition module is used to perform semantic segmentation using the segmentation model to obtain semantic segmentation results; The final segmented image output module is used to merge the semantic segmentation results and obtain the final segmented image.

8. A device for cross-resolution semantic segmentation of SAR images based on contrastive self-supervised learning, characterized by: include: Memory: a computer-readable device storing a computer program for a method for cross-resolution semantic segmentation of SAR images based on contrastive self-supervised learning according to any one of claims 1 to 6; Processor: configured to implement the cross-resolution semantic segmentation method for SAR images based on contrastive self-supervised learning as described in any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, can implement the cross-resolution semantic segmentation method for SAR images based on contrastive self-supervised learning as described in any one of claims 1 to 6.

Images