A SAR image pre-training large model ground object extraction method, system, device and medium based on global-local semantic alignment

Abstract

A SAR image pre-training large model ground object extraction method, system, device and medium based on global-local semantic alignment, the method: collect massive SAR image data and pretreatment, make data set, divide the data set into pre-training data set and downstream fine-tuning data set, data enhancement is carried out to the pre-training data set, the two views of the same image and the corresponding local frame between the views are generated; Construct a global-local semantic alignment pre-training network, use the two views of the same image and the corresponding local frame between the views, network training is carried out, and pre-training weight is obtained; Load the pre-training weight to the backbone network in the downstream ground object extraction network, fine-tune using the downstream fine-tuning data set, and obtain the ground object extraction model; The ground object extraction model is tested on the test set of downstream fine-tuning data, and the performance of the obtained prediction result graph is evaluated; The system, device and medium are used to realize the method; The present application significantly improves the accuracy and reliability of SAR ground object extraction task.

Description

Technical Field

[0001] This invention belongs to the field of SAR image interpretation technology, specifically relating to a method, system, device, and medium for extracting ground features from a large SAR image pre-training model based on global-local semantic alignment. Background Technology

[0002] Remote sensing image feature extraction has immense research value and significance in both military and civilian fields. With the continuous advancement of remote sensing imaging technology, imaging methods are becoming increasingly diversified. Synthetic Aperture Radar (SAR) is a crucial technology in the field of remote sensing. It is a microwave imaging radar that employs active imaging, unaffected by weather, lighting, or cloud cover, overcoming the limitations of optical remote sensing due to external conditions such as weather. Therefore, SAR plays an irreplaceable role in remote sensing. Consequently, SAR is an indispensable remote sensing tool for Earth observation.

[0003] SAR image land cover extraction is a popular topic in remote sensing. Its objective is to predict the corresponding category of each pixel in an image to obtain a map of land cover categories for the entire image. SAR image land cover extraction has wide applications in global land cover, urban planning, and disaster prevention. For example, it can be used for dynamic monitoring and assessment of land resources such as farmland and forests, enabling precise agricultural management and ecological protection; it can be used for spatial analysis and optimization of urban construction and land use planning, improving the scientific rigor and effectiveness of urban planning and management; and it can be used for risk assessment and response to natural disasters and climate change, providing support for disaster reduction and prevention.

[0004] In recent years, with the development of deep learning technology, neural network-based classification algorithms have demonstrated excellent performance in SAR image feature extraction tasks. However, these neural network-based feature extraction algorithms are usually supervised, requiring a large amount of labeled data for supervised training to correctly adjust network parameters and obtain good classification results. However, obtaining pixel-level labels for SAR images not only requires specialized knowledge but is also very time-consuming, consuming significant human and material resources. With the rapid development of remote sensing technology, a large amount of remote sensing data is generated every moment. How to fully utilize this unlabeled data to provide useful information for downstream SAR image feature extraction tasks, enabling the training of effective feature extraction models even with insufficient labeled data, is an urgent problem to be solved.

[0005] Patent application CN116977744A discloses a small-sample cross-domain hyperspectral image classification method based on contrastive learning and subspace. This method uses a prototype network with subspace improvement to classify the embedded features of the source and target domains, improving the utilization rate of embedded feature information. It also constructs a holistic model connecting the source and target domains through a domain discriminator, achieving domain alignment while reducing training costs. Furthermore, contrastive learning is introduced to analyze the correlation and differences between source domain samples, and combined with subspace and domain discriminator, the classification accuracy of small cross-domain samples is further improved. However, this method has a drawback: it is only applicable to hyperspectral images. Since SAR image features are significantly different from hyperspectral images, this method lacks robustness when dealing with SAR images, resulting in poor classification results.

[0006] Patent application CN114724030A discloses a polarimetric SAR land cover classification method based on contrastive learning. This method designs a feasible data augmentation approach based on the unique scattering characteristics of polarimetric SAR data. It pre-trains unlabeled data using an auxiliary task to obtain a relatively good pre-trained model, and then fine-tunes the model using a small amount of labeled data to better apply it to land cover classification tasks in polarimetric SAR images. However, this method has shortcomings: its data augmentation method is only applicable to polarimetric SAR images and cannot be generalized to other SAR images; secondly, it does not consider the characteristics of the land cover extraction task, only using contrastive learning to learn instance-level features while ignoring local semantic information, leading to a loss in land cover extraction performance; and thirdly, the method has insufficient pre-training data, failing to fully mine the potential information in the data, resulting in insufficient generalization performance. Summary of the Invention

[0007] To overcome the shortcomings of the prior art, the present invention aims to provide a method, system, device and medium for pre-training large-scale SAR image features extraction based on global-local semantic alignment. Through the designed instance branches and local semantic branches, the deep network can be pre-trained by fully utilizing the constructed massive unlabeled SAR image dataset, learning potential global and local semantic information from the data, and improving the performance of SAR image features extraction tasks.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0009] A method for extracting ground features from a large SAR image pre-trained model based on global-local semantic alignment includes the following steps:

[0010] Step 1: Collect massive amounts of SAR image data, perform preprocessing, create a dataset, divide the dataset into a pre-training dataset and a downstream fine-tuning dataset, and further divide the downstream fine-tuning dataset into a training set, a validation set, and a test set.

[0011] Step 2: Perform data augmentation on the pre-trained dataset from Step 1 to generate two views of the same image and corresponding local bounding boxes between the views;

[0012] Step 3: Construct a global-local semantic alignment pre-trained network. Use the two views of the same image generated in Step 2 and the corresponding local bounding boxes between the views to train the network and obtain the pre-trained weights.

[0013] Step 4: Load the pre-trained weights from Step 3 into the backbone network of the downstream feature extraction network, and fine-tune the model using the training and validation sets of the downstream fine-tuning dataset from Step 1 to obtain the feature extraction model.

[0014] Step 5: Test the land feature extraction model obtained in Step 4 on the test set of the downstream fine-tuning data in Step 1 to obtain the prediction result map, and evaluate the performance of the prediction result map.

[0015] The specific process of step 1 is as follows:

[0016] Step 1.1: Collect massive amounts of SAR image data;

[0017] Step 1.2: Perform multi-view processing, orthorectification, and depth-level conversion on the SAR image data obtained in Step 1.1, and select some of the processed SAR images for artificial land cover category labeling.

[0018] Step 1.3: Divide the SAR image data processed in Step 1.2 into a pre-training dataset and a downstream fine-tuning dataset;

[0019] Step 1.4: Divide the downstream fine-tuning dataset obtained in Step 1.3 into a training set, a validation set, and a test set.

[0020] The specific process of step 2, which involves data augmentation to generate two views of the same image and corresponding local bounding boxes between the views, is as follows:

[0021] Step 2.1: Generate two views of the same image: Let I represent the image in the pre-training dataset. Data augmentation strategies include random cropping, scaling, brightness and contrast jitter, Gaussian blur and random horizontal flipping. Let rc(.) represent the random cropping operation and aug(.) represent the other data augmentation operations. First, perform two random cropping operations on the image in the pre-training dataset in Step 1.3 to obtain r1 = rc1(I) and r2 = rc2(I). Then, perform subsequent data augmentation operations on the cropped image to obtain v1 = aug1(r1) and v2 = aug2(r2), thus obtaining two views of the same image.

[0022] Step 2.2, Generate corresponding local bounding boxes between views: To align local semantic information, that is, when two local regions correspond to the same spatial region, minimize the feature representations of these two local regions to make them have the same semantic information, it is necessary to generate local bounding boxes with the same spatial region. The specific process is as follows:

[0023] Step 2.2.1: First, find the overlapping region IS(r1,r2) of two random crop views (r1,r2), where IS(r1,r2) represents the overlapping spatial region generated in random crop views r1 and r2;

[0024] Step 2.2.2: After finding the overlapping spatial region, randomly sample a box B = (x,y,w,h) in the overlapping region IS, where (x,y,w,h) is used to represent the coordinates of box B;

[0025] Step 2.2.3: Convert box B into corresponding coordinates in each view v1 and v2, which can be denoted as B. i =(x i ,y i ,w i ,h i ), i∈{1,2};

[0026] Step 2.2.4 generates more local bounding boxes and obtains multiple pairs of local representations; that is, it obtains the coordinate boxes. and Where k = {1, ..., K}, K is the total number of generated local boxes in the image. Calculate the IoU between the newly generated candidate box and the already generated boxes, that is, the ratio of the intersection and union. If it is greater than 50%, reject the candidate box. Repeat this process until the number of retained boxes reaches K.

[0027] Step 3 involves constructing a global-local semantic alignment pre-trained network and training the network as follows:

[0028] Step 3.1, Constructing the pre-trained network framework: The pre-trained network framework includes two neural networks, namely an online network defined by a set of parameters θ and a target network defined by parameters ξ; the target network provides the regression target for training the online network, and the parameters ξ of the target network follow the parameters θ of the online network through an exponential moving average with a decaying parameter τ, i.e., ξ←τξ+(1-τ)θ; the encoding network used is ResNet-101;

[0029] Step 3.2, Constructing Instance Branches: During training, corresponding data of the same sample are augmented into a pair of positive samples, while other samples are negative samples. Assuming the batch size is N, then each sample has at most 2(N-1) negative samples. For the samples in the queue, denoted as {k0,k1,…,k...} MThe queue stores M negative samples; each iteration updates N negative samples according to the "first-in, first-out" principle. The specific process for constructing the instance branch is as follows:

[0030] Step 3.2.1: Input the two views (v1, v2) obtained in Step 2.1 into the two encoding networks (f) constructed in Step 3.1. θ ,f ξ In the process, the feature map m1 = f is obtained. θ (v1) and m2 = f ξ (v2);

[0031] Step 3.2.2: Global average pooling (GAP) is performed on the feature maps m1 and m2 obtained in step 3.2.1 to obtain the instance-level features g1 and g2 of the image.

[0032] Step 3.2.3: Use a projector to map the extracted global features to a high-dimensional feature representation space. Use a multilayer perceptron (MLP) with one hidden layer as the projector, which can be represented as:

[0033] p1=p(g1=W (2) σ(W (1) g1),p2=p(g2)

[0034] Where p(.) represents the projection head, W represents the fully connected layer, and σ(.) represents the ReLU activation function;

[0035] Step 3.2.4, use the InforNCE loss as the loss function for the instance branch:

[0036]

[0037] Where τ>0 is the temperature hyperparameter, used to adjust the loss;

[0038] Step 3.3, Constructing Local Semantic Branches: The local bounding boxes matching the overlapping regions IS(r1,r2) between the two views obtained in Step 2.2.1 are used to construct local semantic branches to enhance the network's local feature learning ability. The specific process is as follows:

[0039] Step 3.3.1, obtain the corresponding coordinate frame in step 2.2.4. and Then, the corresponding region is cropped from the feature map and called the Region of Interest (RoI). The cropped feature map is then locally pooled using 1×1 RoIAlign. Where k = {1, ..., K}, and K is the total number of bounding boxes generated in the image;

[0040] Step 3.3.2, similar to the instance branch, after obtaining the pooling features, passes them through a projection head to obtain... and In the local semantic branch, a prediction header is attached after the online network for further mapping. get Used to predict the output of the target branch Align local semantic information to enhance the local feature representation of the network;

[0041] Step 3.3.3 defines the semantic consistency loss as the mean squared error between the normalized prediction and the normalized target projection, as shown below:

[0042]

[0043] in, Used to normalize feature vectors;

[0044] Step 3.4, perform network training: The overall loss function of the pre-trained network framework can be defined as:

[0045] L=L1+λL2

[0046] Where λ is a hyperparameter used to balance the loss of instance branches and local semantic branches;

[0047] The gradient propagates only in the online network. The parameters ξ of the target network follow the parameters θ of the online network through an exponential moving average with a decaying parameter τ, i.e., ξ←τξ+(1-τ)θ.

[0048] During pre-training, the loss changes are recorded. If the loss does not decrease for m consecutive epochs, training is stopped and the weights of the pre-trained network are saved.

[0049] The specific process of loading the pre-trained weights into the backbone network of the downstream land cover extraction network for fine-tuning is as follows:

[0050] Step 4.1: Use DeeplabV3+ as the downstream ground feature extraction network, and use cross-entropy loss to train the downstream ground feature extraction network, which is defined as follows:

[0051]

[0052] Where C represents the number of categories, y i p represents the true label set for the i-th class. i This represents the prediction result for the i-th class;

[0053] Step 4.2: Load the pre-trained weights obtained in step 3.4 into the backbone network of the downstream feature extraction network in step 4.1;

[0054] Step 4.3: Fine-tune the downstream feature extraction network from Step 4.2 using the training and validation sets from the downstream fine-tuning dataset in Step 1.4. The following process is adopted during the fine-tuning:

[0055] Step 4.3.1: Try a series of different learning rates to find the learning rate that best suits the model;

[0056] Step 4.3.2: After every l training cycles, evaluate the model's performance using the validation set in the downstream fine-tuning dataset from Step 1.4;

[0057] Step 4.3.3: If the results of 1 consecutive validations show no improvement in accuracy, then stop training at the current learning rate;

[0058] Step 4.3.4: Select the best-performing model from these learning rates and save it as the final ground feature extraction model.

[0059] The specific process of performance evaluation in step 5 is as follows:

[0060] The land cover extraction model obtained in step 4 is tested on the test set of the downstream fine-tuning data in step 1.4 to obtain the prediction result map, and the classification indices OA, Kappa, MIoU and FWIoU are calculated.

[0061] This invention also provides a SAR image pre-training large model ground feature extraction system based on global-local semantic alignment, comprising:

[0062] The massive SAR image data acquisition module is used to acquire massive SAR image data, perform preprocessing, create datasets, divide the datasets into pre-training datasets and downstream fine-tuning datasets, and further divide the downstream fine-tuning datasets into training sets, validation sets, and test sets.

[0063] The pre-trained dataset processing module is used to perform data augmentation on the pre-trained dataset, generating two views of the same image and corresponding local bounding boxes between the views;

[0064] The Global-Local Semantic Alignment Pre-trained Network Building Module is used to build a global-local semantic alignment pre-trained network. It uses two views of the same generated image and the corresponding local bounding boxes between the views to train the network and obtain the pre-trained weights.

[0065] The feature extraction model acquisition module is used to load pre-trained weights into the backbone network of the downstream feature extraction network, and to fine-tune the model using the training and validation sets of the downstream fine-tuning dataset in step 1 to obtain the feature extraction model.

[0066] The prediction result map performance evaluation module is used to test the ground feature extraction model on the test set of the downstream fine-tuning data in step 1 to obtain the prediction result map and evaluate its performance.

[0067] This invention also provides a SAR image pre-trained large model ground feature extraction device based on global-local semantic alignment, comprising:

[0068] Memory: A computer program that stores the above-mentioned method for extracting ground features from a large model of SAR image pre-training based on global-local semantic alignment, and is a computer-readable device;

[0069] Processor: Used to implement the method for extracting ground features from a large SAR image pre-training model based on global-local semantic alignment when executing the computer program.

[0070] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the aforementioned method for extracting ground features from a large SAR image pre-training model based on global-local semantic alignment.

[0071] Compared with the prior art, the present invention has the following advantages:

[0072] First, for SAR imagery, we collected massive amounts of data to create a pre-training dataset, enabling the pre-trained large model to fully explore the potential information in SAR imagery and improve the performance of downstream ground feature extraction tasks.

[0073] Second, considering the characteristics of SAR ground feature extraction tasks, a dual-branch structure was designed. The instance branch enhances feature discrimination ability by learning the distinctiveness of image instances, while the local semantic branch captures the local semantic consistency of remote sensing images. By combining the instance branch and the local semantic branch, global and local semantic alignment is achieved, and the synergy between the two helps the model understand the overall scene structure and fine-grained details.

[0074] In summary, this invention designs a dual-branch pre-training framework, including instance branches and local semantic branches, specifically for the characteristics of SAR ground feature extraction tasks. It also collects massive amounts of SAR data to create a pre-training dataset, enabling the pre-trained large model to fully mine and understand the global structure and local details of images from massive amounts of data. This improves the model's overall scene recognition ability and fine-grained ground feature discrimination ability, allowing the model to achieve excellent performance even when trained with only limited labeled data. As a result, the accuracy and reliability of SAR ground feature extraction tasks are significantly improved. Attached Figure Description

[0075] Figure 1 This is a flowchart of the implementation method of the present invention.

[0076] Figure 2 This is a schematic diagram of the data augmentation method used in this invention to generate two views and a partial bounding box.

[0077] Figure 3 This is a schematic diagram of the global-local semantic alignment pre-trained network framework of the present invention.

[0078] Figure 4 This is a schematic diagram of the downstream feature extraction network architecture of the present invention.

[0079] Figure 5 These are the ground feature extraction results images obtained through the pre-training method of this invention and through other pre-training methods. Detailed Implementation

[0080] The embodiments and effects of the present invention will be further described below with reference to the accompanying drawings.

[0081] like Figure 1 As shown, a method for extracting ground features from a large SAR image pre-trained model based on global-local semantic alignment includes the following steps:

[0082] Step 1: Collect a large amount of 3m resolution SAR images, preprocess them, and create a dataset. Divide the dataset into a pre-training dataset and a downstream fine-tuning dataset, and further divide the downstream fine-tuning dataset into a training set, a validation set, and a test set.

[0083] The specific process of step 1 is as follows:

[0084] Step 1.1: Acquire massive amounts of SAR images with a resolution of 3m, obtained by the Gaofen-3 remote sensing satellite, with HH polarization and UFS striping mode;

[0085] Step 1.2: The SAR image data obtained in Step 1.1 is preprocessed using professional remote sensing processing software, including multi-view processing, orthorectification and depth conversion, and some depth-converted SAR images are selected for artificial land cover category labeling.

[0086] Step 1.3: Divide the SAR images processed in Step 1.2 into a pre-training dataset and a downstream fine-tuning dataset, and crop all data to a size of 512×512 for easier subsequent training and testing. The pre-training dataset contains 339,489 unlabeled images, and the downstream fine-tuning dataset contains 33,111 labeled images.

[0087] Step 1.4: Divide the downstream fine-tuning dataset obtained in Step 1.3 into a training set, a validation set, and a test set in a ratio of 8:1:1.

[0088] Step 2: Perform data augmentation on the pre-trained dataset from Step 1 to generate two views of the same image and corresponding local bounding boxes between the views;

[0089] The specific process of step 2, which performs data augmentation to generate two views of the same image and corresponding local bounding boxes between the views, is as follows:

[0090] Step 2.1, generate two views of the same image: as follows Figure 2 As shown, I represents the image in the pre-training dataset. The data augmentation strategies include random cropping, scaling, brightness and contrast jitter, Gaussian blur and random horizontal flipping. rc(.) represents the random cropping operation, and aug(.) represents the remaining data augmentation operations. First, the image in the pre-training dataset in step 1.3 is randomly cropped twice to obtain r1 = rc1(I) and r2 = rc2(I). Then, the cropped image is subjected to subsequent data augmentation operations to obtain v1 = aug1(r1) and v2 = aug2(r2), which are two views of the same image.

[0091] Step 2.2, generating corresponding local bounding boxes between views: To align local semantic information, this invention constructs a local semantic branch, that is, when two local regions correspond to the same spatial region, the feature representations of these two local regions are minimized so that they have the same semantic information. For this purpose, local bounding boxes with the same spatial region need to be generated. The specific process is as follows:

[0092] Step 2.2.1, as follows Figure 2 As shown, firstly, the overlapping region IS(r1,r2) of two random cropped views (r1,r2) is found, where IS(r1,r2) represents the overlapping spatial region generated in random cropped views r1 and r2.

[0093] Step 2.2.2: After finding the overlapping spatial region, randomly sample a box B = (x,y,w,h) in the overlapping region IS, where (x,y,w,h) is used to represent the coordinates of box B; since ResNet101 downsamples by 32 times, in this invention, the minimum length and width of this box is 32, and this box cannot exceed the size of the overlapping region.

[0094] Step 2.2.3: After this, convert box B to its corresponding coordinates in each view v1 and v2, which can be denoted as B. i =(x i ,y i ,w i ,h i ), i∈{1,2};

[0095] Step 2.2.4 can generate more local bounding boxes to simultaneously obtain multiple pairs of local representations. This results in the coordinate boxes. and Where k = {1, ..., K}, and K is the total number of generated local boxes in the image, which is set to 10 in this invention. Furthermore, the diversity of generated boxes can be improved by considering overlapping regions between different boxes. Specifically, the IoU (Intersection over Union) between newly generated candidate boxes and already generated boxes is calculated; if it is greater than 50%, the candidate box is rejected; this process is repeated until the number of retained boxes reaches K.

[0096] Step 3: Construct a global-local semantic alignment pre-trained network. Use the two views of the same image generated in Step 2 and the corresponding local bounding boxes between the views to train the network and obtain the pre-trained weights.

[0097] Step 3 involves constructing a global-local semantic alignment pre-trained network and training the network as follows:

[0098] Step 3.1, construct the pre-trained network framework: such as Figure 3 As shown, this invention uses two neural networks: an online network defined by a set of parameters θ and a target network defined by parameters ξ. The target network provides the regression target for training the online network, and the parameters ξ of the target network follow the parameters θ of the online network through an exponential moving average with a decay parameter τ, i.e., ξ←τξ+(1-τ)θ. In this invention, the encoding network used is ResNet-101, which performs excellently in extracting image features due to its depth and structural design. It can effectively capture multi-scale and multi-level features in SAR images, improving the accuracy and reliability of ground feature extraction.

[0099] Step 3.2 Constructing Instance Branches: Instance branches utilize the idea of ​​contrastive learning, which involves bringing the feature representations between positive sample pairs closer together and distancing the feature representations between negative sample pairs to align the global information between positive samples. During training, corresponding data of the same sample are augmented into a pair of positive samples, while other samples become negative samples. Assuming the batch size is N, each sample has at most 2(N-1) negative samples. To obtain more negative samples, this invention maintains a queue to store negative samples, thereby increasing the number of negative samples. The samples in the queue are denoted as {k0, k1, ..., k M The queue stores M negative samples. Each iteration updates N negative samples according to the "first-in, first-out" principle. In this invention, we set M = 65536 and N = 256. The specific process for constructing the instance branch is as follows:

[0100] Step 3.2.1, as follows Figure 3 As shown, the two views (v1, v2) obtained in step 2.1 are each input into the two encoding networks (f) constructed in step 3.1. θ ,f ξ In the process, the feature map m1 = f is obtained.θ (v1) and m2 = f ξ (v2);

[0101] Step 3.2.2: Global average pooling (GAP) is performed on the feature maps m1 and m2 obtained in step 3.2.1 to obtain the instance-level features g1 and g2 of the image.

[0102] Step 3.2.3 uses a projection head to map the extracted global features to a high-dimensional feature representation space. This mapping helps to better separate and distinguish features. Through feature projection, instance branching reduces the potential impact of data augmentation on downstream tasks, making the learned feature representation more robust. Using a multilayer perceptron (MLP) with one hidden layer as the projection head, it can be represented as:

[0103] p1=p(g1=W (2) σ(W (1) g1),p2=p(g2)

[0104] Where p(.) represents the projection head, W represents the fully connected layer, and σ(.) represents the ReLU activation function;

[0105] Step 3.2.4, use the InforNCE loss as the loss function for the instance branch:

[0106]

[0107] Wherein, τ>0 is a temperature hyperparameter used to adjust the loss; in this invention, τ=0.2 is set.

[0108] Step 3.3 Constructing Local Semantic Branches: The local bounding boxes matching the overlapping regions IS(r1,r2) between the two views obtained in Step 2.2.1 are used to construct local semantic branches to enhance the network's local feature learning ability. The specific process is as follows:

[0109] Step 3.3.1, obtain the corresponding coordinate frame in step 2.2.4. and Then, the corresponding region is cropped from the feature map and called the Region of Interest (RoI). The cropped feature map is then locally pooled using 1×1 RoIAlign. Where k = {1, ..., K}, and K is the total number of bounding boxes generated in the image;

[0110] Step 3.3.2, similar to the instance branch, after obtaining the pooling features, passes them through a projection head to obtain... and The difference is that, in the local semantic branch, the online network is followed by a prediction head, which further maps... get Used to predict the output of the target branch Aligning local semantic information to enhance the local feature representation of the network is beneficial for downstream land feature extraction tasks.

[0111] Step 3.3.3 defines the semantic consistency loss as the mean squared error between the normalized prediction and the normalized target projection, as shown below:

[0112]

[0113] in, Used to normalize eigenvectors.

[0114] Step 3.4 Network Training: The instance branch enhances feature discrimination by learning the distinctiveness of image instances, while the local semantic branch captures the semantic consistency of remote sensing images. The combination of the instance branch and the local semantic branch achieves global and local semantic alignment. Their synergistic effect helps the model understand the overall scene structure and fine-grained details. Therefore, the overall loss function of the pre-trained network framework of this invention can be defined as:

[0115] L=L1+λL2

[0116] Here, λ is a hyperparameter used to balance the loss of instance branches and local semantic branches. In this invention, λ is set to 5.

[0117] The gradient propagates only in the online network. The parameters ξ of the target network follow the parameters θ of the online network through an exponential moving average with a decaying parameter τ, i.e., ξ←τξ+(1-τ)θ.

[0118] During pre-training, the loss changes are recorded. If the loss does not decrease for m consecutive epochs, training is stopped and the weights of the pre-trained network are saved. In this invention, m is set to 10.

[0119] like Figure 4 As shown, in step 4, the pre-trained weights from step 3 are loaded into the backbone network of the downstream feature extraction network, and fine-tuned using the training and validation sets of the downstream fine-tuning dataset from step 1 to obtain the feature extraction model.

[0120] The specific process of loading the pre-trained weights into the backbone network of the downstream land cover extraction network for fine-tuning is as follows:

[0121] Step 4.1: Use DeeplabV3+ as the downstream ground feature extraction network, and use cross-entropy loss to train the downstream ground feature extraction network, which is defined as follows:

[0122]

[0123] Where C represents the number of categories, y i p represents the true label set for the i-th class. i This represents the prediction result for the i-th class;

[0124] Step 4.2: Load the pre-trained weights obtained in step 3.4 into the backbone network of the downstream feature extraction network in step 4.1;

[0125] Step 4.3: Fine-tune the downstream feature extraction network from Step 4.2 using the training and validation sets from the downstream fine-tuning dataset in Step 1.4. During this fine-tuning process, the present invention employs the following procedure:

[0126] Step 4.3.1: Try a series of different learning rates to find the learning rate that best suits the model;

[0127] Step 4.3.2: After every l training cycles, the validation set in the downstream fine-tuning dataset in step 1.4 is used to evaluate the performance of the model. In this invention, l is set to 5.

[0128] Step 4.3.3: If the results of 1 consecutive validations show no improvement in accuracy, then stop training at the current learning rate;

[0129] Step 4.3.4: Finally, select the best-performing model from these learning rates and save it as the final land cover extraction model. This method aims to ensure that the model achieves optimal extraction accuracy through meticulous tuning and rigorous performance monitoring.

[0130] Step 5: Test the land feature extraction model obtained in Step 4 on the test set of the downstream fine-tuning data in Step 1 to obtain the prediction result map, and evaluate the performance of the prediction result map.

[0131] The specific process of performance evaluation in step 5 is as follows:

[0132] The land feature extraction model obtained in step 4 is tested on the test set of the downstream fine-tuning data in step 1.4 to obtain the prediction result map, and the classification indices OA, Kappa, MIoU, and FWIoU are calculated.

[0133] like Figure 5 The image shown is the final result. Figure 5 (a) and Figure 5 (d) It can be seen that the results derived using the pre-training method of this invention are closer to the label map and the results are more complete. Figure 5 (b) and Figure 5 (c) It can be seen that the results for small targets such as roads and wetlands are more complete when using the pre-training method of this invention. This is due to the local perception capability of local semantic branches.

[0134] The effects of the present invention will be further explained below with reference to simulation experiments.

[0135] 1. Simulation experimental conditions:

[0136] The hardware testing platform for the simulation experiment of this invention is: CPU 192 cores 768G; graphics card is Ascend 910 Premium A32G, using 8 cards for parallel training.

[0137] The software platform for the simulation experiment of this invention is as follows: Operating system: EulerOS 2.0 (SP8); Online platform: ModelArts; Notebook integrated development environment, Python 3.7.6; Hardware platform: Ascend 910

[0138] Experimental Methods: A pre-trained large model for SAR image land cover extraction based on global-local semantic alignment

[0139] 2. Simulation content and result analysis:

[0140] 2.1 Simulation Content

[0141] Figure 2 This is a schematic diagram of the data augmentation method used in this invention to generate two views and a partial bounding box.

[0142] Figure 3 This is a schematic diagram of the global-local semantic alignment pre-trained network framework of the present invention;

[0143] Figure 4 This is a schematic diagram of the downstream feature extraction network architecture of the present invention;

[0144] Figure 5 These are the ground feature extraction results images obtained through the pre-training method of this invention and through other pre-training methods.

[0145] The SAR imagery used in the simulation experiments of this invention was captured by the Gaofen-3 remote sensing satellite. The data was divided into a pre-training dataset and a downstream fine-tuning dataset. The pre-training dataset included 339,489 unlabeled images, while the downstream fine-tuning dataset contained 33,111 labeled images. The downstream fine-tuning dataset was randomly divided into training, validation, and test sets in an 8:1:1 ratio. The pre-training method used in this invention, along with two existing methods, BYOL and MoCoV2, was pre-trained on the pre-training dataset to derive the backbone network weights. These weights were then transferred to the downstream pre-training network and fine-tuned using the training and validation sets of the downstream fine-tuning dataset to obtain the final ground feature extraction model. Finally, ground feature extraction and performance evaluation were performed on the test set of the downstream fine-tuning dataset.

[0146] 2.2 Simulation Result Analysis

[0147] See Figure 5 , Figure 5 The images show the ground feature extraction results after using the pre-training method of this invention and other pre-training methods. Table 1 shows the overall accuracy of the test set using the methods of this invention and the two existing methods, BYOL and MoCoV2.

[0148] Table 1 Comparison of Simulation Results

[0149] OA 0.84826 0.85509 0.85723 Kappa 0.79559 0.80479 0.80785 MIoU 0.52127 0.53408 0.53730 FWIoU 0.74209 0.75192 0.75564

[0150] BYOL comes from the paper "Bootstrap your ownlatent - a new approach to self-supervised learning" published by Grill JB, Strub F, et al.

[0151] MOCO v2 comes from the paper "Improved baselines with momentum contrastive learning" published by Chen X, Fan H, et al.

[0152] As shown in Table 1, the present invention achieved an accuracy of 85.723%, a Kappa score of 0.80785, a MIoU score of 0.53730, and a FWIoU score of 0.75564 on the test set. The segmentation visual effect conforms to the real ground features and is superior to some existing methods.

[0153] The overall qualitative evaluation indicators are as follows: Figure 5 , Figure 5 (a) and Figure 5 (d) It can be seen that the results derived using this pre-training method are closer to the label map and the results are more complete. Figure 5 (b) and Figure 5 (c) It can be seen that the results for small targets such as roads and wetlands are more complete when using this pre-training method. This is due to the local perception capability of local semantic branches.

[0154] The simulation results above demonstrate that this invention collects a massive amount of unlabeled SAR imagery to construct a pre-training dataset, enabling the pre-trained large model to fully exploit the potential information in SAR imagery and improve the performance of downstream ground feature extraction tasks. Furthermore, this invention introduces local semantic branches to capture the local semantic consistency of images, improving the classification completeness of small targets, specifically tailored to the characteristics of SAR image ground feature extraction tasks.

[0155] This invention also provides a SAR image pre-training large model ground feature extraction system based on global-local semantic alignment, comprising:

[0156] The massive SAR image data acquisition module is used to acquire massive SAR image data in step 1, perform preprocessing, create a dataset, divide the dataset into a pre-training dataset and a downstream fine-tuning dataset, and further divide the downstream fine-tuning dataset into a training set, a validation set, and a test set.

[0157] The pre-trained dataset processing module is used to perform data augmentation on the pre-trained dataset in step 1 in step 2, and generate two views of the same image and corresponding local bounding boxes between the views.

[0158] The global-local semantic alignment pre-trained network construction module is used to build the global-local semantic alignment pre-trained network in step 3. It uses two views of the same image generated in step 2 and the corresponding local boxes between the views to train the network and obtain the pre-trained weights.

[0159] The feature extraction model acquisition module is used to load the pre-trained weights from step 3 into the backbone network of the downstream feature extraction network in step 4, and to fine-tune the model using the training and validation sets of the downstream fine-tuning dataset from step 1 to obtain the feature extraction model.

[0160] The prediction result map performance evaluation module is used to test the land feature extraction model obtained in step 4 on the test set of the downstream fine-tuning data in step 1 in step 5, obtain the prediction result map, and evaluate the performance of the prediction result map.

[0161] This invention also provides a SAR image pre-trained large model ground feature extraction device based on global-local semantic alignment, comprising:

[0162] Memory: A computer program that stores the above-mentioned method for extracting ground features from a large model of SAR image pre-training based on global-local semantic alignment, and is a computer-readable device;

[0163] Processor: Used to implement the method for extracting ground features from a large SAR image pre-training model based on global-local semantic alignment when executing the computer program.

[0164] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the aforementioned method for extracting ground features from a large SAR image pre-training model based on global-local semantic alignment.

Claims

1. A method for extracting ground features from a large SAR image pre-trained model based on global-local semantic alignment, characterized in that, Includes the following steps: Step 1: Collect massive amounts of SAR image data, perform preprocessing, create a dataset, divide the dataset into a pre-training dataset and a downstream fine-tuning dataset, and further divide the downstream fine-tuning dataset into a training set, a validation set, and a test set. Step 2: Perform data augmentation on the pre-trained dataset from Step 1 to generate two views of the same image and corresponding local bounding boxes between the views; Step 3: Construct a global-local semantic alignment pre-trained network framework. The global-local semantic alignment pre-trained network framework includes two branches: an instance branch and a local semantic branch. By combining the instance branch and the local semantic branch, global and local semantic alignment is achieved. Using the two views of the same image generated in Step 2 and the corresponding local boxes between the views, the network is trained to obtain the pre-trained weights. Step 4: Load the pre-trained weights from Step 3 into the backbone network of the downstream feature extraction network, and fine-tune the model using the training and validation sets of the downstream fine-tuning dataset from Step 1 to obtain the feature extraction model. Step 5: Test the land feature extraction model obtained in Step 4 on the test set of the downstream fine-tuning data in Step 1 to obtain the prediction result map, and evaluate the performance of the prediction result map.

2. The method for extracting ground features from a large SAR image pre-training model based on global-local semantic alignment according to claim 1, characterized in that, The specific process of step 1 is as follows: Step 1.1: Collect massive amounts of SAR image data; Step 1.2: Perform multi-view processing, orthorectification, and depth-level conversion on the SAR image data obtained in Step 1.1, and select some of the processed SAR images for artificial land cover category labeling. Step 1.3: Divide the SAR image data processed in Step 1.2 into a pre-training dataset and a downstream fine-tuning dataset; Step 1.4: Divide the downstream fine-tuning dataset obtained in Step 1.3 into a training set, a validation set, and a test set.

3. The method for extracting ground features from a large SAR image pre-training model based on global-local semantic alignment according to claim 1, characterized in that, The specific process of step 2, which involves data augmentation to generate two views of the same image and corresponding local bounding boxes between the views, is as follows: Step 2.1, generate two views of the same image: using This represents images in a pre-training dataset. Data augmentation strategies include random cropping, scaling, brightness and contrast jitter, Gaussian blur, and random horizontal flipping. To indicate a random cropping operation, use The remaining data augmentation operations involve first performing two random cropping operations on the images in the pre-training dataset from step 1.3, resulting in... and Then, perform subsequent data augmentation operations on the cropped image to obtain... and That is, to obtain two views of the same image; Step 2.2, Generate corresponding local bounding boxes between views: To align local semantic information, that is, when two local regions correspond to the same spatial region, minimize the feature representations of these two local regions to make them have the same semantic information, it is necessary to generate local bounding boxes with the same spatial region. The specific process is as follows: Step 2.2.1: First, find two random cropped views. overlapping areas ( , ),in ( , ) indicates the random cropping view and The overlapping spatial regions are generated in the middle; Step 2.2.2: After finding the overlapping spatial regions, randomly select from the overlapping regions. Sampling a box ,in Used to represent a box The coordinates; Step 2.2.3, place the box Convert to each view and The corresponding coordinates in can be denoted as ; Step 2.2.4 generates more local bounding boxes and obtains multiple pairs of local representations; that is, it obtains the coordinate boxes. and ,in , Given the total number of bounding boxes generated in the image, calculate the IoU (Intersection over Union) between the newly generated candidate boxes and the already generated boxes. If the IoU is greater than 50%, reject the candidate box. Repeat this process until the number of retained boxes reaches a certain threshold. .

4. The method for extracting ground features from a large SAR image pre-training model based on global-local semantic alignment according to claim 3, characterized in that, Step 3 involves constructing a global-local semantic alignment pre-trained network and training the network as follows: Step 3.1, Constructing the pre-trained network framework: The pre-trained network framework consists of two neural networks, namely, networks composed of a set of parameters. Defined online networks and by The target network defines the parameters; the target network provides the regression objective for training the online network, and the parameters of the target network... Through a parameter with attenuation Exponential moving averages are used to track parameters of online networks. ,Right now The encoding network used is ResNet-101. Step 3.2, Constructing Instance Branches: During training, corresponding data for the same sample are augmented into a pair of positive samples, while other samples are negative samples. Assume the batch size is... So each sample has at most 100 samples. There are n negative samples, and for each sample in the queue, let them be denoted as n. That is, the queue stores One negative sample; each iteration updates according to the "first-in, first-out" principle. The specific process for constructing an instance branch from a given negative sample is as follows: Step 3.2.1, combine the two views obtained in step 2.1 ( , Each is input into the two encoding networks constructed in step 3.

1. In the process, the feature map is obtained. and ; Step 3.2.2, take the feature map obtained in step 3.2.

1. and Global average pooling (GAP) yields instance-level features of an image. and ; Step 3.2.3: Use a projector to map the extracted global features to a high-dimensional feature representation space. Use a multilayer perceptron (MLP) with one hidden layer as the projector, which can be represented as: in, Indicates the projection head. Indicates a fully connected layer. Represents the ReLU activation function; Step 3.2.4, use the InforNCE loss as the loss function for the instance branch: in, This is a temperature hyperparameter used to adjust for losses; Step 3.3, Construct local semantic branches: Combine the overlapping areas between the two views obtained in Step 2.2.

1. ( , The matched local bounding boxes are used to construct local semantic branches, which enhances the network's ability to learn local features. The specific process is as follows: Step 3.3.1, obtain the corresponding coordinate frame in step 2.2.

4. and Then, the corresponding region is cropped from the feature map and called the Region of Interest (RoI). The cropped feature map is then locally pooled using 1×1 RoI Align. , ,in , This represents the total number of bounding boxes generated in the image. Step 3.3.2, similar to the instance branch, after obtaining the pooling features, passes them through a projection head to obtain... and In the local semantic branch, a prediction head is appended to the online network for further mapping. ,get Used to predict the output of the target branch Align local semantic information to enhance the local feature representation of the network; Step 3.3.3 defines the semantic consistency loss as the mean squared error between the normalized prediction and the normalized target projection, as shown below: in, , , used to normalize the feature vector; Step 3.4, perform network training: The overall loss function of the pre-trained network framework can be defined as: in, It is a hyperparameter used to balance the loss between instance branches and local semantic branches; The gradient only propagates in the online network, while the parameters of the target network... Through a parameter with attenuation Exponential moving averages are used to track parameters of online networks. ,Right now ; During pre-training, record the changes in loss; if continuous... If the loss does not decrease after one epoch, training is stopped and the weights of the pre-trained network are preserved.

5. The method for extracting ground features from a large SAR image pre-training model based on global-local semantic alignment according to claim 1, characterized in that, The specific process of loading the pre-trained weights into the backbone network of the downstream land cover extraction network for fine-tuning in step 4 is as follows: Step 4.1: Use DeeplabV3+ as the downstream ground feature extraction network, and use cross-entropy loss to train the downstream ground feature extraction network, which is defined as follows: in, Indicates the number of categories. Indicates the first The actual label set by the class Indicates the first The prediction results for the class; Step 4.2: Load the pre-trained weights obtained in step 3.4 into the backbone network of the downstream feature extraction network in step 4.1; Step 4.3: Fine-tune the downstream feature extraction network from Step 4.2 using the training and validation sets from the downstream fine-tuning dataset in Step 1.

4. The following process is adopted during the fine-tuning: Step 4.3.1: Try a series of different learning rates to find the learning rate that best suits the model; Step 4.3.2, each time passing through In each training cycle, the performance of the model is evaluated using the validation set in the downstream fine-tuning dataset from step 1.

4. Step 4.3.3, if continuous If the validation results show no improvement in accuracy, then training at the current learning rate should be stopped. Step 4.3.4: Select the best-performing model from these learning rates and save it as the final ground feature extraction model.

6. The method for extracting ground features from a large SAR image pre-training model based on global-local semantic alignment according to claim 1, characterized in that, The specific process of performance evaluation in step 5 is as follows: The land cover extraction model obtained in step 4 is tested on the test set of the downstream fine-tuning data in step 1.4 to obtain the prediction result map, and the classification indices OA, Kappa, MIoU and FWIoU are calculated.

7. A SAR image pre-training large model ground feature extraction system based on global-local semantic alignment, according to any one of claims 1 to 6, characterized in that, include: The massive SAR image data acquisition module is used to acquire massive SAR image data, perform preprocessing, create datasets, divide the datasets into pre-training datasets and downstream fine-tuning datasets, and further divide the downstream fine-tuning datasets into training sets, validation sets, and test sets. The pre-trained dataset processing module is used to perform data augmentation on the pre-trained dataset, generating two views of the same image and corresponding local bounding boxes between the views; The Global-Local Semantic Alignment Pre-trained Network Building Module is used to build a global-local semantic alignment pre-trained network. It uses two views of the same generated image and the corresponding local bounding boxes between the views to train the network and obtain the pre-trained weights. The feature extraction model acquisition module is used to load pre-trained weights into the backbone network of the downstream feature extraction network, and to fine-tune the model using the training and validation sets of the downstream fine-tuning dataset in step 1 to obtain the feature extraction model. The prediction result map performance evaluation module is used to test the ground feature extraction model on the test set of the downstream fine-tuning data in step 1 to obtain the prediction result map and evaluate its performance.

8. A SAR image pre-training large model ground feature extraction device based on global-local semantic alignment, characterized in that, include: Memory: A computer program for a method of extracting ground features from a large model of SAR image pre-training based on global-local semantic alignment as described in any one of claims 1-6, and is a computer-readable device; Processor: Used to implement the method for extracting ground features from a large SAR image pre-training model based on global-local semantic alignment as described in any one of claims 1-6 when executing the computer program.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, enables the implementation of the SAR image pre-training large model ground feature extraction method based on global-local semantic alignment as described in any one of claims 1-6.

Images