Remote sensing image model parameter training method and system

Abstract

The invention relates to a remote sensing image model parameter training method, which comprises the following steps of: processing an original data set to obtain a merged data set; converting the merged data set into a format suitable for training, cutting the merged data set, and dividing the merged data set into a training set, a verification set and a test set in proportion; generating a synthetic data set from the preprocessed combined data set through data set distillation, so that a model trained on the synthetic data set and a model trained on the original data set have similar test performance; and finely adjusting the remote sensing image model by using the synthesized data set obtained by distillation. The invention further relates to a remote sensing image model parameter training system. According to the invention, the capability of the remote sensing image model in a remote sensing image accurate interpretation task can be improved.

Description

Technical Field

[0001] The invention relates to a remote sensing image model parameter training method and system. Background Art

[0002] Parameter-Efficient Fine-Tuning (PEFT) is a fine-tuning strategy for deep learning models that aims to quickly adapt models to specific tasks with limited training data and computing resources. The core idea is to perform further training on the basis of a pre-trained model. In this process, the parameters of the original pre-trained model are used as initial parameters and then fine-tuned using labeled data related to the specific task. This helps the model adapt to new tasks while retaining the knowledge of the pre-trained model on general visual features. Parameter-Efficient Fine-Tuning generally uses techniques such as small batch training, low learning rate, and regularization to avoid overfitting while effectively utilizing the knowledge of the pre-trained model. This strategy is of great significance in fields such as remote sensing image processing, which can reduce the time and hardware resources required for model fine-tuning and improve the usability and adaptability of the model. The key to parameter-efficient fine-tuning is to balance the parameter update of the model and avoid overfitting to achieve the best performance on a limited dataset. Based on whether to improve the encoding structure (Encoder) or the decoding structure (Decoder) of the pre-trained model for efficient parameter fine-tuning, the existing technology divides the efficient parameter fine-tuning technology into two categories: additive methods and re-parameterization methods.

[0003] The core idea of ​​additive methods is to add additional structures or parameters to the pre-trained large model to better map to the target task space. These methods usually assume that the encoder part of the large model is effective, freeze the original parameters of the pre-trained model, and mainly enhance or adjust the decoder part to adapt to specific tasks. According to whether additional layers or additional parameters are added, typical representatives of additive methods are 1.1) Adapters; 1.2) Soft Prompts.

[0004] 1.1) The adapter method is an important branch of the additive method. It was proposed by Houlsby et al. in 2019. The adapter method mainly adapts downstream tasks by adding learnable network modules to the pre-trained model based on the Transformer architecture. The core idea is to achieve adaptation of downstream tasks by adding additional layers when the encoder part of the pre-trained model is fixed. The adapter method believes that the pre-trained large model already has general knowledge of the target space, and the parameter space of the large model can be fixed. The mapping between the target space and the parameter space of the large model is achieved by embedding the adapter method module. The key to the adapter method is to add adapter modules in different layers of the model. These modules only learn information about specific tasks without large-scale fine-tuning of the parameters of the entire model. Many studies have since improved and expanded the adapter module. For example, Wang et al. proposed Adamix, which improves the performance of the adapter module by adding an expert group to achieve better adaptability. However, although the adapter method improves the training speed, it introduces a new feedforward neural network (FFN) layer, which incurs additional inference overhead in the inference phase, resulting in a decrease in the prediction speed in the prediction phase. Secondly, the adapter layer needs to store additional weights, which increases the memory requirements of the model.

[0005] 1.2) Soft hints are another important branch of additive methods. Soft hints assume that the parameter space of the pre-trained model already contains the knowledge of the target space. By adding hints to the input data, the mapping between the parameter space of the pre-trained model and the target space can be guided. Different from the contextual learning of hard hints, it introduces learnable hint variables. During model training, the entire model remains unchanged and only soft hints can be trained. Representative studies include the prompt tuning method proposed by Lester et al. and the prefix tuning method proposed by Li et al. These soft hints can be used for different downstream tasks, thereby enhancing the versatility of the model. Recent studies have shown that soft hints can also achieve significant results in multimodal tasks. It should be noted that the longer the hint is, the better the performance is, but it is also accompanied by additional computing resource overhead. If the length or complexity of the hint is too high, there will also be overfitting problems. In addition, in order to train learnable soft hints, large-scale datasets are usually required, which may be a challenge for tasks with scarce data.

[0006] In summary, the additive method hopes to freeze the entire original model parameters after pre-training a model, and adapt to downstream tasks by adding additional layers or parameters and only training these additional parameters. However, adding additional parameters or layers will bring additional overhead. For example, the adapter method introduces a new feedforward neural network layer, so there will be additional inference overhead in the inference phase. For soft prompt word method, if the length of the prefix sequence reaches 20 to 100 tokens, the computational complexity of the pre-trained large model in the inference phase will increase significantly. This is mainly due to the computational complexity of Transformer is O(n2), where is the input sequence length. For computational efficiency reasons, the existence of prefix sequences will limit the length of prompt words used to process downstream tasks. In addition, since soft prompts are usually high-dimensional vectors, and the pre-trained model parameters themselves also constitute a high-dimensional space, there is a high-dimensional parameter space in the parameter optimization problem, which leads to the problem of difficulty in optimization of the soft prompt method.

[0007] Another major category of efficient parameter fine-tuning methods is reparameterization methods. Since no new parameters or network layers are introduced, there is generally no additional overhead. This type of method believes that the pre-trained large model needs further adjustment in the encoder to better fit the target task. The core idea is to efficiently fine-tune the original parameters of the large model through different strategies. Reparameterization methods are divided into two subcategories based on whether to choose to update some parameters or to represent the overall parameters in a low-rank manner: 2.1) Selective methods; 2.2) Low-Rank Representation methods.

[0008] 2.1) The selective method is to select the depth of a layer, the type of layer, or a parameter for fine-tuning based on the pre-trained model. For example, in the work of BitFit, it is believed that the internal parameter space of the pre-trained model is similar to the fitting curve of the target task space. By adjusting the orientation of the fitting curve, the mapping of the large model parameter space and the target space can be achieved. BitFit freezes most of the parameters. For each linear or convolutional layer, the weight matrix W remains unchanged, and only the bias term b is fine-tuned. The original paper of BitFit shows that this method can achieve similar or better performance than full fine-tuning in the case of low data of the BERT model (<1B parameters) by only updating about 0.05% of the model parameters. However, further research evaluated this method on larger networks, such as T0 or GPT-3 with 3B parameters, and the performance of BitFit was significantly lower than that of full fine-tuning and other PEFT methods. For most selective methods, usually only part of the network parameters are modified, so there are limitations, especially for models with a large number of parameters or tasks that require more complex adaptability; at the same time, these methods need to determine in advance which parameters should be modified, which usually requires some prior knowledge or heuristic methods. If the choice is inappropriate, the model performance may deteriorate.

[0009] 2.2) The low-rank representation rule represents the original parameter space by decomposing the original parameter matrix into the product of two or more low-rank matrices. Such a decomposition can significantly reduce the number of trainable parameters in the model while retaining the representation ability of the model. For example, in the work of Low-Rank Adaptation (LoRA), researchers were inspired by IntrinsicSAID: for large pre-trained models, their parameter space is highly redundant, that is, the intrinsic rank of the model is very small, so these parameters can be represented by low-rank approximation. LoRA assumes that the change in parameters in the process of adapting to downstream tasks is low-rank, and these changes can be decomposed into low-rank to simplify the number of parameters. The specific implementation of LoRA is mainly used for the Wk and Wv matrices in the self-attention module. For the selected target matrix, singular value decomposition or other low-rank matrix decomposition methods are used for low-rank approximation. During the fine-tuning process, the matrix after low-rank approximation is used to replace the original matrix. Therefore, compared with other methods, LoRA has no reasoning delay problem and will not reduce the model reasoning speed; secondly, LoRA has a lower training threshold. For each fully connected layer, LoRA only optimizes two low-rank matrices, so only a small number of parameter gradient calculations and their corresponding optimizer states need to be retained. In addition, LoRA is compatible with most PEFT methods and can be combined with other parameter efficient fine-tuning methods to further optimize model performance. However, there are still some problems with LoRA itself: 1) LoRA needs to specify the intrinsic rank of the parameter matrix in advance. If the intrinsic rank is not selected properly, it may lead to a decrease in model performance. Usually, finding this intrinsic rank requires experiments and adjustments, which increases the complexity of development. 2) LoRA's method is to set all matrices to be optimized to the same intrinsic rank. However, different parameter matrices may have different information contents, so restricting them to the same intrinsic rank may limit the expressive power of the model.

[0010] In summary, in recent years, basic large models have shown strong generalization capabilities. In the field of computer vision, adapter methods and prompt learning methods have been explored to improve the data generalization of large models on natural images while reducing the need for large-scale fine-tuning. Visual-language large model prompt learning finds the optimal prompts instead of fine-tuning the entire visual-language large model to efficiently adapt the visual-language large model to downstream task scenarios. In addition, adapter methods have also been widely explored, and their core idea is to fine-tune the large model by using an additional lightweight feature adapter. In this way, only a small number of task-specific parameters need to be introduced and updated to adapt to downstream tasks, greatly improving the practicality of pre-trained models.

[0011] These two types of methods can be collectively classified as additive methods of PEFT. However, additive methods have some problems. For example, the adapter method introduces a new FFN layer, which increases the inference time, and the prompt fine-tuning method is difficult to converge the prompt learning during prompt fine-tuning because the prompt words and pre-trained model parameters are high-dimensional vectors. It also consumes input word units and increases the computational complexity of the inference stage.

[0012] For the reparameterization method, no additional parameters or structures are introduced, which can well avoid the common problems derived from the above-mentioned additive methods. Reparameterization methods generally do not have additional inference delays. Selection methods are a subclass of reparameterization methods. This type of method adapts downstream tasks by selecting and fine-tuning a small part of the parameters of the pre-trained model. Although the effectiveness of such methods has been verified in some scenarios or studies (such as BitFit only fine-tuning the bias term), such methods require to determine in advance which parameters need to be trained, which usually requires certain prior knowledge. Therefore, the generalization of such methods is weak and there is uncertainty in the improvement of model performance. Another subclass of reparameterization methods, low-rank representation methods, minimize the number of parameters by low-rank representation of parameter matrices. The representative low-rank representation method LoRA only needs to optimize two low-rank matrices, which significantly reduces the number of training parameters and reduces the memory requirements, greatly reducing the training threshold. Moreover, the LoRA method is orthogonal to other efficient parameter fine-tuning methods and can be used in combination to further improve model performance.

[0013] In summary, there are some problems when applying the basic visual big model to the remote sensing field in the existing technology: the current visual big model is difficult to directly adapt to complex remote sensing data when using the efficient parameter fine-tuning method; this method fails to fully utilize the information of each channel when processing multi-source data input. Because the mainstream visual big model limits the input to the three channels of red, green and blue when designing, it fails to consider the differences in the spectral response functions of multi-source remote sensing data, and directly inputting multi-source remote sensing data into the model leads to the loss of other channel information. Summary of the invention

[0014] In view of this, it is necessary to provide a remote sensing image model parameter training method and system, which can improve the ability of the remote sensing image model in the task of accurate interpretation of remote sensing images.

[0015] The present invention provides a remote sensing image model parameter training method, which includes the following steps: S1, processing an original data set to obtain a merged data set; S2, converting the merged data set into a format suitable for training, cropping, and dividing it into a training set, a verification set, and a test set in proportion; S3, generating a synthetic data set by performing data set distillation on the preprocessed merged data set, so that the model trained on the synthetic data set has similar test performance to the model trained on the original data set; S4, fine-tuning the remote sensing image model using the synthetic data set obtained by distillation.

[0016] Preferably, the step S1 comprises:

[0017] The original data set is a high-resolution data set.

[0018] Preferably, the step S1 comprises:

[0019] Step S11, use ArcMap to load the original dataset, open the working directory, create a Shapefile file as a label layer in the folder where the dataset is stored, and name the file mask_shp;

[0020] Step S12, load mask_shp, open the attribute table of mask_shp, add the cls field as the category attribute, then edit the elements in mask_shp, symbolize and classify the data, and mark the required categories in mask_shp;

[0021] Step S13: Use the conversion tool in the ArcMap toolbar to convert the annotated label layer into a raster, align the resolution of the original dataset, and then use the band synthesis in the data management tool to synthesize the original data and the label layer into a merged dataset.

[0022] Preferably, the step S2 comprises:

[0023] Step S21, creating a script code to check for outliers, duplicates or damaged files in the data set, and repair or remove them, and confirming that all image files can be correctly read by the PIL image processing package in Python;

[0024] Step S22, using PIL to take the last channel of the merged data set as a label, and then merge the image and the label and save them as a new file;

[0025] Step S23, using the GDAL library in Python to crop the images and labels of the merged dataset into size, and then save the images and labels as image folders and label folders respectively;

[0026] Step S24: divide the merged data set into a training set, a validation set, and a test set in a ratio of 70:15:15, and save them in corresponding folders respectively.

[0027] Preferably, the step S3 comprises:

[0028] Step S31, using UNet network training on the original data set, training and saving the model as a teacher model pool according to different learning rates, and the entire link of a single teacher model training is called an expert trajectory;

[0029] Step S32, in each distillation cycle: 1) train the student model on the synthetic data set, and the initial parameters of the student model are randomly selected from a random step of the teacher model; 2) calculate the loss based on the parameters of the student model after training and the final parameters of the selected teacher model; 3) update the synthetic data set and learning rate based on the loss;

[0030] Step S33, iteratively re-randomly select expert trajectories, continue to train the student network with the synthetic data set, and optimize the synthetic data set so that the training effect of the student network on the synthetic data set is closer to the effect of training on the original data set.

[0031] Preferably, in step S31:

[0032] The UNet network is replaced by other deep networks or deep convolutional network structures.

[0033] Preferably, the step S32 comprises:

[0034] Los s The calculation formula is:

[0035]

[0036] Preferably, the step S4 comprises:

[0037] Step S41, in the preparation stage, install the necessary segment-anything and pytorch-lightning libraries, define the pre-trained image encoder freeze in the SAM model in the configuration file, and configure the location, batch size, and learning rate of the dataset;

[0038] Step S42: Load the original SAM model that has been downloaded locally, and inject each Q, K, and V matrix in the SAM image encoder into the LoRA block;

[0039] Step S43, freezing the mask generator and image encoder of the SAM model, and unfreezing the LoRA block injected into the Q, K, and V matrices;

[0040] Step S44: Use the synthetic data set obtained in step S3 to train the SAM model. The parameter update ΔW is represented by a low-rank decomposition matrix, namely:

[0041] W0+ΔW=W0+BA

[0042] in,

[0043] Preferably, in step S4:

[0044] The SAM model is replaced by the visual macro model.

[0045] The present invention also provides a remote sensing image model parameter training system, which includes a data set preparation module, a data preprocessing module, a data set distillation module, and a model fine-tuning module, wherein:

[0046] The data set preparation module is used to process the original data set to obtain a merged data set;

[0047] The data preprocessing module is used to convert the merged data set into a format suitable for training, perform cropping, and divide it into a training set, a validation set, and a test set in proportion;

[0048] The data set distillation module is used to generate a synthetic data set by data set distillation of the preprocessed merged data set, so that the model trained on the synthetic data set has similar test performance to the model trained on the original data set;

[0049] The model fine-tuning module is used to fine-tune the remote sensing image model using the synthetic data set obtained by distillation.

[0050] The present invention proposes an efficient parameter fine-tuning model for remote sensing multi-channels, thereby improving the ability of the visual large model in the task of accurately interpreting remote sensing images. The beneficial effects of the present invention specifically include:

[0051] (1) The present invention can better adapt to complex remote sensing data, especially multi-source data, and improve the applicability and accuracy of the model in the task of intelligent interpretation of remote sensing images.

[0052] (2) Through the efficient parameter fine-tuning method, the present invention significantly reduces the computing resource requirements for model training and reasoning while maintaining model performance.

[0053] (3) Dataset distillation technology enables the model to be effectively trained with limited real data, reducing dependence on large amounts of labeled data and improving training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1It is a flow chart of the remote sensing image model parameter training method of the present invention;

[0055] Figure 2 A schematic diagram of a process for preparing a data set provided in an embodiment of the present invention;

[0056] Figure 3 A schematic diagram of a single data set distillation process provided by an embodiment of the present invention;

[0057] Figure 4 A schematic diagram of the overall framework for remote sensing image model parameter training provided by an embodiment of the present invention;

[0058] Figure 5 A schematic diagram of a parameter training process provided by an embodiment of the present invention;

[0059] Figure 6 The hardware architecture diagram of the remote sensing image model parameter training system of the present invention. DETAILED DESCRIPTION

[0060] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0061] See also Figure 1 FIG. 1 is a flowchart of a preferred embodiment of the remote sensing image model parameter training method of the present invention. Please refer to FIG. Figure 4 .

[0062] Step S1, data set preparation. That is: the original data set is processed to obtain a merged data set to prepare the data set required for fine-tuning, in preparation for the subsequent UNet data set distillation. The original data set is a high-resolution data set. Please refer to Figure 2 , specifically:

[0063] Step S11, use ArcMap to load the original dataset, open the working directory, create a Shapefile file as a label layer in the folder where the dataset is stored, and name the file mask_shp.

[0064] Step S12, load mask_shp, open the attribute table of mask_shp, add the cls field as the category attribute, then edit the elements in mask_shp, symbolize and classify the data, and mark the required categories in mask_shp.

[0065] Step S13: Use the conversion tool in the ArcMap toolbar to convert the annotated label layer into a raster, align the resolution of the original dataset, and then use the band synthesis in the data management tool to synthesize the original data and the label layer into a merged dataset.

[0066] Step S2, preprocessing the merged data set. That is, converting the merged data set into a format suitable for training, trimming it, and dividing it into a training set, a validation set, and a test set in proportion. Specifically:

[0067] Step S21, create a script code to check the outliers, duplicates or damaged files in the data set, and repair or remove them, and confirm that all image files can be correctly read by PIL (image processing package) in Python.

[0068] Step S22: Use PIL to take the last channel of the merged data set as a label, and then merge the image and the label and save them as a new file.

[0069] Step S23: Use the GDAL library in Python to crop the images and labels of the merged dataset to size, and then save the images and labels as image folders and label folders respectively.

[0070] Step S24: divide the merged data set into a training set, a validation set, and a test set in a ratio of 70:15:15, and save them in corresponding folders respectively.

[0071] Step S3, data set distillation. That is, the preprocessed merged data set is distilled to generate a synthetic data set, so that the model trained on the synthetic data set has similar test performance to the model trained on the original data set. Specifically:

[0072] Step S31, this embodiment uses UNet network training on the original data set, trains and saves the model as a teacher model pool according to different learning rates, and the entire link of a single teacher model training is called an "expert trajectory".

[0073] In other embodiments, the UNet network can be replaced by other deep networks or deep convolutional network structures.

[0074] Step S32, please refer to Figure 3 In each distillation cycle: 1) Train the student model on the synthetic dataset, and the initial parameters of the student model are randomly selected from a random step of the teacher model; 2) Calculate the Loss based on the parameters of the student model after training and the final parameters of the selected teacher model; 3) Update the synthetic dataset and learning rate based on the Loss;

[0075] Among them, the calculation formula of Loss is:

[0076]

[0077] Step S33, iteratively re-randomly select expert trajectories, continue to train the student network with the synthetic data set, and optimize the synthetic data set so that the training effect of the student network on the synthetic data set is closer to the effect of training on the original data set. Figure 3 shown.

[0078] Step S4, fine-tune the Segment Anything Model (SAM). That is, use the synthetic dataset obtained by distillation to fine-tune the SAM model. The core idea is to add a small bypass matrix LoRA to the original model parameters to simulate the change in parameters. Freeze the original model parameters during the training phase, and only train the LoRA matrix to achieve efficient parameter fine-tuning.

[0079] In other embodiments, the segmentation macromodel SAM may be replaced by a visual macromodel.

[0080] Specifically:

[0081] Step S41, preparation phase, install the necessary segment-anything and pytorch-lightning libraries, define the pre-trained image encoder freeze in the SAM model in the configuration file, and configure the location of the dataset, batch size, and learning rate.

[0082] Step S42: Load the original SAM model that has been downloaded locally, and inject each Q, K, and V matrix in the SAM image encoder into the LoRA block.

[0083] Step S43, freezing the mask generator and image encoder of the SAM model, and unfreezing the LoRA block injected into the Q, K, and V matrices;

[0084] Step S44: Use the synthetic data set obtained in step S3 to train the model. For the pre-trained weight matrix The parameter update ΔW is represented by a low-rank decomposition matrix (LoRA block), namely:

[0085] W0+ΔW=W0+BA

[0086] in,

[0087] During the training process, the parameter W0 is frozen and only the parameters in A and B are trained, such as Figure 5 As shown, for h = W0x, the forward propagation process becomes:

[0088] h=W0x+ΔWx=W0x+BAx

[0089] See also Figure 6, which is a hardware architecture diagram of the remote sensing image model parameter training system 10 of the present invention. The system includes: a data set preparation module 101, a data preprocessing module 102, a data set distillation module 103, and a model fine-tuning module 104. Among them:

[0090] The dataset preparation module 101 is used for dataset preparation. That is, the original dataset is processed to obtain a merged dataset to prepare the dataset required for fine-tuning, in preparation for the subsequent dataset distillation by UNet. The original dataset is a high-resolution dataset. Figure 2 , specifically:

[0091] The dataset preparation module 101 uses ArcMap to load the original dataset, opens the working directory, creates a Shapefile file as a label layer in the folder storing the dataset, and names the file mask_shp.

[0092] The data set preparation module 101 loads mask_shp, opens the attribute table of mask_shp, adds the cls field as a category attribute, and then edits the elements in mask_shp, symbolically classifies the data, and marks the required categories in mask_shp.

[0093] The dataset preparation module 101 uses the conversion tool in the ArcMap toolbar to convert the annotated label layer into a raster, align the resolution of the original dataset, and then synthesize the original data and the label layer into a merged dataset through band synthesis in the data management tool.

[0094] The data preprocessing module 102 is used to perform data preprocessing on the merged data set. That is, converting the merged data set into a format suitable for training, cutting it, and dividing it into a training set, a validation set, and a test set in proportion. Specifically:

[0095] The data preprocessing module 102 creates script codes to check for outliers, duplicates or damaged files in the data set, and repairs or removes them, and confirms that all image files can be correctly read by PIL (image processing package) in Python.

[0096] The data preprocessing module 102 uses PIL to take the last channel of the merged data set as a label, and then merges the image and the label and saves them as a new file.

[0097] The data preprocessing module 102 uses the GDAL library in Python to crop the images and labels of the merged data set into sizes, and then saves the images and labels into image folders and label folders respectively.

[0098] The data preprocessing module 102 divides the merged data set into a training set, a validation set, and a test set in a ratio of 70:15:15, and saves them in corresponding folders respectively.

[0099] The data set distillation module 103 is used for data set distillation. That is, the preprocessed merged data set is distilled to generate a synthetic data set, so that the model trained on the synthetic data set has similar test performance to the model trained on the original data set. Specifically:

[0100] The data set distillation module 103 uses UNet network training on the original data set, trains and saves the model as a teacher model pool according to different learning rates, and the entire link of a single teacher model training is called an "expert trajectory".

[0101] See also Figure 3 In each distillation cycle, the data set distillation module 103: 1) trains a student model on the synthetic data set, where the initial parameters of the student model are a random step in a randomly selected teacher model; 2) calculates the Loss based on the parameters of the student model after training and the final parameters of the selected teacher model; 3) updates the synthetic data set and the learning rate based on the Loss;

[0102] Among them, the calculation formula of Loss is:

[0103]

[0104] The data set distillation module 103 iteratively re-randomly selects expert trajectories, continues to train the student network with the synthetic data set, and optimizes the synthetic data set so that the training effect of the student network on the synthetic data set is closer to the effect of training on the original data set. The process of single data set distillation is as follows: Figure 3 shown.

[0105] The model fine-tuning module 104 is used to fine-tune the visual large model SAM (Segment Anything Model). That is, the SAM model is fine-tuned using the synthetic data set obtained by distillation. The core idea is to add a small bypass matrix LoRA to the original model parameters to simulate the change in parameters. The original model parameters are frozen during the training phase, and the purpose of efficient parameter fine-tuning is achieved only by training the LoRA matrix. Specifically:

[0106] In the preparation stage, install the necessary segment-anything and pytorch-lightning libraries, define the pre-trained image encoder freeze in the SAM model in the configuration file, and configure the location of the dataset, batch size, and learning rate.

[0107] The model fine-tuning module 104 loads the original SAM model that has been downloaded locally, and each Q, K, and V matrix in the SAM image encoder is injected into the LoRA block.

[0108] The model fine-tuning module 104 freezes the mask generator and image encoder of the SAM model, and unfreezes the LoRA blocks injected into the Q, K, and V matrices;

[0109] The model fine-tuning module 104 uses the obtained synthetic data set to train the model. The parameter update ΔW is represented by a low-rank decomposition matrix (LoRA block), namely:

[0110] W0+ΔW=W0+BA

[0111] in,

[0112] During the training process, the parameter W0 is frozen and only the parameters in A and B are trained, such as Figure 5 As shown, for h = W0x, the forward propagation process becomes:

[0113] h=W0x+ΔWx=W0x+BAx

[0114] This application designs a new parameter efficient fine-tuning framework and uses the low-rank representation LoRA to fine-tune the large visual model, which not only reduces the number of model parameters, but also reduces the computational complexity during training and reasoning. At the same time, this application uses the UNet network to distill the original data set into a synthetic data set, which can integrate the differences of multi-source data and provide more suitable training samples to the large visual model SAM, thereby improving the training efficiency and generalization ability of the model.

[0115] Although the present invention has been described with reference to the current preferred embodiments, those skilled in the art should understand that the above-mentioned preferred embodiments are only used to illustrate the present invention and are not used to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A remote sensing image model parameter training method, characterized in that: The method comprises the following steps: S1, process the original data set to obtain a merged data set; S2, convert the merged dataset into a format suitable for training, crop it, and divide it into training set, validation set, and test set in proportion; S3, the preprocessed merged dataset is distilled to generate a synthetic dataset, so that the model trained on the synthetic dataset has similar test performance to the model trained on the original dataset; S4, fine-tuning the remote sensing image model using the synthetic dataset obtained by distillation.

2. The method according to claim 1, characterized in that The step S1 comprises: The original data set is a high-resolution data set.

3. The method according to claim 2, characterized in that The step S1 comprises: Step S11, use ArcMap to load the original dataset, open the working directory, create a Shapefile file as a label layer in the folder where the dataset is stored, and name the file mask_shp; Step S12, load mask_shp, open the attribute table of mask_shp, add the cls field as the category attribute, then edit the elements in mask_shp, symbolize and classify the data, and mark the required categories in mask_shp; Step S13: Use the conversion tool in the ArcMap toolbar to convert the annotated label layer into a raster, align the resolution of the original dataset, and then use the band synthesis in the data management tool to synthesize the original data and the label layer into a merged dataset.

4. The method according to claim 3, characterized in that The step S2 comprises: Step S21, creating a script code to check for outliers, duplicates or damaged files in the data set, and repair or remove them, and confirming that all image files can be correctly read by the PIL image processing package in Python; Step S22, using PIL to take the last channel of the merged data set as a label, and then merge the image and the label and save them as a new file; Step S23, using the GDAL library in Python to crop the images and labels of the merged dataset into size, and then save the images and labels as image folders and label folders respectively; Step S24: divide the merged data set into a training set, a validation set, and a test set in a ratio of 70:15:15, and save them in corresponding folders respectively.

5. The method according to claim 4, characterized in that The step S3 comprises: Step S31, using UNet network training on the original data set, training and saving the model as a teacher model pool according to different learning rates, and the entire link of a single teacher model training is called an expert trajectory; Step S32, in each distillation cycle: 1) train the student model on the synthetic data set, and the initial parameters of the student model are randomly selected from a random step of the teacher model; 2) calculate the loss based on the parameters of the student model after training and the final parameters of the selected teacher model; 3) update the synthetic data set and learning rate based on the loss; Step S33, iteratively re-randomly select expert trajectories, continue to train the student network with the synthetic data set, and optimize the synthetic data set so that the training effect of the student network on the synthetic data set is closer to the effect of training on the original data set.

6. The method according to claim 5, characterized in that In the step S31: The UNet network is replaced by other deep networks or deep convolutional network structures.

7. The method according to claim 6, characterized in that The step S32 comprises: The calculation formula of Loss is:

8. The method according to claim 7, characterized in that The step S4 comprises: Step S41, in the preparation stage, install the necessary segment-anything and pytorch-lightning libraries, define the pre-trained image encoder freeze in the SAM model in the configuration file, and configure the location, batch size, and learning rate of the dataset; Step S42: Load the original SAM model that has been downloaded locally, and inject each Q, K, and V matrix in the SAM image encoder into the LoRA block; Step S43, freezing the mask generator and image encoder of the SAM model, and unfreezing the LoRA block injected into the Q, K, and V matrices; Step S44: Use the synthetic data set obtained in step S3 to train the SAM model. The parameter update ΔW is represented by a low-rank decomposition matrix, namely: W0+ΔW=W0+BA in, and r<<min(d,k). .

9. The method according to claim 8, characterized in that In the step S4: The SAM model is replaced by the visual macro model.

10. A remote sensing image model parameter training system, characterized in that: The system includes a data set preparation module, a data preprocessing module, a data set distillation module, and a model fine-tuning module, among which: The data set preparation module is used to process the original data set to obtain a merged data set; The data preprocessing module is used to convert the merged data set into a format suitable for training, perform cropping, and divide it into a training set, a validation set, and a test set in proportion; The data set distillation module is used to generate a synthetic data set by data set distillation of the preprocessed merged data set, so that the model trained on the synthetic data set has similar test performance to the model trained on the original data set; The model fine-tuning module is used to fine-tune the remote sensing image model using the synthetic data set obtained by distillation.

Images