A method for updating on-board processing parameters based on low-rank adaptation

Abstract

The application provides an on-orbit processing parameter updating method based on low-rank adaptation, and belongs to the field of on-orbit remote sensing image processing. First, a ground intelligent information extraction module is constructed. Second, activation maps corresponding to convolution feature layers in each network model in the module are calculated, and a low-rank adaptation matrix is implanted according to the activation maps. Then, the on-orbit intelligent information extraction module is deployed. Next, the parameters of the ground intelligent information extraction module are fine-tuned according to the on-orbit processing result. Finally, model parameter updating is performed. The application can ensure that the model performance remains unchanged, reduce the number of on-orbit and ground model updating parameters, reduce the bandwidth occupation of on-orbit and ground model updating, and improve the on-orbit and ground model updating efficiency.

Description

Technical Field

[0001] This invention belongs to the field of on-orbit remote sensing image processing, and more specifically relates to a method for updating on-board processing parameters based on low-rank adaptation. Background Technology

[0002] With the increasing number of space-based satellites and the improvement of satellite spatial and temporal resolution, the amount of satellite remote sensing data has exploded. Its application scope in earthquake prevention and disaster reduction, environmental monitoring, urban planning, and national security is becoming wider and wider. However, the traditional data application model hinders the information mining of satellite remote sensing data. The traditional application model requires a series of steps such as "command generation - command uploading - satellite imaging - data downloading - image processing - information extraction - data distribution", which has low timeliness and high bandwidth consumption.

[0003] The new model of "user demand issuance - mission planning - command uploading - on-orbit processing - real-time transmission" will significantly improve the timeliness of satellite data applications. Among these, on-orbit processing is the most critical link. Existing AI-based intelligent processing algorithms can improve the intelligence of information extraction. However, the number of parameters in intelligent algorithms is huge, and updating model parameters between satellite and ground is costly. Therefore, a low-rank adaptation-based on-orbit processing parameter update method is adopted to reduce the number of updated parameters, thereby facilitating the large-scale application of on-orbit processing. Summary of the Invention

[0004] The purpose of this invention is to improve the efficiency of satellite-to-ground model updates by reducing the number of update parameters and bandwidth usage while ensuring model performance remains unchanged. This invention proposes a low-rank adaptation-based on-board processing parameter update method.

[0005] The technical solution adopted in this invention is as follows:

[0006] A method for updating on-board processing parameters based on low-rank adaptation includes the following steps:

[0007] Step 1: Construct a ground intelligent information extraction module in the ground processing center, including a target detection network model, a ground feature classification network model, a change detection network model, and a scene description network model based on convolutional neural networks;

[0008] Step 2: Extract all convolutional feature layers of each network model in the ground intelligent information extraction module, and calculate all activation maps corresponding to each convolutional feature layer;

[0009] Step 3: Select a set number of convolutional feature layers with the highest contribution in each network model based on the activation map, and implant a low-rank adaptation matrix into them;

[0010] Step 4: Initialize the parameters of the implanted low-rank adaptation matrix, deploy the ground intelligent information extraction module and network model configuration file to the onboard processor to become the onboard intelligent information extraction module, and launch it into the predetermined orbit with the satellite.

[0011] Step 5: After the satellite acquires data in orbit, it transmits the raw data and the information extracted by the on-board intelligent information extraction module to the ground processing center. The ground processing center verifies the information extracted by the on-board intelligent system and fine-tunes the parameters of the ground intelligent information extraction module based on the verification results.

[0012] Step 6: Upload the low-rank adaptation matrix weights and network model configuration file of the ground-based intelligent information extraction module after fine-tuning the training to the onboard processor via the satellite-to-ground link to update the parameters of the onboard intelligent information extraction module.

[0013] Furthermore, in step 1, the target detection network model adopts the YOLOv4-tiny model, the land cover classification network model adopts the DeepLab-v3 model, the change detection network model adopts the UCDFormer model, and the scene description network model adopts the ClipCap-Chinese model.

[0014] Furthermore, the activation map calculation method in step 2 is as follows: the values ​​of each channel in the convolutional feature layer are weighted and summed in the channel dimension through linear fusion, and then normalized and interpolated to obtain the activation map. The value range of the activation map is scaled to the [0,1] interval, and the size is scaled to be the same as the input image size of the intelligent information extraction module.

[0015] Furthermore, the contribution calculation method in step 3 is as follows: calculate the structural similarity between the activation map and the input image; the higher the similarity, the higher the contribution.

[0016] Furthermore, the low-rank adaptation matrix implantation method in step 3 is as follows: Let the parameters of the convolutional feature layer be W, where W is an m×n dimensional matrix, and set low-rank adaptation matrices A and B, where A is an m×d dimensional matrix and B is a d×n dimensional matrix, where d < m and d < n. The implantation method is to replace W with A×B.

[0017] Furthermore, the initialization method described in step 4 is as follows: the low-rank adaptation matrix A is initialized to a matrix of all 1s, and the low-rank adaptation matrix B is initialized to a matrix of all 0s.

[0018] Furthermore, the network model configuration file mentioned in step 4 includes the network model structure, the loading address of the weights, and the loading method.

[0019] Furthermore, the verification method described in step 5 is as follows: re-label the erroneous results in the information extracted by the satellite intelligence, freeze other parameters during fine-tuning, and only train the implanted low-rank adaptation matrix.

[0020] The advantages of this invention over the prior art are:

[0021] (1) The present invention can reduce the number of update parameters of the satellite-ground model while ensuring that the model performance remains unchanged;

[0022] (2) The present invention can significantly reduce the communication bandwidth usage for updating on-board processing parameters;

[0023] (3) The present invention can improve the timeliness of remote sensing information processing applications. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating the overall technical process of the method proposed in this invention.

[0025] Figure 2 This is a schematic diagram of low-rank adaptation for an object detection network model. Detailed Implementation

[0026] The specific embodiments and basic principles of the present invention will be further explained below with reference to the accompanying drawings.

[0027] This invention proposes an on-board processing parameter update method based on low-rank adaptation, the specific flowchart of which is shown below. Figure 1 As shown, the specific process includes the following:

[0028] Step 1: Construct a ground-based intelligent information extraction module at the ground processing center. The intelligent information extraction module includes a target detection network model, a land cover classification network model, a change detection network model, and a scene description network model, all based on convolutional neural networks. The target detection network model uses the YOLOv4-tiny model, the land cover classification network model uses the DeepLab-v3 model, the change detection network model uses the UCDFormer model, and the scene description network model uses the ClipCap-Chinese model.

[0029] Step 2, Calculate activation maps: Extract all convolutional feature layers of each network model in the intelligent information extraction module, calculate all activation maps corresponding to each convolutional feature layer, and sum the values ​​of each channel in the convolutional feature layer in a linear fusion manner, weighted in the channel dimension, and then normalize and interpolate to obtain the activation map, so that the value range of the activation map is scaled to the [0,1] interval, and the size is scaled to be the same as the input image size of the intelligent information extraction module;

[0030] Step 3, Implanting a Low-Rank Adaptation Matrix: Calculate the structural similarity between the activation map and the input image. The higher the similarity, the higher the contribution. Based on the activation map, select a set number (10 in this example) of the highest-contributing convolutional feature layers in each network model, and implant a low-rank adaptation matrix into them, such as... Figure 2 Taking the target detection network model in the intelligent information extraction module as an example, let the parameters of the convolutional feature layer be W, where W is an m×n dimensional matrix, and set low-rank adaptation matrices A and B, where A is an m×d dimensional matrix and B is a d×n dimensional matrix, where d < m and d < n. The implantation method is to replace W with A×B.

[0031] Step 4, Deploy the on-board intelligent information extraction module: Initialize the parameters of the implanted low-rank adaptation matrix, initialize A as an all-1 matrix and B as an all-0 matrix, deploy the ground intelligent information extraction module and network model configuration file to the on-board processor. The network model configuration file includes the network model structure, the loading address of the weights and the loading method, and becomes the on-board intelligent information extraction module, which is launched into the predetermined orbit with the satellite.

[0032] Step 5, fine-tuning of ground intelligent information extraction module parameters: After the satellite collects data in orbit, it transmits its raw data and the information extracted by the satellite intelligent information extraction module to the ground processing center. The ground processing center verifies the information extracted by the satellite intelligent system, re-labels the erroneous and inaccurate results in the information extracted by the satellite intelligent system, and fine-tunes the parameters of the ground intelligent information extraction module based on the verification results. During the fine-tuning training, other parameters are frozen and only the implanted low-rank adaptation matrix is ​​trained.

[0033] Step 6, Model Parameter Update: Upload the fine-tuned low-rank adaptation matrix weights and network model configuration file to the onboard processor via the satellite-to-ground link to update the parameters of the onboard intelligent information extraction module.

Claims

1. A method for on-board processing parameter update based on low-rank adaptation, characterized in that, The method comprises the following steps: Step 1: constructing a ground intelligent information extraction module in a ground processing center, including a target detection network model, a ground feature classification network model, a change detection network model and a scene description network model based on a convolutional neural network; Step 2: extracting all convolutional feature layers of each network model in the ground intelligent information extraction module, and calculating all activation maps corresponding to the convolutional feature layers; Step 3: selecting the highest contribution degree of a set number of convolutional feature layers in each network model according to the activation map, and implanting a low-rank adaptive matrix therein; wherein the contribution degree calculation method is: calculating the structural similarity between the activation map and the input image, and the higher the similarity, the higher the contribution degree; Step 4: initializing the implanted low-rank adaptive matrix parameters, deploying the ground intelligent information extraction module and the network model configuration file to the on-board processor to become an on-board intelligent information extraction module, and launching into the predetermined orbit with the satellite; Step 5: after the satellite on-orbit data acquisition, the original data and the information extracted by the on-board intelligent information extraction module are downloaded to the ground processing center, the ground processing center verifies the information extracted by the on-board intelligent extraction, and adjusts and trains the parameters of the ground intelligent information extraction module according to the verification result; Step 6: uploading the low-rank adaptive matrix weight and the network model configuration file of the ground intelligent information extraction module after fine-tuning to the on-board processor through the satellite-ground link, and updating the parameters of the on-board intelligent information extraction module.

2. The method according to claim 1, wherein, The target detection network model in step 1 adopts a YOLOv4-tiny model, the ground feature classification network model adopts a DeepLab-v3 model, the change detection network model adopts a UCDFormer model, and the scene description network model adopts a ClipCap-Chinese model.

3. The method of claim 1, wherein, The activation map calculation method in step 2 is: the numerical value of each channel in the convolutional feature layer is weighted and summed in the channel dimension through linear fusion, and then normalized and interpolated to obtain the activation map, and the activation map value range is scaled to the [0, 1] interval, and the size is scaled to the same size as the input image of the intelligent information extraction module.

4. The method of claim 1, wherein, The implantation method of the low-rank adaptive matrix in step 3 is: setting the convolutional feature layer parameter as W, W is an m*n matrix, setting the low-rank adaptive matrix A and B, A is an m*d matrix, B is a d*n matrix, where d<m and d<n, and the implantation method is to replace W with A*B.

5. The method of claim 4, wherein, The initialization method in step 4 is: initializing the low-rank adaptive matrix A as a full 1 matrix, and initializing the low-rank adaptive matrix B as a full 0 matrix.

6. The method of claim 1, wherein, The network model configuration file in step 4 includes the network model structure, the loading address and the loading method of the weight.

7. The method of claim 1, wherein, The verification method in step 5 is: re-labeling the incorrect results in the information extracted by the on-board intelligent extraction, freezing other parameters during fine-tuning, and only training the implanted low-rank adaptive matrix.

Images