A prompt learning and SAR-assisted autonomous calibration method for optical image extrinsic parameters

Abstract

The present invention relates to a prompt learning and SAR data assisted optical image external parameter autonomous calibration method, which achieves getting rid of ground control point dependence and fusing multi-source information compared with the prior art, and constructing an optical image external parameter compensation model. The present invention includes the following steps: preparation of the prompt learning and SAR assisted optical image external parameter autonomous calibration method; SAR satellite image conversion of pseudo optical image; construction of optical image external parameter autonomous calibration model based on prompt learning; training of optical image external parameter autonomous calibration model based on prompt learning; application of optical image external parameter autonomous calibration model. By introducing SAR images and prompt learning technology, the present invention has particularly outstanding performance in extrinsic parameter calibration in areas that cannot be covered by traditional calibration methods, such as polar regions, deserts and mountain forests.

Description

Technical Field

[0001] The invention belongs to the field of ground preprocessing applications of optical remote sensing images, and specifically is a prompt learning and SAR-assisted optical image external parameter autonomous calibration method. Background Art

[0002] The geometric accuracy of optical images is crucial in many remote sensing applications, especially in the fields of geographic information systems, disaster monitoring, urban planning, and environmental assessment. These applications require images to have high-precision geolocation capabilities, which usually rely on accurate image extrinsic parameters, such as camera attitude and position parameters. However, due to the influence of multiple complex factors during the on-orbit imaging process, including satellite motion errors, atmospheric refraction, and camera installation offset, the extrinsic parameters of traditional optical images often have errors, resulting in a decrease in the geometric positioning accuracy of the image. Traditional extrinsic parameter calibration methods mainly rely on the assistance of ground control points (GCPs), which has significant limitations in areas with complex geographical locations and scarce ground information. For example, in terrains such as deserts, oceans, or mountains, it is very difficult and costly to obtain enough ground control points. In addition, the uneven distribution of GCPs may also lead to low calibration accuracy in local areas. Therefore, developing an optical image extrinsic parameter calibration method that does not rely on GCPs has become a key research direction for improving the positioning accuracy of remote sensing images.

[0003] As a representative of active remote sensing technology, synthetic aperture radar (SAR) can collect high-precision images under various weather and lighting conditions. SAR images have advantages in geometric positioning because they directly measure the distance and reflection characteristics of objects by transmitting and receiving electromagnetic waves, and are less affected by weather, atmosphere and lighting conditions. At the same time, SAR images usually have good terrain penetration and object shape capture capabilities, and can provide accurate geometric structure information. With the help of geometric features in SAR images, they can play an important role in multi-sensor data fusion. By combining SAR images with optical images, the problem of insufficient geometric accuracy of optical images under certain complex conditions can be compensated. However, due to the differences in the characteristics of SAR images and optical images, such as different reflection mechanisms and imaging principles, the direct fusion of the two still has certain technical challenges. This requires the use of specific algorithms or models to achieve feature intercommunication and similarity conversion between SAR images and optical images.

[0004] As an emerging deep learning method, prompt learning has the characteristics of efficient use of multimodal data and prompt information. This technology has made breakthrough progress in the field of natural language processing, but its application in the field of remote sensing is still in its infancy. Through prompt learning, the multi-source information of remote sensing images, such as location information, imaging conditions, timestamps, etc., can be quantified and encoded, and input into the deep learning model, thereby improving the model's adaptability to complex environments. In the present invention, prompt learning technology is introduced into the process of optical image external parameter calibration, and the pseudo-optical image generated by SAR image conversion is innovatively used as auxiliary information. This method not only fully exploits the geometric features of SAR images, but also enables the model to more accurately identify and correct the external parameter errors of optical images through the guidance of prompt information. This calibration method based on prompt learning and SAR assistance provides a new technical path for autonomous calibration of remote sensing images. Summary of the invention

[0005] The purpose of the present invention is to solve the problems in the prior art of traditional geometric calibration methods that rely on control points and have difficulty in simultaneously processing the synergy between optical images and other non-image data, and to provide a prompt learning and SAR-assisted autonomous calibration method for optical image extrinsic parameters to solve the above problems.

[0006] In order to achieve the above object, the technical solution of the present invention is as follows:

[0007] A method for autonomous calibration of optical image extrinsic parameters with prompt learning and SAR assistance, comprising the following steps:

[0008] 11) Prompt learning and SAR-assisted autonomous calibration of optical image extrinsic parameters Preparation: Collect multi-polarization synthetic aperture radar SAR images under different geographical locations, terrains and imaging conditions and the corresponding optical image , while annotating the optical image Location information , Satellite in-orbit time , terrain information covered by the image , Sun altitude angle , solar azimuth , Imaging attitude angle , and the external parameters of the initial installation matrix of the camera on the satellite body Metadata;

[0009] 12) SAR satellite images Converting Pseudo-Optical Images :First, the collected multi-polarization SAR images with accurate positioning are preprocessed, and the shape, texture, boundary geometric structure characteristics, multi-polarization characteristics and spatial distribution law information of the objects are extracted from the preprocessed SAR images using feature extraction algorithms. This information is input into the diffusion model to generate a pixel value distribution close to the real optical image;

[0010] 13) Autonomous calibration model of optical image extrinsic parameters based on prompt learning Construction: Structural processing of the collected prompt information data, and construction of an optical image external parameter autonomous calibration model based on prompt learning The model includes feature extraction module, prompt learning module, fusion module and output module. The main loss is designed in the model training stage. And auxiliary loss function ;

[0011] 14) Autonomous calibration model of optical image extrinsic parameters based on prompt learning Training: The labeled optical image and pseudo-optical image pairs, as well as the prompt information corresponding to the optical image, are simultaneously input into the model. 80 rounds of iterations are used, and the compensation effect is evaluated in the validation set in each round. In addition, the prompt information is dynamically adjusted during the training process to simulate the situation where the prompt information is missing.

[0012] 15) Autonomous calibration model of optical image external parameters Application: Acquisition of optical images to be calibrated externally And its corresponding prompt learning information data, input In the model, we get Corresponding external parameter error compensation parameters , and Compensation to In the rigorous geometric positioning model constructed.

[0013] SAR satellite imagery Converting Pseudo-Optical Images The following steps are involved:

[0014] 21) SAR satellite images Preprocessing is first performed by using high-precision orbit parameters and ground control points, combined with the rational function RPC positioning model and AW3D30 digital elevation DEM data to perform strict geometric correction on the SAR image; then the amplitude information of the SAR image is converted into backscattering coefficients, and the non-local mean filter algorithm is used for noise reduction; finally, scattering decomposition analysis is performed on the multi-polarization channel to extract surface scattering, volume scattering and volume reflection components;

[0015] 22) Edge information extraction: First, the Canny edge detection algorithm is used to extract the boundary information of the object, and the boundary connectivity is optimized by combining morphological dilation and erosion operations. For complex objects, the active contour model can also be used for dynamic boundary tracking; secondly, the gray-level co-occurrence matrix is ​​used to calculate the texture features of the SAR image, including contrast, correlation, energy, and entropy, to analyze the surface roughness and texture pattern of the object; and wavelet transform processing is used to extract the optical image Texture change information at 0.5x, 1x, and 2x resolution scales; finally, based on the Cloude polarization decomposition method, feature images reflecting the type of ground objects and the characteristics of vegetation coverage are generated;

[0016] 23) Generate pseudo-optical images based on diffusion model :The input layer of the diffusion model receives the SAR image feature vector, and the feature dimension is determined according to the preprocessing output; the diffusion process of the diffusion model uses the diffusion probability model to gradually generate pseudo-optical images, and each iteration adjusts the pixel value distribution to approximate the real optical image; the output layer of the diffusion model generates a pseudo-optical image that is similar to the real optical image in pixel distribution; when training the diffusion model, the collected real optical image and SAR image pairs are used as supervision signals, and the model parameters are optimized through adversarial loss and L2 error. Finally, the pre-processed SAR image is input into the model using the pre-trained diffusion model to obtain the pseudo-optical image.

[0017] The autonomous calibration model of optical image extrinsic parameters based on prompt learning The build consists of the following steps:

[0018] 31) Prompt multi-level encoding and modeling of learning information. The specific steps are:

[0019] 311) Position information encoding, using geographic coordinate standardization method to convert the latitude and longitude information of the image into a regularized vector form;

[0020] 312) Time information is encoded as a timestamp and processed in conjunction with sine and cosine transforms to handle periodic changes;

[0021] 313) Perform maximum and minimum normalization processing on the solar altitude angle and azimuth angle data to form a characteristic input vector;

[0022] 314) Terrain information is generated through SRTM DEM interpolation Elevation data within the image range;

[0023] 32) Autonomous calibration model of optical image external parameters Design, including optimization of the prompt information embedding layer, prompt information, , The attention mechanism between data is enhanced, and the weighted feature fusion strategy is adopted. The specific steps are as follows:

[0024] 321) , Input into the feature extraction module to generate the image feature embedding vector. The specific structure of the feature extraction module is as follows;

[0025] 3211) will , The input is sent to the first convolution layer with a kernel size of 7×7 and a stride of 2, and then a ReLU activation function is used to get the output.

[0026] 3212) The output in 3211) is passed through four consecutive residual modules, each of which is composed of two layers of convolution modules, each of which is composed of a convolution layer with a convolution kernel size of 3×3, a batch normalization layer and a ReLU activation function; after each residual module, a convolution operation with a stride of 2 is used for dimensionality reduction;

[0027] 3213) Use global average pooling to pool the feature maps output by four consecutive residual modules and obtain a fixed-length image feature vector ;

[0028] 322) Input the prompt information into the prompt learning module for multi-head embedding processing, map various types of information into a high-dimensional feature space, and generate an embedding vector containing multimodal features; the specific structure of the prompt learning module is:

[0029] 3221) The embedding vector of each modality is mapped to a unified high-dimensional feature space through a linear mapping layer to obtain a combined single-modal feature vector ,in The shape is , where N represents the number of modes and D represents the dimension size:

[0030] 3222) and a shape of The information of all modalities is integrated by concatenating the vectors of the modalities; the information between modalities interacts through the cross-modal attention mechanism to dynamically adjust the weights of different prompt information;

[0031] 3223) Generate query matrix, key matrix and value matrix through linear transformation, calculate self-attention weight for each modality separately, and capture the correlation between modalities; the information between modalities interacts through the multi-head attention mechanism, and dynamically adjusts the weight of different prompt information;

[0032] 3224) The last output vector The vector is extracted as the multimodal prompt feature vector ;

[0033] 323) Image feature vector and the multimodal cue feature vector Input into the fusion module; its specific structure is as follows:

[0034] 3231) Input is image feature vector and the multimodal cue feature vector ;right Perform linear mapping to generate a query matrix ;right Perform linear mapping to generate a key matrix Sum Matrix ; as shown in the following formula:

[0035] ,

[0036] in , , is a learnable weight matrix;

[0037] 3232) Using the query matrix and key matrix Calculate the attention weights:

[0038] ,

[0039] in Represents the image feature vector and the multimodal cue feature vector The interaction weight between

[0040] 3233) The interaction weight Matrix of values ​​for the multimodal cue feature vector , generate interactive fusion features ;

[0041] 324) The features of the prompt information and the image features are fused through the Self-attention Weighted Fusion strategy;

[0042] 3241) Image feature vector , multimodal cue feature vector and interactive fusion features Splice to form joint features ;

[0043] 3242) Using a fully connected network Process and generate fusion weights :

[0044] ,

[0045] in and is a learnable parameter, is the Sigmoid activation function;

[0046] 3243) Based on dynamically generated weights , perform weighted fusion and output the final fusion feature :

[0047] ;

[0048] 33) Fusion Features The extrinsic parameter error compensation vector is predicted through a fully connected network, and the output is dynamically coupled with the geometric positioning model to achieve extrinsic parameter correction.

[0049] The autonomous calibration model of optical image extrinsic parameters based on prompt learning The training consists of the following steps:

[0050] 41) The optical image features are input into the feature extraction module 321) to obtain the image feature vector ; Input the prompt information into the prompt learning module of 322) to obtain the multimodal prompt feature vector ; Then the image feature vector and the multimodal cue feature vector Input to the fusion module 323) to obtain interactive fusion features ; Then the image feature vector , multimodal cue feature vector and interactive fusion features The fusion feature is obtained through the Self-attention Weighted Fusion weighted fusion strategy of 324) ;

[0051] 42) Fusion Features The extrinsic parameter error compensation vector is predicted through a fully connected network. The compensation vector represents the offset of the corrected extrinsic parameter. The compensation vector output by the model is dynamically coupled with the geometric positioning model to achieve accurate correction of the extrinsic parameters of the optical image.

[0052] 43) During the model training process, a multi-objective loss function is used to optimize the error compensation accuracy and geometric consistency, including:

[0053] 431) Main target loss Measures the mean square error loss between the predicted external parameter error compensation vector and the true error;

[0054] ,

[0055] in represents the number of training samples, represents the extrinsic parameter error compensation vector predicted by the model, represents the true external parameter error, Represents the calculation of L2 norm;

[0056] 432) Auxiliary target loss The structural similarity loss is used to measure the geometric consistency and visual similarity between the rectified image and the geographical real image. The formula is:

[0057] ,

[0058] Where SSIM stands for structural similarity index;

[0059] 44) The model training adopts an iterative optimization method. In each round, the compensation effect is evaluated in the validation set, and the model learning rate and weight initialization strategy hyperparameters are adjusted. In addition, in order to improve the robustness of the model, the prompt information is dynamically adjusted during the training process to simulate the possible missing or disturbed scenarios of the prompt information.

[0060] Beneficial Effects

[0061] The present invention realizes efficient correction of optical image external parameter errors by constructing a prompt learning model and combining the geometric features provided by SAR images. This method significantly improves the uncontrolled positioning accuracy of optical images, especially in areas that traditional calibration methods cannot cover, such as polar regions, deserts and high mountain areas. Traditional calibration methods are highly dependent on the distribution and quality of ground control points, while the present invention gets rid of the dependence on ground control points by introducing SAR images and prompt learning technology. The present invention is widely adapted to optical images in different geographical locations, terrains and imaging conditions, and has extremely high robustness. The introduction of prompt learning technology enables the present invention to efficiently process and fuse multi-source information. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 It is a sequence diagram of the method of the present invention;

[0063] Figure 2 A schematic diagram of an optical image extrinsic parameter autonomous calibration model based on prompt learning according to the present invention;

[0064] Figure 3 It is a schematic diagram of a fusion module in the optical image extrinsic parameter autonomous calibration model based on prompt learning involved in the present invention;

[0065] Figure 4It is a schematic diagram of a weighted fusion strategy in the optical image extrinsic parameter autonomous calibration model based on prompt learning involved in the present invention; DETAILED DESCRIPTION

[0066] In order to have a further understanding and recognition of the structural features and the effects achieved by the present invention, a preferred embodiment and accompanying drawings are used for detailed description as follows:

[0067] like Figure 1 As shown, the present invention provides a prompt learning and SAR-assisted optical image extrinsic parameter autonomous calibration method, comprising the following steps:

[0068] The first step is to prepare the method for autonomous calibration of optical image extrinsic parameters assisted by prompt learning and SAR: collect multi-polarization synthetic aperture radar SAR images under different geographical locations, terrains and imaging conditions. and the corresponding optical image , while annotating the optical image Location information , Satellite in-orbit time , terrain information covered by the image , Sun altitude angle , solar azimuth , Imaging attitude angle , and the external parameters of the initial installation matrix of the camera on the satellite body Metadata.

[0069] Step 2: SAR satellite imagery Converting Pseudo-Optical Images :First, preprocess the collected multi-polarization SAR images with accurate positioning, and use the feature extraction algorithm to extract the shape, texture, boundary geometric structure characteristics, multi-polarization characteristics and spatial distribution law information of the objects from the preprocessed SAR images, and input this information into the diffusion model to generate a pixel value distribution close to the real optical image. The specific steps are as follows:

[0070] (1) SAR satellite images Preprocessing is first performed by using high-precision orbit parameters and ground control points, combined with the rational function RPC positioning model and AW3D30 digital elevation DEM data to perform strict geometric correction on the SAR image; then the amplitude information of the SAR image is converted into backscattering coefficients, and the non-local mean filter algorithm is used for noise reduction; finally, scattering decomposition analysis is performed on the multi-polarization channel to extract surface scattering, volume scattering and volume reflection components;

[0071] (2) Edge information extraction: First, the Canny edge detection algorithm is used to extract the boundary information of the object, and the boundary connectivity is optimized by combining morphological dilation and erosion operations. For complex objects, the active contour model can also be used for dynamic boundary tracking. Secondly, the gray-level co-occurrence matrix is ​​used to calculate the texture features of the SAR image, including contrast, correlation, energy, and entropy, to analyze the surface roughness and texture pattern of the object. Wavelet transform is then used to extract the optical image. Texture change information at 0.5x, 1x, and 2x resolution scales; finally, based on the Cloude polarization decomposition method, feature images reflecting the type of ground objects and the characteristics of vegetation coverage are generated;

[0072] (3) Generate pseudo-optical images based on diffusion model :The input layer of the diffusion model receives the SAR image feature vector, and the feature dimension is determined according to the preprocessing output; the diffusion process of the diffusion model uses the diffusion probability model to gradually generate pseudo-optical images, and each iteration adjusts the pixel value distribution to approximate the real optical image; the output layer of the diffusion model generates a pseudo-optical image that is similar to the real optical image in pixel distribution; when training the diffusion model, the collected real optical image and SAR image pairs are used as supervision signals, and the model parameters are optimized through adversarial loss and L2 error. Finally, the pre-processed SAR image is input into the model using the pre-trained diffusion model to obtain the pseudo-optical image.

[0073] The third step is Figure 2 , Figure 3 and Figure 4 As shown, the autonomous calibration model of optical image extrinsic parameters based on prompt learning Construction: Structural processing of the collected prompt information data, and construction of an optical image external parameter autonomous calibration model based on prompt learning The model includes feature extraction module, prompt learning module, fusion module and output module. The main loss is designed in the model training stage. And auxiliary loss function The specific steps are:

[0074] (1) Prompt multi-level encoding and modeling of learning information. The specific steps are as follows:

[0075] (1-1) Location information encoding: using geographic coordinate standardization method, the latitude and longitude information of the image is converted into a regularized vector form;

[0076] (1-2) Time information is encoded as a timestamp and processed using sine and cosine transforms to handle periodic changes;

[0077] (1-3) Perform maximum and minimum normalization processing on the solar altitude angle and azimuth angle data to form a feature input vector;

[0078] (1-4) Topographic information is generated through SRTM DEM interpolation Elevation data within the image range;

[0079] (2) Autonomous calibration model of optical image external parameters Design, including optimization of the prompt information embedding layer, prompt information, , The attention mechanism between data is enhanced, and the weighted feature fusion strategy is adopted. The specific steps are as follows:

[0080] (2-1) , Input into the feature extraction module to generate the image feature embedding vector. The specific structure of the feature extraction module is as follows: , The input is fed into the first convolutional layer with a kernel size of 7×7 and a stride of 2, followed by a ReLU activation function to get the output; the output is then passed through four consecutive residual modules, each of which consists of two layers of convolutional modules, each of which consists of a convolutional layer with a kernel size of 3×3, a batch normalization layer, and a ReLU activation function; after each residual module, a convolution operation with a stride of 2 is used for dimensionality reduction; the feature maps output by the four consecutive residual modules are pooled using global average pooling to obtain a fixed-length image feature vector ;

[0081] (2-2) Input the prompt information into the prompt learning module for multi-head embedding processing, map various types of information into a high-dimensional feature space, and generate an embedding vector containing multimodal features; the specific structure of the prompt learning module is: map the embedding vector of each modality into a unified high-dimensional feature space through a linear mapping layer to obtain a combined single-modal feature vector ,in The shape is , where N represents the number of modes and D represents the dimension size: and a shape of The information of all modalities is integrated by concatenating the vectors of the modalities; the information between the modalities interacts through the cross-modal attention mechanism, and the weights of different prompt information are dynamically adjusted; the query matrix, key matrix and value matrix are generated through linear transformation, and the self-attention weight is calculated for each modality to capture the correlation between the modalities; the information between the modalities interacts through the multi-head attention mechanism, and the weights of different prompt information are dynamically adjusted; the last one of the last output vector is The vector is extracted as the multimodal prompt feature vector ;

[0082] (2-3) The image feature vector and the multimodal cue feature vector Input to the fusion module; its specific structure is as follows: Input is the image feature vector and the multimodal cue feature vector ;right Perform linear mapping to generate a query matrix ;right Perform linear mapping to generate a key matrix Sum Matrix ; as shown in the following formula:

[0083] ,

[0084] in , , is a learnable weight matrix;

[0085] Using the query matrix and key matrix Calculate the attention weights:

[0086] ,

[0087] in Represents the image feature vector and the multimodal cue feature vector The interaction weight between Matrix of values ​​for the multimodal cue feature vector , generate interactive fusion features ;

[0088] (2-4) The features of the prompt information and the image features are fused through the Self-attention Weighted Fusion strategy; The features of the prompt information and the image features are fused through the Self-attention Weighted Fusion strategy; Use a fully connected network to Process and generate fusion weights : ,in and is a learnable parameter, is the Sigmoid activation function;

[0089] Based on dynamically generated weights , perform weighted fusion and output the final fusion feature :

[0090] ;

[0091] (3) Fusion features The extrinsic parameter error compensation vector is predicted through a fully connected network, and the output is dynamically coupled with the geometric positioning model to achieve extrinsic parameter correction.

[0092] Step 4: Autonomous calibration model of optical image external parameters based on prompt learning Training: The labeled optical image and pseudo-optical image pairs, as well as the prompt information corresponding to the optical image, are simultaneously input into the model. 80 rounds of iterations are used, and the compensation effect is evaluated in the validation set in each round. In addition, the prompt information is dynamically adjusted during the training process to simulate the situation where the prompt information is missing. The specific steps are as follows:

[0093] (1) The optical image features are input into the feature extraction module in the third step to obtain the image feature vector ; Input the prompt information into the prompt learning module in the third step to obtain the multimodal prompt feature vector ; Then the image feature vector and the multimodal cue feature vector Input into the fusion module in the third step to obtain interactive fusion features ; Then the image feature vector , multimodal cue feature vector and interactive fusion features The fusion features are obtained through the Self-attention Weighted Fusion weighted fusion strategy in the third step ;

[0094] (2) Fusion features The extrinsic parameter error compensation vector is predicted through a fully connected network. The compensation vector represents the offset of the corrected extrinsic parameter. The compensation vector output by the model is dynamically coupled with the geometric positioning model to achieve accurate correction of the extrinsic parameters of the optical image.

[0095] (3) During the model training process, a multi-objective loss function is used to optimize the error compensation accuracy and geometric consistency, including:

[0096] (3-1) Main target loss Measures the mean square error loss between the predicted external parameter error compensation vector and the true error;

[0097] ,

[0098] in represents the number of training samples, represents the extrinsic parameter error compensation vector predicted by the model, represents the true external parameter error, Represents the calculation of L2 norm;

[0099] (3-2) Auxiliary target loss The structural similarity loss is used to measure the geometric consistency and visual similarity between the rectified image and the geographical real image. The formula is:

[0100] ,

[0101] Where SSIM stands for structural similarity index;

[0102] (4) The model training adopts an iterative optimization method. In each round, the compensation effect is evaluated in the validation set, and the model learning rate and weight initialization strategy hyperparameters are adjusted. In addition, in order to improve the robustness of the model, the prompt information is dynamically adjusted during the training process to simulate the possible missing or disturbed scenarios of the prompt information.

[0103] Step 5: Autonomous calibration model of optical image external parameters Application: Acquisition of optical images to be calibrated externally And its corresponding prompt learning information data, input In the model, we get Corresponding external parameter error compensation parameters , and Compensation to In the rigorous geometric positioning model constructed.

[0104] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions only describe the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.

Claims

1. A method for autonomous calibration of optical image extrinsic parameters with prompt learning and SAR assistance, characterized in that: The following steps are involved: 11) Prompt learning and SAR-assisted autonomous calibration of optical image extrinsic parameters Preparation: Collect multi-polarization synthetic aperture radar SAR images under different geographical locations, terrains and imaging conditions and the corresponding optical image , while annotating the optical image Location information , Satellite in-orbit time , terrain information covered by the image , Sun altitude angle , solar azimuth , Imaging attitude angle , and the external parameters of the initial installation matrix of the camera on the satellite body Metadata; 12) SAR satellite images Converting Pseudo-Optical Images :First, the collected multi-polarization SAR images with accurate positioning are preprocessed, and the shape, texture, boundary geometric structure characteristics, multi-polarization characteristics and spatial distribution law information of the objects are extracted from the preprocessed SAR images using feature extraction algorithms. This information is input into the diffusion model to generate a pixel value distribution close to the real optical image; 13) Autonomous calibration model of optical image extrinsic parameters based on prompt learning Construction: Structural processing of the collected prompt information data, and construction of an optical image external parameter autonomous calibration model based on prompt learning The model includes feature extraction module, prompt learning module, fusion module and output module. The main loss is designed in the model training stage. And auxiliary loss function ; 14) Autonomous calibration model of optical image extrinsic parameters based on prompt learning Training: The labeled optical image and pseudo-optical image pairs, as well as the prompt information corresponding to the optical image, are simultaneously input into the model. 80 rounds of iterations are used, and the compensation effect is evaluated in the validation set in each round. In addition, the prompt information is dynamically adjusted during the training process to simulate the situation where the prompt information is missing. 15) Autonomous calibration model of optical image external parameters Application: Acquisition of optical images to be calibrated externally And its corresponding prompt learning information data, input In the model, we get Corresponding external parameter error compensation parameters , and Compensation to In the rigorous geometric positioning model constructed.

2. The method for autonomous calibration of optical image extrinsic parameters using prompt learning and SAR assistance according to claim 1, characterized in that: SAR satellite imagery Converting Pseudo-Optical Images The following steps are involved: 21) SAR satellite images Preprocessing is first performed by using high-precision orbit parameters and ground control points, combined with the rational function RPC positioning model and AW3D30 digital elevation DEM data to perform strict geometric correction on the SAR image; then the amplitude information of the SAR image is converted into backscattering coefficients, and the non-local mean filter algorithm is used for noise reduction; finally, scattering decomposition analysis is performed on the multi-polarization channel to extract surface scattering, volume scattering and volume reflection components; 22) Edge information extraction: First, the Canny edge detection algorithm is used to extract the boundary information of the object, and the boundary connectivity is optimized by combining morphological dilation and erosion operations. For complex objects, the active contour model can also be used for dynamic boundary tracking; secondly, the gray-level co-occurrence matrix is ​​used to calculate the texture features of the SAR image, including four indicators: contrast, correlation, energy and entropy, to analyze the surface roughness and texture pattern of the object; and wavelet transform is used to extract the optical image Texture change information at 0.5x, 1x, and 2x resolution scales; finally, based on the Cloude polarization decomposition method, feature images reflecting the type of ground objects and the characteristics of vegetation coverage are generated; 23) Generate pseudo-optical images based on diffusion model :The input layer of the diffusion model receives the SAR image feature vector, and the feature dimension is determined according to the preprocessing output; the diffusion process of the diffusion model uses the diffusion probability model to gradually generate pseudo-optical images, and each iteration adjusts the pixel value distribution to approximate the real optical image; the output layer of the diffusion model generates a pseudo-optical image that is similar to the real optical image in pixel distribution; when training the diffusion model, the collected real optical image and SAR image pairs are used as supervision signals, and the model parameters are optimized through adversarial loss and L2 error. Finally, the pre-processed SAR image is input into the model using the pre-trained diffusion model to obtain the pseudo-optical image.

3. The method for autonomous calibration of optical image extrinsic parameters using prompt learning and SAR assistance according to claim 1, characterized in that: The autonomous calibration model of optical image extrinsic parameters based on prompt learning The build consists of the following steps: 31) Prompt multi-level encoding and modeling of learning information. The specific steps are: 311) Position information encoding, using geographic coordinate standardization method to convert the latitude and longitude information of the image into a regularized vector form; 312) Time information is encoded as a timestamp and processed in conjunction with sine and cosine transforms to handle periodic changes; 313) Perform maximum and minimum normalization processing on the solar altitude angle and azimuth angle data to form a characteristic input vector; 314) Terrain information is generated through SRTM DEM interpolation Elevation data within the image range; 32) Autonomous calibration model of optical image external parameters Design, including optimization of the prompt information embedding layer, prompt information, , The attention mechanism between data is enhanced, and the weighted feature fusion strategy is adopted. The specific steps are as follows: 321) , Input into the feature extraction module to generate the image feature embedding vector. The specific operation of the feature extraction module is as follows: 3211) will , The input is sent to the first convolution layer with a kernel size of 7×7 and a stride of 2, and then a ReLU activation function is used to get the output. 3212) The output in 3211) is passed through four consecutive residual modules, each of which is composed of two layers of convolution modules, each of which is composed of a convolution layer with a convolution kernel size of 3×3, a batch normalization layer and a ReLU activation function; after each residual module, a convolution operation with a stride of 2 is used for dimensionality reduction; 3213) Use global average pooling to pool the feature maps output by four consecutive residual modules and obtain a fixed-length image feature vector ; 322) Input the prompt information into the prompt learning module for multi-head embedding processing, map various types of information into a high-dimensional feature space, and generate an embedding vector containing multimodal features; the specific structure of the prompt learning module is: 3221) The embedding vector of each modality is mapped to a unified high-dimensional feature space through a linear mapping layer to obtain a combined single-modal feature vector ,in The shape is , where N represents the number of modes and D represents the dimension size: 3222) and a shape of The information of all modalities is integrated by concatenating the vectors of the modalities; the information between modalities interacts through the cross-modal attention mechanism to dynamically adjust the weights of different prompt information; 3223) Generate query matrix, key matrix and value matrix through linear transformation, calculate self-attention weight for each modality separately, and capture the correlation between modalities; the information between modalities interacts through the multi-head attention mechanism, and dynamically adjusts the weight of different prompt information; 3224) Output the last The vector is extracted as the multimodal prompt feature vector ; 323) Image feature vector and the multimodal cue feature vector Input into the fusion module, the specific operations are as follows: 3231) Input is image feature vector and the multimodal cue feature vector ;right Perform linear mapping to generate a query matrix ;right Perform linear mapping to generate a key matrix Sum Matrix ; as shown in the following formula: , in , , is a learnable weight matrix; 3232) Using the query matrix and key matrix Calculate the attention weights: ， in Represents the image feature vector and the multimodal cue feature vector The interaction weight between them; 3233) The interaction weight Matrix of values ​​for multimodal cue feature vectors , generate interactive fusion features ; 324) The features of the prompt information and the image features are weighted fused through Self-attention Weighted Fusion: 3241) Image feature vector , multimodal cue feature vector and interactive fusion features Splice to form joint features ; 3242) Using a fully connected network Process and generate fusion weights : ， in and is a learnable parameter, is the Sigmoid activation function; 3243) Based on dynamically generated weights , perform weighted fusion and output the final fusion feature : ； 33) Fusion Features The extrinsic parameter error compensation vector is predicted through a fully connected network, and the output is dynamically coupled with the geometric positioning model to achieve extrinsic parameter correction.

4. The method for autonomous calibration of optical image extrinsic parameters using prompt learning and SAR assistance according to claim 1, characterized in that: The autonomous calibration model of optical image extrinsic parameters based on prompt learning The training consists of the following steps: 41) The optical image features are input into the feature extraction module 321) to obtain the image feature vector ; Input the prompt information into the prompt learning module of 322) to obtain the multimodal prompt feature vector ; Then the image feature vector and the multimodal cue feature vector Input to the fusion module 323) to obtain interactive fusion features ; Then the image feature vector , multimodal cue feature vector and interactive fusion features The fusion feature is obtained through the Self-attention Weighted Fusion weighted fusion strategy of 324) ; 42) Fusion Features The extrinsic parameter error compensation vector is predicted through a fully connected network. The compensation vector represents the offset of the corrected extrinsic parameter. The compensation vector output by the model is dynamically coupled with the geometric positioning model to achieve accurate correction of the extrinsic parameters of the optical image. 43) During the model training process, a multi-objective loss function is used to optimize the error compensation accuracy and geometric consistency, including: 431) Main target loss Measures the mean square error loss between the predicted external parameter error compensation vector and the true error; ， in represents the number of training samples, represents the extrinsic parameter error compensation vector predicted by the model, represents the true external parameter error, Represents the calculation of L2 norm; 432) Auxiliary target loss The structural similarity loss is used to measure the geometric consistency and visual similarity between the rectified image and the geographical real image. The formula is: ， Where SSIM stands for structural similarity index; 44) The model training adopts an iterative optimization method. In each round, the compensation effect is evaluated in the validation set, and the model learning rate and weight initialization strategy hyperparameters are adjusted. In addition, in order to improve the robustness of the model, the prompt information is dynamically adjusted during the training process to simulate the possible missing or disturbed scenarios of the prompt information.

Images