A text description method for full polarimetric synthetic aperture radar and visible light images

Abstract

The present invention discloses a method for textual description of fully polarimetric synthetic aperture radar (SAR) and visible light images. The method comprises the following steps: collecting fully polarimetric SAR images and visible light images; performing Lipschitz filtering to denoise the images; fusing multiple polarimetric channel features and inputting them into a visual transformer encoder for feature extraction; and performing multi-layer transformer feature extraction on the visible light images; performing weighted fusion on the multi-layer features of the visible light image and the SAR image; and inputting the fused multimodal features into a text-to-text transfer transformer model text encoder to generate a natural language description. This method leverages deep learning technology to fully leverage the complementary advantages of fully polarimetric SAR and visible light images, achieving dynamic fusion and semantic interaction of multimodal features. This significantly improves the accuracy and robustness of automatic textual description of remote sensing images and is suitable for analyzing and understanding multimodal remote sensing images in complex environments.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing and natural language generation, and in particular to a method for describing a full-polarization synthetic aperture radar and a visible light image in text. Background Art

[0002] With the rapid development of remote sensing technology and artificial intelligence, polarimetric synthetic aperture radar (SAR) and visible light imagery, two important remote sensing data sources, are playing a key role in surface monitoring, environmental surveillance, military reconnaissance, and other fields. Polarimetric SAR data, with its cloud-penetrating and all-weather capabilities, can provide information on the electromagnetic scattering characteristics of targets, while visible light imagery provides rich texture and color information, facilitating intuitive identification of surface features. The combination of these two technologies enables the complementary development of multimodal information, enhancing the understanding and analysis of remote sensing imagery.

[0003] Traditional multimodal remote sensing image fusion methods often rely on hand-crafted feature extraction and fusion strategies, which struggle to fully capture the complex semantic relationships between different modalities. They also suffer from low efficiency, limited fusion accuracy, and insufficient robustness when processing high-dimensional, large-scale data. Furthermore, significant differences between synthetic aperture radar and visible light imagery in terms of resolution, observation angle, and noise properties lead to widespread information loss, registration errors, and noise interference during the data fusion process, severely restricting the comprehensive application of multimodal remote sensing imagery.

[0004] In recent years, the development of deep learning technology has provided new solutions for the fusion and understanding of multimodal remote sensing data. The encoder-decoder architecture based on deep neural networks automatically learns multi-level, multi-scale feature representations to effectively mine the inherent connections and complementary information between different modal data. Furthermore, advanced technologies such as adaptive weighting mechanisms, attention mechanisms, and generative adversarial networks can dynamically adjust modal weights, compensate for information loss, and correct for registration errors, significantly improving the accuracy and robustness of fusion.

[0005] However, existing methods still have shortcomings in the fine-grained level of multimodal feature fusion and cross-modal information exchange, making it difficult to fully leverage the advantages of fully polarimetric synthetic aperture radar (SAR) and visible light imagery. In particular, they are limited in achieving accurate and natural image text descriptions. To address this challenge, a multimodal fusion text description method based on a dual encoder-decoder architecture is designed. This method integrates the deep semantic features of fully polarimetric SAR and visible light imagery. Leveraging advanced cross-modal self-attention mechanisms and text generation architectures, it will significantly advance the development and application of intelligent remote sensing image interpretation technology. Summary of the Invention

[0006] The purpose of this invention is to provide a method for textual description of fully polarimetric synthetic aperture radar (SAR) and visible light images. This method leverages deep learning technology to achieve efficient fusion and semantic understanding of multimodal data. This method addresses issues such as resolution discrepancies, registration errors, and noise interference between SAR and visible light images, improving the expressiveness of fused features and the accuracy and naturalness of textual descriptions. This method is adaptable to complex remote sensing environments, enabling intelligent semantic analysis and high-quality text generation for multimodal remote sensing imagery. It has broad applications in environmental monitoring, disaster assessment, target identification, and other fields.

[0007] To achieve the above object, the present invention provides a method for describing a full-polarization synthetic aperture radar and a visible light image in text, comprising the following steps:

[0008] S1. Collect visible light images and full-polarimetric synthetic aperture radar images of the same scene, where the full-polarimetric synthetic aperture radar image contains multiple polarization channels (such as HH, HV, VH, and VV).

[0009] S2. Performing Lipschitz filtering denoising on each polarization channel of the full-polarization synthetic aperture radar image to remove noise and speckles;

[0010] S3, fusing the denoised multiple polarization channel features to form a multi-channel fusion feature, and inputting it into the visual transformer encoder for multi-layer transformer feature extraction;

[0011] S4, using the contrastive language-image pre-trained model encoder to perform multi-layer transformer feature extraction on visible light images;

[0012] S5. Using a layered feature fusion method with adaptive weighting coefficients, weighted fusion is performed on the multi-layer transformer features of the visible light image and the synthetic aperture radar image, and the contribution weight of each layer of features is dynamically adjusted;

[0013] S6. Design a cross-modal self-attention mechanism to achieve deep bidirectional information interaction between visible light image fusion features and synthetic aperture radar image fusion features, further improving the expressive power of multimodal fusion features.

[0014] S7, inputting the fused multimodal features into a text encoder based on a text-to-text transfer transformer model architecture to generate a corresponding natural language text description;

[0015] S8. Combine the temperature regulation mechanism with the reinforcement learning strategy based on the reinforcement learning policy gradient algorithm to optimize the diversity and accuracy of the generated text and improve the final description quality.

[0016] Preferably, the step of performing Lipschitz filtering denoising on each polarization channel of the full polarization synthetic aperture radar image in S2 is:

[0017] S2.1-1. Collect full-polarization synthetic aperture radar image data for the same ground feature scene, including four polarization channels: horizontal transmission-horizontal reception HH, horizontal transmission-vertical reception HV, vertical transmission-horizontal reception VH, and vertical transmission-vertical reception VV. The original images of these four polarization channels are expressed as follows: , ;

[0018] Each polarization channel reflects the different scattering characteristics of the ground objects and is the basis for subsequent fusion analysis;

[0019] S2.1-2. For each polarization channel image, the Lipschitz filter algorithm is used to suppress noise and remove speckles. The Lipschitz filter is based on local statistical characteristics. By calculating the local mean and variance, the filter weight is dynamically adjusted to distinguish the signal and noise parts of the image. The specific filter calculation formula is as follows:

[0020] ;

[0021] in, Represents the original polarization channel pixel value to be processed; Indicated in pixels The mean value of pixels in a small window centered on , which reflects the average intensity of the local image; It represents the variance of pixels within the small window, reflecting the degree of change in local intensity; K is the noise variance estimate, which is usually obtained through statistical methods or prior knowledge.

[0022] The core idea of ​​this filter is to use local statistical information to determine the ratio of signal to noise in pixel values ​​and calculate a weight factor to balance the relationship between maintaining signal edges and suppressing noise, thereby effectively reducing the influence of speckle noise in synthetic aperture radar images.

[0023] S2.1-3. Perform Lipschitz filtering on each of the four polarization channels to obtain the denoised polarization channel images:

[0024] ;

[0025] Each denoised image has a high signal-to-noise ratio, retaining polarization-specific scattering information and image details, providing quality assurance for subsequent polarization channel fusion and high-level feature extraction.

[0026] Preferably, the step of fusing the denoised multiple polarization channel features in S3 specifically includes:

[0027] S3.1-1. After completing the Lipschitz filter denoising process on each polarization channel (HH, HV, VH, VV) of the full polarization synthetic aperture radar image, we obtained four high-quality denoised polarization features, which are expressed as , these four polarization channels reflect the electromagnetic wave scattering characteristics of the ground objects from different polarization angles, and have obvious complementary information. Processing these channel features alone is difficult to fully explore the intrinsic correlation between each polarization information, which limits the semantic understanding and target detection performance of subsequent remote sensing images. By introducing a learnable weight parameter Different importance weights are assigned to the four polarization channels. The weight parameters are automatically optimized through model training to adapt to different scenarios and task requirements. The weights meet the normalization constraints:

[0028] ;

[0029] S3.1-2. Concatenate the weighted four polarization channel features in the channel dimension to form a complete fusion feature representation:

[0030] ;

[0031] The dimension of the spliced ​​feature vector is four times that of a single-channel feature, preserving the fine-grained information of each polarization channel and enhancing the richness and expressiveness of the features. This weighted splicing and fusion method overcomes the information redundancy and feature conflict that may result from simple splicing or summation, fully preserving the diversity and complementarity of polarization channels. At the same time, it enhances the discriminability and robustness of the features through adaptive weight adjustment.

[0032] S3.1-3. The visual transformer encoder is mainly composed of multiple layers of transformer modules. Each layer of transformer abstracts and enhances the input features through a multi-head self-attention mechanism and a feedforward network. Assuming that the visual transformer encoder contains L layers of transformer modules, the output features of each layer are expressed as:

[0033] ;

[0034] in, Represents the polarimetric characteristics of synthetic aperture radar data after denoising and fusion.

[0035] The feature dimensions of each layer of transformer modules are kept consistent, which facilitates the transmission and fusion of information between subsequent layers. Multi-layer transformer modules capture the local and global context information of the image in turn, and gradually extract more advanced and more semantic visual features. The final multi-layer feature sequence is As a multi-scale and multi-level feature representation of synthetic aperture radar images.

[0036] Preferably, the step of performing multi-layer transformer feature extraction on the visible light image in S4 specifically includes:

[0037] S4.1-1. Collect a sequence of visible light images of the same scene. Assume that the original input data is represented as ,Since the pixel value range and resolution of visible light images may be different, ,before feature extraction, the input image is preprocessed, including normalization processing. ,The specific normalization formula is :

[0038] ;

[0039] in, and are the mean and standard deviation of visible light image pixels, respectively, used to unify the numerical scale between different images;

[0040] S4.1-2. Input the normalized visible light image into the pre-trained contrastive language-image pre-training model encoder for feature extraction. The contrastive language-image pre-training model encoder is mainly composed of multiple layers of transformer modules. Each layer of transformer abstracts and enhances the input features through a multi-head self-attention mechanism and a feedforward network. Assuming that the contrastive language-image pre-training model encoder contains L layers of transformer modules, the output features of each layer are expressed as:

[0041] ;

[0042] in, Represents the input image features after preliminary linear transformation and position encoding;

[0043] S4.1-3, the feature dimensions of each layer of transformer modules are kept consistent, which facilitates the transmission and fusion of information between subsequent layers. The multi-layer transformer modules capture the local and global context information of the image in turn, and gradually extract more advanced and more semantic visual features. The final multi-layer feature sequence is obtained As a multi-scale, multi-level feature representation of visible light images.

[0044] Preferably, the steps of the hierarchical feature fusion method using adaptive weighting coefficients in S5 specifically include:

[0045] S5.1-1, the visible light image encoder (using the contrastive language-image pre-training model architecture) and the full polarimetric synthetic aperture radar image encoder (using the visual transformer architecture) both contain L layers of transformer modules. The output features of each layer carry the layer's expression of the input image at different scales and semantic levels, forming a serialized feature representation:

[0046] ;

[0047] Where N and M are the lengths of visible light and synthetic aperture radar feature sequences, respectively, and d is the feature dimension;

[0048] S5.1-2. To overcome the drawback of uneven contribution of feature expressions at each layer to the final task, a dynamic weight is assigned to each layer of features. , the weights are learned through training, specifically by designing a multi-layer perceptron (MLP) network for each layer of features to calculate the weight scores:

[0049] ;

[0050] in, and They represent the weight scores of the kth layer in visible light and synthetic aperture radar modes respectively. To ensure the rationality and comparability of the weights, the softmax function is used to normalize the weight scores of all layers:

[0051] ;

[0052] In this way, the sum of the weights of all layers is 1, ensuring stability during fusion;

[0053] S5.1-3. Use normalized weights to perform weighted summation on the features of each layer to obtain the single-modal fusion feature:

[0054] ;

[0055] This fusion method comprehensively considers the contributions of different layers, taking into account both shallow detail information and deep semantic information, thereby improving the richness and discriminability of fusion features.

[0056] S5.1-4. To avoid inconsistent feature scales after weighted fusion, further layer normalization is performed on the fusion results:

[0057] ;

[0058] Normalization enhances the numerical stability of features and facilitates subsequent cross-modal fusion calculations.

[0059] Preferably, the step of using a cross-modal self-attention mechanism in S6 to achieve deep fusion of visible light image and full polarimetric synthetic aperture radar image features specifically includes:

[0060] S6.1-1. Assume that after the hierarchical feature fusion of the adaptive weighted coefficients, the visible light modality fusion features are obtained respectively and synthetic aperture radar modal fusion features , where N and M are the lengths of visible light and synthetic aperture radar feature sequences respectively, and d is the feature dimension. The two-modal fusion features contain the high-level semantic information extracted by the multi-layer transformer modules of their respective encoders, which is the basis for subsequent cross-modal fusion;

[0061] S6.1-2. To achieve deep semantic interaction between the visible light modality and the synthetic aperture radar modality, a cross-modal self-attention mechanism is designed. Specifically, the visible light modality fusion features are mapped into a query matrix, and the synthetic aperture radar modality fusion features are mapped into a key and value matrix. The calculation formula is as follows:

[0062] ;

[0063] in, The linear projection matrix learned by the model projects the input features into the attention space;

[0064] S6.1-3, based on the above , calculate the cross-modal self-attention weights, the query of the visible light modality weights the key of the synthetic aperture radar modality to produce an attention response:

[0065] ;

[0066] S6.1-4. Perform a residual connection between the attention response and the fusion feature of the visible light modality to avoid information loss and promote gradient propagation during model training, forming an improved fusion feature:

[0067] ;

[0068] S6.1-5, the final fusion feature It is passed as input to the subsequent text encoder to generate a natural language description that combines multimodal information, improving the accuracy and richness of the description.

[0069] Preferably, the step of inputting the fused multimodal features into a text encoder based on a text-to-text transfer transformer model architecture in S7 specifically includes:

[0070] S7.1-1. Assume that the final multimodal fusion feature obtained by deep fusion of the cross-modal self-attention mechanism is expressed as:

[0071] ;

[0072] Where L represents the length of the feature sequence, and d is the feature dimension. This fused feature comprehensively reflects the multi-layer semantic information extracted by the visible light and full polarimetric synthetic aperture radar image encoders. It contains rich multimodal contextual semantics and serves as the basic input for the subsequent text encoder to generate descriptive text.

[0073] S7.1-2. Input the fused features into a text encoder based on the text-to-text transfer transformer model architecture. The text encoder consists of multiple layers of transformer encoding layers. Each layer performs deep processing and semantic abstraction of the input features through a self-attention mechanism and a feedforward network. The output of the i-th layer is recorded as:

[0074] ;

[0075] in, is the total number of encoder layers, and the initial input is ;

[0076] S7.1-3, encoder last layer output The context vector is passed to the text decoder. The text decoder generates natural language descriptions based on the autoregressive mechanism and predicts each word in turn. Probability of:

[0077] ;

[0078] in, Represents the sequence of words that have been generated in the previous order.

[0079] Preferably, the step of optimizing text generation by combining the temperature adjustment mechanism and the reinforcement learning strategy based on the reinforcement learning policy gradient algorithm in S8 specifically includes:

[0080] S8.1-1. In the process of text generation by the text decoder based on the text-to-text transfer transformer model architecture, in order to improve the diversity of the generated text and control the randomness in the sampling process, a temperature adjustment mechanism is introduced. Specifically, the unnormalized prediction value of the i-th word in the vocabulary generated by the decoder at time step t is , adjust its probability distribution by temperature parameter T>0, and the calculation formula is:

[0081] ;

[0082] Where V represents the vocabulary size, and the temperature parameter T adjusts the smoothness of the probability distribution. When T>1, the probability distribution becomes flatter and the sampling diversity increases; when T<1, the probability distribution becomes sharper, tending to select the words with the highest probability, and the generated text is more certain.

[0083] S8.1-2. In order to directly optimize the quality and semantic relevance of the generated text, we define the reward function by combining the reinforcement learning strategy based on the reinforcement learning policy gradient algorithm. , used to evaluate the generated sequence The quality of the generated text can be rewarded based on bilingual evaluation alternatives, recall-oriented summary evaluation indicators, or task-related indicators. The optimization goal of the model is to maximize the expected reward of the generated text:

[0084] ;

[0085] in, represents the model parameters, is the generated probability distribution based on the current parameters;

[0086] S8.1-3. Calculate the gradient of the model parameters using the reinforcement learning policy gradient algorithm:

[0087] ;

[0088] Use the sampled text sequence and its corresponding reward to estimate the gradient and optimize the model parameters through gradient ascent;

[0089] S8.1-4. In order to reduce the variance of the gradient estimate, a baseline function b is introduced and the gradient estimate is improved as follows:

[0090] ;

[0091] in, is the number of sampling times, is the mth sampled text sequence, the baseline The mean of the rewards is usually selected to help stabilize the training process.

[0092] By combining the temperature regulation mechanism with the reinforcement learning strategy gradient algorithm reinforcement learning strategy, the text generation process is made rich and diverse while being optimized in quality and semantics, significantly improving the intelligent text description effect of multimodal remote sensing images.

[0093] Beneficial effects: The present invention collects full-polarimetric synthetic aperture radar image and visible light image data in the same scene; uses Lipschitz filtering to denoise each polarization channel of the collected full-polarimetric synthetic aperture radar image to reduce speckle noise interference, and at the same time performs normalization preprocessing and multi-layer transformer feature extraction on the visible light image; adopts a hierarchical feature fusion method with adaptive weighting coefficients to fuse the features of each polarization channel of the full-polarimetric synthetic aperture radar and the multi-layer features of visible light; designs a cross-modal self-attention mechanism to achieve deep interaction between the fusion features of visible light and full-polarimetric synthetic aperture radar, and enhances the semantic complementarity between modalities; inputs the fused multimodal features into a text encoder based on a text-to-text transfer transformer model architecture to perform high-level semantic encoding and natural language text generation; introduces a temperature regulation mechanism to control the sampling diversity of text generation, and combines it with a reinforcement learning strategy based on a reinforcement learning policy gradient algorithm to optimize the quality and diversity of the generated text. Through the method of the present invention, remote sensing image data of different modalities can be effectively fused, the expressive power of fused features and the accuracy of text descriptions can be significantly improved, and it can adapt to multimodal information processing and intelligent semantic understanding in complex remote sensing environments, and is widely used in environmental monitoring, disaster assessment, target identification and other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0094] Figure 1 It is a schematic flow diagram of the present invention;

[0095] Figure 2 This is a diagram of the dual encoder network model structure;

[0096] Figure 3 This is a schematic diagram of the structure of the cross-modal self-attention mechanism;

[0097] Figure 4 It is a flowchart of text generation based on the text-to-text transfer transformer model. DETAILED DESCRIPTION

[0098] The present invention will be further described below with reference to the accompanying drawings.

[0099] like Figure 1 As shown, a method for describing a full polarization synthetic aperture radar and a visible light image in text form includes the following steps:

[0100] S1. Collect visible light images and full-polarimetric synthetic aperture radar images of the same scene, where the full-polarimetric synthetic aperture radar image contains multiple polarization channels (such as HH, HV, VH, and VV).

[0101] S2. Performing Lipschitz filtering denoising on each polarization channel of the full-polarization synthetic aperture radar image to remove noise and speckles;

[0102] S3, fusing the denoised multiple polarization channel features to form a multi-channel fusion feature, and inputting it into the visual transformer encoder for multi-layer transformer feature extraction;

[0103] S4, using the contrastive language-image pre-trained model encoder to perform multi-layer transformer feature extraction on visible light images;

[0104] S5. Using a layered feature fusion method with adaptive weighting coefficients, weighted fusion is performed on the multi-layer transformer features of the visible light image and the synthetic aperture radar image, and the contribution weight of each layer of features is dynamically adjusted;

[0105] S6. Design a cross-modal self-attention mechanism to achieve deep bidirectional information interaction between visible light image fusion features and synthetic aperture radar image fusion features, further improving the expressive power of multimodal fusion features.

[0106] S7, inputting the fused multimodal features into a text encoder based on a text-to-text transfer transformer model architecture to generate a corresponding natural language text description;

[0107] S8. Combine the temperature regulation mechanism with the reinforcement learning strategy based on the reinforcement learning policy gradient algorithm to optimize the diversity and accuracy of the generated text and improve the final description quality.

[0108] Furthermore, the step of performing Lipschitz filtering denoising on each polarization channel of the full polarization synthetic aperture radar image in S2 is as follows:

[0109] S2.1-1. Collect full-polarization synthetic aperture radar image data for the same ground feature scene, including four polarization channels: horizontal transmission-horizontal reception HH, horizontal transmission-vertical reception HV, vertical transmission-horizontal reception VH, and vertical transmission-vertical reception VV. The original images of these four polarization channels are expressed as follows: ;

[0110] Each polarization channel reflects the different scattering characteristics of the ground objects and is the basis for subsequent fusion analysis;

[0111] S2.1-2. For each polarization channel image, the Lipschitz filter algorithm is used to suppress noise and remove speckles. The Lipschitz filter is based on local statistical characteristics. By calculating the local mean and variance, the filter weight is dynamically adjusted to distinguish the signal and noise parts of the image. The specific filter calculation formula is as follows:

[0112] ;

[0113] in, Represents the original polarization channel pixel value to be processed; Indicated in pixels The mean value of pixels in a small window centered on , which reflects the average intensity of the local image; It represents the variance of pixels within the small window, reflecting the degree of change in local intensity; K is the noise variance estimate, usually obtained through statistical methods or prior knowledge;

[0114] The core idea of ​​this filter is to use local statistical information to determine the ratio of signal to noise in pixel values ​​and calculate a weight factor to balance the relationship between preserving signal edges and suppressing noise, thereby effectively reducing the influence of speckle noise in synthetic aperture radar images.

[0115] S2.1-3. Perform Lipschitz filtering on each of the four polarization channels to obtain the denoised polarization channel images:

[0116] ;

[0117] Each denoised image has a high signal-to-noise ratio, preserving polarization-specific scattering information and image details, ensuring quality for subsequent polarization channel fusion and high-level feature extraction.

[0118] Furthermore, the step of fusing the denoised multiple polarization channel features in S3 specifically includes:

[0119] S3.1-1. After completing the Lipschitz filter denoising process on each polarization channel (HH, HV, VH, VV) of the full polarization synthetic aperture radar image, we obtained four high-quality denoised polarization features, which are expressed as These four polarization channels reflect the electromagnetic wave scattering characteristics of the ground objects from different polarization angles, and have obvious complementary information. Processing these channel features alone is difficult to fully explore the intrinsic correlation between the polarization information, which limits the semantic understanding and target detection performance of subsequent remote sensing images. By introducing learnable weight parameters Different importance weights are assigned to the four polarization channels. The weight parameters are automatically optimized through model training to adapt to different scenarios and task requirements. The weights satisfy the normalization constraints:

[0120] ;

[0121] S3.1-2. Concatenate the weighted four polarization channel features in the channel dimension to form a complete fusion feature representation:

[0122] ;

[0123] The dimension of the spliced ​​feature vector is four times that of a single-channel feature, preserving the fine-grained information of each polarization channel and enhancing the richness and expressiveness of the features. This weighted splicing and fusion method overcomes the information redundancy and feature conflict that may result from simple splicing or summation, fully preserving the diversity and complementarity of polarization channels. At the same time, it enhances the discriminability and robustness of the features through adaptive weight adjustment.

[0124] S3.1-3. The visual transformer encoder is mainly composed of multiple layers of transformer modules. Each layer of transformer abstracts and enhances the input features through a multi-head self-attention mechanism and a feedforward network. Assuming that the visual transformer encoder contains L layers of transformer modules, the output features of each layer are expressed as:

[0125] ;

[0126] in, Represents the polarization characteristics of synthetic aperture radar data after denoising and fusion;

[0127] The feature dimensions of each layer of transformer modules are kept consistent, which facilitates the transmission and fusion of information between subsequent layers. Multi-layer transformer modules capture the local and global context information of the image in turn, and gradually extract more advanced and more semantic visual features. The final multi-layer feature sequence is As a multi-scale and multi-level feature representation of synthetic aperture radar images.

[0128] Furthermore, the step of performing multi-layer transformer feature extraction on the visible light image in S4 specifically includes:

[0129] S4.1-1. Collect a sequence of visible light images of the same scene. Assume that the original input data is represented as ,Since the pixel value range and resolution of visible light images may be different, ,before feature extraction, the input image is preprocessed, including normalization processing. ,The specific normalization formula is :

[0130] ;

[0131] in, and are the mean and standard deviation of visible light image pixels, respectively, used to unify the numerical scale between different images;

[0132] S4.1-2. Input the normalized visible light image into the pre-trained contrastive language-image pre-training model encoder for feature extraction. The contrastive language-image pre-training model encoder is mainly composed of multiple layers of transformer modules. Each layer of transformer abstracts and enhances the input features through a multi-head self-attention mechanism and a feedforward network. Assuming that the contrastive language-image pre-training model encoder contains L layers of transformer modules, the output features of each layer are expressed as:

[0133] ;

[0134] in, Represents the input image features after preliminary linear transformation and position encoding;

[0135] S4.1-3, the feature dimensions of each layer of transformer modules are kept consistent, which facilitates the transmission and fusion of information between subsequent layers. The multi-layer transformer modules capture the local and global context information of the image in turn, and gradually extract more advanced and more semantic visual features. The final multi-layer feature sequence is obtained As a multi-scale, multi-level feature representation of visible light images.

[0136] Furthermore, the steps of the hierarchical feature fusion method using adaptive weighting coefficients in S5 specifically include:

[0137] S5.1-1, the visible light image encoder (using the contrastive language-image pre-training model architecture) and the full polarimetric synthetic aperture radar image encoder (using the visual transformer architecture) both contain L layers of transformer modules. The output features of each layer carry the layer's expression of the input image at different scales and semantic levels, forming a serialized feature representation:

[0138] ;

[0139] Where N and M are the lengths of visible light and synthetic aperture radar feature sequences, respectively, and d is the feature dimension;

[0140] S5.1-2. To overcome the drawback of uneven contribution of feature expressions at each layer to the final task, a dynamic weight is assigned to each layer of features. , the weights are learned through training, specifically by designing a multi-layer perceptron (MLP) network for each layer of features to calculate the weight scores:

[0141] ;

[0142] in, and They represent the weight scores of the kth layer in visible light and synthetic aperture radar modes respectively. To ensure the rationality and comparability of the weights, the softmax function is used to normalize the weight scores of all layers:

[0143] ;

[0144] The sum of the weights of all layers is 1 to ensure stability during fusion;

[0145] S5.1-3. Use normalized weights to perform weighted summation on the features of each layer to obtain the single-modal fusion feature:

[0146] ;

[0147] This fusion method comprehensively considers the contributions of different layers, taking into account both shallow detail information and deep semantic information, thereby improving the richness and discriminative power of the fused features.

[0148] S5.1-4. To avoid inconsistent feature scales after weighted fusion, further layer normalization is performed on the fusion results:

[0149] ;

[0150] Normalization enhances the numerical stability of features and facilitates subsequent cross-modal fusion calculations.

[0151] Furthermore, the step of using the cross-modal self-attention mechanism in S6 to achieve deep fusion of visible light image and full polarimetric synthetic aperture radar image features specifically includes:

[0152] S6.1-1. Assume that after the hierarchical feature fusion of the adaptive weighted coefficients, the visible light modality fusion features are obtained respectively and synthetic aperture radar modal fusion features , where N and M are the lengths of visible light and synthetic aperture radar feature sequences respectively, and d is the feature dimension. The two-modal fusion features contain the high-level semantic information extracted by the multi-layer transformer modules of their respective encoders, which is the basis for subsequent cross-modal fusion;

[0153] S6.1-2. To achieve deep semantic interaction between the visible light modality and the synthetic aperture radar modality, a cross-modal self-attention mechanism is designed. Specifically, the visible light modality fusion feature is mapped into a query Q matrix, and the synthetic aperture radar modality fusion feature is mapped into a key K and value V matrix. The calculation formula is as follows:

[0154] ;

[0155] in, The linear projection matrix learned by the model projects the input features into the attention space;

[0156] S6.1-3, based on the above , calculate the cross-modal self-attention weights, the query of the visible light modality weights the key of the synthetic aperture radar modality to produce an attention response:

[0157] ;

[0158] S6.1-4. Perform a residual connection between the attention response and the fusion feature of the visible light modality to avoid information loss and promote gradient propagation during model training, forming an improved fusion feature:

[0159] ;

[0160] S6.1-5, the final fusion feature It is passed as input to the subsequent text encoder to generate a natural language description that combines multimodal information, improving the accuracy and richness of the description.

[0161] Furthermore, the step of inputting the fused multimodal features into a text encoder based on a text-to-text transfer transformer model architecture in S7 specifically includes:

[0162] S7.1-1. Assume that the final multimodal fusion feature obtained by deep fusion of the cross-modal self-attention mechanism is expressed as:

[0163] ;

[0164] Where L represents the length of the feature sequence, and d is the feature dimension. This fused feature comprehensively reflects the multi-layer semantic information extracted by the visible light and full polarimetric synthetic aperture radar image encoders. It contains rich multimodal contextual semantics and serves as the basic input for the subsequent text encoder to generate descriptive text.

[0165] S7.1-2, fusion features The input is sent to a text encoder based on the text-to-text transfer transformer model architecture. The text encoder consists of multiple layers of transformer encoding layers. Each layer performs deep processing and semantic abstraction of input features through a self-attention mechanism and a feedforward network. The output of the i-th layer is recorded as:

[0166] ;

[0167] in, is the total number of encoder layers, and the initial input is ;

[0168] S7.1-3, encoder last layer output As the context vector is passed to the text decoder, the text decoder generates a natural language description based on the autoregressive mechanism and predicts each word in turn. Probability of:

[0169] ;

[0170] in, Represents the sequence of words that have been generated in the previous order.

[0171] Furthermore, the step of optimizing text generation by combining the temperature adjustment mechanism and the reinforcement learning strategy based on the reinforcement learning policy gradient algorithm in S8 specifically includes:

[0172] S8.1-1. In the process of text generation by the text decoder based on the text-to-text transfer transformer model architecture, in order to improve the diversity of the generated text and control the randomness in the sampling process, a temperature adjustment mechanism is introduced. Specifically, the unnormalized prediction value of the i-th word in the vocabulary generated by the decoder at time step t is , adjust its probability distribution by temperature parameter T>0, and the calculation formula is:

[0173] ;

[0174] Where V represents the vocabulary size, and the temperature parameter T adjusts the smoothness of the probability distribution. When T>1, the probability distribution becomes flatter and the sampling diversity increases; when T<1, the probability distribution becomes sharper, tending to select the words with the highest probability, and the generated text is more certain.

[0175] S8.1-2. In order to directly optimize the quality and semantic relevance of the generated text, we define the reward function by combining the reinforcement learning strategy based on the reinforcement learning policy gradient algorithm. , used to evaluate the generated sequence The quality of the generated text can be rewarded based on bilingual evaluation alternatives, recall-oriented summary evaluation indicators, or task-related indicators. The optimization goal of the model is to maximize the expected reward of the generated text:

[0176] ;

[0177] in, represents the model parameters, is the generated probability distribution based on the current parameters;

[0178] S8.1-3. Calculate the gradient of the model parameters using the reinforcement learning policy gradient algorithm:

[0179] ;

[0180] Use the sampled text sequence and its corresponding reward to estimate the gradient and optimize the model parameters through gradient ascent;

[0181] S8.1-4. In order to reduce the variance of the gradient estimate, a baseline function b is introduced to improve the gradient estimate as follows:

[0182] ;

[0183] in, is the number of sampling times, is the mth sampled text sequence, the baseline Typically, the mean reward value is selected to help stabilize the training process. By combining a temperature regulation mechanism with a reinforcement learning policy gradient algorithm, the text generation process is both rich and diverse, while also being optimized in terms of quality and semantics, significantly improving the intelligent text description of multimodal remote sensing images.

Claims

1. A method for describing full polarimetric synthetic aperture radar and visible light images, characterized in that: The steps include: S1. Acquire a visible light image and a full-polarimetric synthetic aperture radar image of the same scene, wherein the full-polarimetric synthetic aperture radar image contains multiple polarization channels; S2. Performing Lipschitz filtering denoising on each polarization channel of the full-polarization synthetic aperture radar image to remove noise and speckles; S3, fusing the denoised multiple polarization channel features to form a multi-channel fusion feature, and inputting it into the visual transformer encoder for multi-layer transformer feature extraction; S4, using the contrastive language-image pre-trained model encoder to perform multi-layer transformer feature extraction on visible light images; S5. Using a layered feature fusion method with adaptive weighting coefficients, weighted fusion is performed on the multi-layer transformer features of the visible light image and the synthetic aperture radar image, and the contribution weight of each layer of features is dynamically adjusted; S6. Design a cross-modal self-attention mechanism to achieve deep bidirectional information interaction between visible light image fusion features and synthetic aperture radar image fusion features; S7. Input the fused multimodal features into a text encoder based on the text-to-text transfer transformer model architecture to generate the corresponding natural language text description. The specific steps are as follows: S7.1-1. Assume that the final multimodal fusion feature obtained by deep fusion of the cross-modal self-attention mechanism is expressed as: ； Where L represents the length of the feature sequence and d is the feature dimension; S7.1-2, fusion features The input is sent to a text encoder based on the text-to-text transfer transformer model architecture. The text encoder consists of multiple layers of transformer encoding layers. Each layer performs deep processing and semantic abstraction of input features through a self-attention mechanism and a feedforward network. The output of the i-th layer is recorded as: ； in, is the total number of encoder layers, and the initial input is ; S7.1-3, encoder last layer output As the context vector is passed to the text decoder, the text decoder generates a natural language description based on the autoregressive mechanism and predicts each word in turn. Probability of: ； in, Represents the word sequence that has been generated in the previous order; S8. Combine the temperature regulation mechanism with the reinforcement learning strategy based on the reinforcement learning policy gradient algorithm to optimize the diversity and accuracy of the generated text and improve the final description quality.

2. The method for describing full polarimetric synthetic aperture radar and visible light images according to claim 1, characterized in that: The steps of performing Lipschitz filtering denoising on each polarization channel of the full polarization synthetic aperture radar image in S2 are as follows: S2.1-1. Collect full-polarization synthetic aperture radar image data for the same ground feature scene, including four polarization channels: horizontal transmission-horizontal reception HH, horizontal transmission-vertical reception HV, vertical transmission-horizontal reception VH, and vertical transmission-vertical reception VV. The original images of these four polarization channels are expressed as follows: ; S2.1-2. For each polarization channel image, the Lipschitz filter algorithm is used to suppress noise and remove speckles. The Lipschitz filter is based on local statistical characteristics. By calculating the local mean and variance, the filter weight is dynamically adjusted to distinguish the signal and noise parts of the image. The specific filter calculation formula is as follows: ； in, Represents the original polarization channel pixel value to be processed; Indicated in pixels The mean value of pixels in a small window centered on , which reflects the average intensity of the local image; represents the variance of pixels within the small window, reflecting the degree of change in local intensity; K is the noise variance estimate, obtained through statistical methods or prior knowledge; S2.1-3. Perform Lipschitz filtering on each of the four polarization channels to obtain the denoised polarization channel images: 。 3. The method for describing full polarimetric synthetic aperture radar and visible light images according to claim 1, characterized in that: The step of fusing the denoised multiple polarization channel features in S3 specifically includes: S3.1-1. By introducing learnable weight parameters , assigning different importance weights to the four polarization channels. The weight parameters are automatically optimized through model training, and the weights satisfy the following normalization constraints: ； S3.1-2. Concatenate the weighted four polarization channel features in the channel dimension to form a complete fusion feature representation: ； S3.1-3. The visual transformer encoder consists of multiple layers of transformer modules. Each layer of transformer abstracts and enhances the input features through a multi-head self-attention mechanism and a feedforward network. Assuming that the visual transformer encoder contains L layers of transformer modules, the output features of each layer are expressed as: ； Wherein, represents the polarization characteristics of synthetic aperture radar data after denoising and fusion.

4. The method for describing full polarimetric synthetic aperture radar and visible light images according to claim 1, wherein: The step of performing multi-layer transformer feature extraction on the visible light image in S4 specifically includes: S4.1-1. Collect a sequence of visible light images of the same scene. Assume that the original input data is represented as Before feature extraction, the input image is preprocessed, including normalization. The formula is: ； in, and are the mean and standard deviation of visible light image pixels, respectively, used to unify the numerical scale between different images; S4.1-2. Input the normalized visible light image into the pre-trained contrastive language-image pre-training model encoder for feature extraction. The contrastive language-image pre-training model encoder consists of multiple layers of transformer modules. Each layer of transformer abstracts and enhances the input features through a multi-head self-attention mechanism and a feedforward network. Assuming that the contrastive language-image pre-training model encoder contains L layers of transformer modules, the output features of each layer are expressed as: ； in, Represents the input image features after preliminary linear transformation and position encoding; S4.1-3, the feature dimensions of each layer of transformer modules are consistent. The multi-layer transformer modules capture the local and global context information of the image in turn, and gradually extract more advanced and more semantic visual features. The final multi-layer feature sequence is As a multi-scale, multi-level feature representation of visible light images.

5. The method for describing full polarimetric synthetic aperture radar and visible light images according to claim 1, characterized in that: The steps of the hierarchical feature fusion method using adaptive weighting coefficients in S5 specifically include: S5.1-1, the visible light image encoder, and the full polarimetric synthetic aperture radar image encoder both contain L layers of transformer modules. The output features of each layer of modules carry the layer's expression of the input image at different scales and semantic levels, forming a serialized feature representation: ； Where N and M are the lengths of visible light and synthetic aperture radar feature sequences, respectively, and d is the feature dimension; S5.1-2. Assign a dynamic weight to each layer of features , the weights are learned through training, specifically by designing a multi-layer perceptron MLP network for each layer of features to calculate the weight scores: ； in, and They represent the weight scores of the kth layer in visible light and synthetic aperture radar modes respectively, and the softmax function is used to normalize the weight scores of all layers: ； The sum of the weights of all layers is 1 to ensure stability during fusion; S5.1-3. Use normalized weights to perform weighted summation on the features of each layer to obtain the single-modal fusion feature: ； S5.1-4. To avoid inconsistent feature scales after weighted fusion, further layer normalization is performed on the fusion results: 。 6. The method for describing full polarimetric synthetic aperture radar and visible light images according to claim 1, characterized in that: The steps of using the cross-modal self-attention mechanism in S6 to achieve deep fusion of visible light image and full polarimetric synthetic aperture radar image features specifically include: S6.1-1. Assume that after the hierarchical feature fusion of the adaptive weighted coefficients, the visible light modality fusion features are obtained respectively and synthetic aperture radar modal fusion features , where N and M are the lengths of visible light and synthetic aperture radar feature sequences, respectively, and d is the feature dimension; S6.1-2. To achieve deep semantic interaction between the visible light modality and the synthetic aperture radar modality, a cross-modal self-attention mechanism is designed to map the visible light modality fusion features into a query Q matrix and the synthetic aperture radar modality fusion features into a key K and value V matrix. The calculation formula is as follows: ； in, The linear projection matrix learned by the model projects the input features into the attention space; S6.1-3, based on the above , calculate the cross-modal self-attention weights, the query of the visible light modality weights the key of the synthetic aperture radar modality to produce an attention response: ； S6.1-4. Perform a residual connection between the attention response and the fusion feature of the visible light modality to form an improved fusion feature: ； S6.1-5, the final fusion feature It is passed as input to the subsequent text encoder to generate a natural language description that combines multimodal information, improving the accuracy and richness of the description.

7. The method for describing full polarimetric synthetic aperture radar and visible light images according to claim 1, characterized in that: The steps of optimizing text generation by combining the temperature regulation mechanism and the reinforcement learning strategy based on the reinforcement learning policy gradient algorithm in S8 specifically include: S8.1-1. In the process of generating text using a text decoder based on the text-to-text transfer transformer model architecture, a temperature adjustment mechanism is introduced to adjust the unnormalized prediction value of the i-th word in the vocabulary generated by the decoder at time step t. , adjust its probability distribution by temperature parameter T>0, and the calculation formula is: ； Among them, V represents the vocabulary size, and the temperature parameter T adjusts the smoothness of the probability distribution; S8.1-2. Define the reward function based on the reinforcement learning strategy based on the reinforcement learning policy gradient algorithm , used to evaluate the generated sequence The reward can be based on bilingual evaluation alternatives, recall-oriented summary evaluation metrics, or task-related metrics. The optimization goal of the model is to maximize the expected reward of the generated text: ； in, represents the model parameters, is the generated probability distribution based on the current parameters; S8.1-3. Calculate the gradient of the model parameters using the reinforcement learning policy gradient algorithm: ； Use the sampled text sequence and its corresponding reward to estimate the gradient and optimize the model parameters through gradient ascent; S8.1-4, introduce the baseline function b and improve the gradient estimation to: ； in, is the number of sampling times, is the mth sampled text sequence, the baseline Select the mean of the rewards.

Images