Satellite remote sensing image information description method in dark environment

Abstract

The invention discloses a satellite remote sensing image information description method in a dark environment, and relates to the technical field of artificial intelligence. According to the method, an intelligence description model comprising a feature extractor, a multi-scale language encoder and a text decoder is constructed: the feature extractor takes a modified ResNet34 convolutional network as a Siamese backbone network, comprises a feature refining module and a multi-scale difference generation and fusion module, and realizes multi-scale visual feature extraction and fusion of the dual-temporal image; a multi-scale language encoder extracts dual-temporal feature difference and related multi-scale language information through difference extraction, multi-scale cross attention and a projection layer to obtain advanced semantic information; the text decoder processes high-level semantic information based on a Transform architecture, generates vocabulary probability distribution and gradually generates description texts. According to the method, the problems of feature extraction and change description in a dark environment are solved in a targeted manner, and the accuracy of information description is effectively improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method for describing satellite remote sensing imagery in low-light environments. Background Technology

[0002] Satellite remote sensing imagery intelligence description refers to the process of comparing remote sensing images of the same area acquired at different times, identifying and analyzing changes in surface features over time, detecting specific location information of these changes, and automatically generating precise language to describe this intelligence. This provides data support for scientific decision-making in fields such as environmental monitoring, disaster assessment, and land planning.

[0003] Currently, satellite remote sensing image intelligence description methods mainly follow an encoder-decoder framework. First, visual features are extracted from dual-temporal images. Then, difference features are calculated based on these visual features, and finally, intelligence descriptions are generated based on these difference features. Existing technology one uses a pre-trained CNN as the encoder and an RNN as the decoder to generate text descriptions, thereby describing changes in different types of objects. Existing technology two decomposes the remote sensing image change description task into two problems: whether a change has occurred and what the change was. An image-level classifier performs binary classification to solve the problem of whether a change has occurred, and a feature-level encoder extracts discriminative features to assist the decoder in generating descriptive text.

[0004] The aforementioned algorithms have three main drawbacks: First, all models directly utilize shared-weight image encoders to independently extract spatial features from satellite remote sensing images at different times, resulting in the models failing to focus more specifically on features of changing areas and ignoring features of irrelevant areas. Second, none of the models consider the large differences in target scales within remote sensing images, and the potential for complex spatial interactions between multiple targets, leading to a decline in model performance. Third, the models only study remote sensing image description methods under normal lighting conditions, lacking research on satellite remote sensing image description methods under low-light conditions. Summary of the Invention

[0005] The main objective of this application is to provide a method for describing satellite remote sensing imagery in low-light environments, which aims to effectively enhance the model's ability to identify key change features, analyze change information at different scales and levels, and effectively improve the accuracy of satellite remote sensing imagery description in low-light environments.

[0006] To achieve the above objectives, this application provides a method for describing satellite remote sensing imagery in low-light environments, comprising: A satellite remote sensing image intelligence description model is constructed for low-light environments. The satellite remote sensing image intelligence description model includes a feature extractor, a multi-scale language encoder, and a text decoder. The feature extractor uses a modified ResNet34 convolutional network as the backbone network of Siamese to extract multi-scale visual features from the acquired dual-temporal images. The modified ResNet34 convolutional network includes a feature refinement module with cross-channel attention and difference enhancement sub-modules, as well as a multi-scale difference generation module and a difference fusion module. The feature refinement module serves as a bridge for information interaction between two temporal images, while the multi-scale difference generation module and the difference fusion module are used to fuse multi-scale visual features. The multi-scale language encoder is used to extract the bi-temporal feature differences in multi-scale visual features through the difference extraction module, and then use multi-scale cross attention to extract the bi-temporal multi-scale language information related to changes in multi-scale visual features. The bi-temporal feature differences and bi-temporal multi-scale language information are further processed through the projection layer to obtain high-level semantic information. The text decoder is used to process high-level semantic information based on the Transformer decoder architecture, obtain the word probability distribution of the vocabulary, and gradually generate words to finally obtain satellite remote sensing image intelligence description text.

[0007] Optionally, the ResNet backbone network is divided into 5 stages. After each stage of the ResNet backbone network, the dual-temporal images are sent to the feature refinement module to exchange information between the dual-temporal images, obtain the dual-temporal intermediate coding features, and then send the dual-temporal intermediate coding features to the next stage.

[0008] Optionally, the cross-channel attention is used for: The global feature channels of each temporal intermediate coding feature are obtained by average pooling. The channel difference information is obtained by subtracting the features of the global feature channels. Then, the channel attention weights are generated by convolution. The channel attention weights are used to enhance the channel features of each temporal intermediate coding feature and residual connection is used to obtain the channel-enhanced temporal coding features.

[0009] Optionally, the difference enhancement submodule is used for: Direct dissimilarity and dissimilarity are calculated separately to generate a change representation. The direct dissimilarity is obtained by calculating the absolute value of the difference between the channel-enhanced bi-temporal coding features to capture the direct dissimilarity in the bi-temporal features. The dissimilarity is obtained by calculating the similarity between the bi-temporal features in the feature space to capture the dissimilarity in the bi-temporal features. The two representations obtained by direct dissimilarity and dissimilarity are fused to generate a final difference representation, which is then applied to the channel-enhanced bi-temporal coding features to obtain the feature-optimized bi-temporal coding features.

[0010] Optionally, the multi-scale difference generation module is used to subtract the features from the optimized dual-time-coded features generated at each stage and extract multi-scale difference information through convolution, and then fuse the multi-scale difference features by convolution and feature addition to generate difference features; The difference fusion module is used to fuse the obtained difference features with the optimized dual-time-coded features obtained in the last stage of the ResNet backbone network through Concat concatenation and convolution to obtain multi-scale visual features.

[0011] Optionally, the multi-scale language encoder is used to extract bi-temporal feature differences in multi-scale visual features through a difference extraction module, including: The difference extraction module is used to perform feature subtraction, concat concatenation and convolution operations on multi-scale visual features to extract the differences in features across two time periods.

[0012] Optionally, the step of using multi-scale cross-attention to extract change-related bi-temporal multi-scale linguistic information from multi-scale visual features includes: Multi-scale cross-attention is used to extract dual-temporal multi-scale linguistic information by using multi-scale visual features as query vectors and dual-temporal feature differences as key and value vectors, respectively.

[0013] Optionally, the step of using multi-scale cross-attention to treat multi-scale visual features as query vectors includes: Each input multi-scale visual feature is divided into multiple non-overlapping local windows and reassembled into a sequence. Multi-head attention computation is performed within each local window to capture feature associations within the local region, resulting in processed window features. The processed window features are reassembled to restore the original feature map size, resulting in the local attention features of the three branches; The query vector is obtained by dynamically learning the fusion weights of the three branches through a gating network and weighting and fusing the local attention features of the three branches.

[0014] Optionally, the step of further processing the bi-temporal feature differences and bi-temporal multi-scale linguistic information through the projection layer to obtain high-level semantic information includes: The dual-temporal multi-scale linguistic information features are processed by subtraction and convolution operations through a projection layer to obtain the processed dual-temporal features. The cosine similarity of the processed bi-temporal features is calculated. The processed bi-temporal features and the cosine similarity are concatenated and then passed through a standard residual network to obtain high-level semantic information.

[0015] Optionally, the text decoder is used to process high-level semantic information based on the Transformer decoder architecture, obtain the word probability distribution of the vocabulary, and gradually generate words to finally obtain satellite remote sensing image intelligence description text, including: The text decoder uses an embedding layer to map text tokens from high-level semantic information to word embeddings and adds positional encoding to obtain the initial input to the Transformer decoder; The Transformer decoder consists of N identical layers stacked together. Each layer contains a multi-head masked self-attention layer, a multi-head cross-attention layer, and a feedforward neural network layer. Each sub-layer is equipped with residual connections and layer normalization. After the initial input is decoded by the Transformer decoder, the probability of each word at that position is obtained through a linear layer and a Softmax activation function, generating satellite remote sensing image intelligence description text.

[0016] This application proposes a satellite remote sensing image intelligence description method for low-light environments. The method constructs a satellite remote sensing image intelligence description model for low-light environments, which includes a feature extractor, a multi-scale language encoder, and a text decoder. The feature extractor uses a modified ResNet34 convolutional network as the backbone network of Siamese to extract multi-scale visual features from the acquired dual-temporal images. The modified ResNet34 convolutional network includes a feature refinement module with cross-channel attention and difference enhancement submodules, as well as a multi-scale difference generation module and a difference fusion module. The feature refinement module is used to refine the features between the dual-temporal images. The system serves as an information exchange bridge, with a multi-scale difference generation module and a difference fusion module used to fuse multi-scale visual features. A multi-scale language encoder extracts bi-temporal feature differences from multi-scale visual features through a difference extraction module, then uses multi-scale cross-attention to extract change-related bi-temporal multi-scale language information from the multi-scale visual features. A projection layer further processes the bi-temporal feature differences and bi-temporal multi-scale language information to obtain high-level semantic information. A text decoder, based on a Transformer decoder architecture, processes the high-level semantic information, obtains the word probability distribution of the vocabulary, and gradually generates words, ultimately producing satellite remote sensing image intelligence description text. This solution addresses the problem of satellite remote sensing image intelligence description methods in low-light environments and implements a method for describing satellite remote sensing image intelligence in low-light conditions. Attached Figure Description

[0017] Figure 1 This is a structural diagram provided in an embodiment of the satellite remote sensing image information description method in low-light environments according to this application; Figure 2 This is a schematic diagram of an embodiment of the satellite remote sensing image information description method in low-light environments provided in this application; Figure 3 This is an example diagram of cross-channel attention provided in an embodiment of the satellite remote sensing image intelligence description method in low-light environments according to this application; Figure 4 This is a schematic diagram of the difference enhancement module provided in an embodiment of the satellite remote sensing image information description method in low-light environments according to this application; Figure 5 This is a schematic diagram of the local attention method of the present invention provided in an embodiment of the satellite remote sensing image information description method in low-light environments; Figure 6 This is a schematic diagram of the projection layer provided in an embodiment of the satellite remote sensing image information description method in low-light environments according to this application.

[0018] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0019] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0020] To overcome the shortcomings of existing technologies, this invention provides a method for describing satellite remote sensing imagery in low-light environments. It constructs a dataset and a model for describing changes in satellite remote sensing images under low-light conditions. The remote sensing image description model includes a feature extractor, a multi-scale language encoder, and a text decoder. The feature extractor incorporates a dual-temporal feature refinement module into the Siamese backbone network to interact and fuse dual-temporal features, thereby aligning channels, reducing the impact of illumination changes, and effectively capturing regions of interest, enhancing the model's ability to identify key change features. The multi-scale language encoder extracts differences between dual-temporal features through a difference extraction module and then uses multi-scale cross-attention to enable the model to extract multi-scale language information related to the changes. The text decoder uses a Transformer decoder to generate change description text. This invention enhances the model's ability to identify key change features, analyzes change information at different scales and levels, and effectively improves the accuracy of satellite remote sensing imagery description in low-light environments.

[0021] Reference Figure 1 and Figure 2 The satellite remote sensing image intelligence description method for low-light environments provided in the first embodiment of this application can be executed by a processor of a terminal or server. The satellite remote sensing image intelligence description method for low-light environments may include: S101. Construct a satellite remote sensing image intelligence description model under low light conditions. The satellite remote sensing image intelligence description model includes a feature extractor, a multi-scale language encoder, and a text decoder. The core application scenario of this model is in low-light environments. The input is a pair of dual-temporal satellite remote sensing images of the same area acquired at different times. The goal is to generate a text sequence that accurately describes the changes in surface features between the two images. In other words, the input to the satellite remote sensing image intelligence description model is a pair of satellite remote sensing images. and The goal is to generate descriptive text sequences. .

[0022] Among them, the feature extractor is responsible for mining effective visual features from dual-temporal images in low-light environments, the multi-scale language encoder realizes the transformation and optimization of visual features into text features, and the text decoder generates coherent and accurate descriptive text based on the optimized features. Obviously, by constructing a model containing these three core components, a structured technical framework is provided for the description of satellite remote sensing imagery in low-light environments, ensuring that the functions of each link are complementary and orderly connected. This fundamentally solves the problem that existing technologies lack design for low-light environments and lays the architectural foundation for improving the accuracy of descriptions in the future.

[0023] In another embodiment of this application, the satellite remote sensing image information description method in low-light environments further includes: This invention constructs a dataset for describing satellite remote sensing imagery in low-light environments, dividing it into training, testing, and validation sets. Specifically, the brightness of the MIL-CC dataset is adjusted to 0.2-0.3, and its green channel is enhanced to construct the dataset. The dataset is then divided into training, testing, and validation sets in a 7:2:1 ratio.

[0024] The MIL-CC dataset includes 1,500 pairs of short-interval (0–10 minutes) dual-temporal remote sensing images, each 512×512 pixels, covering 12 typical military changes such as aircraft concealment / appearance / formation / evacuation / taxiing, ship navigation / formation / evacuation / departure / arrival, and bridge / building destruction. Each image pair is accompanied by 5 descriptive sentences, with a total vocabulary of 132 words.

[0025] S102. The feature extractor uses a modified ResNet34 convolutional network as the backbone network of Siamese to extract multi-scale visual features from the acquired dual-temporal images. The modified ResNet34 convolutional network includes a feature refinement module with cross-channel attention and difference enhancement sub-modules, as well as a multi-scale difference generation module and a difference fusion module. The feature refinement module serves as a bridge for information interaction between two temporal images, while the multi-scale difference generation module and the difference fusion module are used to fuse multi-scale visual features to obtain multi-scale visual features.

[0026] It should be noted that the feature extractor uses a pre-trained ResNet convolutional neural network as the backbone network, and embeds the feature refinement module into the ResNet backbone network as an information exchange bridge between the two temporal images.

[0027] In one embodiment of this application, the ResNet backbone network is divided into 5 stages. After each stage of the ResNet backbone network, the dual-temporal images are sent to the feature refinement module to perform information exchange between the dual-temporal images, obtain the dual-temporal intermediate coding features, and then send the dual-temporal intermediate coding features to the next stage.

[0028] Specifically, ResNet34's five stages correspond to feature extraction at different scales, from low-level texture features in the first stage to high-level semantic features in the fifth stage, achieving layer-by-layer capture of multi-dimensional visual information. Information exchange occurs between the two-phase images after each stage, rather than a one-time exchange after all stages, enabling timely extraction of temporally correlated information at each scale.

[0029] The model in this application can continuously enhance the correlation between two temporal images while extracting multi-scale features, effectively avoiding interference from irrelevant regional features, and providing high-quality basic feature support for subsequent accurate identification of features in changed regions.

[0030] In one embodiment of this application, cross-channel attention is used to: obtain the global feature channels of each of the two temporal intermediate coding features through average pooling, subtract the features of the global feature channels to obtain channel difference information, generate their respective channel attention weights through convolution, enhance the channel features of each temporal intermediate coding feature using the channel attention weights and use residual connections to obtain channel-enhanced two temporal coding features.

[0031] In one embodiment of this application, the difference enhancement submodule is used to: calculate direct difference and dissimilarity respectively to generate a change representation; the direct difference is obtained by calculating the absolute value of the difference between the channel-enhanced dual-temporal coding features to capture the direct difference in the dual-temporal features; the dissimilarity is obtained by calculating the similarity between the dual-temporal features in the feature space to capture the dissimilarity in the dual-temporal features; the two representations obtained by direct difference and dissimilarity are fused to generate the final difference representation, and applied to the channel-enhanced dual-temporal coding features to obtain the feature-optimized dual-temporal coding features.

[0032] For example, the feature refinement module includes a cross-channel attention and difference enhancement module. (See reference) Figure 3 Given bi-temporal intermediate encoded features at a certain stage of visual encoding , Cross-channel attention obtains the global feature channels of each image through average pooling. , Channel difference information is obtained by subtracting features from global feature channels of dual-temporal features. Then, convolution is used to generate separate channel attention weights to enhance the respective channel features, and residual connections are used to obtain channel-enhanced bi-temporal coding features. , The above process can be represented as:

[0033]

[0034]

[0035]

[0036]

[0037] refer to Figure 4 The difference enhancement module calculates "direct difference" and "dissimilarity" respectively to generate change representations, and uses a fusion approach to refine the change regions of the input features. Direct difference Inconsistencies in bi-temporal features are captured by the absolute value of feature differences. Dissimilarity. By measuring the similarity between bi-temporal features in the feature space, the above process can be expressed as:

[0038]

[0039] in It is a learnable linear transformation matrix. Used to extract the diagonal elements of a matrix, i.e., the similarity of features at the same position.

[0040] To combine the advantages of both direct dissimilarity and non-dissimilarity measurement methods, the module fuses the two representations obtained from direct dissimilarity and non-dissimilarity to generate a final dissimilarity representation, which is then applied to the original features to optimize the feature representation. The above process can be represented as:

[0041]

[0042]

[0043]

[0044] Then, the bi-temporal features generated at each stage are used to extract multi-scale difference information through feature subtraction and convolution, and multi-scale difference features are fused through convolution and feature addition. Given the bi-temporal features obtained in the i-th stage of visual encoding... , The above process can be represented as:

[0045]

[0046] The difference features obtained by the multi-scale difference generation module Bi-temporal features obtained from the final stage of the ResNet backbone network , Multi-scale visual features are obtained by fusion through concat stitching and convolution. , .

[0047] It should be noted that direct dissimilarity, calculated by the absolute value of feature differences, can intuitively capture the numerical inconsistencies of features in both time phases, making it suitable for quickly identifying obvious regions of change. Dissimilarity, on the other hand, extracts the diagonal elements of the feature matrix through a learnable linear transformation matrix, accurately measuring the similarity of features at the same location in a high-dimensional feature space, and can uncover deep, implicit change information. Fusing these two dissimilarity representations can capture both explicit and implicit changes, and then applying them to the channel-enhanced features achieves secondary feature optimization.

[0048] In one embodiment of this application, the multi-scale difference generation module is used to subtract the features of the dual-time-coded features generated at each stage and extract multi-scale difference information through convolution, and then perform multi-scale difference feature fusion by convolution and feature addition to generate difference features; the difference fusion module is used to fuse the obtained difference features with the dual-time-coded features of the feature optimization obtained in the last stage of the ResNet backbone network through Concat concatenation and convolution to obtain multi-scale visual features.

[0049] Understandably, the optimized dual-time-encoded features output by ResNet34 at each stage correspond to visual information at different scales. The multi-scale difference generation module separates the difference information at each scale through feature subtraction, and then performs convolution enhancement and feature addition to achieve preliminary aggregation of multi-scale difference information. The difference fusion module concatenates the aggregated difference features with the high-level semantic features from the final stage through Concat to integrate dimensional information, and then performs convolution operations to achieve deep feature fusion. The final multi-scale visual features contain difference information at each level and integrate high-level semantic features, resulting in richer information dimensions. Specifically, the cross-channel attention and difference enhancement sub-modules in the feature refinement module work together to achieve channel-level difference alignment and feature optimization, while the multi-scale difference generation and fusion module further integrates difference information at different scales. The entire feature extraction process is progressive, effectively reducing the interference of lighting changes on feature extraction in low-light environments, significantly enhancing the model's ability to identify key changing features, and solving the deficiency of existing technologies in targeting features in changing regions.

[0050] S103, the multi-scale language encoder is used to extract the bi-temporal feature differences in multi-scale visual features through the difference extraction module, and then use multi-scale cross attention to extract the bi-temporal multi-scale language information related to changes in multi-scale visual features. The bi-temporal feature differences and bi-temporal multi-scale language information are further processed through the projection layer to obtain high-level semantic information. In one embodiment of this application, a multi-scale language encoder is used to extract bi-temporal feature differences in multi-scale visual features through a difference extraction module, including: using the difference extraction module to perform feature subtraction, concat concatenation, and convolution operations on the multi-scale visual features to extract bi-temporal feature differences.

[0051] Among them, the feature subtraction operation targets the dual-temporal features in multi-scale visual features and directly locates the core difference region between the two; the concat concatenation integrates the difference information by dimension and enriches the expression of the difference features; the convolution operation further enhances the representation ability of the difference features, ensuring that the extracted dual-temporal feature differences are accurate and discriminative, and providing a focus for subsequent language information extraction.

[0052] In one embodiment of this application, multi-scale cross-attention is used to extract bi-temporal multi-scale linguistic information related to changes in multi-scale visual features, including: using multi-scale cross-attention to treat multi-scale visual features as query vectors and bi-temporal feature differences as key vectors and value vectors, respectively, to extract bi-temporal multi-scale linguistic information.

[0053] The method employs multi-scale cross-attention to use multi-scale visual features as query vectors. This includes: dividing each input multi-scale visual feature into multiple non-overlapping local windows and recombining them into a sequence; performing multi-head attention calculation within each local window to capture feature associations within the local region, resulting in processed window features; reassembling the processed window features to restore the original feature map size, resulting in local attention features for the three branches; and dynamically learning the fusion weights of the three branches through a gating network and performing weighted fusion of the local attention features of the three branches to obtain the query vector.

[0054] For example, a multi-scale language encoder uses a difference extraction module to process multi-scale visual features. , Feature subtraction, concatenation, and convolution operations are performed to extract the differences in features between the two temporal phases. Then, multi-scale cross-attention is used to separately process the multi-scale visual features. , As Differences in characteristics between two time phases As , Extract deep language features. (Reference) Figure 6 The processor then performs subtraction and convolution operations on the bi-temporal language features and calculates the cosine similarity of the bi-temporal features. The two results are then concatenated through a projection layer, and a standard residual network is used to extract the high-level semantic information of the bi-temporal features. This process can be represented as:

[0055]

[0056]

[0057]

[0058]

[0059] in This represents multi-scale cross-attention, which divides each input feature into a window, as shown in the reference. Figure 5 The feature map is segmented into multiple non-overlapping local windows and then reassembled into a sequence. Multi-head attention computation is performed within each local window to capture feature associations within the local region. The processed window features are then reassembled to restore the original feature map size, resulting in local attention features. Finally, a gating network is used to dynamically learn the fusion weights of the three branches, thereby performing weighted fusion to obtain the multi-scale cross-attention result. The specific fusion process can be represented as follows:

[0060]

[0061]

[0062] in The window size represents the size of the feature map.

[0063] Understandably, this application decomposes global multi-scale visual features into multiple local windows through window partitioning, which can effectively capture feature associations within local regions and avoid the loss of local details caused by global attention calculation; multi-head attention calculation mines feature information within local windows from multiple dimensions, improving the comprehensiveness of feature representation; the gating network dynamically learns and fuses weights, which can adaptively adjust the contribution of local attention features of each branch according to the actual importance of features in dim environments, ensuring that the query vector can highlight key information.

[0064] In one embodiment of this application, the bi-temporal feature differences and bi-temporal multi-scale linguistic information are further processed through a projection layer to obtain high-level semantic information, including: The dual-temporal multi-scale linguistic information features are subtracted and convolved by a projection layer to obtain processed dual-temporal features. The cosine similarity of the processed dual-temporal features is calculated, and the processed dual-temporal features and cosine similarity are concatenated and passed through a standard residual network to obtain high-level semantic information.

[0065] It's important to note that the core function of the projection layer is to facilitate the transition of features from the visual domain to the textual domain. Subtraction and convolution of dual-temporal, multi-scale linguistic information features further enhance the differential representation between the two phases. The calculation of cosine similarity accurately measures the similarity of the processed dual-temporal features, providing a quantitative basis for semantic association. After concatenation, a standard residual network extracts high-level semantic information, ensuring the depth and accuracy of the semantic features. In fact, the multi-scale language encoder accurately locates the core differences through the difference extraction module, focuses on relevant linguistic information through multi-scale cross-attention, and optimizes the query vector through window partitioning and dynamic fusion. The projection layer achieves cross-domain feature transition and advanced semantic extraction. The entire process fully considers the characteristics of large target scale differences and complex spatial interactions in remote sensing images, effectively extracting semantic information related to changes at different scales, thus solving the problem of decreased model performance caused by neglecting this characteristic in existing technologies.

[0066] S104, the text decoder is used to process high-level semantic information based on the Transformer decoder architecture, obtain the word probability distribution of the vocabulary, and gradually generate words to finally obtain satellite remote sensing image intelligence description text. In one embodiment of this application, a text decoder is used to process high-level semantic information based on a Transformer decoder architecture, obtain the word probability distribution of the vocabulary, and gradually generate words to finally obtain satellite remote sensing image intelligence description text. This includes: using the embedding layer of the text decoder to map text tags in the high-level semantic information to word embeddings and adding positional encoding to obtain the initial input of the Transformer decoder; wherein the Transformer decoder consists of N identical layers stacked together, each layer containing a multi-head masked self-attention layer, a multi-head cross-attention layer, and a feedforward neural network layer, and each sub-layer is equipped with residual connections and layer normalization; after the Transformer decoder decodes the initial input, the probability of each word at that position is obtained through a linear layer and a Softmax activation function to generate the satellite remote sensing image intelligence description text.

[0067] For the text decoder, to accurately describe the changes between image pairs, this algorithm uses a Transformer-based decoder, widely adopted in various natural language processing tasks, to generate change description text. The Transformer decoder structure consists of N identical stacked layers, each containing three main sub-layers: a multi-head masked self-attention layer, a multi-head cross-attention layer, and a feedforward neural network layer. Each sub-layer is equipped with residual connections and layer normalization. First, an embedding layer maps text tokens to word embeddings. Specifically, this algorithm uses a function... Map the original token t to word embeddings The initial input to the Transformer decoder is obtained by adding positional encoding. The specific formula is as follows:

[0068] in It is a positional encoding, calculated using a sine function for each marked position.

[0069] The input is then fed into a multi-head masked self-attention layer. The j-th masked multi-head attention sublayer in an N-layer stack can be represented as:

[0070]

[0071] After passing through the multi-head masking self-attention layer, the system enters the cross-attention layer. The cross-attention layer also employs a multi-head attention mechanism; K and V come from the attention encoder, while Q comes from the previously encoded multi-head attention sub-layer. The output of the j-th cross-attention layer can be represented as:

[0072]

[0073] After passing through the Transformer decoder, the probability of each word at that position is obtained through a linear layer and a Softmax activation function, thereby generating the descriptive text. The above process can be represented as:

[0074] in The probability of generating word v at position t.

[0075] Understandably, the embedding layer transforms text tags into low-dimensional, dense word embedding vectors, making them computable. Positional encoding calculates the positional information of each tag using a sine function and incorporates it into the word embedding, ensuring the temporal logic of the generated text and avoiding word order confusion. In the N-layer stacked structure of the Transformer decoder, the multi-head masked self-attention layer captures the dependencies within the generated text, ensuring text fluency; the multi-head cross-attention layer associates high-level semantic information with text features, ensuring that the generated text is highly correlated with image changes; the feedforward neural network layer performs non-linear transformations on features, enhancing semantic representation capabilities; residual connections and layer normalization effectively alleviate the gradient vanishing and feature shift problems in deep network training. Finally, the linear layer maps the decoder output features to the vocabulary dimension, and the Softmax activation function transforms them into word probability distributions, achieving the gradual generation of words. Clearly, the text generation method based on the Transformer decoder architecture can make full use of the high-level semantic information output by the multi-scale language encoder, accurately capture the temporal relationship and semantic association of the text, and the gradually generated words can accurately and coherently describe the changes in satellite remote sensing images in low-light environments. Combined with the collaborative optimization of the modules mentioned above, the accuracy of satellite remote sensing image intelligence description in low-light environments is significantly improved.

[0076] In another embodiment of this application, a training method for the model is also included. The training method and the model's performance are as follows: The satellite remote sensing imagery description model under low-light conditions is trained using a training set; model parameters are adjusted using a validation set; the cross-entropy function is used to calculate the loss between the generated description text and the corresponding real description text, and this loss function is used to optimize the model. In the multi-head attention mechanism used by the model, the hidden layer dimension is set to 512, the number of attention heads is set to 8, dropout is set to 0.1, and the maximum length of the generated text is set to 40. During training, the optimizer uses the Adam optimizer with default parameters and is trained for 100 epochs. To better adjust the learning rate, a warm-up learning rate strategy and a cosine decay strategy are combined during training, with the peak learning rate set to... .

[0077] To verify the effectiveness of the designed satellite remote sensing image intelligence description model in low-light environments, tests were conducted on our created dataset for satellite remote sensing image intelligence description in low-light environments and the LEVIR-CC dataset. The results in Tables 1 and 2 show that the algorithm significantly improves performance compared to existing methods.

[0078] Table 1. Performance of the present invention on the dataset of satellite remote sensing imagery description in low-light environments.

[0079] Table 2. Performance of the present invention on the LEVIR-CC dataset.

[0080] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for describing satellite remote sensing imagery in low-light environments, characterized in that, include: A satellite remote sensing image intelligence description model is constructed for low-light environments. The satellite remote sensing image intelligence description model includes a feature extractor, a multi-scale language encoder, and a text decoder. The feature extractor uses a modified ResNet34 convolutional network as the backbone network of Siamese to extract multi-scale visual features from the acquired dual-temporal images. The modified ResNet34 convolutional network includes a feature refinement module with cross-channel attention and difference enhancement sub-modules, as well as a multi-scale difference generation module and a difference fusion module. The feature refinement module serves as a bridge for information interaction between two temporal images, while the multi-scale difference generation module and the difference fusion module are used to fuse multi-scale visual features. The multi-scale language encoder is used to extract the bi-temporal feature differences in multi-scale visual features through the difference extraction module, and then use multi-scale cross attention to extract the bi-temporal multi-scale language information related to changes in multi-scale visual features. The bi-temporal feature differences and bi-temporal multi-scale language information are further processed through the projection layer to obtain high-level semantic information. The text decoder is used to process high-level semantic information based on the Transformer decoder architecture, obtain the word probability distribution of the vocabulary, and gradually generate words to finally obtain satellite remote sensing image intelligence description text.

2. The satellite remote sensing image information description method in low-light environments as described in claim 1, characterized in that, The ResNet backbone network is divided into 5 stages. After each stage of the ResNet backbone network, the bi-temporal images are sent to the feature refinement module to exchange information between the bi-temporal images, obtain the bi-temporal intermediate coding features, and then send the bi-temporal intermediate coding features to the next stage.

3. The satellite remote sensing image information description method in low-light environments as described in claim 2, characterized in that, The cross-channel attention is used for: The global feature channels of each temporal intermediate coding feature are obtained by average pooling. The channel difference information is obtained by subtracting the features of the global feature channels. Then, the channel attention weights are generated by convolution. The channel attention weights are used to enhance the channel features of each temporal intermediate coding feature and residual connection is used to obtain the channel-enhanced temporal coding features.

4. The satellite remote sensing image information description method in low-light environments as described in claim 3, characterized in that, The difference enhancement submodule is used for: Direct dissimilarity and dissimilarity are calculated separately to generate change representations. Direct dissimilarity is calculated by taking the absolute value of the differences between the channel-enhanced bi-temporal encoded features to capture the direct dissimilarity in the bi-temporal features. Dissimilarity is calculated by taking the similarity between the bi-temporal features in the feature space to capture the dissimilarity in the bi-temporal features. The two representations obtained from direct dissimilarity and dissimilarity are fused to generate a final difference representation, which is then applied to the channel-enhanced bi-temporal encoded features. The optimized dual-time-coded features are obtained.

5. The satellite remote sensing image information description method in low-light environments as described in claim 4, characterized in that, The multi-scale difference generation module is used to subtract the dual-time-coded features generated at each stage and extract multi-scale difference information through convolution. Then, it performs multi-scale difference feature fusion by convolution and feature addition to generate difference features. The difference fusion module is used to fuse the obtained difference features with the optimized dual-time-coded features obtained in the last stage of the ResNet backbone network through Concat concatenation and convolution to obtain multi-scale visual features.

6. The satellite remote sensing image information description method in low-light environments as described in claim 1, characterized in that, The multi-scale language encoder is used to extract bi-temporal feature differences in multi-scale visual features through a difference extraction module, including: The difference extraction module is used to perform feature subtraction, concat concatenation and convolution operations on multi-scale visual features to extract the differences in features across two time periods.

7. The satellite remote sensing image information description method in low-light environments as described in claim 1, characterized in that, The method of extracting change-related bi-temporal multi-scale linguistic information from multi-scale visual features using multi-scale cross-attention includes: Multi-scale cross-attention is used to extract dual-temporal multi-scale linguistic information by using multi-scale visual features as query vectors and dual-temporal feature differences as key and value vectors, respectively.

8. The satellite remote sensing image information description method in low-light environments as described in claim 1, characterized in that, The method of using multi-scale cross-attention to treat multi-scale visual features as query vectors includes: Each input multi-scale visual feature is divided into multiple non-overlapping local windows and reassembled into a sequence. Multi-head attention computation is performed within each local window to capture feature associations within the local region, resulting in processed window features. The processed window features are reassembled to restore the original feature map size, resulting in the local attention features of the three branches; The query vector is obtained by dynamically learning the fusion weights of the three branches through a gating network and weighting and fusing the local attention features of the three branches.

9. The satellite remote sensing image information description method in low-light environments as described in claim 1, characterized in that, The process of further processing bi-temporal feature differences and bi-temporal multi-scale linguistic information through a projection layer yields high-level semantic information, including: The dual-temporal multi-scale linguistic information features are processed by subtraction and convolution operations through a projection layer to obtain the processed dual-temporal features. The cosine similarity of the processed bi-temporal features is calculated. The processed bi-temporal features and the cosine similarity are concatenated and then passed through a standard residual network to obtain high-level semantic information.

10. The satellite remote sensing image information description method in low-light environments as described in claim 1, characterized in that, The text decoder is used to process high-level semantic information based on the Transformer decoder architecture, obtain the word probability distribution of the vocabulary, and gradually generate words to finally obtain satellite remote sensing image intelligence description text, including: The text decoder uses an embedding layer to map text tokens from high-level semantic information to word embeddings and adds positional encoding to obtain the initial input to the Transformer decoder; The Transformer decoder consists of N identical layers stacked together. Each layer contains a multi-head masked self-attention layer, a multi-head cross-attention layer, and a feedforward neural network layer. Each sub-layer is equipped with residual connections and layer normalization. After the initial input is decoded by the Transformer decoder, the probability of each word at that position is obtained through a linear layer and a Softmax activation function, generating satellite remote sensing image intelligence description text.

Images