A remote sensing image natural language description generation method for complex scene information prior

Abstract

The application discloses a remote sensing image natural language description generation method for complex scene information prior, and belongs to the field of image processing and analysis. The method comprises the following steps: constructing a Chinese and English combined dataset; constructing visual global features and local features; constructing information prior features; performing intensive processing on the global features and the local features; performing multi-feature cross fusion; and performing natural language description generation on the cross-fused features. The method improves the accuracy of remote sensing image description involving a large number of complex scenes.

Description

Technical Field

[0001] This application relates to the fields of deep learning and image processing, and in particular to a method for generating natural language descriptions of remote sensing images based on prior information about complex scenes. Background Technology

[0002] With the rapid development of science and technology, remote sensing technology has become an indispensable tool in fields such as Earth observation, environmental monitoring, urban planning, and disaster assessment. Remote sensing image analysis, as a core component of this technology, has undergone a profound transformation from traditional manual interpretation to automated processing based on machine learning and deep learning. This shift has not only significantly improved the speed and accuracy of image processing but also greatly expanded the application boundaries of remote sensing technology. Natural language description generation for remote sensing images, as an emerging direction in the field of remote sensing image analysis, aims to transform complex image information into human-understandable natural language descriptions, thereby greatly improving the convenience and intuitiveness of information acquisition. The realization of this technology is of great significance for promoting the widespread application of remote sensing data, enhancing decision support capabilities, and driving interdisciplinary integration; it serves as an important bridge connecting the technological world and human cognition.

[0003] The development of natural language description (NLP) generation for remote sensing images can be traced back to early rule-based methods. These methods attempted to map specific features in images to corresponding NLP descriptions using pre-defined templates and rules. However, as the complexity and diversity of image content increased, the limitations of these methods became increasingly apparent. Subsequently, with the cross-fertilization of natural language processing and computer vision technologies, deep learning-based models began to emerge. These models were able to learn the potential relationships between images and text, generating richer and more accurate descriptions. This advancement marked the beginning of a new era of intelligent NLP generation technology for remote sensing images, propelling the leap from simple feature descriptions to complex scene narratives.

[0004] While existing methods have achieved some success in generating natural language descriptions for remote sensing images, they still face numerous challenges when dealing with complex and ever-changing remote sensing scenes. Complex scenes often contain multi-layered and multi-category ground feature information, as well as intricate spatial relationships, placing higher demands on the model's understanding, generalization, and detail capture capabilities. Existing methods often struggle to accurately capture and express all key information when handling such scenes, resulting in incomplete, inaccurate, or even misleading descriptions. Therefore, we propose an innovative method for generating natural language descriptions for remote sensing images based on prior information for complex scenes. This method fully leverages the advantages of deep learning models in handling large-scale data and incorporates prior knowledge to enhance the model's understanding and parsing capabilities for complex scenes. The aim is to generate more accurate, detailed, and insightful natural language descriptions, effectively overcoming the limitations of existing technologies and promoting the further development of natural language description generation technology for remote sensing images. Summary of the Invention

[0005] This application provides a method for generating natural language descriptions of remote sensing images based on prior information of complex scenes, in order to improve the generation quality of current methods for generating natural language descriptions of remote sensing images.

[0006] To solve the above-mentioned technical problems, the technical solution adopted in this application is as follows:

[0007] A method for generating natural language descriptions of remote sensing images based on prior information in complex scenes, the method comprising the following steps:

[0008] Step 1: Construct a joint Chinese and English natural language description dataset for remote sensing images;

[0009] Step 2: Use the backbone network of a pre-trained image recognition neural network to extract shallow coding features at multiple scales and deep coding features at multiple scales from the input image;

[0010] Visual global features are obtained by fusing shallow coding features from multiple scales, and visual local features are obtained by fusing deep coding features from multiple scales.

[0011] Step 3: Based on prior information, construct a module to extract text prior features from the visual global features;

[0012] Step 4: Enhance the global visual features and local visual features respectively to obtain enhanced global visual features and local visual features;

[0013] Step 5: Use a multi-feature cross-fusion fusion tool to fuse the text prior features and the enhanced visual global features to obtain global fused features, and fuse the text prior features and the enhanced visual local features to obtain local fused features; then fuse the global fused features and the local fused features to obtain the final encoded features.

[0014] Step 6: Decode the final encoded features based on the pre-trained natural language model to generate a natural language description of the remote sensing image.

[0015] Furthermore, step 1 includes: obtaining a publicly available natural language description dataset of remote sensing images as the initial dataset, obtaining an English dataset based on the English descriptions of the remote sensing images in the initial dataset; and re-annotating the English dataset using matching Chinese characters, and constructing a corresponding Chinese dataset based on the Chinese annotations.

[0016] Furthermore, in step 2, the shallow coding features and deep coding features each include two scales, corresponding to two stages.

[0017] Furthermore, in step 2, the pre-trained backbone network is a Transformer-based network structure or a convolutional neural network-based network structure.

[0018] Furthermore, in step 2, the number of channels of the extracted shallow coding features and deep coding features at multiple scales gradually increases, while the height and width of the features gradually decrease, according to the forward propagation direction.

[0019] Furthermore, step 2 also includes fine-tuning some network parameters in the backbone network, wherein the fine-tuned network parameters are network parameters of the network layer used to extract deep coding features.

[0020] Furthermore, in step 2, a layer-by-layer fusion method is used to fuse shallow coding features at multiple scales. Following the forward propagation direction of the backbone network, the shallow / deep coding features at each scale are traversed sequentially. Through convolution and downsampling operations, the feature dimension of the shallow / deep coding feature at the current scale is adjusted to the dimension of the next shallow / deep coding feature. Then, the features are superimposed, and the superimposed shallow / deep coding feature is used as the new next shallow / deep coding feature. This process of feature adjustment and superposition continues until the shallow / deep coding features at the last scale are reached.

[0021] For example, for shallow encoded features that include two scales, the dimension of the first scale is adjusted to the second scale and then the features are superimposed to obtain the visual global features.

[0022] Furthermore, in step 3, the prior information construction module's processing includes:

[0023] Step 3-1: Define the visual global feature as G∈R C×H×W ;

[0024] The visual global features G are processed by global max pooling and global average pooling respectively to obtain global vectors g1 and g2;

[0025] The global vectors g1 and g2 are then passed through a pre-trained multi-attribute predictor with shared weight parameters to obtain the attribute prediction probability p1∈R of the scene attributes of the remote sensing image. d and p2∈R d , where d represents the number of scene attributes in the remote sensing image;

[0026] Step 3-2: Sort the elements in the multi-attribute prediction probabilities p1 and p2, then select the top K largest image attribute prediction probabilities and convert them into one-hot vector representations to obtain matrix U∈R. d×k ;

[0027] Step 3-3: Construct a learnable matrix M∈R t×d Make such that the sum of matrix U∈R d×k Perform matrix multiplication to obtain a new matrix E∈R t×k , where t is the set encoding length;

[0028] Steps 3-4: Expand matrix E and then copy it spatially to obtain feature map E. f ∈R B×H×W Wherein, the number of feature map channels B = t × k;

[0029] Feature map E f The text prior features P∈R are obtained by adjusting the dimensions through a 1×1 convolutional layer. C ×H×W .

[0030] Prior information features can provide more information about scene categories, thus playing an auxiliary role in understanding the semantic and spatial relationships of ground objects in complex scenes.

[0031] Furthermore, in step 4, the visual global features are enhanced through the set global feature enhancement module, including:

[0032] Step 4-1-A: Define the visual global feature as G∈R C×H×W Where C is the number of channels, and H×W is the height and width of the feature;

[0033] We perform convolution operations on the global feature G using parallel separable convolutions with kernel sizes of 3x3, 5x5, and 7x7, respectively.

[0034] The results of the three convolution operations are stacked according to the channel dimension, and then the number of feature channels is adjusted to be consistent with the visual global feature G through a convolutional layer with a 1x1 kernel, resulting in feature S∈R. C×H×W ;

[0035] Split feature S into two new features S1∈R according to the channels. C / 2×H×W and S2∈R C / 2×H×W ;

[0036] Feature S1 is obtained by horizontal global average pooling. _h ∈R C / 2×H×1 Feature S2 is obtained by vertical global average pooling. _w ∈R C / 2×1×W ;

[0037] For feature S _h Dot product of S1 and S1 yields feature T1∈R C / 2×H×W , for S _w Dot product of S2 and S2 yields feature T2∈R C / 2×H×W ;

[0038] The enhanced visual global feature T∈R is obtained by superimposing features T1 and T2. C×H×W .

[0039] Furthermore, in step 4, the visual local features are enhanced using the set local feature enhancement module, including:

[0040] Step 4-2-A: Define the visual local features as L∈R C×H×W Where C is the number of channels, and H×W is the height and width of the feature;

[0041] The visual local feature L is passed sequentially through a 3x3 convolutional layer and a ReLU activation function to obtain feature L. / ∈R C×H×W ;

[0042] Step 4-2-B: Transfer feature L / ∈R C×H×W They are divided into four features L1∈R according to the channel order. C / 4×H×W L2∈R C / 4×H×W L3∈R C / 4×H×W and L4∈R C / 4×H×W ;

[0043] L is obtained by superimposing features L1 and L3. 1-3 ∈R C / 2×H×WThe sum of features L2 and L4 is used to obtain L. 2-4 ∈R C / 2×H×W The sum of features L1 and L4 is used to obtain L. 1-4 ∈R C / 2×H×W The sum of features L2 and L3 is used to obtain L. 2-3 ∈R C / 2×H×W This process involves cross-overlaying features along the channel dimension in local features, which can enhance the perception of features between channels, thereby enhancing local features.

[0044] Step 4-2-C: For feature L 1-3 L 2-4 L 1-4 and L 2-3 Channel attention will be performed separately to obtain enhanced features. and and

[0045] Step 4-2-D: Enhance the features and The feature U∈R is obtained by channel superposition. C×H×W Enhanced features and The feature V∈R is obtained by channel superposition. C×H×W ;

[0046] Then, the features U and V are added pixel by pixel to obtain the enhanced visual local features O∈R. C×H×W .

[0047] Furthermore, the specific execution process of the multi-feature cross-fusion processor in step 5 includes:

[0048] Let the prior features of the text be P∈R C×H×W The enhanced visual global features are T∈R C×H×W The enhanced visual local features are O∈R C×H×W ;

[0049] Perform a joint spatial attention operation on the enhanced visual global features T and the text prior features P to obtain the features.

[0050] A joint spatial attention operation is performed on the enhanced visual local feature O and the text prior feature P to obtain the feature.

[0051] feature and characteristics The first fused coding feature Y∈R is obtained by stacking the channels and then passing it through a 3x3 convolutional layer and an activation function. C×H×WBy fusing multiple features, a deep understanding of the semantic relationships between ground objects in complex scenes can be effectively achieved.

[0052] The first fused encoded feature Y is passed through a dilated spatial convolutional pooling pyramid (ASPP) module to obtain the final encoded feature Q∈R. C×H×W By further mining multi-scale information based on fused features, a more comprehensive understanding of complex scenes can be achieved.

[0053] Furthermore, a joint spatial attention operation is performed on the enhanced visual global features T and the text prior features P to obtain the features. include:

[0054] The visual global feature T is obtained by passing it through two convolutional layers with a kernel of 1×1 to obtain features T1 and T2 respectively;

[0055] Adjust the dimension of feature T2 to H×W×C, and then perform matrix multiplication with feature P to obtain the spatial attention graph map1 = (H×W)×(H×W); and then apply the softmax function to activate the spatial attention graph map1 with weights.

[0056] Then, the feature T1 and the spatial attention graph map2 after weight activation are added pixel by pixel to obtain the feature.

[0057] Furthermore, a joint spatial attention operation is performed on the enhanced visual local feature O and the text prior feature P to obtain the feature.

[0058] Visual local features O are obtained by passing them through two convolutional layers with 1×1 kernels to obtain features O1 and O2 respectively;

[0059] Adjust the dimension of O2 to H×W×C, and then perform matrix multiplication with feature P to obtain the spatial attention graph map2=(H×W)×(H×W), and then use the softmax function to perform weight activation on the spatial attention graph map2;

[0060] Then, the features O1 and the spatial attention graph map2 after weight activation are added pixel by pixel to obtain the feature.

[0061] The technical solution provided in this application brings at least the following beneficial effects:

[0062] This application utilizes a pre-trained neural network to construct global and local features, and builds prior information features based on the global features. These prior information features enable a better understanding of the spatial and semantic relationships between ground objects in complex scenes. Furthermore, the cross-fusion of global and local features with prior information features can effectively improve the accuracy of image description of remote sensing images. Attached Figure Description

[0063] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0064] Figure 1 This is a structural diagram of the global feature enhancement module in an embodiment of this application;

[0065] Figure 2 This is a structural diagram of the prior information construction module in an embodiment of this application;

[0066] Figure 3 This is a structural diagram of a multi-cross fusion device according to an embodiment of this application. Detailed Implementation

[0067] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions of the embodiments of this application will be described in detail and completely below with reference to the accompanying drawings. Obviously, the embodiments described with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0068] This application provides a method for generating natural language descriptions of remote sensing images based on prior information of complex scenes, aiming to improve the performance of current methods for generating natural language descriptions of remote sensing images.

[0069] In one embodiment, the remote sensing image natural language description generation method proposed in this application includes the following steps:

[0070] Step 1: Construct a combined Chinese and English dataset.

[0071] We obtained open-source and publicly available natural language description datasets for remote sensing images from both domestic and international sources. We retained the original English datasets, referenced the English descriptions, and then re-annotated them using appropriate Chinese.

[0072] In this embodiment, to further ensure annotation quality, the following requirements should be noted when constructing the joint Chinese and English dataset:

[0073] 1) In the process of annotating remote sensing images, it is not only necessary to accurately identify and annotate the target objects, scene backgrounds, and their hierarchical relationships in the image, but also to use deep learning technology to deeply mine the implicit information in the image, such as the shape features, texture details, and color attributes of the target objects. This information will be structurally integrated into the annotation text, thereby generating a more detailed and comprehensive image description.

[0074] 2) First, preliminary annotation text is automatically generated using natural language generation technology. Then, it is manually reviewed and optimized by professionals to ensure that the annotation text conforms to language standards, accurately and clearly conveys key information from the remote sensing image, and maintains fluent and natural written expression.

[0075] 3) In the final stage of determining the labeled text, each piece of text will be carefully checked to remove redundant information and vague descriptions, ensuring that each label is closely related to the image content and can accurately reflect the target object and scene features in the image.

[0076] 4) To ensure annotation quality, a comprehensive annotation quality evaluation system will be established to regularly evaluate and inspect completed annotation deliverables. Based on the evaluation results, targeted feedback and improvement suggestions will be provided to continuously improve the accuracy and overall quality of annotation work, ensuring high-level output of annotation results.

[0077] Step 2: Construction of global and local visual features.

[0078] In this embodiment, a neural network based on a Transformer structure or CNN pre-trained on the ImageNet dataset is used to obtain encoded features of the input image at multiple scales, and based on this, global and local visual features are constructed.

[0079] The specific steps for constructing global and local visual features in the above process are as follows:

[0080] Step 2-1: Remove the last average pooling layer and fully connected layer of the pre-trained neural network based on Transformer or CNN; that is, extract the encoding features of the input image at multiple scales based on the backbone network of the pre-trained image recognition neural network. This backbone network constitutes the pre-trained encoder of this application. In this embodiment, the encoding features of four stages are extracted from the backbone network in the forward propagation direction. The number of channels of the four stages of encoding features gradually increases, and the feature resolution of the encoding features gradually decreases, that is, the height and width gradually decrease.

[0081] Step 2-2: Freeze the parameters of the first and third stages in the pre-trained model, and fine-tune the parameters of the second and fourth stages during training.

[0082] Steps 2-3: Obtain the remote sensing image coding features F of the four stages in the encoder. n Then, based on the encoded feature F n Construct global and local visual features.

[0083] Given a remote sensing image, input I∈R C×H×W Visual features F from remote sensing images at different scales are extracted using a pre-trained encoder. n :

[0084] F n =Encoder(I; W1, W2)

[0085] Where n represents the different feature extraction stages, W1 represents the weight parameters frozen in the pre-trained model, and W2 represents the weight parameters that need to be fine-tuned in the pre-trained model.

[0086] Visual feature F n It includes four stages of encoded features from the pre-trained model: F1, F2, F3, and F4. In these four stages, the number of channels gradually increases, while the height and width gradually decrease. The feature obtained by combining F1 and F2 is used as the local feature, and the result of combining F3 and F4 is used as the global feature. Considering the inconsistent dimensions of the feature maps, before combining the features, convolution and downsampling operations are needed to adjust the size of F1 to be consistent with F2, and the size of F3 to be consistent with F4.

[0087] Step 3: Construct prior information features.

[0088] Based on the constructed visual global features, information prior features are constructed through the information prior construction module.

[0089] See Figure 1 In this embodiment, the specific execution process of the information prior construction module includes:

[0090] Step 3-A: Set the global feature to G∈R C×H×W Where C is the number of channels, and H×W is the feature resolution, i.e., height and width. First, the global features are processed by global max pooling and global average pooling respectively, yielding global vectors g1 and g2. Then, global vectors g1 and g2 are each processed by a pre-trained multi-attribute predictor with shared weight parameters to obtain the multi-attribute prediction probability p1∈R of the image scene. d and p2∈R d Where d represents the number of scene attributes, such as airport, forest, school, city, etc.

[0091] Step 3-B: Calculate the multi-attribute prediction probability p1∈R predicted in the above steps.d and p2∈R d The elements in the matrix are sorted in descending order. Then, the top k (preset values) attribute prediction probabilities are selected by combining the two probability prediction results and transformed into one-hot vector representations to obtain matrix U∈R. d×k .

[0092] Step 3-C: Construct a learnable matrix M∈R t×d Make such that the sum of matrix U∈R d×k Perform matrix multiplication to obtain a new matrix E∈R t×k Where t is the set encoding length.

[0093] Step 3-D: Expand matrix E and then copy it spatially to obtain feature map E. f ∈R B×H×W Where B = t × k. Next, the feature map E... f By adjusting the feature map to a dimension of C×H×W through a 1×1 convolution, we obtain the text prior features P∈R. C×H×W .

[0094] Step 4: Enhancement processing of global and local features.

[0095] The global feature enhancement module and the local feature enhancement module are used to enhance the constructed global features and local features respectively.

[0096] See Figure 2 In this embodiment, the specific execution process of the global feature enhancement module is as follows:

[0097] Step 4-1-A: Set the global feature to G∈R C×H×W Global features are processed using parallel depthwise separable convolutions with kernel sizes of 3x3, 5x5, and 7x7, respectively. The process is expressed as follows:

[0098] G1 = DWConv3 × 3(G)

[0099] G2 = DWConv5 × 5(G)

[0100] G3 = DWConv7 × 7(G)

[0101] Here, G1, G2, and G3 represent the features processed by depthwise separable convolutions with kernel sizes of 3x3, 5x5, and 7x7, respectively, and all dimensions are consistent with G. DWConv3×3, DWConv5×5, and DWconv7×7 represent depthwise separable convolutions with kernel sizes of 3x3, 5x5, and 7x7, respectively.

[0102] Step 4-1-B: Stack the features G1, G2, and G3 obtained in the above steps according to the channel dimension, and adjust the number of channels to match the feature G using a 1x1 convolution. This is expressed as follows:

[0103] S=Conv1×1(ConaCat(G1,G2,G3))

[0104] Step 4-1-C: Apply the features S∈R obtained in the above steps C×H×W According to the channel split, it is divided into two new features S1∈R C / 2×H×W and S2∈R C / 2×H×W .

[0105] Next, feature S1 will undergo horizontal global average pooling to obtain feature S. _h ∈R C / 2×H×1 Feature S2 will be obtained by vertical global average pooling. _w ∈R C / 2×1×W Then, S _h Dot product of S1 and S1 yields feature T1∈R C / 2×H×W S _w Dot product of S2 and S2 yields feature T2∈R C / 2×H×W Finally, the features are superimposed to obtain the output feature T∈R. C×H×W .

[0106] In one embodiment, the specific execution process of the local feature enhancement module in step 4 is as follows:

[0107] Step 4-2-A: Set the local features as L∈R C×H×W Feature L is first processed through a 3x3 convolution and a ReLU activation function to reassemble the spatial features, resulting in feature L. / ∈R C×H×W .

[0108] Step 4-2-B: The feature L obtained in the above steps / ∈R C×H×W The channels are divided into four features L1∈R according to their order. C / 4×H×W L2∈R C / 4×H×W L3∈R C / 4×H×W and L4∈R C / 4×H×W Then, L1 and L3 are superimposed to obtain L. 1-3 ∈R C / 2×H×W L2 and L4 are superimposed to obtain L 2-4 ∈R C / 2×H×W The sum of features L1 and L4 is used to obtain L. 1-4 ∈R C / 2×H×W L2 and L3 are superimposed to obtain L 2-3 ∈R C / 2×H×WThe above process involves cross-overlaying features along the channel dimension in local features, which can enhance the perception of features between channels, thereby enhancing local features.

[0109] Step 4-2-C: Feature L obtained from the above steps 1-3 L 2-4 L 1-4 and L 2-3 Channel attention will be performed separately to obtain enhanced features. and and

[0110] Step 4-2-D: Then apply the features obtained in the above steps and The feature U∈R is obtained by channel superposition. C ×H×W ,feature and The feature V∈R is obtained by channel superposition. C×H×W The features U and V are superimposed to obtain the final output feature O∈R. C×H×W .

[0111] Step 5: Multi-feature fusion. A multi-feature cross-fusion tool is used to fuse global features, local features, and prior textual features.

[0112] See Figure 3 The specific execution process of the multi-cross fusion processor in this embodiment is as follows:

[0113] Step 5-A: Define the enhanced global features as T∈R C×H×W The prior features of the text are P∈R C×H×W Next, features T and P undergo a joint spatial attention operation. Specifically, first, the global feature T is processed by two 1x1 convolutions to obtain features T1 and T2. Then, feature T2 is multiplied by feature P to obtain a spatial attention graph map = (H×W)×(H×W). The spatial attention heatmap map is then weighted and activated using a softmax function. Finally, the feature obtained by multiplying feature T1 and the spatial attention heatmap is summed pixel-by-pixel with T to obtain the output.

[0114] Step 5-B: Define the enhanced local features as O∈R C×H×W The prior features of the text are P∈R C×H×WNext, features O and P undergo a joint spatial attention operation. Specifically, first, local feature O is processed through two 1x1 convolutions to obtain features O1 and O2. Then, feature O2 is multiplied by feature P to obtain a spatial attention graph map = (H×W)×(H×W). The spatial attention heatmap map is then weighted and activated using the softmax function. Finally, the feature obtained by multiplying feature O1 and the spatial attention graph pixel by pixel is added to O to obtain the output. Through the above-mentioned multi-feature cross-fusion, a deep understanding of the semantic relationships of ground objects in complex scenes can be effectively achieved.

[0115] Step 5-C: Features obtained from the above steps and characteristics The channel-dimensional data is stacked and then subjected to 3x3 convolution and activation functions to obtain the fused encoded features Y∈R. C×H×W .

[0116] Step 5-D: The features Y obtained in the above steps are passed through a dilated spatial convolutional pooling pyramid (ASPP) module to obtain the final encoded features Q∈R. C×H×W By further mining multi-scale information based on fused features, a more comprehensive understanding of complex scenes can be achieved.

[0117] Step 6: Natural Language Description Generation. Based on the current pre-trained natural language model, the features fused in Step 5 are decoded to generate the final natural language description.

[0118] Example

[0119] In one embodiment, the method for generating natural language descriptions of remote sensing images based on prior information of complex scenes provided in this application specifically includes the following steps:

[0120] Step S1: Construct a joint Chinese and English dataset. Obtain publicly available open-source natural language description datasets for remote sensing images from both domestic and international sources. Retain the original English dataset, refer to the English descriptions, and re-annotate using appropriate Chinese.

[0121] Step S2: Construction of global and local visual features. A pre-trained neural network based on a Transformer or CNN architecture, trained on the ImageNet dataset, is used to acquire the encoded features of the input image, and these features are used to construct global and local visual features. In practice, a pre-trained ResNe50 network can be chosen as the feature extraction network. The input remote sensing image size is I∈R. 3×512×512 The input image, after passing through ResNe50, will yield four stages of encoded features: F1∈R256×128×128 F2∈R 512×64×64 F3∈R 1024×32×32 and F4∈R 2048×16×16 Next, F1 will downsample and convolve using max pooling to make the feature dimension consistent with F2, and then superimpose it with F2 along the channel dimension to obtain the local feature L∈R. 1024×64×64 Similarly, F3 uses max pooling for downsampling and convolution to keep the feature dimension consistent with F4, and then stacks it with F4 along the channel dimension to obtain the global feature G∈R. 4096×16×16 .

[0122] Step S3: Construction of prior information features. First, global features G∈R 4096×16×16 After undergoing global max pooling and global average pooling respectively, a global vector g1∈R is obtained. 4096×1×1 and g2∈R 4096×1×1 Then, the global vectors g1 and g2 are respectively passed through a pre-trained multi-attribute predictor with shared weight parameters to obtain the multi-attribute prediction probability p1∈R. d and p2∈R d Where d represents the number of attributes. The multi-attribute prediction probability p1∈R predicted in the above steps is... d and p2∈R d The elements in the matrix are sorted in descending order. Then, the top k attribute prediction probabilities are selected by combining the two probability prediction results and transformed into one-hot vector representations to obtain matrix U∈R. d×k Next, we construct a learnable matrix M∈R. t×d Make such that the sum of matrix U∈R d×k Perform matrix multiplication to obtain a new matrix E∈R t×k Where t is the set encoding length. Next, the matrix E is expanded and then spatially copied to obtain the feature map E∈R. B×H×W Where B = t × k. Finally, the feature map E is adjusted to have the dimension P ∈ R through a 1 × 1 convolution. 4096×16×16 .

[0123] Step S4: Enhancement processing of global and local features. Global features are enhanced through the global feature enhancement module. Specifically, the global features are G∈R 4096×16×16 Features G1∈R were obtained by processing the data through parallel depthwise separable convolutions with kernel sizes of 3x3, 5x5, and 7x7, respectively. 4096×16×16 G2∈R 4096×16×16 and G3∈R 4096×16×16 The results are stacked according to the channel dimension, and the number of channels is adjusted to be consistent with G through 1x1 convolution to obtain the feature S∈R. 4096×16×16 Next, feature S is split into two new features S1∈R according to the channels.2048×16×16 and S2∈R 2048×16×16 Next, feature S1 will undergo horizontal global average pooling to obtain feature S. _h ∈R 2048×16×1 Feature S2 will be obtained by vertical global average pooling. _w ∈R 2048×1×16 Then, S _h Dot product of S1 and S1 yields the same features as S. _w The features obtained by performing a dot product with S2 are superimposed to obtain the output feature T∈R. 4096×16×16 .

[0124] Local features are enhanced through a local feature enhancement module. Local features L∈R 1024×64×64 First, spatial features are re-integrated using a 3x3 convolution and ReLU activation function to obtain feature L. / ∈R 1024×64×64 Next, feature L / The channels are divided into four features L1∈R according to their order. 256×64×64 L2∈R 256×64×64 L3∈R 256×64×64 and L4∈R 256×64×64 Then, L1 and L3 are superimposed to obtain L. 1-3 ∈R 512×64×64 L2 and L4 are superimposed to obtain L 2-4 ∈R 512×64×64 The sum of features L1 and L4 is used to obtain L. 1-4 ∈R 512×64×64 L2 and L3 are superimposed to obtain L 2-3 ∈R 512×64×64 Next, feature L 1-3 L 2-4 L 1-4 and L 2-3 Channel attention will be performed separately to obtain enhanced features. and and Finally, features and The feature U∈R is obtained by channel superposition. 512×64×64 ,feature and The feature V∈R is obtained by channel superposition. 512×64×64 The features U and V are superimposed to obtain the final output feature O∈R. 1024×64×64 .

[0125] Step S5: Multi-feature cross-fusion. The enhanced global features are T∈R 4096×16×16 And the enhanced local features are O∈R 1024×64×64 The information prior features are respectively P∈R4096×16×16 Cross-fusion is performed. Taking the cross-fusion of global features and text prior features as an example, firstly, the global feature T is processed by two 1x1 convolutions to obtain feature T1∈R. 4096×16×16 and T2∈R 4096×16×16 Then, feature T2 is multiplied by feature P using a matrix multiplication process with its dimensions changed to HxWxC, resulting in a spatial attention graph map (map = (16×16)×(16×16)). This spatial attention heatmap map is then weighted and activated using a softmax function. Next, the feature obtained by matrix multiplying feature T1 and the spatial attention heatmap map is summed pixel-by-pixel with T to obtain the output. The cross-fusion process of enhanced local features and prior text features is consistent with the above description, yielding the output features. However, before cross-fusion, the local features O need to be adjusted to match the dimensions of P through convolution and upsampling. Finally, features and characteristics The final encoded feature Y is obtained by adding pixels one by one. The feature Y obtained from the above steps is then passed through a dilated spatial convolutional pooling pyramid (ASPP) module to obtain the final encoded feature Q∈R. 4096×16×16 .

[0126] Step S6: Based on the current pre-trained natural language model, decode the fused features from Step 5 to generate the final natural language description.

[0127] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0128] Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include at least one of those features.

[0129] Any process or method described in the flowchart or otherwise in this specification can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order according to the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0130] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0131] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0132] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the application, and are not intended to limit them; although the application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the application.

Claims

1. A method for generating natural language description of remote sensing image based on complex scene information prior, characterized in that, The method comprises the following steps: Step 1: constructing a remote sensing image natural language description dataset combined with Chinese and English; Step 2: extracting shallow layer coding features of multiple scales and deep layer coding features of multiple scales of the input image by using a backbone network of a pre-trained image recognition neural network; Fusing the shallow layer coding features of multiple scales to obtain visual global features, and fusing the deep layer coding features of multiple scales to obtain visual local features; Step 3: extracting text prior features from the visual global features based on a prior information construction module; Step 4: respectively performing reinforcement processing on the visual global features and the visual local features to obtain reinforced visual global features and reinforced visual local features; Step 5: performing feature fusion on the text prior features and the reinforced visual global features by using a multi-feature cross fusion device to obtain global fusion features, and performing feature fusion on the text prior features and the reinforced visual local features to obtain local fusion features; and then performing feature fusion on the global fusion features and the local fusion features to obtain final coding features; Step 6: decoding the final coding features based on a pre-trained natural language model to generate a natural language description of the remote sensing image. In step 3, the processing of the prior information construction module comprises: Step 3-1: Define visual global features as ; obtaining visual global features respectively through global maximum pooling and global average pooling processing, obtaining global vectors and ; The global vector and The attribute prediction probability of the scene attribute of the remote sensing image is obtained by a multi-attribute predictor with a shared weight parameter and pre-training respectively and wherein, represents the number of scene attributes of the remote sensing image; Step 3-2: Synthesize multi-attribute prediction probability and Sort the elements in the matrix and select the top K largest image attribute prediction probabilities, and convert them into one-hot vector representation to obtain the matrix ; Step 3-3: Construct a learnable matrix and the matrix Matrix multiplication is performed to obtain a new matrix wherein, is the set encoding length; Step 3-4: matrix unfolding and then spatially replicating to get feature map ; wherein the number of channels of the feature map ; The feature map By a convolution kernel for 1 1 convolution layer dimension adjustment, get text prior feature .

2. The method of claim 1, wherein, Step 1 comprises: obtaining a public remote sensing image natural language description dataset as an initial dataset, obtaining an English dataset based on English descriptions of remote sensing images in the initial dataset; and based on the English dataset, using matched Chinese to perform re-labeling, and constructing a corresponding Chinese dataset based on the Chinese labeling.

3. The method of claim 1, wherein, In step 2, the pre-trained backbone network is a network structure based on a Transformer or a network structure based on a convolutional neural network.

4. The method of claim 1, wherein, In step 2, part of the network parameters in the backbone network are subjected to fine-tuning learning, wherein the part of the network parameters subjected to fine-tuning learning are network parameters of network layers used for extracting deep layer coding features.

5. The method of claim 1, wherein, In step 2, the multi-scale shallow layer coding features are fused by using a layer-by-layer fusion method, and each scale of shallow / deep layer coding features is sequentially traversed in a forward propagation direction of the backbone network, the feature dimension of the current scale of shallow / deep layer coding features is adjusted to the dimension of the next shallow / deep layer coding feature through convolution and down-sampling operation, then the shallow / deep layer coding features after the current superposition are taken as new next shallow / deep layer coding features, and the feature adjustment and superposition of the new next shallow / deep layer coding features and the next shallow / deep layer coding features are continued until the last scale of shallow / deep layer coding features.

6. The method of claim 1, wherein, In step 4, the visual global features are reinforced by using a set global feature reinforcement module, which comprises: Step 4-1-A: Define visual global features as where C is the number of channels, is the height and width of the feature; The global feature is subjected to convolution operation by parallel depth separable convolutions with kernel sizes of 3x3, 5x5 and 7x7 respectively ; The three convolution operation results are stacked in the channel dimension, and a convolution layer with a kernel of 1x1 is used to adjust the number of feature channels to be consistent with the visual global feature , obtaining the feature ; Splitting a feature Splitting a feature according to a channel into two new features and ; obtaining features obtaining features ; obtaining features obtaining features ; dot product of features and dot product of features , and dot product of features dot product of features ; characteristics with superimposed to obtain enhanced visual global features .

7. The method of claim 1, wherein, In step 4, the visual local features are reinforced by using a set local feature reinforcement module, which comprises: Step 4-2-A: Define visual local features as where C is the number of channels, is the height and width of the feature; The visual local feature is The visual local feature is ; Step 4-2-B: Feature Divide into four features in channel order , , and ; Features and are superimposed to obtain , the feature and are superimposed to obtain , the feature and are superimposed to obtain , the feature and are superimposed to obtain ; Step 4-2-C: Channel attention on features , , and will be respectively performed on the channel attention, obtaining the enhanced features and , and ; Step 4-2-D: enhanced features and features from channel overlay , enhanced features and features from channel overlay ; The features U and V are then added pixel by pixel to obtain the enhanced visual local feature .

8. The method of claim 1, wherein, The specific execution process of the multi-feature cross fusion device in step 5 comprises: The set text prior feature is , the enhanced visual global feature is , and the enhanced visual local feature is ; enhanced visual global features and text prior features perform joint spatial attention operation to obtain features ; The enhanced visual local features are combined with the text prior features to perform a joint spatial attention operation to obtain feature and text prior features ;​ Features And features According to the channel dimension superposition, sequentially through the convolution layer with the convolution kernel of 3x3 and the activation function to obtain the first fusion encoding feature ; obtaining a first fused encoded feature obtaining a final encoded feature through a cavity spatial convolution pyramid module .

9. The method of claim 8, wherein, enhanced visual global features and text prior features perform a joint spatial attention operation to obtain features comprising: Visual global features two convolution kernels are 1 convolution layer with 1 and ; adjusting features of the dimension , and matrix multiplication is performed on the features to obtain a spatial attention relationship graph ; and the spatial attention relationship graph is activated by a softmax function Again, the features and the spatial attention graph after weight activation pixel-wise addition to obtain features ; The enhanced visual local features are combined with the text prior features to perform a joint spatial attention operation to obtain feature and text prior features ;​ Visual local features two convolution kernels are 1 convolution layer with 1 and ; adjustment dimension of the feature , and the feature is multiplied by the matrix to obtain a spatial attention relationship graph , and the spatial attention relationship graph is activated by a softmax function. Again, the features and the spatial attention graph after weight activation are added pixel by pixel to obtain the features .

Images