A remote sensing image description generation method based on a two-stage generation and sorting process

Abstract

This invention provides a two-stage remote sensing image description generation method based on generation and ranking. First, addressing the issues of large differences in target scale, uneven distribution, and weak yet dense targets in remote sensing images, a multi-scale target detection network is proposed to extract regional features of targets. Second, to address the problem that existing image generation mechanisms only consider local target features and lack global features, an end-to-end image generation model integrating global and local features is proposed. Third, to address the exposure bias problem in the decoding process of existing generation models, ranking is introduced into the image generation model, proposing a two-stage model based on generation followed by ranking. Through these three steps, this invention can automatically perform semantic descriptions of remote sensing images, generating highly readable text descriptions that reflect the activity of targets.

Description

Technical Field

[0001] This invention relates to an image description generation method, and more particularly to a remote sensing image description generation method based on a two-stage process of generation and sorting. Background Technology

[0002] The goal of remote sensing image description generation is to convert images into text. This involves not only comprehensively capturing the content of the image but also understanding scene information, object attributes, and relationships between objects. This semantic understanding provides a higher-level abstract representation of the image, making the generated text easier for humans to understand and accept. Therefore, remote sensing image description generation is a cross-modal conversion technology that enables effective information exchange and conversion between images and text.

[0003] Existing image generation mechanisms do not fully consider the inherent properties of remote sensing images, such as large differences in target scale, uneven distribution, and small and dense targets. In addition, existing schemes only consider local features of the target and lack the fusion of global features. Furthermore, there are exposure bias issues in the decoding process of existing generation models. Summary of the Invention

[0004] Purpose of the invention: The technical problem to be solved by the present invention is to provide a remote sensing image description generation method based on a two-stage generation and sorting process, which addresses the shortcomings of the existing technology.

[0005] To address the aforementioned technical problems, this invention discloses a remote sensing image description generation method based on a two-stage process of generation and sorting, comprising the following steps:

[0006] Step 1: Construct a multi-scale target detection network to extract regional features of the target, i.e., the remote sensing image. Specific methods include:

[0007] Step 1-1: Based on the Swin transformer skeleton network, extract the multi-scale feature map of the target, i.e., the remote sensing image, specifically including:

[0008] Using a multi-scale Swing transformer as the backbone network, the remote sensing image is downsampled through the backbone network into feature maps of 1 / 8, 1 / 16, 1 / 32 and 1 / 64 of the original remote sensing image size, which are the multi-scale feature maps.

[0009] Let the resolution of the input remote sensing image be H*W, then the output multi-size feature map is: Among them, feature map V l The resolutions are 8×8, 16×16, 32×32, and 64×64, respectively, where l represents the l-th feature map, L b Indicates the number of feature maps;

[0010] Step 1-2: Input the multi-size feature map extracted in Step 1-1 into the deformabledetr-based target detection network to obtain the region features of the remote sensing image, as follows:

[0011] Input the multi-scale feature map extracted in step 1-1 into the deformabledetr-based object detection network. and N target query vectors R d Let r represent a set of d-dimensional real vectors, where d represents the vector dimension and r is the vector dimension. i This represents the i-th target query vector;

[0012] The deformabledetr-based object detection network updates the target query vector after three or more deformable layers, taking the hidden state of the last layer. The target region feature R.

[0013] Step 2: Construct an end-to-end image-text generation model that integrates global and local features. Use this model to generate different descriptive texts for the remote sensing image, specifically including:

[0014] Step 2-1: Based on a multi-layer self-attention mechanism, extract the grid features of the remote sensing image. Specific methods include:

[0015] The feature map obtained from the last layer of the Swing Transformer backbone network is taken as input, and the dimension of this feature map is... Where M is a 64-fold downsampling factor, i.e., M = H / 64 * W / 64. Representing the dimension of the feature vector, the above feature map passes through a fully connected layer. Map its vector dimension to a d-dimensional vector After a group containing L g The self-attention transformer module of the layer updates G0, and takes the hidden state vector of the last layer as the grid feature G∈R of the remote sensing image. M*d ;

[0016] Step 2-2: Use a feature fusion network to fuse grid features and region features. Specific methods include:

[0017] The feature fusion network is an L-layer network. For the l-th layer cross-attention network, its input consists of three parts: the text decoded by the decoder, i.e., the word vector denoted as X′. l ={x′ i}, the target's regional features R, and the grid features G of the remote sensing image;

[0018] The specific process of fusion is as follows:

[0019] Step 2-2-1, convert the word vector {x′} i As a query, the region feature R is used as the key and value for multi-head attention interaction to generate attention weights. Similarly, the word vector {x′ i The attention weights are obtained by performing multi-head attention interaction between the grid feature G and the grid feature G.

[0020] Step 2-2-2, perform vector concatenation to obtain and The concatenated vector is then projected onto d dimensions and transformed into probabilities using the sigmoid function. and Specifically as follows:

[0021]

[0022]

[0023]

[0024] Among them, b g Indicates grid feature bias, b r LN() indicates regional feature bias, and LN() indicates layer normalization. W represents element-wise multiplication. r Represents the weight of regional features. This represents the attention value of time step j for layer l;

[0025] Step 2-2-3, for each vector in the l-th layer After weighted summation and layer normalization, the attention weights for layer l are obtained.

[0026] Steps 2-3 involve training an autoregressive image-text generation model g(.) as the decoder, which selects the text with the highest probability as the decoded text. Specifically, this includes:

[0027] The training objective is to maximize the likelihood of a given input image and a reference description text. conditional probability Specifically as follows:

[0028]

[0029] Where I represents the input remote sensing image, c represents the encoded representation of global and local features of the image, m represents the text length, and y represents the input remote sensing image. k This represents the decoding probability at the current time step k. This represents the text encoding of the current character up to the current time step k. This indicates the network parameters for decoding.

[0030] Step 2-4: Repeat steps 2-2 to 2-3 until candidate description text is generated;

[0031] Steps 2-5: Generate other candidate description texts, specifically including:

[0032] The text is generated using a bundle constraint method, with a similarity penalty factor added to ensure the diversity of the generated text. This results in n candidate descriptive texts, namely, candidate descriptive texts S1,...,S... n And calculate the difference between each candidate description text and the actual labeled description text. The CIDER values ​​are then sorted in descending order as follows:

[0033]

[0034] in, This represents the candidate description text with a ranking of m.

[0035] Step 3: Introduce the sorting method into the end-to-end image and text generation model described in Step 2, and construct a two-stage model based on generation followed by sorting. That is, sort the different descriptive texts generated in Step 2 according to the sorting method, and select the optimal descriptive text.

[0036] Step 3-1: Construct a ranking network and perform ranking, that is, use a single-stream multimodal network as a ranking machine to rank according to the semantic similarity scores calculated by it.

[0037] The single-stream multimodal network takes an image and descriptive text as input and outputs a semantic similarity score between them, specifically including:

[0038] The descriptive text is encoded using word embedding to obtain a word vector representation;

[0039] The image is processed through a swin transformer to obtain a semantic representation of the image, and then through a projection layer to obtain the same dimension as the word vector representation of the summary text.

[0040] Word vector representations and semantic representations are concatenated and input into the character encoding layer of the BERT pre-trained model. After BERT transformer encoding, the last hidden layer is taken as the final semantic representation. After passing through a fully connected layer with an output of 1, the semantic distance between the image and the descriptive text, i.e. the semantic similarity score, is obtained.

[0041] Step 3-2: Calculate the contrastive learning loss:

[0042] A ranking loss based on contrastive learning is used to measure the distance L between the image and the text summary, specifically including:

[0043]

[0044] in, It is the triplet loss that maximizes the boundary. This represents the ranking loss; its optimization objective is: for a given image, the greater the ranking difference between two candidate text summaries, the greater the aforementioned distance;

[0045] in the formula and Let m1, p, and q represent the candidate abstracts ranked in descending order of cider, respectively. This represents the actual text used to annotate the corresponding image, λ. ij = (ji)*λ is a hyperparameter used to distinguish the top-ranked summary from other candidate summaries, and D represents the given image;

[0046] Among all candidate summaries, the one with the highest semantic similarity score as described in step 3-1 is selected as the best summary S:

[0047]

[0048] Step 4: Using the above model, extract the descriptive text from the remote sensing image.

[0049] Beneficial effects:

[0050] This invention proposes a multi-scale target detection network to extract regional features of targets in remote sensing images that exhibit large differences in target scale, uneven distribution, and are small and dense.

[0051] This invention addresses the problem that existing image and text generation mechanisms only consider local features of the target and lack global features, and proposes an end-to-end image and text generation model that integrates global and local features.

[0052] This invention addresses the exposure bias problem in the decoding process of existing generative models by introducing sorting into the image and text generation model, and proposes a two-stage model based on generation followed by sorting.

[0053] This invention can automatically perform semantic descriptions on remote sensing images, generating highly readable text descriptions that reflect the activity of the target. Attached Figure Description

[0054] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0055] Figure 1This is a flowchart of the overall process of the method of the present invention.

[0056] Figure 2 Generate an end-to-end network structure diagram for the image and text. Detailed Implementation

[0057] This invention provides a remote sensing image description generation method based on a two-stage process of generation and ranking. First, addressing the challenges of large differences in target scale, uneven distribution, and weak yet dense targets in remote sensing images, a multi-scale target detection network is proposed to extract regional features of the targets. Second, to address the problem that existing image generation mechanisms only consider local target features and lack global features, an end-to-end image generation model that integrates global and local features is proposed. Third, to address the exposure bias problem in the decoding process of existing generation models, ranking is introduced into the image generation model, proposing a two-stage model based on generation followed by ranking. Through these three steps, this invention automatically performs semantic descriptions on remote sensing images, generating highly readable text descriptions that reflect the activity of the targets.

[0058] This invention aims to semantically describe remote sensing images, and to this end, proposes a remote sensing image description generation method based on a two-stage process of generation and sorting, comprising the following steps:

[0059] Step 1: To address the characteristics of large differences in target scale, uneven distribution, and small and dense targets in remote sensing images, a multi-scale target detection network is proposed to extract regional features of targets.

[0060] Step 1-1, skeleton network based on swin transformer (reference: https: / / arxiv.org / pdf / 2103.14030.pdf):

[0061] Because targets in remote sensing images exhibit large scale differences, we employ a multi-scale Swintransformer as the backbone network. Feature maps downsampled to 1 / 8, 16 / 1, 32 / 1, and 1 / 64 of the original image are input into the target detection network. Let the input image resolution be H*W, and the output feature map be labeled as... Their resolutions are H / 8×W / 8, H / 16×W / 16, H / 32×W / 32, and H / 64×W / 64, respectively.

[0062] Steps 1-2: Object detection network based on deformabledetr (reference: https: / / arxiv.org / pdf / 2010.04159.pdf):

[0063] The object detection network input includes the multi-scale feature map {V} from the previous step. l} and N learnable target query vectors The network updates the target query vector through multiple deformable layers, and then retrieves the hidden state of the last layer. R is the feature of the target region.

[0064] Step 2: To address the issue that existing image and text generation mechanisms only consider local features of the target and lack global features, an end-to-end image and text generation model that integrates global and local features is proposed. The specific steps are as follows:

[0065] Step 2-1: Mesh feature extraction based on multi-layer self-attention mechanism:

[0066] We take the feature map of the last layer of the swin transformer (reference: https: / / arxiv.org / abs / 2103.14030) as input, and its dimension is... Where M is a 64-fold downsampling factor, i.e., M = H / 64 * W / 64, and this vector passes through a fully connected layer. Map its vector dimension to a d-dimensional vector After a group containing L g The self-attention transformer of the layer updates G0, and takes the hidden state vector of the last layer as the grid feature G∈R. M*d .

[0067] Step 2-2: Fusing mesh features and region features:

[0068] The feature fusion network is a cross-attention-based transformer network (reference: https: / / arxiv.org / pdf / 1810.04805.pdf). For layer l, its input consists of three parts: word vectors denoted as X′. l ={x′ i}, target region features R, image grid features G. First, the word vector {x′} is... i As a query, the region feature R is used as the key and value for multi-head attention interaction to generate attention weights. Similarly, the word vector {x′} i The attention weights are obtained by interacting with the grid feature G.

[0069] Then, vector concatenation is performed to obtain... and The concatenated vector is then projected onto the d-dimensional plane and transformed into probabilities using a sigmoid function. and

[0070]

[0071]

[0072]

[0073] Finally, the vectors are weighted and summed, and after layer normalization, the attention weights for layer l are obtained.

[0074] Steps 2-3: Autoregressive image-text generation model g(.)(https: / / arxiv.org / abs / 1910.04396)

[0075] We train the generative model g(.) using an autoregressive approach, with the training objective being to maximize the likelihood of the input image I. (See abstract) Conditional probability:

[0076]

[0077] Where I represents the input remote sensing image, and c represents the encoded global and local features of the image.

[0078] Step 3: To address the exposure bias issue in the decoding process of existing generative models, this paper introduces sorting into the image-text generation model and proposes a two-stage model based on generation followed by sorting, as follows:

[0079] Step 3-1: Candidate Abstract Generation:

[0080] A grouping approach with similarity penalty and diverse beam search is used as the generation strategy in the text generation process to generate n candidate summaries S1,...,S n And calculate the difference between each candidate summary and the ground truth labeled summary. The cider values ​​are then sorted in descending order. in This represents the candidate summary with rank i.

[0081] Step 3-2, Sorting Network

[0082] We employ a single-stream multimodal network as the sorter. This network takes an image and a summary text as input and outputs a semantic similarity score between the two. The text is word-embedded to obtain word vector representations; the image is processed by a Swin transformer to obtain image semantic representations, and then passed through a projection layer to obtain the same dimension as the text word vectors. The two are concatenated as the token embedding layer of the BERT pre-trained model, encoded by a BERT transformer, and the last layer of encoding is taken as the semantic representation. Finally, a fully connected layer with an output of 1 is passed to obtain the semantic distance between the two.

[0083] Step 3-3: Calculation of contrastive learning loss

[0084] We employ a ranking loss based on contrastive learning to measure the semantic distance between candidate summaries and ground truth annotations and the image.

[0085]

[0086] The first part is the triplet loss that maximizes the boundary, and the second part is the ranking loss; its optimization objective is to ensure that candidates with a large ranking difference need to have a large difference.

[0087] in the formula This represents the candidate abstract with the i-th rank after descending order of cider. This represents the actual text used to annotate the corresponding image, λ. ij = (ji)*λ is a hyperparameter used to distinguish the difference between good and bad (the first ranked one is good, and the rest are bad) candidate summaries, and D represents the given image;

[0088] During the inference phase, the best summary S is selected from all candidate summaries based on the network output from step 3-2.

[0089]

[0090] Example:

[0091] like Figure 1 As shown, this invention provides a remote sensing image description generation method based on a two-stage generation and sorting process, specifically including the following steps:

[0092] Step 1 proposes a multi-scale target detection network to extract regional features of targets.

[0093] Step 1-1, Backbone network based on Swing transformer:

[0094] Because targets in remote sensing images exhibit large scale differences, we employ a multi-scale Swintransformer as the backbone network. Feature maps downsampled to 1 / 8, 16 / 1, 32 / 1, and 1 / 64 of the original image are input into the target detection network. Let the input image resolution be H*W, and the output feature map be labeled as... Their resolutions are H / 8×W / 8, H / 16×W / 16, H / 32×W / 32, and H / 64×W / 64, respectively.

[0095] Steps 1-2: Object detection network based on deformabledetr:

[0096] The object detection network input includes the multi-scale feature map {V} from the previous step. l} and N learnable target query vectors The network updates the target query vector through multiple deformable layers, and then retrieves the hidden state of the last layer. R is the feature of the target region.

[0097] Step 2 proposes an end-to-end image and text generation model that integrates global and local features, such as... Figure 2 As shown.

[0098] Step 2-1: Mesh feature extraction based on multi-layer self-attention mechanism:

[0099] We take the last layer feature map of the swin transformer as input, and its dimension is... Where M is a 64-fold downsampling factor, i.e., M = H / 64 * W / 64, and this vector passes through a fully connected layer. Map its vector dimension to a d-dimensional vector After a group containing L g The self-attention transformer of the layer updates G0, and takes the hidden state vector of the last layer as the grid feature G∈R. M*d .

[0100] Step 2-2: Fusing mesh features and region features:

[0101] The feature fusion network is a multi-layer network. For layer l, its input consists of three parts: word vectors denoted as X′. l ={x′ i}, target region features R, image grid features G. First, the word vector {x′} is... i As a query, the region feature R is used as the key and value for multi-head attention interaction to generate attention weights. Similarly, the word vector {x′} iThe attention weights are obtained by interacting with the grid feature G.

[0102] Then, vector concatenation is performed to obtain... and The concatenated vector is then projected onto the d-dimensional plane and transformed into probabilities using a sigmoid function. and

[0103]

[0104]

[0105]

[0106] Finally, the vectors are weighted and summed, and after layer normalization, the attention weights for layer l are obtained.

[0107] Steps 2-3: Autoregressive image and text generation model g(.)

[0108] We train the generative model g(.) using an autoregressive approach, with the training objective being to maximize the likelihood of the input image I. (See abstract) Conditional probability:

[0109]

[0110]

[0111] Where I represents the input remote sensing image, and c represents the encoded global and local features of the image.

[0112] Step 3 introduces sorting into the image and text generation model, proposing a two-stage model based on first generating and then sorting.

[0113] Step 3-1: Candidate Abstract Generation:

[0114] A grouping and similarity-penalized diversity bundle search algorithm is adopted as the generation strategy in the text generation process (reference: DIVERSE BEAM SEARCH: DECODING DIVERSE SOLUTIONS FROM NEURAL SEQUENCEMODELS) to generate n candidate summaries S1,...,S n And calculate the difference between each candidate summary and the ground truth labeled summary. The cider values ​​are then sorted in descending order. in This represents the candidate summary with rank i.

[0115] Step 3-2, Sorting Network

[0116] We employ a single-stream multimodal network (see: https: / / openaccess.thecvf.com / content / CVPR2021 / papers / Miech_Thinking_Fast_and_Slow_Efficient_Text-to-Visual_Retrieval_With_Transformers_CVPR_2021_paper.pdf) as the sorter. This network takes an image and a summary text as input and outputs a semantic similarity score between them. Specifically, the text undergoes word embedding (see: https: / / arxiv.org / pdf / 1810.04805.pdf) to obtain word vector representations; the image undergoes a Swin transformer (see: https: / / arxiv.org / pdf / 2103.14030.pdf) to obtain image semantic representations, which are then passed through a projection layer to achieve the same dimension as the text word vectors. The two are concatenated as the token embedding layer of a BERT pre-trained model (see: https: / / arxiv.org / pdf / 1810.04805.pdf), and then processed by BERT. Transformer encoding is used, and the last layer of encoding is taken as the semantic representation. The semantic distance between the two is obtained by passing the encoding through a fully connected layer with an output of 1.

[0117] Step 3-3: Calculation of contrastive learning loss

[0118] We employ a ranking loss based on contrastive learning to measure the semantic distance between candidate summaries and ground truth annotations and the image.

[0119]

[0120] The first part is the triplet loss that maximizes the boundary, and the second part is the ranking loss; its optimization objective is to ensure that candidates with a large ranking difference need to have a large difference.

[0121] in the formula This represents the candidate abstract with the i-th rank after descending order of cider. This represents the actual text used to annotate the corresponding image, λ. ij = (ji)*λ is a hyperparameter used to distinguish the difference between good and bad candidate summaries, and D represents the given image;

[0122] The highest-scoring abstract among all candidate abstracts is selected as the best abstract, S.

[0123]

[0124] In its specific implementation, this application provides a computer storage medium and a corresponding data processing unit. The computer storage medium is capable of storing a computer program, which, when executed by the data processing unit, can run the invention's content regarding a two-stage remote sensing image description generation method based on generation and sorting, as well as some or all of the steps in various embodiments. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0125] Those skilled in the art will clearly understand that the technical solutions in the embodiments of the present invention can be implemented using computer programs and their corresponding general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of computer programs, i.e., software products. These computer program software products can be stored in a storage medium and include several instructions to cause a device containing a data processing unit (which may be a personal computer, server, microcontroller, MUU, or network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.

[0126] This invention provides a method for generating remote sensing image descriptions based on a two-stage process of generation and sorting. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A remote sensing image description generation method based on a two-stage process of generation and sorting, characterized in that, Includes the following steps: Step 1: Construct a multi-scale target detection network to extract the regional features of the target, i.e., the remote sensing image. Step 2: Construct an end-to-end image-text generation model that integrates global and local features, and use the model to generate different descriptive texts for the remote sensing image; Step 3: Introduce the sorting method into the end-to-end image and text generation model described in Step 2, and construct a two-stage model based on generation followed by sorting. That is, sort the different descriptive texts generated in Step 2 according to the sorting method, and select the optimal descriptive text. Step 4: Using the above model, extract descriptive text from the remote sensing image; Furthermore, step 3 specifically includes: Step 3-1: Construct a ranking network and perform ranking, that is, use a single-stream multimodal network as a ranking machine to rank according to the semantic similarity scores calculated by it. The single-stream multimodal network takes an image and descriptive text as input and outputs a semantic similarity score between them, specifically including: The descriptive text is encoded using word embedding to obtain a word vector representation; The image is processed through a swin transformer to obtain a semantic representation of the image, and then through a projection layer to obtain the same dimension as the word vector representation of the summary text. Word vector representations and semantic representations are concatenated and input into the character encoding layer of the BERT pre-trained model. After BERT transformer encoding, the last hidden layer is taken as the final semantic representation. After passing through a fully connected layer with an output of 1, the semantic distance between the image and the descriptive text, i.e. the semantic similarity score, is obtained. Step 3-2: Calculate the contrastive learning loss: A ranking loss based on contrastive learning is used to measure the distance L between the image and the text summary, specifically including: in, It is the triplet loss that maximizes the boundary. This represents the ranking loss; its optimization objective is: for a given image, the greater the ranking difference between two candidate text summaries, the greater the aforementioned distance; in the formula and Let m1, p, and q represent the candidate abstracts ranked in descending order of cider, respectively. This represents the actual text used to annotate the corresponding image, λ. ij = (ji)*λ is a hyperparameter used to distinguish the top-ranked summary from other candidate summaries, and D represents the given image; Among all candidate summaries, the one with the highest semantic similarity score as described in step 3-1 is selected as the best summary S:

2. The remote sensing image description generation method based on a two-stage generation and sorting process according to claim 1, characterized in that, The specific method for extracting the target, i.e., the regional features of the remote sensing image, as described in step 1 includes: Step 1-1: Based on the skeleton network of the swin transformer, extract the multi-scale feature map of the target, i.e., the remote sensing image. Step 1-2: Input the multi-size feature map extracted in Step 1-1 into the target detection network based on deformabledetr to obtain the regional features of the remote sensing image.

3. The remote sensing image description generation method based on a two-stage generation and sorting process according to claim 2, characterized in that, The skeleton network based on the Swin transformer described in step 1-1 extracts the multi-scale feature map of the target, i.e., the remote sensing image, specifically including: Using a multi-scale Swing transformer as the backbone network, the remote sensing image is downsampled through the backbone network into feature maps of 1 / 8, 1 / 16, 1 / 32 and 1 / 64 of the original remote sensing image size, which are the multi-scale feature maps. Let the resolution of the input remote sensing image be H*W, then the output multi-size feature map is: Among them, feature map V l The resolutions are 8×8, 16×16, 32×32, and 64×64, respectively, where l represents the l-th feature map, L b Indicates the number of feature maps.

4. The remote sensing image description generation method based on a two-stage generation and sorting process according to claim 3, characterized in that, In step 1-2, the multi-size feature map extracted in step 1-1 is input into a deformabledetr-based target detection network to obtain the region features of the remote sensing image, as detailed below: Input the multi-scale feature map extracted in step 1-1 into the deformable DTR-based object detection network. and N target query vectors R d Let r represent a set of d-dimensional real vectors, where d represents the vector dimension and r is the vector dimension. i This represents the i-th target query vector; The deformabledetr-based object detection network updates the target query vector after three or more deformable layers, taking the hidden state of the last layer. The target region feature R.

5. The remote sensing image description generation method based on a two-stage generation and sorting process according to claim 4, characterized in that, Step 2 involves constructing an end-to-end image-text generation model that integrates global and local features, and using this model to generate different descriptive texts for the remote sensing image. Specifically, this includes: Step 2-1: Extract the grid features of the remote sensing image based on a multi-layer self-attention mechanism; Step 2-2: Use a feature fusion network to fuse grid features and region features; Steps 2-3: Train an autoregressive-based image-text generation model g(.) as a decoder to select the text with the highest probability as the decoded text. Step 2-4: Repeat steps 2-2 to 2-3 until candidate description text is generated; Steps 2-5: Generate other candidate description texts.

6. The remote sensing image description generation method based on a two-stage generation and sorting process according to claim 5, characterized in that, Step 2-1 describes the extraction of grid features from the remote sensing image based on a multi-layer self-attention mechanism. The specific method includes: The feature map obtained from the last layer of the Swing Transformer backbone network is taken as input. The dimension of this feature map is... Where M is a 64-fold downsampling factor, i.e., M = H / 64 * W / 64. Representing the dimension of the feature vector, the above feature map passes through a fully connected layer. Map its vector dimension to a d-dimensional vector After a group containing L g The self-attention transformer module of the layer updates G0, and takes the hidden state vector of the last layer as the grid feature G∈R of the remote sensing image. M*d .

7. The remote sensing image description generation method based on a two-stage generation and sorting process according to claim 6, characterized in that, Step 2-2 describes using a feature fusion network to fuse grid features and region features. The specific method includes: The feature fusion network is an L-layer network. For the l-th layer cross-attention network, its input consists of three parts: the text decoded by the decoder, i.e., the word vector denoted as X′. l ={x′ i }, the target's regional features R, and the grid features G of the remote sensing image; The specific process of fusion is as follows: Step 2-2-1, convert the word vector {x′} i As a query, the region feature R is used as the key and value for multi-head attention interaction to generate attention weights. Similarly, the word vector {x′ i The attention weights are obtained by performing multi-head attention interaction between the grid feature G and the grid feature G. Step 2-2-2, perform vector concatenation to obtain and The concatenated vector is then projected onto d dimensions and transformed into probabilities using the sigmoid function. and Specifically as follows: Among them, b g Indicates grid feature bias, b r LN() indicates regional feature bias, and LN() indicates layer normalization. W represents element-wise multiplication. r Represents the weight of regional features. This represents the attention value of time step j for layer l; Step 2-2-3, for each vector in the l-th layer After weighted summation and layer normalization, the attention weights for layer l are obtained.

8. The remote sensing image description generation method based on a two-stage generation and sorting process according to claim 7, characterized in that, Steps 2-3 describe training an autoregressive image-text generation model g(.) as the decoder, used to select the text with the highest probability as the decoded text. Specifically, this includes: The training objective is to maximize the likelihood of a given input image and a reference description text. conditional probability Specifically as follows: Where I represents the input remote sensing image, c represents the encoded representation of global and local features of the image, m represents the text length, and y represents the input remote sensing image. k This represents the decoding probability at the current time step k. This represents the text encoding of the current character up to the current time step k. This indicates the network parameters for decoding.

9. A remote sensing image description generation method based on a two-stage generation and sorting process according to claim 8, characterized in that, Steps 2-5, which involve generating other candidate descriptive texts, specifically include: The text is generated using a bundle constraint method, with a similarity penalty factor added to ensure the diversity of the generated text. This results in n candidate descriptive texts, namely, candidate descriptive texts S1,...,S... n And calculate the difference between each candidate description text and the actual labeled description text. The CIDER values ​​are then sorted in descending order as follows: in, This represents the candidate description text with a ranking of m.

Images