Wide remote sensing image semantic description method and system based on adaptive region segmentation

Abstract

The present application relates to a kind of wide remote sensing image semantic description method and system based on adaptive region segmentation, belong to the cross-application technical field of natural language processing and computer vision.The present application first for wide remote sensing image, the semantic distribution characteristics of image are analyzed, semantic estimation point is generated, clustering is carried out to semantics, and image is segmented according to cluster center, to obtain target sub-region.For target sub-region, a small scale remote sensing image description model is used to generate the description of corresponding region, and multiple single sentence descriptions are merged according to their clustering relationship, to generate complete, clear final description.

Description

Technical Field

[0001] This invention relates to a method and system for semantic description of remote sensing images, specifically a method and system for semantic description of wide-span remote sensing images based on adaptive region segmentation, belonging to the technical field of interdisciplinary applications of natural language processing and computer vision. Background Technology

[0002] Image description is a task in which a computer automatically generates descriptive text based on an image. The computer must not only detect objects in the image but also understand the relationships between them and finally express them in fluent natural language. Existing technologies have achieved good results in image description of natural images.

[0003] Remote sensing images, acquired by satellites or aircraft from high altitudes, offer a unique bird's-eye view. Furthermore, most remote sensing images contain numerous semantic objects with difficult-to-distinguish primary and secondary elements, making them more challenging to interpret than natural images. Researchers need to improve upon descriptive models used for natural images to adapt to the characteristics of remote sensing images.

[0004] Among the many improvement proposals put forward by researchers, they all share the common feature of designing a mechanism to strengthen the relationship between semantics, while using an attention mechanism to ensure the understanding of a part of the image to generate vocabulary, and focusing on the information describing the whole image.

[0005] Currently, with the development and widespread application of remote sensing technology, remote sensing images are evolving towards higher resolution and wider swaths. Wide-swath remote sensing images, in particular, contain multiple consecutive small-scale images, which contain richer and more complex semantic and positional information compared to small-scale images. However, existing semantic description models for remote sensing images are not suitable for wide-swath remote sensing images. Summary of the Invention

[0006] The purpose of this invention is to address the technical problem that current remote sensing image semantic description techniques are only applicable to single, small-scale remote sensing images and cannot be used to describe wide-swath, large-scale remote sensing images. This invention creatively proposes a method and system for semantic description of wide-swath remote sensing images based on adaptive region segmentation. By utilizing an adaptive region segmentation strategy to extract key descriptive regions, generating descriptions for sub-regions, and integrating and sorting them, a more comprehensive, chapter-based description of the wide-swath remote sensing image is generated.

[0007] The innovations of this invention include: for wide-span remote sensing images, by analyzing the semantic distribution features of the images and selecting different partitioning strategies, adaptive clustering and segmentation of regions are performed to extract multiple target sub-regions. For each target sub-region, a small-scale remote sensing image description model is used to generate a description of the corresponding region, and multiple single-sentence descriptions are summarized and merged according to their clustering relationships to generate a final description containing completeness and clarity.

[0008] The present invention is achieved using the following technical solution.

[0009] A semantic description method for wide-span remote sensing images based on adaptive region segmentation includes the following steps:

[0010] Step 1: Input a wide-span remote sensing image and generate the sub-region to be described.

[0011] Specifically, it includes the following steps:

[0012] Step 1.1: Divide the image evenly into multiple sub-regions (e.g., 16). Calculate the variance of the three color channels in each sub-region and take their mean.

[0013] Step 1.2: Determine whether the input image contains obvious clustering features by threshold truncation.

[0014] Step 1.3: For images with variance less than or equal to a set threshold a (e.g., 20), a fixed pattern segmentation process is used; for images with variance greater than the set threshold a, semantic clustering is first used to determine the semantic center, and then segmentation is performed based on the semantic center.

[0015] Specifically, the following methods can be used to achieve this:

[0016] Step 1.3.1: Using the sub-region variance value obtained in Step 1.1 as a guide, randomly project var points in the k-th tile as "pseudo-semantic points".

[0017] Step 1.3.2: For all “pseudo-semantic points”, use K-Means to cluster them to obtain better cluster centers.

[0018] Step 1.3.3: Take the cluster center plus the center of the fixed minimum variance block as the primary semantic center, and move it 64 pixels in each of the four directions (up, down, left, and right) to extract four secondary semantic centers. Finally, use the secondary semantic center points as the centers of the selected subgraphs to be described.

[0019] Step 2: Generate a single-sentence description for the target sub-region obtained in Step 1.

[0020] Specifically, it includes the following steps:

[0021] Step 2.1: Input a segmented small-scale image of size (256,256) pixels, use ResNet (Residual Network) pre-trained on ImageNet to extract image features, and then use FCN (Fully Convolutional Neural Network) to extract pixel-level classification features.

[0022] Step 2.2: Input the two different scale image features obtained in Step 2.1 into a specially modified multi-scale LSTM (Long Short-Term Memory Neural Network) to obtain the corresponding text description of the image.

[0023] Step 3: Integrate and sort the single-sentence descriptions generated in Step 2.

[0024] Specifically, it includes the following steps:

[0025] Step 3.1: For the multiple single sentences generated in Step 2, the descriptions are merged according to the clustering relationship to obtain the integrated summary sentences.

[0026] Step 3.2: Sort the summary sentences from Step 3.1 according to spatial relationships and insert descriptive words in relative positions.

[0027] Based on the above method, this invention further proposes a wide-span remote sensing image semantic description system based on adaptive region segmentation, including a target sub-region generation module, a target sub-region description module, and a description sentence integration module.

[0028] The target sub-region generation module is used to adaptively segment and extract key descriptive regions based on the semantic distribution features of the image.

[0029] The target sub-region description module is used to generate descriptive text for the target sub-region using a small-scale remote sensing image semantic description model, where small-scale refers to an image with a size of (256, 256) pixels.

[0030] The description sentence integration module is used to merge multiple single sentences according to their clustering relationships to obtain a complete and clear final description;

[0031] Furthermore, the target sub-region generation module includes an image semantic distribution feature analysis unit, a semantic clustering unit, and an image segmentation unit. The image semantic distribution feature analysis unit analyzes whether the image semantics exhibit obvious clustering features; the semantic clustering unit generates semantic centers through clustering; and the image segmentation unit segments multiple target sub-regions to be described based on the semantic centers.

[0032] Furthermore, the target sub-region description module includes an image encoding unit and a text generation unit. The image encoding unit extracts image features at different levels using a ResNet network and an FCN; the text generation unit uses a multi-scale LSTM to generate a single-sentence description of the image.

[0033] Furthermore, the descriptive sentence integration module includes a summary generation unit and a summary sorting unit. The summary generation unit merges single-sentence descriptions corresponding to multiple sub-regions based on their clustering relationships; the summary sorting unit sorts the integrated sub-region descriptions according to spatial relationships and generates a final description containing the relative positional information of each region.

[0034] The connections between the above components are as follows:

[0035] The input of the target sub-region description module is connected to the output of the target sub-region generation module;

[0036] The input of the description sentence integration module is connected to the output of the target sub-region description module;

[0037] In the target sub-region generation module, the input of the image segmentation unit is connected to the output of the semantic clustering unit.

[0038] In the target sub-region description module, the input end of the text generation unit is connected to the output end of the image encoding unit;

[0039] In the descriptive sentence integration module, the input of the summary sorting unit is connected to the output of the summary generation unit.

[0040] Beneficial effects

[0041] Compared with the prior art, the present invention has the following advantages.

[0042] 1. This invention selects an appropriate partitioning strategy based on the semantic distribution characteristics of the image, and is applicable to various wide-span remote sensing images, such as those with uniform semantic distribution and those with uneven semantic distribution.

[0043] 2. By integrating sentence descriptions of similar regions together, this invention effectively increases the descriptive range of remote sensing image regions by the small-scale description model, making the descriptions generated for each target semantic region more detailed. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of a small-scale remote sensing image description model according to an embodiment of the present invention;

[0045] Figure 2 This is a schematic diagram of a multi-scale LSTM according to an embodiment of the present invention. Detailed Implementation

[0046] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0047] Example

[0048] like Figure 1As shown, a semantic description method for wide-span remote sensing images based on adaptive region segmentation includes the following steps:

[0049] Step 1: Input a wide-span remote sensing image and generate the sub-region to be described.

[0050] Specifically as follows:

[0051] Step 1.1: Divide the image into 16 sub-regions evenly, and calculate the variance of the three color channels in each sub-region and take the mean;

[0052] Specifically, the input image is first resized to 1024*1024, and then evenly divided into 16 sub-regions of size 256*256. Next, the variance of each sub-region's three color channels is calculated and the mean is taken, as shown in the following formula:

[0053]

[0054] Where var represents the variance of the sub-region. I represents a segmented sub-region, and i represents the color channel of the image, where 0, 1, and 2 represent the red, green, and blue color channels of an image in RGB mode, respectively. p and q represent the pixel-level coordinates (p, q) of the sub-region. mean(I(i)) represents the mean of image I on color channel i.

[0055] Calculate the variance (var) for each of the 16 sub-regions, and then calculate the variance again for the total of 16 var values ​​using the following formula:

[0056]

[0057] Where VAR represents the variance of the image, var(k) represents the var value of the k-th sub-region, and mean(var) represents the mean of the 16 var values.

[0058] The variable var measures, to some extent, the density of semantic distribution in a sub-region of an image, thereby determining whether the semantic distribution in the image has obvious clustering characteristics.

[0059] Step 1.2: By comparing thresholds, determine whether the input image contains obvious clustering features.

[0060] Specifically, if the VAR value is above 20, the image is classified as having clustering features and needs to be adaptively searched and extracted as a sub-image; if the VAR value is 20 or below, the semantics in the image are considered to be relatively uniform and can be processed using a fixed pattern segmentation.

[0061] Step 1.3: For images with a VAR value of 20 or less, a fixed-pattern segmentation process is used; for images with a VAR value of more than 20, semantic clustering is used to determine the semantic center first, and then segmentation is performed based on the semantic center.

[0062] Specifically, using the sub-region variance value obtained in step 1.1 as a guide, var points are randomly projected in the k-th tile as "pseudo-semantic points".

[0063] Then, K-Means clustering is performed on all the "pseudo-semantic points". Specifically, the image is first divided into 16 fixed modules, and five fixed points are used to initialize the cluster centers. These five points are selected at the center of the image, the center of the four images in the upper left corner, the center of the four images in the upper right corner, the center of the four images in the lower left corner, and the center of the four images in the lower right corner. In addition, the sub-image with the lowest var value among the 16 sub-images is selected as a fixed descriptive region as a supplement. The semantic center is the center of the sub-image. After clustering, five better cluster centers are obtained.

[0064] Finally, the five cluster centers plus the center of the fixed minimum variance block are used as the primary semantic centers, and four secondary semantic centers are extracted by moving the centers 64 pixels in each of the four directions (up, down, left, and right). Ultimately, the secondary semantic centers are used as the centers of the selected subgraphs to be described. A total of 24 subgraphs can be selected.

[0065] Step 2: Generate a single-sentence description for the target sub-region obtained in Step 1.

[0066] Specifically as follows:

[0067] Step 2.1: Input the segmented small-scale image, extract image features using a ResNet network (such as ResNet101), and extract pixel-level classification features using FCN.

[0068] like Figure 1 As shown. Specifically, the encoder uses two encoding methods simultaneously, passing the image through a ResNet and an FCN respectively:

[0069] v = Resnet(I) (3)

[0070] V = argmax(FCN(I), 0) (4) where I ∈ R 3*256*256 ,v∈R 8*8*2048 ,FCN(I)∈R 31*256*256 , representing a small-scale image of size (256, 256) pixels, image features encoded by a ResNet network, and pixel-level classification output obtained by an FCN, respectively. The final result is obtained by selecting the classification with the highest probability for each pixel. argmax(·, i) represents extracting the largest index value in the i-th dimension. V∈R256*256 , represents the pixel-level classification result. R represents the set of real numbers.

[0071] Step 2.2: Input the two different scale image features obtained in Step 2.1 into a specially modified multi-scale LSTM to obtain the text description of the corresponding image;

[0072] Specifically, such as Figure 2 As shown, the decoder employs a specially modified multi-scale LSTM. Compared to a regular LSTM, it allocates an additional memory cell to store image features at another scale. These features are computed separately at each time step, and the values ​​from both memory cells are integrated as a new hidden layer. Here, the features v obtained from ResNet and the image classification features V obtained from FCN are simultaneously input into the multi-scale LSTM.

[0073] In multi-scale LSTM, two memory cells C and c are initialized with image features obtained from FCN and ResNet, respectively. Their function is to store the image features at two different levels.

[0074] c -1 =W init_c v+b init_c (5)

[0075] C -1 =W init_CV +b init_C (6)

[0076] H -1 =W init_hv +b init_h (7)

[0077] Among them, W init_c and b init_c W represents the weight matrix and offset of memory cell c; init_C and b init_c W represents the weight matrix and offset of memory cell C; init_h and b init_h This represents the weight matrix and offset of the hidden layer H.

[0078] In each time step, the two features are calculated relatively independently, but they share a set of gating information. The calculation of the three gates is the same as that of a regular LSTM, as follows:

[0079] I t =σ(W i [X t H t-1 ]+b i (8)

[0080] Ft =σ(W f [X t H t-1 ]+b f (9)

[0081] O t =σ(W o [X t H t-1 ]+b o (10)

[0082] Where σ represents the Sigmoid function, X t H represents the input at the current time step. t-1 I represents the hidden layer features of the previous time step. t W represents the input gate. i and b i F represents the weight matrix and offset of the input gate; t W represents the Forgotten Gate. f and b f Represents the weight matrix and offset of the forget gate; O t Indicates the output gate, W o and b o This represents the weight matrix and offset of the output gate.

[0083] The input and forgetting calculation processes for memory cells c and C are basically the same as those for LSTM:

[0084]

[0085]

[0086]

[0087]

[0088] Among them, W C and b C and W c and b c These represent the weight matrix and offset of the two linear layers, respectively. This represents the input to memory cell C at time t. C represents the input of memory cell c at time t. t This represents the output of memory cell C at time t after passing through the input gate and the forget gate. t-1 This represents the output of memory cell C at time t-1, where c t This represents the output of memory cell c after passing through the input gate and the forget gate. t-1 F represents the output of memory cell c at time t-1. tRepresents the forgetting gate, I t This indicates the input gate.

[0089] In equations 11 and 12, W C and W c This corresponds to two different linear layer weight matrices. After calculating C... t and c t The values ​​are concatenated into a vector, which is then mapped to the same size as the hidden layer vector through an affine transformation and activation layer. Finally, the output is controlled by an output gate and fed into the current hidden layer H. t Strength:

[0090] H t =O t ⊙tanh(W h [C t ,ct]+b h (15)

[0091] Among them, O t Indicates the output gate, W h and b h This represents the weight matrix and offset of the linear layer.

[0092] Then, use a fully connected layer to connect H t A vector mapped to the size of the vocabulary is used to generate the output at that time step, serving as a prediction of the probability distribution of the next word across the entire dictionary:

[0093] Pred t =softmax(W pred H t +b pred (16)

[0094] Among them, Pred t W represents the generated prediction vector. pred and b pred This represents the weight matrix and offset of the fully connected layer.

[0095] Step 3: Integrate and sort the single-sentence descriptions generated in Step 2.

[0096] Specifically as follows:

[0097] Step 3.1: Summarize the multiple single-sentence descriptions generated in Step 2 according to clustering relationships to obtain the integrated summary sentences;

[0098] Specifically, the Meansum text summarization model is used to integrate multiple sentences into a summary. This model includes an autoencoder and a summarization module.

[0099] The role of the autoencoder is to learn the representation of the input sentence (Review) and restrict the generated summary to the linguistic domain;

[0100] The summarizing module is responsible for learning to generate summaries that are semantically similar to each input sentence. Ideally, the autoencoder encodes the input sentence into an intermediate vector, and the summarizing module decodes this intermediate vector to generate a summary. Both the autoencoder and the summarizing module contain an encoder and a decoder, both implemented using LSTM. The encoder and decoder in the autoencoder and summarizing modules have the same model parameters; similarly, their decoders also have the same parameters. During model learning, the autoencoder and summarizing module each incur two losses: an autoencoder reconstruction loss and a summarizing generator similarity loss.

[0101] Let V represent the set of tokenized sentences in the dataset, with the longest sentence denoted as L. The autoencoder encodes the input sentences using an LSTM, and the encoder's Φ... E Map the sentence from V to Φ E (x j ) is denoted as z j Input x at each time step j The values ​​of the hidden layer and memory cells generated from the final sequence are concatenated [h,c] to obtain the encoding of a sentence x. The mapping direction at the decoding end is exactly the opposite of that at the encoding end, Φ D From Mapped to V, the tokens generated by the LSTM are given in the form of a probability distribution over a vocabulary. This introduces a reconstruction loss: the encoder and decoder are ideally functionally inverse, so that the intermediate representation of the sentence input to the encoder can be reconstructed back to the original sentence when passed through a decoder.

[0102] Since the words reconstructed by the decoder follow a probability distribution, the reconstruction loss is given by the form of cross-entropy in Equation 17:

[0103]

[0104] in, Let x represent the reconstruction loss. k Φ represents the k-th token. E Indicates encoder, Φ D Indicates decoder, Denotes cross-entropy loss, x j This represents the j-th token.

[0105] The summarization module receives multiple intermediate vectors [h, c] encoded by the autoencoder, where h represents the hidden layer output and c represents the memory cell output. The mean values ​​of the hidden layer and memory cell outputs for all sentences are then calculated to obtain the sum. in This represents the mean of the hidden layer output. This represents the mean output of memory cells. The process of calculating this mean is viewed as summarizing all sentences. Then, after calculating the mean... Decoding yields the textual output of the summary. Similarly, the learning process of the summarizing module also incurs a loss value. The summarizing module learns by re-encoding the decoded token summary, calculating the similarity between the resulting intermediate vector and the intermediate vectors obtained from all sentences in the original input. A higher similarity indicates that the summary contains more complete information from the original input. This similarity is measured using the mean cosine distance, and the loss value is given.

[0106] Where, d cos (·,·) represents the cosine distance between two vectors, and the LSTM hidden state after the token-formatted digest is re-encoded by the encoder is denoted as h. s Sentence x j The encoded hidden state is denoted as h. j The similarity loss is obtained by calculating the average cosine distance between the hidden states of all k sentences and the summary. The total loss function that the model needs to optimize is the sum of the two losses.

[0107]

[0108] Step 3.2: Sort the summary sentences from Step 3.1 according to spatial relationships and insert descriptive words in relative positions.

[0109] Specifically, if the semantic distribution of the image is determined to be uniform in step 1.2 (i.e., VAR <= 20), then a fixed pattern of segmentation and integration is used. For example, the image is divided into 16 blocks, and four adjacent blocks are integrated into one sentence.

[0110] First, generate a description of the semantic clustering features of the image: "The semantic distribution in this image is relatively uniform." Then, describe the four directions of Upper left, Upper right, Lower left, and Lower right in sequence.

[0111] If the image semantics are determined to have obvious clustering features in step 1.2, the description order is determined as follows:

[0112] Let V be the set of all sentences to be sorted. Divide V into multiple subsets, where C∈V represents the sentence whose corresponding primary semantic center falls in the CENTER region, and L, R, D, T represent the sets of sentences that fall in the LEFT, RIGHT, DOWNSIDE, and TOP regions, respectively.

[0113] First, based on the VAR value of the image, generate a sentence to describe the characteristics of the image's semantic distribution. With VAR > a threshold (e.g., 20), the generated sentence is: "The semantic distribution of the image is relatively scattered, showing clustering features."

[0114] If C is not empty, output the fixed sentence "On the center area of ​​image," and output the sentences in C in sequence.

[0115] If L and R are not empty, insert the sentence "On the left / right area of ​​image," and output the sentences in L or R in sequence.

[0116] If T and D are not empty, insert the sentence "On the top / bottom area of ​​image," and output the sentences in T or D in sequence.

[0117] If any of the primary semantic centers fall in the four corners, insert the sentence "On the upper / lower left / right corner of image," and output the corresponding sentence.

[0118] Finally, the final description text is obtained.

[0119] In another aspect, the present invention proposes a wide-swath remote sensing image semantic description system based on adaptive region segmentation, including a target sub-region generation module, a target sub-region description module, and a description sentence integration module;

[0120] The target sub-region generation module is used to adaptively segment and extract key descriptive regions based on the semantic distribution features of the image;

[0121] The target sub-region description module is used to generate descriptive text for target sub-regions using a small-scale remote sensing image semantic description model;

[0122] The description sentence integration module is used to merge multiple single sentences according to their clustering relationships to obtain a complete and clear final description;

[0123] Furthermore, the target sub-region generation module includes an image semantic distribution feature analysis unit, a semantic clustering unit, and an image segmentation unit;

[0124] The image semantic distribution feature analysis unit is used to analyze whether there are obvious clustering features in the image semantics; the semantic clustering unit is used to generate semantic centers through clustering; and the image segmentation unit is used to segment multiple target sub-regions to be described based on the semantic centers.

[0125] The target sub-region description module includes an image encoding unit and a text generation unit;

[0126] The image coding unit extracts image features at different levels using ResNet and FCN; the text generation unit uses a multi-scale LSTM to generate a single-sentence description of the image.

[0127] The descriptive sentence integration module includes a summary generation unit and a summary sorting unit;

[0128] The abstract generation unit is used to merge the single-sentence descriptions corresponding to multiple sub-regions according to their clustering relationships; the abstract sorting unit is used to integrate the sub-region descriptions, sort them according to spatial relationships, and generate a final description containing the relative position information of each region.

Claims

1. A semantic description method for wide-swath remote sensing images based on adaptive region segmentation, characterized in that, Includes the following steps: Step 1: Input a wide-span remote sensing image and generate the sub-region to be described; Step 1.1: Divide the image into multiple sub-regions evenly, calculate the variance of the three color channels in each sub-region, and take their mean; Step 1.2: Determine whether the input image contains obvious clustering features by threshold truncation; Step 1.3: For images with variance less than or equal to a set threshold, a fixed-pattern segmentation process is used; for images with variance greater than a set threshold, semantic clustering is first used to determine semantic centers, and then segmentation is performed based on the semantic centers. Step 2: Generate a single-sentence description for the target sub-region obtained in Step 1; In step 2, two encoding methods are used simultaneously at the encoding end, passing the image through a ResNet and an FCN respectively: v = Resnet(I) (3) V = argmax(FCN(I), 0)(4) Where, I∈R 3*256*256 ,v∈R 8*8*2048 ,FCN(I)∈R 31*256*256 , representing a small-scale image with input size (256, 256) pixels, image features encoded by a ResNet network, and pixel-level classification output obtained by an FCN, respectively; the final result after selecting the classification with the highest probability for each pixel; argmax(·, i) represents extracting the largest index value in the i-th dimension; V∈R 256*256 , represents the pixel-level classification result; R represents the set of real numbers. At the decoder, a modified multi-scale LSTM is used. Compared with LSTM, it opens an additional memory cell to store image features of another scale. The features are calculated separately at each time step and the values ​​of the two memory cells are integrated as a new hidden layer. The features v obtained from ResNet and the image classification features V obtained from FCN are simultaneously input into the multi-scale LSTM. In multi-scale LSTM, two memory cells C and c are initialized with image features obtained from FCN and ResNet, respectively. Their function is to store the image features at two different levels. c -1 =W init_c v+b init_c (5) C -1 =W init_C V+b init_C (6) H -1 =W init_h v+b init_h (7) Among them, W init_c and b init_c W represents the weight matrix and offset of memory cell c; init_C and b init_C W represents the weight matrix and offset of memory cell C; init_h and b init_h This represents the weight matrix and offset of the hidden layer H; In each time step, the two features are calculated relatively independently, but share a set of gating information; Step 2.1: Input the segmented small-scale image, use the pre-trained residual network ResNet to extract image features, and then use the fully convolutional neural network FCN to extract pixel-level classification features; Step 2.2: Input the two different scale image features obtained in Step 2.1 into a specially modified multi-scale long short-term memory neural network LSTM to obtain the text description of the corresponding image; Step 3: Integrate and sort the single-sentence descriptions generated in Step 2; Step 3.1: For the multiple single sentences generated in Step 2, the descriptions are merged according to the clustering relationship to obtain the integrated summary sentences; Step 3.2: Sort the summary sentences from Step 3.1 according to spatial relationships and insert descriptive words in relative positions.

2. The semantic description method for wide-swath remote sensing images based on adaptive region segmentation as described in claim 1, characterized in that, In step 1.1, the input image is first resized to 1024*1024 and then evenly divided into 16 sub-regions of size 256*256. Calculate the variance and mean for each of the three color channels in each sub-region, as shown in the following formula: Where var represents the variance of the sub-region; I represents a sub-region obtained by segmentation; i represents the color channel of the image, where 0, 1, and 2 represent the red, green, and blue color channels of an image in RGB mode, respectively; p and q represent the pixel-level coordinates (p, q) of the sub-region; mean(I(i)) represents the mean of image I on color channel i. Calculate the variance (var) for each of the 16 sub-regions, and then calculate the variance again for the total of 16 var values ​​using the following formula: Where VAR represents the variance of the image, var(k) represents the var value of the k-th sub-region, and mean(var) represents the mean of the 16 var values.

3. The wide-swath remote sensing image semantic description method based on adaptive region segmentation as described in claim 1, characterized in that, Step 1.3 includes the following steps: Step 1.3.1: Using the sub-region variance value obtained in Step 1.1 as a guide, randomly project var points in the k-th tile as "pseudo-semantic points"; Step 1.3.2: For all "pseudo-semantic points", use K-Means to cluster them to obtain better cluster centers; Step 1.3.3: Take the cluster center plus the center of the fixed minimum variance block as the first-level semantic center, and move 64 pixels in each of the four directions (up, down, left, and right) to extract four second-level semantic centers; Finally, the secondary semantic center point is used as the center of the selected subgraph to be described.

4. The wide-swath remote sensing image semantic description method based on adaptive region segmentation as described in claim 1, characterized in that, In each time step, the two features are calculated relatively independently, but they share a set of gating information. The calculation of the three gates is the same as that of a regular LSTM, as follows: I t =σ(W i [X t ,H t-1 ]+b i ) (8) F t =σ(W f [X t ,H t-1 ]+b f ) (9) The t =σ(W o [X t ,H t-1 ]+b o ) (10) Where σ represents the Sigmoid function, X t H represents the input at the current time step. t-1 I represents the hidden layer features of the previous time step. t W represents the input gate. i and b i F represents the weight matrix and offset of the input gate; t W represents the Forgotten Gate. f and b f Represents the weight matrix and offset of the forget gate; O t Indicates the output gate, W o and b o This represents the weight matrix and offset of the output gate; The input and forgetting calculation processes for memory cells c and C are consistent with those of LSTM: Among them, W C and b C and W c and b c These represent the weight matrix and offset of the two linear layers, respectively. This represents the input to memory cell C at time t. C represents the input of memory cell c at time t. t This represents the output of memory cell C at time t after passing through the input gate and the forget gate. t-1 This represents the output of memory cell C at time t-1, where c t This represents the output of memory cell c after passing through the input gate and the forget gate. t-1 F represents the output of memory cell c at time t-1. t Represents the forgetting gate, I t Indicates the input gate; In equations 11 and 12, W C and W c Corresponding to two different linear layer weight matrices; after calculating C t and c t The values ​​are concatenated into a vector, which is then mapped to the same size as the hidden layer vector through an affine transformation and activation layer. Finally, the output is controlled by an output gate and fed into the current hidden layer H. t Strength: H t =O t ⊙tanh(W h [C t ,c t ]+b h ) (15) Among them, O t Indicates the output gate, W h and b h This represents the weight matrix and offset of the linear layer; Then, use a fully connected layer to connect H t A vector mapped to the size of the vocabulary is used to generate the output at that time step, serving as a prediction of the probability distribution of the next word across the entire dictionary: Pred t =softmax(W pred H t +b pred ) (16) Among them, Pred t W represents the generated prediction vector. pred and b pred This represents the weight matrix and offset of the fully connected layer.

5. The wide-swath remote sensing image semantic description method based on adaptive region segmentation as described in claim 1, characterized in that, In step 3.1, the text summarization model Meansum is used to integrate multiple sentences into a summary. This model includes an autoencoder and a summarization module. The role of the autoencoder is to learn the representation of the input sentence "Review" and restrict the generated summary to the linguistic domain. The summarization module is responsible for learning to generate summaries that are semantically similar to each input sentence. Ideally, the autoencoder encodes the input sentence into an intermediate vector, and the summarization module decodes the intermediate vector to generate a summary. Both the autoencoder and the summarization module contain an encoder and a decoder, both implemented using LSTM. The encoder and the encoder of the autoencoder module have the same model parameters, and the decoder of the autoencoder module also has the same parameters. During model learning, the autoencoder and the summarization module each generate two losses: the autoencoder reconstruction loss and the summary generator similarity loss. Let V represent the set of tokenized sentences in the dataset, with the longest sentence denoted as L; the autoencoder encodes the input sentences using LSTM, and the encoder's Φ... E Map the sentence from V to Φ E (x j ) is denoted as z j Input x at each time step j The values ​​of the hidden layer and memory cells generated at the end of the sequence are concatenated [h,c] to obtain the encoding of a sentence x; the mapping direction at the decoding end is exactly opposite to that at the encoding end, Φ D From Mapped to V, the tokens generated by LSTM are given in the form of a probability distribution over a vocabulary; here a reconstruction loss is introduced, namely: it is hoped that the encoder and decoder are functionally inverse, so that the intermediate representation of the sentence input to the encoder can be reconstructed back to the original sentence when passed through a decoder. Since the words reconstructed by the decoder follow a probability distribution, the reconstruction loss is given by the form of cross-entropy in Equation 17: in, Let x represent the reconstruction loss. k Φ represents the k-th token. E Indicates encoder, Φ D Indicates decoder, Denotes cross-entropy loss, x j This represents the j-th token; The summarization module receives multiple intermediate vectors [h, c] encoded by the autoencoder, where h represents the hidden layer output and c represents the memory cell output; it then calculates the mean of the hidden layer and memory cell values ​​for all sentences to obtain the sum. in This represents the mean of the hidden layer output. This represents the mean output of memory cells. The process of calculating this mean is viewed as summarizing all sentences. Then, after calculating the mean... Decoding will produce a textualized summary output. Similarly, the learning process of the summarization module also incurs a loss value. The summarization module learns by re-encoding the decoded token summary, calculating the similarity between the resulting intermediate vector and the intermediate vectors obtained from all sentences in the original input. A higher similarity indicates that the summary contains more complete information from the original input. This similarity is measured using the mean cosine distance. The similarity loss function is given below. as follows: Where, d cos (·,·) represents the cosine distance between two vectors, and the LSTM hidden state after the token-formatted digest is re-encoded by the encoder is denoted as h. s Sentence x j The encoded hidden state is denoted as h. j The similarity loss is obtained by calculating the average cosine distance between the hidden states of all k sentences and the summary; the total loss function that the model needs to optimize is the sum of the two losses.

6. The wide-swath remote sensing image semantic description method based on adaptive region segmentation as described in claim 1, characterized in that, In step 3.2, if the semantic distribution of the image is determined to be uniform in step 1.2, then a fixed pattern of segmentation and integration is adopted; First, generate a description of the semantic clustering features of the image, and then describe the four directions of Upper left, Upper right, Lower left, and Lower right in sequence. If the image semantics are determined to have obvious clustering features in step 1.2, the description order is determined as follows: Let V be the set of all sentences to be sorted; divide V into multiple subsets, where C∈V represents the sentence whose corresponding first-level semantic center falls in the CENTER region, and L, R, D, T represent the sets of sentences that fall in the LEFT, RIGHT, DOWNSIDE, and TOP regions, respectively. First, based on the VAR value of the image, a sentence is generated to describe the characteristics of the image's semantic distribution. If VAR > a threshold is set, the sentence "The semantic distribution of the image is relatively scattered, showing clustering features" is generated. If C is not empty, output the fixed sentence "On the center area of ​​image," and output the sentences in C in sequence; If L and R are not empty, insert the sentence "On the left / right area of ​​image," and output the sentences in L or R in sequence; If T and D are not empty, insert the sentence "On the top / bottom area of ​​image," and output the sentences in T or D in sequence; If any of the primary semantic centers fall in the four corners, insert the sentence "On the upper / lower left / right corner of image," and output the corresponding sentence. Finally, the final description text is obtained.

7. A system employing the wide-swath remote sensing image semantic description method based on adaptive region segmentation according to any one of claims 1-6, characterized in that, It includes a target sub-region generation module, a target sub-region description module, and a description sentence integration module; The target sub-region generation module is used to adaptively segment and extract key descriptive regions based on the semantic distribution features of the image. The target sub-region description module is used to generate descriptive text for target sub-regions using a small-scale remote sensing image semantic description model; The description sentence integration module is used to merge multiple single sentences according to their clustering relationships to obtain a complete and clear final description; The input of the target sub-region description module is connected to the output of the target sub-region generation module; The input of the description sentence integration module is connected to the output of the target sub-region description module.

8. The wide-swath remote sensing image semantic description system based on adaptive region segmentation as described in claim 7, characterized in that: The target sub-region generation module includes an image semantic distribution feature analysis unit, a semantic clustering unit, and an image segmentation unit. The image semantic distribution feature analysis unit is used to analyze whether there are obvious clustering features in the image semantics. The semantic clustering unit is used to generate semantic centers through clustering. The image segmentation unit is used to segment multiple target sub-regions to be described based on the semantic centers. The target sub-region description module includes an image encoding unit and a text generation unit; the image encoding unit extracts image features at different levels through a ResNet network and an FCN; the text generation unit uses a multi-scale LSTM to generate a single-sentence description for the image; The description sentence integration module includes a summary generation unit and a summary sorting unit. The summary generation unit is used to merge the single-sentence descriptions corresponding to multiple sub-regions according to their clustering relationships. The summary sorting unit is used to sort the integrated sub-region descriptions according to spatial relationships and generate a final description containing the relative position information of each region. In the target sub-region generation module, the input of the image segmentation unit is connected to the output of the semantic clustering unit; In the target sub-region description module, the input end of the text generation unit is connected to the output end of the image encoding unit; In the descriptive sentence integration module, the input of the summary sorting unit is connected to the output of the summary generation unit.

Images