Cross-modal pre-training method and device based on remote sensing image and long text

Abstract

The application provides a cross-modal pre-training method and device based on remote sensing images and long texts, and is applied to the field of artificial intelligence.The method comprises the following steps: inputting long text word vectors and short text word vectors into a text encoder to obtain long text feature tensors and short text feature tensors; inputting remote sensing image slices into an image encoder to obtain image feature tensors; performing dimension reduction on the image feature tensors to obtain coarse-grained image features; determining long text similarity between the long text feature tensors and the image feature tensors, and short text similarity between the short text feature tensors and the coarse-grained image features; determining total loss between the remote sensing image slices and the text data according to a cross-entropy loss function; and performing back propagation based on the total loss to obtain a pre-trained text encoder and a pre-trained image encoder; and the application can train a model that can understand long texts and short texts simultaneously, and improve the fine-grained alignment capability of remote sensing images and texts.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a cross-modal pre-training method and apparatus based on remote sensing images and long text. Background Technology

[0002] Multimodal learning aims to integrate information from different modalities of data (such as text and images). In the field of remote sensing, multimodal learning can be used to align short texts with remote sensing images, thereby improving data understanding and analysis capabilities.

[0003] However, short text image captions cannot provide sufficient detailed descriptions of remote sensing images. The ambiguity and domain specificity between short text and remote sensing images exacerbate the illusion problem, meaning that although some search results may seem correct from a subjective point of view, the actual matching degree between text and image is still not high.

[0004] This demonstrates that the text data and remote sensing image processing methods in related technologies suffer from inaccurate text-image matching. Summary of the Invention

[0005] This invention provides a cross-modal pre-training method and apparatus based on remote sensing imagery and long text, which addresses the shortcomings of existing text data and remote sensing imagery processing methods in terms of inaccurate text-image matching. It enables the training of a basic model that can simultaneously understand both long and short texts, thereby improving the fine-grained alignment capability between remote sensing imagery and text.

[0006] This invention provides a cross-modal pre-training method based on remote sensing images and long text, comprising the following steps.

[0007] The text data of remote sensing image slices is input into a large language model to obtain long and short texts output by the large language model. A keyword phrase segmentation word segmenter is used to encode the long and short texts respectively, obtaining long text word vectors and short text word vectors. The long and short text word vectors are then input into a Transformer-based text encoder to obtain the target size long text feature tensor and short text feature tensor output by the text encoder. Finally, the remote sensing image slices are input into a Transformer-based image encoder to obtain the target size output by the image encoder. The image feature tensor is used; the image feature tensor is reduced in dimensionality based on principal component analysis to obtain coarse-grained image features; the long text similarity between the long text feature tensor and the image feature tensor, and the short text similarity between the short text feature tensor and the coarse-grained image features are determined; based on the long text similarity and the short text similarity, the total loss between the remote sensing image slice and the text data is determined according to the cross-entropy loss function; backpropagation is performed based on the total loss to update the weights of the text encoder and the image encoder to obtain a pre-trained text encoder and a pre-trained image encoder.

[0008] According to the present invention, a cross-modal pre-training method based on remote sensing imagery and long text is provided. The step of inputting text data of remote sensing image slices into a large language model to obtain long text and short text output by the large language model includes: acquiring long text remote sensing image data and short text remote sensing image data of the remote sensing image slices; summarizing the long text remote sensing image data by the large language model to obtain short text; and generating long text by the large language model based on the short text remote sensing image data.

[0009] According to the present invention, a cross-modal pre-training method based on remote sensing imagery and long text is provided. The method involves encoding the long text and the short text respectively to obtain long text word vectors and short text word vectors, comprising: segmenting the long text and the short text respectively based on a position threshold to obtain a first long text and a second long text, and a first short text and a second short text, wherein the first long text and the first short text are located before the position threshold, and the second long text and the second short text are located after the position threshold; interpolating the first long text based on the second long text according to a keyword phrase segmentation formula to obtain long text word vectors; and interpolating the first short text based on the second short text according to the keyword phrase segmentation formula to obtain short text word vectors.

[0010] The keyword phrase segmentation formula includes:

[0011] ;

[0012] in, This indicates the positional encoding after keyword phrase segmentation. This indicates the positional code of the keyword phrase before segmentation. Indicates the word vector position index. This indicates the preset fixed interpolation ratio. Indicates the interpolation position weight. This represents the location threshold.

[0013] According to the present invention, a cross-modal pre-training method based on remote sensing imagery and long text is provided. The method for determining the long text similarity between the long text feature tensor and the image feature tensor, and the short text similarity between the short text feature tensor and the coarse-grained image features, includes: determining a first long text similarity between the long text feature tensor and the image feature tensor; determining a second long text similarity between the image feature tensor and the long text feature tensor; determining a first short text similarity between the short text feature tensor and the coarse-grained image features; and determining a second short text similarity between the coarse-grained image features and the short text feature tensor.

[0014] According to the present invention, a cross-modal pre-training method based on remote sensing imagery and long text is provided. The method involves determining the total loss between the remote sensing image slice and the text data based on the long text similarity and the short text similarity using a cross-entropy loss function. The method includes: obtaining target labels, wherein the target labels are used for correct pairing of the remote sensing image slice and the text data; determining the long text loss between the long text and the remote sensing image slice based on the target labels, the first long text similarity, and the second long text similarity using the cross-entropy loss function; determining the short text loss between the short text and the remote sensing image slice based on the target labels, the first short text similarity, and the second short text similarity using the cross-entropy loss function; and performing a weighted summation of the long text loss and the short text loss to obtain the total loss between the remote sensing image slice and the text data.

[0015] According to the present invention, a cross-modal pre-training method based on remote sensing imagery and long text is provided. After backpropagation based on the total loss to update the weights of the text encoder and the image encoder to obtain a pre-trained text encoder and a pre-trained image encoder, the method further includes: using the pre-trained text encoder and the pre-trained image encoder as a backbone network; evaluating downstream tasks based on the backbone network to obtain downstream task evaluation results, wherein the downstream tasks include at least one of the following: cross-modal retrieval of long text and remote sensing imagery, cross-modal retrieval of short text and remote sensing imagery, zero-shot classification, and semantic localization.

[0016] This invention also provides a cross-modal pre-training device based on remote sensing imagery and long text, comprising the following modules: an input module for inputting text data of remote sensing image slices into a large language model to obtain long and short texts output by the large language model; an encoding module for encoding the long and short texts respectively using a keyword phrase segmentation word segmenter to obtain long text word vectors and short text word vectors; a text module for inputting the long and short text word vectors into a Transformer-based text encoder to obtain target-size long text feature tensors and short text feature tensors output by the text encoder; and an image module for inputting the remote sensing image slices into a Transformer-based image encoder to obtain... The system comprises: an image feature tensor of the target size output by the image encoder; a dimensionality reduction module for performing dimensionality reduction on the image feature tensor based on principal component analysis to obtain coarse-grained image features; a similarity module for determining the long text similarity between the long text feature tensor and the image feature tensor, and the short text similarity between the short text feature tensor and the coarse-grained image features; a loss module for determining the total loss between the remote sensing image slice and the text data based on the long text similarity and the short text similarity, according to the cross-entropy loss function; and an update module for performing backpropagation based on the total loss to update the weights of the text encoder and the image encoder, thereby obtaining a pre-trained text encoder and a pre-trained image encoder.

[0017] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the cross-modal pre-training method based on remote sensing images and long text as described above.

[0018] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the cross-modal pre-training method based on remote sensing images and long text as described above.

[0019] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the cross-modal pre-training method based on remote sensing images and long text as described above.

[0020] This invention provides a cross-modal pre-training method and apparatus based on remote sensing imagery and long text. It utilizes a large language model to perform semantic understanding on text data from remote sensing image slices, generating more descriptive long texts and concise short texts. A keyword phrase segmentation word segmenter is used to encode the text, extracting key information and reducing redundancy. The generated long text word vectors and short text word vectors capture global semantics and local key information, respectively. A Transformer-based text encoder captures long-distance dependencies in the text, generating high-quality long text feature tensors and short text feature tensors. A Transformer-based image encoder effectively extracts global and local features from remote sensing images, generating feature tensors of similar size to the text feature tensors. Consistent image feature tensors are used; dimensionality reduction is achieved through principal component analysis to reduce the dimensionality of image features, extract the most salient features, and reduce computational complexity; similarity between long text features and image features is calculated to capture global semantic alignment; similarity between short text features and coarse-grained image features is calculated to focus on the alignment of key information; the difference between text and image features is quantified using the cross-entropy loss function, and the similarity between long and short texts is combined to ensure that the model achieves accurate alignment at both the global and local levels; model parameters are updated through backpropagation to optimize the performance of the text encoder and image encoder; the final pre-trained encoder can better capture the semantic relationship between text and images, providing powerful feature representations for downstream tasks (such as remote sensing image classification, object detection, etc.). Attached Figure Description

[0021] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating the cross-modal pre-training method based on remote sensing images and long text provided by the present invention.

[0023] Figure 2 This is a flowchart of the cross-modal pre-training method provided by the present invention.

[0024] Figure 3 This is a model structure diagram provided by the present invention.

[0025] Figure 4This is a text encoding flowchart provided by the present invention.

[0026] Figure 5 This is an image encoding flowchart provided by the present invention.

[0027] Figure 6 This is a flowchart of the loss calculation provided by the present invention.

[0028] Figure 7 This is a schematic diagram of the cross-modal pre-training device based on remote sensing images and long text provided by the present invention.

[0029] Figure 8 This is a schematic diagram of the physical structure of the electronic device provided by the present invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0031] In this embodiment of the invention, cross-modal pre-training based on remote sensing imagery and long text aims to enable the model to understand and align complex semantic relationships between images and text by jointly training remote sensing imagery and long text descriptions. Unlike short text, long text contains more detailed information and can describe objects, attributes, and spatial relationships in remote sensing images, which helps improve the model's cross-modal matching ability. Through the joint representation of remote sensing imagery and long text, the model can achieve more accurate image retrieval, classification, and localization functions in various downstream tasks, especially demonstrating higher understanding and reasoning capabilities when dealing with complex remote sensing scenes.

[0032] CLIP (Contrastive Language-Image Pretraining) is an important cross-modal pre-training model that maps images and text to the same vector space through contrastive learning, enabling them to be matched using similarity metrics. CLIP is trained with a large number of image-text pairs, allowing the model to perform zero-shot learning on various visual and language tasks. This means it can effectively perform tasks such as image classification, retrieval, and question answering even without dedicated training data. Due to its strong versatility, CLIP has also achieved excellent results in cross-modal tasks involving remote sensing images and long texts, particularly demonstrating significant advantages in improving the semantic alignment between images and text.

[0033] Related technologies all maintain the basic pre-training framework of the original CLIP and then fine-tune it on a dataset in the remote sensing domain. Therefore, they can only align short texts and remote sensing images within the 77-token limit of CLIP's text encoder.

[0034] Aligning short text with remote sensing imagery presents several challenges. First, short text image captions fail to provide sufficient detail in the remote sensing imagery, making it difficult to accurately capture information about multiple objects, attributes, and spatial relationships within the image, resulting in insufficient fine-grained alignment capabilities. Second, the ambiguity and domain specificity between short text and remote sensing imagery exacerbate the illusion problem; that is, although some search results may appear correct from a subjective perspective, the actual matching degree between text and image remains low. Finally, existing short text-based models cannot globally understand image and text information, affecting the accuracy of downstream tasks such as cross-modal retrieval. Therefore, there is an urgent need to improve the alignment accuracy of models and reduce the illusion problem through richer long text descriptions.

[0035] The purpose of this invention is to address the shortcomings of existing cross-modal pre-training techniques for remote sensing and text, such as insufficient short text descriptions, illusion problems, and limited effective length of CLIP-based model text encoders. This invention proposes a cross-modal pre-training method based on remote sensing imagery and long text. By constructing the LRS2M dataset containing 2M remote sensing image-text pairs and combining it with image captioning data of both short and long texts, this invention aims to train a basic model capable of simultaneously understanding both long and short texts, thereby improving the fine-grained alignment capability between remote sensing imagery and text. This model aligns long text with the feature representations of remote sensing images, enabling it to understand both global and local information of the imagery and utilize information at different scales in different tasks, thus overcoming the deficiencies of existing models in cross-modal retrieval and image-text alignment.

[0036] Optionally, the cross-modal pre-training method based on remote sensing images and long text in this embodiment can be executed by a server, by a terminal device, or by both a server and a terminal device. For example, the cross-modal pre-training method based on remote sensing images and long text in this embodiment can be executed by a server.

[0037] Figure 1 This is a flowchart illustrating the cross-modal pre-training method based on remote sensing imagery and long text provided by the present invention, as shown below. Figure 1 As shown, the method includes the following steps.

[0038] Step 101: Input the text data of the remote sensing image slices into the large language model to obtain the long text and short text output by the large language model.

[0039] In some embodiments, the text data of remote sensing image slices typically includes descriptive information corresponding to the remote sensing image slices, such as geographical location, time, scene type, target object, etc. The text data can be manually annotated or generated by other automated methods.

[0040] Text data is input into large language models (such as GPT, BERT, etc.). Through semantic understanding of the input text, large language models generate richer long texts and more concise short texts.

[0041] The long text generated by the large language model includes global descriptive information, such as global information, contextual relationships, and semantic details of remote sensing image slices. The short text generated by the large language model includes key descriptive information, such as main targets, salient features, or core semantics.

[0042] According to the present invention, a cross-modal pre-training method based on remote sensing imagery and long text is provided, which inputs text data of remote sensing image slices into a large language model to obtain long and short texts output by the large language model, including:

[0043] Obtain long-text remote sensing image data and short-text remote sensing image data for remote sensing image slices;

[0044] Short texts are obtained by summarizing long text remote sensing image data using a large language model;

[0045] Long texts are generated from short text remote sensing image data using a large language model.

[0046] In this embodiment of the invention, during the data preparation process, based on existing publicly available remote sensing short text or long text image datasets, the text data of a remote sensing image slice is supplemented into a long text and a short text using a large language model.

[0047] In other words, the long text-based remote sensing image dataset uses a large language model to summarize and condense the long text to generate a short text; the short text-based dataset uses a multimodal large language model to use the short text as a generation prompt to generate long text from remote sensing image slices.

[0048] In some embodiments, image tiles and their corresponding text descriptions are obtained from remote sensing image datasets. Long text remote sensing image data typically contains detailed descriptions, such as geographic location, time, scene type, target objects, and contextual information. Short text remote sensing image data, on the other hand, provides concise descriptions of the image, usually focusing on key information, such as the main target or salient features.

[0049] Long-text remote sensing image data is input into a large language model. The large language model extracts key information and generates concise short texts through semantic understanding of the long texts. Short-text remote sensing image data is input into the large language model. Based on the semantic information of the short texts, the large language model generates more detailed descriptions of the long texts.

[0050] Through the embodiments of the present invention, the generated short text focuses on core information, which facilitates rapid matching with the coarse-grained features of remote sensing images; the generated long text contains richer semantic information, provides a more comprehensive contextual description, enhances the diversity of text data, and provides richer input for subsequent multimodal alignment tasks.

[0051] Step 102: Encode the long text and short text separately using a keyword phrase segmentation word segmenter to obtain the word vectors of the long text and the word vectors of the short text.

[0052] In this embodiment of the invention, long text and short text are encoded into long text word vectors and short text word vectors respectively by a word segmenter that includes KeyPhrase Extraction (KPS) method.

[0053] Keywords and phrases are converted into vector representations using pre-trained word embedding models (such as Word2Vec, GloVe, FastText) or context-aware word vector models (such as BERT, RoBERTa).

[0054] According to the present invention, a cross-modal pre-training method based on remote sensing imagery and long text is provided, which encodes long text and short text separately to obtain word vectors for long text and word vectors for short text, including:

[0055] The long text and short text are segmented based on the position threshold to obtain the first long text and the second long text, as well as the first short text and the second short text. The first long text and the first short text are located before the position threshold, and the second long text and the second short text are located after the position threshold.

[0056] Based on the keyword phrase segmentation formula, the first long text is interpolated based on the second long text to obtain the word vector of the long text;

[0057] Based on the keyword phrase segmentation formula, the first short text is interpolated based on the second short text to obtain the word vector of the short text;

[0058] Keyword phrase segmentation formulas include:

[0059] ;

[0060] in, This indicates the positional encoding after keyword phrase segmentation. This indicates the positional code of the keyword phrase before segmentation. Indicates the word vector position index. This indicates the preset fixed interpolation ratio. Indicates the interpolation position weight. This represents the location threshold.

[0061] In this embodiment of the invention, the position threshold is 20. At that time, the standard position encoding function is used directly. This is applicable to short texts or keyword phrases that are placed relatively early (before a preset position threshold).

[0062] exist In this process, the positional encoding of long texts is dynamically adjusted, and interpolation positional weights are calculated. ( ), for word vector position index After scaling, we obtain two reference positions: the floor position and the floor position. And the rounding up position: And use linear interpolation to calculate the dynamic encoded value ( ).

[0063] Here, This represents a preset fixed interpolation ratio, used to control the granularity of positional encoding in long texts, i.e., a larger... This reduces the fine-grainedness of positional encoding, making it suitable for longer texts. Smaller It retains more location information and is suitable for medium-length text. This represents the interpolation position weight, used to smooth the encoded value between two reference positions.

[0064] In this embodiment of the invention, for keyword phrases whose positions are less than the position threshold, standard position encoding is used directly to retain fine-grained position information; for keyword phrases whose positions are greater than the position threshold, position encoding is dynamically adjusted by scaling and interpolation to reduce the position encoding complexity of long texts; and interpolation is used to smooth the transition and avoid abrupt changes in the position encoding of long texts.

[0065] Through the embodiments of the present invention, by dynamically adjusting the positional encoding, the problem of keyword and phrase segmentation of long and short texts can be effectively handled. It not only preserves the fine-grained positional information of short texts, but also reduces the complexity of positional encoding of long texts, making it suitable for text encoding processing in multimodal tasks.

[0066] Step 103: Input the long text word vectors and short text word vectors into the Transformer-based text encoder to obtain the target-size long text feature tensor and short text feature tensor output by the text encoder.

[0067] In this embodiment of the invention, long text word vectors and short text word vectors are input into a Transformer-based text encoder for encoding, and a long text feature tensor with a fixed size of 1×512 is output. With short text feature tensor .

[0068] Step 104: Input the remote sensing image slices into the Transformer-based image encoder to obtain the image feature tensor of the target size output by the image encoder.

[0069] In this embodiment of the invention, the remote sensing image slices are preprocessed by adjusting the input remote sensing image slices to a size of 3×224×224, and then performing center cropping, RGB conversion and standardization to obtain the preprocessed remote sensing image slices.

[0070] The preprocessed remote sensing image slices are input into a Transformer-based image encoder for encoding, and the output is a feature code with a fixed size of 1×512. (i.e., image feature tensor).

[0071] Step 105: Dimensionality reduction of the image feature tensor is performed based on principal component analysis to obtain coarse-grained image features.

[0072] In this embodiment of the invention, since short text represents coarse-grained features of an image, principal component analysis (PCA) will be used to transform the image feature tensor into a more coarse-grained feature vector to achieve better alignment. Dimensionality reduction to coarse-grained image features .

[0073] PCA is sensitive to the scale of the data. Therefore, before performing PCA, the image feature tensor usually needs to be standardized, that is, the mean is subtracted and the standard deviation is divided so that the mean of each feature is 0 and the variance is 1.

[0074] For the standardized image feature tensor, its covariance matrix is ​​calculated. Eigenvalue decomposition is then performed on the covariance matrix to obtain eigenvalues ​​and eigenvectors. The eigenvalues ​​correspond to the variance of each principal component, while the eigenvectors define the orientation of the principal components. Based on the magnitude of the eigenvalues, the eigenvectors corresponding to the k largest eigenvalues ​​are selected as the principal components. The choice of k is usually based on the desired degree of dimensionality reduction or determined by some criterion (such as information gain, reconstruction error, etc.).

[0075] The original feature tensor is projected onto the space spanned by the selected principal components (feature vectors) to obtain the dimensionality-reduced feature vectors (i.e., coarse-grained image features).

[0076] Step 106: Determine the long text similarity between the long text feature tensor and the image feature tensor, and the short text similarity between the short text feature tensor and the coarse-grained image features.

[0077] In this embodiment of the invention, cosine similarity is used to calculate the image feature tensor. and long text feature tensor Similarity between them, and coarse-grained image features and short text feature tensor The similarity between them.

[0078] According to the present invention, a cross-modal pre-training method based on remote sensing imagery and long text is provided to determine the long text similarity between long text feature tensors and image feature tensors, and the short text similarity between short text feature tensors and coarse-grained image features, including:

[0079] Determine the first long text similarity between the long text feature tensor and the image feature tensor;

[0080] Determine the second long text similarity between the image feature tensor and the long text feature tensor;

[0081] Determine the first short text similarity between the short text feature tensor and the coarse-grained image features;

[0082] Determine the second short text similarity between the coarse-grained image features and the short text feature tensor.

[0083] In this embodiment of the invention, the image feature tensor and long text feature tensor Coarse-grained image features and short text feature tensor Perform L2 normalization to ensure that the norm of all features is 1 for subsequent similarity calculations. Refer to the following formula for details:

[0084] ;

[0085] in, Represents the original vector, The norm of the original vector This represents the normalized vector.

[0086] The first long-text similarity between the long-text feature tensor and the image feature tensor can be expressed by the following formula:

[0087]

[0088] in, Indicates the similarity of the first long text. Represents the feature tensor of long text. The transpose matrix of the image feature tensor.

[0089] The second long text similarity between the image feature tensor and the long text feature tensor can be expressed by the following formula:

[0090]

[0091] in, Indicates the similarity of the second longest text. Represents the image feature tensor. The transpose matrix representing the feature tensor of a long text

[0092] The first short text similarity between the short text feature tensor and the coarse-grained image features can be expressed by the following formula:

[0093]

[0094] in, Indicates the similarity of the first short text. Represents the feature tensor of short text. The transpose matrix representing the coarse-grained image features.

[0095] The second short text similarity between the coarse-grained image features and the short text feature tensor can be expressed by the following formula:

[0096]

[0097] in, Indicates the similarity of the second short text. Represents coarse-grained image features. The transpose of the short text feature tensor.

[0098] In this embodiment of the invention, normalization aims to ensure that all feature vectors have a length of 1, thereby eliminating differences in numerical magnitude between different feature vectors and making similarity calculation more fair. In vector space, the dot product of two vectors can represent their similarity or correlation. When vectors are normalized, their dot product is equal to their cosine similarity.

[0099] Step 107: Based on the similarity of long text and short text, determine the total loss between remote sensing image slices and text data according to the cross-entropy loss function.

[0100] In this embodiment of the invention, cross-entropy loss is used to calculate the loss between long text and remote sensing image slices. And the loss of short texts and remote sensing image slices. The total loss is calculated by weighted summation. L .

[0101] Cross-entropy loss function is commonly used for classification tasks, where the model outputs the probability distribution of the classes, and the true labels are represented in the form of one-hot encoding.

[0102] According to the present invention, a cross-modal pre-training method based on remote sensing imagery and long text is provided. Based on the similarity between long and short texts, the total loss between remote sensing image slices and text data is determined using a cross-entropy loss function, including:

[0103] Obtain target labels, which are used for the correct pairing of remote sensing image tiles and text data;

[0104] Based on the cross-entropy loss function, the long text loss between the long text and the remote sensing image slice is determined based on the target label, the first long text similarity, and the second long text similarity.

[0105] Based on the cross-entropy loss function, the short text loss between the short text and the remote sensing image slice is determined based on the target label, the first short text similarity, and the second short text similarity.

[0106] The total loss between remote sensing image slices and text data is obtained by weighted summation of long text loss and short text loss.

[0107] In this embodiment of the invention, target labels are constructed, and target labels (targets) are set for each batch of images. These labels correspond to the correct pairing of images and text. The target labels (targets) are a sequence from rank × bs to rank × bs + bs-1, where bs is the batch size and rank is the rank of the current process (used in distributed training).

[0108] The total loss between remote sensing image slices and text data is calculated using the cross-entropy loss function. The goal is to maximize the similarity between each image and the correct text pairing, while minimizing the similarity with mismatched text.

[0109] The loss of long text and remote sensing image tiles can be calculated using the following formula:

[0110]

[0111] in, This represents the loss between long text and remote sensing image slices (long text loss). Represents the cross-entropy function. Indicates the similarity of the second longest text. Indicates the similarity of the first long text. Indicates the target label.

[0112] The loss of short text and remote sensing image tiles can be calculated using the following formula:

[0113]

[0114] in, This represents the loss between short text and remote sensing image slices (short text loss). Represents the cross-entropy function. Indicates the similarity of the second short text. This indicates the similarity of the first short text.

[0115] The total loss is calculated by weighting and summing the losses of long text and remote sensing image slices with those of short text and remote sensing image slices. This sum can be expressed by the following formula:

[0116]

[0117] in, Indicates the total loss. The weights representing the loss between long text and remote sensing image slices. This indicates the loss in long text and remote sensing image slices; The weights representing the loss between the short text and the remote sensing image slice. This indicates the loss of short text and remote sensing image slices.

[0118] Through the embodiments of this invention, by simultaneously considering the similarity between long and short texts and remote sensing image slices, the model can more comprehensively understand the relationship between images and text, thereby improving the accuracy of pairing. Cross-entropy loss can effectively measure the difference between the probability distribution predicted by the model and the true labels, thus guiding model optimization.

[0119] Step 108: Perform backpropagation based on the total loss to update the weights of the text encoder and the image encoder, and obtain the pre-trained text encoder and the pre-trained image encoder.

[0120] In this embodiment of the invention, the backpropagation algorithm first calculates the gradient of the total loss with respect to the model parameters (i.e., the weights of the text encoder and image encoder). The gradient indicates the rate of change of the loss function relative to each weight, i.e., the slope of the loss function in the weight space. Once the gradient is obtained, optimization algorithms (such as stochastic gradient descent, Adam, etc.) can be used to update the weights. The direction of weight updates is the opposite of the gradient because the goal is to minimize the loss function. The magnitude of the updates is typically controlled by the learning rate.

[0121] This process is iterative, with each iteration using a batch (or all) of the training data to calculate the loss and gradients, and updating the weights. As the iterations continue, the model's performance on the training data typically improves gradually.

[0122] A text encoder is used to convert text data into feature vectors. During backpropagation, the weights of the text encoder are updated based on the gradient of the total loss with respect to those weights.

[0123] The image encoder is used to convert remotely sensed image slices into feature vectors. Similarly, the weights of the image encoder are updated based on the gradient of the total loss with respect to these weights.

[0124] After a sufficient number of iterations, the text encoder and image encoder will reach a certain performance level on the training data; at this point, they can be considered to have been pre-trained, that is, they have learned how to extract useful features from text and images.

[0125] Pre-trained text and image encoders can be used for various downstream tasks, such as cross-modal retrieval, multimodal fusion, and zero-shot learning. In these tasks, we can use pre-trained encoders as feature extractors or fine-tune them to suit specific application scenarios.

[0126] Through the steps described above in this embodiment of the invention, semantic understanding of the text data of remote sensing image slices is performed using a large language model to generate more descriptive long texts and concise short texts; a keyword phrase segmentation word segmenter is used to encode the text to extract key information and reduce redundancy, and the generated long text word vectors and short text word vectors capture global semantics and local key information, respectively; the Transformer-based text encoder can capture long-distance dependencies in the text and generate high-quality long text feature tensors and short text feature tensors; the Transformer-based image encoder can effectively extract global and local features of remote sensing images and generate image feature tensors with the same size as the text feature tensors. The model employs several techniques: Principal component analysis (PCA) is used to reduce the dimensionality of image features, extracting the most salient features and reducing computational complexity; similarity between long text features and image features is calculated to capture global semantic alignment; similarity between short text features and coarse-grained image features is calculated to focus on the alignment of key information; cross-entropy loss function is used to quantify the differences between text and image features, and the similarity between long and short texts is combined to ensure accurate alignment at both global and local levels; model parameters are updated through backpropagation to optimize the performance of the text encoder and image encoder; the resulting pre-trained encoder can better capture the semantic relationships between text and images, providing powerful feature representations for downstream tasks such as remote sensing image classification and object detection.

[0127] According to the cross-modal pre-training method based on remote sensing imagery and long text provided by the present invention, after backpropagation based on the total loss to update the weights of the text encoder and image encoder to obtain the pre-trained text encoder and pre-trained image encoder, the method further includes:

[0128] The pre-trained text encoder and the pre-trained image encoder are used as the backbone network.

[0129] The downstream tasks are evaluated based on the backbone network to obtain the downstream task evaluation results. The downstream tasks include at least one of the following: cross-modal retrieval of long text and remote sensing image, cross-modal retrieval of short text and remote sensing image, zero-shot classification, and semantic localization.

[0130] In this embodiment of the invention, the pre-trained text encoder and image encoder are used as the backbone network for downstream tasks. They are evaluated in four downstream tasks: cross-modal retrieval of long text and remote sensing images, cross-modal retrieval of short text and remote sensing images, zero-shot classification, and semantic localization, in order to analyze the cross-modal pre-training effect of remote sensing images and long text.

[0131] The following describes an example of a cross-modal pre-training method based on remote sensing images and long text provided by this invention in a practical application.

[0132] refer to Figure 2 , Figure 2 This is a flowchart of the cross-modal pre-training method provided by the present invention, which includes: data preparation (acquiring images, long text, and short text), image encoding (encoding images through an image encoder to obtain image features), long text encoding (encoding long text through a text encoder to obtain long text features), short text encoding (encoding short text through a text encoder to obtain short text features), loss calculation, backpropagation, and downstream task evaluation.

[0133] refer to Figure 3 , Figure 3This is a model structure diagram provided by the present invention, which includes: input remote sensing image slices, image encoder, long image feature, principal component analysis (PCA), short image feature, short text feature, long text feature, text encoder, and keyword phrase segmentation method (KPS), input long text (The image shows an aerial view of an airport with numerous airplanes parked near a terminal building…) and input short text (A white big plane are on the runway.).

[0134] This invention provides a cross-modal pre-training method based on remote sensing imagery and long text, which consists of steps S1 to S5, as follows: Figure 2 As shown, the model structure involved in the core steps S2~S4 of the method is as follows: Figure 3 As shown.

[0135] Step S1: Data preparation. Based on existing publicly available remote sensing short text or long text image datasets, a large language model is used to supplement the text data of a remote sensing image slice into a long text and a short text. That is, the long text remote sensing image dataset is summarized and condensed by the large language model to generate a short text; while the short text dataset is generated by using the short text as a generation prompt by the multimodal large language model to generate a long text from the remote sensing image slice.

[0136] Step S2: Text Encoding. First, the long and short text data are encoded into word vectors using a tokenizer incorporating the KPS method. Then, these vectors are fed into a Transformer-based text encoder for encoding, outputting a long text feature tensor of fixed size 1×512. With short text feature tensor .

[0137] refer to Figure 4 , Figure 4 This is a text encoding flowchart provided by the present invention, which includes: input text, word segmenter and KPS, word vectors, text encoder, and text feature encoding.

[0138] Specifically, the encoding process described in step S2, text encoding, includes two sub-steps: text segmentation and word vector encoding. Figure 4 As shown.

[0139] Step S21: Text segmentation. Using the KPS method, the true encoding of the word vectors at the first 20 positions is retained, while the positions after position 20 are interpolated to the first 20 positions. The mathematical expression of KPS is:

[0140]

[0141] in, The position code after KPS. This is the position code before KPS. For word vector position index, For a relatively large fixed interpolation ratio, Values ​​between 0 and 1, when A larger value indicates that the new embedding is closer to a later position. To enable the model to effectively process both long and short texts, this process involves simultaneous word segmentation of both texts. and .

[0142] Step S22: Word vector encoding, ... and Each input is fed into the same Transformer text encoder for encoding, and the output is a long text feature tensor with a fixed size of 1×512. With short text feature tensor .

[0143] Step S3: Image encoding. First, the remote sensing image slices are preprocessed into a 3×224×224 tensor, then fed into a Transformer-based image encoder for encoding, outputting a feature code with a fixed size of 1×512. (i.e., image feature tensor), Coarse-grained image features were obtained through PCA dimensionality reduction. .

[0144] refer to Figure 5 , Figure 5 This invention provides an image encoding flowchart, which includes remote sensing image slicing, image preprocessing (including: resizing, center cropping, RGB conversion, and normalization), image encoder, image feature encoding, PCA, and coarse-grained image feature encoding.

[0145] Specifically, the encoding process described in step S3, image encoding, includes three sub-steps: image preprocessing, image encoding, and PCA dimensionality reduction. Figure 5 As shown.

[0146] Step S31: Image preprocessing. Adjust the input image to a size of 3×224×224, and then perform center cropping, RGB conversion, and standardization. The mean values ​​of the three channels after standardization are 0.48145466, 0.4578275, and 0.40821073, and the variances are 0.26862954, 0.26130258, and 0.27577711.

[0147] Step S32: Image encoding. The preprocessed image is input into the Transformer image encoder for encoding, and the output is a feature code with a fixed size of 1×512. .

[0148] Step S33: PCA dimensionality reduction. Since short text represents coarse-grained features of the image, PCA will be used to reduce the dimensionality of the text to a finer level to achieve better alignment. Dimensional reduction .

[0149] Step S4: Loss calculation, using cosine similarity to calculate separately. and , and similarity , , as well as Then, cross-entropy loss is used to calculate the loss between the long text and the remote sensing image slice. And the loss of short texts and remote sensing image slices. The total loss is calculated by weighted summation. L Simultaneously, backpropagation is performed on both the image encoder and the text encoder to update the weights.

[0150] refer to Figure 6 , Figure 6 This is a flowchart of the loss calculation provided by the present invention.

[0151] Specifically, the calculation process described in step S4, loss calculation, includes five sub-steps: feature normalization, similarity calculation, target label construction, loss calculation, and weighted summation. Figure 6 As shown.

[0152] Step S41: Feature normalization. In this step, the features will be normalized separately. and , and Perform L2 feature normalization to ensure that the norm of all features is 1 for subsequent similarity calculations:

[0153] ;

[0154] in, Represents the original vector, The norm of the original vector This represents the normalized vector.

[0155] Step S42: Similarity calculation, using the inner product to calculate the similarity between the image and the text:

[0156]

[0157] in, Representation of features With features similarity, This represents the transpose of the matrix.

[0158] Similarity between images and long text:

[0159]

[0160] in, This represents the similarity between the image and the long text (i.e., the second longest text similarity). Represents the image feature tensor. The transpose of the feature tensor of the long text.

[0161] Similarity between long text and images:

[0162]

[0163] in, This represents the similarity between the long text and the image (i.e., the first long text similarity). Represents the feature tensor of long text. The transpose matrix of the image feature tensor.

[0164] Similarity between coarse-grained images and short texts:

[0165]

[0166] in, This represents the similarity between the coarse-grained image and the short text (i.e., the second short text similarity). Represents coarse-grained image features. The transpose of the short text feature tensor.

[0167] Similarity between short texts and coarse-grained images:

[0168]

[0169] in, This represents the similarity between the short text and the coarse-grained image (i.e., the first short text similarity). Represents the feature tensor of short text. The transpose matrix representing the coarse-grained image features.

[0170] Step S43: Construct target labels. Set target labels (targets) for each batch of images. These labels correspond to the correct pairing of images and text. The target labels (targets) are a sequence from rank × bs to rank × bs + bs-1, where bs is the batch size and rank is the rank of the current process (used in distributed training).

[0171] Step S44: Calculate the loss using the cross-entropy loss function. The goal is to maximize the similarity between each image and the correct text pairing, while minimizing the similarity with unmatched text. Loss for long text and remote sensing image tiles:

[0172]

[0173] in, This represents the loss between long text and remote sensing image slices (i.e., long text loss). Represents the cross-entropy function. Indicates the similarity of the second longest text. Indicates the similarity of the first long text. Indicates the target label.

[0174] Losses in short text and remote sensing image tiles:

[0175]

[0176] in, This represents the loss between short text and remote sensing image slices (i.e., short text loss). Represents the cross-entropy function. Indicates the similarity of the second short text. This indicates the similarity of the first short text.

[0177] Step S45: Weighted summation: The total loss is calculated by weighting and summing the losses of long text and remote sensing image slices with those of short text and remote sensing image slices, and then used for backpropagation.

[0178]

[0179] in, Indicates the total loss. The weights representing the loss between long text and remote sensing image slices. This indicates the loss in long text and remote sensing image slices; The weights representing the loss between the short text and the remote sensing image slice. This indicates the loss of short text and remote sensing image slices.

[0180] Step S5: Downstream task evaluation. The pre-trained text encoder and image encoder are used as the backbone network for downstream tasks, specifically in cross-modal retrieval of long text and remote sensing images, and short text and remote sensing images. The implementation steps of the four downstream tasks in step S5 are as follows:

[0181] Step S51: Cross-modal retrieval of long text and remote sensing images. This task will use pre-trained text encoders and image encoders to fine-tune the training set of the retrieval dataset according to steps S2-S4, so as to better adapt to the data distribution of the retrieval dataset.

[0182] During the inference phase, long texts and remote sensing image slices are feature-encoded using a text encoder and an image encoder, respectively, to obtain feature matrices for all images and texts. and , and The size is n×512, where n is the number of data to be retrieved and 512 is the dimension of the feature vector. Then calculate... and The similarity specifically includes image-to-text similarity. Similarity between text and images :

[0183]

[0184]

[0185] in, Indicates the similarity between images and text. Represents the image feature matrix. This represents the transpose of the text feature matrix. Indicates the similarity between text and images. Represents the text feature matrix. This represents the transpose of the image feature matrix.

[0186] The similarity matrices are all of size n×n. Finally, by counting the elements on the main diagonal of the similarity matrix, the number of maximum values ​​is calculated, representing the number of correct searches:

[0187]

[0188] in, Indicates the number of maximum values. Indicates the total number of objects. , indicates a conditional statement, which checks the object Does similarity to itself equal that of an object? With all other objects The maximum similarity.

[0189] in, It is an indicator function that returns 1 if the condition is true, and 0 otherwise.

[0190] Step S52: Cross-modal retrieval of short text and remote sensing imagery is consistent with cross-modal retrieval of long text and remote sensing imagery. For specific steps, please refer to step S51.

[0191] Step S53: Zero-shot classification. This task involves compiling the image-level labels of remote sensing images into a sentence, i.e., "a satellite photo of {class_name}.", where class_name is the image-level label, and then performing classification retrieval by referring to the method in step S51.

[0192] First, all categories of statements are encoded into category feature matrices using a text encoder. The feature vector has a size of c×512, where c is the number of categories and 512 is the dimension of the feature vector. The image to be classified is then encoded using an image encoder to form an image feature matrix. The feature vector has a size of n×512, where n is the number of images to be classified and 512 is the dimension of the feature vector. Then, matrix multiplication is used to calculate the similarity between the image to be classified and the category. :

[0193]

[0194] in, This indicates the similarity between the image to be classified and the category. Represents the image feature matrix. This represents the transpose of the category feature matrix. The size is n×c.

[0195] Finally, the classification category of each image is determined by finding the maximum similarity between the feature vector of each image to be classified and the category vector of each image. Then, the number of correctly classified images is counted using this formula:

[0196]

[0197] in, This represents a counting function (used to calculate the number of maximum values). Indicates the total number of objects. It is the first The real label of the image Representation Object With object Similarity between them Indicates finding Index of objects that reach the maximum value , It is an indicator function that returns 1 if the condition is true, and 0 otherwise.

[0198] Step S54: Semantic localization. For this task, the pre-trained text encoder and image encoder are used as the backbone network to replace the original text encoder and image encoder. Finally, the performance of the pre-trained text encoder and image encoder is evaluated according to the four evaluation metrics mentioned above. The four evaluation metrics are: The purpose is to calculate the probability ratio between the ground truth (GT) region and other regions, which approaches 1 when the model focuses its attention on the GT region and approaches 0 otherwise. : Used to quantify the distance between the center of the ground truth region and the approximate previous attention. When the model attention is close to the center of the ground truth, it is close to 0, and vice versa. It can measure the degree of attention deviation. : A metric used to quantify the attention divergence of a model, which can be used to assess the stability of a retrieval model and determine whether an image contains the query target or relationship; Taking into account the above three indicators, the SeLo task is comprehensively evaluated from multiple aspects. The calculation formula is as follows:

[0199]

[0200] in, , , .

[0201] This invention designs a visual language cross-modal pre-training method that simultaneously takes into account the understanding of both long and short texts. Unlike traditional visual language cross-modal pre-training methods such as CLIP, this method effectively expands the text length that the model can process through the KPS module, while maintaining the ability to understand the main information of short texts or images.

[0202] This invention designs a model that is pre-trained on 2 million image-text pairs, possessing strong prior knowledge. Without further training of the model, it can be seamlessly transferred to various downstream tasks, and all of them achieve good results.

[0203] Compared with existing technologies, this invention not only expands the processing capabilities for long texts, but also surpasses other pre-trained foundational model backbone networks in multiple downstream tasks. To evaluate the effectiveness of this invention, cross-modal retrieval of long and short texts will be performed on the RSITMD dataset, with the main evaluation metrics being the average of text-to-image and image-to-text recall and the ratio of Top-1 recall to correctness. Zero-shot classification will be performed on the RS_C11 dataset, with accuracy as the evaluation metric. Semantic localization will be performed on the AIR-SLT dataset, with the main evaluation metric being... Furthermore, this invention will also compare with other pre-trained base models RemoteCLIP and GeoRSCLIP under the same configuration, and the results are shown in Table 1.

[0204] Table 1 Downstream Task Evaluation Table

[0205]

[0206] The cross-modal pre-training device based on remote sensing images and long text provided by the present invention will be described below. The cross-modal pre-training device based on remote sensing images and long text described below can be referred to in correspondence with the cross-modal pre-training method based on remote sensing images and long text described above.

[0207] refer to Figure 7 , Figure 7 This is a schematic diagram of the cross-modal pre-training device based on remote sensing images and long text provided by the present invention.

[0208] The input module 701 is used to input the text data of remote sensing image slices into the large language model to obtain the long text and short text output by the large language model.

[0209] The encoding module 702 is used to encode long text and short text separately through a keyword phrase segmentation word segmenter to obtain word vectors for long text and word vectors for short text.

[0210] The text module 703 is used to input long text word vectors and short text word vectors into a Transformer-based text encoder to obtain a long text feature tensor and a short text feature tensor of the target size output by the text encoder.

[0211] Image module 704 is used to input remote sensing image slices into a Transformer-based image encoder to obtain an image feature tensor of target size output by the image encoder;

[0212] The dimension reduction module 705 is used to reduce the dimension of the image feature tensor based on principal component analysis to obtain coarse-grained image features.

[0213] The similarity module 706 is used to determine the long text similarity between the long text feature tensor and the image feature tensor, and the short text similarity between the short text feature tensor and the coarse-grained image features.

[0214] Loss module 707 is used to determine the total loss between remote sensing image slices and text data based on long text similarity and short text similarity, according to the cross-entropy loss function;

[0215] The update module 708 is used to perform backpropagation based on the total loss to update the weights of the text encoder and the image encoder, so as to obtain the pre-trained text encoder and the pre-trained image encoder.

[0216] Specifically, the cross-modal pre-training device based on remote sensing images and long text provided by the present invention can realize all the method steps implemented in the above-mentioned cross-modal pre-training method embodiment based on remote sensing images and long text, and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0217] Figure 8 This is a schematic diagram of the physical structure of the electronic device provided by the present invention, such as... Figure 8As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a cross-modal pre-training method based on remote sensing images and long text. This method includes: inputting text data of remote sensing image slices into a large language model to obtain long text and short text output by the large language model; encoding the long text and short text separately through a keyword phrase segmentation word segmenter to obtain long text word vectors and short text word vectors; inputting the long text word vectors and short text word vectors into a Transformer-based text encoder to obtain long text feature tensors and short text feature tensors of target size output by the text encoder; inputting remote sensing image slices into a Transformer-based... The nsformer image encoder is used to obtain the image feature tensor of the target size output by the image encoder. Principal component analysis is used to reduce the dimensionality of the image feature tensor to obtain coarse-grained image features. Long-text similarity is determined between the long-text feature tensor and the image feature tensor, and short-text similarity is determined between the short-text feature tensor and the coarse-grained image features. Based on the long-text similarity and short-text similarity, the total loss between the remote sensing image slice and the text data is determined using the cross-entropy loss function. Backpropagation is performed based on the total loss to update the weights of the text encoder and the image encoder, resulting in a pre-trained text encoder and a pre-trained image encoder.

[0218] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0219] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the cross-modal pre-training method based on remote sensing images and long text provided by the above methods. The method includes: inputting text data of remote sensing image slices into a large language model to obtain long text and short text output by the large language model; encoding the long text and short text respectively through a keyword phrase segmentation word segmenter to obtain long text word vectors and short text word vectors; inputting the long text word vectors and short text word vectors into a Transformer-based text encoder to obtain the target scale output by the text encoder. The process involves: generating long and short text feature tensors; inputting remote sensing image slices into a Transformer-based image encoder to obtain the target-size image feature tensor output by the image encoder; performing dimensionality reduction on the image feature tensor using principal component analysis to obtain coarse-grained image features; determining the long text similarity between the long text feature tensor and the image feature tensor, and the short text similarity between the short text feature tensor and the coarse-grained image features; determining the total loss between the remote sensing image slices and text data based on the long and short text similarities using the cross-entropy loss function; and performing backpropagation based on the total loss to update the weights of the text encoder and the image encoder, resulting in a pre-trained text encoder and a pre-trained image encoder.

[0220] Furthermore, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, this computer program implements the cross-modal pre-training method based on remote sensing images and long text provided by the methods described above. This method includes: inputting text data from remote sensing image slices into a large language model to obtain long and short texts output by the large language model; encoding the long and short texts separately using a keyword phrase segmentation word segmenter to obtain word vectors for the long and short texts; and inputting the word vectors for the long and short texts into a Transformer-based text encoder to obtain a target-size long text feature tensor and short text feature tensor output by the text encoder. Tensor; Input remote sensing image slices into a Transformer-based image encoder to obtain an image feature tensor of target size output by the image encoder; Perform dimensionality reduction on the image feature tensor based on principal component analysis to obtain coarse-grained image features; Determine the long text similarity between the long text feature tensor and the image feature tensor, and the short text similarity between the short text feature tensor and the coarse-grained image features; Based on the long text similarity and short text similarity, determine the total loss between the remote sensing image slices and the text data according to the cross-entropy loss function; Perform backpropagation based on the total loss to update the weights of the text encoder and the image encoder to obtain a pre-trained text encoder and a pre-trained image encoder.

[0221] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0222] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0223] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cross-modal pre-training method based on remote sensing images and long texts, characterized in that, The method comprises the following steps: inputting text data of a remote sensing image slice into a large language model to obtain long text and short text output by the large language model; encoding the long text and the short text respectively through a keyword phrase segmentation tokenizer to obtain long text word vectors and short text word vectors; inputting the long text word vectors and the short text word vectors into a Transformer-based text encoder to obtain long text feature tensors and short text feature tensors of a target size output by the text encoder; inputting the remote sensing image slice into a Transformer-based image encoder to obtain image feature tensors of the target size output by the image encoder; performing dimension reduction on the image feature tensors based on principal component analysis to obtain coarse-grained image features; determining a long text similarity between the long text feature tensors and the image feature tensors, and a short text similarity between the short text feature tensors and the coarse-grained image features; determining a total loss between the remote sensing image slice and the text data according to a cross-entropy loss function based on the long text similarity and the short text similarity; performing back propagation based on the total loss to update weights of the text encoder and the image encoder to obtain a pre-trained text encoder and a pre-trained image encoder.

2. The cross-modal pre-training method based on remote sensing images and long texts according to claim 1, characterized in that, The method of inputting text data of a remote sensing image slice into a large language model to obtain long text and short text output by the large language model comprises the following steps: obtaining long text remote sensing image-text data of a remote sensing image slice and short text remote sensing image-text data of the remote sensing image slice; summarizing the long text remote sensing image-text data based on a large language model to obtain short text; generating long text based on the short text remote sensing image-text data through the large language model.

3. The cross-modal pre-training method based on remote sensing images and long texts according to claim 1, characterized in that, The method of encoding the long text and the short text respectively to obtain long text word vectors and short text word vectors comprises the following steps: segmenting the long text and the short text based on a position threshold to obtain a first long text and a second long text, and a first short text and a second short text, wherein the first long text and the first short text are located before the position threshold, and the second long text and the second short text are located after the position threshold; interpolating the first long text based on the second long text according to a keyword phrase segmentation formula to obtain long text word vectors; interpolating the first short text based on the second short text according to the keyword phrase segmentation formula to obtain short text word vectors; The keyword phrase segmentation formula comprises: ； wherein, represents the position encoding after keyword phrase splitting, represents the position encoding before keyword phrase splitting, represents the word vector position index, represents the preset fixed interpolation ratio, represents the interpolation position weight, represents the position threshold.

4. The cross-modal pre-training method based on remote sensing images and long texts according to claim 1, characterized in that, The method of determining a long text similarity between the long text feature tensors and the image feature tensors, and a short text similarity between the short text feature tensors and the coarse-grained image features comprises the following steps: determining a first long text similarity between the long text feature tensors and the image feature tensors; determining a second long text similarity between the image feature tensors and the long text feature tensors; determining a first short text similarity between the short text feature tensors and the coarse-grained image features; determine a second short text similarity between the coarse-grained image features and the short text feature tensor.

5. The cross-modal pre-training method based on remote sensing images and long texts according to claim 4, characterized in that, determine a total loss between the remote sensing image slice and the text data according to a cross-entropy loss function based on the long text similarity and the short text similarity, including: obtain a target label, wherein the target label is used for correct pairing of the remote sensing image slice and the text data; determine a long text loss between the long text and the remote sensing image slice based on the target label, the first long text similarity, and the second long text similarity according to a cross-entropy loss function; determine a short text loss between the short text and the remote sensing image slice based on the target label, the first short text similarity, and the second short text similarity according to the cross-entropy loss function; weight and sum the long text loss and the short text loss to obtain the total loss between the remote sensing image slice and the text data.

6. The cross-modal pre-training method based on remote sensing images and long texts according to claim 1, characterized in that, After the total loss is back propagated to update the weights of the text encoder and the image encoder to obtain a pre-trained text encoder and a pre-trained image encoder, the method further includes: using the pre-trained text encoder and the pre-trained image encoder as a backbone network; evaluating a downstream task based on the backbone network to obtain a downstream task evaluation result, wherein the downstream task includes at least one of the following: long text and remote sensing image cross-modal retrieval, short text and remote sensing image cross-modal retrieval, zero-shot classification, and semantic positioning.

7. A cross-modal pre-training device based on remote sensing images and long texts, characterized in that, including: an input module configured to input text data of a remote sensing image slice into a large language model to obtain long text and short text output by the large language model; an encoding module configured to encode the long text and the short text respectively through a keyword phrase segmentation tokenizer to obtain long text word vectors and short text word vectors; a text module configured to input the long text word vectors and the short text word vectors into a text encoder based on a Transformer to obtain a target size long text feature tensor and a target size short text feature tensor output by the text encoder; an image module configured to input the remote sensing image slice into an image encoder based on a Transformer to obtain an image feature tensor of the target size output by the image encoder; a dimension reduction module configured to reduce the dimension of the image feature tensor based on principal component analysis to obtain coarse-grained image features; a similarity module configured to determine a long text similarity between the long text feature tensor and the image feature tensor, and a short text similarity between the short text feature tensor and the coarse-grained image features; a loss module configured to determine a total loss between the remote sensing image slice and the text data according to a cross-entropy loss function based on the long text similarity and the short text similarity; an update module configured to back propagate based on the total loss to update the weights of the text encoder and the image encoder to obtain a pre-trained text encoder and a pre-trained image encoder.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The computer program is executed by the processor to implement the cross-modal pre-training method based on remote sensing images and long texts according to any one of claims 1-6. 9.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the cross-modal pre-training method based on remote sensing images and long texts according to any one of claims 1-6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the cross-modal pre-training method based on remote sensing images and long texts according to any one of claims 1-6.

Images