Remote sensing image changing subtitle generation method, system and device and medium

Abstract

The invention relates to the technical field of remote sensing image processing, in particular to a method, a system, equipment and a medium for generating remote sensing image changing subtitles, and the method comprises the following steps: acquiring a to-be-processed dual-time remote sensing image, and inputting the to-be-processed dual-time remote sensing image into a constructed changing subtitle generator to obtain dual-time image changing subtitles; wherein the changed subtitle generator comprises an image processing model and a changed subtitle generation model; the image processing model is obtained based on a neural network model and is used for processing the input dual-time remote sensing image to obtain a first feature, the first feature is input to the changed subtitle generation model, and the first feature is processed according to an instruction to generate changed subtitles corresponding to the dual-time image. According to the method, the details, the accuracy and the context correlation of the changed subtitles generated by the dual-time remote sensing image are remarkably improved.

Description

Technical Field

[0001] The invention relates to the technical field of remote sensing image processing, and in particular to a method, system, device and medium for generating remote sensing image changing subtitles. Background Art

[0002] Remote sensing image processing has important practical applications in meteorology, agriculture, urban planning, environmental monitoring, resource management, geological exploration, disaster management and homeland security. Remote Sensing Image Change Caption (RSICC) is a technology that uses natural language to describe the differences between remote sensing images at two time points. However, traditional remote sensing image change captioning methods usually rely on convolutional neural networks to extract image features and use recurrent neural networks for text generation, which may result in the inability to capture the details of complex changes. The introduction of the Transformer algorithm enables the use of attention mechanisms to better generate remote sensing image change captions and capture more interesting change information.

[0003] However, the integration of Transformer models, while improving feature extraction through self-attention mechanisms, still requires a large amount of annotated data. And in current methods, there is no approach to extract more unique and discriminative representation information from the perspective of the image itself. Most importantly, current RSICC methods rarely fully exploit the potential of combining visual data processing with high-level language modeling, resulting in a disconnect between detected changes and their descriptions, which emphasizes the need for more integrated and adaptive approaches in RSICC methods. Summary of the invention

[0004] In view of the shortcomings in the prior art, the present invention provides a method, system, device and medium for generating remote sensing image changing subtitles, which solves the problems of low accuracy and context relevance of current remote sensing image changing subtitles, and improves the effect of dual-time remote sensing image changing subtitles.

[0005] The objective of the present invention is achieved by the following technical solutions:

[0006] In a first aspect, the present invention provides a method for generating remote sensing image changing subtitles, comprising:

[0007] Obtaining a dual-time remote sensing image to be processed, inputting the dual-time remote sensing image to be processed into a constructed change subtitle generator, and obtaining a dual-time image change subtitle;

[0008] The changing subtitle generator includes an image processing model and a changing subtitle generation model; the image processing model is obtained based on a neural network model and is used to process the input dual-time remote sensing image to obtain a first feature, the first feature is input to the changing subtitle generation model, and the first feature is processed according to the instruction to generate changing subtitles corresponding to the dual-time image;

[0009] The changing subtitle generator is trained by a remote sensing image data set, and the remote sensing image data set includes a plurality of paired dual-time remote sensing images.

[0010] As a further improvement of the present invention, the image processing model includes a visual encoder, a change perception module and a multi-layer perception module connected in sequence;

[0011] The visual encoder is used to extract features from the input dual-time remote sensing image to obtain a visual token;

[0012] The change perception module is used to process the visual tokens to obtain category tokens and difference tokens, and to splice the category tokens and difference tokens to obtain spliced ​​feature tokens;

[0013] The multi-layer perception module is used to linearize the feature vectors of the concatenated feature tokens, introduce nonlinearity through an activation function, and then perform linear transformation again to obtain the first feature.

[0014] As a further improvement of the present invention, the change perception module specifically includes:

[0015] In each layer of the visual encoder output, category tokens are selected from the visual tokens, the cosine similarity between the tokens excluding the category tokens is calculated, the token with the smallest cosine similarity is selected as the difference token, the difference token is concatenated with the category token, and the concatenated feature token obtained after concatenation is used as the input of the multi-layer perception module.

[0016] As a further improvement of the present invention, the image processing model also includes a visual change adaptation module, which is connected to the visual encoder and is used to obtain dual-time remote sensing images and adjust the dual-time remote sensing images so that the adjusted dual-time remote sensing images are adapted to the visual encoder.

[0017] As a further improvement of the present invention, the changing subtitle generation model adopts the LLM algorithm model, including an adapter module and a multi-head attention mechanism module, and adopts a feedforward neural network as a network framework. The adapter module is embedded in each layer of the attention mechanism in the multi-head attention mechanism module, and is used to adjust the first input feature and the instruction feature to adapt each layer of the attention mechanism; the changing subtitle generation model specifically includes:

[0018] Performing root mean square normalization processing on the first feature data sample and the corresponding data label to obtain the second feature;

[0019] The second feature is transposed, and the transposed feature is convolved. After being processed by the adapter module and the multi-head attention mechanism module, the dual-time image changing subtitles are output.

[0020] As a further improvement of the present invention, a mask mechanism is introduced into the processing process of the multi-head attention mechanism module, and the mask mechanism is used to set the elements below the mask diagonal to 0, and keep the elements on and above the diagonal to negative infinity. The mask mechanism is used to set the attention weight after the current position to negative infinity.

[0021] As a further improvement of the present invention, the remote sensing image dataset adopts the LEVIR-CC dataset.

[0022] In a second aspect, the present invention provides a remote sensing image changing subtitle generation system, which is used to implement a remote sensing image changing subtitle generation method, comprising:

[0023] The subtitle generation module is used to obtain the dual-time remote sensing image to be processed, input the dual-time remote sensing image to be processed into the constructed change subtitle generator, and obtain the dual-time image change subtitle;

[0024] A model building module, wherein the changing subtitle generator is trained by a remote sensing image data set, wherein the remote sensing image data set includes a plurality of paired dual-time remote sensing images;

[0025] A changing subtitle generator includes an image processing model and a changing subtitle generation model; the image processing model is obtained based on a neural network model, and is used to process an input dual-time remote sensing image to obtain a first feature, the first feature is input to the changing subtitle generation model, and the first feature is processed according to an instruction to generate changing subtitles corresponding to the dual-time image.

[0026] In a third aspect, the present invention further provides an electronic device, comprising a processor and a memory, wherein the processor is used to execute a computer program stored in the memory to implement the method for generating changing subtitles of remote sensing images as described above.

[0027] In a fourth aspect, the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores at least one instruction, and when the at least one instruction is executed by a processor, the method for generating changing subtitles of remote sensing images as described above is implemented.

[0028] The beneficial effects of the present invention are as follows: a method for generating changing subtitles of remote sensing images of the present invention can accurately realize the generation of changing subtitles of dual-time remote sensing images through a changing subtitle generator, and the changing subtitle generator includes an image processing model and a changing subtitle generation model. The image processing model is used to deeply analyze the details of the dual-time remote sensing images and extract features that characterize the changes. These features are then converted into specific descriptive language by the changing subtitle generation model, so that it is possible to accurately point out which areas in the image have undergone what kind of changes, providing important clues for further detailed analysis. The changing subtitle generation model can perform analysis and processing based on the output of the image processing model, can identify subtle changes, and improve the accuracy of change detection. The present invention significantly improves the details, accuracy, and contextual relevance of the changing subtitles generated from dual-time remote sensing images.

[0029] Furthermore, the present invention ensures that the system focuses on the most critical changes by effectively managing the selection of visual tokens and fine-tuning the interaction between visual data and text descriptions through a change perception module and a change adaptation module, and effectively illustrates the important changes in dual-time images. In addition, by freezing the parameters of the pre-trained large language model, the present invention significantly reduces training overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0031] Figure 1 It is a schematic flow chart of a method for generating changing subtitles of remote sensing images in an embodiment of the present invention;

[0032] Figure 2 It is a structural schematic diagram of a method for generating changing subtitles of remote sensing images in an embodiment of the present invention;

[0033] Figure 3 Schematic diagram of the structure of a system for generating captions of remote sensing images according to an embodiment of the present invention;

[0034] Figure 4 It is a schematic diagram of the structure of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to make the purpose and technical solution of the present invention clearer and easier to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. The specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0036] Terminology explanation:

[0037] CLIP: The full name is Contrastive Language-Image Pre-training. It is a multimodal learning model proposed by OpenAI in 2021. It uses large-scale image and text data pairs for self-supervised learning, and can map text and images into the same embedding space, so that semantically related text and images are close to each other in the vector space. The architecture of the CLIP model consists of two main components: a text encoder and an image encoder. The text encoder uses the Transformer architecture, while the image encoder tries two different models: ResNet-50 and Vision Transformer (ViT). The CLIP model is trained by contrastive learning, and the goal is to learn an embedding space so that similar sample pairs are close to each other, while dissimilar samples are far apart.

[0038] Example: a single data point or sample;

[0039] Lable: data label;

[0040] Transformer: Attention mechanism;

[0041] RMSNorm: root mean square normalization;

[0042] Dropout: A commonly used neural network regularization method that aims to reduce overfitting. Its basic principle is to randomly select a part of neurons with a certain probability during each training process and set their output values ​​to 0, that is, these neurons are "discarded" in this training and do not participate in forward propagation and backpropagation;

[0043] LLaMA-7B: Large Language Model in the Meta AI, which is one of the LLaMA family of models designed to understand and generate natural language text through large-scale self-supervised learning;

[0044] AdamW: An optimization algorithm based on adaptive estimation, which combines the advantages of the momentum and RMSprop algorithms. It adjusts the learning rate by calculating the first-order moment (mean) and second-order moment (unsquared mean) of the gradient;

[0045] Dual-Time Images: Dual-Time Images;

[0046] Vision Encoder: Vision encoder;

[0047] Difference Tokens: Difference tokens;

[0048] Change Aware Moudle: Change awareness module;

[0049] Vision Change Adapter: Vision change adapter module;

[0050] Class Tokens: Class tokens are used to represent categories or category-related information;

[0051] Visual Tokens: Visual tokens represent the visual features of an image, extracted by a visual encoder;

[0052] MLP: Multilayer Perceptron, used to process and transform features;

[0053] LLM: Large Language Model;

[0054] Change Caption: change the subtitles and output them in the end;

[0055] Instruction: Instructions given by the user to the model;

[0056] Multi-Head Attention: Multi-head attention;

[0057] Feed Forward Network: Feedforward neural network, used to further process and transform features to generate the final description;

[0058] Tokenizer: A tool that breaks text into smaller units such as words or characters.

[0059] LEVIR-CC: A large-scale building change detection dataset. It is mainly used for building change detection tasks in the fields of computer vision and remote sensing, helping researchers and developers to train and test change detection algorithms.

[0060] The concept of the present invention is to provide a method, system, device and medium for generating remote sensing image changing subtitles, wherein the method mainly includes:

[0061] Obtaining a dual-time remote sensing image to be processed, inputting the dual-time remote sensing image to be processed into a constructed change subtitle generator, and obtaining a dual-time image change subtitle;

[0062] The changing subtitle generator includes an image processing model and a changing subtitle generation model; the image processing model is obtained based on a neural network model and is used to process the input dual-time remote sensing image to obtain a first feature, the first feature is input to the changing subtitle generation model, and the first feature is processed according to the instruction to generate changing subtitles corresponding to the dual-time image;

[0063] The changing subtitle generator is trained by a remote sensing image data set, and the remote sensing image data set includes a plurality of paired dual-time remote sensing images.

[0064] Among them, the image processing model includes a visual encoder, a change perception module and a multi-layer perception module connected in sequence.

[0065] The visual encoder is used to extract features from the input dual-time remote sensing image to obtain visual tokens; the change perception module is used to process the visual tokens to obtain category tokens and difference tokens, and to splice the category tokens and difference tokens to obtain spliced ​​feature tokens; the multi-layer perception module is used to linearize the feature vector of the spliced ​​feature token, introduce nonlinearity through an activation function, and then perform linear transformation again to obtain the first feature.

[0066] The change perception module specifically includes: in each layer of the visual encoder output, category tokens are selected from the visual tokens, the cosine similarity between tokens excluding the category tokens is calculated, the token with the smallest cosine similarity is selected as the difference token, the difference token is concatenated with the category token, and the concatenated feature token obtained after concatenation is used as the input of the multi-layer perception module.

[0067] The image processing model also includes a visual change adaptation module, which is connected to the visual encoder and is used to obtain the dual-time remote sensing image and adjust the dual-time remote sensing image to make the adjusted dual-time remote sensing image adapt to the visual encoder.

[0068] The changing subtitle generation model adopts the LLM algorithm model, including an adapter module and a multi-head attention mechanism module, and adopts a feedforward neural network as the network framework. The adapter module is embedded in each layer of the attention mechanism in the multi-head attention mechanism module, and is used to adjust the first input feature and the instruction feature to adapt each layer of the attention mechanism; the changing subtitle generation model specifically includes:

[0069] The first feature data sample and the corresponding data label are root mean square normalized to obtain the second feature; the second feature is transposed, and the transposed feature is convolved, and after being processed by the adapter module and the multi-head attention mechanism module, the dual-time image change subtitles are output.

[0070] The multi-head attention mechanism module processing process introduces a mask mechanism, which is used to set the elements below the mask diagonal to 0, and keep the elements on the diagonal and above to negative infinity. The mask mechanism is used to set the attention weight after the current position to negative infinity.

[0071] The changing subtitle generator can accurately generate changing subtitles for dual-time remote sensing images.

[0072] The technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings and specific embodiments, wherein the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments.

[0073] Example 1

[0074] like Figure 1 , Figure 2 As shown, this embodiment provides a method for generating changing subtitles of remote sensing images, and the following is a specific implementation method.

[0075] Obtain the popular general dual-temporal remote sensing image dataset LEVIR-CC. Use the dual-temporal remote sensing images in the LEVIR-CC dataset as the training-test-validation set of the model, for example, 7590 pairs of dual-temporal remote sensing images as the training set, 2135 pairs of dual-temporal remote sensing images as the test set, and 1438 pairs of dual-temporal remote sensing images as the test set.

[0076] Construct a changing subtitle generator based on a deep learning network architecture. The changing subtitle generator includes an image processing model and a changing subtitle generation model.

[0077] The feature map of the input dual-time remote sensing image pair is input into the visual encoder for feature extraction to obtain the dual-time image encoding (i.e., visual tokens), and then extract 6 category tokens [CLS] from every fourth layer of the visual encoder through the change perception module, and then calculate X A and X B The cosine similarity of the D difference tokens [DIF] with the smallest cosine similarity is selected. The visual encoder in this embodiment adopts ViT-L / 14. A and X B The cosine similarity calculation expression is:

[0078]

[0079] Where |||| is the vector modulus, CosineSimilarity is the cosine similarity, A and B are the input features of the dual-temporal remote sensing images, respectively.

[0080] In order to match the input of the large language model, [CLS] and [DIF] are concatenated and input into the multi-layer perception module hθ The multi-layer perception module is transformed through a linear layer, then nonlinearity is introduced using the SiLU activation function, and finally transformed through a linear layer again to obtain the final output. Generate dimensions that meet the requirements of large language models

[0081] Y=Linear(SiLU(Linear(cat[CLS,DIF]))) (2)

[0082] In the formula, SiLU is the activation function, linear is the fully connected layer (i.e., linear layer), and cat is the concatenation operation.

[0083] This embodiment also bridges the gap between the visual task and the large language model through the visual change adaptation module, and adapts the large language model to the remote sensing image change caption task by learning a small number of parameters. Before the first feature Y obtained by the multi-layer perceptron is input into the large language model (LLM), in order to enable the large language model to deeply understand the difference information in the dual-time remote sensing image, the change adaptation module is used to enhance the large language model's understanding of the difference in the dual-time image and integrate the difference or comparison features into the understanding of the input data. First, the first feature Y of the remote sensing dual-time image is calculated A and Y B The difference is calculated and the result is combined with the example and label to generate a new example and label. The original example is an encoding of a "question" and an "answer". The original label and example have the same dimension, but the encoding of the "question" part is all -1. Then a new sample example' and data label label' are formed.

[0084] New samples are input into the large language model for network training. First, example′ is input into an RMSNorm layer for normalization to reduce the complexity of subsequent calculations and improve network stability and training efficiency. The normalization steps are:

[0085]

[0086] Where H is the input feature vector (i.e., the first feature), W is a trainable matrix, ∈ is a very small constant to prevent division by 0, and d is the feature dimension of each token in the LLM.

[0087] Adapt the changes to module h φ (·) is integrated into the attention mechanism of each layer of the multi-head attention mechanism module. In order to facilitate the subsequent convolution operation of the change adaptation module, H is first transposed and then two convolution operations are performed, including:

[0088] The first convolutional layer downsamples H and applies a dropout layer, which helps prevent the network from overfitting during the training phase;

[0089] The second convolutional layer performs upsampling to restore the input data to its original dimension and adds residual links, which helps reduce information loss and link gradient smallness problems.

[0090] Finally, the upsampled output is transposed back to the original input to facilitate further processing by the subsequent network. The specific steps are shown in formula (4):

[0091]

[0092] In the formula, H′ is the first convolutional layer, H″ is the second convolutional layer, Perform downsampling operation on the transposed H, Conv up (H′) represents upsampling the input result of the first convolutional layer.

[0093] At the same time, the change adaptation module with the same structure is also applied to the attention mechanism of the visual encoder, which also allows the pre-trained visual encoder to adapt to the RSICC task more effectively.

[0094] In the training process of the variation subtitle generator of this embodiment, the iteration round is set to 40, the initial learning rate is 0.0001, the pre-trained ViT-L / 14 is used as the visual encoder, 6 category tokens and difference tokens are extracted from every fourth layer of the visual encoder, the large language model adopts LLaMA-7B, and the variation adapter module h φ The hidden dimension of (·) is set to 8, the batch size is set to 8, the AdamW optimizer is used to update the network parameters, and the model weights are saved. A half-precision training model is used throughout the training phase, which not only reduces memory overhead but also improves the training speed. During the test process, the selected model weights are loaded, the test set images are input, and the dual-time image change subtitles are output to complete the remote sensing image change subtitle task.

[0095] During the training process, in order to ensure that the current result depends only on the previous position of the input sequence and is not affected by the subsequent position of the sequence when the self-attention mechanism is calculated, a mask mechanism is introduced, that is, a mask M is introduced. m×m , set the elements below the diagonal of M to 0, and keep the elements above the diagonal to negative infinity, that is, the final M can set the attention weight after the current position to negative infinity. Among them, the calculation expression of the mask mechanism M is:

[0096]

[0097] Guiding attention via masks in multi-head attention modules of large language models:

[0098]

[0099] Where Q is the query vector, K is the key vector, V is the value vector, M is the mask, and W is the q is the query matrix, W k is the bond matrix, W v is the value matrix and m is the mask dimension.

[0100] Following the paradigm of large language models, a 32,000-dimensional vector is finally output, appropriate words are selected in the form of classification, one word is predicted at a time, and the next-token prediction method is used to generate variable subtitles. During the entire training process, the pre-trained large language model network structure is frozen, and only the variable adapter module h is trained. φ (·) and feature adaptation multilayer perceptron h θ For training, the standard cross entropy loss function is used to calculate the loss:

[0101]

[0102] In the formula, represents the cross entropy loss function, R represents the updated data label label′, represents the predicted subtitles, φ and θ represent the learnable parameters that need to be optimized.

[0103] In order to further verify the practicality and effectiveness of the present method, this embodiment also conducts simulation experiments on the present invention and four prior arts (DUDA method, RSICCFormer method, PSNet method, Prompt-CC method).

[0104] In order to verify the simulation experiment effect of the present invention, the present invention and four prior arts are tested on samples using Bleu-N (Bilingual Evaluation Understudy, N = 1, 2, 3, 4), Rouge l The four evaluation indicators, Recall-Oriented Understudy for Gisting Evaluation, Meteor (Metric for Evaluation of Translation with Explicit ORdering) and CIDEr (Consensus-based Image Description Evaluation), are used to evaluate the quality of the remote sensing image changing subtitle results of the simulation experiment of the present invention. The higher the four indicators are, the better the effect of the changing subtitles is.

[0105] The comparison results of the average values ​​of all objective evaluation indicators in the present invention and the above four existing methods are shown in Table 1, and the trainable parameters of the entire network training are shown in Table 2:

[0106] Table 1: Objective evaluation value table of the present invention and the RSICC method on the LEVIR-CC dataset

[0107]

[0108] Table 2: Comparison of trainable parameters between the present invention and the RSICC method on the LEVIR-CC dataset

[0109]

[0110]

[0111] It can be seen from Tables 1 and 2 that the caption generation effect of the present invention on the LEVIR-CC dataset is better than the existing method with the least amount of parameter training. Therefore, by using the training of a large language model and combining the large language model with advanced visual processing technology, the details, accuracy and context relevance of the changing captions generated from dual-time remote sensing images are significantly improved.

[0112] Through the change perception module and the change adaptation module, by effectively managing the selection of visual tokens and fine-tuning the interaction between visual data and text descriptions, we ensure that the system focuses on the most critical changes and effectively illustrates the important changes in dual-time images. In addition, by freezing the parameters of the pre-trained large language model, the training overhead is significantly reduced.

[0113] Example 2

[0114] like Figure 3 As shown, this embodiment provides a system for generating changing subtitles of remote sensing images, which can implement a method for generating changing subtitles of remote sensing images in embodiment 1. The system mainly includes:

[0115] The subtitle generation module is used to obtain the dual-time remote sensing image to be processed, input the dual-time remote sensing image to be processed into the constructed change subtitle generator, and obtain the dual-time image change subtitle;

[0116] A model building module, wherein the changing subtitle generator is trained by a remote sensing image data set, wherein the remote sensing image data set includes a plurality of paired dual-time remote sensing images;

[0117] A changing subtitle generator includes an image processing model and a changing subtitle generation model; the image processing model is obtained based on a neural network model, and is used to process an input dual-time remote sensing image to obtain a first feature, the first feature is input to the changing subtitle generation model, and the first feature is processed according to an instruction to generate changing subtitles corresponding to the dual-time image.

[0118] The implementation principles between the modules have been described in detail in the method for generating remote sensing image changing subtitles in Example 1, and will not be repeated here.

[0119] Example 3

[0120] like Figure 4 The embodiment shown provides an electronic device for implementing the method for generating changing subtitles of remote sensing images in the above-mentioned embodiment 1. The electronic device 100 includes at least one processor 102, a memory 101, a computer program 103 stored in the memory 101 and executable on at least one processor 102, and at least one communication bus 104. The memory 101 can be used to store the computer program 103. The processor 102 implements the steps of the method for generating changing subtitles of remote sensing images in embodiment 1 by running or executing the computer program stored in the memory 101 and calling the data stored in the memory 101. The memory 101 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data (such as audio data) created according to the use of the electronic device 100, etc. In addition, the memory 101 may include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices.

[0121] At least one processor 102 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 102 may be a microprocessor or any conventional processor, etc. The processor 102 is the control center of the electronic device 100, and uses various interfaces and lines to connect various parts of the entire electronic device 100.

[0122] The memory 101 in the electronic device 100 stores a plurality of instructions to implement a method for generating a remote sensing image changing subtitle, and the processor 102 can execute the plurality of instructions to implement:

[0123] Obtaining a dual-time remote sensing image to be processed, inputting the dual-time remote sensing image to be processed into a constructed change subtitle generator, and obtaining a dual-time image change subtitle;

[0124] The changing subtitle generator includes an image processing model and a changing subtitle generation model; the image processing model is obtained based on a neural network model and is used to process the input dual-time remote sensing image to obtain a first feature, the first feature is input to the changing subtitle generation model, and the first feature is processed according to the instruction to generate changing subtitles corresponding to the dual-time image;

[0125] The changing subtitle generator is trained by a remote sensing image data set, and the remote sensing image data set includes a plurality of paired dual-time remote sensing images.

[0126] Example 4

[0127] If the module / unit integrated in the electronic device 100 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. Computer-readable media may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory and read-only memory (ROM, Read-Only Memory).

[0128] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0129] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0130] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for generating remote sensing image changing subtitles, characterized in that: include: Obtaining a dual-time remote sensing image to be processed, inputting the dual-time remote sensing image to be processed into a constructed change subtitle generator, and obtaining a dual-time image change subtitle; The changing subtitle generator includes an image processing model and a changing subtitle generation model; the image processing model is obtained based on a neural network model and is used to process the input dual-time remote sensing image to obtain a first feature, the first feature is input to the changing subtitle generation model, and the first feature is processed according to the instruction to generate changing subtitles corresponding to the dual-time image; The changing subtitle generator is trained by a remote sensing image data set, and the remote sensing image data set includes a plurality of paired dual-time remote sensing images.

2. The method for generating remote sensing image changing subtitles according to claim 1, characterized in that: The image processing model includes a visual encoder, a change perception module and a multi-layer perception module connected in sequence; The visual encoder is used to extract features from the input dual-time remote sensing image to obtain a visual token; The change perception module is used to process the visual tokens to obtain category tokens and difference tokens, and to splice the category tokens and difference tokens to obtain spliced ​​feature tokens; The multi-layer perception module is used to linearize the feature vectors of the concatenated feature tokens, introduce nonlinearity through an activation function, and then perform linear transformation again to obtain the first feature.

3. The method for generating remote sensing image changing subtitles according to claim 2, characterized in that: The change perception module specifically includes: In each layer of the visual encoder output, category tokens are selected from the visual tokens, the cosine similarity between the tokens excluding the category tokens is calculated, the token with the smallest cosine similarity is selected as the difference token, the difference token is concatenated with the category token, and the concatenated feature token obtained after concatenation is used as the input of the multi-layer perception module.

4. The method for generating remote sensing image changing subtitles according to claim 2, characterized in that: The image processing model also includes a visual change adaptation module, which is connected to the visual encoder and is used to obtain the dual-time remote sensing image and adjust the dual-time remote sensing image to make the adjusted dual-time remote sensing image adapt to the visual encoder.

5. The method for generating remote sensing image changing subtitles according to claim 1, characterized in that: The variable subtitle generation model adopts the LLM algorithm model, including an adapter module and a multi-head attention mechanism module, and adopts a feedforward neural network as the network framework. The adapter module is embedded in each layer of the attention mechanism in the multi-head attention mechanism module, and is used to adjust the first input feature and the instruction feature to adapt each layer of the attention mechanism; The changing subtitle generation model specifically includes: Performing root mean square normalization processing on the first feature data sample to obtain a second feature; The second feature is transposed, and the transposed feature is convolved. After being processed by the adapter module and the multi-head attention mechanism module, the dual-time image changing subtitles are output.

6. The method for generating remote sensing image changing subtitles according to claim 5, characterized in that: The multi-head attention mechanism module processing process introduces a mask mechanism, which is used to set the elements below the mask diagonal to 0, and keep the elements on and above the diagonal to negative infinity. The mask mechanism is used to set the attention weight after the current position to negative infinity.

7. The method for generating remote sensing image changing subtitles according to claim 1, characterized in that: The remote sensing image dataset adopts the LEVIR-CC dataset.

8. A system for generating captions of remote sensing images, characterized in that: The method for generating a remote sensing image changing subtitle according to any one of claims 1 to 7 comprises: The subtitle generation module is used to obtain the dual-time remote sensing image to be processed, input the dual-time remote sensing image to be processed into the constructed change subtitle generator, and obtain the dual-time image change subtitle; A model building module, wherein the changing subtitle generator is trained by a remote sensing image data set, wherein the remote sensing image data set includes a plurality of paired dual-time remote sensing images; A changing subtitle generator includes an image processing model and a changing subtitle generation model; the image processing model is obtained based on a neural network model, and is used to process an input dual-time remote sensing image to obtain a first feature, the first feature is input to the changing subtitle generation model, and the first feature is processed according to an instruction to generate changing subtitles corresponding to the dual-time image.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the processor is used to execute a computer program stored in the memory to implement the method for generating changing subtitles of remote sensing images as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, and when the at least one instruction is executed by the processor, the method for generating changing subtitles of remote sensing images according to any one of claims 1 to 7 is implemented.

Images