An interactive image-text alignment method for remote sensing image caption generation
By employing an interactive image-text alignment method, utilizing a lightweight interactive Fourier transform and Fourier layers for multi-scale feature extraction, the problems of insufficient semantic alignment and feature redundancy in remote sensing image caption generation are solved, achieving efficient caption generation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HEILONGJIANG CYBERSPACE RESEARCH CENTER (HEILONGJIANG INFORMATION SECURITY EVALUATION CENTER HEILONGJIANG ACADEMY OF NATIONAL DEFENSE SCIENCE & TECHNOLOGY)
- Filing Date
- 2026-03-17
- Publication Date
- 2026-06-05
AI Technical Summary
Existing methods for generating captions for remote sensing images have limitations in multi-scale feature processing when dealing with differences in spatial resolution, resulting in insufficient semantic alignment between visual content and natural language descriptions. Furthermore, directly using the original remote sensing images leads to data redundancy, increased computational burden, and low generation efficiency.
An interactive image-text alignment method is adopted, which uses a lightweight interactive Fourier transform (IFT) and Fourier layers to extract features at multiple scales. It combines a cross-attention mechanism to achieve accurate semantic alignment between visual content and natural language description, and performs feature selection and compression in the frequency domain.
It achieves precise semantic alignment between visual content and natural language description, reduces feature redundancy, and improves the accuracy and efficiency of subtitle generation.
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image captioning technology, and relates to an interactive image-text alignment method (BITA) for generating captions for remote sensing images. Background Technology
[0002] With the rapid development of remote sensing technology, Remote Sensing Image Captioning (RSIC) technology has shown broad application potential in fields such as geographic information systems, disaster monitoring, and environmental change analysis. However, existing RSIC methods still face many technical bottlenecks in practical applications. First, current methods have significant limitations in handling multi-scale features caused by differences in spatial resolution, failing to achieve accurate semantic alignment between visual content and natural language descriptions, resulting in insufficient accuracy and poor relevance of the generated captions. Second, directly using the original remote sensing image as input not only leads to data redundancy and increased computational burden but also reduces the efficiency of caption generation. Summary of the Invention
[0003] This invention addresses the problems of insufficient semantic alignment and severe feature redundancy in existing RSIC methods by providing an interactive image-text alignment method, comprising the following steps:
[0004] The process involves acquiring the image and text to be aligned, and then feeding the image into a trained IFT model for alignment. The IFT model mainly consists of an image converter, a text converter, and a cross-attention module.
[0005] The image converter encodes the image, the text converter encodes the text, and the cross-attention alignment module interacts with the visual features of the encoded image and the text embedding features of the encoded text to focus the text on the relevant image region, ultimately generating aligned features to decode accurate remote sensing image captions.
[0006] Furthermore, the image converter includes convolutional layers, Fourier transform layers, and a multilayer perceptron;
[0007] The image converter process includes: extracting local details from the image through convolutional layers, capturing global structural information using Fourier transform layers, fusing features through a multilayer perceptron, and outputting visual features rich in spatial and semantic information.
[0008] Furthermore, the text converter includes a token embedding layer and a Fourier layer;
[0009] The text converter process includes: the text converter transforms the input caption text into a token embedding sequence, and captures the long-distance semantic dependencies between words through a Fourier layer to generate text embedding features containing contextual information.
[0010] Furthermore, the cross-attention module includes a cross-attention mechanism;
[0011] The processing steps of the cross-attention module include: receiving visual features from the image converter and text embedding features from the text converter, calculating the correspondence between image regions and text words through the cross-attention mechanism, fusing the information from both, and outputting the aligned joint features.
[0012] Furthermore, the IFT model is pre-trained, and the training process includes:
[0013] Step 1: Obtain multiple remote sensing image datasets and perform uniform processing on all images: adjust the spatial resolution of the images, normalize the color channels of the images, and standardize the caption text.
[0014] Step 2: Convert the format of the remote sensing image dataset, apply random data augmentation to the images, and save the processed image data as a standardized format file to obtain the enhanced standardized remote sensing image dataset.
[0015] Step 3: Divide the enhanced remote sensing image dataset into a training set and a test set. Extract training batches from the training set, construct training groups, and divide the batches.
[0016] Step 4: Input the training batch data into the IFT model for training. Extract the visual features of the image and generate text embedding features through the image converter and text converter. Achieve interactive alignment of image and text features through the cross attention mechanism. Save the final model parameters.
[0017] Step 5: Input the test set into the trained IFT model and generate image captions. Then, evaluate the semantic relevance, structural fluency, and visual consistency of the generated captions.
[0018] Step 6: Determine whether to retrain and optimize based on the evaluation results of the test set. If retraining and optimization are required, expand the training set, adjust the model parameters, optimize the feature extraction module, retrain and optimize the model, and determine the final deployment version.
[0019] Furthermore, in step one, the size of all images is standardized to 224×224 pixels, and a bilinear interpolation algorithm is used; the pixel value range is uniformly scaled to [0, 1].
[0020] Furthermore, in step two, all image files in the remote sensing image dataset are formatted into JPEG files. After the format conversion, color dithering is added to each image to enhance data diversity. Then, all the enhanced image data is saved as NumPy format files.
[0021] Furthermore, in step three, the data-augmented remote sensing image dataset is divided into a training set and a test set in an 8:2 ratio; the data size of each mini-batch is fixed at 32.
[0022] Furthermore, in step five, the semantic relevance index specifically uses BLEU, CIDEr, and ROUGE, mainstream natural language processing evaluation metrics; the manual scoring specifically focuses on the grammatical structure of letters, sentence organization, and the standardization of language expression; and the visual consistency evaluation specifically checks the accuracy of the subtitles' description of the target object's type, location, and scene background.
[0023] Furthermore, in step six, the method of expanding the training set includes adding image samples for the corresponding scene and generating richer caption descriptions for them; the method of adjusting the model parameters includes adjusting the learning rate, batch size, and parameter settings of the Fourier transform layer.
[0024] The beneficial effects of this invention are:
[0025] 1. This invention introduces a lightweight interactive Fourier transform (IFT) and a multi-scale feature extraction mechanism of Fourier layers to capture multi-scale features of images in the frequency domain. It achieves accurate semantic alignment between visual content and natural language description through a cross-attention mechanism, thus solving the technical problem of insufficient semantic alignment in existing methods.
[0026] 2. Significantly reduces feature redundancy. This invention maps image features to the frequency domain through Fourier transform, and performs feature selection and compression in the frequency domain, effectively reducing the redundancy of remote sensing visual features, lowering the computational burden, and improving the efficiency of caption generation tasks. Detailed Implementation
[0027] This embodiment is an interactive image-text alignment method for generating captions for remote sensing images, specifically including the following steps:
[0028] S1. Preprocessing the remote sensing image dataset:
[0029] First, we obtained multiple remote sensing image datasets from publicly available resources, such as the UCM-caption dataset, the RSICD dataset, and the NWPU-caption dataset. These datasets cover a variety of remote sensing scenes, including urban buildings, natural landscapes, and transportation facilities. Each image is accompanied by a corresponding caption description, providing a mapping relationship between visual and semantic information.
[0030] After obtaining the dataset, all images are standardized:
[0031] The spatial resolution of the images is adjusted, and the size of all images is standardized to 224×224 pixels. A bilinear interpolation algorithm is used to minimize the distortion that may occur during image scaling.
[0032] The color channels of the image are normalized to uniformly scale the pixel value range to [0, 1], thereby reducing noise caused by differences in brightness or contrast in the data and ensuring the consistency of input features.
[0033] The subtitle text is standardized by removing redundant spaces, meaningless characters, and special symbols to facilitate subsequent processing. At the same time, any grammatical errors are corrected to enhance the semantic expressiveness of the subtitles.
[0034] S2. Enhancement processing of remote sensing image datasets:
[0035] All image files in the remote sensing image dataset are standardized to ensure that the input format is JPEG. During the format conversion process, the original image quality is preserved to avoid information loss or image quality degradation caused by format conversion.
[0036] After format conversion, color dithering is added to each image to enhance data diversity: by adjusting the brightness, contrast and saturation of the image, changes in lighting conditions are simulated; and Gaussian blur is applied to some images to generate a blur effect through pixel smoothing, simulating focus blur or motion blur scenes, and improving the model's performance on low-quality images.
[0037] All enhanced image data is saved as NumPy format files. During the saving process, efficient data storage tools are used to compress the file size while preserving the complete data structure, facilitating subsequent loading and management.
[0038] S3. Divide the augmented dataset and construct training batches:
[0039] The data-augmented remote sensing image dataset was divided into training and testing sets in an 8:2 ratio to meet data usage requirements. During the partitioning process, to ensure the quality of model training, the pairing relationship between each group of images and its corresponding caption description was strictly maintained to avoid training errors or model performance degradation due to data mismatch. In addition to the proportional partitioning, stratified sampling was performed based on the metadata of the image data (such as geographic location, scene labels, etc.) to improve the diversity and coverage of the training and testing sets.
[0040] After the partitioning is complete, the training and test sets are saved to separate folders. During the saving process, ensure that each folder contains image data and corresponding caption files, and name them according to a standardized format for subsequent loading and management.
[0041] The partitioned training set is fed into the IFT model for learning. Through multiple iterations, the semantic alignment between the input image and the caption description is extracted and optimized. The training set covers diverse scenes and semantic information, enabling the model to learn the correspondence between different image features and caption descriptions. During each training iteration, the training set data is loaded in mini-batches, and the model optimizes the alignment of images and text in the feature space, ultimately improving the model's generation performance.
[0042] The test set is input into the trained IFT model to evaluate its generalization performance. The quality of the generated captions is assessed to verify whether the model can accurately describe unseen remote sensing image content. Evaluation metrics for the test set include semantic relevance (such as BLEU, CIDEr, etc.) and generation fluency; these quantitative metrics are used to judge the model's performance in real-world applications.
[0043] During training, several image-caption pairs are randomly selected from the training set to form a training batch. Each batch covers different scenes and semantic categories, ensuring the model is exposed to diverse data samples during training. Special attention must be paid to the balance of data distribution during batch construction; for multi-class datasets, such as images of different terrain scenes or target objects, the proportion of each class in the training batch should be relatively consistent to avoid oversampling or undersampling of certain classes. The selected training data is randomly grouped and further divided into multiple smaller batches to meet the model's training needs, with each smaller batch having a fixed size of 32.
[0044] The training batches that have undergone data augmentation need to be validated again for their diversity and representativeness. During the validation process, the distribution of different scenarios, categories and data types in the batches is checked to ensure that they can cover all possible feature distributions in the training dataset.
[0045] S4. Train the IFT model using the training set:
[0046] During the training phase, the constructed training batch data is input into the IFT model for training. The structure of the IFT model includes an image converter, a text converter, and a cross-attention alignment module.
[0047] The processing steps for the IFT model include:
[0048] First, the input image is processed by an image converter: the image converter extracts the visual features of the image, transforms them into an embedding representation that can be parsed by the IFT model, and captures multi-scale image features through Fourier transform. The image converter includes convolutional layers, Fourier transform layers, and a multilayer perceptron, where the Fourier transform layers transform image features from the spatial domain to the frequency domain, and perform feature selection and enhancement in the frequency domain.
[0049] After image feature extraction is complete, the caption data from the training batch is input into a text converter. The text converter first preprocesses the caption text, converting it into a token embedding sequence. Subsequently, a Fourier transform layer is used to capture the complex dependencies between text tokens in the frequency domain, generating text embedding features with stronger semantic expressiveness. The generation of text features is performed simultaneously with image feature extraction, providing semantic information for the subsequent cross-attention mechanism.
[0050] After image and text features enter the cross-attention mechanism, the model uses visual cue embeddings and attention weight allocation to achieve interactive alignment of features. In this process, the cross-attention mechanism focuses on visual feature regions related to caption descriptions, while combining semantic information from text embeddings with image features to establish semantic connections between images and text.
[0051] After each training iteration, the model is evaluated using a validation set to assess its performance and convergence status. During evaluation, the model generates captions for the validation set images, and the quality of the generated captions is quantified using metrics such as BLEU and CIDEr. Furthermore, the learning rate, optimization strategy, and regularization parameters are dynamically adjusted based on the validation results to ensure the model gradually converges to its optimal state, avoiding overfitting or underfitting.
[0052] After training, the final optimized model parameters are saved to a file for subsequent testing and practical applications. During the saving process, the model's hyperparameter settings, the number of training iterations, and the validation set evaluation results are recorded to provide a basis for subsequent performance analysis and model improvement.
[0053] S5. Evaluate the trained IFT model using the test set:
[0054] The prepared test set data is fed into the trained IFT model one by one, and the model generates corresponding image captions. Each set of data in the test set includes an image and a reference caption. Based on the input image features and semantic alignment capabilities, the model generates a natural language description that matches the image content. In this process, the model's text generation module plays a role, outputting high-quality image captions through the learned semantic mapping relationships.
[0055] Semantic relevance metrics are used to quantitatively evaluate the generated subtitles and reference subtitles. Specifically, mainstream natural language processing evaluation metrics such as BLEU, CIDEr, and ROUGE are used to compare the overlap between the generated subtitles and reference subtitles at the word and phrase levels.
[0056] After semantic relevance assessment, professional reviewers score the generated subtitles for structural fluency. Human review primarily focuses on the correctness of the subtitle's grammatical structure, the rationality of sentence organization, and the conformity of language expression to natural language norms. Reviewers assign scores to each subtitle according to the scoring criteria, evaluating the model's language ability in generating natural language descriptions.
[0057] The generated subtitles are compared with the corresponding input image content to verify whether the subtitles accurately describe the key visual information in the image. Visual consistency assessment mainly checks whether the subtitles correctly describe the type, location, and background of the target object.
[0058] The overall performance of the model is comprehensively analyzed based on the evaluation results of semantic relevance, structural fluency, and visual consistency. The scores of the automatic evaluation metrics and the human scoring results are weighted and calculated to generate a comprehensive performance report, which serves as the basis for measuring the model's generation capability.
[0059] S6. Optimize model performance:
[0060] By combining quantitative metrics from the test set evaluation with human scoring results, the model's performance in different scenarios is systematically analyzed. The focus is on the accuracy, fluency, and semantic consistency of the generated subtitles with the image content. Shortcomings in the model's output are identified, and the analysis provides clear guidance for further model improvement.
[0061] When the evaluation results revealed that the model was underlearning certain scenes (such as rare terrain types or special target objects), the model's performance was improved by expanding the training set. Expansion methods included adding image samples for the corresponding scenes and generating richer caption descriptions. The expanded training set was then reused for model training to further optimize the model's adaptability and generative capabilities across multiple scenes.
[0062] Based on the analysis results, key parameters of the model were adjusted to optimize its performance, including adjusting the learning rate, batch size, and parameter settings of the Fourier transform layer. To address issues such as inaccurate subtitle generation or poor extraction of specific features, the feature extraction module of the IFT model was optimized, for example, by increasing the number of channels in the Fourier transform layer and adjusting the number of attention heads.
[0063] After model optimization, the adjusted model is retrained using an expanded and optimized training set for multiple rounds. During training, performance metrics are continuously monitored to ensure gradual improvement on new data. After training, the model is comprehensively evaluated again using a test set to verify the effectiveness of the optimization measures. By comparing performance data before and after optimization, it is confirmed whether the optimized model meets the design requirements.
[0064] After ensuring that the model performance meets the expected goals, the optimized model is defined as the final deployment version. The final model needs to undergo multiple verifications and tests. After the deployment version is determined, all optimization steps, parameter settings, and test results are recorded to provide a complete technical archive for subsequent model maintenance and version updates.
Claims
1. An interactive image-text alignment method for generating captions on remote sensing images, characterized in that, Includes the following steps: The process involves acquiring the image and text to be aligned, and then feeding the image into a trained IFT model for alignment. The IFT model mainly consists of an image converter, a text converter, and a cross-attention module. The image converter encodes the image, the text converter encodes the text, and the cross-attention alignment module interacts with the visual features of the encoded image and the text embedding features of the encoded text to focus the text on the relevant image region, ultimately generating aligned features to decode accurate remote sensing image captions.
2. The interactive image-text alignment method for generating letters in remote sensing images according to claim 1, characterized in that: The image converter includes convolutional layers, Fourier transform layers, and a multilayer perceptron; The image converter process includes: extracting local details from the image through convolutional layers, capturing global structural information using Fourier transform layers, fusing features through a multilayer perceptron, and outputting visual features rich in spatial and semantic information.
3. The interactive image-text alignment method for generating letters in remote sensing images according to claim 1, characterized in that: The text converter includes a token embedding layer and a Fourier layer; The text converter process includes: the text converter transforms the input caption text into a token embedding sequence, and captures the long-distance semantic dependencies between words through a Fourier layer to generate text embedding features containing contextual information.
4. The interactive image-text alignment method for generating letters in remote sensing images according to claim 1, characterized in that: The cross-attention module includes a cross-attention mechanism; The processing steps of the cross-attention module include: receiving visual features from the image converter and text embedding features from the text converter, calculating the correspondence between image regions and text words through the cross-attention mechanism, fusing the information from both, and outputting the aligned joint features.
5. The interactive image-text alignment method for generating letters in remote sensing images according to claim 1, characterized in that: The IFT model is pre-trained, and the training process includes: Step 1: Obtain multiple remote sensing image datasets and perform uniform processing on all images: adjust the spatial resolution of the images, normalize the color channels of the images, and standardize the caption text. Step 2: Convert the format of the remote sensing image dataset, apply random data augmentation to the images, and save the processed image data as a standardized format file to obtain the enhanced standardized remote sensing image dataset. Step 3: Divide the enhanced remote sensing image dataset into a training set and a test set. Extract training batches from the training set, construct training groups, and divide the batches. Step 4: Input the training batch data into the IFT model for training. Extract the visual features of the image and generate text embedding features through the image converter and text converter. Achieve interactive alignment of image and text features through the cross attention mechanism. Save the final model parameters. Step 5: Input the test set into the trained IFT model and generate image captions. Then, evaluate the semantic relevance, structural fluency, and visual consistency of the generated captions. Step 6: Determine whether to retrain and optimize based on the evaluation results of the test set. If retraining and optimization are required, expand the training set, adjust the model parameters, optimize the feature extraction module, retrain and optimize the model, and determine the final deployment version.
6. The interactive image-text alignment method for generating letters in remote sensing images according to claim 5, characterized in that: In step one, the size of all images is standardized to 224×224 pixels, and a bilinear interpolation algorithm is used; the pixel value range is uniformly scaled to [0, 1].
7. The interactive image-text alignment method for generating letters in remote sensing images according to claim 5, characterized in that: In step two, all image files in the remote sensing image dataset are formatted into JPEG files. After the format conversion, color dithering is added to each image to enhance data diversity. Then, all the enhanced image data is saved as NumPy format files.
8. The interactive image-text alignment method for generating letters in remote sensing images according to claim 5, characterized in that: In step three, the data augmented remote sensing image dataset is divided into a training set and a test set in an 8:2 ratio; the size of each batch of data is fixed at 32.
9. The interactive image-text alignment method for generating letters in remote sensing images according to claim 5, characterized in that: In step five, the semantic relevance index specifically uses BLEU, CIDEr, and ROUGE, mainstream natural language processing evaluation metrics; the manual scoring specifically focuses on the grammatical structure of letters, sentence organization, and the standardization of language expression; and the visual consistency evaluation specifically checks the accuracy of the subtitles' description of the target object's type, location, and scene background.
10. The interactive image-text alignment method for generating letters in remote sensing images according to claim 5, characterized in that: In step six, the method of expanding the training set includes adding image samples of the corresponding scene and generating richer caption descriptions for them; the method of adjusting the model parameters includes adjusting the learning rate, batch size, and parameter settings of the Fourier transform layer.