Remote sensing visual positioning method based on multi-scale progressive reasoning
By adopting a multi-scale progressive inference method in remote sensing visual positioning, combining the differential multi-scale feature aggregation module and the progressive cross-modal attention interaction module, the problem of different target scales and low target significance in high-resolution remote sensing images is solved, and higher positioning accuracy and model generalization capabilities are achieved.
Patent Information
- Application Number
- CN202510161145.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-11-04
- Filing Date
- 2025-02-13
- Publication Date
- 2025-06-06
AI Technical Summary
Existing remote sensing visual positioning methods are difficult to effectively deal with the problems of different target scales and low target significance in high-resolution remote sensing images, and ignore the hierarchical connections between cross-modal information.
The multi-scale progressive inference method is adopted to extract and aggregate the multi-scale information of remote sensing images through the differential multi-scale feature aggregation module, and the progressive cross-modal attention interaction module is used to learn language-guided visual features and visually enhanced text features to explore the connection between cross-modal information.
It effectively solves the problems of different target scales and low target significance in high-resolution remote sensing images, improves positioning accuracy and model generalization capabilities, adapts to complex scenarios, and reduces errors.
Smart Images

Figure CN120107354A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of image processing and natural language processing, and in particular relates to a remote sensing visual positioning method based on multi-scale progressive reasoning. Background Art
[0002] Visual localization methods are also called reference expression understanding, phrase localization, and natural language target retrieval. Remote sensing image visual localization methods perform interactive target query and localization based on text and remote sensing images. Since remote sensing images have the characteristics of large target scale span and low target feature significance, traditional target detection methods are difficult to break through the bottleneck of light. Therefore, by introducing text to enhance the detailed understanding of remote sensing visual image features, high-precision, fine-grained, and interactive visual target detection effects can be achieved.
[0003] Existing research on remote sensing visual localization (RSVG) is still in its early stages of development. Previous visual language models in the remote sensing field generally use features extracted from visual backbone networks to interactively learn with text. Among them, the paper Visual Grounding in Remote Sensing Images was published in the 2022 ACM International Conference on Multimedia journal, which introduced the visual localization task into the remote sensing field for the first time. It proposed a method GeoVG consisting of a language encoder that can learn complex geographic spatial relationships, an image encoder with adaptive regional attention, and a module that fuses text and image features. In 2023, the paper RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Data was published in the IEEE Transactions on Geoscience and Remote Sensing journal. It directly used the features of the convolutional neural network backbone and text features for multimodal fusion, designed a Transformer-based multi-granularity visual language fusion module, and built a new benchmark DIOR-RSVG based on the DIOR dataset, which promoted comprehensive research on RSVG. In 2024, the IEEE Transactions on Geoscience and Remote Sensing journal published the paper Language-Guided Progressive Attention for Visual Grounding in Remote Sensing Images, proposing a progressive attention mechanism and a multi-level feature enhancement module, aiming to focus on spatial and channel information, aggregate visual context information, and construct a new large-scale benchmark dataset OPT-RSVG to promote comprehensive understanding of complex scenes.
[0004] However, the above methods do not perform multi-scale feature extraction and aggregation based on the large scale span of remote sensing image targets and the low significance of target features. The structural design of visual language interaction also tends to be one-way interaction, such as using only visual features as queries and text features as keys and values. This method only focuses on the visual features guided by language, and does not study the language features of visual cues. Although some studies have used visual features and text features as queries to perform cross-attention interaction to obtain two interactive features, and then performed secondary interaction on them, this method ignores the connection between the above two interactive features and does not explore them. Summary of the invention
[0005] The purpose of the present invention is to overcome the problem of visual positioning in the case of high resolution of remote sensing and low target saliency, and propose a remote sensing visual positioning method based on multi-scale progressive reasoning.
[0006] In order to achieve the above object, the present invention adopts the following technical scheme: A remote sensing visual positioning method based on multi-scale progressive reasoning includes the following steps: Obtain remote sensing visual positioning dataset; Constructing a multimodal visual language model, wherein the multimodal visual language model includes a differential multi-scale feature aggregation module and a progressive cross-modal attention interaction module; The remote sensing visual positioning dataset is input into the multimodal visual language model to obtain the target boundary coordinate prediction box.
[0007] Furthermore, the number of pairs of remote sensing images and text instructions annotated in the remote sensing visual positioning data set is no less than 10,000 pairs; The remote sensing image is a remote sensing high-resolution visible light modality image; The text instruction is a text description of the target in the remote sensing high-resolution visible light modality image.
[0008] Furthermore, the remote sensing visual positioning data set is input into the multimodal visual language model to obtain the target boundary coordinate prediction box, specifically: The remote sensing visual positioning dataset is divided into a training set and a validation set; the training set and the validation set are divided in a ratio of 8:2; Use the training set to extract four layers of multi-scale visual features and text features; The four-layer multi-scale visual features are multi-scale aggregated through the differential multi-scale feature aggregation module to obtain visual aggregate features; The visual aggregation features and text features are enhanced through the progressive cross-modal attention interaction module to obtain new visual features and new text features; The new visual features and new text features are decoded to obtain the target boundary coordinate prediction box.
[0009] Furthermore, the method of extracting four-layer multi-scale visual features and text features using the training set is specifically as follows: ResNet50 is used to extract visual features from the images in the training set to obtain four-layer multi-scale visual features. Bert-base is used to extract text features from the instruction texts in the images in the training set to obtain text features. The four-layer multi-scale visual features are multi-scale aggregated through a differential multi-scale feature aggregation module to obtain visual aggregate features, specifically: Input the four-layer multi-scale visual features into the differential multi-scale feature aggregation module for aggregation. Input two adjacent features in the four-layer multi-scale visual features each time. Aggregate three times to output three output features. Add the three output features to obtain the visual aggregation feature. Aggregation includes the following steps: The two adjacent features in the four-layer multi-scale visual features are the first input feature and the second input feature. The differential multi-scale feature aggregation module performs layer normalization, 1×1 convolution and 3×3 depth convolution on the first input feature and the second input feature respectively; the output of the first input feature is used as the query, and the output of the second input feature is used as the key and value. After inputting into the cross attention module of the differential multi-scale feature aggregation module, an output feature is obtained after a 1×1 convolution adjustment channel; The visual aggregation features and text features are enhanced through a progressive cross-modal attention interaction module to obtain new visual features and new text features, specifically: The visual aggregation feature is used as the query, and the text feature is used as the key and value for cross attention calculation to obtain the interactive feature. The interactive feature is used to perform pixel-level weight calculation on the visual aggregation feature through the pixel-level weight calculation formula to obtain the weight of the visual feature. The visual aggregation feature is weighted based on the weight of the visual feature to obtain the visual feature map. The pixel-level weight calculation formula is as follows:
[0010] Among them, α and σ are learnable parameters, Represents the position coordinates in the visual feature map, is the visual aggregation feature, is the interactive feature, and The visual aggregation features and interaction features are mapped to the same space by linear layer and normalization layer respectively; The weighting is as follows:
[0011] in, is the visual feature map, is the weight of the visual feature, It is the visual aggregation feature; The text features are used as the query and the visual feature map is used as the key and value to perform cross-attention calculation to obtain visually enhanced text features; The visual feature map and the visually enhanced text features are used as visual features and text features to cycle through the above steps twice to obtain new visual features and new text features; The new visual features and the new text features are decoded to obtain the target boundary coordinate prediction box, specifically: The new visual feature is input into the 4-layer Transformer decoder to obtain the first decoding feature, the new text feature is linearly mapped to obtain the second decoding feature, and the first decoding feature and the second decoding feature are input into the 4-layer Transformer decoder to obtain the target boundary coordinate prediction box.
[0012] Furthermore, the multimodal visual language model is iteratively optimized after construction, and the iterative optimization includes loss calculation, gradient optimization and round number judgment; The loss calculation includes using the target boundary coordinate prediction box and the real target boundary coordinate value to calculate the loss through a loss calculation formula; The loss calculation formula is as follows:
[0013] in, is the loss target, λ is the weight coefficient, set to 1, For smooth loss, is the GIoU loss, g is the label of the data; The gradient optimization includes optimizing and updating the network weights of the multimodal visual language model using the AdamW optimizer and the Step learning rate strategy according to the loss. The learning rate for the first 60 training rounds is , the learning rate for the last 30 training rounds is ; The round number judgment includes judging whether the current training round number Epoch reaches the preset round number 90. If so, the model parameters are saved and the iterative optimization of the multimodal visual language model is ended. If not, the model verification of the multimodal visual language model is entered, and Epoch=Epoch+1 is set.
[0014] Furthermore, the model verification of the multimodal visual language model is specifically as follows: Input the validation set of the remote sensing visual localization dataset into the multimodal visual language model; After normalizing the input remote sensing visual positioning data set, the visual features and visual masks output by the last layer of the feature extraction network and the fixed-length text features and text masks are extracted, and the visual-text cross-modal feature fusion is performed to obtain the fused features; the fused features are sent to the decoder to obtain the decoded features, and the decoded features are input into the target box regressor for regression to obtain four coordinate values; Select the result presentation method according to the task type. If it is a model reasoning task, draw a coordinate box on the remote sensing image of the input remote sensing visual positioning dataset according to the four coordinate values and save the image; if it is a model evaluation task, calculate the model evaluation index based on the regressed four coordinate values and the read real coordinates, and compare the current model evaluation index with the previous model's highest evaluation index. If it is greater than the model's highest evaluation index, update the weight to the parameters of this round of multimodal visual language model, otherwise enter the next round of iterative optimization of the multimodal visual language model.
[0015] Furthermore, the normalization is to normalize the scale of the input remote sensing image to be less than or equal to 2000×2000 pixels; The model evaluation index calculation includes: Pr@0.5, Pr@0.6, Pr@0.7, Pr@0.8, Pr@0.9, mIoU and cumIoU; The IoU is the value of the intersection of the four regressed coordinate values and the read real coordinates divided by the union, I represents the intersection of the four regressed coordinate values and the read real coordinates, and U represents the union of the four regressed coordinate values and the read real coordinates; The calculation method of Pr@0.5 is the ratio of the number of samples with IoU>0.5 between the predicted value and the true label in the image-text pair to the number of all samples in the dataset; The calculation method of Pr@0.6 is the ratio of the number of samples with IoU>0.6 between the predicted value and the true label in the image-text pair to the number of all samples in the dataset; The calculation method of Pr@0.7 is the ratio of the number of samples with IoU>0.7 between the predicted value and the true label in the image-text pair to the number of all samples in the dataset; The calculation method of Pr@0.8 is the ratio of the number of samples with IoU>0.8 between the predicted value and the true label in the image-text pair to the number of all samples in the dataset; The calculation method of Pr@0.9 is the ratio of the number of samples with IoU>0.9 between the predicted value and the true label in the image-text pair to the number of all samples in the dataset; The mIoU indicator is the mean of all target IoUs in each sample; The cumIoU indicator is the ratio of the sum of I and the sum of U in all prediction results.
[0016] A remote sensing visual positioning system based on multi-scale progressive reasoning, comprising: The data set acquisition module is used to acquire remote sensing visual positioning data sets; A model building module, used to build a multimodal visual language model, wherein the multimodal visual language model includes a differential multi-scale feature aggregation module and a progressive cross-modal attention interaction module; A target prediction module is obtained, which is used to input the remote sensing visual positioning dataset into the multimodal visual language model to obtain the target boundary coordinate prediction box.
[0017] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, a remote sensing visual positioning method based on multi-scale progressive reasoning is implemented.
[0018] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the remote sensing visual positioning method based on multi-scale progressive reasoning is implemented.
[0019] Compared with the prior art, the present invention has the following beneficial technical effects: The present invention provides a remote sensing visual positioning method based on multi-scale progressive reasoning. Aiming at the problems of different target scales and low target saliency in high-resolution remote sensing images, a differential multi-scale feature aggregation module is used to fully extract and aggregate multi-scale information of remote sensing images. Aiming at the problem of ignoring the hierarchical connection between cross-modal information in the existing framework, a progressive cross-modal attention interaction module is proposed to learn language-guided visual features and visually enhanced text features in a way of image-text information fusion, effectively learn the connection between cross-modal information, and learn the contextual connection between visual and text interaction features with a progressive reasoning structure to locate targets in remote sensing high-resolution images.
[0020] The present invention proposes a remote sensing visual positioning system based on multi-scale progressive reasoning. Aiming at the problems of different target scales and low target saliency in high-resolution remote sensing images, the system fully extracts and aggregates multi-scale information of remote sensing images, learns language-guided visual features and visually enhanced text features based on a progressive image-text information fusion method, effectively learns the connection between cross-modal information, and learns the contextual connection between visual and text interaction features with a progressive reasoning structure to locate targets in remote sensing high-resolution images. This can effectively solve the visual positioning problem in the case of high resolution remote sensing images and low target saliency. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] The drawings described herein are for explanation purposes only and are not intended to limit the scope of the present invention in any way. In addition, the shapes and proportional dimensions of the components in the drawings are only for illustration purposes to help understand the present invention, and are not intended to specifically limit the shapes and proportional dimensions of the components of the present invention. In the drawings: Figure 1The present invention is a flowchart of a remote sensing visual positioning method based on multi-scale progressive reasoning.
[0022] Figure 2 This is a structural diagram of a remote sensing visual positioning system based on multi-scale progressive reasoning of the present invention.
[0023] Figure 3 The present invention is a schematic diagram of an electronic device for a remote sensing visual positioning method based on multi-scale progressive reasoning.
[0024] Figure 4 The present invention is a flowchart of a remote sensing visual positioning method based on multi-scale progressive reasoning in an embodiment of the present invention.
[0025] Figure 5 Schematic diagram of a differential multi-scale feature aggregation module according to an embodiment of the present invention. DETAILED DESCRIPTION
[0026] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0027] Embodiment 1 See also Figure 1 , a remote sensing visual positioning method based on multi-scale progressive reasoning, comprising the following steps: Obtain remote sensing visual positioning dataset; Construct a multimodal visual language model, which includes a differential multi-scale feature aggregation module and a progressive cross-modal attention interaction module; The remote sensing visual positioning dataset is input into the multimodal visual language model to obtain the target boundary coordinate prediction box.
[0028] The remote sensing visual positioning method based on multi-scale progressive reasoning provided in this embodiment can improve positioning accuracy, enhance model generalization ability, improve computational efficiency and adapt to complex scenarios by combining a multimodal visual language model and the design of a differential multi-scale feature aggregation module and a progressive cross-modal attention interaction module; through the differential multi-scale feature aggregation module, feature information from different scales can be fused to more accurately locate the target, and the progressive cross-modal attention interaction module allows the model to gradually pay attention to and fuse information from different modalities when processing remote sensing images, thereby reducing errors.
[0029] The multimodal visual language model can process data from different modalities, such as images, text, etc., so that the model can understand the information in remote sensing images more comprehensively, thereby improving its generalization ability. The combination of differential processing and progressive reasoning enables the model to gradually adapt to and respond to changes in the data, further enhancing the generalization ability of the model. The design of the differential multi-scale feature aggregation module and the progressive cross-modal attention interaction module enables the model to utilize computing resources more efficiently when processing remote sensing images. Each module is optimized for specific tasks, thereby improving overall computing efficiency.
[0030] The parallel processing capability of the multimodal visual language model also enables it to process large-scale remote sensing image datasets faster. The differential multi-scale feature aggregation module enables the model to perceive feature information of different scales, and the progressive cross-modal attention interaction module allows the model to fuse information from different modalities. The model can capture more contextual information when processing complex scenes.
[0031] The number of pairs of remote sensing images and text instructions annotated in the remote sensing visual positioning dataset is no less than 10,000; The remote sensing image is a remote sensing high-resolution visible light modality image; The text instructions are text descriptions of targets in remote sensing high-resolution visible light modality images.
[0032] The remote sensing visual positioning dataset is input into the multimodal visual language model to obtain the target boundary coordinate prediction box, specifically: The remote sensing visual positioning dataset is divided into a training set and a validation set; the training set and the validation set are divided in a ratio of 8:2; Use the training set to extract four layers of multi-scale visual features and text features; The four-layer multi-scale visual features are multi-scale aggregated through the differential multi-scale feature aggregation module to obtain visual aggregate features; The visual aggregation features and text features are enhanced through the progressive cross-modal attention interaction module to obtain new visual features and new text features; The new visual features and new text features are decoded to obtain the target boundary coordinate prediction box.
[0033] The four-layer multi-scale visual features and text features extracted using the training set are as follows: ResNet50 is used to extract visual features from the images in the training set to obtain four-layer multi-scale visual features. Bert-base is used to extract text features from the instruction texts in the images in the training set to obtain text features. The four-layer multi-scale visual features are aggregated at multiple scales through the differential multi-scale feature aggregation module to obtain visual aggregation features, specifically: Input the four-layer multi-scale visual features into the differential multi-scale feature aggregation module for aggregation. Input two adjacent features in the four-layer multi-scale visual features each time. Aggregate three times to output three output features. Add the three output features to obtain the visual aggregation feature. Aggregation includes the following steps: The two adjacent features in the four-layer multi-scale visual features are the first input feature and the second input feature. The differential multi-scale feature aggregation module performs layer normalization, 1×1 convolution and 3×3 depth convolution on the first input feature and the second input feature respectively; the output of the first input feature is used as the query, and the output of the second input feature is used as the key and value. After inputting into the cross attention module of the differential multi-scale feature aggregation module, an output feature is obtained after a 1×1 convolution adjustment channel; The differential multi-scale feature fusion module provided in this embodiment uses a plug-and-play feature aggregation module to input the original features and the difference between the original features and the next layer of features, which is conducive to highlighting feature differences and better aggregating multi-scale features in high-resolution remote sensing images. Compared with cross-attention to each layer of features separately, this hierarchically aggregated feature fully considers the connection between low-level features and high-level features, which is conducive to the integration of local information and the understanding of global information.
[0034] The progressive cross-modal attention interaction module provided in this embodiment first uses visual features as queries and text features as keys and values to obtain language-guided visual features, and then uses text features as queries and language-guided visual features as keys and values to obtain visually enhanced text features. This progressive cross-modal attention interaction is different from the method of using only one modality as a query and the parallel structure of using two modalities to query simultaneously. It helps to use language-guided prior information during the second cross-attention to obtain features with better modal integration.
[0035] The visual aggregation features and text features are enhanced through the progressive cross-modal attention interaction module to obtain new visual features and new text features, specifically: The visual aggregation feature is used as the query, and the text feature is used as the key and value for cross attention calculation to obtain the interactive feature. The interactive feature is used to perform pixel-level weight calculation on the visual aggregation feature through the pixel-level weight calculation formula to obtain the weight of the visual feature. The visual aggregation feature is weighted based on the weight of the visual feature to obtain the visual feature map. The pixel-level weight calculation formula is as follows:
[0036] Among them, α and σ are learnable parameters, Represents the position coordinates in the visual feature map, is the visual aggregation feature, is the interactive feature, and The visual aggregation features and interaction features are mapped to the same space by linear layer and normalization layer respectively; The weighting is as follows:
[0037] in, is the visual feature map, is the weight of the visual feature, It is the visual aggregation feature; The text features are used as the query and the visual feature map is used as the key and value to perform cross-attention calculation to obtain visually enhanced text features; The visual feature map and the visually enhanced text features are used as visual features and text features to cycle through the above steps twice to obtain new visual features and new text features; The new visual features and new text features are decoded to obtain the target boundary coordinate prediction box, specifically: The new visual feature is input into the 4-layer Transformer decoder to obtain the first decoding feature, the new text feature is linearly mapped to obtain the second decoding feature, and the first decoding feature and the second decoding feature are input into the 4-layer Transformer decoder to obtain the target boundary coordinate prediction box.
[0038] After the multimodal visual language model is built, iterative optimization is performed, including loss calculation, gradient optimization, and round number determination. The loss calculation includes using the target boundary coordinate prediction box and the real target boundary coordinate value to calculate the loss through the loss calculation formula; The loss calculation formula is as follows:
[0039] in, is the loss target, λ is the weight coefficient, set to 1, For smooth loss, is the GIoU loss, g is the label of the data; Gradient optimization includes optimizing and updating the network weights of the multimodal visual language model using the AdamW optimizer and the Step learning rate strategy according to the loss. The learning rate for the first 60 training rounds is , the learning rate for the last 30 training rounds is ; The round number judgment includes judging whether the current training round number Epoch reaches the preset round number 90. If so, the model parameters are saved and the iterative optimization of the multimodal visual language model is ended. If not, the model verification of the multimodal visual language model is entered, and Epoch=Epoch+1 is set.
[0040] The model validation of the multimodal visual language model is as follows: Input the validation set of the remote sensing visual localization dataset into the multimodal visual language model; After normalizing the input remote sensing visual positioning data set, the visual features and visual masks output by the last layer of the feature extraction network and the fixed-length text features and text masks are extracted, and the visual-text cross-modal feature fusion is performed to obtain the fused features; the fused features are sent to the decoder to obtain the decoded features, and the decoded features are input into the target box regressor for regression to obtain four coordinate values; Select the result presentation method according to the task type. If it is a model reasoning task, draw a coordinate box on the remote sensing image of the input remote sensing visual positioning dataset according to the four coordinate values and save the image; if it is a model evaluation task, calculate the model evaluation index based on the regressed four coordinate values and the read real coordinates, and compare the current model evaluation index with the previous model's highest evaluation index. If it is greater than the model's highest evaluation index, update the weight to the parameters of this round of multimodal visual language model, otherwise enter the next round of iterative optimization of the multimodal visual language model.
[0041] Normalization is to normalize the input remote sensing image scale to less than or equal to 2000×2000 pixels; Model evaluation index calculations include: Pr@0.5, Pr@0.6, Pr@0.7, Pr@0.8, Pr@0.9, mIoU and cumIoU; IoU is the value of the intersection of the four regressed coordinate values and the read real coordinates divided by the union, I represents the intersection of the four regressed coordinate values and the read real coordinates, and U represents the union of the four regressed coordinate values and the read real coordinates; Pr@0.5 is calculated as the ratio of the number of samples with IoU>0.5 between the predicted value and the true label in the image-text pair to the number of all samples in the dataset; Pr@0.6 is calculated as the ratio of the number of samples with IoU>0.6 between the predicted value and the true label in the image-text pair to the number of all samples in the dataset; Pr@0.7 is calculated as the ratio of the number of samples with IoU>0.7 between the predicted value and the true label in the image-text pair to the number of all samples in the dataset; Pr@0.8 is calculated as the ratio of the number of samples with IoU>0.8 between the predicted value and the true label in the image-text pair to the number of all samples in the dataset; Pr@0.9 is calculated as the ratio of the number of samples with IoU>0.9 between the predicted value and the true label in the image-text pair to the number of all samples in the dataset; The mIoU metric is the mean of all target IoUs in each sample; The cumIoU metric is the ratio of the sum of I and U in all prediction results.
[0042] Embodiment 2 See also Figure 2 , a remote sensing visual positioning system based on multi-scale progressive reasoning, comprising: The data set acquisition module is used to acquire remote sensing visual positioning data sets; A model building module, used to build a multimodal visual language model, wherein the multimodal visual language model includes a differential multi-scale feature aggregation module and a progressive cross-modal attention interaction module; A target prediction module is obtained, which is used to input the remote sensing visual positioning dataset into the multimodal visual language model to obtain the target boundary coordinate prediction box.
[0043] Embodiment 3 See also Figure 3 An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, a remote sensing visual positioning method based on multi-scale progressive reasoning described in Example 1 is implemented.
[0044] Embodiment 4 A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, a remote sensing visual positioning method based on multi-scale progressive reasoning described in the first embodiment of the present invention is implemented.
[0045] Embodiment 5 The remote sensing visual positioning method in this embodiment refers to obtaining the bounding box of the corresponding target in the remote sensing image given a text prompt under the remote sensing high-resolution image and its matching text data. It is necessary to perform effective visual positioning under the conditions that the target scales in the high-resolution remote sensing image are different and the target significance is small, so as to avoid the problem of missed detection of the target.
[0046] This embodiment provides a remote sensing visual positioning method based on differential multi-scale progressive reasoning, and the flow chart is as follows: Figure 4 As shown, including: Step S1: Construct a dataset. Collect high-resolution visible light modality images from the Gaofen-1 satellite, and provide a text description of the target in each image. One image can have multiple targets and their corresponding text descriptions to generate a usable visual positioning dataset. Alternatively, use a public remote sensing visual positioning dataset. To ensure further optimization and learning of the subsequent model, the number of remote sensing image-text pairs annotated in the dataset should be no less than 10,000.
[0047] Remote sensing high-resolution visible light modal images refer to visible light remote sensing images with high spatial resolution obtained by imaging ground objects using remote sensing technology through high-resolution visible light sensors carried on remote sensing platforms (such as satellites, aircraft, etc.). High-resolution visible light remote sensing images can clearly show the detailed features of ground targets, such as the outline of buildings, the direction of roads, the distribution of vegetation, etc. This type of image is imaged in the visible light band (0.38~0.76 microns), so it can reproduce the real scene information that the human eye can see, which is more intuitive and clear. This technology mainly uses the visible spectrum of the human eye (wavelength 0.4~0.7 microns) for spatial remote sensing. In this band, the color sensitivity range of the photographic film just matches it, so black and white full-color or color images with high ground resolution and interpretation and mapping performance can be obtained. Since the visible light band is sensitive to the reflection and scattering characteristics of surface objects, high-resolution visible light remote sensing images can contain rich ground object information, such as the color, texture, shape, etc. of the ground objects.
[0048] Step S2: Model training: The obtained remote sensing visual positioning data set is passed through the multimodal visual language network designed in this embodiment to obtain a multimodal visual language model that can process high-resolution remote sensing images.
[0049] The multimodal visual language network is an information processing system that integrates multiple perceptual modalities (such as vision, language, etc.). Multimodality refers to the way of using two or more senses (such as vision, hearing, touch, smell, etc.) to interact with information at the same time. In the field of artificial intelligence, multimodal technology is the fusion of different types of data and information to achieve more accurate and efficient artificial intelligence applications. The technical principle of the multimodal visual language network is mainly based on algorithms such as deep learning. By training models to learn the common semantics between different modal data, cross-modal understanding and generation are achieved. Specifically, these networks usually include two parts: a visual encoder and a language encoder, which are used to process visual and language information respectively. The visual encoder converts images or videos into embeddings (lists of numbers), while the language encoder converts text into embeddings. The two encoders are trained on batches of image-text pairs, where the text describes the image. For each image-text pair, the image and text embeddings are "close" to each other, thereby achieving joint understanding and processing of visual and language information.
[0050] Furthermore, the model in step S2 is obtained by the following steps: Step S21: Divide the data set. If you collect data yourself, randomly sample the obtained visual positioning data set into a local training set and a validation set, where the training set and the validation set are divided in a ratio of 8:2. The training set is used for model optimization and convergence, and the validation set is used for optimal model selection. If a public data set is used, keep the data division of the public data set.
[0051] Step S22: Build a model. Use the training set to train the multimodal visual language model; further, the model in step S22 is obtained by the following steps: Step S221: Visual feature extraction. Use ResNet50 to extract visual features from the training set images to obtain four layers of multi-scale visual features.
[0052] Step S222: Text feature extraction. Use Bert-base to extract text features from the text corresponding to the image. ; Step S223: Multi-scale aggregation of visual features. Aggregate the four layers of visual features in step S221 Input differential multi-scale feature aggregation module, the module is as follows Figure 5 As shown, each time two adjacent features are input for feature aggregation, for example, the input feature and Features , respectively perform layer normalization, 1x1 convolution, and 3x3 depth convolution modules to transform the features The output of is used as Query, and the feature The output of is used as the Key and Value, input into the cross attention module, and then the channel output is adjusted through 1x1 convolution, the four features are aggregated three times, and the three output features are added to get the final aggregated feature .
[0053] Step S224: Cross-modal attention interaction.
[0054] The first step is to aggregate the visual features in step S222 and text features The progressive cross-modal attention interaction module designed together is input. Specifically, As a query, the text feature As Key and Value, perform cross attention calculation to obtain interactive features , and then perform pixel-level weight calculation, the calculation formula is:
[0055] in, and is a learnable parameter, Represents the position coordinates in the visual feature map, and The visual aggregation features and interaction features are mapped to the same space by linear layer and normalization layer respectively. After the above calculation, the weight result of the visual feature is obtained. , then based on For feature maps Weighted to obtain a language-guided visual feature map ,Right now:
[0056] The second step is to transform the text features As a query, the visual feature map As Key, Value, perform cross attention calculation to obtain visually enhanced text features .
[0057] The third step is to and As new visual features and text features, the first and second steps are repeated twice to obtain the final features. and .
[0058] Step S225: Feature decoding.
[0059] The first step is preprocessing. Send it to the 4-layer Transformer Encoder and get , the text features Linear mapping to ,in, , where c is the feature dimension, Make four copies .
[0060] The second step is to and Input into the 4-layer Transformer Decoder to get the output prediction box .
[0061] Step S3: training the model. The model is subjected to loss calculation and gradient optimization. Further, step S3 includes the following steps: Step S31: Calculate the loss. Compare the obtained b with the true value Perform loss calculation and set the smoothing based Loss and GIoU loss training objectives , the formula is as follows:
[0062] This objective function is to balance the high loss, adding the generalized IoU loss (GIoU) that is not affected by the scale, where is the weight coefficient, which is set to 1 here.
[0063] The loss function is used to quantify the difference between the model's predicted value and the actual value. The smaller the loss value, the more accurate the model's prediction and the better the performance. The loss value is used as an input to the optimization process to guide the adjustment direction of the model parameters.
[0064] Step S32: Gradient optimization. The network weights are optimized and updated according to the loss. The AdamW optimizer and Step learning rate strategy are used. The learning rate for the first 60 epochs is set to , the learning rate for the next 30 epochs is set to .
[0065] The gradient optimization algorithm uses the gradient of the loss function to the model parameters to update the parameters. The gradient indicates the direction in which the loss function value changes fastest with the parameters. By updating the parameters in the opposite direction of the gradient, the loss value can be gradually reduced. The goal of the gradient optimization algorithm is to find the parameter configuration that minimizes the loss function value, that is, the optimal solution. An efficient gradient optimization algorithm can speed up the model training process and reduce the training time. Appropriate optimization algorithms and parameter settings can also improve the generalization ability of the model and reduce the risk of overfitting.
[0066] Step S33: Determine whether the current training round number Epoch reaches the set round number 90. If so, save the model parameters and end the training. If not, enter the verification phase and set Epoch=Epoch+1.
[0067] Loss calculation and gradient optimization are usually performed alternately in an iterative process. In each iteration, the loss value under the current parameter configuration is first calculated, then the gradient of the loss function with respect to the parameters is calculated, and the parameters are updated according to the gradient. Through multiple iterations, the model parameters gradually converge to the configuration that minimizes the loss function value. When the loss value no longer decreases significantly, the model can be considered to have been trained.
[0068] In model training, "epoch" represents the process of completely traversing the entire dataset once. This process includes inputting all samples in the dataset into the neural network and completing one forward propagation and one backpropagation. Simply put, an epoch is the process of training all training samples in the neural network once. In practical applications, the entire dataset is divided into several batches, each of which contains a part of the training samples. Each time a batch of data is fed into the network for training, it is an iteration. In each iteration, the model predicts the data based on the current parameters, then calculates the error between the predicted result and the true value, and backpropagates the error to update the model's parameters. Repeat this process until all batches have been traversed once, that is, one epoch of training is completed. During the model training process, multiple epochs need to be set so that the model can gradually optimize its parameters by traversing the dataset multiple times.
[0069] Step S4: Validate the model. Use the validation set to evaluate the generated multimodal visual language model; further, step S4 includes the following steps: Step S41: Input one or more sets of remote sensing images and text instruction pairs in visible light mode. Normalize the input image. If the scale of the input remote sensing image is greater than 2000×2000, and the scale unit is the number of pixels, that is, the width of the image is 2000 pixels and the height of the image is also 2000 pixels, then scale scaling is performed without other data enhancement.
[0070] Step S42: Extract visual and text features from the processed image-text pair, take the visual features and fixed-length text features output by the last layer of the feature extraction network, and the corresponding visual mask and text mask, and perform visual-text cross-modal feature fusion. Send the fused features to the decoder to obtain decoded features, and finally input them into the target box regressor to regress four coordinate values.
[0071] The target box regressor usually uses a four-dimensional vector (x, y, w, h) to represent the window (i.e., candidate box or target box), where x and y represent the coordinates of the center point of the window, and w and h represent the width and height of the window. The goal of the regressor is to find a relationship so that the input original candidate box P is mapped to a regression window G' that is closer to the true target box G. During the training phase, the regressor receives a large number of samples, each of which includes a feature vector of a candidate box and the corresponding true target box. By minimizing the loss function (such as IoU loss, GIoU loss, etc.), the regressor learns how to predict the required translation and scaling from the feature vector of the candidate box in order to adjust the candidate box to a position closer to the true target box. During the testing phase, the regressor receives a new candidate box feature vector and applies the learned mapping relationship to predict the adjustment amount to obtain a more accurate target box position.
[0072] Step S43: Select the result presentation method according to the task type. If it is a model reasoning task, draw a coordinate box on the input remote sensing image according to the obtained coordinate values and save the image to the specified directory.
[0073] If it is a model evaluation task, the indicator is calculated based on the coordinate value obtained by regression and the real coordinate value read. The intersection of the real value and the predicted value is IoU. The higher the IoU value, the better the alignment between the predicted area and the actual area, reflecting the more accurate the model. IoU is a common indicator for visual positioning tasks. I represents the intersection of the predicted area and the real area, and U represents the union of the two. IoU is the value of the intersection divided by the union. Evaluation indicator calculation includes: Model evaluation index calculations include: Pr@0.5, Pr@0.6, Pr@0.7, Pr@0.8, Pr@0.9, mIoU and cumIoU; IoU is the value of the intersection of the four regressed coordinate values and the read real coordinates divided by the union, I represents the intersection of the four regressed coordinate values and the read real coordinates, and U represents the union of the four regressed coordinate values and the read real coordinates; Pr@0.5 is calculated as the ratio of the number of samples with IoU>0.5 between the predicted value and the true label in the image-text pair to the number of all samples in the dataset; Pr@0.6 is calculated as the ratio of the number of samples with IoU>0.6 between the predicted value and the true label in the image-text pair to the number of all samples in the dataset; Pr@0.7 is calculated as the ratio of the number of samples with IoU>0.7 between the predicted value and the true label in the image-text pair to the number of all samples in the dataset; Pr@0.8 is calculated as the ratio of the number of samples with IoU>0.8 between the predicted value and the true label in the image-text pair to the number of all samples in the dataset; Pr@0.9 is calculated as the ratio of the number of samples with IoU>0.9 between the predicted value and the true label in the image-text pair to the number of all samples in the dataset; cumIoU: This indicator is for the entire data set. It sums the I in the prediction results of all pictures to get I', sums the U to get U', and then divides I' by U'; mIoU: the mean of all target IoUs in each sample; Compare the current round model evaluation index with the previous highest evaluation index. If it is greater than the highest index, update the weight to the model parameter of this round, otherwise enter the next round of training.
[0074] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0075] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0076] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0077] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention is described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation of the present invention can still be modified or replaced by equivalents, and any modification or equivalent replacement that does not deviate from the spirit and scope of the present invention should be included in the protection scope of the claims of the present invention.
Claims
1. A remote sensing visual positioning method based on multi-scale progressive reasoning, characterized in that: The following steps are involved: Obtain remote sensing visual positioning dataset; Constructing a multimodal visual language model, wherein the multimodal visual language model includes a differential multi-scale feature aggregation module and a progressive cross-modal attention interaction module; The remote sensing visual positioning dataset is input into the multimodal visual language model to obtain the target boundary coordinate prediction box.
2. The remote sensing visual positioning method based on multi-scale progressive reasoning according to claim 1 is characterized in that: The number of remote sensing image and text instruction pairs annotated in the remote sensing visual positioning dataset is no less than 10,000; The remote sensing image is a remote sensing high-resolution visible light modality image; The text instruction is a text description of the target in the remote sensing high-resolution visible light modality image.
3. The remote sensing visual positioning method based on multi-scale progressive reasoning according to claim 2 is characterized in that: The remote sensing visual positioning data set is input into the multimodal visual language model to obtain the target boundary coordinate prediction box, specifically: The remote sensing visual positioning dataset is divided into a training set and a validation set; the training set and the validation set are divided in a ratio of 8:2; Use the training set to extract four layers of multi-scale visual features and text features; The four-layer multi-scale visual features are multi-scale aggregated through the differential multi-scale feature aggregation module to obtain visual aggregate features; The visual aggregation features and text features are enhanced through the progressive cross-modal attention interaction module to obtain new visual features and new text features; The new visual features and new text features are decoded to obtain the target boundary coordinate prediction box.
4. The remote sensing visual positioning method based on multi-scale progressive reasoning according to claim 3 is characterized in that: The method of using the training set to extract four layers of multi-scale visual features and text features is specifically as follows: ResNet50 is used to extract visual features from the images in the training set to obtain four-layer multi-scale visual features. Bert-base is used to extract text features from the instruction texts in the images in the training set to obtain text features. The four-layer multi-scale visual features are multi-scale aggregated through a differential multi-scale feature aggregation module to obtain visual aggregate features, specifically: Input the four-layer multi-scale visual features into the differential multi-scale feature aggregation module for aggregation. Input two adjacent features in the four-layer multi-scale visual features each time. Aggregate three times to output three output features. Add the three output features to obtain the visual aggregation feature. Aggregation includes the following steps: The two adjacent features in the four-layer multi-scale visual features are the first input feature and the second input feature. The differential multi-scale feature aggregation module performs layer normalization, 1×1 convolution and 3×3 depth convolution on the first input feature and the second input feature respectively; the output of the first input feature is used as the query, and the output of the second input feature is used as the key and value. After inputting into the cross attention module of the differential multi-scale feature aggregation module, an output feature is obtained after a 1×1 convolution adjustment channel; The visual aggregation features and text features are enhanced through a progressive cross-modal attention interaction module to obtain new visual features and new text features, specifically: The visual aggregation feature is used as the query, and the text feature is used as the key and value for cross attention calculation to obtain the interactive feature. The interactive feature is used to perform pixel-level weight calculation on the visual aggregation feature through the pixel-level weight calculation formula to obtain the weight of the visual feature. The visual aggregation feature is weighted based on the weight of the visual feature to obtain the visual feature map. The pixel-level weight calculation formula is as follows: Among them, α and σ are learnable parameters, Represents the position coordinates in the visual feature map, is the visual aggregation feature, is the interactive feature, and The linear layer and the normalization layer are used to map the visual aggregation features and the interaction features to the features of the same space respectively; The weighting is as follows: in, is the visual feature map, is the weight of the visual feature, It is the visual aggregation feature; The text features are used as the query and the visual feature map is used as the key and value to perform cross-attention calculation to obtain visually enhanced text features; The visual feature map and the visually enhanced text features are used as visual features and text features to cycle through the above steps twice to obtain new visual features and new text features; The new visual features and the new text features are decoded to obtain the target boundary coordinate prediction box, specifically: The new visual feature is input into the 4-layer Transformer decoder to obtain the first decoding feature, the new text feature is linearly mapped to obtain the second decoding feature, and the first decoding feature and the second decoding feature are input into the 4-layer Transformer decoder to obtain the target boundary coordinate prediction box.
5. The remote sensing visual positioning method based on multi-scale progressive reasoning according to claim 4 is characterized in that: The multimodal visual language model is iteratively optimized after construction, and the iterative optimization includes loss calculation, gradient optimization and round number judgment; The loss calculation includes using the target boundary coordinate prediction box and the real target boundary coordinate value to calculate the loss through a loss calculation formula; The loss calculation formula is as follows: in, is the loss target, λ is the weight coefficient, set to 1, For smooth loss, is the GIoU loss, g is the label of the data; The gradient optimization includes optimizing and updating the network weights of the multimodal visual language model using the AdamW optimizer and the Step learning rate strategy according to the loss. The learning rate for the first 60 training rounds is , the learning rate for the last 30 training rounds is ; The round number judgment includes judging whether the current training round number Epoch reaches the preset round number 90. If so, the model parameters are saved and the iterative optimization of the multimodal visual language model is ended. If not, the model verification of the multimodal visual language model is entered, and Epoch=Epoch+1 is set.
6. The remote sensing visual positioning method based on multi-scale progressive reasoning according to claim 5 is characterized in that: The model verification of the multimodal visual language model is specifically as follows: Input the validation set of the remote sensing visual localization dataset into the multimodal visual language model; After normalizing the input remote sensing visual positioning data set, the visual features and visual masks output by the last layer of the feature extraction network and the fixed-length text features and text masks are extracted, and the visual-text cross-modal feature fusion is performed to obtain the fused features; the fused features are sent to the decoder to obtain the decoded features, and the decoded features are input into the target box regressor for regression to obtain four coordinate values; Select the result presentation method according to the task type. If it is a model reasoning task, draw a coordinate box on the remote sensing image of the input remote sensing visual positioning dataset according to the four coordinate values and save the image; if it is a model evaluation task, calculate the model evaluation index based on the regressed four coordinate values and the read real coordinates, and compare the current model evaluation index with the previous model's highest evaluation index. If it is greater than the model's highest evaluation index, update the weight to the parameters of this round of multimodal visual language model, otherwise enter the next round of iterative optimization of the multimodal visual language model.
7. The remote sensing visual positioning method based on multi-scale progressive reasoning according to claim 6 is characterized in that: The normalization is to normalize the scale of the input remote sensing image to be less than or equal to 2000×2000 pixels; The model evaluation index calculation includes: Pr@0.5, Pr@0.6, Pr@0.7, Pr@0.8, Pr@0.9, mIoU and cumIoU; The IoU is the value of the intersection of the four regressed coordinate values and the read real coordinates divided by the union, I represents the intersection of the four regressed coordinate values and the read real coordinates, and U represents the union of the four regressed coordinate values and the read real coordinates; The calculation method of Pr@0.5 is the ratio of the number of samples with IoU>0.5 between the predicted value and the true label in the image-text pair to the number of all samples in the dataset; The calculation method of Pr@0.6 is the ratio of the number of samples with IoU>0.6 between the predicted value and the true label in the image-text pair to the number of all samples in the dataset; The calculation method of Pr@0.7 is the ratio of the number of samples with IoU>0.7 between the predicted value and the true label in the image-text pair to the number of all samples in the dataset; The calculation method of Pr@0.8 is the ratio of the number of samples with IoU>0.8 between the predicted value and the true label in the image-text pair to the number of all samples in the dataset; The calculation method of Pr@0.9 is the ratio of the number of samples with IoU>0.9 between the predicted value and the true label in the image-text pair to the number of all samples in the dataset; The mIoU is the mean of all target IoUs in each sample; The cumIoU is the ratio of the sum of I to the sum of U in all prediction results.
8. A remote sensing visual positioning system based on multi-scale progressive reasoning, characterized in that: include: The data set acquisition module is used to acquire remote sensing visual positioning data sets; A model building module, used to build a multimodal visual language model, wherein the multimodal visual language model includes a differential multi-scale feature aggregation module and a progressive cross-modal attention interaction module; A target prediction module is obtained, which is used to input the remote sensing visual positioning dataset into the multimodal visual language model to obtain the target boundary coordinate prediction box.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a remote sensing visual positioning method based on multi-scale progressive reasoning as described in any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements a remote sensing visual positioning method based on multi-scale progressive reasoning as described in any one of claims 1 to 7.
Citation Information
Cited By
Free text guided remote sensing image anaphora segmentation method and system
CN120340034A
Video understanding description generation method, device and equipment based on remote sensing time sequence data
CN121214055A
Cross-modal olfaction-vision-language-action decision-making method and system
CN122100179A
