Text-guided cross-scale collaborative gating remote sensing target detection method

Abstract

The invention relates to the technical field of remote sensing images, in particular to a text-guided cross-scale collaborative gating remote sensing target detection method, which comprises the following steps of: generating fine-grained text description corresponding to remote sensing image contents; respectively extracting image multi-scale features and text hierarchical semantic features; constructing a language guide query vector based on the text hierarchical semantic features; performing dynamic fusion on the image multi-scale features, the text hierarchical semantic features and the language guide query vectors to obtain visual memory features and text enhancement features after cross-modal fusion; and based on the language guide query vector, the visual memory features and the text enhancement features, iteratively optimizing a prediction bounding box through a decoder, and outputting a final remote sensing target detection result. The method aims at overcoming the limitations of the existing remote sensing image visual detection technology in the aspects of semantic description coarseness, cross-modal feature alignment noise sensitivity, weak spatial relationship modeling and the like.

Description

Technical Field

[0001] The embodiments of this application relate to the field of remote sensing image technology, and in particular to a text-guided cross-scale collaborative gating remote sensing target detection method. Background Technology

[0002] With the rapid development of high-resolution remote sensing satellites and UAVs, remote sensing images are playing an increasingly important role in key areas such as military reconnaissance, urban planning, disaster monitoring, and resource exploration. The visual grounding task of remote sensing images aims to accurately locate and identify corresponding target objects in images based on given natural language descriptions. It is a core technology for achieving accurate "language-to-pixel" mapping and is of great significance for improving the intelligent level of automatic interpretation of remote sensing information. Currently, cutting-edge research in this field mostly adopts visual-language models based on the Transformer architecture and draws on the visual grounding framework in the field of natural imagery. Through two-stage (generating candidate boxes and then filtering) or single-stage (end-to-end prediction) methods, it attempts to achieve cross-modal alignment and target localization of text and vision in remote sensing scenarios.

[0003] However, existing technologies still suffer from several inherent bottlenecks when adapting to complex and ever-changing open remote sensing scenarios, making it difficult to meet the requirements of high-precision decision-making in terms of detection accuracy, robustness, and practicality. At the data level, the problem of coarse-grained semantic description is prominent. Even with VLM fine-tuning, the text remains limited to target categories or simple phrases, failing to capture target attributes, background elements, and scene relationships, resulting in insufficient semantic understanding depth. At the model level, cross-modal fusion mechanisms are imperfect. Early stitching, late classification layer integration, and Transformer attention fusion are all prone to imbalance, either obscuring visual characteristics or losing key information. Furthermore, they are sensitive to noise such as image blurring and target occlusion, making it difficult to bridge the semantic gap. At the spatial level, single / two-stage methods have not broken through traditional detection logic, relying on bounding box geometric coordinates to determine location, and cannot model the response orientation requirement of "the target is located in the upper right corner of the airport." In summary, existing remote sensing visual grounding and target detection technologies have significant shortcomings in terms of the richness of semantic description, the anti-interference capability of cross-modal fusion, the ability to model spatial relationships, and the degree of automation. Summary of the Invention

[0004] In view of this, embodiments of this application propose a text-guided cross-scale collaborative gating remote sensing target detection method, which aims to overcome the limitations of existing remote sensing image visual detection technologies in terms of coarse-grained semantic description, sensitivity to noise in cross-modal feature alignment, and weak spatial relationship modeling. By constructing a data-model-space collaborative optimization framework, it achieves high-precision and robust remote sensing target detection.

[0005] To achieve the above objectives, embodiments of this application propose a text-guided cross-scale collaborative gating remote sensing target detection method, the method comprising the following steps: The system acquires the remote sensing image to be detected and generates a fine-grained text description corresponding to the content of the remote sensing image; the fine-grained text description includes spatial orientation information. Based on remote sensing images and fine-grained text descriptions, a dual-stream coding architecture is used to extract multi-scale features of the images and hierarchical semantic features of the text, respectively. Based on the hierarchical semantic features of text, a language-guided query vector is constructed; the language-guided query vector has spatial awareness and semantic alignment attributes. By using a cross-scale gated bidirectional attention interaction module, multi-scale image features, hierarchical semantic features of text, and language-guided query vectors are dynamically fused to obtain cross-modal fusion visual memory features and text enhancement features. Based on language-guided query vectors, visual memory features, and text-enhanced features, the predicted bounding boxes are iteratively optimized through a decoder to output the final remote sensing target detection results.

[0006] To achieve the above objectives, embodiments of this application also propose a text-guided cross-scale collaborative gated remote sensing target detection system, the system comprising: The fine-grained semantic generation module is used to acquire the remote sensing image to be detected and generate a fine-grained text description corresponding to the content of the remote sensing image; wherein, the fine-grained text description includes spatial orientation information; The dual-stream encoder module is used to extract multi-scale features of the image and hierarchical semantic features of the text based on remote sensing images and fine-grained text descriptions through a dual-stream coding architecture. The language-guided query construction module is used to construct language-guided query vectors based on text hierarchical semantic features; the language-guided query vectors have spatial awareness and semantic alignment attributes. The cross-scale gated bidirectional attention fusion module is used to dynamically fuse multi-scale features of images, hierarchical semantic features of text, and language-guided query vectors through the cross-scale gated bidirectional attention interaction module to obtain cross-modal fused visual memory features and text enhancement features. The iterative decoding and localization output module is used to iteratively optimize the predicted bounding boxes based on language-guided query vectors, visual memory features, and text enhancement features, and output the final remote sensing target detection results.

[0007] To achieve the above objectives, embodiments of this application also propose an electronic device, including a processor and a memory, wherein the memory stores instructions executable by the processor, and the processor is configured to execute the instructions such that the electronic device can implement a text-guided cross-scale collaborative gating remote sensing target detection method as described above.

[0008] To achieve the above objectives, embodiments of this application also propose a computer-readable storage medium storing a computer program that, when executed by a processor, enables a text-guided cross-scale collaborative gating remote sensing target detection method as described above.

[0009] This application proposes a text-guided cross-scale collaborative gating remote sensing target detection method. First, it acquires the remote sensing image to be detected and generates a fine-grained text description corresponding to the image content, containing spatial orientation information. Then, based on the remote sensing image and the fine-grained text description, it extracts multi-scale features of the image and hierarchical semantic features of the text using a dual-stream coding architecture. Next, it constructs a language-guided query vector based on the hierarchical semantic features of the text, which possesses spatial awareness and semantic alignment attributes. Then, through a cross-scale gating bidirectional attention interaction module, it dynamically fuses the multi-scale features of the image, the hierarchical semantic features of the text, and the language-guided query vector to obtain cross-modal fusion-based visual memory features and text enhancement features. Finally, based on the language-guided query vector, visual memory features, and text enhancement features, iteratively optimizes the predicted bounding box through a decoder to output the final remote sensing target detection result. This scheme aims to overcome the limitations of existing remote sensing image visual detection technologies in terms of coarse-grained semantic descriptions, sensitivity to noise in cross-modal feature alignment, and weak spatial relationship modeling. By constructing a data-model-space collaborative optimization framework, it achieves high-precision and robust remote sensing target detection.

[0010] Optionally, the remote sensing image to be detected is acquired, and a fine-grained text description containing spatial orientation information corresponding to the content of the remote sensing image is generated. This includes: using a large language vision model to parse the content of the remote sensing image and automatically generating a structured long text description containing target attributes and scene relationships; automatically filtering the structured long text description to remove abstract concepts and non-visualizable words to obtain a semantically pure text description; calculating the overlap rate between the target bounding box and the preset image grid in the remote sensing image, and determining structured position markers representing the primary and secondary absolute positions based on the maximum and second-largest overlap rates; and concatenating the structured position markers representing the primary and secondary absolute positions with the semantically pure text description to obtain a fine-grained text description containing spatial orientation information.

[0011] Optionally, the dual-stream coding architecture includes a visual encoder and a text encoder; the extraction of multi-scale image features and hierarchical semantic features of text based on remote sensing images and fine-grained text descriptions through the dual-stream coding architecture includes: Multi-scale features of the image are extracted using a visual encoder. ;in, For batch size, The feature map space size, Number of visual feature channels; Text hierarchical semantic features are extracted using a text encoder. ;in, The length of the text word. For semantic dimension.

[0012] Optionally, based on the hierarchical semantic features of the text, a language-guided query vector is constructed, including: obtaining an initial query and reference point through a learnable embedding layer; normalizing the reference point and generating a corresponding sinusoidal position code to enable the query to have spatial awareness; and using a text cross-attention mechanism to encode the query combined with the position and then interacting with the hierarchical semantic features of the text to obtain the language-guided query vector.

[0013] Optionally, the initial query is , The reference point is , ; For the number of queries, For model feature dimensions; The text cross-attention mechanism is used to encode the query combination position, which is then interacted with the hierarchical semantic features of the text to obtain a language-guided query vector. The formula is expressed as follows: ; in, For sinusoidal position encoding, Text features after incorporating visual information. This is a text feature mask.

[0014] Optionally, a cross-scale gating bidirectional attention interaction module is used to dynamically fuse multi-scale image features, hierarchical semantic features of text, and language-guided query vectors to obtain cross-modal fusion-derived visual memory features and text enhancement features. This includes: establishing a bidirectional attention framework to calculate visually guided text-focused attention scores and language-guided visual enhancement attention scores respectively; using an attention gating mechanism, dynamically generating language gating values ​​and language interaction strengths based on global features of visual queries and language keys, and simultaneously dynamically generating visual gating values ​​based on global features of language queries and visual keys to adjust the interaction strength; introducing a hierarchical spatial enhancement strategy for different levels of multi-scale image features, achieving cross-scale bidirectional interaction between high-level semantics and low-level details through upsampling and downsampling operations, and updating attention maps at each level; and introducing residual gating fusion to dynamically balance the new features generated by cross-modal fusion with the original features, thereby obtaining visual memory features and text enhancement features.

[0015] Optionally, residual gating fusion is introduced to dynamically balance the new features generated by cross-modal fusion with the original features, thereby obtaining visual memory features and text enhancement features. Text Enhancement Features The formula is expressed as follows: ; ; in, Original visual features Features of the original text The mask corresponding to the original visual features. This is the mask corresponding to the original text features. and is the residual gating coefficient.

[0016] Optionally, based on the language-guided query vector, visual memory features, and text-enhanced features, the predicted bounding box is iteratively optimized through a decoder to output the final remote sensing target detection result. This includes: inputting the language-guided query vector, visual memory features, and text-enhanced features into the decoder for layer-by-layer iterative optimization; wherein each layer iteration includes: refining the query features through query self-attention; deeply binding the refined query with the text-enhanced features through text cross-attention; interacting the query bound to text semantics with the visual memory features through visual deformable cross-attention; adaptively sampling target region features; performing nonlinear optimization of the fused features through a feedforward network; and dynamically updating the bounding box coordinates of the reference points based on the optimized query features; after completing the layer iteration, based on the final query features, the target category probability and the final bounding box coordinates are output through the classification head and bounding box head. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies of this application will be briefly introduced below. Obviously, the following drawings are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. The drawings described herein are only used to explain this application and are not intended to limit this application.

[0018] Figure 1 This is a flowchart of a text-guided cross-scale collaborative gating remote sensing target detection method provided in one embodiment of this application; Figure 2 This is a schematic diagram of an automated text prompt generation process provided in one embodiment of this application; Figure 3 This is a schematic diagram of a location guidance and prompting mechanism provided in one embodiment of this application; Figure 4 This is a schematic diagram of an overall algorithm structure provided in one embodiment of this application; Figure 5 This is a schematic diagram of a cross-scale bidirectional gating attention mechanism provided in one embodiment of this application; Figure 6 This is a schematic diagram of the structure of a text-guided cross-scale collaborative gating remote sensing target detection system provided in another embodiment of this application; Figure 7 This is a schematic diagram of the structure of an electronic device provided in another embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. Those skilled in the art will understand that many technical details have been presented in the embodiments of this application to facilitate better understanding. However, the technical solutions claimed in this application can be implemented even without these technical details and various variations and modifications based on the following embodiments. The division of the following embodiments is for ease of description and should not constitute any limitation on the specific implementation of this application. The following embodiments can be combined with and referenced by each other without contradiction.

[0020] With the rapid development of high-resolution remote sensing satellites and UAVs, remote sensing images are playing an increasingly important role in key areas such as military reconnaissance, urban planning, disaster monitoring, and resource exploration. The visual grounding task of remote sensing images aims to accurately locate and identify corresponding target objects in images based on given natural language descriptions. It is a core technology for achieving accurate "language-to-pixel" mapping and is of great significance for improving the intelligent level of automatic interpretation of remote sensing information. Currently, cutting-edge research in this field mostly adopts visual-language models based on the Transformer architecture and draws on the visual grounding framework in the field of natural imagery. Through two-stage (generating candidate boxes and then filtering) or single-stage (end-to-end prediction) methods, it attempts to achieve cross-modal alignment and target localization of text and vision in remote sensing scenarios.

[0021] However, existing technologies still suffer from several inherent bottlenecks when adapting to complex and ever-changing open remote sensing scenarios, making it difficult to meet the requirements of high-precision decision-making in terms of detection accuracy, robustness, and practicality. At the data level, the problem of coarse-grained semantic description is prominent. Even with VLM fine-tuning, the text remains limited to target categories or simple phrases, failing to capture target attributes, background elements, and scene relationships, resulting in insufficient semantic understanding depth. At the model level, cross-modal fusion mechanisms are imperfect. Early stitching, late classification layer integration, and Transformer attention fusion are all prone to imbalance, either obscuring visual characteristics or losing key information. Furthermore, they are sensitive to noise such as image blurring and target occlusion, making it difficult to bridge the semantic gap. At the spatial level, single / two-stage methods have not broken through traditional detection logic, relying on bounding box geometric coordinates to determine location, and cannot model the response orientation requirement of "the target is located in the upper right corner of the airport." In summary, existing remote sensing visual grounding and target detection technologies have significant shortcomings in terms of the richness of semantic description, the anti-interference capability of cross-modal fusion, the ability to model spatial relationships, and the degree of automation.

[0022] In view of this, embodiments of this application propose a text-guided cross-scale collaborative gating remote sensing target detection method, which aims to overcome the limitations of existing remote sensing image visual detection technologies in terms of coarse-grained semantic description, sensitivity to noise in cross-modal feature alignment, and weak spatial relationship modeling. By constructing a data-model-space collaborative optimization framework, it achieves high-precision and robust remote sensing target detection.

[0023] One embodiment of this application proposes a text-guided cross-scale collaborative gating remote sensing target detection method applied to an electronic device, wherein the electronic device can be a terminal or a server. This embodiment and the following embodiments will use a server as an example for description. The implementation details of the text-guided cross-scale collaborative gating remote sensing target detection method proposed in this embodiment will be described in detail below. The following implementation details are provided for ease of understanding and are not necessary for implementing this solution.

[0024] The specific process of the text-guided cross-scale collaborative gating remote sensing target detection method proposed in this embodiment can be as follows: Figure 1 As shown, it includes: Step 101: Obtain the remote sensing image to be detected and generate a fine-grained text description corresponding to the content of the remote sensing image.

[0025] Among them, fine-grained text descriptions include spatial orientation information.

[0026] In one possible embodiment, step 101 includes: using a large language vision model to perform content parsing on the remote sensing image, automatically generating a structured long text description containing target attributes and scene relationships; automatically filtering the structured long text description to remove abstract concepts and non-visualizable words, obtaining a semantically pure text description; calculating the overlap rate between the target bounding box and a preset image grid in the remote sensing image, and determining the target structured location marker based on the maximum and second-largest overlap rates; and concatenating the target structured location marker with the semantically pure text description to obtain a fine-grained text description containing spatial orientation information.

[0027] Among them, the target structured location marker is used to represent the primary absolute location and the secondary absolute location.

[0028] For example, the pre-trained large language-visual model Qwen-VL-Plus can be used to guide the model to deeply understand image content by designing prompt words, automatically generating structured long text descriptions covering target attributes, background elements, and scene relationships without relying on manual annotation. At the same time, through a three-level automated filtering mechanism, abstract concepts are removed, core parts of speech are retained, and non-visualizable words are filtered out in turn, improving semantic purity and visual consistency, ensuring that the generated descriptions accurately correspond to the image content, thereby solving the problems of brief text descriptions and missing semantic information in traditional methods, and providing a rich semantic foundation for subsequent cross-modal alignment.

[0029] For example, such as Figure 2 As shown, 800×800 resolution original images were obtained from DIOR-RSVG remote sensing image data. The Qwen-VL-Plus visual language model was used to perform deep semantic parsing of the images, automatically generating structured long-text descriptions containing target attributes, background elements, and scene relationships, covering multi-dimensional semantic details. This approach, through the model's global understanding of image content, avoids reliance on manual annotation, significantly improving the completeness and scene adaptability of semantic descriptions, and providing a high-quality semantic input foundation for subsequent cross-modal alignment.

[0030] For the generated text description, a three-level automated filtering mechanism is implemented: First, abstract concepts are eliminated through regular expression matching; second, the part-of-speech tagging tool spaCy is used to segment the text and tag its parts of speech, retaining nouns, adjectives, and quantifiers; finally, non-visualizable words are filtered based on a remote sensing professional dictionary. This measure eliminates semantic discrepancies between text and image content, ensuring that descriptions contain only visually perceptible semantics and avoiding noise interference in the cross-modal alignment process, thereby establishing semantic purity and visual consistency and providing precise semantic constraints for feature fusion.

[0031] To further enhance the spatial orientation of pure semantic text, a location guidance and prompting mechanism is constructed to supplement the semantic description with a spatial constraint dimension, thereby marking the absolute position of the target in the image. The specific process is as follows: like Figure 3 As shown, the original input image size is The image is divided into 9 equal sub-regions, each with a pixel range of 3×3 standard grid. ; The target bounding box is: ,calculate With the Pixel overlap rate of each grid ; This represents the pixel range and area corresponding to the bounding box of a grid or target. Represents the grid points used to divide the image; take The grid position corresponding to the maximum value is taken as the main absolute position of the target (e.g., top left). The second largest value is Generate structured location markers: ; The final enhanced tooltip text is generated as follows: .

[0032] Understandably, this operation enables the mathematical encoding of sub-region-level spatial constraints, replacing the traditional center point judgment. This allows the text description to contain both semantic details and spatial orientation information, enhancing the subsequent text encoder's spatial understanding of the orientation description and providing both semantic and spatial constraints for cross-modal feature alignment.

[0033] Step 102: Based on remote sensing images and fine-grained text descriptions, multi-scale features of the images and hierarchical semantic features of the text are extracted using a dual-stream coding architecture.

[0034] For example, a dual-stream coding architecture is adopted, using ResNet50 to extract multi-scale pyramid features of images and BERT to extract hierarchical semantic representations of text. The visual features retain sufficient spatial details and cover multi-scale abstract information, while the text representations can achieve adaptive matching with the visual features at the level of abstraction. This provides the optimal level of abstraction during the feature extraction process, avoids the modal feature overload caused by early fusion and the information loss caused by late fusion, and lays a solid foundation for cross-modal interaction.

[0035] For example, such as Figure 4 As shown, Figure 4 A dual-stream coding architecture is adopted, which extracts multi-scale pyramid features of images and hierarchical semantic representations of text through ResNet50 visual encoder and BERT text encoder respectively. The core is to provide the optimal abstraction level in the feature extraction process, so as to avoid the overwhelming of modal characteristics caused by early fusion or the loss of information in late fusion.

[0036] In one possible embodiment, step 102 includes: a dual-stream coding architecture comprising a visual encoder and a text encoder; the step of extracting multi-scale image features and hierarchical semantic features of text based on remote sensing images and fine-grained text descriptions using the dual-stream coding architecture includes: extracting multi-scale image features using the visual encoder. The hierarchical semantic features of the text are extracted using a text encoder. .

[0037] in, For batch size, The feature map space size, Number of visual feature channels; The length of the text word. For semantic dimension.

[0038] For example, text hierarchical semantic features can be used to obtain word-level, phrase-level, and sentence-level hierarchical semantic representations. The two types of features achieve adaptive matching at the abstract level, laying the foundation for subsequent cross-modal interaction.

[0039] Step 103: Construct a language-guided query vector based on the hierarchical semantic features of the text.

[0040] Among them, language-guided query vectors have spatial awareness and semantic alignment attributes.

[0041] For example, based on dual-stream features, queries and reference points are initialized through a learnable embedding layer to generate sinusoidal position codes to give queries spatial awareness. Then, text cross-attention is used to achieve accurate alignment between queries and text semantics, constructing a query vector with both spatial awareness and semantic alignment attributes. This realizes the transformation from text semantics to query space, breaking the limitation of traditional queries relying solely on visual features and providing proactive query-driven cross-modal fusion.

[0042] In one possible embodiment, step 103 includes: obtaining an initial query and reference point through a learnable embedding layer; normalizing the reference point and generating a corresponding sinusoidal position code to enable the query to have spatial awareness; and using a text cross-attention mechanism to encode the query combined with the position and interact with the text hierarchical semantic features to obtain a language-guided query vector.

[0043] For example, based on textual and visual features, a language-guided query vector and initial reference points are constructed to realize the transformation of textual semantics into the query space. First, a learnable embedding layer is used to initialize the query and reference points.

[0044] In one possible implementation, the initial query is: , The reference point is , ; For the number of queries, This represents the feature dimension of the model.

[0045] For example, It can be 300. This refers to the unnormalized initial reference point, such as the bounding box coordinates.

[0046] Generate sinusoidal position codes based on reference points to enable spatial awareness in queries: ; ; ; in, As the initial reference point, with a dimension of 4, the corresponding bounding box pixel coordinates are [x1, y1, x2, y2]. , representing the effective area proportion at each scale. For visual feature scales, To correct the spatial bias caused by padding during element-wise multiplication in broadcasting, sine is a sinusoidal positional coding mapping function. The obtained sinusoidal positional encoding, MLP(·) is a 2-layer fully connected network, maps the sinusoidal encoding to the feature dimension, and obtains a positional encoding consistent with the feature dimension of the query space. .

[0047] The text cross-attention mechanism is used to encode the query combination position, which is then interacted with the hierarchical semantic features of the text to obtain a language-guided query vector. The formula is expressed as follows: ; in, For sinusoidal position encoding, Text features after incorporating visual information. This is a text feature mask.

[0048] Understandably, by constructing a dual-attribute query vector that combines spatial awareness and semantic alignment, the limitation of traditional queries relying solely on visual features is broken. This provides an active query driver for cross-modal fusion in subsequent embodiments, enabling the fusion process to focus on the visual regions corresponding to the text semantics and significantly improving the accuracy of cross-modal alignment.

[0049] Step 104: Through the cross-scale gating bidirectional attention interaction module, the multi-scale features of the image, the hierarchical semantic features of the text, and the language-guided query vector are dynamically fused to obtain the cross-modal fused visual memory features and text enhancement features.

[0050] For example, a cross-scale gating bidirectional attention interaction module is embedded to dynamically calculate the three-dimensional interaction weights of visual regions, text terms, and query vectors. The multi-scale attention weights are optimized by combining a hierarchical spatial enhancement strategy. The interaction intensity and feature fusion ratio are adjusted through a two-layer gating mechanism. At the same time, parallel dual-path independent query projection is designed to avoid the shortcomings of insufficient fine-grained control of standard bidirectional attention. Independent residual connections and masking are introduced to enhance noise suppression. Spatial consistency modeling is achieved through top-down and bottom-up cross-scale bidirectional updates while preserving the original feature discriminability. Finally, visual guidance for text focusing and language guidance for visual enhancement are achieved, solving the problems of semantic gap and insufficient attention to small objects, and completing the progressive optimization from basic alignment to spatial consistency alignment.

[0051] In one possible embodiment, step 104 includes: establishing a bidirectional attention framework to calculate visually guided text-focused attention scores and language-guided visual enhancement attention scores, respectively; utilizing an attention gating mechanism, dynamically generating language gating values ​​and language interaction strengths based on global features of visual queries and language keys, and simultaneously dynamically generating visual gating values ​​based on global features of language queries and visual keys to adjust the interaction strength; introducing a hierarchical spatial enhancement strategy for different levels of multi-scale image features, achieving cross-scale bidirectional interaction between high-level semantics and low-level details through upsampling and downsampling operations, and updating attention maps at each level; introducing residual gating fusion to dynamically balance the new features generated by cross-modal fusion with the original features, thereby obtaining visual memory features and text enhancement features.

[0052] For example, such as Figure 5 As shown, a multi-scale gated bidirectional attention interaction module is embedded in the dual-stream feature extraction process to dynamically calculate the interaction weight between visual regions and text words, thereby achieving visually guided text focusing and language-guided visual enhancement, bridging the semantic gap between visual and language representations. Unlike the problem of insufficient fine-grained control caused by standard bidirectional attention sharing QKV query projection, the embodiments of this application achieve progressive alignment optimization through innovative design.

[0053] First, a basic bidirectional attention framework is established to achieve initial alignment of visual and linguistic features, and the visually guided text-focused attention score is calculated: ; in, For visual query projection, For learnable projection matrices, Projecting language keys For feature dimension, This is a scaling factor used to mitigate the explosion of numerical values ​​in high dimensions. These represent visual regions and text lexical indexes, respectively.

[0054] To address semantic bias caused by noise interference, a gating mechanism is introduced to dynamically adjust the interaction intensity and improve alignment robustness. ; in, The Sigmoid function outputs the gate value. ; , For learnable gating parameters; , These are the global mean features of visual queries and language keys, respectively. This is a feature concatenation operation that enables the model to focus on semantic fragments relevant to the target.

[0055] Simultaneously, computational language-guided visual enhancement attention scores are used to complete two-way interaction: ; in For language query projection, For learnable projection matrix; For visual key projection, The learningable projection matrix is ​​used to apply the gating mechanism described above.

[0056] And the same gating mechanism is used to achieve visual enhancement for language-guided communication: ; By using bidirectional interaction to guide visual feature enhancement with linguistic features, the alignment accuracy is improved a second time. At the same time, independent residual connections and masking are introduced to enhance feature preservation and noise suppression capabilities.

[0057] To address the issue of insufficient attention to small objects caused by inconsistent spatial feature representations across layers, the method provided in the embodiments of this application introduces a hierarchical spatial enhancement strategy into the cross-scale gated bidirectional attention mechanism, and achieves multi-scale bidirectional updates based on a list of spatial shapes. ; A list of spatial shapes is represented as follows: ; in, For upsampling operations (bilinear interpolation), the high-rise buildings... Upsample the low-resolution attention map to the current scale; For downsampling operations, the lower layer Downsample the high-resolution attention map to the current scale; As the scale weight coefficient, this mechanism enables cross-scale bidirectional interaction of high-level semantics, integrates multi-scale features under language constraints, and completes the progression from basic alignment to spatial consistency alignment.

[0058] Finally, residual gating fusion is used to preserve the discriminative power of the original features and ensure feature integrity. See the following example for details: In one possible embodiment, residual-gated fusion is introduced to dynamically balance the new features generated by cross-modal fusion with the original features, thereby obtaining visual memory features and text enhancement features. Text Enhancement Features The formula is expressed as follows: ; ; in, Original visual features Features of the original text The mask corresponding to the original visual features. This is the mask corresponding to the original text features. and is the residual gating coefficient.

[0059] and For residual gating coefficients, To ensure learnable parameters, the new features are dynamically balanced with the original features during the fusion process, avoiding modal feature overload caused by feature fusion, and outputting the fused visual memory features. Text Enhancement Features This achieves the ultimate progression from spatial consistency alignment to feature integrity preservation.

[0060] Step 105: Based on the language-guided query vector, visual memory features, and text enhancement features, the predicted bounding box is iteratively optimized through the decoder to output the final remote sensing target detection result.

[0061] For example, by integrating enhanced text description, location markers, and cross-modal fusion features, the query continuously absorbs textual semantics and visual spatial information through multi-layer iteration of the decoder. The visual deformable attention can adaptively sample target region features and dynamically update the bounding box by combining reference points to achieve refinement. Finally, the target bounding box coordinates and category probabilities are output through the detection head, realizing accurate mapping from language commands to visual positioning. Through the collaborative optimization of data and models, the accuracy, robustness, and positioning accuracy of remote sensing target detection in open scenarios are significantly improved.

[0062] For example, query vector Initial reference point With fusion features , Through multi-layer iterative optimization of the decoder and bi-directional refinement of query and fused features through multi-layer feature interaction, the focus is ultimately on the dynamic adjustment of bounding box coordinates, forming a complete closed loop from semantic alignment to spatial localization, and outputting high-precision detection results. See the following example for details: In one possible embodiment, step 105 includes: inputting the language-guided query vector, visual memory features, and text enhancement features into the decoder for layer-by-layer iterative optimization; After completing the layer iteration, based on the final query features, the target class probability and the final bounding box coordinates are output through the classification header and bounding box header.

[0063] Each iteration includes: refining query features through query self-attention; deeply binding the refined query with text enhancement features through text cross-attention; interacting the query bound to text semantics with visual memory features through visual deformable cross-attention; adaptively sampling target region features; performing nonlinear optimization of fused features through a feedforward network; and dynamically updating the bounding box coordinates of reference points based on the optimized query features. For example, the decoder may include six iterative update layers: first, it refines the query's own feature representation through a self-attention mechanism; second, it strengthens the deep binding between the query and text semantics through text cross-attention; third, it uses visually deformable cross-attention to focus the query on the visual target region; fourth, it performs non-linear optimization of the interacted features through a feedforward network; and finally, it dynamically updates the reference point coordinates based on the optimized query features, achieving progressive refinement of the bounding box. The layer iteration formula is as follows: Step 1: Query self-attention: We refine the feature representation of the query vector itself, explore the feature associations and dependencies within the query, and retain the spatial location encoding information of the query, laying a high-quality feature foundation for subsequent cross-modal cross-attention interaction.

[0064] ; ; in, This is the query vector refined by self-attention, where the superscript sa is an abbreviation for self-attention. No. The query vector output by the layer iteration. No. Layer query sine position encoding, For multi-head self-attention operators, the first... The query vector output after the layer iteration is completed serves as the three inputs (Query, Key, and Value) for multi-head self-attention, enabling the mining of feature associations within the query vector and refining feature representation.

[0065] Step 2: Use text cross-attention to enhance language guidance Let the self-attention refined query vector and text augmentation features Deep interaction strengthens the deep binding between queries and text semantics, ensuring that subsequent visual feature interactions are always guided by text descriptions, avoiding deviation from semantic goals.

[0066] ; ; The query vector is refined through self-attention. As the query of cross-attention, the text-enhanced features output after cross-scale gating bidirectional attention fusion serve as the key and value of cross-attention, providing pure text semantic information for the query; Text feature masking masks the feature information of invalid words in the text, avoiding noise interference with cross-modal alignment. The query vector enhanced by text cross-attention; the superscript "text" indicates that the vector has been deeply integrated with text semantics. Step 3: Visually Deformable Cross-Attention Focuses on the Visual Target The query vector bound to the text semantics interacts with the visual memory feature Vout from step 4. By adaptively sampling the visual features of the target region in the remote sensing image through multi-scale deformable attention, the query can accurately focus on the real visual target location, thus realizing the feature mapping from text semantics to visual space.

[0067] ; ; in, For the first The reference point input of the layer provides sampling location guidance for deformable attention, allowing feature sampling to focus on the target area around the reference point; Features of visual memory A list of spatial shapes for multi-scale visual features. Visual scale starting index, marking the hierarchical division of multi-scale visual features, in conjunction with To achieve accurate cross-scale feature sampling, For visual feature masking, The query vector after visual deformable cross-attention fusion, with the superscript "cross" indicating that the vector has completed text-visual cross-modal feature fusion.

[0068] Step 4: Feedforward Network Optimization Query features after cross-modal fusion Nonlinear transformation and feature refinement are performed by using a two-layer fully connected feedforward network (FFN) to enhance the nonlinear expressive power of features, uncover finer-grained semantic spatial relationships in the features, and provide optimal features for subsequent bounding box updates.

[0069] ; ; Step 5: Update reference points (bounding box) Based on optimized query features The coordinates of the normalized reference point (boundary box) are dynamically corrected to achieve progressive refinement of the boundary box; each iteration makes a small correction to the boundary box, and after 6 iterations, the boundary box accurately matches the actual location of the target in the remote sensing image.

[0070] ; ; ; in, For the first The normalized reference point after layer iteration has a value range of [0,1]. The inverse Sigmoid function, with the normalized reference point Restoring to the original numerical space provides a basis for coordinate correction. For the first Layer bounding box prediction head, For the first The bounding box correction for layer iterations, for non-normalized reference points Make numerical corrections. No. The normalized reference point updated after each layer iteration is used as the reference point for the next layer. Iteration Step 3 This enables iterative refinement of the bounding box.

[0071] Final detection output: After After layer iteration, the results are output using the classification header and bounding box header: ; ; in, For the number of categories, For class probabilities, To normalize the bounding box coordinates, the entire process of transformation from the original image to high-precision detection results is completed.

[0072] Through multi-layer iteration of the decoder, the language-guided query continuously absorbs textual semantics and visual spatial information, gradually refining the bounding box coordinates, achieving a precise mapping from language instructions to visual positioning, and significantly improving the accuracy and robustness of remote sensing target detection.

[0073] This application proposes a text-guided cross-scale collaborative gating remote sensing target detection method. First, it acquires the remote sensing image to be detected and generates a fine-grained text description corresponding to the image content, containing spatial orientation information. Then, based on the remote sensing image and the fine-grained text description, it extracts multi-scale features of the image and hierarchical semantic features of the text using a dual-stream coding architecture. Next, it constructs a language-guided query vector based on the hierarchical semantic features of the text, which possesses spatial awareness and semantic alignment attributes. Then, through a cross-scale gating bidirectional attention interaction module, it dynamically fuses the multi-scale features of the image, the hierarchical semantic features of the text, and the language-guided query vector to obtain cross-modal fusion-based visual memory features and text enhancement features. Finally, based on the language-guided query vector, visual memory features, and text enhancement features, iteratively optimizes the predicted bounding box through a decoder to output the final remote sensing target detection result. This scheme aims to overcome the limitations of existing remote sensing image visual detection technologies in terms of coarse-grained semantic descriptions, sensitivity to noise in cross-modal feature alignment, and weak spatial relationship modeling. By constructing a data-model-space collaborative optimization framework, it achieves high-precision and robust remote sensing target detection.

[0074] To verify the effectiveness of the methods and technical solutions provided in the embodiments of this application, the experiment selected Swin-Tiny, which was pre-trained on datasets such as O365, GoldG, and Cap4M, as the base model. Further fine-tuning and performance verification were performed on the DIOR-RSVG remote sensing dataset. This dataset is divided into three categories: training, validation, and testing. The training set contains 12,340 images, the validation set contains 1,690 images, and the test set contains 3,372 images, covering targets at different scales and meeting the needs of fine-grained performance analysis.

[0075] Both the experimental hardware and software environments adopted industry-standard configurations to ensure the reliability of the results: For hardware, an RTX 3090 GPU with 24GB of dedicated video memory was selected; for software, a deep learning environment was built on the Ubuntu 20.04 operating system, configured with the CUDA 11.3 computing framework, PyTorch 1.11, and TorchVision 0.12.0 deep learning library. All experimental schemes were implemented using Python 3.9. During training, Adam was used as the optimizer, and the core hyperparameters were specifically set: the learning rate was 1e^(-1 / 2)^2. 3. Set the batch size to 2 and the number of training epochs to 10 to ensure the model converges fully and avoids overfitting.

[0076] To verify the effectiveness of the method provided in this application in addressing the text-visual semantic gap, a rigorous ablation validation design was employed. The evaluation criteria followed common standards in the object detection field, specifically the standard COCO evaluation protocol. Average precision and average recall were calculated using IoU thresholds of 0.5:0.95, and fine-grained performance decomposition analysis was performed according to target scale (small, medium, and large). Simultaneously, to quantify the impact of text description on cross-modal alignment, three text description comparison groups were constructed: the Baseline group directly concatenated the original short text description; the Word group retained only the target category name; and the Phase group utilized the Qwen-VL-Plus model to expand and generate fine-grained semantic descriptions. Through multi-dimensional comparative experiments, the superiority and reliability of the technical solution in this application's embodiments were comprehensively verified.

[0077] Table 1 Ablation comparison experiments on the DIOR-RSVG dataset

[0078] As shown in Table 1, red indicates the best performance and green indicates the second best. This experiment included three ablation validation sets, focusing on the progressive effects of data augmentation (Phase, Word), cross-scale bidirectional attention (BI), and the gate module. All sets were compared on the validation and test sets. On the DIOR-RSVG test set, using GroudingDINO as the baseline, the mAP@0.5:0.95 was 0.3897. After data augmentation, mAP@0.5:0.95 increased to 0.3946 (+0.49 percentage points). After introducing the cross-scale bidirectional attention (BI) mechanism, mAP@0.5:0.95 increased to 0.439 (4.93 percentage points higher than the baseline), effectively bridging the visual-text semantic gap, especially improving accuracy by 4.43 percentage points in small object detection. With the further introduction of a dynamic gating mechanism, the mAP@0.5:0.95 of the Phase+BI+gate combination jumped to 0.4716 (an improvement of 8.19 percentage points compared to the baseline). Among them, the AP for small targets improved by 0.0506 (+5.06%), medium targets by 0.0789 (+7.89%), and large targets by 0.1152 (+11.52%), which fully verified the gating mechanism's effect on suppressing redundant noise.

[0079] Understandably, while meeting the accuracy requirements for remote sensing target detection, the embodiments of this application rely on a complete chain of automated fine-grained semantic generation and purification, dual-stream feature extraction, cross-scale gated bidirectional attention fusion, and cross-modal detection. They embed a cross-scale bidirectional attention module and combine it with dynamic gating adjustment for collaborative innovation, constructing a data-model-space collaborative optimization system. This effectively overcomes the limitations of traditional methods, such as shallow semantics and inefficient fusion. Experimental verification shows that this module can significantly improve the alignment performance of text and visual features, reduce alignment deviation, and effectively improve the accuracy of multi-scale target detection, especially optimizing the detection effect of small-sized, dense targets in remote sensing scenes. The method provided by the embodiments of this application can automatically enrich the semantic dimension of text, improve cross-modal alignment stability, reduce manual intervention, and provide a high-precision, low-threshold, and robust solution for remote sensing target detection. It is both innovative and practical, providing reliable support for the implementation of visual grounding tasks for remote sensing images.

[0080] In summary, the method provided by the embodiments of this application achieves efficient and accurate alignment of visual and linguistic semantics through the aforementioned steps. From feature extraction to cross-scale gating bidirectional interaction, dynamic attention adjustment, and then to semantic enhancement and residual fusion, each step aims to bridge the cross-modal semantic gap, significantly improving the accuracy and robustness of target detection. Through continuous data feedback and model optimization, the method provided by the embodiments of this application can continuously improve its detection performance in complex remote sensing scenarios, providing strong technical support for remote sensing target detection, intelligent monitoring, autonomous driving, and other fields.

[0081] The steps described above are merely for clarity in describing the technical solution. In actual implementation, they can be combined into one step, or certain steps can be broken down into multiple steps, as long as they involve the same logical relationship, they are all within the scope of protection of this application. Any insignificant modifications or designs added to the algorithm or process, as long as they do not change the core of the algorithm or process, are also within the scope of protection of this application.

[0082] Another embodiment of this application proposes a text-guided cross-scale collaborative gated remote sensing target detection system. The details of this text-guided cross-scale collaborative gated remote sensing target detection system are described below. The following implementation details are provided for ease of understanding and are not essential for implementing this example. Figure 6 This is a schematic diagram of the structure of a text-guided, cross-scale collaborative gating remote sensing target detection system proposed in this embodiment, including: The fine-grained semantic generation module 210 is used to acquire the remote sensing image to be detected and generate a fine-grained text description corresponding to the content of the remote sensing image; wherein, the fine-grained text description includes spatial orientation information; The dual-stream encoder module 220 is used to extract multi-scale features of the image and hierarchical semantic features of the text based on remote sensing images and fine-grained text descriptions through a dual-stream coding architecture. The language-guided query construction module 230 is used to construct language-guided query vectors based on text hierarchical semantic features; wherein, the language-guided query vectors have spatial awareness and semantic alignment attributes; The cross-scale gated bidirectional attention fusion module 240 is used to dynamically fuse multi-scale features of images, hierarchical semantic features of text, and language-guided query vectors through the cross-scale gated bidirectional attention interaction module to obtain cross-modal fusion visual memory features and text enhancement features. The iterative decoding and localization output module 250 is used to iteratively optimize the predicted bounding box through the decoder based on the language-guided query vector, visual memory features and text enhancement features, and output the final remote sensing target detection result.

[0083] It is worth noting that all modules involved in this embodiment are logical modules. In practical applications, a logical module can be a physical module, a part of a physical module, or an organic combination of multiple physical modules. Furthermore, to highlight the innovative aspects of this application, this embodiment does not introduce modules that are not closely related to solving the technical problems proposed in this application. However, this does not mean that other modules are absent from this embodiment.

[0084] It is not difficult to see that this embodiment is a system embodiment corresponding to the above method embodiments, and this embodiment can be implemented in conjunction with the above method embodiments. The relevant technical details and technical effects mentioned in the above method embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above method embodiments.

[0085] Another embodiment of this application provides an electronic device, such as Figure 7 As shown, it includes a processor 31 and a memory 32. The memory 32 stores instructions that the processor 31 can execute. When the processor 31 is configured to execute the instructions, the electronic device can implement a text-guided cross-scale collaborative gating remote sensing target detection method as described in the above method embodiment.

[0086] The memory and processor are connected via a bus, which includes any number of interconnecting buses and bridges, connecting various circuits of one or more processors and the memory. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single component or multiple components, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0087] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0088] Another embodiment of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, enables a text-guided cross-scale collaborative gating remote sensing target detection method as described in the above method embodiments.

[0089] That is, those skilled in the art will understand that all or part of the steps in the above method embodiments can be implemented by a program instructing related hardware. The program is stored in a storage medium and includes several instructions to cause a device (such as a microcontroller, chip, etc.) or processor to execute all or part of the steps of the method described in the method embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0090] It will be understood by those skilled in the art that the above embodiments are specific implementations of this application, and various changes in form and detail can be made in practical applications without departing from the spirit and scope of this application. For those skilled in the art, several improvements and modifications can be made without departing from the principles of this application, and these improvements and modifications are also considered to be within the scope of protection of this application.

Claims

1. A text-guided, cross-scale collaborative gating remote sensing target detection method, characterized in that, The method includes: The system acquires the remote sensing image to be detected and generates a fine-grained text description corresponding to the content of the remote sensing image; the fine-grained text description includes spatial orientation information. Based on remote sensing images and fine-grained text descriptions, a dual-stream coding architecture is used to extract multi-scale features of the images and hierarchical semantic features of the text, respectively. Based on the hierarchical semantic features of text, a language-guided query vector is constructed; the language-guided query vector has spatial awareness and semantic alignment attributes. By using a cross-scale gated bidirectional attention interaction module, multi-scale image features, hierarchical semantic features of text, and language-guided query vectors are dynamically fused to obtain cross-modal fusion visual memory features and text enhancement features. Based on language-guided query vectors, visual memory features, and text-enhanced features, the predicted bounding boxes are iteratively optimized through a decoder to output the final remote sensing target detection results.

2. The method according to claim 1, characterized in that, The step of acquiring the remote sensing image to be detected and generating a fine-grained text description corresponding to the content of the remote sensing image includes: By using a large language vision model to perform content analysis on remote sensing images, a structured long text description containing target attributes and scene relationships is automatically generated. Automated filtering of structured long text descriptions is performed to remove abstract concepts and non-visualizable words, resulting in semantically pure text descriptions. The overlap rate between the target bounding box and the preset image grid in the remote sensing image is calculated, and the target structured location marker is determined based on the maximum overlap rate and the second largest overlap rate; wherein, the target structured location marker is used to represent the primary absolute position and the secondary absolute position; By concatenating the target's structured location markers with semantically clean text descriptions, a fine-grained text description containing spatial orientation information is obtained.

3. The method according to claim 2, characterized in that, The dual-stream coding architecture includes a visual encoder and a text encoder; based on remote sensing images and fine-grained text descriptions, the dual-stream coding architecture extracts multi-scale features of the images and hierarchical semantic features of the text, including: Multi-scale features of the image are extracted using a visual encoder. ;in, For batch size, The feature map space size, Number of visual feature channels; Text hierarchical semantic features are extracted using a text encoder. ;in, The length of the text word. For semantic dimension.

4. The method according to claim 3, characterized in that, The construction of language-guided query vectors based on text-layered semantic features includes: The initial query and reference point are obtained through the learnable embedding layer; The reference point is normalized and a corresponding sinusoidal position code is generated to enable the query to have spatial awareness. By utilizing the text cross-attention mechanism, the query combination position is encoded and then interacted with the text hierarchical semantic features to obtain the language-guided query vector.

5. The method according to claim 4, characterized in that, Initialize the query as , The reference point is , ; For the number of queries, For model feature dimensions; The text cross-attention mechanism is used to encode the query combination position, which is then interacted with the hierarchical semantic features of the text to obtain a language-guided query vector. The formula is expressed as follows: ； in, For sinusoidal position encoding, Text features after incorporating visual information This is a text feature mask.

6. The method according to claim 5, characterized in that, The method utilizes a cross-scale gating bidirectional attention interaction module to dynamically fuse multi-scale image features, hierarchical semantic features of text, and language-guided query vectors to obtain cross-modal fused visual memory features and text enhancement features, including: A bidirectional attention framework is established to calculate visually guided text-focused attention scores and language-guided visually enhanced attention scores separately. By utilizing the attention gating mechanism, language gating values ​​and language interaction intensity are dynamically generated based on the global features of visual queries and language keys. At the same time, visual gating values ​​are dynamically generated based on the global features of language queries and visual keys to adjust the interaction intensity. For different levels of multi-scale features in images, a hierarchical spatial enhancement strategy is introduced. Through upsampling and downsampling operations, cross-scale bidirectional interaction between high-level semantics and low-level details is achieved, and attention maps of each layer are updated. By introducing residual gating fusion, the new features generated by cross-modal fusion are dynamically balanced with the original features, thereby obtaining visual memory features and text enhancement features.

7. The method according to claim 6, characterized in that, The introduction of residual-gated fusion dynamically balances the new features generated by cross-modal fusion with the original features, thereby obtaining visual memory features and text enhancement features. Text Enhancement Features The formula is expressed as follows: ； ； in, Original visual features Features of the original text The mask corresponding to the original visual features. This is the mask corresponding to the original text features. and is the residual gating coefficient.

8. The method according to claim 7, characterized in that, The method, based on language-guided query vectors, visual memory features, and text-enhanced features, iteratively optimizes the predicted bounding boxes through a decoder, outputting the final remote sensing target detection results, including: The language-guided query vector, visual memory features, and text-enhanced features are input into the decoder for layer-by-layer iterative optimization. Each layer-by-layer iteration includes: refining the query features through query self-attention; deeply binding the refined query with text-enhanced features through text cross-attention; interacting the query with the bound text semantics through visual deformable cross-attention; adaptively sampling target region features; performing non-linear optimization of the fused features through a feedforward network; and dynamically updating the bounding box coordinates of the reference points based on the optimized query features. After completing the layer iteration, based on the final query features, the target class probability and the final bounding box coordinates are output through the classification header and bounding box header.

9. An electronic device, characterized in that, include: The processor and memory, wherein the memory stores instructions executable by the processor, and the processor is configured to, when executing the instructions, enable the electronic device to implement a text-guided cross-scale collaborative gating remote sensing target detection method as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it can implement a text-guided cross-scale collaborative gating remote sensing target detection method as described in any one of claims 1 to 8.

Images