Remote sensing rotating target self-attention mechanism construction method based on geometric compatibility perception

Abstract

This invention discloses a method for constructing a self-attention mechanism for remote sensing rotating targets based on geometric compatibility perception, belonging to the fields of computer vision and deep learning technology. First, the invention performs feature analysis on the input image, extracting rotation bounding box parameters for candidate target regions to construct semantic tokens and geometric tokens. Then, it calculates the geometric consistency constraint between any two geometric tokens using a geometric relationship function and optimizes the attention weight model with a geometric compatibility perception matrix. Finally, it completes feature collaborative iteration and weighted aggregation through a Transformer update mechanism, thereby improving the ability to model the orientation consistency of rotating targets. This invention overcomes the limitations of pure semantic modeling, taking into account both long-distance semantic dependencies and local geometric consistency. It can alleviate problems such as feature confusion and localization offset in rotating target detection and remote sensing image fusion, improving the geometric fidelity and task accuracy of the model, and is applicable to various geometrically sensitive visual engineering scenarios.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, remote sensing image processing, and deep learning, specifically to a method for constructing a self-attention mechanism for remote sensing rotating targets based on geometric compatibility perception. This method introduces structured reasoning with geometric consistency modeling into the self-attention mechanism, and is applicable to visual tasks sensitive to geometric deformation, target pose, and spatial structure, such as high-resolution remote sensing image fusion, rotating target detection, dense scene target recognition, and multimodal image registration. Background Technology

[0002] In deep learning vision tasks, self-attention mechanisms, with their advantages of capturing long-range feature dependencies and adaptively focusing on key information, have become a core module of the Transformer architecture and various deep networks, achieving excellent performance in tasks such as image classification, object detection, and image fusion. Traditional self-attention mechanisms generate attention weights by calculating the similarity of query-key features, thereby achieving weighted aggregation of value features. While focusing on modeling the semantic correlation of features, they neglect the inherent geometric structure attributes and spatial pose compatibility of visual data, making them difficult to adapt to complex vision tasks such as remote sensing images and dense scenes containing arbitrarily rotated, scale-deformed, and elongated dense targets.

[0003] Currently, conventional self-attention mechanisms have several inherent flaws: First, they only focus on semantic relevance of features, failing to incorporate geometric information such as target geometry, spatial orientation, and scale into the weighting modeling logic. This leads to a disconnect between attention weight allocation and the actual geometric structure of the target, making it prone to feature confusion and positioning misalignment for rotating targets and elongated targets (such as ships, vehicles, and airport runways in remote sensing images). Second, the global weighting method of traditional self-attention lacks geometric constraints. When processing multi-source image fusion and target detection tasks, it easily disrupts the local geometric consistency of the image, causing fusion artifacts, detection box misalignment, and pose misjudgment, reducing the robustness of the model in geometrically sensitive tasks. Third, existing improved self-attention mechanisms mostly focus on complexity optimization and receptive field expansion, rarely designing specifically for geometrically compatible perception characteristics. They cannot achieve collaborative modeling of semantic feature association and geometric structural constraints, making it difficult to meet the actual needs of tasks such as high-precision remote sensing interpretation and dense target detection.

[0004] In summary, existing self-attention mechanisms, lacking geometric compatibility perception capabilities, struggle to meet the stringent requirements of geometric fidelity and accurate pose perception in complex visual tasks, thus limiting their application in remote sensing image processing, rotating target detection, and other fields. Therefore, developing a self-attention mechanism construction method that can balance semantic association modeling and geometric compatibility perception, overcoming the bottleneck of the lack of geometric characteristics in traditional self-attention mechanisms, and improving the performance of deep models in geometrically sensitive visual tasks has become a pressing technical challenge in this field. Summary of the Invention

[0005] To address the inherent drawbacks of existing traditional self-attention mechanisms that focus solely on semantic feature association modeling and lack geometric compatibility perception capabilities, this invention aims to overcome the shortcomings of existing technologies and provide a method for constructing a remote sensing rotating target self-attention mechanism based on geometric compatibility perception. This method solves the technical problems of existing self-attention mechanisms neglecting geometric attributes such as target geometry, spatial orientation, and scale ratio in weight modeling, leading to a disconnect between attention weight allocation and actual geometric structure, loss of geometric consistency, and low attitude perception accuracy. It is particularly suitable for the application needs of geometrically sensitive visual tasks such as remote sensing image fusion and rotating target detection.

[0006] The above objectives are achieved through the following technical solutions:

[0007] The present invention provides a method for constructing a self-attention mechanism for remotely sensed rotating targets based on geometrically compatible sensing, comprising the following steps:

[0008] S1. Perform feature parsing on the input image, extract the rotation bounding box parameters of the candidate target region, and construct semantic tokens and geometric tokens;

[0009] S2. Calculate any two geometric tokens using geometric relationship functions. and Token Geometric consistency constraints between ;

[0010] S3, Apply the geometric consistency constraint Introduced as a structured geometry modulation term into Transformer self-attention computation, resulting in geometry-compatible perceptual attention weights. Geometric consistency constraints As an explicit prior, it is added to the original attention score to generate the geometric attention score. Geometric attention score Attention weights are normalized using softmax. Ultimately, the output tokens of each token pair are controlled during the feature aggregation stage. The degree of contribution is used to achieve geometry-guided context modeling;

[0011] S4. Based on the Transformer update rules based on geometry-compatible awareness self-attention, the final geometry-aware Transformer (referred to as GeoFormer) module is generated, following the standard Transformer architecture with residual connections.

[0012] S5. Based on the geometrically compatible perceptual attention weights Modeling the geometric relationships between targets improves the ability to model the orientation consistency of rotating targets.

[0013] Further, the semantic token mentioned in step S1 is represented as:

[0014] Geometric tokens are represented as: in, Represents a semantic token. Represents geometric tokens, This represents the semantic feature projection function. Represents the geometric feature projection function. For the input feature map, Indicates target scale information. Indicates the target rotation angle. Indicates the spatial location of the target;

[0015] Furthermore, the geometric consistency constraint described in step S2 It is expressed as follows: in, Represents the spatial proximity weight. Indicates the directional consistency weight. Indicates the shape similarity weight; Represents any two geometric tokens. and Token Spatial proximity between them; Represents any two geometric tokens. and Token Consistency in direction between them; Represents any two geometric tokens. and Token The similarity in shape between them; specifically,

[0016] Spatial nearest neighbor calculation: in, Geometric Token Spatial location characteristics; Geometric Token Spatial location characteristics, It is an exponential function with the natural constant e as its base, used to map distances to similarity values ​​between 0 and 1; It is a spatial scale hyperparameter used to control the sensitivity of spatial proximity; It is a spatial scale hyperparameter in the range of (0,1], used to control the sensitivity of spatial proximity; The larger the similarity, the slower the decay of distant tokens, and the more the model tends to focus on a broader spatial context; conversely, The smaller the value, the more the model focuses on locally neighboring tokens.

[0017] Directional consistency calculation: in, Geometric Token The target rotation orientation angle, Geometric Token The target rotation orientation angle;

[0018] Shape similarity calculation: in, Geometric Token The target aspect ratio shape features Geometric Token The target aspect ratio and shape characteristics.

[0019] Furthermore, the geometrically compatible perceptual attention weights described in step S3 The calculation is as follows: in, These are the attention query matrix, key matrix, and value matrix, respectively. The feature dimension is T, where the superscript T denotes the transpose operation of the matrix; G is a matrix with elements of 10 ... The geometric compatibility matrix, i.e.: Where N is the total number of geometric tokens. G represents an N x N real matrix used to model the spatial, orientation, and shape geometric relationships between any two tokens;

[0020] Any two geometric tokens and Token Geometric attention scores between The calculation is as follows:

[0021] Normalized attention weights The calculation is as follows: in, Geometric Token Corresponding semantic query vector and geometric token Geometric perceptual attention scores between corresponding semantic key vectors Represented as a summation index;

[0022] Geometric Token Output feature tokens after geometric perception attention weighting Represented as in, Geometric Token The corresponding value vector; Characteristic matrix The 1 eigenvector.

[0023] Furthermore, the Transformer update rule based on geometry-compatible perceptual self-attention described in step S4 is as follows: in, The intermediate features are represented by geometric attention and residual connections, and F represents the input feature map. This indicates the enhanced features that this module ultimately outputs. This represents a feedforward neural network.

[0024] Compared with existing object detection or visual Transformer methods, the self-attention mechanism construction method based on geometric compatibility perception proposed in this invention achieves collaborative modeling of semantic information and geometric structure information by explicitly introducing geometric consistency constraints between targets during the self-attention calculation process, and has the following beneficial effects:

[0025] 1. Enhance the ability to model the geometric structure of rotating targets.

[0026] This invention constructs geometric tokens that include spatial location, orientation angle, and scale ratio, and uses geometric relationship functions to establish a geometric compatibility matrix between tokens. This enables attention computing to explicitly perceive the spatial and directional relationships between targets, thereby significantly improving the ability to express rotating targets and orientation-sensitive targets.

[0027] 2. Improve the accuracy of target detection in complex scenarios.

[0028] Traditional self-attention mechanisms rely primarily on semantic features for global modeling, lacking constraints on the geometric relationships of the target and easily susceptible to background interference. This invention modulates the attention weights through geometric consistency constraints, giving higher weights to tokens with reasonable geometric relationships, thereby effectively suppressing noisy features and improving detection accuracy, especially in the detection of dense targets, elongated targets, and targets in remote sensing scenes.

[0029] 3. Achieve efficient integration of geometric priors and the Transformer framework.

[0030] This invention directly incorporates the geometric compatibility matrix as a structured modulation term into the original attention score without altering the overall computational framework of the Transformer. This allows the proposed GeoFormer module to be seamlessly embedded into existing visual Transformers or object detection networks (such as Faster R-CNN, DETR, etc.), exhibiting good compatibility and scalability.

[0031] 4. Enhance the ability to model the contextual relationships between objectives.

[0032] By employing a geometrically compatible attention mechanism, this invention not only captures semantic relevance but also establishes geometric relationships between targets, enabling the network to more accurately model the spatial distribution structure of target groups and improve overall scene understanding.

[0033] 5. Low computational cost and easy to implement in engineering.

[0034] This invention achieves geometric perception modeling simply by constructing a lightweight geometrically compatible matrix and modulating the attention score, without introducing complex network structures or additional detection branches. Therefore, while ensuring improved detection performance, the increase in computational overhead is small, making it of good engineering application value.

[0035] In summary, this invention effectively enhances the model's ability to model rotating targets and complex scenes by introducing a geometrically compatible perceptual self-attention mechanism. While maintaining the simplicity of the network structure, it significantly improves detection performance and has good practical value and prospects for promotion. Attached Figure Description

[0036] Figure 1 is a schematic diagram of the overall module structure of the geometric perception Transformer of the present invention;

[0037] Figure 2 is a schematic diagram of the dual-token construction process of the present invention;

[0038] Figure 3 is a schematic diagram of the geometrically compatible self-attention calculation process of the present invention. Detailed Implementation

[0039] To enable those skilled in the art to more clearly understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the following embodiments are only for illustrating the present invention and are not intended to limit the scope of protection of the present invention. Various modifications or substitutions made to the present invention by those skilled in the art without departing from the technical concept of the present invention should fall within the scope of protection of the present invention.

[0040] This invention proposes a method for constructing a self-attention mechanism for remote sensing rotating targets based on geometric compatibility perception, which enhances the geometric relationship modeling capability of visual Transformers in rotating target detection and complex scene vision tasks. This method constructs target geometric information tokens and establishes geometric consistency constraints between tokens, explicitly introducing geometric prior information into the Transformer's self-attention computation process, thereby achieving joint modeling of semantic features and geometric structure information.

[0041] The method of the present invention includes the following steps:

[0042] S1. Perform feature parsing on the input image, extract the rotation bounding box parameters of the candidate target region, and construct semantic tokens and geometric tokens; the semantic token is represented as:

[0043] Geometric tokens are represented as: in, Represents a semantic token; Represents a geometric token; The semantic feature projection function is obtained through a linear layer transformation operation; This represents the geometric feature projection function; in this embodiment, CNN convolutional transformation is used. Input feature map; Indicates target scale information; Indicates the target rotation angle. Indicates the spatial location of the target.

[0044] S2. Calculate any two geometric tokens using geometric relationship functions. and Token Geometric consistency constraints between , It is expressed as follows: in, Represents the spatial proximity weight. Indicates the directional consistency weight. Indicates the shape similarity weight; Represents any two geometric tokens. and Token Spatial proximity between them; Represents any two geometric tokens. and Token Consistency in direction between them; Represents any two geometric tokens. and Token The similarity in shape between them; specifically,

[0045] Spatial nearest neighbor calculation: in, Geometric Token Spatial location characteristics; Geometric Token Spatial location characteristics, It is an exponential function with the natural constant e as its base, used to map distances to similarity values ​​between 0 and 1; It is a spatial scale hyperparameter used to control the sensitivity of spatial proximity; It is a spatial scale hyperparameter in the range of (0,1], used to control the sensitivity of spatial proximity; The larger the similarity, the slower the decay of distant tokens, and the more the model tends to focus on a broader spatial context; conversely, The smaller the value, the more the model focuses on locally neighboring tokens.

[0046] Directional consistency calculation: in, Geometric Token The target rotation orientation angle, Geometric Token The target rotation orientation angle;

[0047] Shape similarity calculation: in, Geometric Token The target aspect ratio shape features Geometric Token The target aspect ratio and shape characteristics.

[0048] S3, Apply the geometric consistency constraint Introduced as a structured geometry modulation term into Transformer self-attention computation, resulting in geometry-compatible perceptual attention weights. Geometric consistency constraints As an explicit prior, it is added to the original attention score to generate the geometric attention score. Geometric attention score Attention weights are normalized using softmax. Ultimately, the output tokens of each token pair are controlled during the feature aggregation stage. The degree of contribution is used to achieve geometry-guided context modeling;

[0049] The geometrically compatible perceptual attention weight The calculation is as follows: in, These are the attention query matrix, key matrix, and value matrix, respectively. The feature dimension is T, where the superscript T denotes the transpose operation of the matrix; G is a matrix with elements of 10 ... The geometric compatibility matrix, i.e.: Where N is the total number of geometric tokens. G represents an N x N real matrix used to model the spatial, orientation, and shape geometric relationships between any two tokens;

[0050] Any two geometric tokens and Token Geometric attention scores between The calculation is as follows:

[0051] Normalized attention weights The calculation is as follows: in, Geometric Token Corresponding semantic query vector and geometric token Geometric perceptual attention scores between corresponding semantic key vectors Represented as a summation index;

[0052] Geometric Token Output feature tokens after geometric perception attention weighting Represented as in, Geometric Token The corresponding value vector; Characteristic matrix The 1 eigenvector.

[0053] S4. Based on the geometry-compatible self-attention-based Transformer update rule, the final GeoFormer module is generated, following the standard Transformer architecture with residual connections; the geometry-compatible self-attention-based Transformer update rule is as follows: in, The intermediate features are represented by geometric attention and residual connections, and F represents the input feature map. This indicates the enhanced features that this module ultimately outputs. This represents a feedforward neural network.

[0054] S5. Based on the geometrically compatible perceptual attention weights Modeling the geometric relationships between targets improves the ability to model the orientation consistency of rotating targets.

[0055] To verify the effectiveness of the present invention, the following experiment was conducted.

[0056] 1. Experimental Dataset:

[0057] This embodiment conducts experiments on the DOTA (Dataset for Object Detection in Aerial Images) dataset. DOTA is one of the most representative publicly available benchmark datasets in the field of remote sensing rotating target detection, widely used to evaluate the performance of detection algorithms in complex aerial imagery scenes. The DOTA dataset has the following characteristics: high-resolution remote sensing images (800×800 to 4000×4000); large target scale variations; dense distribution; targets rotating in any direction. The dataset contains 15 target categories:

[0058] Airplane (PL), baseball field (BD), bridge (BR), runway (GTF), small vehicle (SV), large vehicle (LV), ship (SH), tennis court (TC), basketball court (BC), storage tank (ST), football field (SBF), ring road (RA), port (HA), swimming pool (SP), helicopter (HC).

[0059] The DOTA dataset is divided into a training set of 1411 images, a validation set of 458 images, and a test set of 937 images. Due to the high resolution of the original images, a sliding window cropping strategy is used to divide the images into 1024×1024 sub-images, with 200 pixels of overlap to avoid truncating the target.

[0060] 2. Evaluation indicators:

[0061] The experiment used mean average precision (mAP) as the main evaluation index.

[0062] For each category, first calculate the average precision (AP): in, This represents the precision-recall curve. This refers to recall.

[0063] The final overall performance passed calculate: Where M is the number of target categories (in DOTA) ), This represents the detection accuracy for class a.

[0064] 3. Experimental setup:

[0065] All experiments were implemented using the PyTorch deep learning framework. The experimental hardware environment was as follows: GPU: NVIDIA RTX 4090D; CUDA version: 11.3; PyTorch version: 1.10; Python version: 3.8; network backbone: ResNet-50 + FPN; input image size: 1024 pixels × 1024 pixels; optimizer: SGD.

[0066] 4. Baseline detection framework:

[0067] To verify the versatility of the GeoFormer module, this experiment embeds it into three typical rotating target detection frameworks:

[0068] Table 1. Experimental Framework Types: method type Rotated Faster R-CNN Two-stage rotating detector Oriented R-CNN Improved two-stage rotating detector Rotated ATSS Single-stage anchorless rotating detector

[0069] The GeoFormer module was inserted between the backbone network and the detector head to enhance geometric relationship modeling capabilities. In the experiment, all other network structures and training parameters were kept completely consistent; only the GeoFormer module was added to ensure experimental fairness.

[0070] 5. The results of the implementation effectiveness verification are shown in Table 2:

[0071] Table 2. Experimental results of embedding this module in different detection frameworks: Model mAP PL BD BR GTF SV LV SH TC BC ST SBF RA HA SP HC Rotated Faster RCNN 63.3 80.6 74.8 41.7 63.1 58.0 74.2 78.4 90.1 54.7 53.6 53.7 51.2 66.3 55.1 54.0 + GeoFormer 63.8 80.7 73.2 41.0 64.8 57.6 75.4 71.2 89.7 61.7 53.4 58.0 52.1 66.8 54.9 56.6 Oriented RCNN 65.1 80.8 75.3 44.0 69.2 59.3 77.8 80.0 90.1 61.5 53.6 59.9 49.7 67.7 53.3 54.4 + GeoFormer 65.6 80.7 73.9 44.3 68.2 58.9 78.1 80.1 90.2 62.0 53.4 60.8 52.3 67.9 53.9 59.2 Rotated ATSS 61.7 80.5 68.7 41.6 65.7 56.3 74.6 75.0 89.5 54.0 59.9 49.2 50.3 61.1 51.8 46.7 + GeoFormer 62.1 80.5 69.0 42.2 63.8 55.0 74.5 75.4 89.4 58.6 60.0 49.5 49.6 60.7 52.6 50.6

[0072] As verified by this embodiment, compared with the traditional self-attention mechanism, the self-attention mechanism constructed in this invention significantly improves the multi-class target localization accuracy of multiple algorithms in remote sensing rotating target detection tasks. It effectively solves the problems of missed detection, false detection, positioning offset, and angle misjudgment that are prone to occur in rotating target detection, and greatly improves the robustness of the model for detecting targets with complex postures.

[0073] As shown in Table 3, the GeoFormer module proposed in this invention can continuously improve the detection performance of different rotation detectors. Specifically, GeoFormer achieves improvements on both two-stage detectors (Rotated Faster R-CNN and Oriented R-CNN) and single-stage anchorless detectors (Rotated ATSS), indicating that the module does not depend on a specific detection framework. The mAP of Rotated Faster R-CNN improved from 63.3% to 63.8% (+0.5), while the mAP of Oriented R-CNN and Rotated ATSS both improved by 0.4. Oriented R-CNN is already a relatively strong rotation detection framework, but it still achieved a +0.4 mAP improvement, indicating that there is still room for improvement in geometric modeling.

[0074] Table 3. Module gain on a typical detection framework: method Baseline mAP +GeoFormer promote Rotated Faster R-CNN 63.3 63.8 +0.5 Oriented R-CNN 65.1 65.6 +0.4 Rotated ATSS 61.7 62.1 +0.4

[0075] Furthermore, significant performance improvements were observed in several categories with distinct geometric features, such as baseball fields, soccer fields, and helicopters, demonstrating that the proposed geometry-aware attention mechanism effectively enhances the representation of orientation-sensitive objects. These results indicate that GeoFormer can capture the spatial geometric compatibility between rotating objects and provide complementary contextual information for accurate object localization.

Claims

1. A geometric compatibility perception based self-attention mechanism construction method for remote sensing rotating targets, characterized in that, Includes the following steps: S1. Perform feature parsing on the input image, extract the rotation bounding box parameters of the candidate target region, and construct semantic tokens and geometric tokens; S2. Calculate any two geometric tokens using geometric relationship functions. and Token Geometric consistency constraints between ; S3, Apply the geometric consistency constraint Introduced as a structured geometry modulation term into Transformer self-attention computation, resulting in geometry-compatible perceptual attention weights. Geometric consistency constraints As an explicit prior, it is added to the original attention score to generate the geometric attention score. Geometric attention score Attention weights are normalized using softmax. Ultimately, the output tokens of each token pair are controlled during the feature aggregation stage. The degree of contribution is used to achieve geometry-guided context modeling; S4. Based on the Transformer update rules based on geometry-compatible self-attention, the final GeoFormer module is generated, which follows the standard Transformer architecture with residual connections. S5. Based on the geometrically compatible perceptual attention weights Modeling the geometric relationships between targets improves the ability to model the orientation consistency of rotating targets.

2. The method for constructing a remote sensing rotating target self-attention mechanism based on geometrically compatible perception according to claim 1, characterized in that, The semantic token mentioned in step S1 is represented as follows: Geometric tokens are represented as: in, Represents a semantic token. Represents geometric tokens, Represents the semantic feature projection function. Represents the geometric feature projection function. For the input feature map, Indicates target scale information. Indicates the target rotation angle. Indicates the spatial location of the target.

3. The method for constructing a remote sensing rotating target self-attention mechanism based on geometrically compatible perception according to claim 1, characterized in that, The geometric consistency constraint in step S2 It is expressed as follows: in, Represents the spatial proximity weight. Indicates the directional consistency weight. Indicates the shape similarity weight; Represents any two geometric tokens. and Token Spatial proximity between them; Represents any two geometric tokens. and Token Consistency in direction between them; Represents any two geometric tokens. and Token The similarity in shape between them; specifically, Spatial nearest neighbor calculation: in, Geometric Token Spatial location characteristics; Geometric Token Spatial location characteristics, It is an exponential function with the natural constant e as its base, used to map distances to similarity values ​​between 0 and 1; It is a spatial scale hyperparameter in the range of (0,1], used to control the sensitivity of spatial proximity; The larger the similarity, the slower the decay of distant tokens, and the more the model tends to focus on a broader spatial context; conversely, The smaller the value, the more the model focuses on locally neighboring tokens; Directional consistency calculation: in, Geometric Token The target rotation orientation angle, Geometric Token The target rotation orientation angle; Shape similarity calculation: in, Geometric Token The target aspect ratio shape features Geometric Token The target aspect ratio and shape characteristics.

4. The method for constructing a remote sensing rotating target self-attention mechanism based on geometrically compatible perception according to claim 1, characterized in that, The geometrically compatible perceptual attention weights described in step S3 The calculation is as follows: in, These are the attention query matrix, key matrix, and value matrix, respectively. The feature dimension is T, where the superscript T denotes the transpose operation of the matrix; G is a matrix with elements of 10 ... The geometric compatibility matrix, i.e.: Where N is the total number of geometric tokens. G represents an N x N real matrix used to model the spatial, orientation, and shape geometric relationships between any two tokens; Any two geometric tokens and Token Geometric attention scores between The calculation is as follows: Normalized attention weights The calculation is as follows: in, Geometric Token Corresponding semantic query vector and geometric token Geometric perceptual attention scores between corresponding semantic key vectors Represented as a summation index; Geometric Token Output feature tokens after geometric perception attention weighting Represented as in, Geometric Token The corresponding value vector; Characteristic matrix The 1 eigenvector.

5. The method for constructing a remote sensing rotating target self-attention mechanism based on geometrically compatible perception according to claim 1, characterized in that, The Transformer update rule based on geometry-compatible self-attention described in step S4 is as follows: in, The intermediate features are represented by geometric attention and residual connections, and F represents the input feature map. This indicates the enhanced features that this module ultimately outputs. This represents a feedforward neural network.

Images