A self-attention method and system based on axial squeezing and window

Abstract

The present invention relates to the field of artificial intelligence technology, and in particular to a self-attention method and system based on axial squeezing and window. The self-attention method based on axial squeezing and window includes: performing a first normalization process on the original feature to obtain a first normalized feature; inputting the first normalized feature into a multi-head self-attention unit based on a window and a multi-head self-attention unit based on axial squeezing respectively, outputting a detail enhancement weight based on the multi-head self-attention unit based on the window, outputting a squeezed attention feature based on the axial squeezing, multiplying and fusing the squeezed attention feature and the detail enhancement weight to obtain an attention feature; performing a first residual connection process, a second normalization process, a full connection process and a second residual connection process on the attention feature in sequence to obtain a target feature. The present invention is at least conducive to reducing the computational complexity of global self-attention.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a self-attention method and system based on axial squeezing and windows. Background Art

[0002] Before the emergence of deep learning algorithms, methods based on handcrafted descriptors were widely used in various registration tasks. However, due to the design limitations of existing knowledge, traditional matching algorithms often perform poorly when dealing with more complex matching tasks due to handcrafted descriptors. With the vigorous development of deep learning algorithms, their deep feature extraction and expression capabilities can make up for the defects of shallow feature instability and improve the robustness of registration algorithms. However, for the registration of remote sensing images, due to the presence of a large number of sparse and repeated texture areas, it is difficult to detect feature points and obtain discriminative feature representations. This is due to the limited receptive field and the difficulty in obtaining global context information.

[0003] Although joint detection and description methods also recognize the importance of a large receptive field, previous studies generally focus more on designing loss functions, and their network structures are usually simpler and lighter.

[0004] Specifically, in the field of image processing, the Transformer model overcomes the shortcoming of traditional convolutional neural networks being limited to local receptive fields. The Transformer model captures the overall semantic information of the image by dividing the image into small local regions and directly extracting global features from these local regions. However, the computational complexity of the Transformer model's global self-attention mechanism is proportional to the quadratic power of the image size, which results in a high cost for both training and reasoning of the Transformer model. Therefore, how to reduce the computational complexity of the Transformer model while maintaining global information has become a key issue in deep learning research.

[0005] The high computational cost of the global self-attention mechanism makes it difficult to apply the Transformer model to lightweight network structures. In order to achieve efficient self-attention operations, a window-based self-attention mechanism has emerged. SwinTransformer is a hierarchical Transformer model structure designed for the visual field. Swin Transformer adopts a shift window scheme to improve computational efficiency by limiting self-attention calculations to non-overlapping local windows and allowing cross-window connections. MaxVit (Multi-Axis Vision Transformer) is a general Transformer model structure. MaxVit adopts a simpler and more effective method, which consists of local attention and expanded global attention, and achieves global perception with linear complexity. However, the above method can only obtain relatively sparse global attention.

[0006] In addition, in order to reduce the computational complexity of the global self-attention mechanism, axial attention is proposed, but axial attention ignores the problem of missing local details due to pooling / averaging operations. For example, SeaFormer (Squeeze-Enhanced Axial Transformer) provides a general attention block that achieves a dense global receptive field through axial squeezing and detail enhancement, but it relies heavily on convolutional neural networks.

[0007] In summary, how to reduce the computational complexity of global self-attention while maintaining global information while getting rid of the dependence on convolutional neural networks is a technical problem that needs to be solved urgently in the field of deep learning research. Summary of the invention

[0008] In view of this, the present invention aims to provide a self-attention method and system based on axial squeezing and window, which utilizes axial squeezing features to perform global self-attention, reduces the computational complexity of global self-attention, and utilizes window-based self-attention to enhance local details.

[0009] To achieve the above object, the technical solution created by the present invention is implemented as follows:

[0010] On the one hand, the present invention provides a self-attention method based on axial squeezing and window, which is characterized in that it includes: performing a first normalization process on the original feature to obtain a first normalized feature; inputting the first normalized feature into a window-based multi-head self-attention unit and an axial squeezing-based multi-head self-attention unit respectively, the window-based multi-head self-attention unit sequentially performs window segmentation processing and a first multi-head self-attention process on the first normalized feature, and outputs a detail enhancement weight, the axial squeezing-based multi-head self-attention unit sequentially performs squeezing processing and a second multi-head self-attention process on the first normalized feature, and outputs a squeezed attention feature, and multiplying and fusion the squeezed attention feature and the detail enhancement weight to obtain an attention feature; performing a first residual connection process, a second normalization process, a full connection process, and a second residual connection process on the attention feature in sequence to obtain a target feature.

[0011] Furthermore, the first residual connection processing includes: aggregating the attention feature and the original feature to obtain the first residual connection feature; performing a second normalization processing, a full connection processing and a second residual connection processing on the first residual connection feature in sequence to obtain the target feature.

[0012] Furthermore, the attention features are sequentially subjected to a first residual connection process, a second normalization process, and a full connection process to obtain a fully connected feature; the second residual connection process includes: aggregating the fully connected feature and the first residual connection feature to obtain the target feature.

[0013] Furthermore, performing window segmentation processing on the first normalized feature includes: segmenting the first normalized feature into a plurality of non-overlapping window features.

[0014] Furthermore, the first multi-head self-attention processing includes: performing a first attention calculation on multiple window features to obtain window attention features, wherein the first attention calculation also includes a similarity calculation, and the similarity calculation includes adding a relative position bias; after performing the first attention calculation, it also includes: restoring the window attention features to obtain detail enhancement weights.

[0015] Furthermore, performing extrusion processing on the first normalized feature includes: performing horizontal extrusion on the first normalized feature to obtain a horizontal extrusion feature; and performing vertical extrusion on the first normalized feature to obtain a vertical extrusion feature.

[0016] Furthermore, the second multi-head self-attention processing includes: performing a second attention calculation on the horizontal squeezing feature to obtain the horizontal squeezing attention feature; performing a second attention calculation on the vertical squeezing feature to obtain the vertical squeezing attention feature; wherein the second attention calculation also includes adding a learnable position bias.

[0017] Furthermore, after the second multi-head self-attention processing, it also includes: using a broadcast mechanism to fuse horizontal squeezed attention features and vertical squeezed attention features to obtain initial squeezed attention features, and convolving the initial squeezed attention features to obtain squeezed attention features.

[0018] On the other hand, the present invention provides a self-attention system based on axial squeezing and window, including: a first normalization layer, the first normalization layer performs a first normalization process on the original feature to obtain a first normalized feature; a parallel window-based multi-head self-attention unit and an axial squeezing-based multi-head self-attention unit, the first normalized feature is input into the window-based multi-head self-attention unit and the axial squeezing-based multi-head self-attention unit respectively, the window-based multi-head self-attention unit sequentially performs window segmentation process and first multi-head self-attention process on the first normalized feature, and outputs detail enhancement weights, the axial squeezing-based multi-head self-attention unit sequentially performs squeezing process and second multi-head self-attention process on the first normalized feature, and outputs squeezed attention feature, the squeezed attention feature and the detail enhancement weight are multiplicatively fused to obtain an attention feature; a post-processing module, the post-processing module sequentially performs a first residual connection process, a second normalization process, a full connection process and a second residual connection process on the attention feature to obtain a target feature.

[0019] Compared with the prior art, the invention can achieve the following beneficial effects:

[0020] In the self-attention method based on axial squeezing and window provided in an embodiment of the present invention, the original features are subjected to a first normalization process and enter into two parallel multi-head self-attentions to obtain attention features, wherein one multi-head self-attention refers to a multi-head self-attention unit based on a window, and the other multi-head self-attention refers to a multi-head self-attention unit based on axial squeezing;

[0021] For the multi-head self-attention unit based on axial squeezing, it performs axial position encoding through horizontal squeezing and vertical squeezing to realize the calculation of axial self-attention, and then fuses the horizontal squeezing attention features and vertical squeezing attention features obtained by axial squeezing through the broadcast mechanism and 1×1 convolution processing. The use of the multi-head self-attention unit based on axial squeezing is conducive to greatly reducing the computational complexity of global self-attention;

[0022] For the window-based multi-head self-attention unit, the first normalized feature is evenly divided into window features, and then self-attention calculation is performed on the window features, which is helpful to enhance the details of local features;

[0023] In addition, the squeezed attention features and detail enhancement weights are multiplicatively fused, and then residual connections, normalization, and full connections are performed to obtain the final output, realizing a Transformer model that can obtain a dense global receptive field at a low computational cost while maintaining structural details. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings constituting part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments and descriptions of the present invention are used to explain the present invention and do not constitute an improper limitation on the present invention. In the drawings:

[0025] Figure 1 An algorithm flow chart of the self-attention method based on axial squeezing and window described in the embodiment of the present invention;

[0026] Figure 2 An algorithm flow chart of the parallel multi-head self-attention module described in the embodiment of the present invention;

[0027] Figure 3 A schematic diagram of the overall architecture of the remote sensing image registration method described in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the invention more clear, the invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described here are only used to explain the invention and do not constitute a limitation of the invention.

[0029] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0030] In the description of the invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing the invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the invention, unless otherwise specified, the meaning of "multiple" is two or more.

[0031] In the description of the invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the invention can be understood according to specific circumstances.

[0032] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0033] On the one hand, the present invention provides a self-attention method based on axial squeezing and window, including: performing a first normalization process on the original feature to obtain a first normalized feature; inputting the first normalized feature into a window-based multi-head self-attention unit and an axial squeezing-based multi-head self-attention unit respectively, the window-based multi-head self-attention unit sequentially performs window segmentation process and first multi-head self-attention process on the first normalized feature, and outputs detail enhancement weights, the axial squeezing-based multi-head self-attention unit sequentially performs squeezing process and second multi-head self-attention process on the first normalized feature, and outputs squeezed attention feature, multiplicatively fuses the squeezed attention feature and the detail enhancement weight to obtain an attention feature; sequentially performs a first residual connection process, a second normalization process, a full connection process and a second residual connection process on the attention feature to obtain a target feature.

[0034] It should be noted that the window-based multi-head self-attention unit and the axial squeezing-based multi-head self-attention unit constitute a multi-head self-attention module in parallel.

[0035] Furthermore, the first residual connection processing includes: aggregating the attention feature and the original feature to obtain the first residual connection feature. The first residual connection feature is sequentially subjected to a second normalization processing, a full connection processing, and a second residual connection processing to obtain the target feature.

[0036] Furthermore, the attention features are sequentially subjected to a first residual connection process, a second normalization process, and a full connection process to obtain a fully connected feature; the second residual connection process includes: aggregating the fully connected feature and the first residual connection feature to obtain the target feature.

[0037] Furthermore, performing window segmentation processing on the first normalized feature includes: segmenting the first normalized feature into a plurality of non-overlapping window features.

[0038] Furthermore, the first multi-head self-attention processing includes: performing a first attention calculation on multiple window features to obtain window attention features, wherein the first attention calculation also includes a similarity calculation, and the similarity calculation includes adding a relative position bias; after performing the first attention calculation, it also includes: restoring the window attention features to obtain detail enhancement weights.

[0039] Furthermore, performing extrusion processing on the first normalized feature includes: performing horizontal extrusion on the first normalized feature to obtain a horizontal extrusion feature; and performing vertical extrusion on the first normalized feature to obtain a vertical extrusion feature.

[0040] Furthermore, the second multi-head self-attention processing includes: performing a second attention calculation on the horizontal squeezing feature to obtain the horizontal squeezing attention feature; performing a second attention calculation on the vertical squeezing feature to obtain the vertical squeezing attention feature; wherein the second attention calculation also includes adding a learnable position bias.

[0041] Furthermore, after the second multi-head self-attention processing, it also includes: using a broadcast mechanism to fuse horizontal squeezed attention features and vertical squeezed attention features to obtain initial squeezed attention features, and convolving the initial squeezed attention features to obtain squeezed attention features.

[0042] A specific embodiment of the self-attention method based on axial squeezing and window provided in an embodiment of the present invention is described below.

[0043] In some embodiments of the present invention, reference Figure 1 and Figure 2 The parallel multi-head self-attention module consists of parallel window-based multi-head self-attention units ( ) and axial squeezing-based multi-head self-attention units ( ), with a second normalization layer LN2 and a fully connected layer MLP after the parallel multi-head self-attention module. There is a first normalization layer LN1 before the parallel multi-head self-attention module, and residual connections are applied after the parallel multi-head self-attention module and after the fully connected layer. The calculation process of the self-attention method based on axial squeezing and window can be expressed as:

[0044] ;

[0045] ;

[0046] Among them, LN1 represents the first normalization process, LN2 represents the second normalization process, Represents the detail enhancement weights of the output of the window-based multi-head self-attention unit, represents the squeezed attention features output by the multi-head self-attention unit based on axial squeezing, is the output of the second normalization layer LN2, is the attention feature processed by the first residual connection (i.e., the first residual connection feature), is the original feature, , H is the height of the original feature, W is the width of the original feature, C is the number of channels of the original feature, R is a real number set, represents the target feature, Figure 2 In the equation, Q, K and V are The linear projection of Q represents the query vector, K represents the key vector, and V represents the value vector. , , ,in, , as well as are learnable weights, , , , ,pass The time complexity of doing dot multiplication to calculate Q is , so the total time complexity of calculating Q, calculating K, and calculating V is .

[0047] refer to Figure 2 , for the processing of the first normalized feature by the multi-head self-attention unit based on axial squeezing, in order to achieve more efficient calculation and aggregate global information at the same time, firstly, the average value of the query Q is taken in the horizontal direction to squeeze Q horizontally, and the average value of the query Q is taken in the vertical direction to squeeze Q vertically. Similarly, corresponding squeezing operations are performed on K and V respectively. After the squeezing operation, the horizontal squeezing feature and the vertical squeezing feature are obtained. The linear projection of the vertical squeezing feature is , and , , the linear projection of the horizontal extrusion feature is , and , , , and They represent the horizontal squeezing of Q, K and V respectively. , and They represent squeezing Q, K, and V along the vertical direction, respectively. The squeezing operation can preserve the global information on a single axis, thereby greatly reducing the computational complexity of subsequent global semantic extraction.

[0048] In order to consider the location information of the feature map, that is, to make , , and Perceiving its position in the squeezed axial feature, for the second multi-head self-attention processing, the embodiment of the present invention introduces a learnable position bias, that is, adding a position encoding vector , , and ,in , , Expressed as The added positional encoding vector, Expressed as The added position encoding vector, similarly, Expressed as The added positional encoding vector, Expressed as The added position encoding vector.

[0049] The second multi-head self-attention processing includes: performing a second attention calculation on the horizontal squeezed feature to obtain the horizontal squeezed attention feature; performing a second attention calculation on the vertical squeezed feature to obtain the vertical squeezed attention feature, and then using the broadcast mechanism to fuse the horizontal squeezed attention feature and the vertical squeezed attention feature to obtain the initial squeezed attention feature, and convolving the initial squeezed attention feature to obtain the squeezed attention feature. It should be noted that the broadcast mechanism is a technique commonly used in mathematics and computer science to process operations between arrays or tensors of different shapes.

[0050] The expression of the squeezed attention feature output by the multi-head self-attention unit based on axial squeezing can be:

[0051] ;

[0052] Here, d represents the hidden dimension.

[0053] It should be noted that for the horizontal squeezed attention features and the vertical squeezed attention features, each position of the feature map only propagates information on the two squeezed axial features. Therefore, feature fusion is achieved through an efficient broadcast mechanism, and the axial information is further fused through 1×1 convolution.

[0054] Next, we calculate the time complexity of multi-head self-attention for squeezing operation along the vertical direction, that is, the time complexity of obtaining vertical squeezing attention features. First, , , , and The time complexity of doing point multiplication is , , Again with The time complexity of doing point multiplication is , the subsequent calculation is equivalent to a matrix Doing point multiplication, the time complexity is , , and The time complexity of the multi-head self-attention for squeezing operation in the vertical direction is obtained by adding the three together: .

[0055] Similarly, replacing H with W, the time complexity of multi-head self-attention for squeezing operation in the horizontal direction is In summary, the time complexity of the multi-head self-attention unit based on axial squeezing is .

[0056] For the window-based multi-head self-attention unit, it should be noted that although the squeezing operation can effectively extract global semantic information, it also sacrifices local details. Therefore, the embodiment of the present invention applies a window-based multi-head self-attention unit to enhance spatial details.

[0057] Continue to refer Figure 2 , the window-based multi-head self-attention unit splits the first normalized feature. Specifically, Q, K, and V are evenly divided into blocks of size Multiple non-overlapping windows, the tensor corresponding to Q is , the tensor corresponding to K is , the tensor corresponding to V is , The segmentation process now applies self-attention in the local spatial dimension, which is equivalent to performing attention calculations within a small window. For window-based multi-head self-attention units, the calculation of similarity includes adding relative position biases , After the first attention calculation, it also includes: restoring the window attention feature to obtain the detail enhancement weight, and multiplying and fusing the squeezed attention feature and the detail enhancement weight to obtain the attention feature.

[0058] The expression of the detail enhancement weight output by the window-based multi-head self-attention unit can be: ,d represents the hidden dimension.

[0059] The time complexity of the window-based multi-head self-attention unit is calculated as follows: , , and The time complexity of doing point multiplication is , , , and The time complexity of doing point multiplication is , , the subsequent calculation is equivalent to another matrix Doing point multiplication, the time complexity is , , and Add together to get , which is just the time complexity of processing a single window, with a total of windows, Multiplying by the number of windows, we get the time complexity of the window-based multi-head self-attention unit: .

[0060] The time complexity of calculating Q, calculating K and calculating V is , the time complexity of multi-head self-attention unit based on axial squeezing And the time complexity of the window-based multi-head self-attention unit Adding them together, the total time complexity of the self-attention method based on axial squeezing and window provided in the embodiment of the present invention is obtained as follows: It should be noted that the time complexity of Swin Transformer is .

[0061] Since the technical solution provided by the present invention adopts a parallel structure, Q, K and V only need to be calculated once, and a multi-head self-attention unit based on axial squeezing with relatively low computational cost is used. If H=512, W=512, P=8, C=128 are set, the floating point number of the self-attention method based on axial squeezing and window provided by the embodiment of the present invention is reduced by about half compared with the floating point number of Swin Transformer, and the embodiment of the present invention only needs about 216 million FLOPs (floating point operations per second). At the same time, the embodiment of the present invention can retain rich structural details while obtaining a dense global receptive field.

[0062] Compared with SeaFormer, the window-based multi-head self-attention unit used in the present invention is more suitable for detail enhancement. The present invention realizes the self-attention calculation of the complete path while further reducing the number of parameters and training memory usage. Specifically, the number of parameters of SeaFormer is 5.2M, while the number of parameters of the present invention is 4.01M. When training on 500 512×512 data sets, the memory usage of SeaFormer is 15480M, while the memory usage of the present invention is 11406M. This is only for a single layer. If multiple layers are stacked, the effect will be more obvious.

[0063] On the other hand, the present invention provides a self-attention system based on axial squeezing and window, including: a first normalization layer, the first normalization layer performs a first normalization process on the original feature to obtain a first normalized feature; a parallel window-based multi-head self-attention unit and an axial squeezing-based multi-head self-attention unit, the first normalized feature is input into the window-based multi-head self-attention unit and the axial squeezing-based multi-head self-attention unit respectively, the window-based multi-head self-attention unit sequentially performs window segmentation process and first multi-head self-attention process on the first normalized feature, and outputs detail enhancement weights, the axial squeezing-based multi-head self-attention unit sequentially performs squeezing process and second multi-head self-attention process on the first normalized feature, and outputs squeezed attention feature, the squeezed attention feature and the detail enhancement weight are multiplicatively fused to obtain an attention feature; a post-processing module, the post-processing module sequentially performs a first residual connection process, a second normalization process, a full connection process and a second residual connection process on the attention feature to obtain a target feature.

[0064] It should be noted that the self-attention method based on axial squeezing and window provided in the embodiment of the present invention can be applied to the registration method of remote sensing images to provide global context features and a larger receptive field for remote sensing images with sparse and repeated texture areas, thereby improving the registration effect.

[0065] The specific steps of the remote sensing image registration method may include: S1: inputting the original remote sensing image pair into the modality adapter to obtain the modality adaptation feature; S2: inputting the modality adaptation feature into the convolution layer of the feature encoder to obtain the original feature; S3: using the self-attention method based on axial squeezing and window provided in the embodiment of the present invention to process the original feature to obtain the target feature; S4: inputting the target feature into the convolution layer through the residual connection to obtain the feature descriptor; S5: inputting the feature descriptor into the original detector and the improved attention layer of the feature detector (the self-attention system based on axial squeezing and window provided in the embodiment of the present invention), respectively, and using conditional probability estimation to globally detect feature points; S6: using the FLANN algorithm for preliminary matching, and combining the RANSAC algorithm for external point elimination to obtain the final registration result.

[0066] Specifically, refer to Figure 3 , Figure 3 The overall architecture of the remote sensing image registration method is demonstrated. The remote sensing image registration method replaces the traditional convolution with improved self-attention (self-attention based on axial squeezing and window provided in an embodiment of the present invention) based on the ReDFeat algorithm, and simultaneously obtains two modal remote sensing images as input, each image is processed by a modality adapter respectively, and then a dense feature descriptor is generated by a weight-shared feature encoder, and a weight-shared feature detector generates feature detection points. Finally, the FLANN algorithm is used for preliminary matching, and the RANSAC algorithm is combined for outlier elimination to obtain the final registration result.

[0067] Specifically, the original remote sensing image pair can be input into the modality adapter to obtain the modality adaptation feature; the modality adaptation feature is input into the convolution layer of the feature encoder to obtain the original feature. The convolution layer can be described as: ;in represents the input features, Represents the first convolution, BN represents batch normalization, and ReLU represents the activation function. The target feature is input into the convolution layer through the residual connection to obtain the feature descriptor ; , ; The input of the feature detector consists of two branches, the same as the network structure of ReDFeat, one of which is the original detector used to simulate the prior probability , which means that the probability of the i-th point being a feature point is directly calculated based on its descriptor. Indicates that point i is a feature point, and represents the descriptor of the i-th point. The structure of the other branch requires the conditional probability Modeling is performed, and the probability represents the probability of detecting it as a feature point according to the global feature descriptor, where Indicates that point i is detected. It is related to global information, so this branch should have a larger receptive field. Therefore, the self-attention based on axial squeezing and window provided by the embodiment of the present invention can obtain a dense global receptive field at a lower computational cost while retaining rich structural details. Indicates the posterior probability that the detected point is a feature point, which can be expressed by Approximate calculations are performed to obtain feature detection points. Finally, the obtained feature descriptors and feature detection points are preliminarily matched using the FLANN algorithm, and then RANSAC is used to eliminate external points to obtain the homography matrix and complete the image registration.

[0068] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the disclosure of the present invention can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document does not limit this.

[0069] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A self-attention method based on axial squeezing and window, characterized in that: The self-attention method based on axial squeezing and window is applied in the registration method of remote sensing images, and the registration method of remote sensing images includes: S1: inputting the original remote sensing image pair into the modality adapter to obtain the modality adaptation feature; S2: inputting the modality adaptation feature into the convolution layer of the feature encoder to obtain the original feature; S3: using the self-attention method based on axial squeezing and window to process the original feature to obtain the target feature; S4: inputting the target feature into the convolution layer through the residual connection to obtain the feature descriptor; S5: inputting the feature descriptor into the original detector of the feature detector and the self-attention system based on axial squeezing and window respectively, and using conditional probability estimation to globally detect feature points; S6: using the FLANN algorithm to perform preliminary matching, and combining the RANSAC algorithm to remove external points to obtain the final registration result; The self-attention method based on axial squeezing and window includes: Performing a first normalization process on the original features to obtain first normalized features; The first normalized feature is respectively input into a window-based multi-head self-attention unit and an axial squeezing-based multi-head self-attention unit, the window-based multi-head self-attention unit sequentially performs window segmentation processing and first multi-head self-attention processing on the first normalized feature, and outputs a detail enhancement weight, the axial squeezing-based multi-head self-attention unit sequentially performs squeezing processing and second multi-head self-attention processing on the first normalized feature, and outputs a squeezed attention feature, and the squeezed attention feature and the detail enhancement weight are multiply fused to obtain an attention feature; The attention feature is sequentially subjected to a first residual connection process, a second normalization process, a full connection process, and a second residual connection process to obtain the target feature.

2. The self-attention method based on axial squeezing and window according to claim 1, characterized in that: The first residual connection processing includes: aggregating the attention feature and the original feature to obtain a first residual connection feature; The first residual connection feature is sequentially subjected to the second normalization processing, the full connection processing and the second residual connection processing to obtain the target feature.

3. The self-attention method based on axial squeezing and window according to claim 2, characterized in that: performing the first residual connection processing, the second normalization processing, and the full connection processing on the attention feature in sequence to obtain a fully connected feature; The second residual connection processing includes: aggregating the fully connected features and the first residual connection features to obtain a target feature.

4. The self-attention method based on axial squeezing and window according to claim 1, characterized in that: Performing the window segmentation processing on the first normalized feature includes: segmenting the first normalized feature into a plurality of non-overlapping window features.

5. The self-attention method based on axial squeezing and window according to claim 4, characterized in that: The first multi-head self-attention processing includes: performing a first attention calculation on a plurality of the window features to obtain a window attention feature, wherein the first attention calculation also includes a similarity calculation, and the similarity calculation includes adding a relative position offset; After performing the first attention calculation, the method also includes: restoring the window attention feature to obtain the detail enhancement weight.

6. The self-attention method based on axial squeezing and window according to claim 1, characterized in that: Performing the extrusion process on the first normalized feature includes: horizontally extruding the first normalized feature to obtain a horizontally extruded feature; The first normalized feature is vertically extruded to obtain a vertically extruded feature.

7. The self-attention method based on axial squeezing and window according to claim 6, characterized in that: The second multi-head self-attention processing includes: Performing a second attention calculation on the horizontal squeezing feature to obtain a horizontal squeezing attention feature; Performing a second attention calculation on the vertical squeezing feature to obtain a vertical squeezing attention feature; Wherein, the second attention calculation also includes adding a learnable position bias.

8. The self-attention method based on axial squeezing and window according to claim 7, characterized in that: After performing the second multi-head self-attention processing, it also includes: using a broadcast mechanism to fuse the horizontal squeeze attention feature and the vertical squeeze attention feature to obtain an initial squeeze attention feature, and convolving the initial squeeze attention feature to obtain the squeeze attention feature.

9. A self-attention system based on axial squeezing and window, characterized in that: The self-attention system based on axial squeezing and window is applied in a remote sensing image registration method, and the remote sensing image registration method includes: S1: inputting the original remote sensing image pair into a modality adapter to obtain modality adaptation features; S2: inputting the modality adaptation features into a convolution layer of a feature encoder to obtain original features; S3: using the self-attention system based on axial squeezing and window to process the original features to obtain target features; S4: inputting the target features into a convolution layer through a residual connection to obtain feature descriptors; S5: inputting the feature descriptors into the original detector of the feature detector and the self-attention system based on axial squeezing and window, respectively, and using conditional probability estimation to globally detect feature points; S6: using the FLANN algorithm for preliminary matching, and combining the RANSAC algorithm for external point elimination to obtain the final registration result; The self-attention system based on axial squeezing and window includes: A first normalization layer, wherein the first normalization layer performs a first normalization process on the original features to obtain first normalized features; A parallel window-based multi-head self-attention unit and an axial squeezing-based multi-head self-attention unit, inputting the first normalized feature into the window-based multi-head self-attention unit and the axial squeezing-based multi-head self-attention unit respectively, the window-based multi-head self-attention unit sequentially performs window segmentation processing and first multi-head self-attention processing on the first normalized feature, and outputs detail enhancement weights, the axial squeezing-based multi-head self-attention unit sequentially performs squeezing processing and second multi-head self-attention processing on the first normalized feature, and outputs a squeezed attention feature, and the squeezed attention feature and the detail enhancement weight are multiply fused to obtain an attention feature; A post-processing module, wherein the post-processing module sequentially performs a first residual connection process, a second normalization process, a full connection process, and a second residual connection process on the attention feature to obtain a target feature.

Images