Unmanned aerial vehicle spatial pose estimation method and device, electronic equipment and storage medium

Abstract

The present application belongs to the field of computer vision and intelligent perception technology, and provides a kind of unmanned aerial vehicle space pose estimation method, device, electronic equipment and storage medium, including: performing feature extraction on RGB input tensor and thermal imaging input tensor, to obtain RGB feature vector and thermal feature vector;Perform feature fusion on RGB feature vector and thermal feature vector with learnable modal weight and global attention, to obtain multi-modal fusion feature vector;According to the frequency domain transformation of RGB feature vector, thermal feature vector and multi-modal fusion feature vector, an enhanced spatial feature map is obtained;Training is performed by local fractal descriptor and enhanced spatial feature map;The enhanced spatial feature map, fractal robust embedding vector and multi-modal fusion feature vector are spliced and predicted to obtain the spatial pose estimation result of the unmanned aerial vehicle, and the precision and stability of the unmanned aerial vehicle in complex environment are improved by the above technical solution.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and intelligent sensing technology, and in particular to a method, apparatus, electronic device and storage medium for estimating the spatial pose of a UAV. Background Technology

[0002] With the rapid development of drones in fields such as security patrol, emergency rescue, smart agriculture, geographic surveying and mapping, and unmanned transportation, achieving high-precision autonomous positioning and attitude estimation in complex environments has become a key issue. Traditional pose estimation methods based on a single modality (such as RGB images, IMU, or LiDAR) perform well in controlled environments, but often suffer from problems such as feature failure, large matching errors, and positioning drift under conditions of low light, strong noise, or occlusion, making it difficult to meet the robustness and real-time requirements of practical applications.

[0003] While existing multimodal fusion methods can improve accuracy to some extent, they still have the following shortcomings:

[0004] Most existing technologies simply overlay features at the feature or decision layers, lacking in-depth mining of the geometric structure and scale characteristics between modes. In low-light, foggy, or highly reflective environments, or in non-textured scenarios (such as nighttime security patrols and disaster relief), existing technologies do not fully utilize frequency domain information, resulting in insufficient feature representation of structural edges and low-texture regions. In complex scenarios such as agricultural and forestry vegetation monitoring and topographic mapping, existing technologies lack self-supervised optimization mechanisms under unlabeled conditions, limiting their ability to handle large-scale data and cross-scenario deployment. For resource-constrained platforms such as drones (e.g., complex indoor navigation and warehousing and transportation scenarios lacking GPS positioning information), existing deep learning models are highly complex and cannot balance real-time performance with power consumption constraints. Summary of the Invention

[0005] Aimed at at least in solving one of the technical problems existing in the prior art, the present invention provides a method, apparatus, electronic device and storage medium for estimating the spatial pose of a UAV, thereby improving the accuracy and stability of UAV pose estimation in complex environments.

[0006] One aspect of the present invention provides a method for estimating the spatial pose of a UAV, comprising:

[0007] Acquire RGB and thermal images of the UAV, perform multi-scale preprocessing and feature extraction on the RGB and thermal images to obtain RGB input tensors and thermal image input tensors;

[0008] Feature extraction is performed on the RGB input tensor and the thermal imaging input tensor to obtain RGB feature vectors and thermal feature vectors;

[0009] The RGB feature vector and the hot feature vector are subjected to learnable modal weights and global attention feature fusion to obtain a multimodal fused feature vector;

[0010] Frequency domain transformation is performed based on the RGB feature vector, the thermal feature vector, and the multimodal fusion feature vector to obtain the amplitude spectrum and the phase spectrum. Phase enhancement and inverse Fourier reconstruction are then performed on the amplitude spectrum and the phase spectrum in sequence. Adaptive spatial coding and residual fusion are then performed on the obtained phase-enhanced spatial features to obtain the enhanced spatial feature map.

[0011] The enhanced spatial feature map is sampled, and intensity map construction and box counting are performed on the obtained sampled points to obtain a local fractal descriptor. Training is performed using the local fractal descriptor and the enhanced spatial feature map to obtain a fractal robust embedding vector.

[0012] The enhanced spatial feature map, the fractal robust embedding vector, and the multimodal fusion feature vector are concatenated. The concatenated result is then subjected to bi-branch prediction of position and orientation to obtain the spatial pose estimation result of the UAV.

[0013] According to the aforementioned UAV spatial pose estimation method, RGB images and thermal images of the UAV are acquired, and multi-scale preprocessing and feature extraction are performed on the RGB images and thermal images to obtain RGB input tensors and thermal image input tensors, including:

[0014] Normalize the RGB image, and expand the thermal imaging image to three channels and normalize it;

[0015] The normalized RGB image and the normalized thermal image are spatially aligned and registered using an affine transformation model to obtain the registered RGB image and the registered thermal image.

[0016] The registered RGB image and the registered thermal image are preprocessed using a multi-scale pyramid to obtain a multi-scale input set.

[0017] The multi-scale input set is mapped to the deep network input space to obtain the RGB input tensor and the thermal imaging input tensor.

[0018] According to the UAV spatial pose estimation method, feature extraction is performed on the RGB input tensor and the thermal imaging input tensor to obtain RGB feature vectors and thermal feature vectors, including:

[0019] Spatial features are extracted from the RGB input tensor and the thermal imaging input tensor using depthwise convolution and independent channels;

[0020] The output of depthwise convolution is combined across channels using pointwise convolution to obtain the output of pointwise convolution;

[0021] The outputs of the depthwise convolution and the pointwise convolution are merged and then activated and normalized to obtain the first layer feature map for each scale:

[0022] The first layer feature map is enhanced using a standard residual structure to obtain the second layer feature map;

[0023] The second-layer feature map is sequentially processed by channel compression, global average pooling, attention mapping, and channel weighting to obtain an attention-weighted feature map.

[0024] The attention-weighted feature map is processed sequentially by standard 3×3 convolution, dilated 3×3 convolution and 1×1 convolution, and then convolution compression is performed to obtain the multi-scale fused output feature map.

[0025] The output feature map after multi-scale fusion is processed by global average pooling or average pooling at each scale to obtain a scale vector. The scale vector is then aggregated and normalized between scales to obtain RGB feature vectors and hot feature vectors.

[0026] According to the UAV spatial pose estimation method, a feature fusion process involving learnable modal weights and global attention is performed on the RGB feature vector and the thermal feature vector to obtain a multimodal fused feature vector, including:

[0027] The RGB feature vector and the hot feature vector are concatenated to obtain a concatenated vector.

[0028] The RGB feature vector and the hot feature vector are weighted and fused using learnable modal weights to obtain a weighted fused vector.

[0029] The concatenated vector is weighted element-wise by modal attention weights to obtain an attention-weighted vector;

[0030] The weighted fusion vector and the attention weighted vector are concatenated, and then a unified feature mapping is performed to obtain a multimodal fusion feature vector.

[0031] According to the UAV spatial pose estimation method, frequency domain transformation is performed based on the RGB feature vector, the thermal feature vector, and the multimodal fusion feature vector to obtain the amplitude spectrum and phase spectrum. Phase enhancement and inverse Fourier reconstruction are then performed sequentially on the amplitude spectrum and phase spectrum. Adaptive spatial coding and residual fusion are then performed on the obtained phase-enhanced spatial features to obtain an enhanced spatial feature map, including:

[0032] Employ learnable global weight scalars Furthermore, modal weights are extracted from the multimodal fusion feature vector using a multilayer perceptron or linear layer, resulting in a learnable global weight scalar. for:

[0033] ;

[0034] in, For learnable weights, For learnable bias, This is a multimodal fusion feature vector. For transpose;

[0035] For RGB feature vectors and thermal eigenvectors The fused spatial feature map is obtained by spatial point-by-point weighted fusion and modal weighting. for:

[0036] ;

[0037] ;

[0038] ;

[0039] in, For the scale Transform into Image height, For overscale Transform into Image height, The number of output channels for pointwise convolution; The spatial coordinates of the image, For input channels, It is a feature map in RGB space. This is a spatial feature map for thermal imaging.

[0040] Fusion spatial feature map Each channel undergoes mean-reduction processing to obtain a mean-reduced channel feature set. ,in for:

[0041] ;

[0042] in, The total number of channels; perform a two-dimensional Fourier transform on the mean-free channel features. The amplitude spectrum and phase spectrum are obtained, where the two-dimensional Fourier transform is:

[0043] ;

[0044] in, For offset, For amplitude spectrum, For phase spectrum, For two-dimensional Fourier transform, The imaginary unit;

[0045] Phase enhancement is performed by applying phase modulation to the phase spectrum. The phase modulation method for the phase spectrum is as follows:

[0046] ;

[0047] in, To enhance the phase spectrum, (A) is the amplitude-normalized nonlinear gate function; For frequency domain phase-sensitive weights, and for:

[0048] ;

[0049] ;

[0050] Where λ>0 and κ>0 are learnable or manually adjustable scaling hyperparameters. This indicates median normalization. This refers to the magnitude of the amplitude spectrum; It is the numerical stability constant; Indicates in the channel The entire frequency domain space Within the range, phase gradient magnitude global maximum value Reconstruct the frequency domain complex numbers from the amplitude spectrum and the enhanced phase spectrum, and perform an inverse Fourier transform. spatial features with phase enhancement were obtained. for:

[0051] ;

[0052] ;

[0053] ;

[0054] in, Obtained by stacking in channels , This indicates taking the real part of a complex number. Indicates the first Each channel is in The reconstructed complex spectrum value, The channel corresponding to the enhanced phase spectrum;

[0055] Local descriptors are calculated for the fused spatial feature map and the spatial features enhanced by phase. A spatial attention map is then calculated based on these local descriptors. The calculation method is as follows:

[0056] ;

[0057] Spatial attention map The calculation method is as follows:

[0058] ;

[0059] in, For activation function, Used to output a single-channel heatmap; This is the Sigmoid activation function, used to obtain the weights at each spatial location;

[0060] Based on the spatial characteristics of phase enhancement Spatial attention map and integrated spatial feature map After performing residual fusion and normalization, and activation processing, the enhanced spatial feature map is obtained as follows:

[0061] ;

[0062] in, As a global scale factor, For learnable weights, This is a learnable bias.

[0063] According to the UAV spatial pose estimation method, the enhanced spatial feature map is sampled, and intensity map construction and box counting are performed on the obtained sampled points to obtain a local fractal descriptor. Training is then performed using the local fractal descriptor and the enhanced spatial feature map to obtain a fractal robust embedding vector, including:

[0064] The enhanced spatial feature map is sampled using one of the following methods: fixed window, sliding window, and interest point sampling, to obtain sampling points, where the sampling points are continuous local regions;

[0065] The multi-channel features of a local region are projected into a single-channel intensity map. Box counting statistics are performed on the single-channel intensity map at multiple scales. The box counting results are then fitted using the least squares method to obtain the local fractal dimension, where the scale set of the box counting is... Determine the fractal descriptor based on the statistics of the local region and the local fractal dimension. for:

[0066] ;

[0067] in, For the local fractal dimension, the mean intensity of the local region. Local area intensity standard deviation , This is a single-channel intensity map. This refers to the local window size, and Configurable; The first one obtained by upsampling from the enhanced spatial feature map A local area, in which for:

[0068] ;

[0069] in, Tensors for local regions;

[0070] Extracting local vectors from local regions using a CNN convolutional neural network. , where local vectors The following results were obtained by processing with global average pooling or small convolutions:

[0071] ;

[0072] By using local vectors With fractal descriptors The original concatenated vector is obtained by concatenating the vectors. for:

[0073] ;

[0074] The original splicing vector is projected through the projection head. After mapping and normalization, the local embedding vector z is obtained:

[0075] ;

[0076] in, For projection networks, used to output dimensions and local embedding vectors ;

[0077] Multiple local embedding vectors from the same source are used to perform optimized embedding learning through either self-supervised contrastive learning or triplet training to obtain the trained projection network. These local embedding vectors represent positive sample pairs from the same physical point or target and negative samples from other samples. The self-supervised contrastive learning and triplet training methods employ triplet loss. Or InfoNCE loss :

[0078] ;

[0079] ;

[0080] in, For positive sample embedding, For negative sample embedding, These are local embedding vectors of samples from the same source, where m>0 indicates a boundary. The distance is Euclidean. Indicates cosine similarity, Temperature coefficient;

[0081] After processing the augmented spatial feature map through the trained projection network, a new augmented spatial feature map and a fractal robust embedding vector are obtained.

[0082] According to the UAV spatial pose estimation method, the enhanced spatial feature map, the fractal robust embedding vector, and the multimodal fusion feature vector are concatenated. The concatenated result is then subjected to bi-branch prediction of position and orientation to obtain the UAV spatial pose estimation result, including:

[0083] The new enhanced spatial feature map, fractal robust embedding vector, and multimodal fusion feature vector are concatenated:

[0084] ;

[0085] ;

[0086] ;

[0087] ;

[0088] in, For the new enhanced spatial feature map The dimensionality reduction or dimensionality increase results of linear mapping, Multimodal fusion feature vector The dimensionality reduction or dimensionality increase results of linear mapping, fractal robust embedding vector The dimensionality reduction or dimensionality increase results of linear mapping, To concatenate vectors, For concatenating vectors Dimensions , and For bias terms, , and This is the weight matrix;

[0089] Using lightweight MLP Mapping to the regression feature space yields the intermediate representation vector. :

[0090] ;

[0091] ;

[0092] For intermediate representation vectors Prediction is performed using a location prediction branch and an orientation prediction branch, where the location prediction branch is as follows:

[0093] ;

[0094] Orientation prediction branch is:

[0095] ;

[0096] ;

[0097] in To predict the location, To predict orientation, Here is the weight matrix for the location prediction branch. For the location prediction branch bias term, The weight matrix is ​​for the predicted branch. For the bias term pointing towards the branch, As an intermediate representation, It is an L2 norm;

[0098] The total loss function used when making predictions in the location prediction branch and the orientation prediction branch is... for:

[0099] ;

[0100] The total loss is weighted by adaptive uncertainty. It is a learnable scalar used to represent the observable uncertainty of position and rotation; For position loss, ; Towards loss, ; For time-series smoothing; For consistency regularization, As an intermediate feature, The mean or target embedding of the local embedding vector. For projection head; This is a hyperparameter.

[0101] Another aspect of the present invention provides a spatial pose estimation device for an unmanned aerial vehicle (UAV), comprising:

[0102] The first module is used to acquire RGB images and thermal images of the UAV, perform multi-scale preprocessing and feature extraction on the RGB images and thermal images, and obtain RGB input tensors and thermal image input tensors.

[0103] The second module is used to perform feature extraction on the RGB input tensor and the thermal imaging input tensor to obtain RGB feature vector and thermal feature vector;

[0104] The third module is used to perform feature fusion of learnable modal weights and global attention on the RGB feature vector and the hot feature vector to obtain a multimodal fusion feature vector;

[0105] The fourth module is used to perform frequency domain transformation based on the RGB feature vector, the thermal feature vector and the multimodal fusion feature vector to obtain the amplitude spectrum and the phase spectrum, perform phase enhancement and inverse Fourier reconstruction on the amplitude spectrum and the phase spectrum in sequence, and perform adaptive spatial coding and residual fusion on the obtained phase-enhanced spatial features to obtain the enhanced spatial feature map.

[0106] The fifth module is used to sample the enhanced spatial feature map, perform intensity map construction and box counting on the obtained sampled points to obtain a local fractal descriptor, and perform training using the local fractal descriptor and the enhanced spatial feature map to obtain a fractal robust embedding vector.

[0107] The sixth module is used to concatenate the enhanced spatial feature map, the fractal robust embedding vector, and the multimodal fusion feature vector. The concatenated result is then used to perform a bi-branch prediction of position and orientation to obtain the spatial pose estimation result of the UAV.

[0108] Another aspect of the present invention provides an electronic device, including a processor and a memory;

[0109] The memory is used to store programs;

[0110] The processor executes the program to implement the method as described above.

[0111] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the methods described above.

[0112] The beneficial effects of this invention are as follows: By quantizing the multimodal fusion structure, RGB and thermal imaging complement each other at the energy and structural levels, ensuring the perception stability of UAVs in low-light, foggy, or highly reflective environments; by employing phase transformation-adaptive spatial coding, phase fine-tuning guided by amplitude in the frequency domain is used to achieve physical enhancement of geometric structures such as edges and contours, which is equivalent to enhancing the geometric continuity and structural constraints of the scene in the spatial domain, thereby significantly improving the geometric interpretability and noise resistance of pose estimation; by using a self-supervised fractal invariant feature embedding (S-FIFE) module to represent local structural complexity with fractal dimension, scale-invariant structural embedding learning is achieved under unlabeled conditions, enabling the network to maintain stable feature responses under multi-view and cross-scale conditions. The synergistic effect of these three aspects enables this invention to simultaneously possess high accuracy, strong robustness, and cross-scene adaptability in six-degree-of-freedom pose estimation. Attached Figure Description

[0113] Figure 1 This is a schematic diagram of the UAV spatial pose estimation process according to an embodiment of the present invention.

[0114] Figure 2 This is a schematic diagram of the multimodal input process according to an embodiment of the present invention.

[0115] Figure 3 This is a schematic diagram of the phase transformation-adaptive spatial coding process according to an embodiment of the present invention.

[0116] Figure 4 This is a flowchart of the degree-of-freedom pose regression according to an embodiment of the present invention.

[0117] Figure 5 This is a schematic diagram of a UAV spatial pose estimation device according to an embodiment of the present invention. Detailed Implementation

[0118] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings. Throughout the description, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions. In the following description, suffixes such as "module," "part," or "unit" used to denote elements are used only for the purpose of illustrative purposes and have no specific meaning in themselves. Therefore, "module," "part," or "unit" can be used interchangeably. Terms such as "first," "second," etc., are used only to distinguish technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the sequential relationship of the indicated technical features. In the following description, the consecutive reference numerals for method steps are for ease of review and understanding. Adjusting the implementation order of steps, in conjunction with the overall technical solution of the present invention and the logical relationship between the various steps, will not affect the technical effect achieved by the technical solution of the present invention. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0119] refer to Figure 1 , Figure 1 This is a schematic flowchart of the UAV spatial pose estimation method according to an embodiment of the present invention, which includes, but is not limited to, steps S100~S600:

[0120] S100 acquires RGB and thermal images of the UAV, performs multi-scale preprocessing and feature extraction on the RGB and thermal images, and obtains RGB input tensors and thermal image input tensors.

[0121] In some embodiments, reference Figure 2 The multimodal input process diagram shown includes the following steps:

[0122] (1) Normalize the RGB image and expand the thermal image to three channels and normalize it.

[0123] In some embodiments, RGB image acquisition and input are performed by acquiring color images using a visible light camera mounted on a drone. Let the input RGB image be:

[0124] ;

[0125] in, Indicates the image height. The value represents the image width, and 3 represents the three RGB color channels (R, G, and B respectively). The RGB image is normalized to maintain data consistency under different lighting conditions.

[0126] In some embodiments, thermal imaging image acquisition and extension are performed by acquiring single-channel images output by a thermal imaging sensor mounted on the UAV via an infrared camera:

[0127] ;

[0128] To align with RGB images, the single-channel image is expanded to three channels:

[0129] ;

[0130] The thermal imaging was then normalized.

[0131] (2) The normalized RGB image and the normalized thermal image are spatially aligned and registered using an affine transformation model to obtain the registered RGB image and thermal image.

[0132] In some embodiments, spatial alignment and registration are performed by inputting a normalized RGB image and a thermal image: To ensure the consistency of the two modes in spatial location, image registration is required, using an affine transformation model:

[0133] ;

[0134] in, , , , Indicates the rotation / scaling parameters. Indicates the translation parameters, outputs the registered RGB image. With thermal imaging images , The coordinates of the thermal imaging image to be registered. The coordinates are for the reference thermal imaging image.

[0135] (3) The registered RGB image and the thermal image are preprocessed using a multi-scale pyramid to obtain a multi-scale input set.

[0136] In some embodiments, multi-scale pyramid processing is employed to obtain image representations at different scales, resulting in a multi-scale input set:

[0137] ;

[0138] in, Indicates the scaling ratio.

[0139] In some embodiments, different scaling ratios are used to construct a multi-layered perception system that covers the global structure, local features and edge details, thereby achieving coverage of features at different resolutions.

[0140] (4) Map the multi-scale input set to the deep network input space to obtain the RGB input tensor and the thermal imaging input tensor.

[0141] In some embodiments, inputs from different modalities are uniformly mapped to the input space of the deep network, using a normalized tensor form:

[0142] ;

[0143] The resulting input tensor, which serves as the feature extraction module, is the final multimodal form, providing semantic and structural priors for subsequent lightweight feature extraction and phase enhancement.

[0144] , ;

[0145] Multimodal input provides a basic input for fusing different sensing bands, RGB images provide texture and color information, and thermal imaging supplements the thermal radiation features that are not sensitive to light. The two work together to effectively solve the problem of insufficient perception in low light, occlusion and cross-scale scenes. In night, haze and strong reflective conditions, the solution of the present invention can significantly improve scene perception and improve feature extraction accuracy.

[0146] S200 performs feature extraction on the RGB input tensor and the thermal imaging input tensor to obtain the RGB feature vector and the thermal feature vector.

[0147] In some embodiments, the following preprocessing is further included before feature extraction of the RGB input tensor and the thermal imaging input tensor:

[0148] Input RGB image tensor: ;

[0149] Thermal imaging tensor (extended to three channels): ;

[0150] in Indicates the first The scale tensor, for each scale Process separately (if only a single scale is used, take...) (That's all), and then normalization is performed.

[0151] In some embodiments, the feature extraction process for RGB input tensors and thermal imaging input tensors includes:

[0152] (1) Spatial features are extracted from the RGB input tensor and the thermal imaging input tensor by using depth convolution and each channel independently.

[0153] In some embodiments, Depthwise-Separable convolution is used to achieve intramodal spatial feature extraction and cross-channel fusion, reducing computational cost while maintaining feature expressive power. The Depthwise-Separable convolution is as follows:

[0154] ;

[0155] The output of a depthwise convolution represents the input channel. and location The value; For depthwise convolution kernels, it means that in the input channel and offset The weights are applied to each channel individually, reducing the number of parameters (compared to standard convolution); where... This represents the offset of the convolution kernel in the height direction (Y-axis). The offset of the convolution kernel in the width direction (X-axis); position. Specifically, spatial location refers to the position of the element in the output feature map. line, number The column position, when involving operations on feature maps, All of these convey the same meaning, and will not be elaborated upon below.

[0156] (2) The output of the depthwise convolution is obtained by performing a cross-channel linear combination of pointwise convolution.

[0157] In some embodiments, pointwise convolution (1×1, cross-channel linear combination) is:

[0158] ;

[0159] For pointwise convolution output, it means at output channel k, position The value is used to fuse the channel information after depthwise convolution and adjust the number of channels; The weights for pointwise convolution represent the weights from the input channels. To output channel This is to achieve weighted combination between channels.

[0160] (3) The outputs of depthwise convolution and pointwise convolution are merged and activation and normalization are performed to obtain the first layer feature map of each scale.

[0161] In some embodiments, merging and performing activation and normalization is as follows:

[0162] ;

[0163] ;

[0164] in The activation function is ; the number of convolution output channels is . Since stride=2, the size becomes Finally, the first layer feature maps of the RGB mode and the thermal imaging mode at the s-th scale are output as follows:

[0165] ;

[0166] ;

[0167] For example, number of channels It has a capacity of 64 and is learnable.

[0168] (4) The first layer feature map is enhanced by the standard residual structure to obtain the second layer feature map.

[0169] In some embodiments, lightweight residual block enhancements include:

[0170] enter , Each modality is processed through residual blocks and standard residual structures to prevent gradient vanishing, improve expressive power, and maintain lightweight operation. Finally, the second-layer feature maps of the RGB modality and the thermal imaging modality at the s-th scale are output. and .

[0171] ;

[0172] ;

[0173] in It consists of two layers of Depthwise-separable convolution + BN + ReLU, and is used to extract non-linear feature transformations.

[0174] (5) Perform channel compression, global average pooling, attention mapping and channel weighting on the second layer feature map in sequence to obtain the attention-weighted feature map.

[0175] In some embodiments, the channel compression and channel attention (lightweight SE) steps are as follows:

[0176] enter , (size is) First, perform 1×1 convolutional dimensionality reduction (channel compression), then perform lightweight channel attention (SE-like). Taking the RGB of the second layer feature map as an example:

[0177] Channel compression (reduced to) ):

[0178] ;

[0179] in The 1×1 convolution weights represent channel compression and are used to define the dimensionality reduction transformation. The processing method for thermal imaging is the same as that for RGB.

[0180] Global average pooling (GAP) yields channel descriptors (channel vectors):

[0181] ;

[0182] For example, number of channels It has 128 and is learnable.

[0183] Attention mapping (lightweight MLP, dimensionality reduction and then dimensionality increase) generates a weight vector s, including and The details are as follows:

[0184] ;

[0185] in The weight matrix of the MLP ( Dimensional reduction (Upward dimension).

[0186] Finally, the feature maps are weighted by channel to obtain the attention-weighted maps:

[0187] ;

[0188] Output: Attention-weighted feature map:

[0189] ;

[0190] (6) After processing the attention-weighted feature map with standard 3×3 convolution, dilated 3×3 convolution and 1×1 convolution in sequence, convolution compression is then performed to obtain the output feature map after multi-scale fusion.

[0191] In some embodiments, input and To increase the local receptive field without significantly increasing the parameters, parallel small kernels and dilated convolution are used, followed by channel stitching. The processing steps for thermal imaging images are as follows (the processing for RGB images is the same):

[0192] Standard 3×3 convolution, ;

[0193] Dilated 3×3 convolution (dilated to 2). ;

[0194] 1×1 convolution (preserving fine granularity). ;

[0195] After splicing, compress using a 1×1 convolution:

[0196] ;

[0197] Finally, the multi-scale fused feature map is output:

[0198] ;

[0199] in, This is a concatenation function.

[0200] (7) Perform global average pooling or average pooling on the output feature map after multi-scale fusion at each scale to obtain the scale vector. Perform inter-scale aggregation on the scale vector and then perform normalization to obtain the RGB feature vector and hot feature vector.

[0201] In some embodiments, the global aggregation and modal feature vector output includes:

[0202] enter , The process involves first performing GAP (or average pooling) on ​​each scale to obtain a scale vector, then aggregating the scales (concatenating or weighted summing), and finally normalizing to obtain a fixed-length final mode vector. The steps are as follows:

[0203] The value of the k-th dimension of a single-scale GAP vector is:

[0204] ;

[0205] Multi-scale aggregation (concatenated into vectors):

[0206] ;

[0207] ;

[0208] in = The number of scales, in order to fuse multi-scale information, in some embodiments, Number of output channels The number is 256 and it is learnable.

[0209] The final mode vector obtained by linear transformation and normalization is:

[0210] ;

[0211] Where D is the expected feature dimension of the regression / fusion module. and The weight matrix and bias of the fully connected layer are used for linear transformation to adjust the aggregated vector to the target dimension; the dimension is transformed through the fully connected layer (FC) and normalized with LayerNorm to ensure vector stability.

[0212] Finally, the RGB modal feature vector and the thermal imaging modal feature vector are output:

[0213] , ;

[0214] The embodiments of this invention construct the foundational energy spectrum for subsequent frequency domain analysis, achieving efficient encoding of intra-modal features and providing a structurally clear representation for subsequent fusion. Unlike traditional convolutional feature extraction, this invention employs a lightweight multi-scale-attention joint structure, providing real-time performance for UAV embedded platforms. These features will serve as inputs to the multimodal fusion and phase transformation modules, enabling dynamic control of global semantics.

[0215] S300 performs feature fusion with learnable modal weights and global attention on RGB feature vectors and hot feature vectors to obtain multimodal fused feature vectors.

[0216] It is understood that this embodiment of the invention effectively fuses the global features of RGB and thermal imaging modalities. First, a unified feature representation is obtained through normalization and stitching. Then, a weighting mechanism and an attention mechanism are combined to adaptively adjust the contribution ratio of different modalities in the fusion process. The final output is a global fusion vector. It not only integrates multimodal semantic information, but also retains key discriminative features in the scene, providing global guidance for subsequent phase transformation and spatial enhancement.

[0217] In some embodiments, the feature fusion step includes:

[0218] (1) The RGB feature vector and the hot feature vector are concatenated to obtain the concatenated vector.

[0219] RGB modal feature vector Thermal imaging modal feature vector The vectors are standardized using L2 or mean-variance methods to make the numerical distributions of the two modes comparable. The feature vectors of the two modes are then concatenated to obtain a 2D vector. This concatenation process ensures that the original feature information of both modes is preserved, resulting in the concatenated vector. .

[0220] (2) The RGB feature vector and the hot feature vector are weighted and fused using learnable modal weights to obtain a weighted fused vector.

[0221] In some embodiments, input To avoid any one mode becoming overly dominant, learnable mode weight parameters are set. It is used to control the ratio of RGB to thermal imaging features.

[0222] ;

[0223] in It can be obtained through training or custom settings, or it can be set manually (such as initial selection). =0.5).

[0224] (3) The splicing vector is weighted element by element by modal attention weights to obtain the attention weighted vector.

[0225] In some embodiments, attention mechanism enhancement includes:

[0226] enter Modal attention weights are generated using a lightweight attention module (MLP+Sigmoid), and the concatenated vector is weighted element-wise according to these weights, then the attention-weighted vector is output. The weights of each dimension (or each group of dimensions) in the fusion vector are dynamically allocated to highlight useful features in the current environment and suppress noise or failure modes.

[0227] (4) The weighted fusion vector and the attention weighted vector are concatenated and then a unified feature mapping is performed to obtain the multimodal fusion feature vector.

[0228] In some embodiments, the unified feature mapping and output include:

[0229] Will The results of the two fusion methods are concatenated and then mapped back to a unified dimension D. This integrates and maps the information obtained from different fusion strategies (linear weighting, attention weighting, etc.) to a unified dimension D, facilitating its use in subsequent modules. The resulting fusion feature vector is then obtained. The final fused features are subjected to LayerNorm or L2 normalization to ensure the stability of subsequent inputs, resulting in a global feature vector fused from multimodal data. This serves as a condition for subsequent modal weight calculations.

[0230] This invention unifies the multimodal information of the perception layer through global weights, providing scene-level control parameters. The subsequent phase enhancement module uses... The enhancement amplitude is controlled to form a semantic-geometric collaborative mechanism: high weight is given to thermal images at night and high weight is given to RGB images during the day, achieving adaptive information flow control. Unlike traditional simple stitching and fusion, this module introduces learnable modal weights and global attention to ensure dynamic selective response to different modalities in the subsequent frequency domain enhancement stage.

[0231] S400 performs frequency domain transformation based on RGB feature vectors, thermal feature vectors, and multimodal fusion feature vectors to obtain amplitude spectrum and phase spectrum. Phase enhancement and inverse Fourier reconstruction are then performed on the amplitude spectrum and phase spectrum in sequence. The obtained phase-enhanced spatial features are then subjected to adaptive spatial coding and residual fusion to obtain an enhanced spatial feature map.

[0232] It is understood that this invention addresses the problem of low contrast and weak structural information in feature maps acquired by UAVs in complex environments by proposing a method combining frequency domain phase enhancement and spatial adaptive coding. The spatial feature map is obtained using lightweight feature extraction. and As input, in the global vector Modal weighted fusion is performed under guidance, and the amplitude and phase are decomposed through Fourier transform. The phase is enhanced and then inversely transformed back to the spatial domain. Subsequently, an adaptive spatial attention mechanism is introduced to highlight regions useful for localization and suppress irrelevant background, finally outputting an enhanced spatial feature map. This provides more robust input for embedded learning and pose regression.

[0233] In some embodiments, reference Figure 3 The schematic diagram of the phase transform-adaptive spatial coding process shown includes:

[0234] (1) Use learnable global weight scalars Furthermore, modal weights are extracted from the multimodal fusion feature vector using a multilayer perceptron or linear layer, resulting in a learnable global weight scalar. for:

[0235] ;

[0236] in, For learnable weights, For learnable bias, This is a multimodal fusion feature vector;

[0237] The fused spatial feature map is obtained by spatially weighting and fusing the RGB feature vectors and hot feature vectors point by point. for:

[0238] ;

[0239] in, The spatial coordinates of the image, For input channels, It is a feature map in RGB space. This is a thermal imaging spatial feature map, where Compared with the above embodiments The meaning is consistent and will not be elaborated further.

[0240] Fusion spatial feature map Each channel undergoes mean-reduction processing to obtain a mean-reduced channel feature set. ,in for:

[0241] ;

[0242] in, Spatial location;

[0243] (2) Perform a two-dimensional Fourier transform on the mean-removed channel features. The amplitude spectrum and phase spectrum are obtained, where the two-dimensional Fourier transform is:

[0244] ;

[0245] in, For offset, For amplitude spectrum, Phase spectrum;

[0246] (3) Phase enhancement is performed on the phase spectrum using phase modulation. The phase modulation method of the phase spectrum is as follows:

[0247] ;

[0248] in, To enhance the phase spectrum, (A) is the amplitude-normalized nonlinear gate function;

[0249] in (A) is the amplitude-normalized nonlinear gated function, for example

[0250] ;

[0251] (A) Nonlinearly compress / amplify the ratio of amplitude to median;

[0252] For frequency domain phase-sensitive weights, and for:

[0253] ;

[0254] λ>0, κ>0 are learnable or manually adjustable scaling hyperparameters; It is the numerical stability constant. Usually set to Small values ​​of the order of magnitude are used to prevent division by zero errors; among which, Indicates the current channel The entire frequency domain space Within the range, phase gradient magnitude The global maximum value, i.e.:

[0255] ;

[0256] This global maximum value is used to normalize the phase gradient at all locations within the current channel, ensuring... The range of values ​​is within Within the interval, this enables enhanced focusing on regions with significant phase gradients.

[0257] Specifically, Calculated using the central difference method in the frequency domain: ;

[0258] ;

[0259] The gradient magnitude is ;

[0260] This weighting ensures that phase enhancement primarily acts on frequencies with dramatic phase changes, typically corresponding to geometric structures such as edges and contours in the image. It should be noted that... (A) The direction of enhancement intensity is determined based on the amplitude; that is, for medium amplitudes, the phase is appropriately enhanced to highlight the texture; for extremely small amplitudes (which are very likely noise), the enhancement is suppressed. Median normalization is used to resist extreme spectral values. The enhancement is concentrated on frequencies with large phase gradients (corresponding to edges and geometric abrupt changes), thereby more accurately recovering geometric information. This embodiment of the invention performs amplitude-to-phase information transfer in the frequency domain, thus selectively using energy information to correct / amplify geometric cues, rather than blindly amplifying phase noise.

[0261] (4) Reconstruct the frequency domain complex numbers from the amplitude spectrum and the enhanced phase spectrum and perform the inverse Fourier transform. spatial features with phase enhancement were obtained. for:

[0262] ;

[0263] ;

[0264] ;

[0265] Stacked through channels Stacking , This indicates taking the real part of a complex number. Indicates the first Each channel is in The reconstructed complex spectrum value, This refers to the channel corresponding to the enhanced phase spectrum.

[0266] (5) Calculate local descriptors for the fused spatial feature map and the spatial features enhanced by phase, and calculate the spatial attention map based on the local descriptors, wherein the local descriptors are... The calculation method is as follows:

[0267] ;

[0268] Spatial attention map The calculation method is as follows:

[0269] ;

[0270] in, Used to output single-channel thermal images; This is the Sigmoid activation function, used to obtain the weights at each spatial location;

[0271] Based on the spatial characteristics of phase enhancement Spatial attention map and integrated spatial feature map Residual fusion and normalization are performed, followed by activation processing, to obtain an enhanced spatial feature map. The process is represented as:

[0272] ;

[0273] in, As a global scale factor, For learnable weights, This is a learnable bias.

[0274] S500 samples the augmented spatial feature map, performs intensity map construction and box counting on the obtained sampled points to obtain a local fractal descriptor, and performs training using the local fractal descriptor and the augmented spatial feature map to obtain a fractal robust embedding vector.

[0275] In some embodiments, a self-supervised fractal invariant feature embedding (S-FIFE) approach is employed to leverage the scale-invariant properties of fractal geometry and a self-supervised learning mechanism to enhance the robustness of features across viewpoints and scales. The enhanced spatial feature map is output using phase transform-adaptive spatial encoding. As input, local regions are first extracted and fractal descriptors are calculated, then concatenated with local convolutional features to form a composite representation. After optimization through self-supervised contrastive learning, the output is an embedding vector that contains both local geometric structure and scale robustness. It can also be further used to form a global representation to assist in pose regression and cross-view matching.

[0276] In some embodiments, the process of self-supervised fractal invariant feature embedding includes:

[0277] (1) The enhanced spatial feature map is sampled using one of the following methods: fixed window, sliding window, and point of interest sampling, to obtain sampling points, where the sampling points are continuous local regions.

[0278] In some embodiments, the enhanced spatial feature map output from the phase transform-adaptive spatial coding... The above uses a fixed window p×p (or key point center) to sample several local continuous local regions (patch), dividing the whole image into local units, which facilitates the calculation of fractal features and the construction of comparable local embeddings (improving cross-view and cross-scale robustness). Sampling can be a sliding window (step size can be p / 2) or based on interest point sampling.

[0279] (2) Project the multi-channel features of the local area into a single-channel intensity map, and use box counting statistics on the single-channel intensity map at multiple scales. Use the least squares method to fit the box counting statistics results to obtain the local fractal dimension.

[0280] In some embodiments, the multi-channel features of a local region are projected as a single-channel intensity map. (Usually channel average), at multiple scales Box counting statistics Let scale set The box dimensions are 2, 4, 8, 16, and 32 pixels respectively. The local fractal dimension is estimated using log-linear regression. It quantitatively measures local self-similarity / complexity, and fractal dimension is naturally robust to minor changes in scale and perspective, serving as a stable local geometric descriptor to supplement CNN features.

[0281] ;

[0282] For each scale calculate ,in The slope (fractal dimension) is obtained by least-squares fitting. ).

[0283] It should be noted that, Characterizing the complexity of local structures, when ≈2 indicates a flat texture, when A larger value indicates a geometrically structured or edge-dense region. In drone images, this fractal feature stably characterizes the geometric complexity of a scene and is insensitive to changes in viewpoint.

[0284] (3) Determine the fractal descriptor based on the statistics of the local region and the local fractal dimension. for:

[0285] ;

[0286] in, For the local fractal dimension, the mean intensity of the local region. Local area intensity standard deviation , This is a single-channel intensity map. The size of the local window, and Configurable; The first one obtained by upsampling from the enhanced spatial feature map A local region; including a single-channel intensity map for:

[0287] ;

[0288] in, It is a tensor for a local region.

[0289] (4) Extracting local vectors from local regions using CNN convolutional neural networks The local vectors are obtained by processing them using global average pooling or small convolutions:

[0290] ;

[0291] By using local vectors With fractal descriptors The original concatenated vector is obtained by concatenating the vectors. for:

[0292] ;

[0293] The original splicing vector is projected through the projection head. After mapping and normalization, the local embedding vector z is obtained:

[0294] ;

[0295] in, As a dimension, It is a projection network.

[0296] It should be noted that projection networks (e.g., two layers of FC + ReLU + FC), output dimension And output the local embedding vector By combining fractal structure information with the local semantics of CNN into the same embedding space, the embedding has both semantic discrimination ability and cross-scale and cross-view stability.

[0297] (5) Multiple local embedding vectors from the same source are optimized using either self-supervised contrastive learning or triplet training to obtain the trained projection network. The multiple local embedding vectors from the same source are used to represent positive sample pairs of the same physical point or the same target and negative samples of other samples. The self-supervised contrastive learning and triplet training use triplet loss. Or InfoNCE loss :

[0298] ;

[0299] ;

[0300] in, For positive sample embedding, For negative sample embedding, These are local embedding vectors of samples from the same source, where m>0 indicates a boundary. The distance is Euclidean. Indicates cosine similarity, This is a temperature coefficient, ranging from 0.1 to 1, used to describe the sensitivity of the loss function to the sample. Specifically, when... When the value is small (e.g.) The loss function is more sensitive to difficult negative samples; the model will work harder to bring the anchor point closer to positive samples while pushing it further away from similar negative samples. This allows the model to learn sharper discriminative boundaries. When the value is large, the loss function responds relatively smoothly to all negative samples, and the penalty is relatively mild, which allows the model to learn smoother feature representations.

[0301] After processing the augmented spatial feature map through the trained projection network, a new augmented spatial feature map and a fractal robust embedding vector are obtained.

[0302] In some embodiments, the fractal dimension of the same structural region remains unchanged under different scales or perturbations, so the embeddings should maintain high similarity; however, the fractal dimensions of different scenes or geometric regions differ greatly, so the embeddings should be separated. This self-supervised constraint enables the network to automatically learn structural invariance.

[0303] The projection obtained after final training in the embodiment of the present invention (and encoder parameters) make the embedding z compact and distinguishable across viewpoints / scales, so that different viewpoint patches of the same physical target can be brought together in the embedding space without manual annotation, while the embeddings of different targets are separated, thereby obtaining robust matching and retrieval capabilities.

[0304] The S600 concatenates the enhanced spatial feature map, fractal robust embedding vector, and multimodal fusion feature vector. The concatenated result is then used to perform a two-branch prediction of position and orientation to obtain the spatial pose estimation result of the UAV.

[0305] In some embodiments, the global feature vector fused from multimodal sources Phase transform-adaptive spatial coding output enhanced global feature map And the fractal robust embedding vector obtained by self-supervised fractal invariant feature embedding. By performing vector concatenation and feature projection on the aforementioned features, a joint representation is established. A lightweight regression network is then used for bi-branch prediction of position and orientation, with orientation represented by unit quaternions to ensure the correctness of rotation constraints. During the training phase, a weighted combination of position loss, rotation loss, and time-series smoothing constraints is introduced to achieve high robustness of the model across scenarios, modalities, and time periods. In the inference phase, confidence estimation and filtering strategies are further combined to output stable pose results that can be directly used for flight control and navigation tasks. This invention not only ensures the positioning accuracy and robustness of the unmanned system in complex environments but also balances real-time performance and computational efficiency, making it suitable for various embedded unmanned platforms.

[0306] In some embodiments, reference Figure 4 The flowchart shown below illustrates the pose regression process for degrees of freedom, which includes:

[0307] (1) The new enhanced spatial feature map, fractal robust embedding vector, and multimodal fusion feature vector are concatenated:

[0308] ;

[0309] ;

[0310] ;

[0311] ;

[0312] in, , and For the corresponding new enhanced spatial feature map Multimodal fusion feature vector and fractal robust embedding vectors The dimensionality reduction or dimensionality increase results of linear mapping, To concatenate vectors, For concatenating vectors Dimensions , and For bias terms, , and This is the weight matrix;

[0313] (2) Using a lightweight MLP Mapping to the regression feature space yields the intermediate representation vector. :

[0314] ;

[0315] ;

[0316] in, This is the output vector of the first hidden layer in an MLP. This refers to the bias term of the first fully connected layer. This is the bias term for the second fully connected layer.

[0317] (3) Input the predicted location Predicting direction with truth position Truth Orientation .

[0318] Basic term, position loss (squared error): .

[0319] Orientation loss (L2): (For unit quaternions).

[0320] For intermediate representation vectors Prediction is performed using a location prediction branch and an orientation prediction branch, where the location prediction branch is as follows:

[0321] ;

[0322] Orientation prediction branch is:

[0323] ;

[0324] ;

[0325] in To predict the location, To predict orientation, Here is the weight matrix for the location prediction branch. For the location prediction branch bias term, The weight matrix is ​​for the predicted branch. For the bias term pointing towards the branch, As an intermediate representation, It is an L2 norm;

[0326] In some embodiments, branched prediction can focus on different statistical characteristics of position and orientation (position is usually affected by scale and depth, while orientation is affected by geometry) and ensure that the rotation prediction values ​​are stable (quaternion normalization avoids angle jumps).

[0327] (4) The total loss function used when making predictions for the location prediction branch and the orientation prediction branch. for:

[0328] ;

[0329] The total loss is weighted by adaptive uncertainty. It is a learnable scalar used to represent the observable uncertainty of position and rotation; For time series smoothing, The difference between the predicted positions of two adjacent frames. The difference in actual position between two adjacent frames; For consistency regularization, As an intermediate feature, The mean or target embedding of the local embedding vector. For projection head, This is a hyperparameter.

[0330] The embodiments of the present invention use an adaptive weighted loss with uncertainty (referencing the idea of ​​uncertainty learning, but presenting a directly implementable form), and add time smoothing and fractal consistency regularization terms to enhance cross-perspective robustness.

[0331] Figure 5 This is a schematic diagram of a UAV spatial pose estimation device according to an embodiment of the present invention. The device includes a first module 510, a second module 520, a third module 530, a fourth module 540, a fifth module 550, and a sixth module 560.

[0332] The system comprises four modules: First, acquiring RGB and thermal images of the UAV, performing multi-scale preprocessing and feature extraction to obtain RGB and thermal input tensors; second, performing feature extraction on the RGB and thermal input tensors to obtain RGB and thermal feature vectors; third, performing learnable modal weights and global attention feature fusion on the RGB and thermal feature vectors to obtain multimodal fusion feature vectors; and fourth, performing frequency domain transformation on the RGB, thermal, and multimodal fusion feature vectors to obtain amplitude and phase spectra, and adjusting the amplitude... The spectrum and phase spectrum are sequentially subjected to phase enhancement and inverse Fourier reconstruction. The spatial features obtained by phase enhancement are then subjected to adaptive spatial coding and residual fusion to obtain an enhanced spatial feature map. The fifth module is used to sample the enhanced spatial feature map, and the intensity map is constructed and box counting is performed on the obtained sampled points to obtain a local fractal descriptor. The local fractal descriptor and the enhanced spatial feature map are then trained to obtain a fractal robust embedding vector. The sixth module is used to concatenate the enhanced spatial feature map, the fractal robust embedding vector and the multimodal fusion feature vector. The concatenated result is then subjected to bi-branch prediction of position and orientation to obtain the spatial pose estimation result of the UAV.

[0333] Exemplarily, with the cooperation of the first, second, third, fourth, fifth, and sixth modules in the device, the embodiment device can implement any of the aforementioned UAV spatial pose estimation methods, namely, acquiring RGB images and thermal images of the UAV; performing multi-scale preprocessing and feature extraction on the RGB images and thermal images to obtain RGB input tensors and thermal imaging input tensors; performing feature extraction on the RGB input tensors and thermal imaging input tensors to obtain RGB feature vectors and thermal feature vectors; performing feature fusion with learnable modal weights and global attention on the RGB feature vectors and thermal feature vectors to obtain multi-modal fusion feature vectors; and performing feature fusion based on the RGB feature vectors, thermal feature vectors, and multi-modal fusion feature vectors. The modal fusion feature vectors are subjected to frequency domain transformation to obtain amplitude and phase spectra. Phase enhancement and inverse Fourier reconstruction are then performed on the amplitude and phase spectra sequentially. The resulting phase-enhanced spatial features are subjected to adaptive spatial coding and residual fusion to obtain an enhanced spatial feature map. The enhanced spatial feature map is sampled, and intensity map construction and box counting are performed on the sampled points to obtain local fractal descriptors. Training is performed using the local fractal descriptors and the enhanced spatial feature map to obtain fractal robust embedding vectors. The enhanced spatial feature map, fractal robust embedding vectors, and multimodal fusion feature vectors are concatenated. The concatenated result is then subjected to bi-branch prediction of position and orientation to obtain the spatial pose estimation result of the UAV. The beneficial effects of this invention are as follows: By quantizing the multimodal fusion structure, RGB and thermal imaging complement each other at the energy and structural levels, ensuring the perception stability of UAVs in low-light, foggy, or highly reflective environments; by employing phase transformation-adaptive spatial coding, phase fine-tuning guided by amplitude in the frequency domain is used to achieve physical enhancement of geometric structures such as edges and contours, which is equivalent to enhancing the geometric continuity and structural constraints of the scene in the spatial domain, thereby significantly improving the geometric interpretability and noise resistance of pose estimation; by using a self-supervised fractal invariant feature embedding (S-FIFE) module to represent local structural complexity with fractal dimension, scale-invariant structural embedding learning is achieved under unlabeled conditions, enabling the network to maintain stable feature responses under multi-view and cross-scale conditions. The synergistic effect of these three aspects enables this invention to simultaneously possess high accuracy, strong robustness, and cross-scene adaptability in six-degree-of-freedom pose estimation.

[0334] This invention also provides an electronic device, which includes a processor and a memory;

[0335] The memory stores the program;

[0336] The processor executes a program to perform the aforementioned UAV spatial pose estimation method; the electronic device has the function of carrying and running the UAV spatial pose estimation software system provided in the embodiments of the present invention, such as a personal computer, minicomputer, main frame, workstation, network or distributed computing environment, standalone or integrated computer platform, or communicating with charged particle tools or other imaging devices, etc.

[0337] This invention also provides a computer-readable storage medium storing a program that is executed by a processor to implement the UAV spatial pose estimation method described above.

[0338] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented in the embodiments of this invention. Alternative embodiments are contemplated, in which the order of various operations is changed and sub-operations described as part of a larger operation are executed independently.

[0339] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned UAV spatial pose estimation method.

[0340] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, considering the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed in the embodiments of the invention, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0341] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0342] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can include, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0343] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0344] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0345] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0346] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0347] The above is a detailed description of the preferred embodiments of the present invention, but the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A method for estimating the spatial pose of a UAV, characterized in that, include: Acquire RGB and thermal images of the UAV, perform multi-scale preprocessing and feature extraction on the RGB and thermal images to obtain RGB input tensors and thermal image input tensors; Feature extraction is performed on the RGB input tensor and the thermal imaging input tensor to obtain RGB feature vectors and thermal feature vectors; The RGB feature vector and the hot feature vector are subjected to learnable modal weights and global attention feature fusion to obtain a multimodal fused feature vector; Frequency domain transformation is performed based on the RGB feature vector, the thermal feature vector, and the multimodal fusion feature vector to obtain the amplitude spectrum and the phase spectrum. Phase enhancement and inverse Fourier reconstruction are then performed on the amplitude spectrum and the phase spectrum in sequence. Adaptive spatial coding and residual fusion are then performed on the obtained phase-enhanced spatial features to obtain the enhanced spatial feature map. The enhanced spatial feature map is sampled, and intensity map construction and box counting are performed on the obtained sampled points to obtain a local fractal descriptor. Training is performed using the local fractal descriptor and the enhanced spatial feature map to obtain a fractal robust embedding vector. The enhanced spatial feature map, the fractal robust embedding vector, and the multimodal fusion feature vector are concatenated. The concatenated result is then subjected to a bi-branch prediction of position and orientation to obtain the spatial pose estimation result of the UAV. The process involves performing frequency domain transformation based on the RGB feature vector, the thermal feature vector, and the multimodal fusion feature vector to obtain the amplitude spectrum and phase spectrum. Phase enhancement and inverse Fourier reconstruction are then performed sequentially on the amplitude spectrum and phase spectrum. Adaptive spatial coding and residual fusion are then applied to the obtained phase-enhanced spatial features to obtain an enhanced spatial feature map, including: Employ learnable global weight scalars Furthermore, modal weights are extracted from the multimodal fusion feature vector using a multilayer perceptron or linear layer, resulting in a learnable global weight scalar. for: in, For learnable weights, For learnable bias, This is a multimodal fusion feature vector. For transpose; For RGB feature vectors and thermal eigenvectors The fused spatial feature map is obtained by spatial point-by-point weighted fusion and modal weighting. for: in, For the scale Transform into Image height, For overscale Transform into Image width, The number of output channels for pointwise convolution; The spatial coordinates of the image, For input channels, It is a feature map in RGB space. This is a spatial feature map for thermal imaging. Fusion spatial feature map Each channel undergoes mean-reduction processing to obtain a mean-reduced channel feature set. ,in for: in, This represents the total number of channels; Perform a two-dimensional Fourier transform on the mean-removed channel features The amplitude spectrum and phase spectrum are obtained, where the two-dimensional Fourier transform is: in, For offset, For amplitude spectrum, For phase spectrum, The imaginary unit; Phase enhancement is performed by applying phase modulation to the phase spectrum. The phase modulation method for the phase spectrum is as follows: in, To enhance the phase spectrum, (A) is the amplitude-normalized nonlinear gate function; For frequency domain phase-sensitive weights, and for: Where λ>0 and κ>0 are learnable or manually adjustable scaling hyperparameters. This indicates median normalization. This refers to the magnitude of the amplitude spectrum; It is the numerical stability constant; Indicates in the channel The entire frequency domain space Within the range, phase gradient magnitude global maximum value Reconstruct the frequency domain complex numbers from the amplitude spectrum and the enhanced phase spectrum, and perform an inverse Fourier transform. spatial features with phase enhancement were obtained. for: in, Obtained by stacking in channels , This indicates taking the real part of a complex number. Indicates the first Each channel is in The reconstructed complex spectrum value, The channel corresponding to the enhanced phase spectrum; Local descriptors are calculated for the fused spatial feature map and the spatial features enhanced by phase. A spatial attention map is then calculated based on these local descriptors. The calculation method is as follows: Spatial attention map The calculation method is as follows: in, For activation function, Used to output a single-channel heatmap; This is the Sigmoid activation function, used to obtain the weights at each spatial location; Based on the spatial characteristics of phase enhancement Spatial attention map and integrated spatial feature map After performing residual fusion and normalization, and activation processing, the enhanced spatial feature map is obtained as follows: in, As a global scale factor, For learnable weights, For learnable bias; The process involves sampling the enhanced spatial feature map, constructing an intensity map and performing box counting on the sampled points to obtain a local fractal descriptor, and then training the local fractal descriptor and the enhanced spatial feature map to obtain a fractal robust embedding vector, including: The enhanced spatial feature map is sampled using one of the following methods: fixed window, sliding window, and interest point sampling, to obtain sampling points, where the sampling points are continuous local regions; The multi-channel features of a local region are projected into a single-channel intensity map. Box counting statistics are performed on the single-channel intensity map at multiple scales. The box counting results are then fitted using the least squares method to obtain the local fractal dimension, where the scale set of the box counting is... Determine the fractal descriptor based on the statistics of the local region and the local fractal dimension. for: in, For the local fractal dimension, the mean intensity of the local region. Local area intensity standard deviation , This is a single-channel intensity map. This refers to the local window size, and Configurable; The first one obtained by upsampling from the enhanced spatial feature map A local area, in which for: in, Tensors for local regions; Extracting local vectors from local regions using a CNN convolutional neural network. , where local vectors The following results were obtained by processing with global average pooling or small convolutions: By using local vectors With fractal descriptors The original concatenated vector is obtained by concatenating the vectors. for: The original splicing vector is projected through the projection head. After mapping and normalization, the local embedding vector z is obtained: in, For projection networks, used to output dimensions and local embedding vectors ; Multiple local embedding vectors from the same source are used to perform optimized embedding learning through either self-supervised contrastive learning or triplet training to obtain the trained projection network. These local embedding vectors represent positive sample pairs from the same physical point or target and negative samples from other samples. The self-supervised contrastive learning and triplet training methods employ triplet loss. : in, For positive sample embedding, For negative sample embedding, These are local embedding vectors of samples from the same source, where m>0 indicates a boundary. The distance is Euclidean. After processing the augmented spatial feature map through the trained projection network, a new augmented spatial feature map and a fractal robust embedding vector are obtained.

2. The UAV spatial pose estimation method according to claim 1, characterized in that, The process of acquiring RGB and thermal images of the UAV, performing multi-scale preprocessing and feature extraction on the RGB and thermal images to obtain RGB and thermal input tensors includes: Normalize the RGB image, and expand the thermal imaging image to three channels and normalize it; The normalized RGB image and the normalized thermal image are spatially aligned and registered using an affine transformation model to obtain the registered RGB image and the registered thermal image. The registered RGB image and the registered thermal image are preprocessed using a multi-scale pyramid to obtain a multi-scale input set. The multi-scale input set is mapped to the deep network input space to obtain the RGB input tensor and the thermal imaging input tensor.

3. The UAV spatial pose estimation method according to claim 1, characterized in that, The step of performing feature extraction on the RGB input tensor and the thermal imaging input tensor to obtain RGB feature vectors and thermal feature vectors includes: Spatial features are extracted from the RGB input tensor and the thermal imaging input tensor using depthwise convolution and independent channels; The output of depthwise convolution is combined across channels using pointwise convolution to obtain the output of pointwise convolution; The outputs of the depthwise convolution and the pointwise convolution are merged and then activated and normalized to obtain the first layer feature map for each scale: The first layer feature map is enhanced using a standard residual structure to obtain the second layer feature map; The second-layer feature map is sequentially processed by channel compression, global average pooling, attention mapping, and channel weighting to obtain an attention-weighted feature map. The attention-weighted feature map is processed sequentially by standard 3×3 convolution, dilated 3×3 convolution and 1×1 convolution, and then convolution compression is performed to obtain the multi-scale fused output feature map. The output feature map after multi-scale fusion is processed by global average pooling or average pooling at each scale to obtain a scale vector. The scale vector is then aggregated and normalized between scales to obtain RGB feature vectors and hot feature vectors.

4. The UAV spatial pose estimation method according to claim 1, characterized in that, The step of performing learnable modal weights and global attention feature fusion on the RGB feature vector and the hot feature vector to obtain a multimodal fused feature vector includes: The RGB feature vector and the hot feature vector are concatenated to obtain a concatenated vector. The RGB feature vector and the hot feature vector are weighted and fused using learnable modal weights to obtain a weighted fused vector. The concatenated vector is weighted element-wise by modal attention weights to obtain an attention-weighted vector; The weighted fusion vector and the attention weighted vector are concatenated, and then a unified feature mapping is performed to obtain a multimodal fusion feature vector.

5. The UAV spatial pose estimation method according to claim 1, characterized in that, The enhanced spatial feature map, the fractal robust embedding vector, and the multimodal fusion feature vector are concatenated. The concatenated result is then subjected to bi-branch prediction of position and orientation to obtain the UAV's spatial pose estimation result, including: The new enhanced spatial feature map, fractal robust embedding vector, and multimodal fusion feature vector are concatenated: in, For the new enhanced spatial feature map The dimensionality reduction or dimensionality increase results of linear mapping, Multimodal fusion feature vector The dimensionality reduction or dimensionality increase results of linear mapping, fractal robust embedding vector The dimensionality reduction or dimensionality increase results of linear mapping, To concatenate vectors, For concatenating vectors Dimensions , and For bias terms, , and This is the weight matrix; Using lightweight MLP Mapping to the regression feature space yields the intermediate representation vector. : For intermediate representation vectors Prediction is performed using a location prediction branch and an orientation prediction branch, where the location prediction branch is as follows: Orientation prediction branch is: in To predict the location, To predict orientation, Here is the weight matrix for the location prediction branch. For the location prediction branch bias term, The weight matrix is ​​for the predicted branch. For the bias term pointing towards the branch, As an intermediate representation, It is an L2 norm; The total loss function used when making predictions in the location prediction branch and the orientation prediction branch is... for: The total loss is weighted by adaptive uncertainty. It is a learnable scalar used to represent the observable uncertainty of position and rotation; For position loss, ; Towards loss, ; For time series smoothing, The difference between the predicted positions of two adjacent frames. The difference in actual position between two adjacent frames; For consistency regularization, As an intermediate feature, The mean or target embedding of the local embedding vector. For projection head; This is a hyperparameter.

6. A spatial pose estimation device for unmanned aerial vehicles (UAVs), characterized in that, include: The first module is used to acquire RGB images and thermal images of the UAV, perform multi-scale preprocessing and feature extraction on the RGB images and thermal images, and obtain RGB input tensors and thermal image input tensors. The second module is used to perform feature extraction on the RGB input tensor and the thermal imaging input tensor to obtain RGB feature vector and thermal feature vector; The third module is used to perform feature fusion of learnable modal weights and global attention on the RGB feature vector and the hot feature vector to obtain a multimodal fusion feature vector; The fourth module is used to perform frequency domain transformation based on the RGB feature vector, the thermal feature vector and the multimodal fusion feature vector to obtain the amplitude spectrum and the phase spectrum, perform phase enhancement and inverse Fourier reconstruction on the amplitude spectrum and the phase spectrum in sequence, and perform adaptive spatial coding and residual fusion on the obtained phase-enhanced spatial features to obtain the enhanced spatial feature map. The fifth module is used to sample the enhanced spatial feature map, perform intensity map construction and box counting on the obtained sampled points to obtain a local fractal descriptor, and perform training using the local fractal descriptor and the enhanced spatial feature map to obtain a fractal robust embedding vector. The sixth module is used to concatenate the enhanced spatial feature map, the fractal robust embedding vector, and the multimodal fusion feature vector, and to perform a two-branch prediction of position and orientation on the concatenated result to obtain the spatial pose estimation result of the UAV. The process involves performing frequency domain transformation based on the RGB feature vector, the thermal feature vector, and the multimodal fusion feature vector to obtain the amplitude spectrum and phase spectrum. Phase enhancement and inverse Fourier reconstruction are then performed sequentially on the amplitude spectrum and phase spectrum. Adaptive spatial coding and residual fusion are then applied to the obtained phase-enhanced spatial features to obtain an enhanced spatial feature map, including: Employ learnable global weight scalars Furthermore, modal weights are extracted from the multimodal fusion feature vector using a multilayer perceptron or linear layer, resulting in a learnable global weight scalar. for: in, For learnable weights, For learnable bias, This is a multimodal fusion feature vector. For transpose; For RGB feature vectors and thermal eigenvectors The fused spatial feature map is obtained by spatial point-by-point weighted fusion and modal weighting. for: in, For the scale Transform into Image height, For overscale Transform into Image width, The number of output channels for pointwise convolution; The spatial coordinates of the image, For input channels, It is a feature map in RGB space. This is a spatial feature map for thermal imaging. Fusion spatial feature map Each channel undergoes mean-reduction processing to obtain a mean-reduced channel feature set. ,in for: in, This represents the total number of channels; Perform a two-dimensional Fourier transform on the mean-removed channel features The amplitude spectrum and phase spectrum are obtained, where the two-dimensional Fourier transform is: in, For offset, For amplitude spectrum, For phase spectrum, The imaginary unit; Phase enhancement is performed by applying phase modulation to the phase spectrum. The phase modulation method for the phase spectrum is as follows: in, To enhance the phase spectrum, (A) is the amplitude-normalized nonlinear gate function; For frequency domain phase-sensitive weights, and for: Where λ>0 and κ>0 are learnable or manually adjustable scaling hyperparameters. This indicates median normalization. This refers to the magnitude of the amplitude spectrum; It is the numerical stability constant; Indicates in the channel The entire frequency domain space Within the range, phase gradient magnitude global maximum value Reconstruct the frequency domain complex numbers from the amplitude spectrum and the enhanced phase spectrum, and perform an inverse Fourier transform. spatial features with phase enhancement were obtained. for: in, Obtained by stacking in channels , This indicates taking the real part of a complex number. Indicates the first Each channel is in The reconstructed complex spectrum value, The channel corresponding to the enhanced phase spectrum; Local descriptors are calculated for the fused spatial feature map and the spatial features enhanced by phase. A spatial attention map is then calculated based on these local descriptors. The calculation method is as follows: Spatial attention map The calculation method is as follows: in, For activation function, Used to output a single-channel heatmap; This is the Sigmoid activation function, used to obtain the weights at each spatial location; Based on the spatial characteristics of phase enhancement Spatial attention map and integrated spatial feature map After performing residual fusion and normalization, and activation processing, the enhanced spatial feature map is obtained as follows: in, As a global scale factor, For learnable weights, For learnable bias; The process involves sampling the enhanced spatial feature map, constructing an intensity map and performing box counting on the sampled points to obtain a local fractal descriptor, and then training the local fractal descriptor and the enhanced spatial feature map to obtain a fractal robust embedding vector, including: The enhanced spatial feature map is sampled using one of the following methods: fixed window, sliding window, and interest point sampling, to obtain sampling points, where the sampling points are continuous local regions; The multi-channel features of a local region are projected into a single-channel intensity map. Box counting statistics are performed on the single-channel intensity map at multiple scales. The box counting results are then fitted using the least squares method to obtain the local fractal dimension, where the scale set of the box counting is... Determine the fractal descriptor based on the statistics of the local region and the local fractal dimension. for: in, For the local fractal dimension, the mean intensity of the local region. Local area intensity standard deviation , This is a single-channel intensity map. This refers to the local window size, and Configurable; The first one obtained by upsampling from the enhanced spatial feature map A local area, in which for: in, Tensors for local regions; Extracting local vectors from local regions using a CNN convolutional neural network. , where local vectors The following results were obtained by processing with global average pooling or small convolutions: By using local vectors With fractal descriptors The original concatenated vector is obtained by concatenating the vectors. for: The original splicing vector is projected through the projection head. After mapping and normalization, the local embedding vector z is obtained: in, For projection networks, used to output dimensions and local embedding vectors ; Multiple local embedding vectors from the same source are used to perform optimized embedding learning through either self-supervised contrastive learning or triplet training to obtain the trained projection network. These local embedding vectors represent positive sample pairs from the same physical point or target and negative samples from other samples. The self-supervised contrastive learning and triplet training methods employ triplet loss. : in, For positive sample embedding, For negative sample embedding, These are local embedding vectors of samples from the same source, where m>0 indicates a boundary. The distance is Euclidean. After processing the augmented spatial feature map through the trained projection network, a new augmented spatial feature map and a fractal robust embedding vector are obtained.

7. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor executes the program to implement the UAV spatial pose estimation method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The storage medium stores a program that is executed by a processor to implement the UAV spatial pose estimation method as described in any one of claims 1-5.

Images