An unmanned aerial vehicle image matching method based on rotation isovist visual features

Abstract

This invention belongs to the field of computer vision and UAV autonomous localization technology. It discloses a UAV image matching method based on rotationally equivariant visual features. By introducing a group-equivariant convolutional structure, a rotationally equivariant feature extraction network is constructed for target matching, ensuring that the feature representation maintains mathematical structural consistency as the input image rotates. The method for generating feature point detection heatmaps is improved, and an offset correction network and a global offset loss function are designed to achieve automatic coordinate alignment of matching points, eliminating projection deviations caused by rotation and improving pose calculation accuracy. The global offset loss function constrains the spatial consistency of feature points under different viewpoints, significantly improving matching accuracy and stability. This invention achieves rotational equivariance and self-correction of matching projection errors from both network structure and training mechanism levels, enabling UAV visual matching capabilities that adapt to continuous rotation. This invention operates quickly and maintains stable performance in complex environments, closely reflecting real-world UAV flight scenarios.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and UAV autonomous localization technology, and in particular to a UAV image matching method based on rotationally variable visual features. Background Technology

[0002] In recent years, unmanned aerial vehicles (UAVs) have been widely used in reconnaissance, disaster relief, basic surveying and mapping, and inspection and monitoring. Traditional UAVs rely on GNSS for positioning, but GNSS signals can be severely limited or even completely unusable in scenarios such as terrain obstruction, dense urban high-rise buildings, underground structures, electromagnetic interference, and deception. In the absence of GNSS positioning, UAVs must rely solely on onboard visual sensors to complete environmental perception, motion estimation, and positioning and navigation, making pure visual positioning a necessary alternative. The most critical basic capabilities are feature point detection, descriptor extraction, and matching between images, and their performance directly determines the robustness and availability of higher-level tasks such as visual SLAM, SfM, and LOAM.

[0003] However, in actual drone flight, image matching faces the following typical difficulties: 1) Continuous changes in camera posture lead to large-angle rotational parallax. When a drone rotates around itself or a target, rotational deviations of 30°, 90° or even more than 180° can occur between two images. Conventional convolutional neural networks (CNNs) do not have rotational equivariance, and feature representations undergo structural drift with rotation, which can easily cause mismatches or overall translational errors.

[0004] 2) Limited onboard computing power makes it difficult to deploy highly complex networks. Although the Transformer architecture has certain global modeling capabilities, it has high memory requirements and slow inference speed, making it unsuitable for integration into flight control, embedded GPU, ARM and other platforms.

[0005] 3) Traditional artificial design features are difficult to adapt to changes in the field of view of UAVs. Artificial features such as SIFT, ORB, and SURF are unstable in areas with high viewing angle differences, changes in lighting, and sparse textures.

[0006] 4) Existing lightweight deep feature methods lack rotation robustness. Existing CNN structures have advantages in translational equivariance, but they do not have natural equivariance for SO(2) rotation groups. Data augmentation alone is insufficient to guarantee feature consistency, leading to global registration shifts in rotation scenarios.

[0007] In summary, current technologies suffer from the inability to simultaneously satisfy the requirements of "deployability, rotational robustness, and matching accuracy." There is a lack of a visual feature matching method that can run in real-time on embedded platforms, possesses strict rotational equivariance, and can correct for feature projection shifts caused by rotation. Therefore, there is an urgent need for a visual matching method that possesses rotational equivariance capabilities, can be deployed on embedded flight control platforms, and adapts to freely rotating UAV aerial photography scenarios to support autonomous positioning tasks in GNSS-denied environments. Summary of the Invention

[0008] The purpose of this invention is to provide a UAV image matching method based on rotationally variable visual features, which overcomes the shortcomings of existing UAV image matching methods such as feature instability, matching error shift, and excessive computational overhead in large-angle rotation scenarios. This method enables UAV image matching to perform feature point detection, descriptor extraction, and image matching with high robustness and low computational cost even under GNSS denial-of-localization conditions, providing basic support for subsequent tasks such as visual localization, SLAM, and 3D reconstruction.

[0009] To achieve the above objectives, this invention provides a UAV image matching method based on rotationally variable visual features, comprising the following steps: Step S1: Obtain a continuous frame image sequence from the UAV's onboard camera and satellite images, and perform size standardization, normalization, and grayscale conversion on the images; Step S2: Based on the E2CNN framework, a rotation-equivariant feature extraction network is built using a group equivariant convolutional structure, so that the output features maintain linear equivariance with any angle rotation. Step S3: Based on the rotational isotropic feature field output by the rotational isotropic feature extraction network, design a lightweight isotropic heatmap prediction head to generate a feature point detection heatmap. Step S4: Using a rotationally equivariant feature extraction network, descriptors are constructed based on a high-dimensional rotationally equivariant feature field. Feature descriptors are extracted for all image feature points to obtain a set of descriptors for matching. Step S5: Use the offset prediction network to correct the projection error caused by rotation; Step S6: Construct a global offset loss constraint to ensure that the position of the same physical point remains consistent before and after the offset under different rotation angles; Step S7: Employ a strategy of mutual nearest neighbor matching combined with cosine similarity thresholding to achieve feature matching and homography matrix calculation, which can be used for UAV localization or map reconstruction.

[0010] Preferably, in step S1, a continuous frame image sequence from the UAV's onboard camera and satellite images are acquired, and the images are normalized, standardized, and converted to grayscale. The specific process is as follows: Step S11: In the end-to-end training phase, supervised and self-supervised hybrid training is performed using a real outdoor scene multi-view dataset, including: 1) MegaDepth dataset: used to learn dense geometric consistency in scenes at different heights, angles and depths, including urban scenes, natural terrain and complex building structures, which constitute the training set of the model; 2) HPatches dataset: used to evaluate the matching ability of explicit rotation and lighting changes in planar scenes, and to verify the ability of rotational isomorphic features; Step S12: In the application phase, acquire continuous frame image sequences from the UAV's onboard camera and satellite images, and perform normalization preprocessing on the acquired images, specifically including: First, the two input images to be matched... and Perform a normalization operation to scale the pixel values ​​from the three-channel [0,255] range to the single-channel [0,1] range; Secondly, the image size is uniformly scaled to the network's preset input size, usually a multiple of 32, to ensure compatibility during batch processing; In addition, data augmentation strategies are applied during the training phase, including random brightness and contrast adjustments to simulate lighting changes, adding Gaussian noise to simulate sensor noise, and random rotation operations. Finally, the preprocessed image and As input to the extraction network.

[0011] Preferably, in step S2, a rotationally equivariant feature extraction network is built based on the group equivariant convolutional structure of the E2CNN framework, so that the output features maintain linear equivariance with any angle rotation; For cyclic groups or continuously rotated groups, the convolutional kernels of E2CNN are constrained to be equivariant kernels, thus ensuring the equivariant property of the rotation-equivariant feature extraction network; let Denotes the rotation group, whose elements Represents a rotation angle For an input image Group elements The rotational equivariance of group convolution is shown below: ; in, This represents a feature extraction network; Indicates the input image Apply by Defined rotation operation; Indicates the input image The feature space; In the feature space Apply to The corresponding transformation operations.

[0012] Preferably, in the design of the rotationally equivariant feature extraction network structure, the equivariant convolutional layer, equivariant nonlinear activation function, and equivariant pooling layer provided by the E2CNN.nn package are used to construct a deep feature pyramid; The rotationally equivariant feature extraction network takes a regular grayscale image as input. After the first convolutional layer, the features are boosted to a specified group representation and maintain equivariance in subsequent layers. Finally, it outputs a set of multi-level feature fields with rotational equivariance. and ; The feature field encodes the semantic and structural information in the image and is inherently robust to arbitrary rotations of the input image.

[0013] Preferably, in step S3, the rotational equivariant feature field output by the rotational equivariant feature extraction network is... and A lightweight isovariant heatmap prediction head built from E2CNN components was designed to ensure rotational isovariance throughout the heatmap generation process. The specific process is as follows: Step S31: First, perform feature transformation by using two consecutive 1×1 equal-variable convolutional layers, with batch normalization and equal-variable ReLU activation functions embedded in between, to transform the input feature field. and Nonlinear transformation and information fusion between channels are performed to enhance its discrimination ability; Step S32: Then, channel mapping and activation are performed. The last 1×1 equivariant convolutional layer maps the geometric type of the feature field from equivariant to invariant, which is defined as a scalar field invariant under the action of the rotation group. The output heatmap is a probability scalar field invariant under rotation. Step S33: Finally, by applying the pointwise Sigmoid activation function, the output values ​​are compressed to the [0,1] interval to obtain the final heatmap. and ; Step S34: Use non-maximum suppression to extract feature point locations from the heatmap, as shown below: Within the local neighborhood of each pixel, a decision is made only if the response value of that pixel is the maximum value within its neighborhood and exceeds a preset confidence threshold. If so, then retain it as a valid feature point; Assuming that it is eventually detected Let the initial positions of the feature points on the feature map scale be denoted as . ;in, Representing feature points The initial coordinates.

[0014] Preferably, in step S4, a lightweight equivariant descriptor extraction head is constructed from two consecutive 1×1 equivariant convolutional layers, which further fuses and transforms the feature field output by the rotational equivariant feature extraction network. The specific process is as follows: Step S41: Map the feature field to the descriptor space for feature matching to obtain a rotationally equivalent descriptor field; Step S42: Perform a group pooling operation on the descriptors to obtain rotation-invariant descriptor vectors. The specific operation is as follows: First, a rotationally invariant descriptor subfield is input, where the features at each spatial location are vectors that transform under group action. Then, pooling is performed on all features at each spatial location that span the group dimension. Finally, a regular rotationally invariant feature map is obtained. and ; Step S43: On the obtained rotation-invariant feature map, for each feature point detected in step S3, its initial position is... By using bilinear interpolation, the corresponding descriptor vector is extracted at that location. ; Step S44: For each descriptor vector Perform L2 normalization to distribute it on the unit hypersphere.

[0015] Preferably, in step S5, a lightweight offset prediction network specifically designed for the E2CNN architecture is introduced. This network does not predict a dense offset vector field, but instead directly predicts a global offset vector applicable to the entire image from global features. This corrects the projection error caused by rotation while reducing computational complexity. The specific process is as follows: Step S51: First, use group pooling to process the isovariant feature field output by the rotation isovariant feature extraction network and convert it into a rotation-invariant feature map. Step S52: Then, global average pooling is applied to the pooled feature map to obtain a global feature vector, which encodes the context information of the entire image. Step S53: Finally, input the global feature vector into a lightweight MLP to directly regress a two-dimensional global offset. This MLP consists of two linear layers. Step S54: Introduce a learnable scaling factor to adjust the predicted offset in order to control the magnitude of the offset, which facilitates network initialization and stable training.

[0016] Preferably, for each input image I, the offset prediction network outputs a global offset. ; For all feature points detected in image I, their initial coordinates Global offset will be used The correction is performed to obtain the final corrected feature point coordinates. As shown below: ; The aforementioned correction process compensates for the systematic geometric projection errors caused by rotation and viewpoint changes, ensuring that the predicted two-dimensional coordinates of the same physical space point remain consistent under different viewpoints, thereby improving the accuracy of subsequent pose solving.

[0017] Preferably, in step S6, a global offset loss constraint is constructed, assuming the true initial coordinates of the matching feature points in the two images are... and The global offset vectors predicted by the offset prediction network for the two images are as follows: and And the rotation-invariant feature map output by the rotation-equivalent feature extraction network, after group pooling, is... and ; Step S61: Broadcast the global offset to each pair of matching points and calculate their offset coordinates on the feature map scale, as shown below: ; ; in, and These represent the offset coordinates; Step S62: Normalize the offset coordinates to the [-1,1] grid coordinates, and then perform the corresponding operations on their respective feature maps. and Bilinear interpolation is performed on the vector, and resampling is used to obtain the offset position feature vector; Step S63: Calculate the cosine similarity of the positional features of all matching point pairs after the offset; Step S64: Introduce an adaptive weighting mechanism to maximize the average similarity and design a loss function. As shown below: ; in, To match the cosine similarity of the positional features of the point pairs after the offset; The temperature coefficient is weighted as follows: ; in, For normalized exponential functions, Temperature coefficient; Step S65: To ensure training stability and improve performance, the loss function also includes two auxiliary terms, specifically: (1) To ensure that the average similarity after offset correction is not lower than the average similarity at the original position and to prevent deviation from the optimization direction, a consistency loss function is added. As shown below: ; in, For activation functions; This represents the average feature similarity of the actual matching point pairs; The average feature cosine similarity of the matching point pairs predicted by the network after the offset; (2) To prevent the total loss from being too large, L2 regularization is applied to the predicted offset, and the regularization loss function is... As shown below: ; in, It is an L2 norm; Step S66: Finally, the total global offset loss is: ; in, This represents the total global offset loss.

[0018] Preferably, in step S7, a strategy combining mutual nearest neighbor matching and cosine similarity thresholding is adopted to achieve feature matching and homography matrix calculation for UAV localization or map reconstruction. The specific process is as follows: Based on the feature point sets of the two obtained images, specifically including the precise coordinates after offset correction. and the corresponding normalized descriptor Establish the correspondence between the two descriptor subsets; Step S71: Based on L2 normalized descriptors, efficiently calculate the cosine similarity matrix between all descriptors of two images using matrix multiplication; Step S72: Use the mutual nearest neighbor criterion to eliminate unstable many-to-one matching. The specific process is as follows: For images Each descriptor in the image Find its nearest neighbor in the image; similarly, for an image... Each descriptor in the image Find its nearest neighbor; then the point pair (A,B) is considered a matching pair. Step S73: Apply a cosine similarity threshold min_cossim to the initially obtained matching pairs for further filtering; Step S74: The algorithm outputs two index arrays, idx0 and idx1, indicating the image... and images The matching pairs formed in The matching point pairs are directly provided to the subsequent robust estimation algorithm for calculating the homography matrix; The homography matrix H describes the projection transformation relationship of a two-dimensional plane under two different camera viewpoints, satisfying: ; Here, (x,y) and (x′,y′) are homogeneous coordinates of a pair of matching points; H is estimated from the matching point pair using a robust estimation algorithm and then used to complete advanced tasks such as visual localization, image stitching, and attitude estimation for the UAV.

[0019] Therefore, the present invention employs the above-mentioned UAV image matching method based on rotationally variable visual features, and the beneficial effects are as follows: (1) By introducing a group equivariant convolutional structure, a feature extraction network with strict rotation equivariance is constructed for target matching; the feature expression has rotation invariance and can adapt to arbitrary angle changes without relying on a large amount of rotation enhancement data, ensuring that the feature expression maintains the same mathematical structure as the input image rotates.

[0020] (2) Improve the generation method of feature point detection heatmap and design offset correction network and global offset loss function to realize automatic coordinate alignment of matching points, eliminate projection deviation caused by rotation, and improve the accuracy of pose solution; use global offset loss to constrain the spatial consistency of feature points under different viewpoints, significantly improve matching accuracy and stability.

[0021] (3) This invention achieves rotational isovariance and self-correction of matching projection error from two levels: network structure and training mechanism, thereby realizing the visual matching capability of UAVs that can adapt to continuous rotation from 0° to 360°.

[0022] (4) This invention does not require self-attention global calculation, runs fast, is compatible with edge devices such as Jetson and RK3588, and maintains stable performance in complex environments such as rotation, scale, lighting and weak texture, which fits the real flight scenario of UAV.

[0023] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0024] Figure 1 This is a flowchart of a UAV image matching method based on rotationally equivariant visual features according to the present invention; Figure 2 This is a schematic diagram of the group-variable convolution structure of the present invention; Figure 3 This is a schematic diagram of the rotational isotropic feature extraction network structure of the present invention; Figure 4This is a schematic diagram of the offset prediction branch structure of the present invention; Figure 5 This is a schematic diagram comparing the matching effects of the present invention in a large-angle rotating image matching scenario. Detailed Implementation

[0025] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0026] Example 1 like Figure 1 As shown, the present invention provides a UAV image matching method based on rotationally variable visual features, comprising the following steps: Step S1: Obtain a continuous frame image sequence from the UAV's onboard camera and satellite images, and perform size standardization, normalization, and grayscale conversion on the images.

[0027] Step S11: In the end-to-end training phase, supervised and self-supervised hybrid training is performed using a real outdoor scene multi-view dataset, mainly including: 1) MegaDepth dataset: used to learn dense geometric consistency in scenes at different heights, angles and depths. It contains a large number of urban scenes, natural terrains and complex building structures, which constitute the training set of the model.

[0028] 2) HPatches dataset: used to evaluate the matching ability of explicit rotation and lighting changes in planar scenes, suitable for verifying the rotation isotropic feature capability proposed in this invention.

[0029] The MegaDepth-1500 dataset is a subset of the training set used to evaluate the training performance of the model.

[0030] Step S12: In the application phase, acquire continuous frame image sequences from the UAV's onboard camera and satellite images, and perform normalization preprocessing on the acquired images.

[0031] Because publicly available datasets and images acquired by drones and satellites often vary in size, lighting conditions, and may contain noise, the preprocessing workflow is as follows: First, the two input images to be matched... and Perform a normalization operation to scale the pixel values ​​from the three-channel [0,255] range to the single-channel [0,1] range.

[0032] Secondly, the image size is uniformly scaled to the network's preset input size, usually a multiple of 32, to ensure compatibility during batch processing.

[0033] To further simulate the various complex situations encountered by drones in actual flight, a series of data augmentation strategies will be applied during the training phase, including but not limited to: random brightness and contrast adjustments to simulate lighting changes; adding Gaussian noise to simulate sensor noise; and the crucial random rotation operation.

[0034] The random rotation here is not a simple data augmentation; its purpose is to force the network to learn invariance to rotation transformations during the training phase, laying the foundation for the effective learning of the subsequent rotation-equivariant feature extraction network.

[0035] Finally, the preprocessed image and It serves as the input for the subsequent rotational isomorphic feature extraction network.

[0036] Step S2: Based on the E2CNN framework, a rotation-equivariant feature extraction network is built using a group equivariant convolutional structure, so that the output features maintain linear equivariance with any angle rotation.

[0037] Traditional CNNs only exhibit translational equivariance, meaning that when the input image is translated, its output feature map also translates accordingly. However, for rotation operations, the feature response of CNNs changes unpredictably, lacking mathematical guarantees. This implies that if two images of the same scene have rotational differences, the features extracted by the CNN may be far apart in the descriptive subspace, greatly increasing the difficulty of matching.

[0038] To address this issue, this invention employs group-equalized convolution based on the E2CNN framework, such as... Figure 2 As shown, a rotation-equivariant feature extraction network is constructed to address the fundamental problem of unstable feature representation in traditional convolutional neural networks under rotational changes. E2CNN is specifically designed to handle rotational equivariance in two-dimensional images. Its core lies in extending traditional feature maps to feature fields, where each feature channel is associated with a specific group representation. For cyclic groups or continuous rotation groups SO(2), the convolutional kernels of E2CNN are constrained to be equivariant kernels, thus ensuring the equivariant properties of the rotation-equivariant feature extraction network.

[0039] set up Let SO(2) be the rotation group, and its elements be... Represents a rotation angle For an input image Group elements The rotational equivariance of group convolution is shown below: ; in, This represents a feature extraction network; Indicates the input image Apply by Defined rotation operation; Indicates the input image The feature space; In the feature space Apply to The corresponding transformation operations.

[0040] The physical meaning of the above formula is that rotating the image before extracting features is equivalent to extracting features first and then transforming the feature map accordingly. This property ensures that regardless of how the input image is rotated, the feature representation learned by the rotation-equivariant feature extraction network remains consistent in mathematical structure, thus fundamentally overcoming the feature instability caused by rotation.

[0041] like Figure 3 As shown, in the design of the rotationally equivariant feature extraction network structure, an equivariant convolutional layer, an equivariant nonlinear activation function, and an equivariant pooling layer provided by the E2CNN.nn package are used to construct a deep feature pyramid. The network input is a regular grayscale image. After the first convolutional layer, the features are boosted to a specified group representation and maintain equivariance in subsequent layers. The network finally outputs a set of multi-level feature fields with strict rotational equivariance. and These feature fields encode rich semantic and structural information in the image and are inherently robust to arbitrary rotations of the input image.

[0042] Step S3: Based on the rotational isotropic feature field output by the rotational isotropic feature extraction network, design a lightweight isotropic heatmap prediction head to generate a feature point detection heatmap.

[0043] Based on the feature field output by the rotation-equal feature extraction network, salient feature points for matching are located. Traditional methods typically generate corner points directly on the original image or use CNN-based detectors, but these methods may result in inconsistent point locations or reduced response when the image is rotated.

[0044] Rotational isotropic feature field output by a rotational isotropic feature extraction network and This invention designs a lightweight isovariant heatmap prediction head, also built from E2CNN components, to ensure that the entire heatmap generation process maintains rotational isovariance.

[0045] Step S31: First, perform feature transformation by using two consecutive 1×1 equal-variable convolutional layers, with batch normalization and equal-variable ReLU activation functions embedded in between, to transform the input feature field. and Nonlinear transformation and information fusion between channels are performed to enhance its discrimination ability.

[0046] Step S32: Then, channel mapping and activation are performed. The last 1×1 equivariant convolutional layer maps the geometric type of the feature field from equivariant to invariant, which is defined as a scalar field invariant under rotation group action. This ensures that no matter how the input image is rotated, the output heatmap is a probability scalar field invariant under rotation.

[0047] Step S33: Finally, by applying the pointwise Sigmoid activation function, the output values ​​are compressed to the [0,1] interval to obtain the final heatmap. and .

[0048] In this heatmap, each pixel value represents the probability of a stable, repeatable keypoint existing at its corresponding spatial location. Since the input feature field and the entire isotropic heatmap prediction head are rotationally isotropic, the generated heatmap is rotationally invariant. This means that when the input image rotates, the heatmap's response pattern rotates uniformly in space, but its probability value at each physical location remains unchanged. This fundamentally avoids the rotation inconsistency problem caused by pixel grid sampling and filter directionality when detecting directly on the original image.

[0049] Step S34: Using non-maximum suppression, extract the specific feature point locations from the heatmap, as shown below: Within the local neighborhood of each pixel, a decision is made only if the response value of that pixel is the maximum value within its neighborhood and exceeds a preset confidence threshold. If so, then it is retained as a valid feature point.

[0050] Assuming that it is eventually detected Let the initial positions of the feature points on the feature map scale be denoted as . ;in, Representing feature points The initial coordinates.

[0051] Step S4: Utilize a rotation-equivariant feature extraction network to construct descriptors based on a high-dimensional rotation-equivariant feature field. This enables the extraction of highly discriminative and rotation-robust feature descriptors for all image feature points, resulting in a set of descriptors for matching.

[0052] This invention designs a lightweight isovariant descriptor extraction head, which consists of two consecutive 1×1 isovariant convolutional layers. Its function is to further fuse and transform the feature field output by the rotation isovariant feature extraction network.

[0053] Step S41: Map the feature field to a descriptor subspace that is more suitable for feature matching to obtain a rotationally equivalent descriptor field.

[0054] Step S42: Group pooling was performed on the descriptors to obtain rotation-invariant descriptor vectors. The specific operation is as follows: First, a rotationally invariant descriptor subfield is input, where the feature at each spatial location is a vector that transforms under the group action. Then, pooling is performed on all features at each spatial location across the group dimension. Finally, a regular, rotation-invariant feature map is obtained. and .

[0055] For each spatial location, the pooled feature vector remains unchanged regardless of the rotation of the input image. This ensures that the same physical point can be extracted with the same descriptor from different rotational viewpoints.

[0056] Step S43: On the obtained rotation-invariant feature map, for each feature point detected in step S3, its initial position is... By using bilinear interpolation, the corresponding descriptor vector is extracted at that location. .

[0057] Step S44: For each descriptor vector L2 normalization is performed to distribute the data across the unit hypersphere, making it easier to use cosine similarity for matching later.

[0058] Step S5: Use the offset prediction network to correct the projection error caused by rotation.

[0059] Although rotationally equivariant feature extraction networks ensure the stability of feature representation, the geometric positions of feature points may undergo a global, systematic projection shift when there are large-angle rotations and significant viewpoint differences (such as when shooting from a tilted drone). This shift is not caused by the imaging itself, but by systematic errors caused by rotationally equivariant convolution during sampling, which directly affects the accuracy of subsequent pose calculations based on feature point positions.

[0060] To this end, this invention introduces a lightweight offset prediction network specifically designed for the E2CNN architecture, such as... Figure 4 As shown, this network does not predict a dense field of offset vectors, but instead predicts a global offset vector applicable to the entire image directly from global features, significantly reducing computational complexity.

[0061] Step S51: First, use group pooling to process the isovariant feature field output by the rotation isovariant feature extraction network and convert it into a rotation-invariant feature map.

[0062] Step S52: Then, global average pooling is applied to the pooled feature map to obtain a global feature vector, which encodes the contextual information of the entire image.

[0063] Step S53: Finally, the global feature vector is input into a lightweight MLP, which directly regresses a two-dimensional global offset. This MLP consists of two linear layers, so the number of parameters is extremely small.

[0064] Step S54: Introduce a learnable scaling factor to adjust the predicted offset in order to control the magnitude of the offset, which facilitates network initialization and stable training.

[0065] For each input image I, the offset prediction network outputs a global offset. For all feature points detected in image I, their initial coordinates are... Global offset will be used The correction is performed to obtain the final corrected feature point coordinates. As shown below: ; This correction process effectively compensates for the systematic geometric projection errors caused by rotation and viewpoint changes, making the predicted two-dimensional coordinates of the same physical space point more consistent under different viewpoints, thereby significantly improving the accuracy of subsequent pose solving.

[0066] Step S6: Construct a global offset loss constraint to ensure that the position of the same physical point remains consistent before and after the offset under different rotation angles, thereby eliminating the accumulation of global errors and improving matching stability.

[0067] To effectively train the offset prediction network and optimize it in conjunction with the entire feature extraction and detection process, this invention designs a novel global offset loss function based on feature similarity weighting. The core idea of ​​this loss function is to drive the global offset predicted by the offset prediction network so that the corresponding feature points in two images have higher feature similarity at their corrected coordinate positions.

[0068] Assume that the true initial coordinates of matching feature points in two images are... and The global offset vectors predicted by the offset prediction network for the two images are as follows: and And the rotation-invariant feature map output by the rotation-equivalent feature extraction network, after group pooling, is... and .

[0069] Step S61: Broadcast the global offset to each pair of matching points and calculate their offset coordinates on the feature map scale, as shown below: ; ; in, and These represent the offset coordinates.

[0070] Step S62: Normalize the offset coordinates to the [-1,1] grid coordinates, and then perform the corresponding operations on their respective feature maps. and Bilinear interpolation is performed on the vector, and resampling is used to obtain the offset position feature vector.

[0071] Step S63: Calculate the cosine similarity of the positional features of all matching point pairs after the offset.

[0072] Step S64: Introduce an adaptive weighting mechanism to maximize the average similarity and design a loss function. As shown below: ; in, To match the cosine similarity of the positional features of the point pairs after the offset; The temperature coefficient is weighted as follows: ; in, For normalized exponential functions, This is the temperature coefficient.

[0073] This design makes the loss function focus more on matching point pairs whose feature similarity is already high after the current offset correction. By assigning higher weights to these point pairs, the network is guided to find a global offset that enables more point pairs to achieve a high similarity alignment, making the optimization objective more explicit and efficient.

[0074] Step S65: To ensure training stability and improve performance, the loss function also includes two auxiliary terms, specifically: (1) To ensure that the average similarity after offset correction is not lower than the average similarity at the original position and to prevent deviation from the optimization direction, a consistency loss function is added. As shown below: ; in, For activation functions; This represents the average feature similarity of the actual matching point pairs; This represents the average feature cosine similarity of the predicted matching point pairs after the offset.

[0075] (2) To prevent the total loss from being too large, L2 regularization is applied to the predicted offset, and the regularization loss function is... As shown below: ; in, It is an L2 norm.

[0076] Step S66: Finally, the total global offset loss is: ; in, This represents the total global offset loss.

[0077] The global offset loss function creates an effective learning signal by directly linking the correction of geometric coordinates with the similarity of semantic features. This drives the offset prediction network to converge quickly to a solution that can significantly eliminate rotation projection errors, thereby strongly constraining the consistency of feature representations from different perspectives.

[0078] Step S7: Employ a strategy of mutual nearest neighbor matching combined with cosine similarity thresholding to achieve feature matching and homography matrix calculation, which can be used for UAV localization or map reconstruction.

[0079] Based on the feature point sets of the two obtained images, specifically including the precise coordinates after offset correction. and the corresponding normalized descriptor This invention establishes a correct and reliable correspondence between two descriptor subsets. It employs a strategy combining mutual nearest neighbor matching with a cosine similarity threshold to maximize the quality of matching pairs while ensuring the quantity of matches.

[0080] Step S71: Based on L2 normalized descriptors, efficiently calculate the cosine similarity matrix between all descriptors of two images using matrix multiplication.

[0081] Step S72: Use the mutual nearest neighbor criterion to eliminate unstable many-to-one matching. The specific process is as follows: For images Each descriptor in the image Find its nearest neighbor (the one with the highest similarity). Similarly, for an image... Each descriptor in the image Find its nearest neighbor.

[0082] Only when "image" Point A in the image is considered to be... Point B in the image is its nearest neighbor. Point B in the image is also considered... A pair of points (A,B) is considered a matching pair only when point A is its nearest neighbor.

[0083] Step S73: Even if the mutual nearest neighbor condition is met, the absolute similarity of some matching pairs may still be low, resulting in poor reliability. Therefore, a strict cosine similarity threshold min_cossim is applied to filter the initially obtained mutual matching pairs.

[0084] Step S74: The algorithm outputs two index arrays, idx0 and idx1, which respectively indicate the image. and images Which feature points constitute reliable matching pairs? These high-precision matching point pairs can be directly provided to subsequent robust estimation algorithms (such as RANSAC) for calculating the homography matrix.

[0085] The homography matrix H describes the projection transformation relationship of a two-dimensional plane under two different camera viewpoints, satisfying: ; Here, (x,y) and (x′,y′) are homogeneous coordinates of a pair of matched points. After robustly estimating H from the matched point pair using RANSAC, it can be used to perform advanced tasks such as UAV visual localization, image stitching, and attitude estimation.

[0086] Example 2 This embodiment validates and tests the publicly available datasets megedepth-1500 and Hpatches. The specific process is as follows: 1. Extract the network that uses the task image to rotate as an input.

[0087] 1. Preprocess the two images in the image pair separately, including grayscale conversion, image size scaling to a factor of 32, and normalization.

[0088] 2. Input the preprocessed image into the rotation equivariance feature extraction network built on E2CNN, and output a multi-layer feature field with rotation equivariance.

[0089] 2. Perform feature point heatmap generation, descriptor extraction, and offset prediction in sequence.

[0090] 1. Input the deep feature field output by the rotational isovariant feature extraction network into the isovariant heatmap prediction head to generate a feature point heatmap, and extract the initial position of the feature points by suppressing non-maximum values.

[0091] 2. The feature field is transformed by another isovariant descriptor extraction head, and rotation-invariant feature descriptors are obtained through group pooling. The feature vectors are then filtered using the initial position points extracted by nonmaximum suppression.

[0092] 3. Input the feature field into the lightweight global offset prediction head to predict a global offset applicable to the entire image, and uniformly correct the initial positions of all feature points to obtain the final refined coordinates.

[0093] III. Perform feature matching and evaluation.

[0094] 1. Based on the mutual nearest neighbor strategy combined with the cosine similarity threshold, the feature points and their descriptors of the two images after correction are matched to generate preliminary matching pairs.

[0095] 2. The RANSAC algorithm is used to estimate the fundamental matrix based on matching pairs. For megedepth-1500, the ACC and AUC of the error between it and the true fundamental matrix are calculated. For Hpatches, the midpoints of the image are projected using the estimated and true fundamental matrices, and the mean projection error accuracy (MHA) between the projected points is calculated. The test metrics are shown in Tables 1 and 2.

[0096] Table 1. Megadepth-1500 Test Results

[0097] Table 2 Hpatches Test Results

[0098] The method proposed in this invention yields the following results for feature point detection and matching on rotated images of UAVs: Figure 5 As shown.

[0099] The test set in this embodiment exhibits a high degree of diversity in scene content, with the model facing landmarks and terrains completely untouched during training. However, even in such a challenging cross-scene generalization task, this invention, thanks to its inherent rotational isovariance and global offset correction mechanism, demonstrates superior generalization ability and robustness in both matching accuracy and pose estimation accuracy.

[0100] Therefore, this invention employs a UAV image matching method based on rotationally equivariant visual features. Addressing the core challenges of rotation and viewpoint changes in UAV image matching, it introduces an E2CNN-based rotationally equivariant feature extraction network, fundamentally improving the stability of feature representation. Simultaneously, through global offset prediction and loss design, systematic projection errors are explicitly corrected. Finally, rigorous cross-scene testing experiments on the MegaDepth-1500 and Hpatches datasets demonstrate that even in novel and unknown scenarios, it maintains high-precision matching performance and stable pose estimation capabilities, fully validating its effectiveness and practicality in real-world UAV applications.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for UAV image matching based on rotationally variable visual features, characterized in that, Includes the following steps: Step S1: Obtain a continuous frame image sequence from the UAV's onboard camera and satellite images, and perform size standardization, normalization, and grayscale conversion on the images; Step S2: Based on the E2CNN framework, a rotation-equivariant feature extraction network is built using a group equivariant convolutional structure, so that the output features maintain linear equivariance with any angle rotation. Step S3: Based on the rotational isotropic feature field output by the rotational isotropic feature extraction network, design a lightweight isotropic heatmap prediction head to generate a feature point detection heatmap. Step S4: Using a rotationally equivariant feature extraction network, descriptors are constructed based on a high-dimensional rotationally equivariant feature field. Feature descriptors are extracted for all image feature points to obtain a set of descriptors for matching. Step S5: Use the offset prediction network to correct the projection error caused by rotation; Step S6: Construct a global offset loss constraint to ensure that the position of the same physical point remains consistent before and after the offset under different rotation angles; Step S7: Employ a strategy of mutual nearest neighbor matching combined with cosine similarity thresholding to achieve feature matching and homography matrix calculation, which can be used for UAV localization or map reconstruction.

2. The UAV image matching method based on rotationally variable visual features according to claim 1, characterized in that, In step S1, a continuous frame image sequence from the UAV's onboard camera and satellite images are acquired. The images are then normalized, standardized, and converted to grayscale. The specific process is as follows: Step S11: In the end-to-end training phase, supervised and self-supervised hybrid training is performed using a real outdoor scene multi-view dataset, including: 1) MegaDepth dataset: used to learn dense geometric consistency in scenes at different heights, angles and depths, including urban scenes, natural terrain and complex building structures, which constitute the training set of the model; 2) HPatches dataset: used to evaluate the matching ability of explicit rotation and lighting changes in planar scenes, and to verify the ability of rotational isomorphic features; Step S12: In the application phase, acquire continuous frame image sequences from the UAV's onboard camera and satellite images, and perform normalization preprocessing on the acquired images, specifically including: First, the two input images to be matched... and Perform a normalization operation to scale the pixel values ​​from the three-channel [0,255] range to the single-channel [0,1] range; Secondly, the image size is uniformly scaled to the network's preset input size, usually a multiple of 32, to ensure compatibility during batch processing; In addition, data augmentation strategies are applied during the training phase, including random brightness and contrast adjustments to simulate lighting changes, adding Gaussian noise to simulate sensor noise, and random rotation operations. Finally, the preprocessed image and As input to the extraction network.

3. The UAV image matching method based on rotationally variable visual features according to claim 1, characterized in that, In step S2, a rotationally equivariant feature extraction network is built based on the group equivariant convolution structure of the E2CNN framework, so that the output features maintain linear equivariance with any angle rotation. For cyclic groups or continuously rotated groups, the convolutional kernels of E2CNN are constrained to be equivariant kernels, thus ensuring the equivariant property of the rotation-equivariant feature extraction network; let Denotes the rotation group, whose elements Represents a rotation angle For an input image Group elements The rotational equivariance of group convolution is shown below: ； in, This represents a feature extraction network; Indicates the input image Apply by Defined rotation operation; Indicates the input image The feature space; In the feature space Apply to The corresponding transformation operations.

4. The UAV image matching method based on rotationally variable visual features according to claim 3, characterized in that, In the design of the rotationally equivariant feature extraction network structure, we use the equivariant convolutional layer, equivariant nonlinear activation function and equivariant pooling layer provided by the E2CNN.nn package to construct a deep feature pyramid. The rotationally equivariant feature extraction network takes a regular grayscale image as input. After the first convolutional layer, the features are boosted to a specified group representation and maintain equivariance in subsequent layers. Finally, it outputs a set of multi-level feature fields with rotational equivariance. and ; The feature field encodes the semantic and structural information in the image and is inherently robust to arbitrary rotations of the input image.

5. The UAV image matching method based on rotationally equivariant visual features according to claim 4, characterized in that, In step S3, the rotational equivariant feature field output by the rotational equivariant feature extraction network is... and A lightweight isovariant heatmap prediction head built from E2CNN components was designed to ensure rotational isovariance throughout the heatmap generation process. The specific process is as follows: Step S31: First, perform feature transformation by using two consecutive 1×1 equal-variable convolutional layers, with batch normalization and equal-variable ReLU activation functions embedded in between, to transform the input feature field. and Nonlinear transformation and information fusion between channels are performed to enhance its discrimination ability; Step S32: Then, channel mapping and activation are performed. The last 1×1 equivariant convolutional layer maps the geometric type of the feature field from equivariant to invariant, which is defined as a scalar field invariant under the action of the rotation group. The output heatmap is a probability scalar field invariant under rotation. Step S33: Finally, by applying the pointwise Sigmoid activation function, the output values ​​are compressed to the [0,1] interval to obtain the final heatmap. and ; Step S34: Use non-maximum suppression to extract feature point locations from the heatmap, as shown below: Within the local neighborhood of each pixel, a decision is made only if the response value of that pixel is the maximum value within its neighborhood and exceeds a preset confidence threshold. If so, then retain it as a valid feature point; Assuming that it is eventually detected Let the initial positions of the feature points on the feature map scale be denoted as . ;in, Representing feature points The initial coordinates.

6. The UAV image matching method based on rotationally variable visual features according to claim 5, characterized in that, In step S4, a lightweight equivariant descriptor extraction head is constructed from two consecutive 1×1 equivariant convolutional layers, which further fuses and transforms the feature field output by the rotational equivariant feature extraction network. The specific process is as follows: Step S41: Map the feature field to the descriptor space for feature matching to obtain a rotationally equivalent descriptor field; Step S42: Perform a group pooling operation on the descriptors to obtain rotation-invariant descriptor vectors. The specific operation is as follows: First, input a rotationally equivalent descriptor subfield, where the features at each spatial location of the descriptor subfield are vectors that transform under group action; then, pool all features at each spatial location that span the group dimension. Finally, the rotation-invariant feature map is obtained. and ; Step S43: On the obtained rotation-invariant feature map, for each feature point detected in step S3, its initial position is... By using bilinear interpolation, the corresponding descriptor vector is extracted at that location. ; Step S44: For each descriptor vector Perform L2 normalization to distribute it on the unit hypersphere.

7. The UAV image matching method based on rotationally variable visual features according to claim 6, characterized in that, In step S5, a lightweight offset prediction network specifically designed for the E2CNN architecture is introduced. This network does not predict a dense offset vector field, but instead directly predicts a global offset vector applicable to the entire image from global features. This corrects the projection error caused by rotation while reducing computational complexity. The specific process is as follows: Step S51: First, use group pooling to process the isovariant feature field output by the rotation isovariant feature extraction network and convert it into a rotation-invariant feature map. Step S52: Then, global average pooling is applied to the pooled feature map to obtain a global feature vector, which encodes the context information of the entire image. Step S53: Finally, input the global feature vector into a lightweight MLP to directly regress a two-dimensional global offset. This MLP consists of two linear layers. Step S54: Introduce a learnable scaling factor to adjust the predicted offset in order to control the magnitude of the offset, which facilitates network initialization and stable training.

8. The UAV image matching method based on rotationally variable visual features according to claim 7, characterized in that, For each input image I, the offset prediction network outputs a global offset. ; For all feature points detected in image I, their initial coordinates Global offset will be used The correction is performed to obtain the final corrected feature point coordinates. As shown below: ； The aforementioned correction process compensates for the systematic geometric projection errors caused by rotation and viewpoint changes, ensuring that the predicted two-dimensional coordinates of the same physical space point remain consistent under different viewpoints, thereby improving the accuracy of subsequent pose solving.

9. The UAV image matching method based on rotationally variable visual features according to claim 8, characterized in that, In step S6, a global offset loss constraint is constructed, assuming the true initial coordinates of the matching feature points in the two images are... and The global offset vectors predicted by the offset prediction network for the two images are as follows: and And the rotation-invariant feature map output by the rotation-equivalent feature extraction network, after group pooling, is... and ; Step S61: Broadcast the global offset to each pair of matching points and calculate their offset coordinates on the feature map scale, as shown below: ； ； in, and These represent the offset coordinates; Step S62: Normalize the offset coordinates to the [-1,1] grid coordinates, and then perform the corresponding operations on their respective feature maps. and Bilinear interpolation is performed on the vector, and resampling is used to obtain the offset position feature vector; Step S63: Calculate the cosine similarity of the positional features of all matching point pairs after the offset; Step S64: Introduce an adaptive weighting mechanism to maximize the average similarity and design a loss function. As shown below: ； in, To match the cosine similarity of the positional features of the point pairs after the offset; The temperature coefficient is weighted as follows: ； in, For normalized exponential functions, Temperature coefficient; Step S65: To ensure training stability and improve performance, the loss function also includes two auxiliary terms, specifically: (1) To ensure that the average similarity after offset correction is not lower than the average similarity at the original position and to prevent deviation from the optimization direction, a consistency loss function is added. As shown below: ； in, For activation functions; This represents the average feature similarity of the actual matching point pairs; The average feature cosine similarity of the matching point pairs predicted by the network after the offset; (2) To prevent the total loss from being too large, L2 regularization is applied to the predicted offset, and the regularization loss function is... As shown below: ； in, It is an L2 norm; Step S66: Finally, the total global offset loss is: ； in, This represents the total global offset loss.

10. The UAV image matching method based on rotationally variable visual features according to claim 9, characterized in that, In step S7, a strategy combining mutual nearest neighbor matching and cosine similarity thresholding is adopted to achieve feature matching and homography matrix calculation, which is used for UAV localization or map reconstruction. The specific process is as follows: Based on the feature point sets of the two obtained images, specifically including the precise coordinates after offset correction. and the corresponding normalized descriptor Establish the correspondence between the two descriptor subsets; Step S71: Based on L2 normalized descriptors, efficiently calculate the cosine similarity matrix between all descriptors of two images using matrix multiplication; Step S72: Use the mutual nearest neighbor criterion to eliminate unstable many-to-one matching. The specific process is as follows: For images Each descriptor in the image Find its nearest neighbor in the image; similarly, for an image... Each descriptor in the image Find its nearest neighbor; then the point pair (A,B) is considered a matching pair. Step S73: Apply a cosine similarity threshold min_cossim to the initially obtained matching pairs for further filtering; Step S74: The algorithm outputs two index arrays, idx0 and idx1, indicating the image... and images The matching pairs formed in The matching point pairs are directly provided to the subsequent robust estimation algorithm for calculating the homography matrix; The homography matrix H describes the projection transformation relationship of a two-dimensional plane under two different camera viewpoints, satisfying: ； Here, (x,y) and (x′,y′) are homogeneous coordinates of a pair of matching points; H is estimated from the matching point pair using a robust estimation algorithm and then used to complete advanced tasks such as visual localization, image stitching, and attitude estimation for the UAV.

Images