A method for matching feature of oblique aerial images and airborne LiDAR point clouds

Abstract

This invention belongs to the field of photogrammetry and computer vision technology. Its purpose is to provide a method for matching features between oblique aerial imagery and airborne LiDAR point clouds. This method solves the problem of matching features between oblique aerial imagery and airborne LiDAR point clouds, avoiding the loss of matching accuracy caused by dimensionality changes in airborne LiDAR point clouds or aerial oblique imagery. This method improves the reliability of feature matching results and is suitable for complex scenarios with large amounts of data and applications requiring high precision.

Description

Technical Field

[0001] The present invention belongs to the technical field of photogrammetry and computer vision, and in particular relates to a method for matching features of oblique aerial images and airborne LiDAR point clouds. Background Art

[0002] The prerequisite and key to fusing aerial oblique imagery with airborne LiDAR (Laser Radar, a radar system that uses laser beams to detect target position, velocity, and other characteristic parameters) point clouds is the registration of LiDAR point clouds with the imagery. The core step in this registration process is feature matching, which can be categorized into the following three categories based on the final registration dimension:

[0003] The first type of 3D-3D matching method first performs feature matching on the sequence of images, then uses SFM (Structure From Motion, a 3D reconstruction algorithm) to restore the sparse 3D point cloud, and then uses MVS (Multi-View Steros, a multi-view stereo 3D reconstruction algorithm) to generate a dense image point cloud. Finally, 3D-3D model registration is performed according to the ICP (Iterative Closest Point, iterative closest point algorithm) algorithm or other point cloud registration methods to establish the relationship between the dense point cloud and the airborne LiDAR point cloud, so as to use the coordinates of the laser points in the LiDAR point cloud that match the image points in the dense point cloud to optimize the internal and external orientation elements of the optical image. This type of method indirectly constructs the matching relationship between the three-dimensional and two-dimensional feature points. However, this type of method cannot perform feature matching between the LiDAR point cloud and a single image. The calculation amount is large, and errors will be generated in the process of generating the dense point cloud, which will lead to a decrease in the final registration accuracy. In addition, the ICP algorithm has high accuracy requirements for the selection of initial values.

[0004] The second type of 2D-2D matching method uses the elevation information or intensity information of LiDAR laser point data to convert the point cloud data into a two-dimensional image, and then uses the traditional image matching method to perform feature matching. Finally, the matched two-dimensional image features are inversely calculated to the three-dimensional point cloud coordinates, thereby indirectly constructing the matching relationship between the three-dimensional features and the two-dimensional features. Among them, the traditional image feature matching method can be divided into three categories based on the geometric features of the matching features: point matching, line matching, and surface matching. In order to enhance the reliability of the matching results, some researchers have combined different geometric primitives (such as point-line features or point-surface features) to achieve feature matching between images. This type of 2D-2D matching method makes full use of the original mature image feature matching algorithm and has a high degree of automation. However, this method has errors in the regularization process, matching errors, and errors in the coordinate inverse calculation process. These errors will reduce the accuracy of matching between point cloud features and image features.

[0005] The third type of 3D-3D matching method directly extracts features from the laser point cloud and image, searching for features with the same name between the two, thereby achieving feature matching between the aerial image and the airborne LiDAR point cloud. However, due to the dimensionality inconsistency between two-dimensional and three-dimensional feature descriptors, traditional methods for calculating feature descriptor similarity are not applicable to cross-dimensional matching. Therefore, some researchers detect image vanishing points and point cloud line features, and take advantage of the property that the vanishing point coordinate transformation is independent of translation. By matching the vanishing point with the corresponding three-dimensional direction, they can directly solve the rotation relationship between the camera and the model.

[0006] With the rise of artificial intelligence, a growing number of researchers are applying deep learning-based methods to image and point cloud matching. Deep learning methods can be broadly categorized into feature descriptor networks, networks that combine feature descriptors with a similarity metric, and networks that jointly describe and detect local features. Feature descriptor networks learn feature descriptors by constructing deep neural networks. This type of network approach not only ensures accurate matching but also offers significant flexibility and can be embedded within other networks. Currently, these networks employ a dual-branch model to encode features for the point cloud and image, respectively, and utilize a triple loss function to obtain matching results. However, the feature descriptors learned by these networks are not robust enough, and they simply use Euclidean distance to measure descriptor similarity, resulting in low volumetric matching accuracy between 2D image patches and 3D point clouds.

[0007] To address the issues with single feature descriptor networks, some researchers have proposed networks that combine feature descriptors and similarity metrics. These networks simultaneously learn feature descriptors and similarity metrics, ultimately outputting a matching relationship between a 2D image patch and a 3D point cloud volume. However, these networks do not establish a matching relationship between the point cloud and the image pixels.

[0008] The input layers of the feature descriptor network and the unified network for joint feature descriptors and similarity metrics are both 2D image patches and 3D point cloud volumes. The difference is that the output layer of the feature descriptor network is the descriptors of the 2D image patches and 3D point cloud volumes, respectively, while the output layer of the unified network for joint feature descriptors and similarity metrics is the matching relationship between the 2D image patches and 3D point cloud volumes. The input and output of the network for joint description and detection of local features differ from the previous two types of networks. Its input layer is a 2D image and a 3D point cloud. It jointly detects image feature points and 3D point cloud feature points in the feature space and directly establishes the correspondence between the 2D and 3D feature points. However, the degree of automation of 2D-3D methods depends on the degree of automation of the extraction and matching of features with the same name, which still requires further research.

[0009] In summary, although researchers have proposed a variety of methods for establishing 3D and 2D relationships, the inventors have discovered that the following problems still exist when applying existing methods to point cloud and image feature matching:

[0010] (1) Existing indirect matching methods based on image generation of dense point clouds and conversion of point clouds into images are difficult to apply to complex scenarios with large amounts of data and application scenarios with high accuracy requirements;

[0011] (2) Although the method of directly establishing 2D-3D correspondence is intuitive and effective, and reduces the error accumulation existing in traditional methods, this type of method still has problems with feature extraction and feature matching, especially in outdoor scenes with complex terrain. Summary of the Invention

[0012] The present invention aims to solve the above technical problems at least to a certain extent, and provides a method for matching features of oblique aerial images and airborne LiDAR point clouds.

[0013] In order to achieve the above object, the present invention adopts the following technical solutions:

[0014] In a first aspect, the present invention provides a method for matching features between oblique aerial images and airborne LiDAR point clouds, comprising:

[0015] Get the key point information of the airborne LiDAR point cloud and the key point information of the oblique aerial image respectively, and save the key point information of the airborne LiDAR point cloud to the point cloud key point set Info C In the image key point information of the oblique aerial image, the image key point information set Info is saved. I middle;

[0016] The image key point information set Info I Image key point information and the point cloud key point set Info C The key point information of the point cloud is encoded and processed to obtain the encoded image feature information set And the encoded point cloud feature information set

[0017] Based on the attention aggregation mechanism, the encoded image feature information is respectively The encoded image feature information and the encoded point cloud feature information set in Update the encoded point cloud feature information in the image to obtain the updated image feature information set (L) X 2D And the updated point cloud feature information set (L) X 3D, and respectively according to the updated image feature information set (L) X 2D And the updated point cloud feature information set (L) X 3D Get the final image key point descriptor information set Dec 2D And the final point cloud key point descriptor information set Dec 3D ;

[0018] According to the final image key point descriptor information set Dec 2D And the final point cloud key point descriptor information set Dec 3D , and obtain the matching matrix of the airborne LiDAR point cloud and the oblique aerial image.

[0019] The present invention can solve the problem of feature matching between aerial oblique images and airborne LiDAR point clouds, avoiding the loss of matching accuracy caused by dimensional changes in airborne LiDAR point clouds or aerial oblique images. The reliability of feature matching results is improved, and it is suitable for complex scenes with large amounts of data and application scenarios with high precision requirements.

[0020] In a possible design, obtaining point cloud key point information of an airborne LiDAR point cloud includes:

[0021] Using the longest distance sampling, randomly sample M nodes S1,…,S in the airborne LiDAR point cloud. M , and store M nodes in the node set S; where the node set S={S1,…,S M};

[0022] Traverse the airborne LiDAR point cloud, select the node closest to each 3D point in the airborne LiDAR point cloud from the node set S, and obtain a clustering result of M nodes; where node S k The clustering result is S k Represents the kth node in the node set S, k∈{1,…,M}, n k Indicates the airborne LiDAR point cloud with node S k The number of corresponding 3D points;

[0023] Get the mean coordinates of all clustering results, and obtain the regularized coordinates of the 3D point cloud corresponding to each node based on the original coordinates of the 3D points in the clustering results of each node and the mean coordinates of its corresponding clustering results; where the regularized coordinates of the 3D point cloud corresponding to any node are the regularized coordinates of all 3D points in the 3D point cloud corresponding to the current node;

[0024] According to the regularized coordinates of the three-dimensional point cloud corresponding to each node, the local features of each node are obtained, and then the local features of each node are vector aggregated to obtain M node features G (M×C) ; Among them, node S k The local feature is F k |S k ;

[0025] According to the node feature G (M×C) Get the feature information of K neighboring nodes of each node; among them, node S k The feature information of K neighboring nodes is

[0026] Regularize the feature information of the K neighboring nodes of each node to obtain the regularized feature information of the K neighboring nodes of each node; k The regularized feature information of the K neighboring nodes is Represents node S k The normalized coordinates of the mth neighbor of , It is node S k The original coordinates of the mth neighbor value;

[0027] The point cloud key point information of the airborne LiDAR point cloud is obtained according to the regularized feature information of K neighboring nodes of each node.

[0028] In a possible design, the regularized coordinates of the 3D point cloud corresponding to each node are the difference between the original coordinates of the 3D point in the clustering result of each node and the mean coordinates of its corresponding clustering result; k The corresponding regularized coordinates of the i-th three-dimensional point are: Is the node S k The original coordinates of the corresponding i-th three-dimensional point, It is node S k The mean coordinates of the clustering results.

[0029] In one possible design, the point cloud key point information of the airborne LiDAR point cloud is obtained based on the regularized feature information of K neighboring nodes of each node, including:

[0030] According to the regularized feature information of K neighboring nodes of each node, the feature vector set {H1,…,H M};

[0031] According to the feature vector set {H1,…,H M}Get the key point coordinate matrix of each node

[0032] According to the key point coordinate matrix The regularized coordinates of each node in and the mean coordinates of the corresponding clustering results are used to obtain the point cloud key point information of the airborne LiDAR point cloud.

[0033] In one possible design, the point cloud key point information of the airborne LiDAR point cloud includes the key point position information of the airborne LiDAR point cloud; the key point position information of the airborne LiDAR point cloud is the key point coordinate matrix The sum of the regularized coordinates of each node and the mean coordinates of the corresponding clustering results; where node S k The corresponding key point position information is is the key point coordinate matrix Middle node S k The regularized coordinates of

[0034] In one possible design, the image key point information of the oblique aerial image includes key point position information and key point descriptor information of the oblique aerial image; wherein obtaining the image key point information of the oblique aerial image includes:

[0035] performing feature coding processing on the oblique aerial image to obtain a coded feature vector;

[0036] Performing a first convolution operation on the encoded feature vector to obtain a first convolution feature vector, and calculating the probability that each feature point in the first convolution feature vector is an image feature, so as to obtain position information of all key points of the oblique aerial image;

[0037] The encoded feature vector is subjected to a second convolution operation to obtain a second convolution feature vector, and the second convolution feature vector is linearly interpolated to obtain a linearly interpolated feature vector, and then all key point descriptor information of the oblique aerial image is obtained based on the linearly interpolated feature vector.

[0038] In a possible design, the following encoding formula is used to encode the image key point information set Info I Image key point information or the point cloud key point set Info C The i-th point cloud key point information in is encoded:

[0039]

[0040] Where, X i Is the image key point information set Info I The encoded image feature information corresponding to the i-th image key point information or the point cloud key point set Info CThe encoded point cloud feature information corresponding to the i-th point cloud key point information; dec i Is the image key point information set Info I The key point descriptor information corresponding to the i-th image key point information or the point cloud key point set Info C The key point descriptor information corresponding to the key point information of the i-th point cloud; Pos i Is the image key point information set Info I The i-th image key point information or the point cloud key point set Info C The key point position information corresponding to the key point information of the i-th point cloud; The key point position information Pos obtained through MLP i The encoding result.

[0041] In a possible design, the encoded image feature information set is respectively The encoded image feature information and the encoded point cloud feature information set in Update the encoded point cloud feature information in the image to obtain the updated image feature information set (L) X 2D And the updated point cloud feature information set (L) X 3D , and respectively according to the updated image feature information set (L) X 2D And the updated point cloud feature information set (L) X 3D Get the final image key point descriptor information set Dec 2D And the final point cloud key point descriptor information set Dec 3D ,include:

[0042] According to the encoded image feature information set Construct a first undirected graph, and in the first undirected graph, perform the encoded image feature information collection through the self-attention mechanism The information between the encoded image feature information is transferred to obtain the first self-attention information;

[0043] According to the encoded point cloud feature information set Construct a second undirected graph, and in the second undirected graph, perform the encoded point cloud feature information collection through the self-attention mechanism The information between the encoded point cloud feature information is transferred to obtain the second self-attention information;

[0044] According to the encoded image feature information set And the encoded point cloud feature information set Construct a third undirected graph, and in the third undirected graph, perform the encoded image feature information collection through a cross attention mechanism The encoded image feature information and the encoded point cloud feature information set The information between the encoded point cloud feature information is transferred to obtain cross attention information;

[0045] Aggregate the first self-attention information and the cross-attention information to obtain the encoded image feature information set Corresponding updated image feature information set (L) X 2D ; and aggregate the second self-attention information and the cross-attention information to obtain the encoded point cloud feature information set Corresponding updated point cloud feature information set (L) x 3D ;

[0046] The updated image feature information set (L) X 2D Perform linear projection on the updated image feature information in to obtain the updated image feature information set (L) X 2D The final image key point descriptor information set Dec corresponding to the image key point information 2D ; and the updated point cloud feature information set (L) X 3D Perform linear projection on the updated point cloud feature information in to obtain the updated point cloud feature information set (L) X 3D The final point cloud key point descriptor information set Dec corresponding to the point cloud key point information 3D .

[0047] In a possible design, according to the final image key point descriptor information set Dec 2D And the final point cloud key point descriptor information set Dec 3D , obtain the matching matrix of the airborne LiDAR point cloud and the oblique aerial image, including:

[0048] Calculate the final image key point descriptor information set Dec 2D And the final point cloud key point descriptor information set Dec 3D The cosine value between

[0049] According to the cosine value, the final image key point descriptor information set Dec is constructed. 2DAnd the final point cloud key point descriptor information set Dec 3D The correlation matrix Cor between them; wherein the cosine value Cor of the i-th row and j-th column in the correlation matrix Cor i,j for:

[0050]

[0051] Where, The final image key point descriptor information set Dec 2D The i-th final image key point descriptor information in, The final point cloud key point descriptor information set Dec 3D The jth final point cloud key point descriptor information in;

[0052] According to the correlation matrix Cor, multiple iterative matching is performed on the point cloud key points of the airborne LiDAR point cloud and the image key points in the oblique aerial image to obtain a matching matrix between the airborne LiDAR point cloud and the oblique aerial image.

[0053] In one possible design, the Sinkhorn algorithm is used to perform multiple iterative matching on the point cloud key points of the airborne LiDAR point cloud and the image key points in the oblique aerial image.

[0054] In a second aspect, the present invention provides a system for matching oblique aerial images with airborne LIDAR point cloud features, for implementing any of the above-described methods for matching oblique aerial images with airborne LIDAR point cloud features; the system for matching oblique aerial images with airborne LIDAR point cloud features comprises:

[0055] The key point information extraction module is used to obtain the point cloud key point information of the airborne LiDAR point cloud and the image key point information of the oblique aerial image, and save the point cloud key point information of the airborne LiDAR point cloud to the point cloud key point set Info C In the image key point information of the oblique aerial image, the image key point information set Info is saved. I middle;

[0056] The information encoding module is connected to the key point information extraction module for respectively encoding the image key point information set Info I Image key point information and the point cloud key point set Info C The key point information of the point cloud is encoded and processed to obtain the encoded image feature information set And the encoded point cloud feature information set

[0057] The attention aggregation module is in communication with the information encoding module and is used to respectively aggregate the encoded image feature information based on the attention aggregation mechanism. The encoded image feature information and the encoded point cloud feature information set in Update the encoded point cloud feature information in the image to obtain the updated image feature information set (L) X 2D And the updated point cloud feature information set (L) X 3D , and respectively according to the updated image feature information set (L) X 2D And the updated point cloud feature information set (L) X 3D Get the final image key point descriptor information set Dec 2D And the final point cloud key point descriptor information set Dec 3D ;

[0058] The key point matching module is connected to the attention aggregation module for matching the key point descriptor information set Dec of the final image. 2D And the final point cloud key point descriptor information set Dec 3D , and obtain the matching matrix of the airborne LiDAR point cloud and the oblique aerial image.

[0059] In a third aspect, the present invention provides an electronic device, comprising:

[0060] a memory for storing computer program instructions; and

[0061] A processor is configured to execute the computer program instructions to thereby complete the operation of the method for matching oblique aerial images with airborne LIDAR point cloud features as described in any one of the above items.

[0062] In a fourth aspect, the present invention provides a computer-readable storage medium for storing computer program instructions readable by a computer, wherein the computer program instructions are configured to execute the operations of the method for matching oblique aerial images with airborne LIDAR point cloud features as described in any one of the above items when run. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 is a module block diagram of an FPN network in an embodiment;

[0064] Figure 2 is a structural diagram of an image key point detection network in an embodiment;

[0065] Figure 3 is a schematic diagram of the structure of attention aggregation in the embodiment;

[0066] Figure 4 is a module block diagram of a feature matching module in an embodiment;

[0067] Figure 5 3 is a block diagram of a system for matching features of oblique aerial images and airborne LIDAR point clouds in an embodiment. DETAILED DESCRIPTION

[0068] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the present invention will be briefly introduced below in conjunction with the drawings and the description of the embodiments or the prior art. Obviously, the following description of the structure of the drawings is only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. It should be noted that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation of the present invention.

[0069] Example 1:

[0070] This embodiment discloses a method for matching features between oblique aerial images and airborne LiDAR point clouds, which can be performed, but is not limited to, by a computer device or virtual machine with certain computing resources, such as a personal computer, smart phone, personal digital assistant, or wearable device, or by a virtual machine.

[0071] As shown, a method for matching oblique aerial images with airborne LiDAR point cloud features may include, but is not limited to, the following steps:

[0072] S1. Obtain the key point information of the airborne LiDAR point cloud and the key point information of the tilted aerial image, and save the key point information of the airborne LiDAR point cloud to the point cloud key point set Info C ={dec 3D ,Pos 3D ,s 3D}, save the image key point information of the oblique aerial image to the image key point information set Info I ={dec 2D ,Pos 2D ,s 2D}; It should be noted that, in this embodiment, the point cloud key point information of the airborne LiDAR point cloud and the image key point information of the oblique aerial image both include key point descriptor information, key point location information and confidence, wherein the point cloud key point set Info C dec in 3D ,Pos 3D ,s 3DRespectively refer to the key point descriptor information, key point location information and confidence in the point cloud key point information of the airborne LiDAR point cloud. Accordingly, the image key point information set Info I dec in 2D ,Pos 2D ,s 2D They respectively refer to the key point descriptor information, key point position information and confidence in the image key point information of the oblique aerial image.

[0073] In this embodiment, the FPN (Feature Proposal Network) network is used to obtain the key point information of the airborne LiDAR point cloud. The structure of the FPN network is shown in the figure below. Figure 1 As shown in the figure, the FPN network is used to obtain the key point information of the airborne LiDAR point cloud, including:

[0074] S101a. Initialize nodes: Use Farthest Point Sampling (FPS) to randomly sample M nodes S1,…,S in the airborne LiDAR point cloud. M , and store M nodes in the node set S; where the node set Among them, R N×3 is the set of airborne LiDAR point clouds, and N is the number of airborne LiDAR point clouds;

[0075] S102a. Point cloud clustering: traverse the airborne LiDAR point cloud, i.e., the original point cloud, and select the node closest to each 3D point in the airborne LiDAR point cloud from the node set S to obtain a clustering result of M nodes; wherein, node S k The clustering result is S k Represents the kth node in the node set S, k∈{1,…,M}, n k Indicates the airborne LiDAR point cloud with node S k The number of corresponding three-dimensional points; it should be noted that, with the node S k The corresponding three-dimensional point is the point in the airborne LiDAR point cloud corresponding to the node S k All the three-dimensional points with the shortest distance between them, that is, the distance between them and other nodes in the node set S is greater than the distance between them and node S k The distance between them.

[0076] S103a. Point cloud coordinate regularization: In order to ensure the translation invariance of the point cloud, the original point cloud is coordinate regularized. Specifically, the mean coordinates of all clustering results are obtained, and the regularized coordinates of the three-dimensional point cloud corresponding to each node are obtained based on the original coordinates of the three-dimensional points in the clustering result of each node and the mean coordinates of its corresponding clustering result; wherein, the regularized coordinates of the three-dimensional point cloud corresponding to any node are the regularized coordinates of all three-dimensional points in the three-dimensional point cloud corresponding to the current node; specifically, the mean coordinates of the clustering result of any node are the mean of the original coordinates of all three-dimensional points in the current clustering result.

[0077] Specifically, in step S103a, the normalized coordinates of the three-dimensional point cloud corresponding to each node are the difference between the original coordinates of the three-dimensional point in the clustering result of each node and the mean coordinates of its corresponding clustering result; k The corresponding regularized coordinates of the i-th three-dimensional point are: Is the node S k The original coordinates of the corresponding i-th three-dimensional point, It is node S k The mean coordinates of the clustering results.

[0078] S104a. Calculate local feature vectors: According to the regularized coordinates of the three-dimensional point cloud corresponding to each node, obtain the local features of each node, and then perform vector aggregation operation on the local features of each node to obtain M node features G (M×C) , where C is the dimension of node features; where node S k The local feature is F k |S k ; Specifically, in this embodiment, the original coordinates of all three-dimensional point clouds are updated according to the regularized coordinates of the three-dimensional point cloud corresponding to each node, and the three-dimensional point cloud after the updated coordinates is used as the clustered point cloud; in this embodiment, the clustered point cloud is input into a model similar to PointNet (64, 64, 64) (a new deep learning model for processing point cloud data), and the local features of each node are calculated by the PointNet (64, 64, 64) model.

[0079] S105a. Hierarchical information fusion: According to the node feature G (M×C) Get the feature information of K neighboring nodes of each node; among them, node S k The feature information of K neighboring nodes is Represents node S k The first neighbor of Represents node S k The feature information of the first neighboring point of Represents node S kThe Kth neighboring point of Represents node S k Specifically, in this embodiment, the KNN (k-Nearest Neighbor, K nearest neighbor classification) algorithm is used to perform KNN calculation on each node, thereby obtaining the feature information of K neighboring nodes of each node.

[0080] S106a. Vector coordinate regularization: Regularize the feature information of the K neighboring nodes of each node to obtain the regularized feature information of the K neighboring nodes of each node; where the node S k The regularized feature information of the K neighboring nodes is Represents node S k The normalized coordinates of the mth neighbor of , It is node S k The original coordinates of the mth neighbor value;

[0081] S107a. Obtain the point cloud key point information of the airborne LiDAR point cloud according to the regularized feature information of the K neighboring nodes of each node, that is, the final point cloud key point information.

[0082] Specifically, in step S107a, the point cloud key point information of the airborne LiDAR point cloud is obtained based on the regularized feature information of the K neighboring nodes of each node, including:

[0083] According to the regularized feature information of K neighboring nodes of each node, the convolution feature vector set {H1,…,H M};

[0084] According to the convolution feature vector set {H1,…,H M}Get the key point coordinate matrix of each node and significant uncertainty;

[0085] According to the key point coordinate matrix The normalized coordinates of each node in and the mean coordinates of the corresponding clustering results are combined to obtain the point cloud key point information of the airborne LiDAR point cloud, that is, the final point cloud key point information. Specifically, in this embodiment, the normalized feature information of the K neighboring nodes of each node is input into the PointNet model, so that the convolution feature vector set {H1,…,H M}, then obtain the key point coordinate matrix of each node through MLP (128, 256, 256) (Multilayer Perceptron) and the significant uncertainty vector ∑={δ1,δ2,…,δ M}.

[0086] In step S107a, the key point information of the airborne LiDAR point cloud includes the key point position information of the airborne LiDAR point cloud; the key point position information of the airborne LiDAR point cloud is the key point coordinate matrix The sum of the regularized coordinates of each node and the mean coordinates of the corresponding clustering results; where node S k The corresponding key point position information is is the key point coordinate matrix Middle node S k The regularized coordinates of

[0087] In this embodiment, the image key point information of the oblique aerial image includes key point position information and key point descriptor information of the oblique aerial image; wherein, obtaining the image key point information of the oblique aerial image includes:

[0088] S101b performs feature encoding on the oblique aerial image to obtain an encoded feature vector;

[0089] S102b performs a first convolution operation on the encoded feature vector to obtain a first convolution feature vector, and calculates the probability that each feature point in the first convolution feature vector is an image feature, so as to obtain the position information of all key points of the oblique aerial image;

[0090] S103b. The encoded feature vector is subjected to a second convolution operation to obtain a second convolution feature vector, and the second convolution feature vector is linearly interpolated to obtain a linearly interpolated feature vector, and then all key point descriptor information of the oblique aerial image is obtained based on the linearly interpolated feature vector.

[0091] It should be noted that, in this embodiment, a preset image key point detection network is used to obtain the image key point information of the oblique aerial image, wherein the image key point detection network includes a feature encoding network and a feature vector decoding network connected in sequence, wherein the feature vector decoding network includes a feature point decoding network and a descriptor decoding network respectively connected to the feature encoding network; specifically, the structure diagram of the image key point detection network is shown in FIG. Figure 2 Specifically, in this embodiment, in step S101b, the feature encoding network is used to perform feature encoding processing on the oblique aerial image with a size of H×W, so as to obtain a shape of The encoded feature vector is then input into the feature point decoding network and the descriptor decoding network respectively; in step S102b, the encoded feature vector is subjected to a first convolution operation by the feature point decoding network, so as to add a channel for storing the position coordinates of the feature-free points on the basis of the encoded feature vector, and a first convolution feature vector is obtained, and the probability of each feature point in the first convolution feature vector being an image feature is calculated by the Softmax function, thereby obtaining the position information of all key points of the oblique aerial image, such as the coordinates of the feature points whose probability of being an image feature is greater than the probability threshold; in step S103b, the encoded feature vector is subjected to a second convolution operation by the descriptor decoding network, and a shape of The second convolution feature vector is obtained by linearly interpolating the second convolution feature vector to obtain a linearly interpolated feature vector, the size of the linearly interpolated feature vector is restored to H×W×256, and then all key point descriptor information of the oblique aerial image is obtained according to the linearly interpolated feature vector, where the vector dimension of the descriptor information is 256.

[0092] S2. respectively for the image key point information set Info I The key point position information and key point descriptor information of the image key point information and the point cloud key point set Info C The key point position information and key point descriptor information of the point cloud key point information in the encoding process are processed to obtain the encoded image feature information set And the encoded point cloud feature information set

[0093] In this embodiment, the following encoding formulas are used to encode the image key point information set Info I Image key point information or the point cloud key point set Info C The i-th point cloud key point information in is encoded:

[0094]

[0095] Where, X i Is the image key point information set Info I The encoded image feature information corresponding to the i-th image key point information or the point cloud key point set Info C The encoded point cloud feature information corresponding to the i-th point cloud key point information in is 256 in dimension, which includes the key point position information and key point descriptor information, so that the similarity between the key point position information and the key point descriptor information can be considered simultaneously in the subsequent matching process; iIs the image key point information set Info I The key point descriptor information corresponding to the i-th image key point information or the point cloud key point set Info C The key point descriptor information corresponding to the key point information of the i-th point cloud; Pos i Is the image key point information set Info I The i-th image key point information or the point cloud key point set Info C The key point position information corresponding to the i-th point cloud key point information, specifically, for the point cloud key point set Info C The three-dimensional key points corresponding to the point cloud key point information in the 3D point cloud coordinate system are represented by the xoy plane coordinates in the 3D point cloud coordinate system, that is, Where x represents the horizontal coordinate of the three-dimensional key point, and y represents the vertical coordinate of the three-dimensional key point; for the image key point information set Info I The two-dimensional key points corresponding to the image key point information in the image coordinate system are represented by the row and column numbers in the image coordinate system, that is, Among them, u represents the row number of the two-dimensional key point, and v represents the column number of the two-dimensional key point; The key point position information Pos is obtained through MLP (Multilayer Perceptron) i Specifically, in this embodiment, the image key point information set Info is encoded using the MLP method. I Image key point information or the point cloud key point set Info C The key point position information corresponding to the point cloud key point information in is encoded into a high-dimensional vector.

[0096] S3. Based on the attention aggregation mechanism, the encoded image feature information is respectively collected The encoded image feature information and the encoded point cloud feature information set in Update the encoded point cloud feature information in the image to obtain the updated image feature information set (L) X 2D And the updated point cloud feature information set (L) X 3D , and respectively according to the updated image feature information set (L) X 2D And the updated point cloud feature information set (L) X 3D Get the final image key point descriptor information set Dec 2D And the final point cloud key point descriptor information set Dec 3D .

[0097] like Figure 3 As shown, in step S3, the encoded image feature information set is respectively The encoded image feature information and the encoded point cloud feature information set in Update the encoded point cloud feature information in the image to obtain the updated image feature information set (L) X 2D And the updated point cloud feature information set (L) X 3D , and respectively according to the updated image feature information set (L) X 2D And the updated point cloud feature information set (L) X 3D Get the final image key point descriptor information set Dec 2D And the final point cloud key point descriptor information set Dec 3D ,include:

[0098] S301. According to the encoded image feature information set Construct a first undirected graph, and in the first undirected graph, perform the encoded image feature information collection through the self-attention mechanism The information transmission between the encoded image feature information is obtained by The first self-attention information corresponding to the image key point information;

[0099] S302. According to the encoded point cloud feature information set Construct a second undirected graph, and in the second undirected graph, perform the encoded point cloud feature information collection through the self-attention mechanism The information transmission between the encoded point cloud feature information is obtained by The second self-attention information corresponding to the key point information of the point cloud;

[0100] Specifically, as an example, the encoded image feature information set For example, the encoded image feature information set Any image key point information i in the first undirected graph ε self Self-attention residual information for:

[0101]

[0102]

[0103]

[0104] Where, α i is the preset self-attention weight, q i , k i , v i They are respectively the encoded image feature information X corresponding to the i-th image key point information i After the projection matrix W (l) The vector used for matching obtained by projection calculation is the query vector, called vector query; the vector to be matched is called vector key; the vector used to describe information is called vector value. The local key point information corresponding to different key point position information has different confidence levels, and the weight vector W can be constructed based on this confidence level. The weight vector W is added to the attention aggregation process to adjust the retrieval and transmission ratio of the local key point information corresponding to different key point position information; d k Indicates the dimension of the vector key, as a scale factor; b k , b q , b v The bias of the linear calculation is used as a hyperparameter for network training. The same method can be used to set the encoded point cloud feature information The self-attention information is calculated based on the key point information of the corresponding point cloud.

[0105] S303. According to the encoded image feature information set And the encoded point cloud feature information set Construct a third undirected graph, specifically, the third undirected graph is an undirected graph between the encoded image feature information and the encoded point cloud feature information, and in the third undirected graph, the encoded image feature information set is performed through the cross attention mechanism The encoded image feature information and the encoded point cloud feature information set The information between the encoded point cloud feature information is transferred to obtain cross attention information; Specifically, similar to the multi-head self-attention mechanism in steps S301 and S302, in step S303, the encoded image feature information set The encoded image feature information and the encoded point cloud feature information set Information is transferred between the encoded point cloud feature information in order to obtain cross-attention information.

[0106] Specifically, as an example, the encoded image feature information set For example, the encoded image feature information set Any key point information i in the third undirected graph ε corss Cross attention residual information for:

[0107]

[0108]

[0109]

[0110]

[0111] α i is the cross attention weight, q i is the encoded image feature information X corresponding to the i-th key point information of the image i After the projection matrix W (l) The vector used for matching obtained by projection calculation; k j is the feature information X of the jth key point of the point cloud j Separate the vector for matching; v j is the feature information X of the jth key point of the point cloud j Separate the vector used to describe the information. Similar to self-attention, the local key point information corresponding to different key point position information has different confidence levels, and the weight vector W can be constructed based on this confidence level; d k Represents vector k j The dimension of b, as a scale factor; k , b q , b v It also represents the bias of the linear calculation as a hyperparameter for network training.

[0112] S304. Aggregate the first self-attention information and the cross-attention information to obtain the encoded image feature information set Corresponding updated image feature information set (L) X 2D ; and aggregate the second self-attention information and the cross-attention information to obtain the encoded point cloud feature information set Corresponding updated point cloud feature information set (L) x 3D ;

[0113] Specifically, in this embodiment, m ε→i According to the value of l, the i-th updated image feature information in the updated image feature information set is:

[0114]

[0115] Where, is the encoded image feature information corresponding to the i-th image key point information on the l-th layer image, [·||·] represents the concatenation operation, that is, l starts from 1. When l is an odd number, the encoded image feature information set is calculated by MLP The first self-attention residual information corresponding to the image key point information When l is an even number, the cross attention residual information is calculated by MLP

[0116] The i-th updated point cloud feature information in the updated point cloud feature information set is:

[0117]

[0118] Where, is the encoded point cloud feature information corresponding to the i-th point cloud key point information on the l-th layer point cloud, [·||·] represents the serial operation, that is, l starts from 1, when l is an odd number, the encoded point cloud feature information set is calculated by MLP The second self-attention residual information corresponding to the key point information of the point cloud. When l is an even number, the cross-attention residual information is calculated by MLP.

[0119] S305. The updated image feature information set (L) X 2D Perform linear projection on the updated image feature information in to obtain the updated image feature information set (L) X 2D The final image key point descriptor information set Dec corresponding to the image key point information 2D ; and the updated point cloud feature information set (L) X 3D Perform linear projection on the updated point cloud feature information in to obtain the updated point cloud feature information set (L) X 3D The final point cloud key point descriptor information set Dec corresponding to the point cloud key point information 3D Specifically, in this embodiment, the final image key point descriptor information set Dec 2D The i-th final image key point descriptor information in for:

[0120]

[0121] Where, The updated image feature information set (L) X 2D The i-th updated image feature information in

[0122] Final point cloud key point descriptor information set Dec 3D The i-th final point cloud key point descriptor information in for:

[0123]

[0124] Where, is the updated point cloud feature information set (L) X 3D The i-th updated point cloud feature information in

[0125] It should be noted that, in this embodiment, the first self-attention information and the cross-attention information are aggregated, and the second self-attention information and the cross-attention information are aggregated, which can make the final matching result more reliable.

[0126] S4. According to the final image key point descriptor information set Dec 2D And the final point cloud key point descriptor information set Dec 3D , get the matching matrix M∈[0,1] of the airborne LiDAR point cloud and the oblique aerial image m×n In this embodiment, the final image key point descriptor information set Dec is obtained. 2D And the final point cloud key point descriptor information set Dec 3D The correlation matrix Cor between them, and the matching matrix M∈[0,1] is obtained according to the correlation matrix Cor. m×n .

[0127] In step S4 of this embodiment, according to the final image key point descriptor information set Dec 2D And the final point cloud key point descriptor information set Dec 3D , get the matching matrix M∈[0,1] of the airborne LiDAR point cloud and the oblique aerial image m×n ,include:

[0128] S401. Calculate the final image key point descriptor information set Dec 2D And the final point cloud key point descriptor information set Dec 3D The cosine value between

[0129] S402. According to the cosine value, construct the final image key point descriptor information set Dec 2D And the final point cloud key point descriptor information set Dec 3D The correlation matrix Cor between the two key point descriptor information sets is used to describe the similarity or matching relationship between them; wherein the cosine value Cor of the i-th row and j-th column in the correlation matrix Cor is i,j for:

[0130]

[0131] Where, The final image key point descriptor information set Dec 2D The i-th final image key point descriptor information in, The final point cloud key point descriptor information set Dec 3D The jth final point cloud key point descriptor information in;

[0132] S403. According to the correlation matrix Cor, perform multiple iterative matching on the key points of the airborne LiDAR point cloud and the key points of the image in the oblique aerial image to obtain the matching matrix M∈[0,1] of the airborne LiDAR point cloud and the oblique aerial image. m×n .

[0133] Specifically, in this embodiment, the key point matching is transformed into a bipartite graph matching problem, and the Sinkhorn algorithm is used to solve the iterative matching problem of the correlation matrix Cor. Specifically, the Sinkhorn algorithm iteratively updates the elements in the correlation matrix Cor so that it gradually approaches the probability distribution that satisfies the constraint conditions. Based on the correlation matrix Cor, the point cloud key points of the airborne LiDAR point cloud and the image key point description information in the oblique aerial image are iteratively matched multiple times to obtain the matching matrix M∈[0,1] of the airborne LiDAR point cloud and the oblique aerial image. m×n , calculated as follows:

[0134]

[0135]

[0136] Due to the sparsity of point cloud data, the image key points in the oblique aerial image do not have corresponding matching points in the airborne LiDAR point cloud. In addition, the camera posture during shooting causes the point cloud key points in the airborne LiDAR point cloud to have no corresponding matching points in the oblique aerial image. Therefore, a column and a row are added to the correlation matrix Cor as a trash can channel d, and the key points that theoretically have no matching relationship are placed in the trash can channel. The module block diagram of the feature matching module is shown as follows: Figure 4 As shown in Figure 2, the trash can channel parameter is a learnable variable parameter, and the matching matrix represents the matching relationship between the image and the point cloud key points.

[0137] The loss function Loss is defined by the Sinkhorn distance to find the best matching matrix that satisfies the following formula:

[0138] Loss=d(M p ,Cor)-d(M t , Cor)

[0139]

[0140]

[0141] Where h(M) is the matching matrix M∈[0,1] m×n The information entropy can make the matching more uniform, and each feature point has a fair chance of matching. Parameters are used to adjust the weight relationship of information; M p is the matching matrix of each round, M t is the true value of the matching matrix.

[0142] M∈U(r,c) represents the set of matching matrices between the key points of the image to be matched in the oblique aerial image and the key points of the point cloud to be matched on the airborne LiDAR cloud. Specifically, the vector corresponding to the key point of the image to be matched in the oblique aerial image is defined as 1 m = r (an m-dimensional vector with 1 elements), where the vector corresponding to the key point of the point cloud to be matched on the airborne LiDAR cloud is 1 n = c (an n-dimensional vector with 1 elements), then the matching possibility set of the key points of the image to be matched in the oblique aerial image and the key points of the point cloud to be matched on the airborne LiDAR cloud can be obtained as follows:

[0143]

[0144] Sinkhorn iterative calculation is used to obtain the optimal matching model between image feature points and point cloud feature points.

[0145] This embodiment can solve the problem of feature matching between aerial oblique images and airborne LiDAR point clouds, avoid the loss of matching accuracy caused by dimensional changes in airborne LiDAR point clouds or aerial oblique images, and improve the reliability of feature matching results. Specifically, this embodiment performs point feature detection on airborne LiDAR point clouds and aerial oblique images to obtain point cloud key point information of airborne LiDAR point clouds and image key point information of oblique aerial images, and saves the point cloud key point information of the airborne LiDAR point clouds to the point cloud key point set Info. C In the image key point information of the oblique aerial image, the image key point information set Info is saved. I Then, the image key point information set Info I Image key point information and the point cloud key point set Info C The key point information of the point cloud is encoded and processed to obtain the encoded image feature information set And the encoded point cloud feature information set And based on the attention aggregation mechanism, the encoded image feature information is respectively collected The encoded image feature information and the encoded point cloud feature information set in Update the encoded point cloud feature information in the image to obtain the updated image feature information set (L) X 2D And the updated point cloud feature information set (L) X 3D , and respectively according to the updated image feature information set (L) X 2D And the updated point cloud feature information set (L) X 3D Get the final image key point descriptor information set Dec 2D And the final point cloud key point descriptor information set Dec 3D ; According to the final image key point descriptor information set Dec 2D And the final point cloud key point descriptor information set Dec 3D , get the matching matrix M∈[0,1] of the airborne LiDAR point cloud and the oblique aerial image m×n , and then obtain the matching matrix, that is, the matching relationship between the airborne LiDAR point cloud and the oblique aerial image. In this process, this embodiment obtains the matching matrix by I Image key point information and the point cloud key point set Info C The encoding process of the point cloud key point information in the image can provide geometric information for subsequent key point matching; in addition, this embodiment proposes a method for matching oblique aerial images and airborne LiDAR point cloud features based on an attention aggregation mechanism, which directly establishes a correspondence between two-dimensional and three-dimensional points, and can solve the problem of difficult matching between aerial oblique images and airborne LiDAR point clouds, and can make the final matching result more accurate.

[0146] Specifically, in this embodiment, the image key point information set Info is processed in step 2. I The key point position information and key point descriptor information of the image key point information and the point cloud key point set Info C The key point position information and key point descriptor information of the point cloud key point information in the encoding process are processed to obtain the encoded image feature information set And the encoded point cloud feature information set Thus, the position information of the image key points and the position information of the point cloud key points are encoded into the feature vector, providing geometric information for subsequent matching. In addition, this embodiment addresses the problem of difficulty in matching aerial oblique images with airborne LiDAR point clouds, and proposes a feature matching method for oblique aerial images and airborne LiDAR point clouds based on an attention aggregation mechanism, which directly establishes a correspondence between two-dimensional and three-dimensional. Specifically, based on the feature encoding in step S2, this embodiment constructs an undirected graph on the image key points (first undirected graph), an undirected graph on the point cloud key points (second undirected graph), and an undirected graph between the image key points and the point cloud key points (third undirected graph) respectively through step S3. On the first undirected graph and the second undirected graph, information transfer and information aggregation are carried out between image key points and image key points, and between point cloud key points and point cloud key points; in the third undirected graph, information transfer and information aggregation are carried out between image key points and point cloud key points through cross attention. This embodiment performs iterative information transmission and information aggregation through an attention aggregation mechanism including a self-attention mechanism and a cross-attention mechanism, which can make the final matching result more reliable and is suitable for complex scenarios with large amounts of data and application scenarios with high precision requirements.

[0147] Example 2:

[0148] This embodiment discloses a system for matching features of oblique aerial images and airborne LIDAR point clouds, which is used to implement the method for matching features of oblique aerial images and airborne LIDAR point clouds in embodiment 1; Figure 5 As shown, the oblique aerial image and airborne LIDAR point cloud feature matching system includes:

[0149] The key point information extraction module is used to obtain the point cloud key point information of the airborne LiDAR point cloud and the image key point information of the oblique aerial image, and save the point cloud key point information of the airborne LiDAR point cloud to the point cloud key point set Info C In the image key point information of the oblique aerial image, the image key point information set Info is saved. I middle;

[0150] The information encoding module is connected to the key point information extraction module for respectively encoding the image key point information set Info I Image key point information and the point cloud key point set Info C The key point information of the point cloud is encoded and processed to obtain the encoded image feature information set And the encoded point cloud feature information set

[0151] The attention aggregation module is connected to the information encoding module and is used to respectively aggregate the encoded image feature information based on the attention aggregation mechanism. The encoded image feature information and the encoded point cloud feature information set in Update the encoded point cloud feature information in the image to obtain the updated image feature information set (L) X 2D And the updated point cloud feature information set (L) X 3D , and respectively according to the updated image feature information set (L) X 2D And the updated point cloud feature information set (L) X 3D Get the final image key point descriptor information set Dec 2D And the final point cloud key point descriptor information set Dec 3D ;

[0152] The key point matching module is connected to the attention aggregation module for matching the key point descriptor information set Dec of the final image. 2D And the final point cloud key point descriptor information set Dec 3D , and obtain the matching matrix of the airborne LiDAR point cloud and the oblique aerial image.

[0153] Example 3:

[0154] Based on Embodiment 1 or 2, this embodiment discloses an electronic device, which may be a smartphone, tablet computer, laptop computer, or desktop computer. The electronic device may be referred to as a terminal, portable terminal, desktop terminal, etc. The electronic device includes:

[0155] a memory for storing computer program instructions; and

[0156] A processor is configured to execute the computer program instructions to thereby perform the operations of the method for matching oblique aerial images with airborne LIDAR point cloud features as described in any one of the first embodiments.

[0157] Example 4:

[0158] Based on any one of Examples 1 to 3, this embodiment discloses a computer-readable storage medium for storing computer-readable computer program instructions, wherein the computer program instructions are configured to execute the operations of the method for matching features of oblique aerial images and airborne LIDAR point clouds as described in Example 1 when running.

[0159] Obviously, those skilled in the art will appreciate that the modules or steps of the present invention described above can be implemented using a general-purpose computing device. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Alternatively, they can be implemented using program code executable by a computing device, which can then be stored in a storage device and executed by the computing device. Alternatively, they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0160] Finally, it should be noted that the above embodiments are intended only to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art will appreciate that modifications may be made to the technical solutions described in the above embodiments, or that some of the technical features may be replaced with equivalents. Such modifications or replacements do not deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for matching oblique aerial images with airborne LiDAR point cloud features, characterized by: include: Get the key point information of the airborne LiDAR point cloud and the key point information of the oblique aerial image respectively, and save the key point information of the airborne LiDAR point cloud to the point cloud key point set Info C In the image key point information of the oblique aerial image, the image key point information set Info is saved. I middle; The image key point information set Info I Image key point information and the point cloud key point set Info C The key point information of the point cloud is encoded and processed to obtain the encoded image feature information set And the encoded point cloud feature information set Based on the attention aggregation mechanism, the encoded image feature information is respectively The encoded image feature information and the encoded point cloud feature information set in Update the encoded point cloud feature information in the image to obtain the updated image feature information set (L) X 2D And the updated point cloud feature information set (L) X 3D , and respectively according to the updated image feature information set (L) X 2D And the updated point cloud feature information set (L) X 3D Get the final image key point descriptor information set Dec 2D And the final point cloud key point descriptor information set Dec 3D ; According to the final image key point descriptor information set Dec 2D And the final point cloud key point descriptor information set Dec 3D , and obtain the matching matrix of the airborne LiDAR point cloud and the oblique aerial image.

2. The method for matching oblique aerial images with airborne LiDAR point cloud features according to claim 1, characterized in that: Obtain key point information of the airborne LiDAR point cloud, including: Using the longest distance sampling, randomly sample M nodes S1,…,S in the airborne LiDAR point cloud. M , and store M nodes in the node set S; where the node set S={S1,…,S M }; Traverse the airborne LiDAR point cloud, select the node closest to each 3D point in the airborne LiDAR point cloud from the node set S, and obtain a clustering result of M nodes; where node S k The clustering result is S k Represents the kth node in the node set S, k∈{1,…,M}, n k Indicates the airborne LiDAR point cloud with node S k The number of corresponding 3D points; Get the mean coordinates of all clustering results, and obtain the regularized coordinates of the 3D point cloud corresponding to each node based on the original coordinates of the 3D points in the clustering results of each node and the mean coordinates of its corresponding clustering results; where the regularized coordinates of the 3D point cloud corresponding to any node are the regularized coordinates of all 3D points in the 3D point cloud corresponding to the current node; According to the regularized coordinates of the three-dimensional point cloud corresponding to each node, the local features of each node are obtained, and then the local features of each node are vector aggregated to obtain M node features G (M×C) ; Among them, node S k The local feature is F k |S k ; According to the node feature G (M×C) Get the feature information of K neighboring nodes of each node; among them, node S k The feature information of K neighboring nodes is Represents node S k The first neighbor of Represents node S k The feature information of the first neighboring point, Represents node S k The Kth neighboring point of Represents node S k Feature information of the Kth neighboring point; Regularize the feature information of the K neighboring nodes of each node to obtain the regularized feature information of the K neighboring nodes of each node; k The regularized feature information of the K neighboring nodes is Represents node S k The normalized coordinates of the mth neighbor of , It is node S k The original coordinates of the mth neighbor value; The point cloud key point information of the airborne LiDAR point cloud is obtained according to the regularized feature information of K neighboring nodes of each node.

3. The method for matching oblique aerial images with airborne LiDAR point cloud features according to claim 2, characterized in that: The regularized coordinates of the three-dimensional point cloud corresponding to each node are the difference between the original coordinates of the three-dimensional point in the clustering result of each node and the mean coordinates of its corresponding clustering result; k The corresponding regularized coordinates of the i-th three-dimensional point are: Is the node S k The original coordinates of the corresponding i-th three-dimensional point, It is node S k The mean coordinates of the clustering results.

4. The method for matching oblique aerial images with airborne LiDAR point cloud features according to claim 2, characterized in that: The point cloud key point information of the airborne LiDAR point cloud is obtained according to the regularized feature information of the K neighboring nodes of each node, including: According to the regularized feature information of K neighboring nodes of each node, the convolution feature vector set {H1,…,H M }; According to the convolution feature vector set {H1,…,H M }Get the key point coordinate matrix of each node According to the key point coordinate matrix The regularized coordinates of each node in and the mean coordinates of the corresponding clustering results are used to obtain the point cloud key point information of the airborne LiDAR point cloud.

5. The method for matching oblique aerial images with airborne LiDAR point cloud features according to claim 4, characterized in that: The key point information of the airborne LiDAR point cloud includes the key point position information of the airborne LiDAR point cloud; the key point position information of the airborne LiDAR point cloud is the key point coordinate matrix The sum of the regularized coordinates of each node and the mean coordinates of the corresponding clustering results; where node S k The corresponding key point position information is is the key point coordinate matrix Middle node S k The regularized coordinates of 6. The method for matching oblique aerial images with airborne LiDAR point cloud features according to claim 1, characterized in that: The image key point information of the oblique aerial image includes key point position information and key point descriptor information of the oblique aerial image; wherein obtaining the image key point information of the oblique aerial image includes: performing feature coding processing on the oblique aerial image to obtain a coded feature vector; Performing a first convolution operation on the encoded feature vector to obtain a first convolution feature vector, and calculating the probability that each feature point in the first convolution feature vector is an image feature, so as to obtain position information of all key points of the oblique aerial image; The encoded feature vector is subjected to a second convolution operation to obtain a second convolution feature vector, and the second convolution feature vector is linearly interpolated to obtain a linearly interpolated feature vector, and then all key point descriptor information of the oblique aerial image is obtained based on the linearly interpolated feature vector.

7. The method for matching oblique aerial images with airborne LiDAR point cloud features according to claim 1, characterized in that: The following encoding formula is used to encode the image key point information set Info I Image key point information or the point cloud key point set Info C The i-th point cloud key point information in is encoded: Where, X i Is the image key point information set Info I The encoded image feature information corresponding to the i-th image key point information or the point cloud key point set Info C The encoded point cloud feature information corresponding to the i-th point cloud key point information; dec i Is the image key point information set Info I The key point descriptor information corresponding to the i-th image key point information or the point cloud key point set Info C The key point descriptor information corresponding to the key point information of the i-th point cloud; Pos i Is the image key point information set Info I The i-th image key point information or the point cloud key point set Info C The key point position information corresponding to the key point information of the i-th point cloud; The key point position information Pos obtained through MLP i The encoding result.

8. The method for matching oblique aerial images with airborne LiDAR point cloud features according to claim 1, characterized in that: Based on the attention aggregation mechanism, the encoded image feature information is respectively The encoded image feature information and the encoded point cloud feature information set in Update the encoded point cloud feature information in the image to obtain the updated image feature information set (L) X 2D And the updated point cloud feature information set (L) X 3D , and respectively according to the updated image feature information set (L) X 2D And the updated point cloud feature information set (L) X 3D Get the final image key point descriptor information set Dec 2D And the final point cloud key point descriptor information set Dec 3D ,include: According to the encoded image feature information set Construct a first undirected graph, and in the first undirected graph, perform the encoded image feature information collection through the self-attention mechanism The information between the encoded image feature information is transferred to obtain the first self-attention information; According to the encoded point cloud feature information set Construct a second undirected graph, and in the second undirected graph, perform the encoded point cloud feature information collection through the self-attention mechanism The information between the encoded point cloud feature information is transferred to obtain the second self-attention information; According to the encoded image feature information set And the encoded point cloud feature information set Construct a third undirected graph, and in the third undirected graph, perform the encoded image feature information collection through a cross attention mechanism The encoded image feature information and the encoded point cloud feature information set The information between the encoded point cloud feature information is transferred to obtain cross attention information; Aggregate the first self-attention information and the cross-attention information to obtain the encoded image feature information set Corresponding updated image feature information set (L) X 2D ; and aggregate the second self-attention information and the cross-attention information to obtain the encoded point cloud feature information set Corresponding updated point cloud feature information set (L) X 3D ; The updated image feature information set (L) X 2D Perform linear projection on the updated image feature information in to obtain the updated image feature information set (L) X 2D The final image key point descriptor information set Dec corresponding to the image key point information 2D ; and the updated point cloud feature information set (L) X 3D Perform linear projection on the updated point cloud feature information in to obtain the updated point cloud feature information set (L) X 3d The final point cloud key point descriptor information set Dec corresponding to the point cloud key point information 3D .

9. The method for matching oblique aerial images with airborne LiDAR point cloud features according to claim 1, characterized in that: According to the final image key point descriptor information set Dec 2D And the final point cloud key point descriptor information set Dec 3D , obtain the matching matrix of the airborne LiDAR point cloud and the oblique aerial image, including: Calculate the final image key point descriptor information set Dec 2D And the final point cloud key point descriptor information set Dec 3D The cosine value between According to the cosine value, the final image key point descriptor information set Dec is constructed. 2D And the final point cloud key point descriptor information set Dec 3D The correlation matrix Cor between them; wherein the cosine value Cor of the i-th row and j-th column in the correlation matrix Cor i,j for: Where, The final image key point descriptor information set Dec 2D The i-th final image key point descriptor information in, The final point cloud key point descriptor information set Dec 3D The jth final point cloud key point descriptor information in; According to the correlation matrix Cor, multiple iterative matching is performed on the point cloud key points of the airborne LiDAR point cloud and the image key points in the oblique aerial image to obtain a matching matrix between the airborne LiDAR point cloud and the oblique aerial image.

10. The method for matching oblique aerial images with airborne LiDAR point cloud features according to claim 9, characterized in that: The Sinkhorn algorithm is used to perform multiple iterative matching on the point cloud key points of the airborne LiDAR point cloud and the image key points in the oblique aerial image.

Images