A method and device for matching drone images and point clouds

Abstract

The present invention provides a method and device for matching drone images and point clouds, belonging to the technical field of computer image processing, the method comprising: obtaining image data to be matched, inputting the image data to be matched into a feature extraction network based on multi-stage deep learning of the Transformer architecture, obtaining feature points and feature descriptions extracted from the image; then performing feature matching on the feature points and feature descriptions of the image data to be matched to obtain matching point pairs; thereby, based on a preset image region network adjustment model, performing absolute orientation of the image region network based on point cloud feature constraints on the matching point pairs, thereby achieving drone image and point cloud matching. The present invention converts a three-dimensional point cloud into a two-dimensional depth map, converts the two types of data to the same dimension, and jointly optimizes the network through a key point detection module and a feature description module, so as to accurately and efficiently obtain the same-name feature mapping relationship between the two, thereby achieving accurate registration of the two types of data.

Description

Technical Field

[0001] The present invention relates to the technical field of computer image processing, and in particular to a method and device for matching unmanned aerial vehicle images and point clouds. Background Art

[0002] With the development of sensor technology, especially the popularization and application of new technologies such as drone low-altitude photography, oblique photography, and drone lidar in railway survey and design in recent years, it is common to collect drone images and drone lidar data in the same survey area during railway survey.

[0003] The geometric reference frames of airborne LiDAR point clouds and drone images are different. Before using them together, it is necessary to convert the two types of data into a unified reference coordinate system, that is, to establish a mapping relationship between the LiDAR point cloud depth map and the drone image to achieve accurate registration of the two types of data. Among them, the core issue of registration lies in the acquisition of same-name control information and the establishment of a geometric transformation model. Since airborne LiDAR point clouds are three-dimensional discrete point sets, and drone images are two-dimensional image data sets, there are significant differences in the acquisition mechanism and data organization structure of the two types of data. The feature expression forms of the same ground object in the two types of data are usually different. Therefore, what method is used to accurately and efficiently obtain the same-name feature mapping relationship between the two is the key to the registration of the two types of data.

[0004] At present, existing methods generally directly establish the same-name mapping relationship between LiDAR point cloud and image features through the selected geometric features, such as points, lines, surfaces and other geometric structures, and calculate the geometric transformation parameters accordingly. However, this type of method requires a sufficient number of distinguishable features in the survey area data. When the number of features is small, it is difficult to obtain sufficient same-name control information; when the distinguishability of features is weak, a large number of mismatches will occur. It has the technical defects that feature-based point cloud image registration is usually restricted by the number and quality of features in the scene and has poor generalization. Summary of the invention

[0005] The present invention provides a method and device for matching unmanned aerial vehicle images and point clouds, so as to solve the defects in the prior art that feature-based point cloud image registration is usually restricted by the number and quality of features in the scene and has poor generalization.

[0006] In a first aspect, the present invention provides a method for matching drone images and point clouds, comprising:

[0007] Acquire image data to be matched, wherein the image data to be matched includes a point cloud depth map and two-dimensional image data;

[0008] Input the image data to be matched into a feature extraction optimization network based on Transformer architecture multi-stage deep learning to obtain feature points and feature descriptions extracted from the image;

[0009] Performing feature matching on the feature points and feature descriptions of the image data to be matched to obtain matching point pairs;

[0010] Based on a preset image region network adjustment model, the matching point pairs are absolutely oriented in the image region network based on point cloud feature constraints to achieve matching between drone images and point clouds.

[0011] According to the drone image and point cloud matching method, the feature extraction optimization network based on Transformer architecture multi-stage deep learning includes:

[0012] The encoder network is used to extract deep features from the image data to be matched;

[0013] A key point detection module, used for extracting key points of an input image based on the deep features, wherein the key point detection module is constructed using a first multi-source loss optimization model;

[0014] A feature description module is used to extract feature description of an input image based on the deep feature, and the feature description module is constructed using a second multi-source loss optimization model.

[0015] According to the described UAV image and point cloud matching method, the second multi-source loss optimization model is constructed using triplet loss, neural reprojection loss and reliability loss.

[0016] According to the described method for matching drone images with point clouds, a first multi-source loss optimization model is constructed using binary cross entropy loss, reprojection loss, and divergence peak loss.

[0017] According to the drone image and point cloud matching method, the step of obtaining the point cloud depth map includes:

[0018] Obtain the multi-view images to be matched taken by the drone, and extract the position information and posture information of the target object in each image;

[0019] Based on the position information and attitude information, POS-assisted bundle adjustment is used to obtain the internal and external orientation elements of the image;

[0020] A point cloud depth map is obtained, and based on the internal and external orientation elements and according to a collinearity condition equation, points in the point cloud data are projected onto an imaging plane to obtain a point cloud depth map.

[0021] According to the method for matching drone images with point clouds, after acquiring the point cloud data, the following steps are also included:

[0022] interpolating the point cloud data using a natural neighbor interpolation method;

[0023] And establish an octree spatial index structure, and perform occlusion detection on the point cloud data in the octree node through the hidden point elimination operator;

[0024] Based on the internal and external orientation elements, according to the collinearity condition equation, the points in the point cloud data are projected onto an imaging plane to obtain a point cloud depth map;

[0025] The point cloud depth map is linearly transformed to calculate the grayscale value of the point cloud depth map and normalize it to 0 to 255.

[0026] According to the drone image and point cloud matching method, the preset image area network adjustment model is constructed based on the LiDAR point constraint primitive.

[0027] In a second aspect, the present invention further provides a drone image and point cloud matching device, comprising:

[0028] A data acquisition module, used to acquire image data to be matched, wherein the image data to be matched includes a point cloud depth map and two-dimensional image data;

[0029] The first processing module is used to input the image data to be matched into a feature extraction optimization network based on Transformer architecture multi-stage deep learning to obtain feature points and feature descriptions extracted from the image;

[0030] A second processing module is used to perform feature matching on the feature points and feature descriptions of the image data to be matched to obtain matching point pairs;

[0031] The third processing module is used to perform image area network absolute orientation based on point cloud feature constraints on the matching point pairs based on a preset image area network adjustment model, so as to achieve matching between the drone image and the point cloud.

[0032] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above-mentioned methods for matching drone images and point clouds are implemented.

[0033] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described methods for matching drone images and point clouds.

[0034] The present invention provides a method and device for matching drone images and point clouds. By acquiring image data to be matched including a point cloud depth map and two-dimensional image data, and inputting the data into a feature extraction optimization network based on multi-stage deep learning of a Transformer architecture, the present invention can accurately and efficiently obtain the same-name feature mapping relationship between the two, thereby achieving accurate alignment of the two types of data.

[0035] By using a lightweight TransFormer network as the encoder of the matching network, and jointly optimizing the network through the key point detection module and the feature description module, high-precision and stable matching results can be achieved when the feature point cloud image registration is restricted by the number and quality of features in the scene and the generalization is not strong. In addition, the central projection model is used to construct the point cloud depth map, and a high-quality depth map is obtained through interpolation and occlusion detection, so that it is close to the scale of the drone image and has small perspective geometric deformation, which is conducive to improving the accuracy of subsequent feature matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 It is a flow chart of the method for matching drone images and point clouds provided by the present invention;

[0038] Figure 2 It is a schematic diagram of the feature extraction optimization network architecture provided by the present invention;

[0039] Figure 3 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] It should be noted that, in the description of the embodiments of the present invention, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "include one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0042] This scheme uses the central projection model to construct the point cloud depth map, obtains a high-quality depth map through interpolation and occlusion detection, and then matches the point cloud depth map with the drone image;

[0043] A lightweight matching network based on TransFormer network is proposed;

[0044] When training the image matching network model, binary cross entropy loss, reprojection loss and divergence peak loss are used to optimize the key point positions predicted by the network model.

[0045] At the same time, triplet loss, neural reprojection loss and reliability loss are used to optimize the feature description predicted by the network model.

[0046] Combine the following Figure 1-Figure 3 The invention describes a method and device for matching drone images and point clouds provided in an embodiment of the invention.

[0047] Figure 1 is a flow chart of the method for matching drone images and point clouds provided by the present invention, such as Figure 1 As shown, including but not limited to the following steps (STEP, referred to as S):

[0048] S1: Obtain image data to be matched;

[0049] The image data to be matched includes point cloud depth map and two-dimensional image data.

[0050] Taking the scenario where drone images and drone lidar data are collected simultaneously in the same survey area during railway survey as an example, the image data to be matched are point cloud depth maps and two-dimensional image data.

[0051] Based on the content of the above embodiment, the step of acquiring the two-dimensional image data in a method for matching drone images and point clouds provided by the present invention includes:

[0052] Obtain the multi-view images to be matched taken by the drone, and extract the position information and posture information of the target object in each image;

[0053] Based on the position information and attitude information, POS-assisted bundle adjustment is used to obtain the internal and external orientation elements of the image;

[0054] A point cloud depth map is obtained, and based on the internal and external orientation elements and according to the collinearity condition equation, points in the point cloud data are projected onto an imaging plane to obtain two-dimensional image data.

[0055] Based on the content of the above embodiment, in a method for matching drone images and point clouds provided by the present invention, after acquiring the point cloud data, the method further includes the following steps:

[0056] interpolating the point cloud data using a natural neighbor interpolation method;

[0057] And establish an octree spatial index structure, and perform occlusion detection on the point cloud data in the octree node through the hidden point elimination operator;

[0058] Based on the internal and external orientation elements, according to the collinearity condition equation, the points in the point cloud data are projected onto an imaging plane to obtain a point cloud depth map;

[0059] The point cloud depth map is linearly transformed to calculate the grayscale value of the point cloud depth map and normalize it to 0 to 255.

[0060] S2: Input the image data to be matched into a feature extraction optimization network based on Transformer architecture multi-stage deep learning to obtain feature points and feature descriptions extracted from the image.

[0061] Preferably, the feature extraction optimization network based on Transformer architecture multi-stage deep learning includes:

[0062] The encoder network is used to extract deep features from the image data to be matched;

[0063] A key point detection module, used for extracting key points of an input image based on the deep features, wherein the key point detection module is constructed using a first multi-source loss optimization model;

[0064] A feature description module is used to extract feature description of an input image based on the deep feature, and the feature description module is constructed using a second multi-source loss optimization model.

[0065] Figure 2 The schematic diagram of the feature extraction optimization network architecture provided in this embodiment is as follows: Figure 2 As shown, the feature extraction optimization network includes an encoder network, a key point detection module and a feature description module.

[0066] In the feature extraction optimization network, the input end of the encoder network is used to input the image data to be matched (input image), and the output end of the encoder network is used to output the deep features extracted from the image data to be matched.

[0067] The output of the encoder network is connected to the input of the key point detection module and the input of the feature description module respectively.

[0068] The input end of the key point detection module is used to input the deep features extracted from the image data to be matched, and the output end of the key point detection module is used to output the key points of the deep features.

[0069] The input end of the feature description module is used to input the deep features extracted from the image data to be matched, and the output end of the feature description module is used to output the feature description of the deep features.

[0070] The key point detection module is constructed by using the first multi-source loss optimization model, and the feature description module is constructed by using the second multi-source loss optimization model.

[0071] Based on the contents of the above embodiments, the first multi-source loss optimization model in a method for matching drone images and point clouds provided by the present invention is constructed using triplet loss, neural reprojection loss and reliability loss.

[0072] Preferably, the second multi-source loss optimization model is constructed using binary cross entropy loss, reprojection loss and divergence peak loss.

[0073] S3: Perform feature matching on the feature points and feature descriptions of the image data to be matched to obtain matching point pairs.

[0074] The features of the image data to be matched are matched, and similar feature descriptions can be found by, for example, similarity matching based on pixel intensity distribution or nearest neighbor search based on feature point position and direction, etc. In this way, a set of matching point pairs can be obtained, which represent similar feature points in the original image data to be matched.

[0075] S4: Based on a preset image region network adjustment model, the image region network absolute orientation based on point cloud feature constraints is performed on the matching point pairs to achieve matching between the drone image and the point cloud.

[0076] Based on the preset image region network adjustment model, the matching point pairs obtained by feature matching are further processed. The preset image region network adjustment model is used to optimize the orientation of the image region network to obtain more accurate position and direction information. Then, these optimized matching point pairs are used to perform absolute orientation of the image region network based on point cloud features. The optimization process can be performed by performing a more detailed analysis of each matching point, including its position, direction, surrounding environment and other information, to determine the exact position and direction of each pixel in three-dimensional space. Finally, through this absolute orientation, the matching between drone images and point clouds can be achieved, that is, the pixels in the drone two-dimensional image data are mapped to the corresponding points in the point cloud depth map.

[0077] The technical solution of the present invention is specifically described below through an example.

[0078] (1) Obtain multi-view images taken by drones, record the position and attitude information of each image, and then obtain the internal and external orientation elements of the images through POS-assisted bundle adjustment.

[0079] In this embodiment, the airborne LiDAR point cloud is obtained and simple preprocessing is performed. A central projection model is used to construct a point cloud depth map based on multi-view images to obtain a point cloud depth map to be matched.

[0080] (2) The exterior orientation elements of the oblique image and the intrinsic parameters of the camera are used as data input, and the LiDAR points in the object space are projected onto the imaging plane according to the collinearity condition equation to obtain two-dimensional image data.

[0081] It is worth noting that in order to obtain a point cloud depth map with better image quality, the LiDAR point cloud data can be interpolated. For example, the LiDAR point cloud data can be interpolated by the natural neighbor interpolation method. After interpolation, the pixel value distribution is continuous while the contour characteristics of the object are better maintained.

[0082] At the same time, in order to avoid the impact caused by occlusion, for example, by establishing an octree spatial index structure, a hidden point elimination operator is introduced to perform occlusion detection on the LiDAR points in the octree nodes.

[0083] Furthermore, based on the internal and external orientation elements and according to the collinearity condition equation, the LiDAR points are projected onto the imaging plane to obtain a point cloud depth map.

[0084] Furthermore, in order to improve the expressiveness of the point cloud depth map, the elevation values ​​of the LiDAR point cloud depth map points are linearly transformed and the grayscale values ​​of the point cloud depth map are calculated and normalized to 0 to 255.

[0085] (3) Use existing multimodal image matching datasets to train deep learning networks.

[0086] Figure 2 The schematic diagram of the feature extraction optimization network architecture provided in this embodiment is as follows: Figure 2 As shown in the figure, the training image data is first input into the encoder network to extract deep features. In order to make the entire network structure lightweight, the encoder network here uses the lightweight TransFormer network RepViT.

[0087] RepViT consists of 4 stages, so the features of its 4 stages {F1, F2, F3, F4} can be obtained.

[0088] (4) Perform a cascade operation on the four features obtained in step (3) to obtain the image deep feature F.

[0089] (5) Input the deep features obtained in step (4) into the differentiable key point detection module to obtain the key point position.

[0090] Among them, the deep feature F is convolved and then normalized by the Sigmoid activation function to serve as the score map S.

[0091] In order to detect sub-pixel key points from the score map S, the key point loss in this embodiment uses binary cross entropy loss, reprojection loss and divergence peak loss.

[0092] By adopting binary cross entropy loss and reprojection loss, the position of the key point is directly optimized, and the maximum score at the key point position is guaranteed by adopting the divergence peak loss.

[0093] Specifically, the reprojection loss is defined as follows:

[0094] dist AB =∥p AB -p B ∥ p ;

[0095]

[0096] Among them, dist AB is the projection key point p of key point A on another image AB The closest detection key point p B The distance, dist AB on the contrary.

[0097] Specifically, the divergence peak loss is defined as follows:

[0098] Consider an N×N score tile, where each pixel [i,j] T To detect key points Distance:

[0099]

[0100] Among them, s′(i,j) is the confidence at the pixel point (i,j).

[0101] (6) The deep features obtained in step (4) are input into the feature description module, and a 128-dimensional feature map is obtained after convolution.

[0102] It is worth mentioning that the feature description adopts triplet loss, neural reprojection loss and reliability loss.

[0103] The sparse descriptors sampled from keypoints are optimized by adopting the triplet loss, and the reliability loss improves the discrimination between descriptors by minimizing the cross entropy between the dense reprojected probability map and the dense matching probability map using the neural reprojection loss.

[0104] Specifically, the triplet loss is defined as:

[0105]

[0106] Among them, p is the distance between positive sample descriptions, n is the distance between negative sample descriptions, and M is an empirical constant.

[0107] (7) back-propagating the total loss with respect to the network parameters to find the derivative, for example, by a gradient descent algorithm;

[0108] (8) Steps (1) to (7) are iterated repeatedly until the training is completed, and a deep learning model based on Transformer architecture multi-stage deep learning for predicting image feature points and feature descriptions is obtained, that is, a feature extraction optimization network.

[0109] (9) The trained deep learning model is used to extract feature points and feature descriptions from images and point cloud depth maps respectively.

[0110] (10) Use the existing matching method to perform feature matching to obtain matching point pairs and eliminate gross errors.

[0111] (11) Construct an image network adjustment model under the LiDAR point constraint primitive, complete the absolute orientation of the image network constrained by the LiDAR point cloud features, and complete the matching of drone images and point clouds.

[0112] Compared with the prior art, the advantages of the above technical solution are:

[0113] (1) The central projection model is used to construct the point cloud depth map. A high-quality depth map is obtained through interpolation and occlusion detection. This makes the depth map similar to the scale of the drone image and has small perspective geometric deformation, which is conducive to improving the accuracy of subsequent feature matching.

[0114] (2) A lightweight TransFormer network is used as the encoder of the matching network. At the same time, the network is jointly optimized through the key point detection module and the feature description module to obtain high-precision and stable matching results.

[0115] The present invention also provides a drone image and point cloud matching device, comprising:

[0116] A data acquisition module, used to acquire image data to be matched, wherein the image data to be matched includes a point cloud depth map and two-dimensional image data;

[0117] The first processing module is used to input the image data to be matched into a feature extraction optimization network based on Transformer architecture multi-stage deep learning to obtain feature points and feature descriptions extracted from the image;

[0118] A second processing module is used to perform feature matching on the feature points and feature descriptions of the image data to be matched to obtain matching point pairs;

[0119] The third processing module is used to perform image area network absolute orientation based on point cloud feature constraints on the matching point pairs based on a preset image area network adjustment model, so as to achieve matching between the drone image and the point cloud.

[0120] It should be noted that the drone image and point cloud matching device provided in the embodiment of the present invention can execute the indoor drone image and point cloud matching method described in any of the above embodiments during specific operation, which will not be elaborated in this embodiment.

[0121] Figure 3 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 3 As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330 and a communication bus 340, wherein the processor 310, the communication interface 320 and the memory 330 communicate with each other through the communication bus 340. The processor 310 may call the logic instructions in the memory 330 to execute the method for matching drone images with point clouds, including: obtaining image data to be matched, wherein the image data to be matched includes a point cloud depth map and two-dimensional image data; inputting the image data to be matched into a feature extraction optimization network based on multi-stage deep learning of the Transformer architecture to obtain feature points and feature descriptions extracted from the image; performing feature matching on the feature points and feature descriptions of the image data to be matched to obtain matching point pairs; performing absolute orientation of the image region network based on point cloud feature constraints on the matching point pairs based on a preset image region network adjustment model to achieve matching of drone images with point clouds.

[0122] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0123] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the drone image and point cloud matching methods provided in the above embodiments, including: obtaining image data to be matched, wherein the image data to be matched includes a point cloud depth map and two-dimensional image data; inputting the image data to be matched into a feature extraction optimization network based on multi-stage deep learning of the Transformer architecture to obtain feature points and feature descriptions extracted from the image; performing feature matching on the feature points and feature descriptions of the image data to be matched to obtain matching point pairs; based on a preset image region network adjustment model, performing absolute orientation of the matching point pairs based on the image region network constrained by point cloud features to achieve matching between drone images and point clouds.

[0124] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the drone image and point cloud matching method provided in the above-mentioned embodiments, including: obtaining image data to be matched, wherein the image data to be matched includes a point cloud depth map and two-dimensional image data; inputting the image data to be matched into a feature extraction optimization network based on multi-stage deep learning of the Transformer architecture to obtain feature points and feature descriptions extracted from the image; performing feature matching on the feature points and feature descriptions of the image data to be matched to obtain matching point pairs; based on a preset image region network adjustment model, performing absolute orientation of the matching point pairs based on the image region network constrained by point cloud features to achieve matching of drone images and point clouds.

[0125] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0126] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for matching drone images and point clouds, characterized in that: include: Acquire image data to be matched, wherein the image data to be matched includes a point cloud depth map and two-dimensional image data; Input the image data to be matched into a feature extraction optimization network based on Transformer architecture multi-stage deep learning to obtain feature points and feature descriptions extracted from the image; Performing feature matching on the feature points and feature descriptions of the image data to be matched to obtain matching point pairs; Based on a preset image region network adjustment model, the matching point pairs are absolutely oriented in the image region network based on point cloud feature constraints to achieve matching between drone images and point clouds; The feature extraction optimization network based on Transformer architecture multi-stage deep learning includes: an encoder network for extracting deep features from the image data to be matched; A key point detection module, used for extracting key points of an input image based on the deep features, wherein the key point detection module is constructed by using a first multi-source loss optimization model; the first multi-source loss optimization model is constructed by using binary cross entropy loss, reprojection loss and divergence peak loss; A feature description module is used to extract feature description of an input image based on the deep feature, and the feature description module is constructed using a second multi-source loss optimization model; the second multi-source loss optimization model is constructed using triplet loss, neural reprojection loss and reliability loss.

2. The method for matching drone images and point clouds according to claim 1, characterized in that: The step of acquiring the point cloud depth map includes: Obtain the multi-view images to be matched taken by the drone, and extract the position information and posture information of the target object in each image; Based on the position information and attitude information, POS-assisted bundle adjustment is used to obtain the internal and external orientation elements of the image; Point cloud data is acquired, and based on the internal and external orientation elements and according to the collinearity condition equation, points in the point cloud data are projected onto an imaging plane to obtain a point cloud depth map.

3. The method for matching drone images and point clouds according to claim 2, characterized in that: After obtaining the point cloud data, the following steps are also included: interpolating the point cloud data using a natural neighbor interpolation method; And establish an octree spatial index structure, and perform occlusion detection on the point cloud data in the octree node through the hidden point elimination operator; Based on the internal and external orientation elements, according to the collinearity condition equation, the points in the point cloud data are projected onto an imaging plane to obtain a point cloud depth map; The point cloud depth map is linearly transformed to calculate the grayscale value of the point cloud depth map and normalize it to 0 to 255.

4. The method for matching drone images and point clouds according to claim 1, characterized in that: The preset image area network adjustment model is constructed based on the LiDAR point constraint primitive.

5. A drone image and point cloud matching device, characterized in that: include: A data acquisition module, used to acquire image data to be matched, wherein the image data to be matched includes a point cloud depth map and two-dimensional image data; The first processing module is used to input the image data to be matched into a feature extraction optimization network based on Transformer architecture multi-stage deep learning to obtain feature points and feature descriptions extracted from the image; A second processing module is used to perform feature matching on the feature points and feature descriptions of the image data to be matched to obtain matching point pairs; The third processing module is used to perform image area network absolute orientation based on point cloud feature constraints on the matching point pairs based on a preset image area network adjustment model, so as to achieve matching between the drone image and the point cloud; The feature extraction optimization network based on Transformer architecture multi-stage deep learning includes: an encoder network for extracting deep features from the image data to be matched; A key point detection module is used to extract key points of an input image based on the deep features, and the key point detection module is constructed using a first multi-source loss optimization model; the first multi-source loss optimization model is constructed using binary cross entropy loss, reprojection loss and divergence peak loss; a feature description module is used to extract feature descriptions of an input image based on the deep features, and the feature description module is constructed using a second multi-source loss optimization model; the second multi-source loss optimization model is constructed using triplet loss, neural reprojection loss and reliability loss.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the drone image and point cloud matching method as described in any one of claims 1 to 4 are implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for matching drone images and point clouds as described in any one of claims 1 to 4 are implemented.

Images