A geolocation method based on 2D-3D matching network

Abstract

The present application relates to the field of SLAM and deep learning, and proposes a geographic positioning method based on a 2D-3D matching network, comprising the following steps: S1, tracking the input laser radar data using a laser radar odometer; S2, intercepting a local satellite map according to the current position data; S3, taking the corrected laser radar point cloud and the local satellite map as network input, and obtaining the matching relationship between the laser radar point and the two-dimensional pixel point on the satellite picture through a 2D-3D matching network; S4, calculating the pose transformation relationship of the laser radar frame relative to the map according to the matching relationship; S5, combining the tracking result of the laser radar odometer with the registration result based on the satellite map to optimize the pose graph, and obtaining the optimized pose transformation result; S6, splicing each frame of laser radar point cloud according to the optimized pose transformation result to obtain a point cloud map. After GPS initialization, only point cloud and satellite map are needed to realize long-time real-time geographic positioning, and high positioning accuracy can be maintained.

Description

Technical Field

[0001] This invention relates to the fields of SLAM and deep learning, specifically to a geolocation method based on a 2D-3D matching network. Background Technology

[0002] Accurate positioning over long periods is a key technology for autonomous driving. The pursuit of precise positioning has not diminished with the advent of GPS; on the contrary, the demand for stable and accurate positioning over extended periods has increased. While odometry based on various sensors can provide relatively accurate pose estimation, it is inevitably affected by cumulative drift. Therefore, vehicle positioning technology typically relies on Global Navigation Satellite Systems (GNSS) to estimate absolute pose. However, in urban environments with numerous high-rise buildings, commercial GPS signals are often unusable due to obstruction. In such cases, prior map-based positioning has become a feasible solution to the positioning problem, compensating for the limitations of GPS signals and providing higher-precision 6-DOF pose estimation. Most prior map-based methods require pre-mapping of the environment before positioning. Numerous studies have suggested using open-source maps for robot positioning, such as Geographic Information Systems (GIS) and satellite imagery. These existing prior maps not only have wider coverage but are also easily accessible. However, existing geolocation methods suffer from inaccurate positioning when GPS signals are poor, such as due to obstruction.

[0003] Feature association methods can generally be divided into image feature matching, 3D point cloud registration, and association between images and point clouds. Traditional image feature matching typically involves corner extraction, feature calculation, and feature matching. Generally, a detector is used to pre-extract features insensitive to illumination and scale from the image, and then the feature points extracted from different images are associated to form a match. In recent years, with the continuous advancement of research, detector-free feature matching methods have attracted widespread interest, and new methods have been proposed, which can output the features in the image for subsequent matching using only the input image. Classic point cloud matching methods include ICP (Iterative Closest Point), which calculates the optimal pose of the point cloud stitching through an iterative method to perform point cloud registration. With the proposal and development of deep learning-based point cloud registration methods, it has become possible to correctly register two point clouds even with small overlapping areas. Summary of the Invention

[0004] To address the aforementioned issues, this invention designs a 2D-3D matching network based on laser point clouds and images, using Simultaneous Localization and Mapping (SLAM) and deep learning methods. It also constructs a geolocation system based on vehicle-mounted LiDAR point clouds and satellite maps, solving the problem that existing GPS solutions are prone to inaccurate positioning under conditions of poor signal, such as obstruction. After the system is initialized, it can maintain high positioning accuracy for a long time without relying on GPS signals.

[0005] To achieve the above objectives, the present invention provides a geolocation method based on a 2D-3D matching network, comprising the following steps:

[0006] S1. Track the input lidar data using a lidar odometry.

[0007] S2. Extract a local satellite map based on the current location data;

[0008] S3. Using the corrected lidar point cloud and the local satellite map as network inputs, the matching relationship between lidar points and two-dimensional pixels on the satellite image is obtained through a 2D-3D matching network.

[0009] S4. Calculate the pose transformation relationship of the lidar frame relative to the map based on the matching relationship between the lidar points and the two-dimensional pixels on the satellite image;

[0010] S5. Combine the tracking results of the lidar odometry with the registration results based on the satellite map to optimize the pose graph and obtain the optimized pose transformation result.

[0011] S6. Based on the optimized pose transformation results, stitch together each frame of the lidar point cloud to obtain a point cloud map.

[0012] Preferably, step S1 specifically includes the following steps:

[0013] S11. Preprocess the input LiDAR points and extract corner points and planar points based on curvature;

[0014] S12. The front end constructs point-line constraints and point-surface constraints based on the points extracted from the previous and next frames to solve the inter-frame pose transformation.

[0015] S13. The backend finds the grid index in the current frame map based on the frontend results, and then constructs a local map to optimize the current frame. It constructs point-line constraints and point-polygon constraints to optimize and update the pose transformation between the odometer and the map.

[0016] Preferably, the curvature calculation formula for each point when extracting corner points and planar points based on curvature is as follows:

[0017]

[0018] Where S is the set of points surrounding point i on the same scan line as i when calculating curvature. Let i be the coordinates of point i in the k-th frame. Let j be the coordinates of point j in the k-th frame.

[0019] Preferably, step S12 specifically includes:

[0020] The residual is calculated from the successfully matched corner point pairs, using the following formula:

[0021]

[0022] Where point i is a corner point in the (k+1)th frame, and points j and l are the two corner points in the kth frame that match point i. Let i be the coordinates of the midpoint i in the (k+1)th frame in the lidar coordinate system of the kth frame. and Let i be the coordinates of point j and point l in the lidar coordinate system of the k-th frame. Geometrically, i is the distance from point i to the straight line formed by points j and l.

[0023] The residual is calculated from the successfully matched planar point pairs using the following formula:

[0024]

[0025] Where point i is a planar point in frame k+1, and points j, l, and m are three planar points in frame k that match point i. Let i be the coordinates of the midpoint i in the (k+1)th frame in the lidar coordinate system of the kth frame.

[0026] and Let i be the coordinates of points j, l, and m in the lidar coordinate system of the k-th frame. Geometrically, i is the distance from point i to the plane formed by points j, l, and m.

[0027] Preferably, step S2 specifically includes the following steps:

[0028] S21. Determine whether system initialization is complete. If system initialization is not complete, take the current GPS location as the initial location and retrieve a satellite map of a certain size.

[0029] S22. When initialization is complete and the system is running normally, use the optimized pose to capture a local satellite image on the satellite map as part of the input to the 2D-3D matching network.

[0030] Preferably, step S3 specifically includes the following steps:

[0031] S31. Use Feature Pyramid Network (FPN) to extract coarse and fine features from a 2D image. The coarse feature map of the extracted 2D image is 1 / 16 the size of the original image, and the fine feature map of the extracted 2D image is 1 / 8 the size of the original image.

[0032] S32. Perform four downsampling operations on the point cloud, use the KPConv backbone to extract multi-scale local features of the point cloud to obtain coarse and fine features, and use Local Pool to aggregate features in the Z-axis direction.

[0033] S33. After the coarse features are extracted by the network, a coarse Transformer module is used to fuse the positions of 2D points and 3D points as well as other contextual cues contained therein to enhance the uniqueness of the features.

[0034] S34. Use the enhanced coarse feature point matching module to perform initial matching selection;

[0035] S35. Use the fine-stage matching module to refine the results of the coarse matching.

[0036] Preferably, step S4 specifically includes the following steps:

[0037] S41. Randomly select three pairs of matching points between lidar and satellite images and calculate the pose transformation from lidar frame to satellite map through reprojection error.

[0038] S42. Verify all matching pairs based on the calculated pose transformation, and calculate the in-model point rate;

[0039] S43. If the in-point ratio of the model is insufficient, repeat steps S41 and S42. Otherwise, construct a least squares problem using the in-point ratio of the obtained model and solve it to obtain the final pose transformation from the lidar frame to the satellite map.

[0040] Preferably, step S5 specifically includes the following steps:

[0041] S51. Using the odometry tracking results, construct a pose graph optimization problem. Calculate the edge error between vertices in the pose graph by comparing the relative pose of the lidar frame with the relative pose calculated based on the global pose.

[0042] S52. The pose graph is optimized by using prior edges and fusing the positioning results of LiDAR frames based on satellite maps.

[0043] Preferably, step S6 specifically involves using the optimized pose transformation result to transform the lidar point cloud frame to the map point cloud coordinate system and stitching them together to obtain an updated point cloud map.

[0044] Compared with the prior art, the beneficial effects of the present invention are:

[0045] The present invention provides a geolocation method based on 2D-3D matching network. After GPS initialization is completed, it only requires point cloud and satellite imagery to achieve long-term real-time geolocation. It can maintain high positioning accuracy without relying on external GPS signals. Attached Figure Description

[0046] Figure 1 This is a flowchart of the method of the present invention;

[0047] Figure 2 This is a schematic diagram illustrating the selection of local satellite images for this invention;

[0048] Figure 3 This is a schematic diagram of the 2D-3D matching network constructed in this invention.

[0049] Figure 4 This is a schematic diagram of the Local pool operation of the present invention;

[0050] Figure 5 This is a matching effect diagram output by the matching network of the present invention;

[0051] Figure 6 This is a diagram showing the registration effect between the lidar frame and a local satellite image of the present invention;

[0052] Figure 7 This is a schematic diagram of the pose graph structure of the present invention;

[0053] Figure 8 This is a rendering of a point cloud map output for this invention. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] 2D-3D matching can generally be understood as associating two-dimensional features in an image with three-dimensional points in a point cloud, representing a cross-modal matching process. There are typically two methods for achieving this matching: one is to transform the point cloud into image coordinates using camera intrinsic and extrinsic parameters, and then perform matching through two-dimensional feature matching; the other is to recover the spatial coordinates of image pixels using triangulation or depth estimation methods, and then use point cloud registration for matching. 2D-3D matching networks utilize deep learning to achieve feature association between images and point clouds. This paper proposes an end-to-end method to achieve feature association between satellite images and LiDAR point clouds, and applies it to maintain accurate positioning even when GPS and other global positioning system signals are blocked, providing a new approach and solution for achieving accurate positioning when GPS is unavailable.

[0056] To address the problems and shortcomings of existing technologies, this invention proposes a geolocation method based on 2D-3D matching networks, which can be combined with laser SLAM algorithms to achieve real-time positioning in satellite maps.

[0057] This invention proposes a geolocation method based on a 2D-3D matching network, such as... Figure 1 As shown, it includes the following steps:

[0058] S1. Track the input lidar data using a lidar odometry.

[0059] S2. Extract a local satellite map based on the current location data;

[0060] S3. Using the corrected lidar point cloud and the local satellite map as network inputs, the matching relationship between lidar points and two-dimensional pixels on the satellite image is obtained through a 2D-3D matching network.

[0061] S4. Calculate the pose transformation relationship of the lidar frame relative to the map based on the matching relationship between the lidar points and the two-dimensional pixels on the satellite image;

[0062] S5. Combine the tracking results of the lidar odometry with the registration results based on the satellite map to optimize the pose graph and obtain the optimized pose transformation result (localization result).

[0063] S6. Based on the optimized pose transformation results, stitch together each frame of the lidar point cloud to obtain a lidar point cloud map.

[0064] The following is a detailed explanation of each step.

[0065] Step S1: Track the input lidar data using a lidar odometry.

[0066] For the input lidar frame, this invention uses odometry technology to track each lidar frame to obtain the local relative positioning result, and then uses the result of global matching positioning to correct it.

[0067] Specifically, it mainly includes the following steps:

[0068] S11. Preprocess the input LiDAR points, extracting corner points and planar points based on curvature. When extracting corner points and planar points from the point cloud, the curvature of the points is used as the basis. The curvature calculation formula for each point is as follows:

[0069]

[0070] Where S is the set of points surrounding point i on the same scan line as i when calculating curvature. Let i be the coordinates of point i in the k-th frame. Let j be the coordinates of point j in the k-th frame.

[0071] S12. The front end constructs point-line constraints and point-surface constraints based on the points extracted from the previous and next frames to solve the inter-frame pose transformation. The formula for calculating the residual from the successfully matched corner point pairs is as follows:

[0072]

[0073] Where point i is a corner point in the (k+1)th frame, and points j and l are two corner points in the kth frame that match point i. Let i be the coordinates of the midpoint i in the (k+1)th frame in the lidar coordinate system of the kth frame. and Let i be the coordinates of point j and point l in the lidar coordinate system of the k-th frame. Geometrically, i is the distance from point i to the straight line formed by points j and l.

[0074] The formula for calculating the residual from successfully matched planar point pairs is:

[0075]

[0076] Where point i is a plane point in the (k+1)th frame, and points j, l, and m are three plane points in the kth frame that match point i. Let i be the coordinates of the midpoint i in the (k+1)th frame in the lidar coordinate system of the kth frame. and Let i be the coordinates of points j, l, and m in the lidar coordinate system of the k-th frame. The geometric meaning of the residual is the distance from point i to the plane formed by points j, l, and m.

[0077] S13. The backend finds the grid index in the current frame map based on the frontend results, and then constructs a local map to optimize the current frame. It constructs point-line constraints and point-polygon constraints to optimize and update the pose transformation between the odometer and the map.

[0078] Step S2: Extract a local satellite map based on the current location data.

[0079] This invention uses the KITTI dataset as its data support. The KITTI dataset is a research dataset for the field of autonomous driving, jointly sponsored by the Karlsruhe Institute of Technology in Germany and Toyota Technological University at Chicago. The authors collected six hours of real-world traffic environment data. The dataset consists of multiple modalities of information, including calibrated and synchronized images, LiDAR scans, high-precision GPS information, and IMU acceleration information. Initial GPS coordinates can be obtained directly from the KITTI dataset. Based on the GPS information in the dataset, this invention pre-loads a global satellite map of the city where the dataset is located from Google, and subsequently extracts a local map from the global map based on location (e.g., ...). Figure 2 (As shown).

[0080] Specifically, it mainly includes the following steps:

[0081] S21. Determine whether system initialization is complete. If system initialization is not complete, obtain the current GPS location from the dataset as the initial location and capture a satellite map of a certain size.

[0082] S22. When initialization is complete and the system is running normally, use the optimized pose to capture a local satellite image on the satellite map as part of the input to the 2D-3D matching network.

[0083] Step S3: Using the corrected lidar point cloud and the local satellite map as network inputs, the matching relationship between the lidar points and the two-dimensional pixels on the satellite image is obtained through a 2D-3D matching network.

[0084] This invention proposes a two-stage corner-free detector 2D-3D matching network (e.g. Figure 3 As shown in the figure, the network can generate a dense correspondence between image I and lidar point cloud P.

[0085] Specifically, it mainly includes the following steps:

[0086] S31. Use a Feature Pyramid Network (FPN) to extract coarse and fine-level features from the 2D image I. The extracted coarse feature map of the 2D image... The extracted 2D image features are 1 / 16 the size of the original image. It is 1 / 8 the size of the original image;

[0087] S32. The distortion-free input point cloud P obtained from the odometer is downsampled four times to obtain four point clouds P with different resolutions. k (k = 1, 2, 3, 4), the KPConv backbone is used to extract multi-scale local features of the point cloud to obtain coarse features. and fine features Local Pool is used to aggregate features in the Z-axis direction, as shown in Figure 4. During feature extraction, for each layer of point cloud output, this invention samples the point cloud features to a 2D grid with a fixed resolution, and then uses maxpool to pool the grid features and then backsamples them back to 3D.

[0088] S33. Coarse features extracted by the network and Subsequently, a coarse-level Transformer module is used to fuse the positions of 2D and 3D points, along with other contextual cues they contain, to enhance feature uniqueness. This invention performs matching on already downsampled feature maps, rather than the original image and point cloud, because registration can be fixed by matching coarser correspondences between subsets. Features at the original resolution are often too dense, making matching inefficient and time-consuming. Matching on low-resolution features significantly reduces the data input to the Transformer module, thereby greatly improving matching efficiency while maintaining accuracy.

[0089] The attention layer typically receives a query Q, a key K, and a value V as input, performing tasks similar to information retrieval. Attention determines weights by calculating the similarity between query Q and key K, then transforms the similarity value V into a set of weights using a softmax layer. Finally, the attention value is obtained by weighted summation of the products of these weights and their corresponding values.

[0090] Attention(Q,K,V) = softmax(QK) T V

[0091] Attention can be broadly interpreted as a vector of importance weights. To predict an element, the attention vector is used to estimate its relationship to other elements, and the weighted sum of these values ​​is used as an approximation of the target. When the input query Q and the value V are both image or point cloud features ( or When the input query Q and value V are image and point cloud features respectively, this attention mechanism is called self-attention. and When this happens, it is called cross-attention.

[0092] The simplest and most effective way to integrate location information into ground features is to directly add or concatenate the location embedding vectors to the features. To consistently embed 2D pixel locations and 3D point clouds, only the x and y positions of the point cloud are embedded, and the true scale q (m / pixel) of the satellite map is recovered using zoomLevel. pixelCoordinate = worldCoordinate × 2 zoomLevel

[0093] The main reason for this is that satellite imagery is obtained by measuring ground-based satellites from the air, ignoring the height information of objects on the ground. Points in a LiDAR point cloud with the same horizontal coordinates (x, y) should correspond to the same point on the satellite image, while maintaining feature invariance along the z-axis.

[0094] S34. Initial matching selection is performed using an enhanced coarse feature point matching module. During coarse-level matching, a 2D feature may match multiple 3D point features. Therefore, this invention performs a softmax operation only on the matching score of the point cloud in the image when extracting matching pairs to obtain the matching probability of each LiDAR point in each image block.

[0095] First, this invention addresses the output of the Transformer module. and The features are normalized, and the score matrix is ​​calculated.

[0096]

[0097] Since not all three-dimensional points can be found on a two-dimensional image, this invention expands the score S by adding a new row and filling it with a learnable variable 'a'. Then, the matching confidence matrix P is obtained through softmax at temperature t.

[0098]

[0099] Finally, select the maximum value in each column of P, and only if the maximum confidence score of the match is higher than the preset threshold th. c Matching is selected only when appropriate. A coarse-level match prediction is represented as:

[0100] M c ={(i c ,j c IP(i) c ,j c )=max(P(.,j c )),P(i c ,j c )>thc )

[0101] S35. Refine the coarse matching results using a fine-stage matching module. Matching is performed directly on low-resolution features of the 2D image and 3D point cloud. The accuracy of the resulting matching pairs is insufficient for some downstream tasks such as localization. Therefore, this invention uses a fine-stage matching module to further process and refine the coarse matching results. For each coarse match (i... c j c Extracting fine features from images and point clouds. and To be precise, in a two-dimensional image, i is first calculated. c In feature map The corresponding position on (u) c v c Then, crop a window of size w×w at that location. For 3D point clouds, according to j c Position in the Euclidean distance, from P 3 Extracting a local point cloud Then from Local point cloud features are obtained from the data.

[0102] Similar to the coarse-level Transformer, this invention uses a Transformer to obtain features ( and This is used to calculate the precise location of points on the image. It's important to note that this part only uses local point features learned from the backbone, without any location embedding. In fact, after resolving global ambiguity through coarse matching, point-level matching primarily depends on the proximity of the points.

[0103] Finally, from Select the center point j from all points in the image patch. f Based on the characteristics, the matching probability is calculated to obtain a heatmap. The pixel position i is obtained by calculating the expected value of the probability distribution. f This serves as the matching location. After processing each point obtained from the coarse matching, the matching results are collected to form the final global dense point correspondence M. f =(i f j f The final matching result is as follows: Figure 5 As shown.

[0104] Step S4: Calculate the pose transformation relationship of the lidar frame relative to the map based on the matching relationship between the lidar points and the two-dimensional pixels on the satellite image.

[0105] Specifically, it mainly includes the following steps:

[0106] S41. Randomly select three pairs of matching points between the lidar and satellite images, and calculate the pose transformation from the lidar frame to the satellite map using the reprojection error. The specific error formula is as follows:

[0107]

[0108] Where R and t represent the position transformation relationship from the center of the LiDAR point cloud to the center of the image, p and uv represent the matching points of the coordinates of the 3D LiDAR point and the pixel point, and π(·) represents the projection transformation relationship. For the input point [x,y,z], its transformation formula is:

[0109] n([x,y,z])=[x / scale+H / 2,y / scale+W / 2]

[0110] Here, scale represents the true scale information of the satellite image, and H and W are the height and width of the satellite image, respectively. This transformation relationship maps three-dimensional points to two-dimensional satellite images. It is worth noting that because the satellite image can be regarded as orthogonal, the z-axis is directly ignored.

[0111] S42. Verify all matching pairs based on the calculated pose transformation, and calculate the in-model point rate;

[0112] S43. If the inlier rate of the model is insufficient, repeat steps S41 and S42; otherwise, construct and solve a least-squares problem using the inlier rate of the obtained model. The final pose transformation from the LiDAR frame to the satellite map (the registration result is as follows) Figure 6 (As shown).

[0113] Step S5: Combine the tracking results of the lidar odometry with the registration results based on the satellite map to optimize the pose graph and obtain the optimized pose transformation result (localization result).

[0114] Specifically, it mainly includes the following steps:

[0115] S51. Using the odometry tracking results, construct a pose graph optimization problem. Calculate the edge error between vertices in the pose graph by comparing the relative pose of the lidar frame with the relative pose calculated based on the global pose.

[0116] S52. The pose graph is optimized by using prior edges and fusing the positioning results of LiDAR frames based on satellite maps.

[0117] This invention uses a pose graph optimization method to fuse odometry results and satellite map-based relocalization results, and can be seamlessly coupled with odometry. The odometry output pose is denoted as... The registration result output by the 2D-3D matching network is denoted as This invention uses odometer output And registration results Constructing a pose graph optimization problem (structure as follows) Figure 7 (As shown), to obtain a more accurate pose. The odometer error edge is defined as follows:

[0118]

[0119] To integrate the global localization obtained from registration, this invention designs a special error edge, defined as follows:

[0120]

[0121] in, and for Translation and rotation components, and for Translation and rotation components, R| z The z-axis rotation component of the rotation matrix R represents the rotation of the region, i.e., the yaw angle.

[0122] Step S6: Based on the optimized pose transformation results, stitch together each frame of the lidar point cloud to obtain a point cloud map.

[0123] This invention uses an optimized pose to transform the point cloud into a map coordinate system. In this way, the new point cloud and the point cloud in the original map combine to form a new radar point cloud map, achieving map updates. The overall point cloud output effect is as follows: Figure 8 As shown.

[0124] While the invention has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the invention. Therefore, it should be understood that many modifications can be made to the exemplary embodiments, and other arrangements can be designed without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that different dependent claims and features described herein can be combined in ways different from those described in the original claims. It is also understood that features described in conjunction with individual embodiments can be used in other described embodiments.

Claims

1. A geolocation method based on a 2D-3D matching network, characterized in that, Includes the following steps: S1. Track the input lidar data using a lidar odometry. S2. Extract a local satellite map based on the current location data; S3. Using the corrected lidar point cloud and the local satellite map as network inputs, the matching relationship between lidar points and two-dimensional pixels on the satellite image is obtained through a 2D-3D matching network. S4. Calculate the pose transformation relationship of the lidar frame relative to the map based on the matching relationship between the lidar points and the two-dimensional pixels on the satellite image; S5. Combine the tracking results of the lidar odometry with the registration results based on the satellite map to optimize the pose graph and obtain the optimized pose transformation result. S6. Based on the optimized pose transformation results, stitch together each frame of the lidar point cloud to obtain a point cloud map. Step S1 specifically includes the following steps: S11. Preprocess the input LiDAR points and extract corner points and planar points based on curvature; S12. The front end constructs point-line constraints and point-surface constraints based on the points extracted from the previous and next frames to solve the inter-frame pose transformation. S13. The backend finds the grid index in the current frame map based on the frontend results, and then constructs a local map to optimize the current frame. It constructs point-line constraints and point-polygon constraints to optimize and update the pose transformation between the odometer and the map. Step S2 specifically includes the following steps: S21. Determine whether system initialization is complete. If system initialization is not complete, take the current GPS location as the initial location and retrieve a satellite map of a certain size. S22. When initialization is complete and the system is running normally, use the optimized pose to capture a local satellite image on the satellite map as part of the input to the 2D-3D matching network.

2. The geolocation method based on a 2D-3D matching network according to claim 1, characterized in that, The formula for calculating the curvature of each point when extracting corner points and planar points based on curvature is as follows: Where S is the set of points surrounding point i on the same scan line as i when calculating curvature. Let i be the coordinates of point i in the k-th frame. Let j be the coordinates of point j in the k-th frame.

3. The geolocation method based on a 2D-3D matching network according to claim 1, characterized in that, Step S12 specifically involves: The residual is calculated from the successfully matched corner point pairs, using the following formula: Where point i is a corner point in the (k+1)th frame, and points j and l are the two corner points in the kth frame that match point i. Let i be the coordinates of the midpoint i in the (k+1)th frame in the lidar coordinate system of the kth frame. and Let i be the coordinates of point j and point l in the lidar coordinate system of the k-th frame. Geometrically, i is the distance from point i to the straight line formed by points j and l. The residual is calculated from the successfully matched planar point pairs using the following formula: Where point i is a planar point in frame k+1, and points j, l, and m are three planar points in frame k that match point i. Let i be the coordinates of the midpoint i in the (k+1)th frame in the lidar coordinate system of the kth frame. and Let i be the coordinates of points j, l, and m in the lidar coordinate system of the k-th frame. Geometrically, i is the distance from point i to the plane formed by points j, l, and m.

4. The geolocation method based on a 2D-3D matching network according to claim 1, characterized in that, Step S3 specifically includes the following steps: S31. Use Feature Pyramid Network (FPN) to extract coarse and fine features from a 2D image. The coarse feature map of the extracted 2D image is 1 / 16 the size of the original image, and the fine feature map of the extracted 2D image is 1 / 8 the size of the original image. S32. Perform four downsampling operations on the point cloud, use the KPConv backbone to extract multi-scale local features of the point cloud to obtain coarse and fine features, and use Local Pool to aggregate features in the Z-axis direction. S33. After the coarse features are extracted by the network, a coarse Transformer module is used to fuse the positions of 2D points and 3D points as well as other contextual cues contained therein to enhance the uniqueness of the features. S34. Use the enhanced coarse feature point matching module to perform initial matching selection; S35. Use the fine-stage matching module to refine the results of the coarse matching.

5. The geolocation method based on a 2D-3D matching network according to claim 1, characterized in that, Step S4 specifically includes the following steps: S41. Randomly select three pairs of matching points between lidar and satellite images and calculate the pose transformation from lidar frame to satellite map through reprojection error. S42. Verify all matching pairs based on the calculated pose transformation, and calculate the in-model point rate; S43. If the in-point ratio of the model is insufficient, repeat steps S41 and S42. Otherwise, construct a least squares problem using the in-point ratio of the obtained model and solve it to obtain the final pose transformation from the lidar frame to the satellite map.

6. The geolocation method based on a 2D-3D matching network according to claim 1, characterized in that, Step S5 specifically includes the following steps: S51. Using the odometry tracking results, construct a pose graph optimization problem. Calculate the edge error between vertices in the pose graph by comparing the relative pose of the lidar frame with the relative pose calculated based on the global pose. S52. The pose graph is optimized by using prior edges and fusing the positioning results of LiDAR frames based on satellite maps.

7. The geolocation method based on a 2D-3D matching network according to claim 1, characterized in that, Specifically, step S6 involves using the optimized pose transformation results to transform the point cloud frame to the map point cloud coordinate system and stitching them together to obtain an updated lidar point cloud map.

Images