Cross-modal matching positioning method and system based on visual image and point cloud map

Abstract

The application provides a cross-modal matching positioning method and system based on a visual image and a point cloud map, comprising: acquiring a visual image and an initial pose of a camera, obtaining a laser point cloud local map based on the visual image initial pose; projecting based on the laser point cloud local map to obtain a point cloud projection depth map under the initial pose visual angle; inputting the point cloud projection depth map, the visual image and the initialized pose update quantity into a trained pose solving network to obtain an optimized pose update quantity; superimposing the pose update quantity on the visual image initial pose to obtain an optimized pose, and through multiple cycles, obtaining a final pose to complete positioning. The application solves the problems of low positioning accuracy and high cost of existing automatic driving.

Description

Technical Field

[0001] This invention relates to the field of visual positioning technology, and in particular to a cross-modal matching positioning method and system based on visual images and point cloud maps. Background Technology

[0002] High-level autonomous driving tasks require high-precision position and attitude (hereinafter referred to as pose) information, thus necessitating intelligent vehicles to possess high-precision positioning capabilities. In high-level autonomous driving tasks, GPS signals are easily interfered with and have poor accuracy; therefore, intelligent vehicles generally rely on high-precision environmental maps for map matching and positioning.

[0003] In the process of building high-precision maps, the mainstream method is to use LiDAR to collect environmental data. Localization algorithms that use LiDAR point clouds as input generally rely on the registration of point cloud shapes for localization. That is, by optimizing the current vehicle pose, the acquired 3D LiDAR point cloud is aligned as closely as possible with the point cloud map of a certain area of ​​the environment, thereby estimating the most probable vehicle pose and achieving localization. LiDAR map matching localization algorithms are not sensitive to environmental disturbances such as weather and seasons that may cause changes in appearance, and have better robustness and higher accuracy. However, high-performance LiDAR is expensive and difficult to use on a large scale. It is only suitable for mounting on a small number of dedicated data collection vehicles for building high-precision maps; it is not suitable for mounting on private and commercial vehicles for localization. Moreover, localization algorithms based on LiDAR point cloud matching, such as the classic ICP and NDT algorithms, are quite sensitive to the initial pose, and the pose optimization process is difficult to converge, making them susceptible to interference in challenging environments.

[0004] Another low-cost solution is to use visual data from a monocular camera for mapping and localization. Visual mapping schemes often use algorithms such as Simultaneous Localization and Mapping (SLAM), Visual Odometry (VO), or Structure of Motion (SfM) to construct a 3D point cloud map of the environment. Unlike laser point cloud maps, visual point cloud maps can also include visual feature information. Visual map matching localization algorithms locate the vehicle by matching features in the currently acquired image with 3D landmarks in the visual point cloud map. In other words, by optimizing the current vehicle pose, the error between the reprojection points of image feature points and their matching 3D landmarks is minimized, thereby estimating the most probable vehicle pose and achieving localization. Visual map matching localization algorithms are low-cost and easy to widely apply; however, the accuracy of visual maps is relatively poor and easily affected by changes in the environmental appearance, making it difficult to meet accuracy requirements in large-scale and variable autonomous driving scenarios. Summary of the Invention

[0005] This invention provides a cross-modal matching and positioning method and system based on visual images and point cloud maps to solve the problems of low positioning accuracy and high cost in existing autonomous driving systems.

[0006] This invention provides a cross-modal matching and localization method based on visual images and point cloud maps, comprising:

[0007] Acquire the visual image and initial pose of the camera, and obtain a local map of the laser point cloud based on the initial pose of the visual image;

[0008] Projecting the laser point cloud local map onto the point cloud yields a point cloud projection depth map under the initial pose view.

[0009] The point cloud projection depth map, visual image, and initialized pose update are input into the trained pose solving network to obtain the optimized pose update.

[0010] The pose update is superimposed on the initial pose of the visual image to obtain the optimized pose. After multiple iterations, the final pose is obtained, and the localization is completed.

[0011] According to the present invention, a cross-modal matching and localization method based on visual images and point cloud maps is provided, wherein acquiring visual images and initial poses of a camera, and obtaining a local laser point cloud map based on the initial poses of the visual images, specifically includes:

[0012] Acquire visual images from the camera and a rough localization of the initial pose;

[0013] Based on the rough location, a search is performed on the pre-acquired laser point cloud map;

[0014] Generate a local map of laser point cloud in the world coordinate system near the camera's initial pose.

[0015] According to the present invention, a cross-modal matching and localization method based on visual images and point cloud maps is provided, which projects the local map of the laser point cloud to obtain a point cloud projection depth map under the initial pose viewpoint, specifically including:

[0016] Based on the camera's initial pose and internal parameters, the local map of the laser point cloud in the world coordinate system is projected onto the camera coordinate system.

[0017] The local map of the laser point cloud in the camera coordinate system is then projected onto the normalized pixel coordinate system to obtain the point cloud projection depth map under the initial pose view of the camera.

[0018] According to the present invention, a cross-modal matching localization method based on visual images and point cloud maps is provided, which inputs the point cloud projection depth map, the visual image, and the initialized pose update into a trained pose solving network to obtain the optimized pose update, specifically including:

[0019] The pose solving network adopts a full attention network, and the visual image and the point cloud projection depth map are input into the full attention network;

[0020] The visual image is processed by a visual feature encoder to obtain a high-dimensional visual feature map;

[0021] The point cloud projection depth map is processed by the point cloud feature encoder to obtain a high-dimensional point cloud feature map.

[0022] High-dimensional visual features and high-dimensional point cloud features belong to different modalities. Taking each visual feature in the high-dimensional visual feature map as a benchmark, the similarity between the visual feature and each point cloud feature in the high-dimensional point cloud feature map is calculated to obtain the three-dimensional feature matching cost.

[0023] The initial pose update value is used as the retrieval value. The 3D feature matching cost is processed to obtain the key value and content value. The retrieval value is then updated to generate the optimized pose update value.

[0024] According to the present invention, a cross-modal matching and localization method based on visual images and point cloud maps is provided, wherein the pose solving network training process is as follows:

[0025] Given a visual image and a laser point cloud map acquired at a certain moment, apply a random pose transformation to the laser point cloud map to obtain a laser point cloud with added noise.

[0026] A new point cloud projection depth map is obtained by transforming and projecting the laser point cloud with added noise using the camera's internal and external parameters.

[0027] The visual image and the new point cloud projection depth map are input into the pose solving network to obtain the relative pose between the visual image and the new point cloud projection depth map.

[0028] The relative pose is supervised to make it approximate the known true relative pose, thereby optimizing the parameters of the pose solving network.

[0029] According to the present invention, a cross-modal matching localization method based on visual images and point cloud maps is provided. The method involves superimposing the pose update amount onto the initial pose of the visual image to obtain an optimized pose, and after multiple iterations, obtaining the final pose to complete the localization. Specifically, the method includes:

[0030] In the first pose optimization process, a rough localization is used as the initial pose;

[0031] In each pose optimization process, the previous optimized pose is used as the initial pose for the current optimization process. The pose is searched and projected from the laser point cloud map to obtain a new point cloud projection depth map. The pose is iteratively optimized to obtain the final pose and complete the localization.

[0032] The present invention also provides a cross-modal matching and localization system based on visual images and point cloud maps, the system comprising:

[0033] The data acquisition module is used to acquire the visual image and initial pose of the camera, and to obtain a local map of the laser point cloud based on the initial pose of the visual image.

[0034] The projection module is used to project the laser point cloud local map to obtain a point cloud projection depth map under the initial pose view.

[0035] The pose solving module is used to input the point cloud projection depth map, visual image and initialized pose update into the trained pose solving network to obtain the optimized pose update.

[0036] The pose optimization module is used to superimpose the pose update amount onto the initial pose of the visual image to obtain the optimized pose. After multiple iterations, the final pose is obtained, and the localization is completed.

[0037] According to the present invention, a cross-modal matching and localization system based on visual images and point cloud maps is provided, wherein the data acquisition module acquires visual images from the camera and a rough localization of the initial pose.

[0038] Based on the rough location, a search is performed on the pre-acquired laser point cloud map;

[0039] A local map of laser point cloud in the world coordinate system is generated near the initial pose of the camera. According to the cross-modal matching and localization system based on visual images and point cloud maps provided by the present invention, the projection module projects the local map of laser point cloud in the world coordinate system onto the camera coordinate system based on the initial pose of the camera and the camera's internal parameters.

[0040] The local map of the laser point cloud in the camera coordinate system is then projected onto the normalized pixel coordinate system to obtain the point cloud projection depth map under the initial pose view of the camera.

[0041] According to the present invention, a cross-modal matching and localization system based on visual images and point cloud maps is provided, wherein the pose solving module adopts a full attention network according to the pose solving network, and inputs the visual image and the point cloud projection depth map into the full attention network input;

[0042] The visual image is processed by a visual feature encoder to obtain a high-dimensional visual feature map;

[0043] The point cloud projection depth map is processed by the point cloud feature encoder to obtain a high-dimensional point cloud feature map.

[0044] High-dimensional visual features and high-dimensional point cloud features belong to different modalities. Taking each visual feature in the high-dimensional visual feature map as a benchmark, the similarity between the visual feature and each point cloud feature in the high-dimensional point cloud feature map is calculated to obtain the three-dimensional feature matching cost.

[0045] The initial pose update value is used as the retrieval value. The 3D feature matching cost is processed to obtain the key value and content value. The retrieval value is then updated to generate the optimized pose update value.

[0046] According to the present invention, a cross-modal matching and localization system based on visual images and point cloud maps is provided, wherein the pose optimization module uses a coarse localization as the initial pose during the first pose optimization process;

[0047] In each pose optimization process, the previous optimized pose is used as the initial pose for the current optimization process. The pose is searched and projected from the laser point cloud map to obtain a new point cloud projection depth map. The pose is iteratively optimized to obtain the final pose and complete the localization.

[0048] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the cross-modal matching localization method based on visual images and point cloud maps as described above.

[0049] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the cross-modal matching localization method based on visual images and point cloud maps as described above.

[0050] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the cross-modal matching and localization method based on visual images and point cloud maps as described above.

[0051] This invention provides a cross-modal matching localization method and system based on visual images and point cloud maps. By inputting visual images and point cloud projection depth maps into a pose solving network, the final pose is obtained through multiple optimizations. In the localization stage, only a low-cost visual camera is required as a sensor, making it cost-effective and suitable for large-scale commercial use. Compared to traditional laser point cloud localization algorithms such as ICP and NDT, this invention can iteratively optimize the pose during the localization stage by iteratively searching for point cloud projection depth maps from the laser point cloud map and optimizing the pose. Furthermore, because it uses an end-to-end trained neural network to solve the pose, the optimization process is smoother, less sensitive to the initial pose, and better adaptable to extreme scenarios with large GPS signal errors. In the mapping stage, LiDAR is used for mapping, resulting in a more accurate point cloud map that is less sensitive to changes in environmental appearance. Since the optimized laser point cloud map does not contain point cloud features, the map storage space required is smaller. Moreover, because the laser point cloud map is more accurate than the visual point cloud map, this invention offers higher localization accuracy, better meeting the accuracy requirements of autonomous vehicles for positioning. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0053] Figure 1 This is one of the flowcharts of the cross-modal matching and localization method based on visual images and point cloud maps provided by the present invention;

[0054] Figure 2 This is the second flowchart of the cross-modal matching and localization method based on visual images and point cloud maps provided by the present invention;

[0055] Figure 3 This is the third flowchart of the cross-modal matching and localization method based on visual images and point cloud maps provided by the present invention;

[0056] Figure 4 This is the fourth flowchart of the cross-modal matching and localization method based on visual images and point cloud maps provided by the present invention;

[0057] Figure 5 This is the fifth flowchart of the cross-modal matching and localization method based on visual images and point cloud maps provided by the present invention;

[0058] Figure 6 This is a schematic diagram of the module connections of the cross-modal matching and positioning system based on visual images and point cloud maps provided by the present invention;

[0059] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention;

[0060] Figure 8 This is a diagram of the optimized positioning framework with multiple iterations provided by the present invention;

[0061] Figure 9 This is a schematic diagram of the pose solving network structure provided by the present invention.

[0062] Figure label:

[0063] 110: Data acquisition module; 120: Projection module; 130: Pose solving module; 140: Pose optimization module;

[0064] 710: Processor; 720: Communication interface; 730: Memory; 740: Communication bus. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0066] The following is combined Figures 1-5 This invention describes a cross-modal matching and localization method based on visual images and point cloud maps, comprising:

[0067] S100: Acquire the visual image and initial pose of the camera, and obtain a local map of the laser point cloud based on the initial pose of the visual image;

[0068] S200. Project the laser point cloud local map to obtain a point cloud projection depth map with the initial pose viewpoint.

[0069] S300. Input the point cloud projection depth map, visual image and initialized pose update amount into the trained pose solving network to obtain the optimized pose update amount.

[0070] S400: The pose update amount is superimposed on the initial pose of the visual image to obtain the optimized pose. After multiple iterations, the final pose is obtained, and the localization is completed.

[0071] This invention presents a visual matching and positioning scheme using camera data within a laser point cloud map, which balances accuracy and cost. Laser point cloud maps offer high accuracy, are constructed using specialized data acquisition vehicles, and have an acceptable initial cost. Visual cameras have low mass production costs and can be commercially deployed on a large scale. By matching camera data from different modalities with laser point cloud data, this invention enables high-precision positioning at low cost.

[0072] Acquire the camera's visual image and initial pose, and obtain a local map of the laser point cloud based on the initial pose of the visual image, specifically including:

[0073] S101. Acquire the visual image from the camera and a rough localization of the initial pose;

[0074] S102. Based on the rough positioning, search in the pre-acquired laser point cloud map;

[0075] S103. Generate a local map of laser point cloud in the world coordinate system near the initial pose of the camera.

[0076] In this invention, an offline laser point cloud map needs to be constructed. This invention does not limit the method for constructing the laser point cloud map. The system input is the visual image I acquired when the vehicle needs to be located, along with its GPS positioning signal. The output is the vehicle's current pose. The system utilizes the camera's initial pose. A rough location is performed to search within the laser point cloud map, yielding Q. k Local map of nearby laser point clouds Where w represents In a global coordinate system, laser point cloud maps can improve positioning accuracy.

[0077] Projecting the laser point cloud local map onto the point cloud yields a point cloud projection depth map from the initial pose viewpoint, specifically including:

[0078] S201. Based on the initial pose of the camera and the camera's internal parameters, project the local map of the laser point cloud in the world coordinate system to the camera coordinate system.

[0079] S202. Project the local map of the laser point cloud in the camera coordinate system to the normalized pixel coordinate system to obtain the point cloud projection depth map under the initial pose view of the camera.

[0080] In this invention, based on the initial pose Q of the camera k and camera internal parameters The local map of the laser point cloud in the world coordinate system is projected onto the camera coordinate system c, and then onto the normalized pixel coordinate system i to obtain the point cloud projection depth map D with the same resolution as the visual image I. The calculation process is as follows:

[0081]

[0082]

[0083] D(y,x)=Z#(3)

[0084] This invention does not impose restrictions on point cloud preprocessing operations (such as outlier removal, motion compensation, etc.) during the point cloud map projection process.

[0085] The point cloud projection depth map, visual image, and initialized pose update are input into the trained pose solving network to obtain the optimized pose update, specifically including:

[0086] S301. The pose solving network adopts a full attention network, and the visual image and the point cloud projection depth map are input into the full attention network.

[0087] S302. The visual image is processed by a visual feature encoder to obtain a high-dimensional visual feature map;

[0088] S303. The point cloud projection depth map is processed by the point cloud feature encoder to obtain a high-dimensional point cloud feature map.

[0089] S304. High-dimensional visual features and high-dimensional point cloud features belong to different modalities. Taking each visual feature in the high-dimensional visual feature map as a benchmark, the similarity between the visual feature and each point cloud feature in the high-dimensional point cloud feature map is calculated to obtain the three-dimensional feature matching cost.

[0090] S305. Using the initialized pose update amount as the retrieval value, process the 3D feature matching cost to obtain the key value and content value, update the retrieval value, and generate the optimized pose update amount.

[0091] In this invention, the pose solving network is a dual-input, single-output network that employs an attention mechanism (a fully attention-based Transformer network) to find data matching relationships and solve for the pose in different modal data. The two input branches take the visual image I and the point cloud projection depth map D as inputs, respectively, and are processed by a visual feature encoder. and point cloud feature encoder Obtain high-dimensional visual feature maps of different modalities and high-dimensional point cloud feature maps Where h and w are the height and width of the feature map, respectively, and c is the dimension of the feature map.

[0092] Each visual feature in the high-dimensional visual feature map Based on this, compute visual features and high-dimensional point cloud feature maps for each point cloud feature. similarity Obtain the cost of 3D feature matching After obtaining the 3D feature matching cost C, the previous pose update value is used in the full attention network. As the retrieval value q, the feature matching cost is processed to obtain the key value k and the content value v. Based on the attention-based decoder model, the retrieval value q is updated to obtain the optimized pose update.

[0093] Full attention networks can be stacked in multiple layers. The first layer of the full attention network randomly initializes the pose update of the input. Each subsequent layer (e.g., the (i+1)th layer) is the pose update amount output by the previous layer. Taking it as input, iteratively optimize it to obtain This process simulates the iterative optimization strategy in the traditional Gauss-Newton method, making it more stable and easier to obtain the optimal optimization result.

[0094] In a full attention network, a linear encoding function Embed is used. q () updates the input pose. Convert to a high-dimensional feature vector, which will be used as the retrieval value q:

[0095]

[0096] Flatten the feature matching cost C and treat it as h×w hw-dimensional feature vectors. Similarly, it is processed by the linear encoding function Embed k (),Embed v (), to obtain the key value k and the content value v:

[0097]

[0098]

[0099] Based on the attention mechanism, the retrieved value q is updated:

[0100] q=BN((q T k)v+q)#(7)

[0101] BN() is used for layer regularization. This invention does not restrict the attention mechanism; various attention mechanisms such as multi-head attention, self-attention, cross-attention, and deformable attention can be applied without hindrance. Finally, the updated retrieval value q is converted back to the pose update value through a linear decoding function.

[0102]

[0103] In this invention, the pose-solving network training process is as follows:

[0104] S401. Given a visual image and a laser point cloud map acquired at a certain moment, apply a random pose transformation to the laser point cloud map to obtain a laser point cloud with added noise.

[0105] S402. Based on the laser point cloud with added noise, a new point cloud projection depth map is obtained by transforming and projecting the laser point cloud using the camera's internal and external parameters.

[0106] S403. Input the visual image and the new point cloud projection depth map into the pose solving network to obtain the relative pose between the visual image and the new point cloud projection depth map.

[0107] S404. Supervise the relative pose to make it approximate the known true relative pose and optimize the parameters of the pose solving network.

[0108] In the cross-modal visual matching and localization system proposed in this invention, the pose solving network requires targeted training of its parameters. The pose solving network needs training data of visual images and laser point clouds, where the visual images and laser point clouds are simultaneously acquired data from different modalities. Furthermore, the intrinsic parameters K of the visual camera and its relative pose with the lidar are obtained through sensor intrinsic and extrinsic parameter calibration tools.

[0109] During training, given a visual image I and a laser point cloud map P acquired at a certain moment... l Based on the internal parameter K and the external parameter T cl The point cloud P in the LiDAR coordinate system is first transformed to the camera coordinate system, and then projected onto the pixel coordinate system to obtain the point cloud projection depth map D corresponding to the visual image I.

[0110]

[0111]

[0112] D(y,x)=Z#(11)

[0113] At this point, the relative pose between I and D should be 0. In actual localization scenarios, the point cloud projection depth map D obtained by projecting the initial pose from the laser point cloud map is generally different from the pose of I. To simulate this situation, this invention trains the pose solving network to calculate the laser point cloud map P. l A random pose transformation T was applied. rand This yields a laser point cloud P with added noise. l′ :

[0114]

[0115] Using internal and external parameters to P l′ Transformation and projection are performed to obtain a new point cloud projected depth map D'. I and D' are then input into the pose solving network to obtain the relative pose Q between I and D'. Since the true relative pose T between I and D' is known... rand It directly supervises the relative pose, making the predicted relative pose Q approximate the true relative pose T. rand This is used to optimize the parameters θ of the pose solving network.

[0116] θ * =argmin θ (‖QT rand ‖)#(13)

[0117] This invention calculates the predicted relative pose Q and the true relative pose T. rand There are no restrictions on the methods used to differentiate between them, such as directly calculating the Euclidean or L1 distance from the transformation matrix, or converting the transformation matrix into translation and rotation quantities and then calculating the Euclidean or L1 distance.

[0118] The pose update is superimposed on the initial pose of the visual image to obtain the optimized pose. After multiple iterations, the final pose is obtained, and localization is completed. Specifically, this includes:

[0119] In the first pose optimization process, a rough localization is used as the initial pose;

[0120] In each pose optimization process, the previous optimized pose is used as the initial pose for the current optimization process. The pose is searched and projected from the laser point cloud map to obtain a new point cloud projection depth map. The pose is iteratively optimized to obtain the final pose and complete the localization.

[0121] In one specific embodiment, system verification was performed. To verify the effectiveness of the invention, it was validated on the publicly available KITTI dataset. Training was conducted on sequences 03, 05, 06, 07, 08, and 09, and testing was performed on sequence 00. The laser point cloud map was constructed using a laser SLAM algorithm. In the experiment, the pose-solving network was used twice, meaning the pose was iteratively optimized twice. The pose-solving network used contained a 6-layer full attention network, meaning the pose was iteratively optimized six times within the network.

[0122] Table 1. Performance of the cross-modal matching localization system

[0123]

[0124] As shown in Table 1, although the initial relative pose error of the visual image is large, the positioning accuracy is significantly improved after one cross-modal matching localization. However, since the point cloud projection depth map is affected by the initial pose, using the optimized pose for map search and projection to obtain a new point cloud projection depth map, followed by matching localization, will further improve the performance. Finally, the system achieves high positioning accuracy, and since it only requires a monocular camera for localization, the positioning cost is low.

[0125] This invention provides a cross-modal matching localization method based on visual images and point cloud maps. By inputting visual images and point cloud projection depth maps into a pose solving network, the final pose is obtained through multiple optimizations. In the localization stage, only a low-cost visual camera is required as a sensor, making it cost-effective and suitable for large-scale commercial use. Compared to traditional laser point cloud localization algorithms such as ICP and NDT, this invention can iteratively optimize the pose during the localization stage by iteratively searching for point cloud projection depth maps from the laser point cloud map and optimizing the pose. Furthermore, because it uses an end-to-end trained neural network to solve the pose, the optimization process is smoother, less sensitive to the initial pose, and better adaptable to extreme scenarios with large GPS signal errors. In the mapping stage, LiDAR is used for mapping, resulting in a more accurate point cloud map that is less sensitive to changes in environmental appearance. Since the optimized laser point cloud map does not contain point cloud features, the map storage space required is smaller. Moreover, because the laser point cloud map is more accurate than the visual point cloud map, this invention offers higher localization accuracy, better meeting the accuracy requirements of autonomous vehicles for positioning.

[0126] refer to Figure 6 , Figure 8 and Figure 9 The present invention also discloses a cross-modal matching and localization system based on visual images and point cloud maps, the system comprising:

[0127] The data acquisition module 110 is used to acquire the visual image and initial pose of the camera, and to obtain a local map of the laser point cloud based on the initial pose of the visual image.

[0128] Projection module 120 is used to project a point cloud projection depth map based on the local map of the laser point cloud, under the initial pose view.

[0129] The pose solving module 130 is used to input the point cloud projection depth map, visual image and initialized pose update amount into the trained pose solving network to obtain the optimized pose update amount.

[0130] The pose optimization module 140 is used to superimpose the pose update amount onto the initial pose of the visual image to obtain the optimized pose. After multiple iterations, the final pose is obtained, and the localization is completed.

[0131] Among them, the data acquisition module 110 acquires the visual images from the camera and the rough positioning of the initial pose;

[0132] Based on the rough location, a search is performed on the pre-acquired laser point cloud map;

[0133] Generate a local map of laser point cloud in the world coordinate system near the camera's initial pose.

[0134] The projection module 120 projects a local map of the laser point cloud in the world coordinate system to the camera coordinate system based on the camera's initial pose and internal parameters.

[0135] The camera coordinate system is then projected onto the normalized pixel coordinate system to obtain the point cloud projection depth map under the initial pose view of the camera.

[0136] The pose solving module 130 uses a full attention network to input the visual image and the point cloud projection depth map into the full attention network.

[0137] The visual image is processed by a visual feature encoder to obtain a high-dimensional visual feature map;

[0138] The point cloud projection depth map is processed by the point cloud feature encoder to obtain a high-dimensional point cloud feature map.

[0139] High-dimensional visual features and high-dimensional point cloud features belong to different modalities. Taking each visual feature in the high-dimensional visual feature map as a benchmark, the similarity between the visual feature and each point cloud feature in the high-dimensional point cloud feature map is calculated to obtain the three-dimensional feature matching cost.

[0140] The initial pose update value is used as the retrieval value. The 3D feature matching cost is processed to obtain the key value and content value. The retrieval value is then updated to generate the optimized pose update value.

[0141] The training process of the pose solving network is as follows:

[0142] Given a visual image and a laser point cloud map acquired at a certain moment, apply a random pose transformation to the laser point cloud map to obtain a laser point cloud with added noise.

[0143] A new point cloud projection depth map is obtained by transforming and projecting the laser point cloud with added noise using the camera's internal and external parameters.

[0144] The visual image and the new point cloud projection depth map are input into the pose solving network to obtain the relative pose between the visual image and the new point cloud projection depth map.

[0145] The relative pose is supervised to make it approximate the known true relative pose, thereby optimizing the parameters of the pose solving network.

[0146] The pose optimization module 140 uses a rough positioning as the initial pose during the first pose optimization process.

[0147] In each pose optimization process, the previous optimized pose is used as the initial pose for the current optimization process. The pose is searched and projected from the laser point cloud map to obtain a new point cloud projection depth map. The pose is iteratively optimized to obtain the final pose and complete the localization.

[0148] This invention provides a cross-modal matching and localization system based on visual images and point cloud maps. By inputting visual images and point cloud projection depth maps into a pose solving network, the final pose is obtained through multiple optimizations. During the localization phase, only a low-cost visual camera is required as a sensor, making it cost-effective and suitable for large-scale commercial use. Compared to traditional laser point cloud localization algorithms such as ICP and NDT, this invention iteratively optimizes the pose during the localization phase by searching for the point cloud projection depth map from the laser point cloud map and optimizing the pose. Furthermore, because it uses an end-to-end trained neural network to solve the pose, the optimization process is smoother, less sensitive to the initial pose, and better adaptable to extreme scenarios with large GPS signal errors. In the mapping phase, LiDAR is used for mapping, resulting in a more accurate point cloud map that is less sensitive to changes in environmental appearance. Since the optimized laser point cloud map does not contain point cloud features, the map storage space required is smaller. Moreover, because the laser point cloud map is more accurate than the visual point cloud map, this invention achieves higher localization accuracy, better meeting the accuracy requirements of autonomous vehicles for localization.

[0149] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7 As shown, the electronic device may include a processor 710, a communications interface 720, a memory 730, and a communication bus 740, wherein the processor 710, communications interface 720, and memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a cross-modal matching and localization method based on visual images and point cloud maps. This method includes: acquiring visual images and initial poses from a camera; and obtaining a local laser point cloud map based on the initial pose of the visual images.

[0150] Projecting the laser point cloud local map onto the point cloud yields a point cloud projection depth map under the initial pose view.

[0151] The point cloud projection depth map, visual image, and initialized pose update are input into the trained pose solving network to obtain the optimized pose update.

[0152] The pose update is superimposed on the initial pose of the visual image to obtain the optimized pose. After multiple iterations, the final pose is obtained, and the localization is completed.

[0153] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0154] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program, the computer program being able to be stored on a non-transitory computer-readable storage medium, the computer program being executed by a processor, the computer being able to execute the cross-modal matching and localization method based on visual images and point cloud maps provided by the above methods, the method including: acquiring visual images and initial poses of a camera, and obtaining a local map of laser point cloud based on the initial poses of the visual images;

[0155] Projecting the laser point cloud local map onto the point cloud yields a point cloud projection depth map under the initial pose view.

[0156] The point cloud projection depth map, visual image, and initialized pose update are input into the trained pose solving network to obtain the optimized pose update.

[0157] The pose update is superimposed on the initial pose of the visual image to obtain the optimized pose. After multiple iterations, the final pose is obtained, and the localization is completed.

[0158] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the cross-modal matching and localization method based on visual images and point cloud maps provided by the above methods, the method comprising: acquiring a visual image and an initial pose of a camera, and obtaining a local map of laser point cloud based on the initial pose of the visual image;

[0159] Projecting the laser point cloud local map onto the point cloud yields a point cloud projection depth map under the initial pose view.

[0160] The point cloud projection depth map, visual image, and initialized pose update are input into the trained pose solving network to obtain the optimized pose update.

[0161] The pose update is superimposed on the initial pose of the visual image to obtain the optimized pose. After multiple iterations, the final pose is obtained, and the localization is completed.

[0162] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0163] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0164] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cross-modal matching and localization method based on visual images and point cloud maps, characterized in that, include: Acquire the visual image and initial pose of the camera, and obtain a local map of laser point cloud based on the visual image and initial pose; Projecting the laser point cloud local map onto the point cloud yields a point cloud projection depth map under the initial pose view. The point cloud projection depth map, visual image, and initialized pose update are input into the trained pose solving network to obtain the optimized pose update. The pose update amount is superimposed on the initial pose to obtain the optimized pose. After multiple iterations, the final pose is obtained, and the localization is completed. The step of inputting the point cloud projection depth map, visual image, and initialized pose update into the trained pose solving network to obtain the optimized pose update specifically includes: The pose solving network adopts a full attention network, and the visual image and the point cloud projection depth map are input to the full attention network. The visual image is processed by a visual feature encoder to obtain a high-dimensional visual feature map; The point cloud projection depth map is processed by the point cloud feature encoder to obtain a high-dimensional point cloud feature map. High-dimensional visual features and high-dimensional point cloud features belong to different modalities. Taking each visual feature in the high-dimensional visual feature map as a benchmark, the similarity between the visual feature and each point cloud feature in the high-dimensional point cloud feature map is calculated to obtain the three-dimensional feature matching cost. The initial pose update value is used as the retrieval value. The 3D feature matching cost is processed to obtain the key value and content value. The retrieval value is then updated to generate the optimized pose update value.

2. The cross-modal matching and localization method based on visual images and point cloud maps according to claim 1, characterized in that, The process of acquiring the camera's visual image and initial pose, and obtaining a local map of the laser point cloud based on the visual image and initial pose, specifically includes: Acquire visual images from the camera and a rough localization of the initial pose; Based on the rough location, a search is performed on the pre-acquired laser point cloud map; Generate a local map of laser point cloud in the world coordinate system near the camera's initial pose.

3. The cross-modal matching and localization method based on visual images and point cloud maps according to claim 1, characterized in that, Projecting the laser point cloud local map onto the point cloud yields a point cloud projection depth map from the initial pose viewpoint, specifically including: Based on the camera's initial pose and internal parameters, the local map of the laser point cloud in the world coordinate system is projected onto the camera coordinate system. The local map of the laser point cloud in the camera coordinate system is then projected onto the normalized pixel coordinate system to obtain the point cloud projection depth map under the initial pose view of the camera.

4. The cross-modal matching and localization method based on visual images and point cloud maps according to claim 1, characterized in that, The training process of the pose solving network is as follows: Given a visual image and a laser point cloud map acquired at a certain moment, apply a random pose transformation to the laser point cloud map to obtain a laser point cloud with added noise. A new point cloud projection depth map is obtained by transforming and projecting the laser point cloud with added noise using the camera's internal and external parameters. The visual image and the new point cloud projection depth map are input into the pose solving network to obtain the relative pose between the visual image and the new point cloud projection depth map. The relative pose is supervised to make it approximate the known true relative pose, thereby optimizing the parameters of the pose solving network.

5. The cross-modal matching and localization method based on visual images and point cloud maps according to claim 1, characterized in that, The pose update is added to the initial pose to obtain the optimized pose. After multiple iterations, the final pose is obtained, and the localization is completed. Specifically, this includes: In the first pose optimization process, a rough localization is used as the initial pose; In each pose optimization process, the previous optimized pose is used as the initial pose for the current optimization process. The pose is searched and projected from the laser point cloud map to obtain a new point cloud projection depth map. The pose is iteratively optimized to obtain the final pose and complete the localization.

6. A cross-modal matching and localization system based on visual images and point cloud maps, characterized in that, The system includes: The data acquisition module is used to acquire the visual image and initial pose of the camera, and to obtain a local map of laser point cloud based on the visual image and initial pose. The projection module is used to project the laser point cloud local map to obtain a point cloud projection depth map under the initial pose view. The pose solving module is used to input the point cloud projection depth map, visual image and initialized pose update into the trained pose solving network to obtain the optimized pose update. The pose optimization module is used to superimpose the pose update amount onto the initial pose to obtain the optimized pose. After multiple iterations, the final pose is obtained, and the localization is completed. The step of inputting the point cloud projection depth map, visual image, and initialized pose update into the trained pose solving network to obtain the optimized pose update specifically includes: The pose solving network adopts a full attention network, and the visual image and the point cloud projection depth map are input to the full attention network. The visual image is processed by a visual feature encoder to obtain a high-dimensional visual feature map; The point cloud projection depth map is processed by the point cloud feature encoder to obtain a high-dimensional point cloud feature map. High-dimensional visual features and high-dimensional point cloud features belong to different modalities. Taking each visual feature in the high-dimensional visual feature map as a benchmark, the similarity between the visual feature and each point cloud feature in the high-dimensional point cloud feature map is calculated to obtain the three-dimensional feature matching cost. The initial pose update value is used as the retrieval value. The 3D feature matching cost is processed to obtain the key value and content value. The retrieval value is then updated to generate the optimized pose update value.

7. The cross-modal matching and localization system based on visual images and point cloud maps according to claim 6, characterized in that, In the first pose optimization process, the pose optimization module uses a rough positioning as the initial pose. In each pose optimization process, the previous optimized pose is used as the initial pose for the current optimization process. The pose is searched and projected from the laser point cloud map to obtain a new point cloud projection depth map. The pose is iteratively optimized to obtain the final pose and complete the localization.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the cross-modal matching localization method based on visual images and point cloud maps as described in any one of claims 1 to 5.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the cross-modal matching and localization method based on visual images and point cloud maps as described in any one of claims 1 to 5.

Images