Multi-sensor fusion vehicle rapid positioning method and device, storage medium and electronic equipment

Abstract

The invention discloses a multi-sensor fusion vehicle rapid positioning method and device, a storage medium and electronic equipment, and belongs to the technical field of vehicle positioning. The positioning method comprises the following steps: firstly, acquiring standard data of a vehicle in a driving process on a preset planned path through multiple sensors, and constructing an offline three-dimensional map containing a key frame image database; secondly, collecting fact pose data in a vehicle driving process as initial pose estimation, collecting online current frame environment image data in real time through a camera, extracting a global feature vector in the online current frame environment image, matching K pictures similar to the image from a key frame image database, and carrying out initial pose estimation on the K pictures; and extracting local feature points and descriptors in the K pictures, screening out the most approximate picture from the K pictures as a target picture, and calculating to obtain real-time pose estimation of the vehicle. According to the method provided by the invention, more efficient, more accurate and more reliable vehicle positioning is realized through the cooperation of online processing and offline processing.

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle positioning, and in particular to a multi-sensor fusion vehicle rapid positioning method, device, storage medium and electronic equipment. Background Art

[0002] The autonomous driving system needs to plan the best route to the destination based on the vehicle's accurate location. It also needs to make driving decisions such as changing lanes, overtaking, and parking based on the vehicle's positioning information. Therefore, in the field of intelligent driving, vehicle positioning is an indispensable part and plays a decisive role in the commercialization and popularization of autonomous driving vehicles.

[0003] At present, single-type sensors are mainly used for vehicle positioning, including GPS, inertial navigation system (INS), lidar positioning and camera visual positioning. In some cases, single-type sensors may fail or their performance may degrade. For example, GPS has wide coverage but poor performance in signal-blocked areas, INS can work in a signal-free environment but the error accumulates over time, lidar may be affected in bad weather, and camera performance may also degrade under large changes in lighting. In addition, when moving at high speed or turning quickly, single-type sensors may not be able to provide stable and reliable positioning information. It can be seen that there are many disadvantages in using single-type sensors for vehicle positioning.

[0004] The invention patent application document with publication number CN117014815A discloses a multi-sensor multi-vehicle collaborative positioning system and method. This method realizes collaborative positioning of vehicles by fusing GPS data, IMU data, radar data and camera data. However, this positioning method performs vehicle positioning based entirely on data collected online in real time, without offline standard data for benchmarking. Therefore, the accuracy and reliability of positioning need to be further verified. In addition, this positioning method requires a large amount of calculation of real-time data when performing vehicle positioning, which will affect the speed and efficiency of positioning. Summary of the invention

[0005] In view of the above-mentioned problems, the present invention aims to provide a multi-sensor fusion vehicle rapid positioning method, device, storage medium and electronic device. Before real-time vehicle positioning, an offline three-dimensional map containing a key frame image database is pre-constructed, and then a more efficient, more accurate and more reliable vehicle positioning is achieved through the coordination of online processing and offline processing.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] A multi-sensor fusion vehicle rapid positioning method comprises the following steps:

[0008] Step 1: Collect standard data of the vehicle driving on the preset planned path through multiple sensors to build an offline three-dimensional map containing a key frame image database;

[0009] Step 2: When positioning the vehicle, the vehicle travels along the preset planned path, collects the real-time posture data of the vehicle as the initial posture estimation, collects the online current frame environment image data around the vehicle, and extracts the global feature vector in the online current frame environment image data;

[0010] Step 3: According to the global feature vector extracted in step 2, K images similar to the online current frame environment image are matched in the offline key frame image database established in step 1, and the target image is selected from the K images;

[0011] Step 4: Calculate the real-time pose estimation of the vehicle based on the target image;

[0012] Step 5: Use the real-time pose estimation obtained in step 5 to update the initial pose estimation in step 2, repeat steps 3 to 5, and obtain the current pose of the vehicle in real time as the vehicle positioning information output.

[0013] Furthermore, the specific operation of step one includes the following steps:

[0014] S1: Collect standard data of the vehicle during driving on a preset planned path through multiple sensors; the multiple sensors include a GPS / IMU combined navigation system, a laser radar and a camera;

[0015] S2: Based on the standard data collected in step S1, an offline three-dimensional map including a key frame image database is constructed.

[0016] Furthermore, the specific operation of step S2 includes the following steps:

[0017] S201: performing spatiotemporal alignment on the pose data, three-dimensional point cloud data and image data collected in step S1;

[0018] S202: Processing the aligned three-dimensional point cloud data to obtain the global pose of the laser radar, and estimating the global pose of the camera in the world reference system;

[0019] S203: Select key frame images from the aligned image data to build an offline key frame image database;

[0020] S204: extracting a global feature vector of each key frame image in the key frame image database, and storing the global feature vector in the offline key frame image database;

[0021] S205: extracting local feature points and descriptors of each key frame image in the key frame image database, and storing them in the offline key frame image database.

[0022] S206: constructing an offline three-dimensional map according to the global pose of the camera in the world coordinate system, and the global feature features, local feature points and descriptors of each key frame image.

[0023] Furthermore, the specific operation of step S201 includes the following steps:

[0024] S2011: Based on the timestamp of the GPS / IMU integrated navigation system, the timestamps of the lidar and camera are aligned through the timestamp difference algorithm to achieve time synchronization;

[0025] S2012: Obtain external parameters from the laser radar to the GPS / IMU integrated navigation system through calibration, and spatially align the pose data with the 3D point cloud data; spatially align the 3D point cloud data with the image data through the external parameters from the laser radar to the camera;

[0026] Among them, the conversion relationship between the pose data and the three-dimensional point cloud data for spatial alignment is P nav =T lidar-to- nav P lidar , the conversion relationship between the spatial alignment of 3D point cloud data and image data is P camera =T lidar-to-camera P lidar ;

[0027] Where P lidar is a point in the laser radar coordinate system, P nav is a point in the combined navigation coordinate system; T lidar-to-nav represents the transformation matrix from the laser radar coordinate system to the integrated navigation coordinate system; P camera is a point in the camera coordinate system; T lidar-to-camera Represents the transformation matrix from the lidar coordinate system to the camera coordinate system.

[0028] Furthermore, the specific operation of step 2 includes the following steps:

[0029] S3: When positioning the vehicle, the vehicle travels along the preset planned path, and the real-time posture data of the vehicle collected by the GPS / IMU integrated navigation system is used as the initial posture estimation;

[0030] S4: The online current frame environment image data around the vehicle is collected in real time through the camera, and the global feature vector in the online current frame environment image data is extracted through the NetVLAD graphic retrieval model.

[0031] 6. The multi-sensor fusion vehicle rapid positioning method according to claim 5 is characterized in that the specific operation of step 3 includes the following steps:

[0032] S5: According to the global feature vector extracted in step S4, K images similar to the online current frame environment image are matched in the offline key frame image database established in step S2;

[0033] S6: Extract local feature points and descriptors from the K images matched in step S5 based on the Superpoint deep network model;

[0034] S7: Based on the global feature vector proposed in step S4 and the local feature points and descriptors extracted from the K pictures in step S5, the most similar picture is selected from the K pictures as the target picture.

[0035] Furthermore, the specific operation of step S7 includes the following steps:

[0036] S701: Calculate the global feature vector distances of K images respectively;

[0037] d i =||F current -F i ||2

[0038] In the formula, F current is the global feature vector of the current frame environment image online, F i is the global feature vector extracted from the i-th (i=1, 2, ..., K)th picture, d i is the global feature vector distance between the i-th image and the online current frame environment image, and ||·||2 represents the Euclidean distance;

[0039] S702: Calculate the descriptor distances of local feature points of K images;

[0040]

[0041] Where D current is the local feature point descriptor set of the online current frame environment image, D i is the local feature point descriptor set of the i-th (i=1, 2, ..., K) picture, d′ ij is the distance between the local feature point descriptor of the i-th image and the online current frame environment image, Represents the jth local descriptor in the local feature point descriptor set of the current frame environment image, D ij represents the jth local descriptor in the local feature point descriptor set of the i-th image, and HD represents the Hamming distance;

[0042] S703: weighting the distance results calculated in step S701 and step S702,

[0043]

[0044] In the formula, d″ i represents the weighted distance, α represents the weight of the global feature vector distance, N i Indicates the number of local feature point descriptors of the i-th image;

[0045] S704: nearest neighbor search, select target image;

[0046] After calculating the weighted distance of K images, the image with the smallest weighted distance is selected as the nearest neighbor, i * = arg min i d″ i , where i * Represents the index of the nearest neighbor image.

[0047] Furthermore, the present invention also includes a multi-sensor fusion vehicle rapid positioning device, which includes an offline data acquisition device, an offline map construction device, an initial posture estimation device, an online data acquisition device, a global feature extraction device, an image matching device, a local feature extraction device, a target image screening device, a real-time posture estimation device and a posture output device;

[0048] The offline data acquisition device, offline map construction device, initial pose estimation device, online data acquisition device, global feature extraction device, image matching device, local feature extraction device, target image screening device, real-time pose estimation device and pose output device are all implemented based on the vehicle rapid positioning method as described above.

[0049] Furthermore, the present invention also includes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the vehicle rapid positioning method as described above.

[0050] Furthermore, the present invention also includes an electronic device, which includes at least one processor and a memory communicatively connected to the processor; wherein the memory stores instructions executable by the processor, and the instructions are executed by the processor so that the processor can execute the vehicle rapid positioning method as described above.

[0051] The beneficial effects of the present invention are:

[0052] 1. The multi-sensor fusion vehicle rapid positioning method of the present invention pre-builds an offline three-dimensional map including an offline key frame image database by fusing data collected by three sensors: a GPS / IMU combined navigation system, a lidar, and a camera. On the one hand, it provides a high-precision three-dimensional map model of vehicle driving, which helps to improve the positioning accuracy in the online stage; on the other hand, the structure of the vehicle driving environment can be captured and analyzed in advance through offline processing, which helps to quickly retrieve and match features in the online stage, thereby providing stable and reliable positioning information when the vehicle moves at high speed or turns quickly.

[0053] 2. In the initial stage of real-time online positioning of the vehicle, the multi-sensor fusion vehicle rapid positioning method of the present invention uses the vehicle posture data collected in real time by the GPS / IMU combined navigation system as the initial posture estimation, providing the ability to quickly start positioning. Moreover, in the multi-sensor fusion process, a good initial posture estimation can improve the efficiency and effect of data fusion.

[0054] 3. In the real-time online positioning stage of the vehicle, the multi-sensor fusion vehicle rapid positioning method of the present invention mainly uses a camera to collect the current environment image information around the vehicle, and then first uses the NetVLAD graphic retrieval model to extract global coarse features so as to filter out K approximate pictures from the offline key frame database, and then uses the Superpoint deep network model to extract the local feature points and descriptors in the K approximate pictures, and then uses the KNN algorithm to filter out the most similar target picture from the K approximate pictures based on the local feature points and descriptors, and then estimates the real-time posture of the vehicle based on the key feature points of the target picture and the coordinates of the three-dimensional space points corresponding to the key feature points, and finally uses the current posture estimate of the vehicle to update the previous moment posture estimate, and marks the current posture of the vehicle on the offline three-dimensional map to achieve vehicle positioning. The method can match the image captured in the online stage with the key frame image in the offline database to determine the current posture of the vehicle, and achieves more efficient, more accurate and more reliable vehicle positioning through the cooperation of online processing and offline processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 The present invention is a flowchart of a method for rapid vehicle positioning using multi-sensor fusion in accordance with a first embodiment of the present invention.

[0056] Figure 2 This is a flowchart of offline three-dimensional map construction in the second embodiment of the present invention.

[0057] Figure 3 Schematic diagram of the principle of a vehicle rapid positioning device with multi-sensor fusion in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0058] In order to enable those skilled in the art to better understand the technical solution of the present invention, the technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0059] Embodiment 1:

[0060] Embodiment 1 provides a multi-sensor fusion vehicle rapid positioning method, as shown in the attached Figure 1 As shown, the following steps are included:

[0061] S1: Collect standard data of the vehicle on the preset planned path through multiple sensors;

[0062] Specifically, the multi-sensor includes a GPS / IMU (Global Positioning System, GPS; Inertial Measurement Unit, IMU) combined navigation system, a laser radar and a camera, and the vehicle's posture data is collected through the combined navigation system; when the vehicle is driving on a preset planned path, the vehicle's posture data is collected through the GPS / IMU combined navigation system, the three-dimensional point cloud data of the vehicle's surrounding environment is collected through the laser radar, and the image data of the vehicle's surrounding environment is collected through the camera.

[0063] In the GPS / IMU combined navigation system, GPS is used to obtain the global position information of the vehicle, and IMU is used to obtain the posture data of the vehicle. The vehicle's position information and posture data are combined as the output data of the combined navigation system. The laser radar uses a 128-line or 64-line laser radar to scan and obtain 3D environmental point cloud data; the camera uses a visual surround camera to capture a 360-degree panoramic image around the vehicle.

[0064] Further, step S2: constructing an offline three-dimensional map including a key frame image database according to the standard data collected in step S1;

[0065] S3: When positioning the vehicle, the vehicle travels along the preset planned path, and the real-time posture data of the vehicle collected by the GPS / IMU integrated navigation system is used as the initial posture estimation;

[0066] S4: The online current frame environment image data around the vehicle is collected in real time through the camera, and the global feature vector in the online current frame environment image data is extracted through the NetVLAD graphic retrieval model;

[0067] Specifically, the process of extracting the global feature vector from the online current frame environment image data using the NetVLAD image retrieval model includes: ① Preprocessing: preprocess the online current frame environment image data, including scaling, normalization and other operations to adapt to the subsequent convolutional neural network processing. ② Feature extraction: use the pre-trained convolutional neural network (CNN) to forward propagate the preprocessed image to extract features. ③ Soft-assignment: use the 1x1 convolution layer in NetVLAD to perform linear transformation on the features extracted by CNN, and then perform soft assignment through the Softmax function to obtain the associated weights of each feature descriptor and cluster center. ④ Cluster center encoding: map the features of CNN to a lower-dimensional encoding space through the cluster centers (codebook) obtained through training. Each cluster center can be regarded as a point in the encoding space. ⑤ Feature aggregation: according to the weights obtained by Soft-assignment, the residuals between the feature descriptors and the corresponding cluster centers are weighted summed to obtain the residual vector of each cluster center. ⑥ Global feature vector construction: the residual vectors of all cluster centers are accumulated to form the final global feature vector. This global feature vector is a compact representation of the entire image content and can be used for subsequent retrieval or matching tasks.

[0068] S5: According to the global feature vector extracted in step S4, K pictures similar to the online current frame environment image are matched in the offline key frame image database established in step S2, and the global feature vectors of the K pictures are respectively extracted;

[0069] S6: Extract local feature points and descriptors from the K images matched in step S5 based on the Superpoint deep network model;

[0070] Specifically, the Superpoint deep network model mainly consists of two parts: a network for feature point detection and a network for descriptor generation. K images are used as input data of the Superpoint deep network model, and each image is processed using the feature point detection network in the Superpoint model to identify significant feature points, such as corner points, edge points, etc. Corner points are one of the most significant feature points in a two-dimensional image, usually appearing at the intersection of the edges of the image; edge points are located in the edge area of ​​the image, and are points in the image where the brightness changes more significantly; for each detected feature point, a local descriptor is extracted using the descriptor generation network in the Superpoint deep network model. The descriptor is an encoding of the local area around the feature point, which is used to capture the texture and structural information of the area.

[0071] S7: Based on the KNN algorithm, according to the global feature vector proposed in step S4 and the local feature points and descriptors extracted from the K pictures in step S5, the most similar picture is selected from the K pictures as the target picture;

[0072] Specifically, the specific operation of selecting the most similar image from K images as the target image based on the KNN algorithm includes the following steps:

[0073] S701: Calculate the global feature vector distances of K images respectively;

[0074] Let F current is the global feature vector of the current frame environment image online, F i is the global feature vector extracted from the i-th (i=1, 2, ..., K)th picture, then the calculation formula of the global feature vector distance is

[0075] d i =||F current -F i ‖2

[0076] Where, d i is the global feature vector distance between the ith image and the online current frame environment image, and ‖·||2 represents the Euclidean distance.

[0077] S702: local feature point descriptor distance calculation;

[0078] Let D current is the local feature point descriptor set of the online current frame environment image, D i is the local feature point descriptor set of the i-th (i=1, 2, ..., K) picture, then the calculation formula of the local feature point descriptor distance is:

[0079]

[0080] Where d′ ij is the distance between the local feature point descriptor of the i-th image and the online current frame environment image, Represents the jth local descriptor in the local feature point descriptor set of the current frame environment image, It represents the jth local descriptor in the set of local feature point descriptors of the i-th image, and HD represents the Hamming distance.

[0081] S703: weighted distance calculation;

[0082] The distance results calculated in step S701 and step S702 are weighted, and the calculation formula of the weighted distance is:

[0083]

[0084] In the formula, d″ i represents the weighted distance, α represents the weight of the global feature vector distance, N i Indicates the number of local feature point descriptors of the i-th image.

[0085] S704: nearest neighbor search, select target image;

[0086] After calculating the weighted distance of K images, the image with the smallest weighted distance is selected as the nearest neighbor, i * = arg min i d″ i , where i * Represents the index of the nearest neighbor image.

[0087] The nearest neighbor index i is determined according to the above formula * , select the corresponding picture from K pictures as the target picture.

[0088] S8: Calculate the real-time pose estimation of the vehicle based on the key feature points of the target image and the three-dimensional space point coordinates corresponding to the key feature points;

[0089] Specifically, the real-time posture estimation calculation formula of the vehicle is:

[0090]

[0091] Where π is a function defined by the camera projection model, and p is used to transform the three-dimensional point p 3D，p Projected onto the two-dimensional image plane by the rotation and translation of the camera, p 2D，p is the two-dimensional coordinate of the pth feature point in the image, p 3D，p is the coordinate of the three-dimensional space point corresponding to the world coordinate system, K is the intrinsic parameter matrix of the camera, which includes the focal length and principal point coordinates of the camera, R is the rotation matrix of the camera relative to the world coordinate system, and t is the translation vector of the camera;

[0092] The camera projection model π is expressed as

[0093]

[0094] Where [R|t] represents a homogeneous coordinate transformation matrix, which is used to transform a 3D point from the world coordinate system to the camera coordinate system.

[0095] S9: Use the real-time pose estimation obtained in step S8 to update the initial pose estimation in step S3, repeat steps S4-S8, and obtain the current pose of the vehicle in real time as the vehicle positioning information output.

[0096] Embodiment 2:

[0097] On the basis of the first embodiment, the second embodiment provides a specific operation method of step S2 in the first embodiment, as shown in the attached Figure 2 As shown, according to the standard data collected in step S1, the specific operation of constructing an offline three-dimensional map including a key frame image database includes the following steps:

[0098] S201: performing spatiotemporal alignment on the pose data, three-dimensional point cloud data and image data collected in step S1;

[0099] More specifically, step S2011: using the timestamp of the GPS / IMU integrated navigation system as a reference, aligning the timestamps of the laser radar and the camera through a timestamp difference algorithm to achieve time synchronization;

[0100] Since the clock sources all have clock drift, and each clock source has a different clock drift, even if the timestamps of each sensor are aligned at the initial moment, after running for a period of time, the previously aligned results will deviate. The solution to this problem is to unify the clock source in hardware. A common practice is to make a pulse generator, and all sensors are triggered by this pulse. Each time it is triggered, its own clock is corrected once, so that the cumulative error of the clock source can be eliminated. In the sensor configuration of autonomous driving, the GPS / IMU combined navigation system is an essential sensor. It comes with a second pulse generator, so it can be used directly. Moreover, when the GPS / IMU combined navigation system signal can meet the positioning requirements, its own clock will also be corrected by the atomic clock on the satellite, thereby further improving the accuracy. In the present invention, the GPS / IMU combined navigation system, laser radar and camera are integrated to perform vehicle positioning. In terms of time synchronization, the timestamp of the GPS / IMU combined navigation system is used as the reference, and the timestamps of the laser radar and the camera are aligned by the timestamp interpolation algorithm to ensure that the data between the three are synchronized in time. It should be noted that the timestamp difference algorithm belongs to the prior art and will not be repeated here.

[0101] Step S2012: The external parameters of the laser radar to GPS / IMU combined navigation system are obtained through calibration. Based on the external parameters of the laser radar to GPS / IMU combined navigation system, each frame of three-dimensional point cloud data obtained by the laser radar is converted from a local coordinate system with the laser radar as the origin to a global coordinate system to achieve spatial alignment of the pose data and the three-dimensional point cloud data. At the same time, the external parameters of the laser radar to the camera are obtained through calibration. Based on the external parameters of the laser radar to the camera, the spatial alignment of the three-dimensional point cloud data and the image data is achieved.

[0102] The spatial alignment method of the pose data and the 3D point cloud data is as follows: Assuming that there is a laser radar coordinate system P lidar The points in need to be converted to the combined navigation coordinate system P nav In the above equation, the conversion relationship is expressed as: nav=T lidar-to-nav P lidar , where T lidar-to-nav It represents the transformation matrix from the laser radar coordinate system to the integrated navigation coordinate system, that is, the external parameters from the laser radar to the GPS / IMU integrated navigation system. The transformation matrix contains the rotation matrix and the translation vector.

[0103] The spatial alignment method of 3D point cloud data and image data is as follows: Assuming there is a laser radar coordinate system P lidar The points in need to be transformed to the camera coordinate system P camera In the above equation, the conversion relationship is expressed as P camera =T lidar-to-camera P lidar , where T lidar-to-camera Represents the transformation matrix from the laser radar coordinate system to the camera coordinate system, that is, the external parameters from the laser radar to the camera. The transformation matrix also contains the rotation matrix and translation vector.

[0104] It should be noted that after achieving spatial alignment between pose data and three-dimensional point cloud data, and three-dimensional point cloud data and image data, spatial alignment between pose data and image data can also be achieved based on the spatial alignment relationship between them, thereby achieving spatial alignment of the three types of data.

[0105] Further, step S202: the aligned three-dimensional point cloud data is processed by a laser radar SLAM algorithm to obtain the global pose of the laser radar, and the global pose of the camera in the world reference system is estimated according to the global pose of the laser radar and the external parameters between the laser radar and the camera;

[0106] Further, step S203: selecting key frame images from the aligned image data to construct an offline key frame image database;

[0107] In this step, the uniqueness of the image content is analyzed, and images containing significant visual information are selected as key frames. Significant visual information refers to edges with drastic brightness changes in the image, corners where two or more edges intersect, areas with rich or complex textures, areas with high color contrast, areas with large brightness differences, patterns with specific shapes or regular arrangements, objects with large scale differences, etc. When selecting key frames, analyzing the uniqueness of the image content and selecting images containing significant visual information can enhance the accuracy of subsequent image retrieval.

[0108] Further, step S204: extracting the global feature vector of each key frame image in the key frame image database through the image retrieval model, and storing it in the offline key frame image database;

[0109] In this step, the NetVLAD graphic retrieval model is used to extract the global feature vector of each key frame image. The method for extracting the global feature vector is the same as the process of extracting the global feature vector in the online current frame environment image data using the NetVLAD graphic retrieval model in step S4 of Example 1.

[0110] Further, step S205: extracting local feature points and descriptors of each key frame image in the key frame image database through a neural network model, and storing them in the offline key frame image database.

[0111] In this step, the Superpoint deep network model is used to extract local feature points and descriptors of each key frame image, and the specific extraction process is the same as step S6 of the first embodiment.

[0112] Further, step S206: construct an offline three-dimensional map according to the global pose of the camera in the world coordinate system, and the global feature features, local feature points and descriptors of each key frame image.

[0113] In this step, 3D reconstruction software (COLMAP with Structure from Motion, COLMAP-SFM) is used to combine the global and local image features extracted by NetVLAD and Superpoint, the camera global pose information generated by LiDAR SLAM, and the closed loop detection and graph optimization technology in SLAM for 3D reconstruction. Finally, an offline high-precision real-scene 3D map containing visual and geometric information is generated. Figure 1 On the one hand, it provides a high-precision three-dimensional map model of the vehicle, which helps to improve the positioning accuracy in the online stage; on the other hand, through offline processing, the structure of the vehicle's driving environment can be captured and analyzed in advance, which helps to quickly retrieve and match features in the online stage, thereby providing stable and reliable positioning information when the vehicle moves at high speed or turns quickly.

[0114] Embodiment three:

[0115] Embodiment 3 provides a multi-sensor fusion vehicle rapid positioning device, as shown in the attached Figure 3 The vehicle rapid positioning device comprises an offline data acquisition device, an offline map construction device, an initial posture estimation device, an online data acquisition device, a global feature extraction device, an image matching device, a local feature extraction device, a target image screening device, a real-time posture estimation device and a posture output device;

[0116] The offline data acquisition device is used to collect standard data of the vehicle during its driving on a preset planned path through multiple sensors;

[0117] The offline map construction device is used to construct an offline three-dimensional map and an offline key frame image database; the offline map construction device includes a spatiotemporal alignment unit, a camera pose estimation unit, an image database construction unit, a global feature extraction unit, a local feature extraction unit and an offline map construction unit; the spatiotemporal alignment unit includes a time alignment module and a space alignment module;

[0118] The initial posture estimation device is used to collect the real-time posture data of the vehicle and set it as the initial posture estimation;

[0119] The online data acquisition device is used to collect online current frame environmental image data around the vehicle in real time;

[0120] The global feature extraction device is used to extract the global feature vector in the online current frame environment image data;

[0121] The image matching device is used to match K images similar to the online current frame environment image from the offline key frame image database;

[0122] The local feature extraction device is used to extract local feature points and descriptors of the K matched images;

[0123] The target image screening device is used to screen out one target image from K images;

[0124] The real-time pose estimation device is used to calculate the real-time pose estimation of the vehicle based on the key feature points of the target image and the three-dimensional space point coordinates corresponding to the key feature points;

[0125] The posture output device is used to update the real-time posture estimation and output the current posture of the vehicle after marking it on the offline three-dimensional map.

[0126] It should be noted that the offline data acquisition device, offline map construction device, initial pose estimation device, online data acquisition device, global feature extraction device, image matching device, local feature extraction device, target image screening device, real-time pose estimation device and pose output device in this embodiment are all implemented based on the vehicle rapid positioning method in Example 1 and Example 2.

[0127] Embodiment 4:

[0128] Embodiment 4 provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the vehicle rapid positioning method described in Embodiment 1 or Embodiment 2.

[0129] The method in Embodiment 1 or Embodiment 2 may be implemented in whole or in part by software, hardware, firmware or any combination thereof. The readable storage medium in the present invention may include computer storage media and communication media, and may also include any medium that can transfer a computer program from one place to another. The storage medium may be any target medium that can be accessed by a computer.

[0130] As a possible design, the readable storage medium in the present invention may include a compact disc read only memory (CD ROM), RAM, ROM, EEPROM or other optical disc storage; the computer readable medium may include a magnetic disk storage or other magnetic disk storage device. Moreover, any connecting line may also be appropriately referred to as a computer readable medium. For example, if the software is transmitted from a website, server or other remote source using a coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology (such as infrared, radio and microwave), the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology such as infrared, radio and microwave are included in the definition of the medium. Disks and optical discs as used herein include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks and Blu-ray discs, where disks usually reproduce data magnetically, while optical discs reproduce data optically using lasers.

[0131] Embodiment five:

[0132] Embodiment 5 provides an electronic device, which includes at least one processor and a memory communicatively connected to the processor; wherein the memory stores instructions executable by the processor, and the instructions are executed by the processor so that the processor can execute the vehicle rapid positioning method in embodiment 1 or embodiment 2.

[0133] In the present invention, the memory may be an electronic memory such as a flash memory, an EEPROM (Electrically Erasable Programmable Read-Only Memory), an EPROM, a hard disk or a ROM. The memory has a storage space for storing a computer program for executing the vehicle rapid positioning method. The computer program may be read from or written into one or more computer program products. These computer program products include program code carriers such as a hard disk, a compact disk (CD), a memory card or a floppy disk. The processor may be a single-core (single CPU) processor or a multi-core (multi CPU) processor.

[0134] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A multi-sensor fusion vehicle rapid positioning method, characterized in that: The following steps are involved: Step 1: Collect standard data of the vehicle driving on the preset planned path through multiple sensors to build an offline three-dimensional map containing a key frame image database; Step 2: When positioning the vehicle, the vehicle travels along the preset planned path, collects the real-time posture data of the vehicle as the initial posture estimation, collects the online current frame environment image data around the vehicle, and extracts the global feature vector in the online current frame environment image data; Step 3: According to the global feature vector extracted in step 2, K images similar to the online current frame environment image are matched in the offline key frame image database established in step 1, and the target image is selected from the K images; Step 4: Calculate the real-time pose estimation of the vehicle based on the target image; Step 5: Use the real-time pose estimation obtained in step 5 to update the initial pose estimation in step 2, repeat steps 3 to 5, and obtain the current pose of the vehicle in real time as the vehicle positioning information output.

2. The multi-sensor fusion vehicle rapid positioning method according to claim 1, characterized in that: The specific operations of step one include the following steps: S1: Collect standard data of the vehicle during driving on a preset planned path through multiple sensors; the multiple sensors include a GPS / IMU combined navigation system, a laser radar and a camera; S2: Based on the standard data collected in step S1, an offline three-dimensional map including a key frame image database is constructed.

3. The multi-sensor fusion vehicle rapid positioning method according to claim 2 is characterized in that: The specific operation of step S2 includes the following steps: S201: performing spatiotemporal alignment on the pose data, three-dimensional point cloud data and image data collected in step S1; S202: Processing the aligned three-dimensional point cloud data to obtain the global pose of the laser radar, and estimating the global pose of the camera in the world reference system; S203: Select key frame images from the aligned image data to build an offline key frame image database; S204: extracting a global feature vector of each key frame image in the key frame image database, and storing the global feature vector in the offline key frame image database; S205: extracting local feature points and descriptors of each key frame image in the key frame image database, and storing them in the offline key frame image database. S206: constructing an offline three-dimensional map according to the global pose of the camera in the world coordinate system, and the global feature features, local feature points and descriptors of each key frame image.

4. The multi-sensor fusion vehicle rapid positioning method according to claim 3 is characterized in that: The specific operation of step S201 includes the following steps: S2011: Based on the timestamp of the GPS / IMU integrated navigation system, the timestamps of the lidar and camera are aligned through the timestamp difference algorithm to achieve time synchronization; S2012: Obtain external parameters from the laser radar to the GPS / IMU integrated navigation system through calibration, and spatially align the pose data with the 3D point cloud data; spatially align the 3D point cloud data with the image data through the external parameters from the laser radar to the camera; Among them, the conversion relationship between the pose data and the three-dimensional point cloud data for spatial alignment is P nav =T lidar-to-nav P lidar , the conversion relationship between the spatial alignment of 3D point cloud data and image data is P camera =T lidar-to-camera P lidar ; Where P lidar is a point in the laser radar coordinate system, P nav is a point in the combined navigation coordinate system; T lidar-to-nav represents the transformation matrix from the laser radar coordinate system to the integrated navigation coordinate system; P camera is a point in the camera coordinate system; T lidar-to-camera Represents the transformation matrix from the lidar coordinate system to the camera coordinate system.

5. The multi-sensor fusion vehicle rapid positioning method according to claim 2, characterized in that: The specific operations of step 2 include the following steps: S3: When positioning the vehicle, the vehicle travels along the preset planned path, and the real-time posture data of the vehicle collected by the GPS / IMU integrated navigation system is used as the initial posture estimation; S4: The online current frame environment image data around the vehicle is collected in real time through the camera, and the global feature vector in the online current frame environment image data is extracted through the NetVLAD graphic retrieval model.

6. A multi-sensor fusion vehicle rapid positioning method according to claim 5, characterized in that: The specific operations of step three include the following steps: S5: According to the global feature vector extracted in step S4, K images similar to the online current frame environment image are matched in the offline key frame image database established in step S2; S6: Extract local feature points and descriptors from the K images matched in step S5 based on the Superpoint deep network model; S7: Based on the global feature vector proposed in step S4 and the local feature points and descriptors extracted from the K pictures in step S5, the most similar picture is selected from the K pictures as the target picture.

7. The multi-sensor fusion vehicle rapid positioning method according to claim 6, characterized in that: The specific operation of step S7 includes the following steps: S701: Calculate the global feature vector distances of K images respectively; d i =||F current -F i ||2 In the formula, F current is the global feature vector of the current frame environment image online, F i is the global feature vector extracted from the i-th (i=1, 2, ..., K)th picture, d i is the global feature vector distance between the i-th image and the online current frame environment image, and ||·||2 represents the Euclidean distance; S702: Calculate the descriptor distances of local feature points of K images; Where D current is the local feature point descriptor set of the online current frame environment image, D i is the local feature point descriptor set of the i-th (i=1, 2, ..., K) picture, d′ ij is the distance between the local feature point descriptor of the i-th image and the online current frame environment image, Represents the jth local descriptor in the local feature point descriptor set of the current frame environment image, D ij represents the ,th local descriptor in the local feature point descriptor set of the ith image, and HD represents the Hamming distance; S703: weighting the distance results calculated in step S701 and step S702, Where, d″ i represents the weighted distance, α represents the weight of the global feature vector distance, N i Indicates the number of local feature point descriptors of the i-th image; S704: nearest neighbor search, select target image; After calculating the weighted distance of K images, the image with the smallest weighted distance is selected as the nearest neighbor, i * =argmin i d″ i , where i * Represents the index of the nearest neighbor image.

8. A multi-sensor fusion vehicle rapid positioning device, characterized in that: The vehicle rapid positioning device includes an offline data acquisition device, an offline map construction device, an initial posture estimation device, an online data acquisition device, a global feature extraction device, an image matching device, a local feature extraction device, a target image screening device, a real-time posture estimation device and a posture output device; The offline data acquisition device, offline map construction device, initial posture estimation device, online data acquisition device, global feature extraction device, image matching device, local feature extraction device, target image screening device, real-time posture estimation device and posture output device are all implemented based on the vehicle rapid positioning method described in claim 6.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable the computer to execute the vehicle rapid positioning method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: The electronic device includes at least one processor and a memory communicatively connected to the processor; wherein the memory stores instructions executable by the processor, and the instructions are executed by the processor so that the processor can execute the vehicle rapid positioning method described in any one of claims 1-7.

Images