Cloud 3D map construction method, system and device

Abstract

The present invention provides a cloud-based three-dimensional map construction method and system, device and computer-readable storage medium, wherein the cloud-based three-dimensional map construction method includes: using different communication devices to respectively collect multiple groups of map data; uploading the collected multiple groups of map data to the cloud; in the cloud, the multiple groups of map data are processed in parallel according to one unit per group by visual inertial mileage calculation method, and the processing results of each group of map data are obtained respectively; the processing results are optimized and the optimized results are obtained; each group of optimized results is subjected to map fusion to construct a three-dimensional map. The technical solution provided by the present invention can support multi-device collection, with high efficiency and high speed, and the algorithm is carried out in the cloud, is not restricted by the computing performance of the platform, can be processed in parallel, and takes into account both efficiency and accuracy.

Description

Technical Field

[0001] The present invention relates to the field of communications, and in particular to a cloud-based three-dimensional map construction method and system, equipment, and computer-readable storage medium. Background Art

[0002] Three-dimensional visual map is one of the key technologies in the fields of augmented reality, robotics, and unmanned driving, and is the core of technologies such as automatic identification of the surrounding environment and visual positioning. With the popularization of various sensors, the enhancement of processing capabilities, and the promotion of various application needs, more and more image data is collected every day, and three-dimensional maps are generated and updated at the same time.

[0003] At present, the accuracy and efficiency of three-dimensional maps in the fields of unmanned driving and augmented reality are increasingly demanding. On the one hand, efficient collection equipment and methods are required, and on the other hand, high-precision computing centers are required to support large-scale data processing. At present, the construction of three-dimensional maps mostly relies on high-precision lidar or high-precision RTK, which is a challenge for the accuracy of mapping for consumer-grade cheap products such as mobile phones, sports cameras, VR devices, etc. At present, most of the mapping systems on the market are independent collection and mapping by a single device, which is inefficient. The existing technology relies on the singleness of external devices and cannot support the joint mapping of data collected by multiple devices such as mobile phones, robots, AR / VR devices, sports cameras, etc. When there are duplicate maps collected by multiple devices, storage and computing resources are wasted, and the fusion mapping of multiple sensors of multiple devices is also a technical problem. The update and fusion of maps in multiple time and space and multiple periods is a complex problem. At the same time, more efficient collection methods are needed when collecting data over a large area. The storage of massive data and the processing capacity of the mapping center are also huge challenges. Due to the massive data and large amount of computing required for large-scale mapping, localized mapping is greatly restricted, and the scale, speed and accuracy of mapping have bottlenecks.

[0004] Therefore, how to meet the requirements of high precision and high efficiency of three-dimensional maps while reducing costs and thresholds is an urgent problem to be solved in the industry. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a cloud-based three-dimensional map construction method and system, equipment and computer-readable storage medium, which supports multi-device collection, has high efficiency and high speed, and the algorithm is performed in the cloud, is not restricted by the platform's computing performance, can be processed in parallel, and takes into account both efficiency and accuracy.

[0006] In one aspect, the present invention provides a cloud-based three-dimensional map construction method, wherein the cloud-based three-dimensional map construction method comprises:

[0007] Use different communication devices to collect multiple sets of map data respectively;

[0008] Uploading the collected multiple sets of map data to the cloud;

[0009] In the cloud, the plurality of groups of map data are processed in parallel according to one unit per group by using a visual inertial odometry method, and processing results of each group of map data are obtained respectively;

[0010] Optimizing the processing result and obtaining an optimized result;

[0011] Each set of optimization results is fused into a map to construct a three-dimensional map.

[0012] Preferably, the step of processing the plurality of groups of map data in parallel as one unit per group by using the visual inertial odometry method comprises:

[0013] Inputting serialized images in each set of map data, a timestamp of each image in the serialized images, and synchronously collecting inertial measurement unit information;

[0014] The input information and the synchronously collected information are calculated through the visual inertial odometry method to obtain the six-degree-of-freedom position and posture of each image, as well as the timestamp, speed and zero bias of the corresponding inertial measurement unit as the processing result for each set of map data.

[0015] Preferably, the step of optimizing the processing result includes:

[0016] Extract feature points and descriptors of each image after parallel processing;

[0017] Perform image matching according to the feature points and the descriptors;

[0018] After matching, the images are screened for key frames;

[0019] Perform closed-loop detection on the selected key frames;

[0020] Input the key frame, the three-dimensional point cloud coordinates, the timestamp, velocity and zero bias of the inertial measurement unit corresponding to each key frame for parallel calculation;

[0021] Output the data after parallel calculation as the optimization result.

[0022] Preferably, the cloud-based three-dimensional map construction method further includes:

[0023] After completing the map fusion, determine whether to update the existing map;

[0024] If it is updated, the map to be updated is merged with the existing map through the above steps;

[0025] If not updated, the completed three-dimensional map is downloaded.

[0026] On the other hand, the present invention further provides a cloud-based three-dimensional map construction system, wherein the cloud-based three-dimensional map construction system comprises:

[0027] A collection module, used to collect multiple sets of map data using different communication devices;

[0028] An uploading module, used for uploading the collected multiple sets of map data to the cloud;

[0029] A visual inertial odometry module, used to process the plurality of groups of map data in parallel in the cloud by a visual inertial odometry calculation method according to one unit per group, and obtain processing results of each group of map data respectively;

[0030] An optimization module, used to optimize the processing result and obtain an optimization result;

[0031] The fusion module is used to map fuse each set of optimization results to construct a three-dimensional map.

[0032] Preferably, the visual inertial odometer module is specifically used for:

[0033] Inputting serialized images in each set of map data, a timestamp of each image in the serialized images, and synchronously collecting inertial measurement unit information;

[0034] The input information and the synchronously collected information are calculated through the visual inertial odometry method to obtain the six-degree-of-freedom position and posture of each image, as well as the timestamp, speed and zero bias of the corresponding inertial measurement unit as the processing result for each set of map data.

[0035] Preferably, the optimization module is specifically used for:

[0036] Extract feature points and descriptors of each image after parallel processing;

[0037] Perform image matching according to the feature points and the descriptors;

[0038] After matching, the images are screened for key frames;

[0039] Perform closed-loop detection on the selected key frames;

[0040] Input the key frame, the three-dimensional point cloud coordinates, the timestamp, velocity and zero bias of the inertial measurement unit corresponding to each key frame for parallel calculation;

[0041] Output the data after parallel calculation as the optimization result.

[0042] Preferably, the cloud-based three-dimensional map construction system further includes:

[0043] An update module is used to determine whether to update the existing map after completing the map fusion. If it is updated, the map to be updated is merged with the existing map through the above steps;

[0044] The download module is used to download the constructed three-dimensional map if it is not updated.

[0045] On the other hand, the present invention also provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the cloud-based three-dimensional map construction method described in any one of the above items are implemented.

[0046] On the other hand, the present invention also provides a device, wherein the device includes a memory and a processor, the memory stores computer processing instructions, and the processor executes any of the above-mentioned cloud-based three-dimensional map construction methods by calling the computer processing instructions.

[0047] The technical solution provided by the present invention has the following advantages: it can support multi-device collection, with high efficiency and high speed, and the algorithm is carried out in the cloud, is not restricted by the computing performance of the platform, can be processed in parallel, and takes into account both efficiency and accuracy; it provides a multi-device, multi-time and multi-space, multi-sensor fusion cloud map construction and update method, and this algorithm utilizes the timestamp, speed, zero bias and other information of the inertial measurement unit during map optimization, fusion and update, thereby maximizing the map construction accuracy; through the multi-device collection solution, efficient and fast collection can be achieved, and the diversity of device support can make collection more flexible; by utilizing the parallel processing and subsequent fusion of multiple devices in the cloud, the map construction and map update speeds are fast. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A schematic diagram of a process of constructing a three-dimensional map on the cloud in one embodiment of the present invention;

[0049] Figure 2 A schematic diagram of key frame screening in one embodiment of the present invention;

[0050] Figure 3 Schematic diagram of the structure of a cloud-based three-dimensional map construction system in one embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0052] The following is a detailed description of a cloud-based three-dimensional map construction method provided by the present invention.

[0053] See also Figure 1 , which is a flow chart of a method for constructing a three-dimensional cloud map in one embodiment of the present invention.

[0054] In step S11, multiple groups of map data are collected respectively using different communication devices;

[0055] In this embodiment, different communication devices are used to collect map data separately or simultaneously, wherein the communication devices belong to various types of communication devices on the user side, such as mobile phones, robots, augmented reality (AR) devices, virtual reality (VR) devices, sports cameras, etc. Such communication devices can collect map data, including: image data or video data collected by cameras, or inertial measurement unit (IMU) data collected by accelerometers and gyroscopes. If the communication device has a GPS function, the GPS data of the communication device is also collected. Of course, the image data or video data collected by the camera and the IMU data collected by the accelerometer and gyroscope are required, and GPS data is optional. In this embodiment, the map data collected by each communication device is counted as a group, and M groups of map data are collected using different communication devices, where M is an integer greater than 1.

[0056] In step S12, the collected multiple groups of map data are uploaded to the cloud.

[0057] In this embodiment, the cloud is a device belonging to the network side, such as a server. The servers in the cloud communicate with different communication devices respectively. Different communication devices upload multiple sets of map data collected by each of them to the servers in the cloud for storage, and then annotate the updated map ID number. If there is no remark, the default is to create a new map instead of updating the map.

[0058] In step S13, in the cloud, multiple groups of map data are processed in parallel as one unit per group by visual inertial odometry, and processing results of each group of map data are obtained respectively.

[0059] In this embodiment, the step of processing the plurality of groups of map data in parallel according to one unit per group by using the visual inertial odometry method comprises:

[0060] Inputting serialized images in each set of map data, a timestamp of each image in the serialized images, and synchronously collecting inertial measurement unit information;

[0061] The input information and the synchronously collected information are calculated through the visual inertial odometry method to obtain the six-degree-of-freedom position and posture of each image, as well as the timestamp, speed and zero bias of the corresponding inertial measurement unit as the processing result for each set of map data.

[0062] In this embodiment, after receiving the uploaded map data in the cloud server, the Visual Inertial Odometry (VIO) module is started to read the data in the server, and multiple groups of map data are processed in parallel according to one unit per group through the VIO algorithm, and each group of map data obtains a set of processing results. In this embodiment, the input of each group of visual inertial odometry modules includes: serialized images (>=10Hz), the timestamp of each image in the serialized images, and the synchronously collected inertial measurement unit (IMU) information (>=100Hz, each frame of IMU information includes accelerometer data and timestamp, gyroscope data and timestamp), and the input information and the synchronously collected information are calculated by the visual inertial odometry calculation method, and the processing results of each group of map data include: the timestamp of each image, the six-degree-of-freedom pose (i.e., position and attitude) of each image, and the corresponding IMU timestamp, speed and zero bias.

[0063] In this embodiment, the VIO algorithm here can be a monocular or multi-eye VIO algorithm, including but not limited to various VIO algorithms based on optimization or filtering such as EKF, MSCKF, VISLAM, etc. In this embodiment, there are many methods and frameworks involved in the VIO algorithm here, and there are many open source frameworks, such as VIORB, ROVIO, VINS-mono, VINS-fusion, S-MSCKF, RVIO, openvins, etc. In this embodiment, how to integrate vision and IMU? There will be some differences for different frameworks. The main body is divided into two types based on optimization and filtering. The optimization method is regarded as a maximum likelihood estimation (ML) formula, in which the state with the highest total probability of measurement is iteratively found; the filtering method is regarded as a maximum a posteriori estimation (MAP) formula, in which the prior distribution of the platform posture is composed of the measurement values ​​of the proprioceptive sensor, and the likelihood distribution is composed of the measurement values ​​of the external sensor. In this embodiment, for nonlinear dynamic models and / or nonlinear measurement models, the iterative EKF is equivalent to an optimization-based method that iteratively updates each step in the EKF. The optimization method can reformulate the ML problem as a MAP problem by adding regularization terms or prior terms from internal sensors or other source measurements. These are online or causal algorithms, where the current estimate depends on the current and previous measurements. Offline or non-stochastic algorithms are batch processes, where the current estimate depends on the complete data set, or it can be said that they depend not only on the current and previous measurements, but also on future measurements. In this embodiment, in the optimization method, the batch process is decomposed into solving a set of linear algebraic equations. In the filtering method, the Kalman smoothing algorithm can obtain the backward Gaussian distribution through both forward and backward methods.

[0064] In this embodiment, the parallel processing process includes: for the M groups of map data uploaded, each group of map data includes serialized images (>=10Hz), the timestamp of each image in the serialized images, and the synchronously collected IMU information (>=100Hz, each frame of IMU information includes accelerometer data and timestamp, gyroscope data and timestamp). The processing of each group of data is regarded as a process, which is placed in a certain server to run the VIO algorithm. The main server can start the process by detecting idle machines, randomly sending each group of map data to idle machines to run the VIO algorithm, and the processing results are returned to the main server. In this embodiment, the visual inertial odometer calculation method can make better use of the sequence information of the video than other methods (such as visual odometer VO / VSLAM method, motion recovery structure SFM (Structure from motion) method, etc.), ensure that the trajectory and map accuracy of a single group are relatively high, and can directly restore the true scale of the scene.

[0065] In step S14, the processing result is optimized and an optimized result is obtained;

[0066] In this embodiment, the step of optimizing the processing result includes:

[0067] Extract feature points and descriptors of each image after parallel processing;

[0068] Perform image matching according to the feature points and the descriptors;

[0069] After matching, the images are screened for key frames;

[0070] Perform closed-loop detection on the selected key frames;

[0071] Input the key frame, the three-dimensional point cloud coordinates, the timestamp, velocity and zero bias of the inertial measurement unit corresponding to each key frame for parallel calculation;

[0072] Output the data after parallel calculation as the optimization result.

[0073] In this embodiment, the processing results of the M groups of map data in step S13 are processed in parallel according to one unit per group to obtain M groups of outputs, wherein each group of inputs includes: each frame image of the group, the timestamp of the image, and the six-degree-of-freedom pose (i.e., position and posture) of each image, and the IMU data corresponding to each image, i.e., the timestamp, speed, and bias of the IMU. The output includes: the keyframe after joint IMU and visual optimization, the three-dimensional point cloud coordinates, and the IMU data corresponding to each keyframe (i.e., the timestamp, speed, and bias of the IMU), wherein the keyframe includes the image pose, 2D point features and descriptors, and the link relationship between the 2D features and the 3D point cloud.

[0074] In this embodiment, the processing results are optimized, and each unit performs the following operations:

[0075] (1) Feature extraction: extract feature points and descriptors from each image, including but not limited to features such as fast, Dog, Harris, orb, Breaf, or deep learning features such as superpoint and loft. Feature points and descriptors are mainly used for subsequent image or feature matching related calculations;

[0076] (2) Image matching: performing image matching based on the feature points and descriptors in step (1), including but not limited to matching based on time series matching, brute force matching, bag of words topN retrieval, deep learning retrieval topN or spatial position matching and then triangulating, wherein image matching refers to matching the feature points of other images based on the features and descriptors of each image. If the difference in the descriptors of the feature points is within a certain threshold, they are considered similar, and the number of matching feature points between the images is obtained. If the number of matching feature points of the two images reaches the threshold, the two images are considered to be successfully matched; wherein, time series matching refers to matching images in the order of their timestamps, and matching each image with other images that are within ta seconds of the current image in time; wherein, brute force matching refers to matching each image with all other images in pairs; wherein, bag of words topN retrieval refers to: BOW (bag of words) model is a document representation method in the field of information retrieval. Similar images are detected through the bag of words model. TopN is to extract the top N images with the highest similarity to the target image, and then match these N images with the target image in pairs. Among them, deep learning retrieval topN means: retrieve images through deep learning algorithms (such as NetVlad, MobileNet, HF-Net, etc.) to obtain the top N images with the highest similarity to the target image, and then match these N images with the target image in pairs. Among them, spatial position matching means: each image and other images that reach a threshold in space with this image (such as position distance A meters, angle difference B degrees) are matched in pairs. Among them, triangulation means: given the matching relationship of the feature points of two images, the coordinate points in three-dimensional space are obtained through stereo geometric triangulation. Here, the matching triangulation is mainly to obtain the spatial 3D point coordinates corresponding to the 2D feature points of the image.

[0077] (3) Key frame screening, such as Figure 2The solid squares are the camera positions and orientations of the filtered key frames, and the dotted squares are the camera positions and orientations of the filtered non-key frames. A portion of the images are filtered to be key frames (keyframes). The key frames include image poses, 2D point features and descriptors, the link relationship between 2D features and 3D point clouds, etc. The filtering principles include but are not limited to uniform sampling according to the number of common viewpoints between frames, uniform sampling of the proportion of common viewpoints between frames, uniform sampling of spatial positions between frames, uniform sampling of time intervals between frames, etc.; wherein, uniform sampling of the number of common viewpoints between frames means: traversing each frame of the image, for each frame of the image, if the number of repetitions of the 3D point corresponding to the 2D feature of the current frame and the 3D point corresponding to the 2D feature of another frame of the image reaches Po (i.e., the empirical value), then the current frame is discarded and not made into a key frame, and the remaining ones are key frames; wherein, uniform sampling of the proportion of common viewpoints between frames means: traversing each frame of the image, for each frame of the image, For an image, the total number of 3D points corresponding to the 2D features of the current frame is Ps, and the number of repetitions of the 3D points of another frame is Pc (i.e., empirical value), where the ratio of Pc to the total number of 3D points corresponding to the 2D features of the current frame is Pc / Ps. If Pc / Ps is greater than Pth1 (i.e., empirical value), the current frame is discarded and not used as a key frame, and the remaining ones are key frames; wherein, uniform sampling of spatial positions between frames means: traversing each frame of the image, for each frame of the image, if the current frame reaches a threshold value with other frames in spatial position (the distance is less than Tb (i.e., empirical value) degrees, and the angle difference is less than Rb (i.e., empirical value) degrees), then it is not used as a key frame, and the remaining ones are key frames; wherein, uniform sampling of time intervals between frames means: traversing each frame of the image, for each frame of the image, if the current frame differs from other frames in time within tb (i.e., empirical value) seconds, then the current frame is discarded and not used as a key frame, and the remaining ones are key frames;

[0078] (4) Closed loop detection, including but not limited to bag-of-words retrieval and deep learning retrieval. After retrieving similar images, the images are matched with 2D features in pairs and then fused with 3D points to establish a common view association relationship. Among them, closed loop detection is also called loop closure detection, which means that a certain scene has been reached through image recognition, and the map generated at this moment or this image is matched with the map generated at other moments or other images. Among them, the closed loop detection method includes: traversing all keyframes, solving the closed loop candidate frames of each keyframe, for example, the keyframe KFa is solved by bag-of-words retrieval (such as BOW retrieval based on orb or sift) or deep learning retrieval (such as NetVlad, MobileNet, HF-Net, etc.), and the topN most similar to KFa among all keyframes are solved as candidate keyframes Candidates1, and then clustered by covisibility Clustering determines whether the candidate is reliable. The common view cluster determination method is: traverse each candidate key frame in the candidate frame Candidates1. If the topM (M=1, 2, 3...) common view frames with this candidate key frame are in the candidate key frame Candidates1 and are not the common view frames of KFa, then the candidate frame is reliable. If two images have common 3D points, they are common view frames. The topM common view frames refer to the frames with the top M common view points in the common view frames. The reliable candidate key frames selected are Cand idates2, each key frame in Candidates2 is matched with the current key frame KFa for image 2D feature points. If the number of matching points reaches S, it is credible. The credible candidate key frame is Candidates3. The 3D points in Candidates3 are assigned to the 2D points of the current frame KFa. Use ransac to solve pose. If the solution is successful, the closed loop detection is completed. Then merge the 3D points of the key frame Kfa and Candidates3, and update the correspondence between Kfa and the 3D points and 2D points of Candidates3.

[0079] (5) Global optimization: global BA is performed by combining keyframes, 3D points, IMU, and GPS (if any). The input is a series of keyframes (including image pose, 2D point features and descriptors, the link between 2D features and 3D point clouds, etc.), 3D point cloud coordinates, and IMU data corresponding to each keyframe, such as IMU timestamp, IMU speed, and IMU bias. The optimization results include keyframes (including image pose, 2D point features and descriptors, the link between 2D features and 3D point clouds, etc.), 3D point clouds, IMU data, IMU timestamp, IMU speed, and IMU bias.

[0080] State vector: where x R Indicates the state of the key frame at a certain moment, including: the position of the key frame in the IMU coordinate system The angle q of the key frame in the IMU coordinate system WS , the speed of the key frame in the IMU coordinate system Gyroscope bias b g and the accelerometer bias b a ;

[0081] Cost function J(x) of joint visual-inertial projection mapping: including the visual projection error e r and the IMU prediction error term e s , Where k is the id index of the key frame, i is the id index of the camera device, and j is the id index of the 3D landmark point. An information matrix representing the measurement values ​​of 3D landmark points, The information matrix representing the IMU error term;

[0082] Visual projection error Projection error: Among them, Z i，j，k represents the observation value of the image coordinate system, h i (·) represents the projection of the camera model, Represents the homogeneous coordinates of the landmark point, j represents the id index of the 3D landmark point, Indicates the conversion relationship from IMU to camera device with id i. Indicates the pose (including position and angle) of the key frame with id k in the IMU coordinate system

[0083] The prediction error term of IMU: in, The sum of the measurement errors of the synchronized IMU between key frames k and k+1 is represented by Indicates the predicted position of the key frame with ID k+1 in the world coordinate system, Indicates the measured position of the key frame with ID k+1 in the world coordinate system. Indicates the predicted angle of the key frame with ID k+1 in the world coordinate system, It represents the measured angle of the key frame with ID k+1 in the world coordinate system. represents the angular difference between the prediction and the measurement, x sb State quantity representing speed and bias

[0084] In step S15, each set of optimization results is subjected to map fusion to construct a three-dimensional map.

[0085] In this embodiment, the cloud-based three-dimensional map construction method further includes:

[0086] After completing the map fusion, determine whether to update the existing map;

[0087] If it is updated, the map to be updated is merged with the existing map through the above steps;

[0088] If not updated, the completed three-dimensional map is downloaded.

[0089] In this embodiment, the input is M groups of results obtained by the global optimization module, each group of results includes a series of key frames (including image pose, 2D point features and descriptors, the link relationship between 2D features and 3D point clouds, etc.), 3D point clouds, IMU data, IMU timestamps, IMU speeds and IMU biases, and the output is a fused set of results, including a series of key frames (including image pose, 2D point features and descriptors, the link relationship between 2D features and 3D point clouds, etc.), 3D point clouds, IMU data, IMU timestamps, IMU speeds and IMU biases.

[0090] The specific method is to put the M groups of results obtained by the global optimization module together for closed-loop detection, which is the same as the closed-loop detection method in step S14 of the global optimization module, and then combine the M groups of key frames (including image pose, 2D point features and descriptors, the link between 2D features and 3D point clouds, etc.), 3D point clouds, IMU, GPS (if any) for global optimization, which is the same as the global optimization method in step S14 of the global optimization module.

[0091] See also Figure 3 , which is a structural diagram of a cloud-based three-dimensional map construction system in one embodiment of the present invention.

[0092] In this embodiment, the cloud-based three-dimensional map construction system 10 includes: a collection module 11, an upload module 12, a visual inertial odometer module 13, an optimization module 14 and a fusion module 15.

[0093] The acquisition module 11 is used to respectively acquire multiple groups of map data using different communication devices.

[0094] In this embodiment, different communication devices are used to collect map data separately or simultaneously, wherein the communication devices belong to various types of communication devices on the user side, such as mobile phones, robots, AR devices, VR devices, sports cameras, etc. Such communication devices can collect map data, including: image data or video data collected by cameras, or IMU data collected by accelerometers and gyroscopes. If the communication device has a GPS function, the GPS data of the communication device is also collected. Of course, the image data or video data collected by the camera and the IMU data collected by the accelerometer and gyroscope are required, and GPS data is optional. In this embodiment, the map data collected by each communication device is counted as a group, and M groups of map data are collected using different communication devices, where M is an integer greater than 1.

[0095] The uploading module 12 is used to upload the collected multiple groups of map data to the cloud.

[0096] In this embodiment, the cloud is a device belonging to the network side, such as a server. The servers in the cloud communicate with different communication devices respectively. Different communication devices upload multiple sets of map data collected by each of them to the servers in the cloud for storage, and then annotate the updated map ID number. If there is no remark, the default is to create a new map instead of updating the map.

[0097] The visual inertial odometry module 13 is used to process the plurality of groups of map data in parallel in the cloud by a visual inertial odometry calculation method according to one unit per group, and obtain processing results of each group of map data respectively.

[0098] In this embodiment, the visual inertial odometer module 13 is specifically used for:

[0099] Inputting serialized images in each set of map data, a timestamp of each image in the serialized images, and synchronously collecting inertial measurement unit information;

[0100] The input information and the synchronously collected information are calculated through the visual inertial odometry method to obtain the six-degree-of-freedom position and posture of each image, as well as the timestamp, speed and zero bias of the corresponding inertial measurement unit as the processing result for each set of map data.

[0101] In this embodiment, after the server in the cloud receives the uploaded map data, the VIO module is started to read the data in the server, and multiple groups of map data are processed in parallel according to one unit per group through the VIO algorithm, and each group of map data obtains a set of processing results. In this embodiment, the input of each group of visual inertial odometer modules includes: serialized images (>=10Hz), timestamps of each image in the serialized images, and synchronously collected IMU information (>=100Hz, each frame of IMU information includes accelerometer data and timestamps, gyroscope data and timestamps), and the input information and synchronously collected information are calculated by the visual inertial odometer calculation method, and the processing results of each group of map data include: timestamps of each image, six degrees of freedom pose (i.e., position and posture) of each image, and corresponding timestamps, speeds, and biases of the IMU.

[0102] In this embodiment, the VIO algorithm here can be a monocular or multi-eye VIO algorithm, including but not limited to various VIO algorithms based on optimization or filtering such as EKF, MSCKF, VISLAM, etc. In this embodiment, there are many methods and frameworks involved in the VIO algorithm here, and there are many open source frameworks, such as VIORB, ROVIO, VINS-mono, VINS-fusion, S-MSCKF, RVIO, openvins, etc. In this embodiment, how to integrate vision and IMU? There will be some differences for different frameworks. The main body is divided into two types based on optimization and filtering. The optimization method is regarded as a maximum likelihood estimation (ML) formula, in which the state with the highest total probability of measurement is iteratively found; the filtering method is regarded as a maximum a posteriori estimation (MAP) formula, in which the prior distribution of the platform posture is composed of the measurement values ​​of the proprioceptive sensor, and the likelihood distribution is composed of the measurement values ​​of the external sensor. In this embodiment, for nonlinear dynamic models and / or nonlinear measurement models, the iterative EKF is equivalent to an optimization-based method that iteratively updates each step in the EKF. The optimization method can reformulate the ML problem as a MAP problem by adding regularization terms or prior terms from internal sensors or other source measurements. These are online or causal algorithms, where the current estimate depends on the current and previous measurements. Offline or non-stochastic algorithms are batch processes, where the current estimate depends on the complete data set, or it can be said that they depend not only on the current and previous measurements, but also on future measurements. In this embodiment, in the optimization method, the batch process is decomposed into solving a set of linear algebraic equations. In the filtering method, the Kalman smoothing algorithm can obtain the backward Gaussian distribution through both forward and backward methods.

[0103] In this embodiment, the parallel processing process includes: for the M groups of map data uploaded, each group of map data includes serialized images (>=10Hz), the timestamp of each image in the serialized images, and the synchronously collected IMU information (>=100Hz, each frame of IMU information includes accelerometer data and timestamp, gyroscope data and timestamp). The processing of each group of data is regarded as a process, which is placed in a certain server to run the VIO algorithm. The main server can start the process by detecting idle machines, randomly sending each group of map data to idle machines to run the VIO algorithm, and the processing results are returned to the main server. In this embodiment, the visual inertial odometer calculation method can make better use of the sequence information of the video than other methods (such as visual odometer VO / VSLAM method, motion recovery structure SFM (Structure from motion) method, etc.), ensure that the trajectory and map accuracy of a single group are relatively high, and can directly restore the true scale of the scene.

[0104] The optimization module 14 is used to optimize the processing result and obtain an optimized result.

[0105] In this embodiment, the optimization module 14 is specifically used for:

[0106] Extract feature points and descriptors of each image after parallel processing;

[0107] Perform image matching according to the feature points and the descriptors;

[0108] After matching, the images are screened for key frames;

[0109] Perform closed-loop detection on the selected key frames;

[0110] Input the key frame, the three-dimensional point cloud coordinates, the timestamp, velocity and zero bias of the inertial measurement unit corresponding to each key frame for parallel calculation;

[0111] Output the data after parallel calculation as the optimization result.

[0112] In this embodiment, the processing results of M groups of map data in the visual inertial odometer module 13 are processed in parallel according to one unit per group to obtain M groups of outputs, wherein each group of inputs includes: each frame image of the group, the timestamp of the image, and the six-degree-of-freedom pose (i.e., position and posture) of each image, and the IMU data corresponding to each image, i.e., the timestamp, speed, and bias of the IMU. The output includes: the keyframe after joint IMU and visual optimization, the three-dimensional point cloud coordinates, and the IMU data corresponding to each keyframe (i.e., the timestamp, speed, and bias of the IMU), wherein the keyframe includes the image pose, 2D point features and descriptors, and the link relationship between the 2D features and the 3D point cloud.

[0113] In this embodiment, the processing results are optimized, and each unit performs the following operations:

[0114] (1) Feature extraction: extract feature points and descriptors from each image, including but not limited to features such as fast, Dog, Harris, orb, Breaf, or deep learning features such as superpoint and loft. Feature points and descriptors are mainly used for subsequent image or feature matching related calculations;

[0115] (2) Image matching: performing image matching based on the feature points and descriptors in step (1), including but not limited to matching based on time series matching, brute force matching, bag of words topN retrieval, deep learning retrieval topN or spatial position matching and then triangulating, wherein image matching refers to matching the feature points of other images based on the features and descriptors of each image. If the difference in the descriptors of the feature points is within a certain threshold, they are considered similar, and the number of matching feature points between the images is obtained. If the number of matching feature points of the two images reaches the threshold, the two images are considered to be successfully matched; wherein, time series matching refers to matching images in the order of their timestamps, and matching each image with other images that are within ta seconds of the current image in time; wherein, brute force matching refers to matching each image with all other images in pairs; wherein, bag of words topN retrieval refers to: BOW (bag of words) model is a document representation method in the field of information retrieval. Similar images are detected through the bag of words model. TopN is to extract the top N images with the highest similarity to the target image, and then match these N images with the target image in pairs. Among them, deep learning retrieval topN means: retrieve images through deep learning algorithms (such as NetVlad, MobileNet, HF-Net, etc.) to obtain the top N images with the highest similarity to the target image, and then match these N images with the target image in pairs. Among them, spatial position matching means: each image and other images that reach a threshold in space with this image (such as position distance A meters, angle difference B degrees) are matched in pairs. Among them, triangulation means: given the matching relationship of the feature points of two images, the coordinate points in three-dimensional space are obtained through stereo geometric triangulation. Here, the matching triangulation is mainly to obtain the spatial 3D point coordinates corresponding to the 2D feature points of the image.

[0116] (3) Key frame screening, such as Figure 2The solid squares are the camera positions and orientations of the filtered key frames, and the dotted squares are the camera positions and orientations of the filtered non-key frames. A portion of the images are filtered to be key frames (keyframes). The key frames include image poses, 2D point features and descriptors, the link relationship between 2D features and 3D point clouds, etc. The filtering principles include but are not limited to uniform sampling according to the number of common viewpoints between frames, uniform sampling of the proportion of common viewpoints between frames, uniform sampling of spatial positions between frames, uniform sampling of time intervals between frames, etc.; wherein, uniform sampling of the number of common viewpoints between frames means: traversing each frame of the image, for each frame of the image, if the number of repetitions of the 3D point corresponding to the 2D feature of the current frame and the 3D point corresponding to the 2D feature of another frame of the image reaches Po (i.e., the empirical value), then the current frame is discarded and not made into a key frame, and the remaining ones are key frames; wherein, uniform sampling of the proportion of common viewpoints between frames means: traversing each frame of the image, for each frame of the image, For an image, the total number of 3D points corresponding to the 2D features of the current frame is Ps, and the number of repetitions of the 3D points of another frame is Pc (i.e., empirical value), where the ratio of Pc to the total number of 3D points corresponding to the 2D features of the current frame is Pc / Ps. If Pc / Ps is greater than Pth1 (i.e., empirical value), the current frame is discarded and not used as a key frame, and the remaining ones are key frames; wherein, uniform sampling of spatial positions between frames means: traversing each frame of the image, for each frame of the image, if the current frame reaches a threshold value with other frames in spatial position (the distance is less than Tb (i.e., empirical value) degrees, and the angle difference is less than Rb (i.e., empirical value) degrees), then it is not used as a key frame, and the remaining ones are key frames; wherein, uniform sampling of time intervals between frames means: traversing each frame of the image, for each frame of the image, if the current frame differs from other frames in time within tb (i.e., empirical value) seconds, then the current frame is discarded and not used as a key frame, and the remaining ones are key frames;

[0117] (4) Closed loop detection, including but not limited to bag-of-words retrieval and deep learning retrieval. After retrieving similar images, the images are matched with 2D features in pairs and then fused with 3D points to establish a common view association relationship. Among them, closed loop detection is also called loop closure detection, which means that a certain scene has been reached through image recognition, and the map generated at this moment or this image is matched with the map generated at other moments or other images. Among them, the closed loop detection method includes: traversing all keyframes, solving the closed loop candidate frames of each keyframe, for example, the keyframe KFa is solved by bag-of-words retrieval (such as BOW retrieval based on orb or sift) or deep learning retrieval (such as NetVlad, MobileNet, HF-Net, etc.), and the topN most similar to KFa among all keyframes are solved as candidate keyframes Candidates1, and then clustered by covisibility Clustering determines whether the candidate is reliable. The common view cluster determination method is: traverse each candidate key frame in the candidate frame Candidates1. If the topM (M=1, 2, 3...) common view frames with this candidate key frame are in the candidate key frame Candidates1 and are not the common view frames of KFa, then the candidate frame is reliable. If two images have common 3D points, they are common view frames. The topM common view frames refer to the frames with the top M common view points in the common view frames. The reliable candidate key frames selected are Cand idates2, each key frame in Candidates2 is matched with the current key frame KFa for image 2D feature points. If the number of matching points reaches S, it is credible. The credible candidate key frame is Candidates3. The 3D points in Candidates3 are assigned to the 2D points of the current frame KFa. Use ransac to solve pose. If the solution is successful, the closed loop detection is completed. Then merge the 3D points of the key frame Kfa and Candidates3, and update the correspondence between Kfa and the 3D points and 2D points of Candidates3.

[0118] (5) Global optimization: global BA is performed by combining keyframes, 3D points, IMU, and GPS (if any). The input is a series of keyframes (including image pose, 2D point features and descriptors, the link between 2D features and 3D point clouds, etc.), 3D point cloud coordinates, and IMU data corresponding to each keyframe, such as IMU timestamp, IMU speed, and IMU bias. The optimization results include keyframes (including image pose, 2D point features and descriptors, the link between 2D features and 3D point clouds, etc.), 3D point clouds, IMU data, IMU timestamp, IMU speed, and IMU bias.

[0119] State vector: where x R Indicates the state of the key frame at a certain moment, including: the position of the key frame in the IMU coordinate system The angle q of the key frame in the IMU coordinate system WS , the speed of the key frame in the IMU coordinate system Gyroscope bias b g and the accelerometer bias b a ;

[0120] Cost function J(x) of joint visual-inertial projection mapping: including the visual projection error e r and the IMU prediction error term e s , Where k is the id index of the key frame, i is the id index of the camera device, and j is the id index of the 3D landmark point. An information matrix representing the measurement values ​​of 3D landmark points, The information matrix representing the IMU error term;

[0121] Visual projection error Projection error: Among them, Z i，j，k represents the observation value of the image coordinate system, h i (·) represents the projection of the camera model, Represents the homogeneous coordinates of the landmark point, j represents the id index of the 3D landmark point, Indicates the conversion relationship from IMU to camera device with id i. Indicates the pose (including position and angle) of the key frame with id k in the IMU coordinate system

[0122] The prediction error term of IMU: in, The sum of the measurement errors of the synchronized IMU between key frames k and k+1 is represented by Indicates the predicted position of the key frame with ID k+1 in the world coordinate system, Indicates the measured position of the key frame with ID k+1 in the world coordinate system. Indicates the predicted angle of the key frame with ID k+1 in the world coordinate system, It represents the measured angle of the key frame with ID k+1 in the world coordinate system. represents the angular difference between the prediction and the measurement, x sb State quantity representing speed and bias

[0123] The fusion module 15 is used to perform map fusion on each set of optimization results to construct a three-dimensional map.

[0124] In this embodiment, the input is M groups of results obtained by the optimization module 14, each group of results includes a series of key frames (including image pose, 2D point features and descriptors, the link relationship between 2D features and 3D point clouds, etc.), 3D point clouds, IMU data, IMU timestamps, IMU speed and IMU bias, and the output is a fused set of results, including a series of key frames (including image pose, 2D point features and descriptors, the link relationship between 2D features and 3D point clouds, etc.), 3D point clouds, IMU data, IMU timestamps, IMU speed and IMU bias.

[0125] The specific approach is to put the M groups of results obtained by the optimization module 14 together for closed-loop detection, which is the same as the closed-loop detection approach in the optimization module 14, and then combine the M groups of key frames (including image pose, 2D point features and descriptors, the link between 2D features and 3D point clouds, etc.), 3D point clouds, IMU, GPS (if any) for global optimization, which is the same as the global optimization approach in the optimization module 14.

[0126] In this embodiment, the cloud-based three-dimensional map construction system further includes:

[0127] An update module is used to determine whether to update the existing map after completing the map fusion. If it is updated, the map to be updated is merged with the existing map through the above steps;

[0128] The download module is used to download the constructed three-dimensional map if it is not updated.

[0129] On the other hand, the present invention also provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the cloud-based three-dimensional map construction method as described above are implemented.

[0130] On the other hand, the present invention also provides a device, wherein the device includes a memory and a processor, the memory stores computer processing instructions, and the processor executes the above-mentioned cloud-based three-dimensional map construction method by calling the computer processing instructions.

[0131] The technical solution provided by the present invention has the following advantages: it can support multi-device collection, with high efficiency and high speed, and the algorithm is carried out in the cloud, is not restricted by the computing performance of the platform, can be processed in parallel, and takes into account both efficiency and accuracy; it provides a multi-device, multi-time and multi-space, multi-sensor fusion cloud map construction and update method, and this algorithm utilizes the timestamp, speed, zero bias and other information of the inertial measurement unit during map optimization, fusion and update, thereby maximizing the map construction accuracy; through the multi-device collection solution, efficient and fast collection can be achieved, and the diversity of device support can make collection more flexible; by utilizing the parallel processing and subsequent fusion of multiple devices in the cloud, the map construction and map update speeds are fast.

[0132] It is worth noting that in the above embodiments, the various units included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0133] In addition, a person skilled in the art can understand that all or part of the steps in the above-mentioned embodiments can be completed by instructing the relevant hardware through a program, and the corresponding program can be stored in a computer-readable storage medium, such as a ROM / RAM, a disk or an optical disk.

[0134] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A cloud-based three-dimensional map construction method, characterized in that: The cloud-based three-dimensional map construction method comprises: Use different communication devices to collect multiple sets of map data respectively; Uploading the collected multiple sets of map data to the cloud; In the cloud, the multiple groups of map data are processed in parallel according to one unit per group by using the visual inertial odometry method, and processing results of each group of map data are obtained respectively; the step of processing the multiple groups of map data in parallel according to one unit per group by using the visual inertial odometry method comprises: inputting serialized images in each group of map data, timestamps of each image in the serialized images, and synchronously collecting inertial measurement unit information; calculating the input information and the synchronously collected information by using the visual inertial odometry method to obtain the six-degree-of-freedom position and posture of each image, as well as the timestamp, speed and zero bias of the corresponding inertial measurement unit, as processing results of each group of map data; Optimizing the processing result and obtaining an optimized result; the step of optimizing the processing result includes: Extract feature points and descriptors of each image after parallel processing; Perform image matching according to the feature points and the descriptors; After matching, the images are screened for key frames; Perform closed-loop detection on the selected key frames; Input the key frame, the three-dimensional point cloud coordinates, the timestamp, velocity and zero bias of the inertial measurement unit corresponding to each key frame for parallel calculation; The data after parallel calculation is output as the optimization result; each set of optimization results is fused into a map to construct a three-dimensional map.

2. The cloud-based three-dimensional map construction method according to claim 1, characterized in that: The cloud-based three-dimensional map construction method further includes: After completing the map fusion, determine whether to update the existing map; If updated, the map to be updated is merged with the existing map by optimizing the processing result; If not updated, the completed three-dimensional map is downloaded.

3. A cloud-based three-dimensional map construction system, characterized in that: The cloud-based three-dimensional map construction system includes: A collection module, used to collect multiple sets of map data using different communication devices; An uploading module, used for uploading the collected multiple sets of map data to the cloud; A visual inertial odometry module is used to process the multiple groups of map data in parallel according to one unit per group in the cloud by using a visual inertial odometry calculation method, and obtain processing results of each group of map data respectively; the visual inertial odometry module is specifically used to: input serialized images in each group of map data, timestamps of each image in the serialized images, and synchronously collect inertial measurement unit information; calculate the input information and the synchronously collected information by using a visual inertial odometry calculation method to obtain the six-degree-of-freedom position and posture of each image, as well as the timestamp, speed and zero bias of the corresponding inertial measurement unit, as processing results of each group of map data; The optimization module is used to optimize the processing result and obtain the optimization result; the optimization module is specifically used to: Extract feature points and descriptors of each image after parallel processing; Perform image matching according to the feature points and the descriptors; After matching, the images are screened for key frames; Perform closed-loop detection on the selected key frames; Input the key frame, the three-dimensional point cloud coordinates, the timestamp, velocity and zero bias of the inertial measurement unit corresponding to each key frame for parallel calculation; Output the data after parallel calculation as the optimization result; The fusion module is used to map fuse each set of optimization results to construct a three-dimensional map.

4. The cloud-based three-dimensional map construction system according to claim 3, characterized in that: The cloud-based three-dimensional map construction system also includes: An updating module, used for determining whether to update the existing map after completing the map fusion, and if so, fusing the map to be updated with the existing map by optimizing the processing result; The download module is used to download the constructed three-dimensional map if it is not updated.

5. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps of the cloud-based three-dimensional map construction method according to any one of claims 1 to 2 are implemented.

6. An electronic device, characterized in that: The device includes a memory and a processor, the memory stores computer processing instructions, and the processor executes the cloud-based three-dimensional map construction method described in any one of claims 1 to 2 by calling the computer processing instructions.

Images