Multi-device collaborative visual-inertial joint mapping method, device and system

Abstract

The present invention provides a multi-device collaborative visual-inertial joint mapping method, device and system, belonging to the field of navigation and positioning technology, the method comprising: determining the current position information of the mobile platform according to the current sensor data collected by the mobile platform for the unknown environment; determining a feature point set according to the current sensor data and the previous sensor data of the current sensor data; based on the current position information of the mobile platform and the feature point set, sending the feature point spatial coordinates and feature descriptors of each feature point solved to a server, so that the server can build a map of the unknown environment based on the feature point spatial coordinates and feature descriptors of each feature point. The multi-device collaborative visual-inertial joint mapping method, device and system provided by the present invention can not only ensure the timeliness of mapping, but also improve the accuracy of mapping, and can especially provide a rapid processing basis for emergencies.

Description

Technical Field

[0001] The present invention relates to the field of navigation and positioning technology, and in particular to a multi-device collaborative visual-inertial joint mapping method, device and system. Background Art

[0002] In the emergency response and rescue work of sudden incidents, it is necessary to quickly understand the scene of the incident, especially the indoor building structure, to facilitate work organization, action and retreat route determination and as a base map for indoor positioning.

[0003] At present, the 3D reconstruction of indoor scenes mainly uses backpack or push-broom 3D data acquisition equipment for simultaneous localization and mapping (SLAM). These devices perceive the surrounding environment and their own information through their own installed inertial navigation, multi-channel cameras, lidar and other sensors, and are operated by dedicated personnel to estimate the robot's posture and build a map consistent with the surrounding environment based on the perceived information. Although the mapping accuracy and fineness are high, the perception data often needs to be collected and processed, and the timeliness of its mapping is low for emergency response to emergencies and emergency rescue. Summary of the invention

[0004] The present invention provides a multi-device collaborative visual-inertial joint mapping method, device and system to solve the defect of low timeliness of mapping in the prior art.

[0005] The present invention provides a multi-device collaborative visual-inertial joint mapping method, which is applied to any mobile platform connected to a server for communication, and the method comprises:

[0006] Determining current position information of the mobile platform according to current sensor data collected by the mobile platform for the unknown environment;

[0007] Determine a feature point set according to the current sensor data and the previous sensor data of the current sensor data;

[0008] Based on the current position information of the mobile platform and the feature point set, the feature point spatial coordinates and feature descriptors of each feature point calculated are sent to the server, so that the server can build a map of the unknown environment based on the feature point spatial coordinates and feature descriptors of each feature point;

[0009] The current sensing data includes one or more of point cloud data, image data, posture data and navigation positioning data.

[0010] According to a multi-device collaborative visual-inertial joint mapping method provided by the present invention, determining a feature point set according to the current sensor data and the previous sensor data of the current sensor data includes:

[0011] When it is determined that the current sensing data includes point cloud data, the current point cloud data and the previous point cloud data are matched and screened using a descriptor matching method to obtain the feature point set.

[0012] According to a multi-device collaborative visual-inertial joint mapping method provided by the present invention, determining a feature point set according to the current sensor data and the previous sensor data of the current sensor data includes:

[0013] When it is determined that the current sensing data includes RGB image data and depth image data, the current RGB image data is matched with the previous RGB image data using a descriptor matching method to obtain candidate feature points;

[0014] The candidate feature points are screened using the depth image data to obtain the feature point set.

[0015] The present invention also provides a multi-device collaborative visual-inertial joint mapping method, which is applied to a server connected to multiple mobile platforms for communication, and the method includes:

[0016] Based on the feature point spatial coordinates and feature descriptors of all feature points uploaded by each mobile platform, matching tracking is performed to obtain matching feature point pairs;

[0017] The matched feature point pairs are merged using the least squares principle, and the feature point spatial coordinates of the obtained merged feature points are used as map points, so that a map of the unknown environment is established based on the map points of all mobile platforms.

[0018] According to a multi-device collaborative visual-inertial joint mapping method provided by the present invention, the merging of the matching feature point pairs using the least squares principle includes the following steps:

[0019] Based on each set of matching feature point pairs, spatial coordinate difference information is obtained;

[0020] The spatial coordinate difference information is minimized by the least squares principle to obtain the feature point spatial coordinates of the merged feature point.

[0021] According to a multi-device collaborative visual-inertial joint mapping method provided by the present invention, matching and tracking feature point spatial coordinates and feature descriptors of all feature points uploaded by each mobile platform to obtain matching feature point pairs includes:

[0022] Taking the feature points that meet the constraint conditions as target feature points, and performing matching tracking based on the feature point spatial coordinates and feature descriptors of the target feature points to obtain the matching feature point pairs;

[0023] The constraint condition is set according to a time window threshold and / or a spatial distance threshold.

[0024] The present invention also provides a multi-device collaborative visual-inertial joint mapping device, which is arranged on any mobile platform connected to a server for communication, and the device comprises:

[0025] An initial posture acquisition module, used to determine the current posture information of the mobile platform according to the current sensor data collected by the mobile platform for the unknown environment;

[0026] A feature point extraction module, used to determine a feature point set based on the current sensor data and the previous sensor data of the current sensor data;

[0027] A spatial coordinate analysis module, for sending the feature point spatial coordinates and feature descriptors of each feature point calculated based on the current position information of the mobile platform and the feature point set to a server, so that the server can establish a map of the unknown environment based on the feature point spatial coordinates and feature descriptors of each feature point;

[0028] The current sensing data includes one or more of point cloud data, image data, posture data and navigation positioning data.

[0029] The present invention also provides a multi-device collaborative visual-inertial joint mapping device, which is arranged on a server connected to multiple mobile platforms for communication, and the device includes:

[0030] A feature point matching module is used to perform matching tracking based on the feature point spatial coordinates and feature descriptors of all feature points uploaded by each mobile platform to obtain matching feature point pairs;

[0031] The mapping module is used to merge the matching feature point pairs using the least squares principle, and use the feature point spatial coordinates of the merged feature points as map points, so as to build a map of the unknown environment based on the map points of all mobile platforms.

[0032] The present invention also provides a multi-device collaborative visual-inertial joint mapping system, comprising a plurality of mobile platforms in an unknown environment, and a remote server; each of the mobile platforms is communicatively connected to the server;

[0033] The mobile platform includes a first control processor and a sensor device fixedly arranged on the mobile platform body and communicatively connected; the sensor device is used to collect corresponding sensor data of the surrounding scene of the mobile platform body; the first control processor is used to implement the multi-device collaborative visual-inertial joint mapping method as described in any one of the above items when executing a program;

[0034] The server comprises a memory, a second control processor, and a computer program stored in the memory and executable on the second control processor, wherein the second control processor implements the multi-device collaborative visual-inertial joint mapping method as described in any one of the above items when executing the program;

[0035] Among them, the mobile platform body includes augmented reality, mixed reality glasses and an intelligent robot; the sensing device includes one or more of a binocular camera, an RGB camera, a depth camera and a laser radar.

[0036] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the multi-device collaborative visual-inertial joint mapping methods described above.

[0037] The multi-device collaborative visual-inertial joint mapping method, device and system provided by the present invention use the mobile platform as the front end, adopt the visual-inertial method to obtain the current posture information of the mobile platform in an unknown environment, and solve the feature point set matched with the previous sensor data extracted from the current sensor data perceived by the mobile platform, and upload the feature descriptor of each feature point obtained by the solution and the spatial coordinates of the feature point in the world coordinate system to the server at the back end, and then estimate the camera motion between adjacent images and construct a local map for mapping. It can separate the execution process of SLAM, and the front-end mobile platform can process the sensor data in real time during the process of perceiving the environment to obtain a coarse-grained posture estimation result, and then the back-end server can optimize the posture estimation of different mobile platforms and splice them into a complete map, which not only ensures the timeliness of mapping, but also improves the accuracy of mapping, and can especially provide a basis for processing emergencies quickly. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 This is one of the flow charts of the multi-device collaborative visual-inertial joint mapping method provided by the present invention;

[0040] Figure 2 This is the second flow chart of the multi-device collaborative visual-inertial joint mapping method provided by the present invention;

[0041] Figure 3 This is one of the structural schematic diagrams of the multi-device collaborative visual-inertial joint mapping device provided by the present invention;

[0042] Figure 4 This is the second structural schematic diagram of the multi-device collaborative visual-inertial joint mapping device provided by the present invention;

[0043] Figure 5 It is a structural schematic diagram of the multi-device collaborative visual-inertial joint mapping system provided by the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] The terms "first", "second", etc. in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more.

[0046] It should be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0047] The terms “include” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0048] Figure 1 This is one of the flow charts of the multi-device collaborative visual-inertial joint mapping method provided by the present invention. Figure 1As shown, the multi-device collaborative visual-inertial joint mapping method provided in an embodiment of the present invention is applied to any mobile platform that is communicatively connected to a server, and the method includes: step 101, determining the current posture information of the mobile platform based on the current sensor data collected by the mobile platform for an unknown environment.

[0049] The current sensing data includes one or more of point cloud data, image data, posture data and navigation positioning data.

[0050] It should be noted that the executor of the multi-device collaborative visual-inertial joint mapping method provided in the embodiment of the present invention is a multi-device collaborative visual-inertial joint mapping device arranged on any mobile platform that is communicatively connected to the server.

[0051] The multi-device collaborative visual-inertial joint mapping method provided in the embodiment of the present application is suitable for carrying an electronic device with a multi-device collaborative visual-inertial joint mapping device on a mobile platform, synchronously positioning and mapping the collected video images, and restoring the 3D structure of the environment around the device.

[0052] The mobile platform used in the multi-device collaborative visual-inertial joint mapping method provided in the embodiment of the present application includes but is not limited to a mobile robot, an augmented reality device, a mixed reality device, and an autonomous driving vehicle. It starts moving from an unknown position in an unknown environment, positions itself according to the position and the map during the movement, and builds an incremental map based on its own positioning.

[0053] The electronic device described above can be implemented in various forms. For example, the electronic device described in the embodiments of the present application can be a terminal device that integrates a multi-device collaborative visual inertial joint mapping device and an image acquisition device, such as a mobile phone, a smart phone, a laptop computer, a digital broadcast receiver, a PDA (personal digital assistant), a PAD (tablet computer), a PMP (portable multimedia player), a navigation device, a smart bracelet, a smart watch, a digital camera, and the like.

[0054] The electronic device described in the embodiment of the present application may also be a terminal device for setting a real-time image stabilization device for video images, and the terminal needs to be connected to the video acquisition device for communication. Fixed terminals such as desktop computers and the like. Below, it is assumed that the electronic device is a mobile terminal. However, it will be understood by those skilled in the art that, in addition to components specifically used for mobile purposes, the construction according to the embodiment of the present application can also be applied to fixed type terminals.

[0055] It should be noted that a variety of sensor devices may be provided on the mobile platform to sense the surrounding environment.

[0056] Optionally, the sensing device may be a three-dimensional imaging sensor for acquiring point cloud data of the surrounding environment. For example, the three-dimensional imaging sensor may be a three-dimensional laser scanner, a binocular camera, an RGB-D camera, a laser radar (Light Detection and Ranging, LiDAR), etc.

[0057] Optionally, the sensing device may be a monocular camera or a binocular camera for acquiring image data of the surrounding environment. Among them, the device for establishing a three-dimensional scene by using images collected by different cameras from their respective viewpoints of the same scene includes, in addition to binocular cameras, multi-eye cameras.

[0058] Optionally, the sensing device may be an inertial navigation device, which is used to obtain the attitude data of the mobile platform itself. For example, the inertial navigation device may be an inertial sensor (Inertial measurement unit, IMU), a gyroscope, an accelerometer, etc.

[0059] Optionally, the sensing device may be a positioning device for acquiring navigation positioning data of the mobile platform itself in a world coordinate system. For example, the positioning device may be a global positioning system (GPS), a Beidou satellite, a global navigation satellite system (GNSS), etc.

[0060] Specifically, in step 101, a multi-device collaborative visual-inertial joint mapping device installed on any mobile platform that is communicatively connected to the server uses a sensor device installed on the mobile platform to sense the schedule according to the unknown environment in which it is located, and uses a visual-inertial algorithm to process the current sensor data obtained by perception to obtain the current posture information of the mobile platform in six degrees of freedom.

[0061] The current posture information refers to the posture information of the mobile platform at the corresponding moment when the sensing device obtains the current sensing data during the movement of the mobile platform. The posture information at any moment includes the rotation change and the translation change.

[0062] Step 102: Determine a feature point set according to the current sensing data and the previous sensing data of the current sensing data.

[0063] Specifically, in step 102, a multi-device collaborative visual-inertial joint mapping apparatus disposed on any mobile platform communicatively connected to the server uses an Orb-Brief descriptor method to match current sensor data with previous sensor data acquired in a previous acquisition cycle of the current sensor data, that is, the feature points are detected using a FAST feature point detection method, and then a Harris corner point measurement method is used to select N feature points with the largest Harris corner point response values ​​from the FAST feature points, and the N feature points matched this time are used as pre-selected map points to form a feature point set output.

[0064] Step 103: Based on the current posture information of the mobile platform and the feature point set, the feature point spatial coordinates and feature descriptors of each feature point calculated are sent to the server, so that the server can build a map of the unknown environment based on the feature point spatial coordinates and feature descriptors of each feature point.

[0065] Specifically, in step 103, a multi-device collaborative visual-inertial joint mapping apparatus disposed on any mobile platform communicatively connected to the server uses the current posture information of the mobile platform in the world coordinate system to perform coordinate system transformation on the position coordinates of each feature point in the feature point set in the image coordinate system, and obtains the feature point spatial coordinates of each feature point in the world coordinate system and its corresponding feature descriptor.

[0066] After integrating the feature point spatial coordinates and feature descriptors of all feature points in the feature point set, they are sent to the server at the back end, so that the server can aggregate the feature point spatial coordinates and feature descriptors of the feature points solved by all mobile platforms, perform local bundle adjustment (Bundle Adjustment, BA), and then seamlessly connect the historical trajectory of the mobile platform itself and integrate it with the trajectories of other mobile platforms to obtain a map of the current unknown environment.

[0067] The embodiment of the present invention uses the mobile platform as the front end, adopts the visual inertial method to obtain the current posture information of the mobile platform in the unknown environment, extracts the feature point set matching the previous sensor data from the current sensor data perceived by the mobile platform, combines the feature point set for solution, and uploads the feature descriptor of each feature point obtained by solution and its feature point spatial coordinates in the world coordinate system to the back-end server, thereby estimating the camera motion between adjacent images and constructing a local map for mapping. By separating the execution process of SLAM, the front-end mobile platform can process the sensor data in real time during the process of perceiving the environment to obtain a coarse-grained posture estimation result, and then the back-end server optimizes the posture estimation of different mobile platforms at the same time and splices them into a complete map, which not only ensures the timeliness of mapping, but also improves the accuracy of mapping, and can especially provide a basis for rapid processing of emergencies.

[0068] On the basis of any of the above embodiments, a descriptor-based matching method is used to match the current sensor data with the previous sensor data of the current sensor data to obtain a set of feature points, including: when it is determined that the current sensor data includes point cloud data, the current point cloud data and the previous point cloud data are matched and screened by using a descriptor matching method to obtain the set of feature points.

[0069] Specifically, in step 102, if the data type of the sensor data received from the sensor device by the multi-device collaborative visual-inertial joint mapping apparatus is point cloud data, that is, the sensor device at least includes a binocular camera or a multi-camera that integrates a three-dimensional imaging function, then a descriptor-based matching method such as ORB or SIFT is used to match and track the current point cloud data with reference to the previous sensor data, and the feature point pairs that can be matched and tracked are stored as pre-selected map points in the feature point set output.

[0070] Among them, any feature point is composed of two parts: feature key point and feature descriptor. The feature key point refers to the position of the feature point in the image, and the feature descriptor refers to the orientation of the key point and the surrounding pixel information.

[0071] When the embodiment of the present invention determines that the sensor configuration on the mobile platform is a binocular or multi-camera solution, the feature points that are still continuously observed from the previous point cloud data are stored in the feature point set output, and different feature point extraction schemes can be executed according to different matching combinations in the sensor device, thereby accelerating the front-end processing efficiency.

[0072] Based on any of the above embodiments, a descriptor-based matching method is used to match the current sensor data with the previous sensor data of the current sensor data to obtain a set of feature points, including: when it is determined that the current sensor data includes RGB image data and depth image data, the current RGB image data and the previous RGB image data are matched using a descriptor matching method to obtain candidate feature points.

[0073] Specifically, in step 102, if the data type of the sensor data received by the multi-device collaborative visual-inertial joint mapping apparatus from the sensor device is image data, that is, the sensor device at least includes other combined three-dimensional imaging devices other than binocular cameras and multi-camera cameras, for example, a combination of RGB cameras with depth scanning devices such as lidar and depth cameras, then a descriptor matching method such as ORB or SIFT is used to first match and track the current RGB image data with reference to the previous RGB image data, and the feature point pairs that can be matched and tracked are used as candidate feature points that meet the continuity of image color information.

[0074] Among them, any feature point is composed of two parts: feature key point and feature descriptor. The feature key point refers to the position of the feature point in the image, and the feature descriptor refers to the orientation of the key point and the surrounding pixel information.

[0075] The candidate feature points are screened using the depth image data to obtain the feature point set.

[0076] Specifically, a multi-device collaborative visual-inertial joint mapping device installed on any mobile platform connected to the server for communication uses the depth image data to screen candidate feature points, and stores the feature points that can be matched and tracked and meet the requirements of image color information continuity and depth color information continuity into a feature point set output.

[0077] When the embodiment of the present invention determines that the sensor configuration on the mobile platform is a scheme for combining 3D imaging with an RGB camera and a depth scanning device, it decides to match and track the current RGB image data sensed by the mobile platform with the previous RGB image data, and selects the feature points that are still continuously observed from the previous RGB image data as candidate feature points, and then combines the depth image data for screening, and stores the screened feature points in the feature point set output. Different feature point extraction schemes can be executed according to different combinations in the sensor device, thereby accelerating the front-end processing efficiency.

[0078] Figure 2 This is the second flow chart of the multi-device collaborative visual-inertial joint mapping method provided by the present invention. Figure 2As shown, based on any of the above embodiments, a multi-device collaborative visual-inertial joint mapping method provided in an embodiment of the present invention is applied to a server connected to multiple mobile platforms for communication, and the method includes: step 201, matching and tracking the feature point spatial coordinates and feature descriptors of all feature points uploaded by each mobile platform to obtain matching feature point pairs.

[0079] It should be noted that the execution subject of the multi-device collaborative visual-inertial joint mapping method provided in the embodiment of the present invention is a multi-device collaborative visual-inertial joint mapping device arranged on a server that is communicatively connected to multiple mobile platforms.

[0080] The multi-device collaborative visual-inertial joint mapping method provided in the embodiment of the present application is suitable for optimizing the initial pose estimation of the mobile platform by using a server to carry an electronic device with a multi-device collaborative visual-inertial joint mapping device, and restoring the 3D structure of the environment around the device.

[0081] The electronic device described above can be implemented in various forms. For example, the electronic device described in the embodiments of the present application can be a terminal device that integrates a multi-device collaborative visual inertial joint mapping device and an image acquisition device, such as a mobile phone, a smart phone, a laptop computer, a digital broadcast receiver, a PDA (personal digital assistant), a PAD (tablet computer), a PMP (portable multimedia player), a navigation device, a smart bracelet, a smart watch, a digital camera, and the like.

[0082] The electronic device described in the embodiments of the present application may also be a fixed terminal such as a desktop computer, etc. Below, it is assumed that the electronic device is a mobile terminal. However, it will be understood by those skilled in the art that, in addition to the components specifically used for mobile purposes, the configuration according to the embodiments of the present application can also be applied to fixed type terminals.

[0083] Specifically, in step 201, a multi-device collaborative visual-inertial joint mapping apparatus disposed on a server connected to multiple mobile platforms receives feature point spatial coordinates and feature descriptors of each feature point in a feature point set uploaded by each of the multiple mobile platforms, and performs the following processing on any target mobile platform:

[0084] The feature point spatial coordinates and feature descriptors of any feature point uploaded by the target mobile platform are used as reference points, a window grayscale error function with the target feature point offset as the optimized amount is constructed, the offset is estimated using the pyramid optical flow method, and matching feature point pairs are obtained to characterize whether the feature points uploaded by the target mobile platform within the period are continuous with the feature points uploaded by other mobile platforms within the period.

[0085] Step 202: Use the least squares principle to merge the matching feature point pairs corresponding to the target mobile platform, and use the feature point spatial coordinates of the merged feature points as the map points of the target mobile platform, so that a map of the unknown environment is established based on the map points of all mobile platforms.

[0086] Specifically, in step 202, the BA optimization restriction of the multi-device collaborative visual-inertial joint mapping apparatus on the server communicatively connected to the multiple mobile platforms is to minimize the reprojection error between the matching feature point pairs formed by the feature points of different mobile platforms, and after merging the matching feature point pairs whose error values ​​are less than the threshold, the feature point spatial coordinates of the iteratively optimized merged feature points are used as new map points, so as to seamlessly connect the new map points with all historical trajectories to obtain a map of the current unknown environment.

[0087] The embodiment of the present invention uses the mobile platform as the front end, adopts the visual inertial method to obtain the current posture information of the mobile platform in the unknown environment, combines the feature point set extracted from the current sensor data perceived by the mobile platform and matched with the previous sensor data for solution, and uploads the feature descriptor of each feature point obtained by solution and its feature point spatial coordinates in the world coordinate system to the back-end server to estimate the camera motion between adjacent images and construct a local map for mapping. It can separate the execution process of SLAM, and the front-end mobile platform can process the sensor data in real time during the process of perceiving the environment to obtain a coarse-grained posture estimation result, and then the back-end server can optimize the posture estimation of different mobile platforms at the same time and splice them into a complete map, which not only ensures the timeliness of mapping, but also improves the accuracy of mapping, and can especially provide a quick processing basis for emergencies.

[0088] On the basis of any of the above embodiments, the following steps are performed when the matching feature point pairs are merged using the least squares principle: spatial coordinate difference information is acquired based on each group of matching feature point pairs corresponding to the target mobile platform.

[0089] Specifically, in step 202, a multi-device collaborative visual-inertial joint mapping apparatus disposed on a server connected to multiple mobile platforms takes one feature point in each set of matching feature point pairs as a reference point, and its spatial coordinates are marked as , another feature point is used as the point to be merged, and its feature point space coordinates are marked as , calculate the spatial coordinate difference information of the two , and its calculation formula is as follows:

[0090]

[0091] in, For the The square distance between the group matching feature point pairs, spatial coordinate difference information for , n+1 represents the number of matching feature point pairs.

[0092] The spatial coordinate difference information is minimized by the least squares principle to obtain the feature point spatial coordinates of the merged feature point.

[0093] Specifically, a multi-device collaborative visual-inertial joint mapping device arranged on a server connected to multiple mobile platforms adopts the least squares principle to minimize the overall translation of the points to be merged, thereby merging the two feature points in the point pair, and outputs the feature point spatial coordinates of the merged feature points as the optimal solution.

[0094] The embodiment of the present invention calculates the difference between the feature points uploaded by different mobile platforms in any matching feature point pair to obtain spatial coordinate difference information, minimizes the spatial coordinate difference information by the least square method, and continuously iterates and optimizes the feature point spatial coordinates of the merged feature points. The index reflecting the difference in feature spatial coordinates, such as the sum of squares of the matched feature coordinate differences collected by different mobile platforms, can be used as a target, and the feature cloud coordinates can be adjusted as a whole to reduce the difference in matched feature coordinates and improve the backend optimization accuracy.

[0095] On the basis of any of the above embodiments, matching and tracking are performed based on the feature point spatial coordinates and feature descriptors of all feature points uploaded by each mobile platform to obtain matching feature point pairs, including: taking feature points that meet the constraint conditions as target feature points, and matching and tracking are performed based on the feature point spatial coordinates and feature descriptors of the target feature points to obtain the matching feature point pairs.

[0096] The constraint condition is set according to a time window threshold and / or a spatial distance threshold.

[0097] Specifically, a multi-device collaborative visual-inertial joint mapping apparatus arranged on a server connected to communication with multiple mobile platforms can limit feature points within a dynamic sliding window. By adding feature points within the sliding window and marginalizing old feature points, the reprojection error between adjacent feature points in the sliding window is minimized as the goal, and relatively continuous target feature points are iteratively optimized. The feature point spatial coordinates and feature descriptors of the target feature points are used for matching and tracking, and the feature points from other mobile platforms that match the target feature points form a group of matching feature point pairs.

[0098] The optimization condition of the sliding window can be set to a time window threshold and / or a spatial distance threshold of the matching feature according to actual needs.

[0099] The embodiment of the present invention limits the acquisition process of matching feature point pairs within a sliding window, and optimizes the selection of feature points by matching the time window threshold and the spatial distance threshold of the feature, which can reduce the amount of calculation and improve the optimization efficiency of the backend.

[0100] Figure 3 This is one of the structural schematic diagrams of the multi-device collaborative visual-inertial joint mapping device provided by the present invention. Figure 3 As shown, on the basis of any of the above embodiments, a multi-device collaborative visual-inertial joint mapping device provided in an embodiment of the present invention is arranged on any mobile platform connected to a server for communication, and the device includes an initial pose acquisition module 310, a feature point extraction module 320 and a spatial coordinate analysis module 330, wherein:

[0101] The initial posture acquisition module 310 is used to determine the current posture information of the mobile platform according to the current sensor data collected by the mobile platform for the unknown environment.

[0102] The feature point extraction module 320 is used to determine a feature point set according to the current sensing data and the previous sensing data of the current sensing data.

[0103] The spatial coordinate parsing module 330 is used to send the feature point spatial coordinates and feature descriptors of each feature point calculated to the server based on the current posture information of the mobile platform and the feature point set, so that the server can build a map of the unknown environment based on the feature point spatial coordinates and feature descriptors of each feature point.

[0104] The current sensing data includes one or more of point cloud data, image data, posture data and navigation positioning data.

[0105] Specifically, the initial posture acquisition module 310, the feature point extraction module 320 and the space coordinate analysis module 330 are electrically connected in sequence.

[0106] The initial posture acquisition module 310 uses the sensor device set on the mobile platform to perceive the unknown environment in which the schedule is located, and uses the visual inertial algorithm to process the current sensor data perceived to obtain the current posture information of the mobile platform in six degrees of freedom.

[0107] The feature point extraction module 320 uses the Orb-Brief descriptor method to match the current sensor data with the previous sensor data obtained in the previous acquisition cycle of the current sensor data, that is, the FAST feature point detection method is used to detect the feature points, and then the Harris corner point measurement method is used to select the N feature points with the largest Harris corner point response values ​​from the FAST feature points, and the N feature points matched this time are used as pre-selected map points to form a feature point set output.

[0108] The spatial coordinate analysis module 330 uses the current position information of the mobile platform in the world coordinate system to convert the position coordinates of each feature point in the feature point set in the image coordinate system, and obtains the feature point spatial coordinates of each feature point in the world coordinate system and its corresponding feature descriptor. The feature point spatial coordinates and feature descriptors of all feature points in the feature point set are integrated and sent to the server at the back end.

[0109] Optionally, the feature point extraction module 320 is specifically configured to, when it is determined that the current sensing data includes point cloud data, match and filter the current point cloud data with the previous point cloud data using a descriptor matching method to obtain the feature point set.

[0110] Optionally, the feature point extraction module 320 is specifically used to match the current RGB image data with the previous RGB image data using a descriptor matching method to obtain candidate feature points when it is determined that the current sensor data includes RGB image data and depth image data.

[0111] The candidate feature points are screened using the depth image data to obtain the feature point set.

[0112] The multi-device collaborative visual-inertial joint mapping apparatus provided in an embodiment of the present invention is used to execute the multi-device collaborative visual-inertial joint mapping method mentioned above. Its implementation method is consistent with the implementation method of the multi-device collaborative visual-inertial joint mapping method provided by the present invention, and can achieve the same beneficial effects, which will not be repeated here.

[0113] The embodiment of the present invention uses the mobile platform as the front end, adopts the visual inertial method to obtain the current posture information of the mobile platform in the unknown environment, combines the feature point set extracted from the current sensor data perceived by the mobile platform and matched with the previous sensor data for solution, and uploads the feature descriptor of each feature point obtained by solution and its feature point spatial coordinates in the world coordinate system to the back-end server to estimate the camera motion between adjacent images and construct a local map for mapping. It can separate the execution process of SLAM, and the front-end mobile platform can process the sensor data in real time during the process of perceiving the environment to obtain a coarse-grained posture estimation result, and then the back-end server can optimize the posture estimation of different mobile platforms at the same time and splice them into a complete map, which not only ensures the timeliness of mapping, but also improves the accuracy of mapping, and can especially provide a quick processing basis for emergencies.

[0114] Figure 4 This is the second structural diagram of the multi-device collaborative visual-inertial joint mapping device provided by the present invention. Figure 4As shown, based on any of the above embodiments, a multi-device collaborative visual-inertial joint mapping device provided in an embodiment of the present invention is arranged in a server connected to multiple mobile platforms for communication, and the device includes a feature point matching module 410 and a mapping module 420, wherein:

[0115] The feature point matching module 410 is used to perform matching tracking based on the feature point spatial coordinates and feature descriptors of all feature points uploaded by each mobile platform to obtain matching feature point pairs.

[0116] The mapping module 420 is used to merge the matching feature point pairs using the least squares principle, and use the feature point spatial coordinates of the merged feature points as map points, so as to build a map of the unknown environment based on the map points of all mobile platforms.

[0117] Specifically, the feature point matching module 410 and the mapping module 420 are electrically connected in sequence.

[0118] The feature point matching module 410 receives the feature point spatial coordinates and feature descriptors of each feature point in the feature point set uploaded by multiple mobile platforms, and performs the following processing on any target mobile platform:

[0119] The feature point spatial coordinates and feature descriptors of any feature point uploaded by the target mobile platform are used as reference points, a window grayscale error function with the target feature point offset as the optimized amount is constructed, the offset is estimated using the pyramid optical flow method, and matching feature point pairs are obtained to characterize whether the feature points uploaded by the target mobile platform within the period are continuous with the feature points uploaded by other mobile platforms within the period.

[0120] The BA optimization of the mapping module 420 is limited to minimizing the reprojection error between the matching feature point pairs formed by the feature points of different mobile platforms. After merging the matching feature point pairs with error values ​​less than the threshold, the feature point spatial coordinates of the iteratively optimized merged feature points are used as new map points to seamlessly connect the new map points with all historical trajectories to obtain a map of the current unknown environment.

[0121] Optionally, the mapping module 420 includes a spatial coordinate difference acquisition unit and an iterative optimization unit, wherein:

[0122] The spatial coordinate difference obtaining unit is used to obtain spatial coordinate difference information based on each set of matching feature point pairs.

[0123] The iterative optimization unit is used to minimize the spatial coordinate difference information through the least squares principle to obtain the feature point spatial coordinates of the merged feature point.

[0124] Optionally, the feature point matching module 410 is specifically configured to use feature points that meet the constraint conditions as target feature points, and perform matching and tracking based on feature point spatial coordinates and feature descriptors of the target feature points to obtain the matching feature point pairs.

[0125] The constraint condition is set according to a time window threshold and / or a spatial distance threshold.

[0126] The multi-device collaborative visual-inertial joint mapping apparatus provided in an embodiment of the present invention is used to execute the multi-device collaborative visual-inertial joint mapping method mentioned above. Its implementation method is consistent with the implementation method of the multi-device collaborative visual-inertial joint mapping method provided by the present invention, and can achieve the same beneficial effects, which will not be repeated here.

[0127] The embodiment of the present invention uses the mobile platform as the front end, adopts the visual inertial method to obtain the current posture information of the mobile platform in the unknown environment, combines the feature point set extracted from the current sensor data perceived by the mobile platform and matched with the previous sensor data for solution, and uploads the feature descriptor of each feature point obtained by solution and its feature point spatial coordinates in the world coordinate system to the back-end server to estimate the camera motion between adjacent images and construct a local map for mapping. It can separate the execution process of SLAM, and the front-end mobile platform can process the sensor data in real time during the process of perceiving the environment to obtain a coarse-grained posture estimation result, and then the back-end server can optimize the posture estimation of different mobile platforms at the same time and splice them into a complete map, which not only ensures the timeliness of mapping, but also improves the accuracy of mapping, and can especially provide a quick processing basis for emergencies.

[0128] Figure 5 Schematic diagram of the structure of the multi-device collaborative visual-inertial joint mapping system provided by the present invention. Figure 5 As shown, based on any of the above embodiments, the multi-device collaborative visual-inertial joint mapping system provided in the embodiment of the present invention includes multiple mobile platforms 510 in the same unknown environment, and a remote server 520. Each of the mobile platforms 510 is in communication connection with the server 520.

[0129] The mobile platform 510 includes a first control processor 512 and a sensor device 513 fixedly arranged and communicatively connected on the mobile platform body 511. The sensor device 513 is used to collect corresponding sensor data of the surrounding scene of the mobile platform body 511. The first control processor 512 is used to implement the multi-device collaborative visual-inertial joint mapping method as described in any of the above items when executing a program.

[0130] The server 520 includes a memory, a second control processor 521, and a computer program stored in the memory and executable on the second control processor 521. When the second control processor 521 executes the program, the multi-device collaborative visual-inertial joint mapping method as described in any one of the above items is implemented.

[0131] The mobile platform body 511 includes augmented reality, mixed reality glasses and intelligent robots. The sensor device 513 includes one or more of a binocular camera, an RGB camera, a depth camera and a laser radar.

[0132] Specifically, the multi-device collaborative visual-inertial joint mapping system consists of a mobile platform 510 at the front end for perception, and a server 520 at the back end.

[0133] The mobile platform 510 is composed of a first control processor 512 and a sensor device 513 which are fixedly arranged on the mobile platform body 511 and can be communicatively connected. During the movement of the mobile platform body 511, the sensor device 513 such as a binocular camera, an RGB camera, a depth camera and a lidar is used to perceive the current unknown environment, and the collected sensor data is transmitted to the first control processor 512.

[0134] The first control processor 512 processes the current sensing data sensed by the visual inertial algorithm to obtain the current position information of the mobile platform in six degrees of freedom.

[0135] The first control processor 512 uses the Orb-Brief descriptor method to match the current sensor data with the previous sensor data obtained in the previous acquisition cycle of the current sensor data, that is, uses the FAST feature point detection method to detect feature points, and then uses the Harris corner point measurement method to select N feature points with the largest Harris corner point response values ​​from the FAST feature points, and uses the N feature points matched this time as pre-selected map points to form a feature point set output.

[0136] The first control processor 512 uses the current posture information of the mobile platform in the world coordinate system to perform coordinate system conversion on the position coordinates of each feature point in the feature point set in the image coordinate system to obtain the feature point spatial coordinates of each feature point in the world coordinate system and its corresponding feature descriptor.

[0137] The first control processor 512 integrates the feature point spatial coordinates and feature descriptors of all feature points in the feature point set, and sends them to the server at the back end, so that the second control processor 521 of the server 520 can aggregate the feature point spatial coordinates and feature descriptors of the feature points solved by all mobile platforms 510, perform local bundle adjustment (Bundle Adjustment, BA), and after seamlessly connecting the historical trajectory of the mobile platform itself, integrate it with the trajectories of other mobile platforms to obtain a map of the current unknown environment.

[0138] The embodiment of the present invention uses the mobile platform as the front end, adopts the visual inertial method to obtain the current posture information of the mobile platform in the unknown environment, combines the feature point set extracted from the current sensor data perceived by the mobile platform and matched with the previous sensor data for solution, and uploads the feature descriptor of each feature point obtained by solution and its feature point spatial coordinates in the world coordinate system to the back-end server to estimate the camera motion between adjacent images and construct a local map for mapping. It can separate the execution process of SLAM, and the front-end mobile platform can process the sensor data in real time during the process of perceiving the environment to obtain a coarse-grained posture estimation result, and then the back-end server can optimize the posture estimation of different mobile platforms at the same time and splice them into a complete map, which not only ensures the timeliness of mapping, but also improves the accuracy of mapping, and can especially provide a quick processing basis for emergencies.

[0139] In addition, the logic instructions can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0140] On the other hand, the present invention also provides a computer program product, which includes a computer program, and the computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the multi-device collaborative visual-inertial joint mapping method provided by the above-mentioned methods, and the method includes: determining the current posture information of the mobile platform based on the current sensor data collected by the mobile platform for the unknown environment; determining a set of feature points based on the current sensor data and the previous sensor data of the current sensor data; based on the current posture information of the mobile platform and the feature point set, sending the feature point spatial coordinates and feature descriptors of each feature point solved to a server, so that the server can establish a map of the unknown environment based on the feature point spatial coordinates and feature descriptors of each feature point; wherein the current sensor data includes one or more of point cloud data, image data, posture data and navigation positioning data. The multi-device collaborative visual-inertial joint mapping method provided by the above methods is also executed, and the method includes: matching and tracking the feature point spatial coordinates and feature descriptors of all feature points uploaded by each mobile platform to obtain matching feature point pairs; merging the matching feature point pairs using the least squares principle, and using the feature point spatial coordinates of the obtained merged feature points as map points, so that a map of the unknown environment is established based on the map points of all mobile platforms.

[0141] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the multi-device collaborative visual-inertial joint mapping method provided by the above-mentioned methods, the method comprising: determining the current posture information of the mobile platform based on the current sensor data collected by the mobile platform for the unknown environment; determining a set of feature points based on the current sensor data and the previous sensor data of the current sensor data; based on the current posture information of the mobile platform and the feature point set, sending the feature point spatial coordinates and feature descriptors of each feature point solved to a server, so that the server can establish a map of the unknown environment based on the feature point spatial coordinates and feature descriptors of each feature point; wherein the current sensor data includes one or more of point cloud data, image data, posture data and navigation positioning data. The multi-device collaborative visual-inertial joint mapping method provided by the above methods is also executed, and the method includes: matching and tracking the feature point spatial coordinates and feature descriptors of all feature points uploaded by each mobile platform to obtain matching feature point pairs; merging the matching feature point pairs using the least squares principle, and using the feature point spatial coordinates of the obtained merged feature points as map points, so that a map of the unknown environment is established based on the map points of all mobile platforms.

[0142] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0143] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-device collaborative visual-inertial joint mapping method, It is characterized in that Applied to any mobile platform in communication with a server, the method comprises: Determining current position information of the mobile platform according to current sensor data collected by the mobile platform for the unknown environment; Determine a feature point set according to the current sensor data and the sensor data before the current sensor data; the front-end mobile platform processes the sensor data in real time during the process of perceiving the environment to obtain a coarse-grained pose estimation result, and then the back-end server optimizes the pose estimation results of different mobile platforms; Based on the current posture information of the mobile platform and the feature point set, the position coordinates of each feature point in the feature point set in the image coordinate system are converted into coordinate systems by using the current posture information of the mobile platform in the world coordinate system, so as to obtain the feature point spatial coordinates of each feature point in the world coordinate system and its corresponding feature descriptor, and send the solved feature point spatial coordinates and feature descriptor of each feature point to the server, so that the server can establish a map of the unknown environment based on the feature point spatial coordinates and feature descriptor of each feature point; Wherein, the current sensing data includes one or more of point cloud data, image data, posture data and navigation positioning data; The determining of a feature point set according to the current sensing data and the previous sensing data of the current sensing data includes: When it is determined that the current sensing data includes point cloud data, the current point cloud data and the previous point cloud data are matched and screened using a descriptor matching method to obtain the feature point set; The determining of a feature point set according to the current sensor data and the sensor data previous to the current sensor data further includes: When it is determined that the current sensing data includes RGB image data and depth image data, the current RGB image data is matched with the previous RGB image data using a descriptor matching method to obtain candidate feature points; The candidate feature points are screened using the depth image data to obtain the feature point set.

2. A multi-device collaborative visual-inertial joint mapping method, It is characterized in that Applied to a server in communication connection with a plurality of mobile platforms, the method comprises: Based on the feature point spatial coordinates and feature descriptors of all feature points uploaded by each mobile platform, matching tracking is performed to obtain matching feature point pairs; The matched feature point pairs are merged using the least squares principle, and the feature point spatial coordinates of the merged feature points are used as map points, so that a map of the unknown environment is established based on the map points of all mobile platforms; The matching and tracking of feature point spatial coordinates and feature descriptors of all feature points uploaded by each mobile platform to obtain matching feature point pairs includes: Taking the feature points that meet the constraint conditions as target feature points, and performing matching tracking based on the feature point spatial coordinates and feature descriptors of the target feature points to obtain the matching feature point pairs; Wherein, the constraint condition is set according to a time window threshold and / or a spatial distance threshold; The feature point spatial coordinates and feature descriptors of the feature points uploaded by the mobile platform are calculated by the mobile platform based on the current posture information of the mobile platform and the feature point set; the feature point set is determined by the mobile platform based on the current sensor data and the previous sensor data of the current sensor data; the current posture information is determined by the mobile platform based on the current sensor data collected from the unknown environment; the front-end mobile platform processes the sensor data in real time during the process of perceiving the environment to obtain a coarse-grained posture estimation result, and then the back-end server optimizes the posture estimation results of different mobile platforms; The specific steps of calculating the feature point spatial coordinates and feature descriptors of the feature points based on the current posture information of the mobile platform and the feature point set include: Based on the current posture information of the mobile platform and the feature point set, the position coordinates of each feature point in the feature point set in the image coordinate system are converted into a coordinate system by using the current posture information of the mobile platform in the world coordinate system, so as to obtain the feature point spatial coordinates of each feature point in the world coordinate system and its corresponding feature descriptor; The specific steps of the mobile platform determining the feature point set according to the current sensing data and the previous sensing data of the current sensing data include: When it is determined that the current sensing data includes point cloud data, the current point cloud data and the previous point cloud data are matched and screened using a descriptor matching method to obtain the feature point set; The specific step of the mobile platform determining the feature point set according to the current sensing data and the previous sensing data of the current sensing data also includes: When it is determined that the current sensing data includes RGB image data and depth image data, the current RGB image data is matched with the previous RGB image data using a descriptor matching method to obtain candidate feature points; The candidate feature points are screened using the depth image data to obtain the feature point set.

3. The multi-device collaborative visual-inertial joint mapping method according to claim 2, It is characterized in that The method of merging the matching feature point pairs by using the least squares principle comprises the following steps: Based on each set of matching feature point pairs, spatial coordinate difference information is obtained; The spatial coordinate difference information is minimized by the least squares principle to obtain the feature point spatial coordinates of the merged feature point.

4. A multi-device collaborative visual-inertial joint mapping device, It is characterized in that The device is arranged on any mobile platform in communication with the server, and comprises: An initial posture acquisition module, used to determine the current posture information of the mobile platform according to the current sensor data collected by the mobile platform for the unknown environment; A feature point extraction module is used to determine a feature point set based on the current sensor data and the previous sensor data of the current sensor data; the front-end mobile platform processes the sensor data in real time during the process of perceiving the environment to obtain a coarse-grained pose estimation result, and then the back-end server optimizes the pose estimation results of different mobile platforms; A spatial coordinate parsing module is used to perform coordinate system conversion on the position coordinates of each feature point in the feature point set in the image coordinate system based on the current posture information of the mobile platform and the feature point set, using the current posture information of the mobile platform in the world coordinate system, to obtain the feature point spatial coordinates of each feature point in the world coordinate system and its corresponding feature descriptor, and send the feature point spatial coordinates and feature descriptor of each feature point calculated to the server, so that the server can establish a map of the unknown environment based on the feature point spatial coordinates and feature descriptor of each feature point; Wherein, the current sensing data includes one or more of point cloud data, image data, posture data and navigation positioning data; The determining of a feature point set according to the current sensing data and the previous sensing data of the current sensing data includes: When it is determined that the current sensing data includes point cloud data, the current point cloud data and the previous point cloud data are matched and screened using a descriptor matching method to obtain the feature point set; The determining of a feature point set according to the current sensor data and the sensor data previous to the current sensor data further includes: When it is determined that the current sensing data includes RGB image data and depth image data, the current RGB image data is matched with the previous RGB image data using a descriptor matching method to obtain candidate feature points; The candidate feature points are screened using the depth image data to obtain the feature point set.

5. A multi-device collaborative visual-inertial joint mapping device, It is characterized in that The device is arranged on a server connected to multiple mobile platforms for communication, and includes: A feature point matching module is used to perform matching tracking based on the feature point spatial coordinates and feature descriptors of all feature points uploaded by each mobile platform to obtain matching feature point pairs; A mapping module, used to merge the matching feature point pairs using the least squares principle, and use the feature point spatial coordinates of the merged feature points as map points, so as to build a map of the unknown environment based on the map points of all mobile platforms; The matching and tracking of feature point spatial coordinates and feature descriptors of all feature points uploaded by each mobile platform to obtain matching feature point pairs includes: Taking the feature points that meet the constraint conditions as target feature points, and performing matching tracking based on the feature point spatial coordinates and feature descriptors of the target feature points to obtain the matching feature point pairs; Wherein, the constraint condition is set according to a time window threshold and / or a spatial distance threshold; The feature point spatial coordinates and feature descriptors of the feature points uploaded by the mobile platform are calculated by the mobile platform based on the current posture information of the mobile platform and the feature point set; the feature point set is determined by the mobile platform based on the current sensor data and the previous sensor data of the current sensor data; the current posture information is determined by the mobile platform based on the current sensor data collected from the unknown environment; the front-end mobile platform processes the sensor data in real time during the process of perceiving the environment to obtain a coarse-grained posture estimation result, and then the back-end server optimizes the posture estimation results of different mobile platforms; The specific steps of calculating the feature point spatial coordinates and feature descriptors of the feature points based on the current posture information of the mobile platform and the feature point set include: Based on the current posture information of the mobile platform and the feature point set, the position coordinates of each feature point in the feature point set in the image coordinate system are converted into a coordinate system by using the current posture information of the mobile platform in the world coordinate system, so as to obtain the feature point spatial coordinates of each feature point in the world coordinate system and its corresponding feature descriptor; The specific steps of the mobile platform determining the feature point set according to the current sensing data and the previous sensing data of the current sensing data include: When it is determined that the current sensing data includes point cloud data, the current point cloud data and the previous point cloud data are matched and screened using a descriptor matching method to obtain the feature point set; The specific step of the mobile platform determining the feature point set according to the current sensing data and the previous sensing data of the current sensing data also includes: When it is determined that the current sensing data includes RGB image data and depth image data, the current RGB image data is matched with the previous RGB image data using a descriptor matching method to obtain candidate feature points; The candidate feature points are screened using the depth image data to obtain the feature point set.

6. A multi-device collaborative visual-inertial joint mapping system, It is characterized in that It includes a plurality of mobile platforms in an unknown environment and a remote server; each of the mobile platforms is in communication connection with the server; The mobile platform includes a first control processor and a sensor device fixedly arranged on the mobile platform body and communicatively connected; the sensor device is used to collect corresponding sensor data of the surrounding scene of the mobile platform body; The first control processor is used to implement the multi-device collaborative visual-inertial joint mapping method as claimed in claim 1 when executing the program; The server comprises a memory, a second control processor, and a computer program stored in the memory and executable on the second control processor, wherein the second control processor implements the multi-device collaborative visual-inertial joint mapping method as claimed in any one of claims 2 to 3 when executing the program; Among them, the mobile platform body includes augmented reality, mixed reality glasses and an intelligent robot; the sensing device includes one or more of a binocular camera, an RGB camera, a depth camera and a laser radar.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the multi-device collaborative visual-inertial joint mapping method as described in any one of claims 1 to 3 is implemented.

Images