A local SLAM construction method, a global SLAM construction method, and a construction device.

Abstract

This invention discloses a local SLAM construction method, a global SLAM construction method, and a construction device. In this embodiment, a cloud map and a real-time map are compared, and an object classification model is used to obtain the object classification of the current scene. Based on the object classification, prediction updates are performed, and a sub-map is obtained based on the prediction updates. The updated sub-maps obtained by the local SLAM construction method are then fused to obtain the global map. By classifying objects into dynamic and static objects, and predicting the motion of dynamic objects, a sub-map is obtained based on the prediction results, enabling real-time identification of dynamic objects.

Description

Technical Field

[0001] This article relates to, but is not limited to, SLAM construction techniques, particularly local and global SLAM construction methods, apparatus, and computer storage media. Background Technology

[0002] Currently, Simultaneous Localization and Mapping (SLAM) is a technology that enables mobile devices to automatically create maps in unknown environments and perform self-localization based on estimates of their own state and the map. With the rapid development of robotics, Virtual Reality (VR), and Augmented Reality (AR) in recent years, SLAM technology has received increasing attention. However, SLAM technology itself is still not very mature, facing technical challenges such as high algorithm complexity, difficulty in fusing input sensor data, and difficulty in eliminating accumulated system errors.

[0003] Especially in the VR field, the main problems with current SLAM mapping in VR scenes are: 1) In current VR applications, the user activity area is generally set manually by drawing circles to define the area. However, these methods of setting the activity area are static and cannot adapt well when the environment changes or other targets enter the activity area. 2) Currently, the SLAM mapping of each VR user is isolated, which cannot fully utilize the mapping information to build an effective multi-user collaborative VR scene. Summary of the Invention

[0004] The following is an overview of the subject matter described in detail herein. This overview is not intended to limit the scope of the claims.

[0005] This invention provides a local SLAM construction method, comprising: inputting a cloud map and a real-time map into a classification model in each cycle to obtain the object classification of the current scene; performing prediction updates based on the object classification; and obtaining a local sub-map based on the prediction updates.

[0006] Furthermore, before inputting the cloud map into the classification model in each cycle, the cloud map is parsed to extract the initial dynamic objects and the initial static objects.

[0007] Furthermore, the step of inputting the cloud map and real-time map into the classification model in each cycle to obtain the object classification of the current scene includes: comparing the positions of the initial dynamic objects and the initial static objects with the objects at the corresponding positions in the real-time map; and updating the classification of the dynamic objects and the static objects based on the comparison results to obtain the updated dynamic objects and the updated static objects.

[0008] Furthermore, the prediction update based on object classification includes predicting and updating dynamically changing objects.

[0009] Furthermore, the local SLAM construction method also includes: determining the updated active area based on the local submap.

[0010] This invention also provides a global SLAM construction method, which includes: fusing multiple local sub-maps obtained by the above method to obtain a global map.

[0011] Furthermore, the fusion includes: using the odometer and measurement information of the second terminal to locate the second terminal on a local sub-map of the first terminal, obtaining a transition matrix of the local sub-maps of the first and second terminals, and fusing the transition matrix to obtain a global map.

[0012] Furthermore, the positioning of the second terminal using the odometer and measurement information of the second terminal on a local sub-map of the first terminal includes positioning using an augmented Monte Carlo positioning method.

[0013] Furthermore, the transition matrix is ​​obtained by comparing the time stamp and variance of the pose information repositioned by the second terminal with the pose information when it originally constructed the sub-map, and iterating through the ICP algorithm.

[0014] In another aspect, embodiments of the present invention also provide a construction apparatus, comprising:

[0015] The input module is used to input cloud maps and real-time maps into the classification model in each cycle to obtain the object classification of the current scene;

[0016] The prediction module is used to update predictions based on object classification.

[0017] The map generation module is used to update and generate local sub-maps based on predictions.

[0018] Furthermore, the construction device also includes:

[0019] The fusion module is used to fuse multiple local sub-maps, including: using the odometer and measurement information of the second terminal to locate the second terminal on the local sub-map of the first terminal, obtaining the transition matrix of the local sub-maps of the first terminal and the second terminal, and fusing them according to the transition matrix to obtain a global map.

[0020] In another aspect, embodiments of the present invention also provide a computer storage medium storing a computer program, which, when executed by a processor, implements the local SLAM construction method and the global SLAM construction method as described above.

[0021] This invention, through its embodiments, inputs cloud-based and real-time maps into a classification model to obtain object classifications for the current scene. These object classifications are then predicted and updated to produce updated sub-maps, enabling real-time identification of dynamic objects and thus revealing real-time movable areas. Furthermore, the local maps are fused to obtain a global map, providing crucial infrastructure for positioning and security.

[0022] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the description, claims, and drawings. Attached Figure Description

[0023] The accompanying drawings are provided to further understand the technical solutions of the present invention and constitute a part of the specification. They are used together with the embodiments of this application to explain the technical solutions of the present invention and do not constitute a limitation on the technical solutions of the present invention.

[0024] Figure 1 This is a flowchart of a local SLAM construction method according to an embodiment of the present invention;

[0025] Figure 2 This is a flowchart of the global SLAM construction method according to an embodiment of the present invention;

[0026] Figure 3 This is a schematic diagram of the construction device according to an embodiment of the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be noted that, unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

[0028] The steps illustrated in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases the steps shown or described may be performed in a different order than that presented here.

[0029] First, it is described that both SLAM construction methods involved in the embodiments of the present invention include a terminal. The terminal can be a mobile terminal device such as a mobile phone, tablet computer, laptop computer, or video recorder, or a virtual scene terminal device (such as VR glasses, MR interactive devices, etc.). The terminal may include: a processor, a memory, a peripheral system, a positioning module, and a communication module. These components can communicate on one or more communication buses.

[0030] The peripheral system is mainly used to realize the terminal's display function and the interaction function between the terminal and the user / external environment. For example, it may include several components such as a display screen, camera, and sensor module. Each controller can be coupled to its corresponding peripheral device, such as the display screen, camera, and sensor. In some embodiments, the display screen is used to display images, which can be two-dimensional or three-dimensional images; for example, in VR, the display screen is a binocular stereo display. In some embodiments, the camera can be a monocular camera, a binocular camera, or a depth camera, used to acquire two-dimensional or three-dimensional images of the real scene. In some embodiments, the sensor may include an inertial measurement unit (IMU), and may also include other sensors (e.g., proximity sensors). It should be noted that the peripheral system may also include other I / O peripherals.

[0031] The memory, coupled to the processor, stores data (e.g., image data, point cloud data, map data, keyframe data, coordinate system transformation data, etc.), various software programs, and / or multiple sets of program instructions. The positioning module is used to physically locate the terminal, collecting its motion data. The positioning module may include one or more of the following: a Wi-Fi positioning module, a Bluetooth positioning module, a base station positioning module, or a satellite positioning module. The communication module integrates a transmitter and receiver for receiving and transmitting radio frequency signals to establish communication connections with other terminals or between the terminal and a cloud server.

[0032] like Figure 1 As shown, the local SLAM construction method of this invention includes the following steps:

[0033] S1. In each cycle, input the cloud map and real-time map into the classification model to obtain the object classification of the current scene.

[0034] Before inputting the cloud map into the classification model in each cycle, the cloud map is parsed to extract initial dynamic and static objects. The positions of the initial dynamic and static objects are compared with the corresponding objects in the real-time map; based on the comparison results, the classification of the dynamic and static objects is updated to obtain updated dynamic and updated static objects.

[0035] The SLAM cycle on the terminal is typically 30Hz. In each cycle, a map acquired from the cloud is input into the classification model. The cloud map is the updated sub-map from the previous cycle. The cloud map is parsed to extract initial dynamic and initial static objects. Except for the first cycle, both initial static and initial dynamic objects are obtained from the updated sub-map of the previous cycle. In the first cycle, all objects obtained from the real-time map are directly treated as static objects. The positions of the initial dynamic and initial static objects are compared with the corresponding objects in the real-time map. Specifically, when the terminal obtains the real-time map, it corresponds to a pose. The cloud map is obtained from the cloud according to this pose information. Since the poses of the cloud map and the real-time map are the same, the positions of their objects should be corresponding. The system compares whether there are still static objects at the positions of the initial static objects to determine whether the initial static object is an updated static object. If the position of the object at the position of the initial static object has changed, it is updated to an updated dynamic object. If the position of the object at the position of the initial static object has not changed, it is kept as an updated static object. Similarly, the system compares whether there are still dynamic objects at the positions of the initial dynamic objects to determine whether the initial dynamic object is an updated dynamic object. If the position of the object at the position of the initial dynamic object has not changed, it is updated to an updated static object. If the position of the object at the position of the initial dynamic object has changed, it is kept as an updated dynamic object.

[0036] The classification model can be obtained through pre-training using deep learning methods, such as the SegNet network structure. The core of SegNet consists of an encoder network, a decoder network, and a pixel-level classifier. The encoder network uses the first 13 layers of the VGG16 convolutional network structure. The decoder network maps the feature maps calculated by the encoder network from the original image from a low resolution to a resolution consistent with the original image size, achieving pixel-by-pixel classification and forming a pixel-level accurate semantic segmentation image. Pre-training on scene image data and training a binary classification network yields a classification model for dynamic object recognition in the scene. This model can classify all objects in the scene into types, such as walls, chairs, dogs, and people. After type classification, objects can be further classified as dynamic or static.

[0037] S2. Prediction updates are based on object classification.

[0038] After obtaining the updated dynamic object and the updated static object in step S1, the updated dynamic object is predicted and updated. Specifically, based on the position of the updated dynamic object on the cloud map in the previous period and its position on the real-time map, its position change parameters are obtained. These position change parameters can be motion values ​​such as velocity and acceleration in the three-dimensional direction. Based on these motion values, the position of the object is predicted and updated in the next period.

[0039] S3. Update the local sub-map based on the prediction.

[0040] The predicted updates yield a local submap, which includes the scene, updated static objects, and updated dynamic objects. This local submap is uploaded to the cloud and can serve as the cloud map for the next cycle of local SLAM construction. For the next cycle, the updated static and dynamic objects correspond to the initial static and dynamic objects resolved from the cloud map in that cycle.

[0041] The local SLAM construction method of this embodiment of the invention further includes:

[0042] S4. Determine the updated active area based on the local submap.

[0043] In the VR field, users may not be able to see the outside scene when wearing a VR display. Existing technology initially draws a movable area; as long as the user doesn't step outside this area, it's considered safe. However, if an obstacle suddenly enters the area, such as a dog, the movable area becomes unsafe. This invention's local sub-map allows for real-time updates to the movable area. When a dynamically moving object is predicted to enter the movable area, the sub-map can be adjusted to delineate the object's location as an inactive area. This adjustment can be performed on the terminal or when the sub-map is uploaded to the cloud. The real-time updated sub-map allows for continuous monitoring of the movable area's safety. When a local area becomes unsafe, it is removed from the movable area, ensuring the user is always in a safe environment.

[0044] In this embodiment of the invention, cloud maps and real-time maps are input into a classification model to obtain the object classification of the current scene. The object classification is then predicted and updated to obtain an updated sub-map, which can distinguish dynamic objects in real time and thus obtain real-time active areas.

[0045] Figure 2 This is a flowchart of a global SLAM construction method according to an embodiment of the present invention. The global SLAM construction method will combine multiple... Figure 1 The local sub-maps obtained by the method are merged to obtain a global map, specifically as follows: Figure 2As shown, it includes the following steps:

[0046] S11. The second terminal is located on a local sub-map of the first terminal using the odometer and measurement information of the second terminal.

[0047] In specific implementation methods, the first terminal can be a mobile terminal device such as a mobile phone, tablet computer, laptop computer, or video recorder; it can also be a virtual scene terminal device (such as VR glasses, MR interactive devices, etc.); or it can be a robot. The second terminal can be the same type of terminal as the first terminal, or it can be a different type of terminal.

[0048] The first and second terminals can move within the same real-world environment, which can be any real-world environment. The first and second terminals each run a virtual scene application (VR, AR, or MR application), and can communicate with each other. The first and second terminals can acquire information about any objects in the environment through cameras and / or sensors to complete SLAM image acquisition. The first and second terminals also include sensors such as IMUs, which can be used to obtain real-time attitude data.

[0049] When the cloud acquires two sub-maps with the same timestamp, corresponding to the updated sub-maps of the poses of the first and second terminals respectively, these two sub-maps are merged in the cloud to obtain the global map. The second terminal's odometry and measurement information are used to locate itself on the sub-map constructed by the first terminal using the Monte Carlo localization method. The augmented Monte Carlo localization method, given a sufficient number of particles, can approximate almost any probability distribution in reality. A strategy for increasing the random particle count is designed, determined by the degree of discrepancy between long-term and short-term observations. When short-term and long-term observations are consistent, no random particle increase is made; when short-term and long-term observations are inconsistent, the particle count is increased until the condition is met.

[0050]

[0051] N is the number of dynamically selected particles, and Nb is the number of bins in the discrete probability distribution.

[0052] S22. Obtain the transition matrix of the local sub-maps of the first terminal and the second terminal.

[0053] By comparing the pose information re-localized by the second terminal with the time stamp and variance of its original pose information when constructing the sub-map, the transition matrix of the two maps can be obtained. A more accurate transition matrix can then be obtained through iteration.

[0054] The iteration employs the ICP algorithm, which matches two partially overlapping point clouds by minimizing the distance between them. ICP mainly consists of two parts: calculating corresponding points in two point clouds and finding the transition matrix that minimizes the distance between these corresponding points. The formula for transferring points from the sub-map obtained by the first terminal to points in the coordinate system of the second terminal's sub-map is as follows:

[0055]

[0056] S33. Perform fusion based on the transition matrix to obtain the global map.

[0057] The sub-maps obtained by the first terminal and the second terminal can be merged in a single coordinate system using a transition matrix to form a global sub-map. When there are multiple terminals (more than 2), the location result of each terminal in the sub-maps of other terminals at that moment can be obtained by intersecting the sub-maps of each pair of terminals. The coordinate system can then be transformed using a transition matrix to obtain a global sub-map for multiple terminals.

[0058] This invention employs the augmented Monte Carlo method to obtain the localization results of each terminal in the sub-maps of other terminals at a given time. The pose covariance matrices of each terminal at the same time are compared across multiple sub-maps to determine the best-matching pose. The best-matching pose is then used to calculate the transition matrix between sub-maps. This transition matrix is ​​used as the initial transition matrix for the ICP algorithm for iterative fusion, ultimately yielding a global map. This global map is stored in the cloud and used as the cloud map during the next cycle when each terminal performs local SLAM localization.

[0059] The global SLAM construction method of this embodiment of the invention further includes:

[0060] S44. Determine and update the active area based on the global submap.

[0061] The merged panoramic map includes all dynamic and static objects in the real-world environment obtained from each terminal. By optimizing the transition matrix between sub-maps, a global map is obtained, and the active area is then updated based on the global map. In the VR field, users wearing VR displays may not be able to see the outside scene. When updating the active area using step S4, the positions of dynamic objects outside that sub-map may not be predictable on the local sub-map. However, when the global sub-map is established, the predicted positions of dynamic objects on the local sub-maps of each terminal are integrated, thus allowing the active area to be divided globally based on the updated positions of dynamic objects. The global map provides crucial infrastructure for positioning and security.

[0062] Figure 3 This is a structural block diagram of the construction device according to an embodiment of the present invention, such as... Figure 3The construction apparatus shown in this embodiment of the invention includes:

[0063] The input module 101 is used to input the cloud map and the real-time map into the classification model in each cycle to obtain the object classification of the current scene;

[0064] Before inputting the cloud map into the classification model in each cycle, the cloud map is parsed to extract initial dynamic and static objects. The positions of the initial dynamic and static objects are compared with the corresponding objects in the real-time map; based on the comparison results, the classification of the dynamic and static objects is updated to obtain updated dynamic and updated static objects.

[0065] Prediction module 102 is used for prediction updates based on object classification;

[0066] After obtaining the updated dynamic object and the updated static object from the input module, the updated dynamic object is predicted and updated. Specifically, based on the position of the updated dynamic object on the cloud map in the previous period and its position on the real-time map, its position change parameters are obtained. These position change parameters can be motion values ​​such as velocity and acceleration in the three-dimensional direction. Based on these motion values, the position of the object is predicted and updated for the next period.

[0067] Map generation module 103 is used to update and generate local sub-maps based on predictions.

[0068] The predicted updates yield a local submap, which includes the scene, updated static objects, and updated dynamic objects. This local submap is then uploaded to the cloud and can serve as the cloud map for the next cycle of local SLAM construction.

[0069] The fusion module 104 is used to fuse multiple local sub-maps, including: locating the second terminal on the local sub-map of the first terminal using the odometer and measurement information of the second terminal; obtaining the transition matrix of the local sub-maps of the first terminal and the second terminal; and fusing the transition matrix to obtain a global map.

[0070] This invention also provides a computer storage medium storing a computer program, which, when executed by a processor, implements the above-described functionality. Figure 1 and Figure 2 The local SLAM construction method and the global SLAM construction method are shown.

[0071] "Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all components may be implemented as software executed by a processor, such as a digital signal processor or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is commonly understood in the art..." As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

Claims

1. A local SLAM construction method, characterized in that, include: In each cycle, the cloud map and the real-time map are input into the classification model to obtain the object classification of the current scene, including: comparing the positions of the initial dynamic objects and the initial static objects with the corresponding objects in the real-time map; updating the classification of the dynamic objects and the static objects based on the comparison results to obtain the updated dynamic objects and the updated static objects; wherein, the initial dynamic objects and the initial static objects are obtained by parsing the cloud map. Prediction updates are based on object classification. The local sub-map is updated based on the prediction and uploaded to the cloud as the cloud map for the next cycle of local SLAM construction. The prediction update based on object classification includes: predicting and updating the position of the dynamic object to be updated in the next cycle.

2. The local SLAM construction method according to claim 1, characterized in that, Also includes: The updated active area is determined based on the local submap.

3. A global SLAM construction method, comprising: Multiple local sub-maps obtained by the method of claim 1 are fused to obtain a global map.

4. The global SLAM construction method according to claim 3, characterized in that, The fusion process includes: using the odometer and measurement information of the second terminal to locate the second terminal on a local sub-map of the first terminal; obtaining a transition matrix between the local sub-maps of the first and second terminals; and fusing the transition matrix to obtain a global map.

5. The global SLAM construction method according to claim 4, characterized in that, The method of locating the second terminal on a local sub-map of the first terminal using the odometer and measurement information of the second terminal includes locating the second terminal using an augmented Monte Carlo positioning method.

6. The global SLAM construction method according to claim 4, characterized in that, The transition matrix is ​​obtained by comparing the pose information repositioned by the second terminal with the time stamp and variance of the pose information when it originally constructed the sub-map, and iterating using the ICP algorithm.

7. A construction apparatus, characterized in that, include: The input module is used to input the cloud map and the real-time map into the classification model in each cycle to obtain the object classification of the current scene, including: comparing the positions of the initial dynamic objects and the initial static objects with the corresponding objects in the real-time map; updating the classification of the dynamic objects and the static objects based on the comparison results to obtain the updated dynamic objects and the updated static objects; wherein, the initial dynamic objects and the initial static objects are obtained by parsing the cloud map. The prediction module is used to update predictions based on object classification. The map generation module is used to update and generate local sub-maps based on predictions and upload the obtained local sub-maps to the cloud as the cloud map for the next cycle of local SLAM construction. The prediction update based on object classification includes: predicting and updating the position of the dynamic object to be updated in the next cycle.

8. The construction apparatus according to claim 7, characterized in that, Also includes: The fusion module is used to fuse multiple local sub-maps, including: using the odometer and measurement information of the second terminal to locate the second terminal on the local sub-map of the first terminal; Obtain the transition matrices of the local sub-maps of the first and second terminals; fuse them according to the transition matrices to obtain the global map.

9. A computer storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the SLAM construction method as described in any one of claims 1 to 2 or 3 to 6.

Images