A distributed slam system and learning method thereof

Abstract

The present application relates to the technical field of artificial intelligence robots, in particular to a distributed SLAM system and a learning method thereof; the present application comprises a distributed robot group for receiving sensor data of each robot, extracting features, thereby constructing a pose graph and performing optimization and loop detection of the pose graph, and outputting bag-of-words data and local maps of each scene; and a cloud workstation for receiving the bag-of-words of the images and local maps transmitted by the distributed robots, training a dynamic dictionary, then performing scene recognition and matching, and fusing the local maps in the cloud and outputting a global pose graph and a global map; the present application uses multiple robots and a central cloud server to achieve the construction of a global map of an unknown environment, improves the mapping efficiency and robustness in a large-scale scene, and ensures the privacy and real-time data transmission.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence robot technology, and particularly relates to a distributed SLAM system and a learning method thereof. BACKGROUND

[0002] The artificial intelligence era is coming, and AI is gradually considered by people as the next "Internet" class disruptive technology, and application scenarios include intelligent medical care, intelligent finance, intelligent security, intelligent home, intelligent marketing, intelligent driving, e-commerce retail, personal assistants, industrial robots, server robots, wearable devices, etc.

[0003] The key technologies of artificial intelligence mainly include machine learning, computer vision, speech and natural language processing. At present, the research mainly carries out the key technology research of artificial intelligence, and realizes commercialization based on the achievements. Artificial intelligence applications gradually penetrate into various fields. In the field of artificial intelligence, natural language processing is a major sensing technology, and its application range is very wide, and it can be used in intelligent home, wearable devices, intelligent cars, intelligent education, intelligent finance and other fields.

[0004] SLAM is the abbreviation of Simultaneous Localization And Mapping, which was first proposed by Hugh Durrant-Whyte and John J. Leonard. The problem can be described as follows: a robot is placed in an unknown environment at an unknown position. Is there a way to let the robot move while gradually mapping the complete map of this environment? A complete map (a consistent map) means that it can enter every corner of the room without obstacles.

[0005] The existing technology usually uses a single sensor or robot device to build a map. In a large-scale scene, it takes a long time to build a map, which seriously affects the deployment and practical application value of the SLAM system. SUMMARY

[0006] The technical problem solved by the present application is to provide a distributed SLAM system and a learning method thereof. Multiple robots and a central cloud server are used to achieve the task of building a global map of an unknown environment, which improves the mapping efficiency and robustness in a large-scale scene while ensuring privacy and real-time data transmission. A distributed SLAM learning method is also provided.

[0007] To solve the above technical problems, one technical solution adopted by the present application is to provide a distributed SLAM system, which comprises:

[0008] A distributed robot group is composed of several robots, which receives sensor data of each robot, extracts features, constructs a pose graph, and performs optimization and loop closure detection of the pose graph, and outputs bag-of-words data and local maps of each scene.

[0009] A cloud workstation, as a central cloud server, receives the bag-of-words of the image and local map transmitted by the distributed robot, trains a dynamic dictionary, performs scene recognition and matching, fuses the local map in the cloud, and outputs a global pose graph and a global map.

[0010] As an improvement of the present application, the robot uses a laser radar and / or a monocular camera for perception.

[0011] As a further improvement of the present application, the distributed robot group extracts features from visual images.

[0012] As a further improvement of the present application, the distributed robot group uses point clouds to construct local maps, and the cloud workstation merges the local maps using position matching relationships.

[0013] A learning method of distributed SLAM, comprising the following steps:

[0014] Step S1, the distributed robot group receives sensor data of each robot, extracts features from the sensor data, and constructs a pose graph;

[0015] Step S2, the pose graph is optimized, a local map is constructed, loop closure detection is performed, and bag-of-words data and local maps of each scene are outputted;

[0016] Step S3, after receiving the image and the bag-of-words of the local map transmitted by the distributed robot, a dynamic dictionary is trained, scene recognition and matching are performed, and the local maps are fused in the cloud;

[0017] Step S4, the parameters are encrypted in the cloud, and then the encrypted parameter updates are transmitted to the cloud parameter server;

[0018] Step S5, the cloud parameter server decrypts the parameter updates and updates the shared model, thereby outputting a global pose graph and a global map.

[0019] As an improvement of the present application, in step S1, the robot obtains single-frame scanning data to form sensor data by laser scanning.

[0020] As a further improvement of the present application, in step S2, the local map is constructed by accumulating scanning.

[0021] As a further improvement of the present application, in step S2, the constraint relationship between the scan and the local map is generated by using pixel-accurate scan matching method, and then the loop closure detection is used to eliminate the accumulated error generated by the local map.

[0022] As a further improvement of the present application, the distributed robot group uses a feature extractor to extract features, and the feature extractor is an encoder composed of a convolutional layer, a pooling layer and an activation function.

[0023] As a further improvement of the present application, the cloud workstation trains the dynamic dictionary according to the real-time data transmitted by the distributed robot group.

[0024] The beneficial effects of the present application are: compared with the prior art, the present application uses multiple robots and a central cloud server to achieve the task of building a global map of an unknown environment, which improves the mapping efficiency and robustness in a large-scale scene while ensuring privacy and real-time data transmission. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 The step block diagram of the distributed SLAM learning method of the present application;

[0026] Figure 2 The overall architecture diagram of the distributed SLAM system of the present application;

[0027] Figure 3 The flow block diagram of the distributed SLAM learning method of the present application;

[0028] Figure 4 The feature extraction network architecture diagram of the present application;

[0029] Figure 5 The encryption and sharing flowchart of the distributed SLAM learning method of the present application;

[0030] Figure 6 The schematic diagram of the dynamic dictionary algorithm of the present application. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application.

[0032] Laser-based SLAM research is earlier, and so far there have been many mature solutions, such as Gmapping based on particle filtering method relying on single-line laser radar, Karto based on graph optimization method, LOAM based on multi-line laser radar and Cartographer based on nonlinear optimization, etc.

[0033] Visual-based SLAM can be divided into two branches of direct method and feature point method, the direct method calculates the camera motion and constructs the surrounding environment map according to the gray information of all pixels, and the recent researches include sparse method PTAM, SVO, semi-dense method LSD-SLAM and dense method DTAM, etc. The feature point method extracts features from the image and matches them first, and then estimates the camera motion and constructs the map according to the matching relationship, and the relatively typical work includes ORB-SLAM, etc. The feature point method is poor in the case of no obvious texture (feature missing), while the direct method is easily affected by light and has a large amount of calculation.

[0034] Deep learning enhanced semantic SLAM has achieved extensive research, some researches are committed to building an end-to-end SLAM system with a deep neural network, such as estimating the monocular camera pose by a deep learning model; and some other researches add semantic information to the SLAM system, optimize the solution of the camera pose by using semantic segmentation information, and establish a semantic map, etc.

[0035] As shown in Figures 1 to 6 The present application provides a kind of distributed SLAM system, comprising:

[0036] Distributed robot group, which is composed of several robots, is used to receive the sensor data of each robot, then extract features, thereby constructing pose graph and re-executing the optimization and loop detection of pose graph, and then outputting the bag-of-words data and local map of each scene;

[0037] Cloud workstation, as a central cloud server, receives the bag-of-words of image and local map transmitted by the distributed robot, then trains a dynamic dictionary, and then performs scene recognition and matching, and finally fuses the local map in the cloud and outputs the global pose graph and global map.

[0038] Wherein, the robot uses laser radar and / or monocular camera for perception.

[0039] Further, the distributed robot group extracts features from visual images.

[0040] In the present application, the distributed robot group uses point cloud to construct local map, and the cloud workstation merges the local map by position matching relationship.

[0041] Specifically, as Figure 2As shown, the distributed SLAM system of the present invention can be divided into two modules: a distributed robot group and a cloud workstation. (1) The front end of the distributed robot group is responsible for receiving sensor data, extracting features, and constructing a pose graph. Its back end performs graph optimization and loop closure detection, and then outputs bag-of-words data and local maps for each scene. (2) In the cloud workstation, after receiving the bag-of-words data of the images and local maps from the distributed robot group, a dynamic dictionary is trained, and then scene recognition and matching are performed. Finally, the local maps are fused in the cloud, and the global pose graph and global map are output. Each uses LiDAR and a monocular camera for perception. The distributed robot group mainly uses point clouds to construct local maps. It uses visual images to extract features and matches images in the same scene. Finally, the cloud workstation uses position matching relationships to merge the local maps.

[0042] The distributed SLAM system of this invention uses Karto SLAM as the basic algorithm for robot map construction. First, it acquires single-frame scan data through laser scanning, then constructs a local map through cumulative scanning, then generates the constraint relationship between the scan and the local map using a pixel-precise scan matching method, and finally uses loop closure detection to eliminate the cumulative error generated by the local map.

[0043] like Figure 3 As shown, the distributed SLAM system of the present invention adopts a federated learning architecture, maintaining private models and images on the robot side, while only sending bag-of-words data to the cloud parameter server on the cloud workstation; specifically, after a parameter update is generated in a certain private model, the parameters are encrypted using an encryption algorithm, and then the encrypted parameter update is transmitted to the cloud parameter server. The server first decrypts the parameter update, and then determines whether to aggregate or directly update the shared model based on the number of parameters.

[0044] like Figure 1 As shown, this invention provides a learning method for distributed SLAM, comprising the following steps:

[0045] Step S1: The distributed robot group receives sensor data from each robot, extracts features from the sensor data, and constructs a pose graph.

[0046] Step S2: Optimize the pose graph, construct a local map, perform loop closure detection, and then output the bag-of-words data and local map for each scene;

[0047] Step S3: After receiving the bag-of-words of the image and local map transmitted by the distributed robot, a dynamic dictionary is trained, then scene recognition and matching are performed, and finally the local map is fused in the cloud.

[0048] Step S4: Encrypt the parameters in the cloud, and then update and send the encrypted parameters to the cloud parameter server;

[0049] Step S5, the cloud parameter server decrypts the parameter update and updates the shared model, thereby outputting a global pose graph and a global map.

[0050] In step S1, the robot acquires single-frame scanning data to form sensor data through laser scanning.

[0051] In step S2, a local map is constructed through accumulated scanning, a constraint relationship between scanning and the local map is generated by using a pixel-accurate scanning matching method, and accumulated errors generated by the local map are eliminated by using loop detection.

[0052] In the present application, the distributed robot group uses a feature extractor to extract features, and the feature extractor is an encoder composed of a convolution layer, a pooling layer and an activation function; the cloud workstation trains a dynamic dictionary according to real-time data transmitted by the distributed robot group.

[0053] Specifically, the present application is based on a federal learning architecture, private models and images are maintained at the robot end, and only bag-of-words data is sent to the cloud parameter server of the cloud workstation, as shown in Figure 5 After a parameter update of a certain private model is generated, the parameter is encrypted by an encryption algorithm, and then the encrypted parameter update is transmitted to the cloud parameter server, the server first decrypts the parameter update, and then determines whether aggregation is needed or direct update of the shared model is needed according to the number of parameters, which includes a cloud parameter server fusion method, a private model parameter update algorithm and a shared model parameter update method, and the details are as follows:

[0054] 1. Cloud parameter server fusion algorithm:

[0055] Formula 1:

[0056]

[0057] Formula 2:

[0058]

[0059] Wherein, f j is the loss function of the model at node j, P j is the data at node j, P = |P j is the union set of all data, || represents the size of the data, W j is the model parameter at node j, and W is the model parameter at the parameter server.

[0060] 2. Private model parameter update algorithm:

[0061] Formula 3:

[0062]

[0063] Where α is the learning rate. For node j, the parameters are those after receiving the global update; if no global update has occurred, then... t represents the current time, and t-1 represents the previous time. The gradient symbol is used here. It is worth noting that this method is not limited to gradient descent. Due to the communication bottleneck, more efficient iterative algorithms, such as Newton's method and momentum-based methods, will be used.

[0064] 3. Shared model parameter update algorithm:

[0065] Formula 4:

[0066]

[0067] Formula 5:

[0068]

[0069] The feature extractor of this invention adopts an architecture similar to VGG, such as... Figure 4 As shown, the network shares an encoder consisting of convolutional layers, pooling layers, and activation functions, and then splits into two networks to extract feature points and descriptors respectively. Due to computational reasons, the feature point extraction network does not use the classic encoder-decoder structure, but instead uses a sub-pixel convolution method to achieve upsampling. The feature extraction network outputs a probability map of the same size as the original image. After extracting the descriptors, in order to have the same format as the 0RB descriptors and facilitate word packet calculation, a binary activation layer is added to convert the descriptors into binary format.

[0070] like Figure 6 As shown, a closed-loop basic method implemented using DBoW is used for scene matching. A dictionary is constructed by calculating the bag-of-words model of the scene, and the similarity between positions is calculated. The dictionary can be represented as a set of words, and the scene observed at time k is represented as Z. k ={z1, z2...z i ...z N}, where z i Let x represent the product of the term frequency (TF) and inverse document frequency (IDF) of the i-th word. The similarity between points x and y can be described by Formula 6:

[0071]

[0072] Wherein, the performance of the dictionary method can be affected by the number and environmental sources of the training images (such as indoor and outdoor), which leads to instability of scene recognition and matching; in contrast to the traditional method, the computing resources of the cloud workstation are fully utilized, and a dynamic dictionary is maintained on the cloud, therefore, the dictionary is trained by real-time data transmitted by the distributed robot group to update the dictionary in different environments, in order to ensure the efficiency of logarithmic search, the vocabulary is constructed by using the k-means tree method, and the specific process of the dynamic dictionary algorithm is as shown in Figure 6

[0073] As shown in Figure 5 After a certain private model generates a parameter update, the parameters are encrypted by an encryption algorithm, and then the encrypted parameter update is transmitted to the cloud parameter server, which first decrypts the parameter update, and then judges whether aggregation is needed or the shared model is directly updated according to the number of parameters; at the same time, a staleness threshold S is set, when the update times of a certain private model exceed the average level S times, the update of the private model is suspended, and the purpose of this step is to prevent the quality of the shared model update from being reduced due to the excessive frequency of the update of a certain private model.

[0074] The present application has the following advantages:

[0075] (1) Since multiple machines participate in mapping, the mapping efficiency is significantly improved;

[0076] (2) Without uploading the original data of the user, and adopting the parameter encryption strategy, the privacy of the user is effectively ensured;

[0077] (3) The real-time performance of data transmission is ensured;

[0078] (4) The accuracy of cloud scene matching is improved.

[0079] The above only describes the embodiments of the present application, and does not limit the patent scope of the present application, any equivalent structure or equivalent process transformation according to the content of the specification and drawings of the present application, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.​

Claims

1. A distributed SLAM system, characterized by, The application relates to a distributed robot group, a cloud server and a dynamic dictionary. The distributed robot group receives sensor data of each robot, extracts features, constructs a pose graph, performs optimization and loop detection of the pose graph, and outputs bag-of-words data and a local map of each scene. The cloud server receives the bag-of-words data and the local map of the image transmitted by the distributed robot group, trains a dynamic dictionary, performs scene recognition and matching, fuses the local map in the cloud, and outputs a global pose graph and a global map. where the dynamic dictionary is represented as a set of words, and the observed scene at the kth time instant is represented as Z k = {z1, z2...z i ...z N}, where z i represents the product of the word frequency and inverse document frequency of the ith word, and the similarity between the x-point and the y-point is described as Equation 6: where N is the total number of words in the dictionary at a certain time; denotes the product of the word frequency and inverse document frequency of the i-th word at the x-point; denotes the product of the word frequency and inverse document frequency of the i-th word at the y-point; The cloud server maintains a dynamic dictionary in the cloud by using the computing resources of the cloud server and training the dynamic dictionary by using real-time data transmitted by the distributed robot group.

2. The distributed SLAM system of claim 1, wherein, The robot uses a laser radar and / or a monocular camera for perception.

3. The distributed SLAM system of claim 2, wherein, The distributed robot group extracts features by using visual images.

4. The distributed SLAM system of claim 3, wherein, The distributed robot group uses a point cloud to construct a local map, and the cloud server merges the local map by using a position matching relationship.

Images