An edge-assisted crowd-sourced distributed visual slam system and method

Abstract

The application relates to an edge-assisted crowdsourcing distributed visual SLAM system and method, which comprises a user end and a server end; the user end comprises a user pose estimation module, a key frame data reduction module and a communication module; the server end comprises a mapping module, a redundant information detection and control module, a multi-map merging module and a communication module; the application can effectively reduce the uploading of redundant information of crowdsourcing users in the multi-user mapping process and reduce resource consumption.

Description

TECHNICAL FIELD

[0001] The application relates to an edge-assisted crowd-sourced distributed visual SLAM system and method, and belongs to the technical fields of unmanned systems, multi-user systems and visual SLAM. BACKGROUND

[0002] In recent years, visual SLAM (simultaneous localization and mapping) systems have shown great potential in various fields. For example, when GPS signals are invalid (such as indoors, near high-rise buildings, etc.), the current most popular visual SLAM systems, such as ORB-SLAM, ORB-SLAM2 and ORB-SLAM3, can complete the mapping task on a single device. Recently, many edge-unloading-based visual SLAM (simultaneous localization and mapping) systems have emerged. Edge-SLAM offloads the part with large computing requirements to the server, only retains the necessary and real-time required part on the user side, and at the same time, guarantees the accuracy of the SLAM system as much as possible. Covins collects all the map data of the collaborative users on the server to merge them into a large global map. CCM-SLAM offloads the computationally intensive module of monocular visual SLAM and simultaneously realizes multi-user mapping.

[0003] However, the above methods do not solve the problem of redundant information between data when the user moves. Although there are systems that use path planning to avoid information redundancy when multiple users establish maps, considering the actual scene, users cannot only complete the single task of mapping, and efficient transmission and processing of redundant information can greatly reduce network transmission, computing and data storage. SUMMARY

[0004] The application solves the problem of overcoming the shortcomings of the prior art and providing an edge-assisted crowd-sourced distributed visual SLAM system and method, which can effectively reduce the uploading of redundant information of crowd-sourced users in the multi-user mapping process and reduce resource consumption.

[0005] The technical solution of the application is an edge-assisted crowd-sourced distributed visual SLAM system, which comprises a user end and a server end. The user end comprises a user pose estimation module, a key frame data reduction module and a communication module. The server end comprises a mapping module, a redundant information detection and control module, a multi-map merging module and a communication module.

[0006] The user pose estimation module estimates the user's current pose, i.e., obtains the user's current position and orientation, and determines keyframes. Using visual sensor and IMU input, it takes the previous frame as the initial pose of the current frame, further optimizes it based on the matching results of feature points in the two frames to obtain the user's current pose, and determines the keyframes. A stack is used to store the user's current pose and keyframes. The user's pose is sent to the server via the communication module, waiting for the redundancy information detection and control module on the server to judge and provide feedback control information.

[0007] The keyframe data reduction module is used to further process the original keyframes on the user side. After the user pose estimation module sends the pose to the server, the server feeds back control information. The user side uses the communication module to receive the feedback and uses the feedback information, combined with KD tree acceleration, to reduce the data in the original keyframes to obtain the keyframe part that can provide new data to the server. The user side then sends the keyframe to the server through the communication module.

[0008] The user communication module is used to send data from the user end and receive data from the server end. It is written in ZeroMQ and is mainly used to send user poses, receive server feedback, and send user data after removing redundancy.

[0009] The mapping module is used to combine keyframes uploaded by users into a map. The map includes all the information of each keyframe uploaded by each independent user. Each user will independently create a map on the server side, which is called a "mini-map" for distinction. Each keyframe uploaded by the user is inserted into the mini-map, redundant keyframes and 3D mappings of map points (i.e., feature points) are removed, and optimization is performed to obtain the user's mini-map.

[0010] The redundancy information detection and control module is used on the server side to detect the redundancy of the keyframes that the user is about to upload based on the poses that the user has already uploaded, and to provide feedback to the user. Based on the poses uploaded by the user, the module simulates the user's viewpoint, that is, the range observed by the user in the keyframe. By establishing a data structure of view frustum, the module uses the regional coverage of the view frustum sampling points to obtain the redundancy judgment. In addition to the necessary "do not transmit", "fully transmit", and "partially transmit" instructions, if it is "partially transmit", the module sends the detected duplicate sampling points to the user communication module in the form of a point set. The keyframe data deletion module on the user side uploads the keyframes based on the feedback information.

[0011] The data structure of the visual cone is as follows: using the CGAL library, based on the visual sensor information, a three-dimensional polyhedron is built with each pose uploaded by the user as the vertex direction. The three-dimensional polyhedron is used as the sampling point area, and uniform sampling is performed. The overlapping range is calculated according to the ratio.

[0012] A multi-map merging module is used to realize multi-user mapping, the module realizes merging of multiple user sub-maps from the mapping module to obtain a map of a complete mapping area; the same position is identified through description information of key frames, the sub-maps are identified and merged together by using the position, and repeated data of the position is removed, and in this way, a merged map of all users is obtained; finally, a complete map of the entire mapping area is obtained, and the complete map is stored on the server side;

[0013] A server communication module is used to accept data of a user side and send data of a server side: receiving a pose uploaded by the user side, sending control information of the server side, and receiving a key frame uploaded by the user side.

[0014] The edge auxiliary-based crowd-sourcing distributed visual SLAM method of the application comprises the following steps:

[0015] Step 1: taking data collected by a visual sensor and an IMU as input, extracting feature points in a picture output by the visual sensor, and calculating a pose of a current frame, i.e. a pose of the sensor; meanwhile, selecting ordinary frames that can summarize some information of ordinary frames from captured frames as key frames, sending a pose of the key frames to a server, and retaining all information of the key frames on a user device;

[0016] Step 2: establishing a view cone on the server by using a pose uploaded by a user, simulating a visual angle range of the user, determining coincidence of a latest view cone and all view cones, determining a coincidence range by using a sample point set, and sending information of the coincident sample points to the user side as control information; establishing a view cone on the server by using a pose uploaded by a user and camera parameters, using the view cone to simulate a visual angle range of the user at the pose, determining coincidence of a newly established view cone and all view cones established by all poses of all users on the server, i.e. uniformly designing sample points in the view cone, if sample points of two view cones are within a certain range, it is judged that the sample points coincide, the number of the coincident sample points is used to determine a coincidence range of the two view cones, and a command of "no transmission", "complete transmission" and "partial transmission" is given, if it is "partial transmission", information of the coincident sample points, i.e. three-dimensional positions of each three-dimensional point, is additionally sent to the user side as control information;

[0017] Step 3: the user end receives the control information from the server end, performs key frame data reduction on the control information, judges whether to transmit and process the key frame, and sends the transmission and processing key frame to the server; the user end accepts the control information transmitted from the server end, finds the key frame to which the control information belongs in the temporary stack, and judges whether to transmit the key frame, if it is "partial transmission", the data of the key frame is reduced, mainly including finding a map point, i.e. a feature point, which is close enough to the sampling point in the control information, deleting the information related to the map point in the key frame, and sending the transmission and processing key frame to the server through the communication module; if it is "complete transmission", the whole necessary information of the key frame is sent to the server through the communication module; if it is "no transmission", the key frame is discarded;

[0018] Step 4: the server end receives the key frame uploaded by the user, combines the key frame uploaded by the user into a map, the map includes all information of each key frame uploaded by an independent user, and each user will independently establish a map on the server end, in order to distinguish, it is called "small map"; each key frame uploaded by the user is inserted into the small map, redundant key frames and map points, i.e. three-dimensional mapping of feature points, are removed, and optimization is performed, to obtain the small map of the user;

[0019] Step 5: when a new user joins, a new small map is established for the user, when the new user and the previous user map have an overlap, a merged map is obtained through multi-map merging, so as to complete the SLAM task and obtain the complete merged map; when a new user joins, a new small map is established for the user on the server end, and the key frame uploaded by the user is stored in the corresponding small map, which is isolated from other users, when the new user and the previous user map have an overlap, i.e. pass through the same area and are detected, a merged map is obtained through multi-map merging, and the same is true for each user, so as to complete the SLAM task and obtain the complete merged map containing the key frames uploaded by multiple users.

[0020] Compared with the prior art, the advantages of the present application are that:

[0021] (1) The user pose estimation module in the present application can calculate the user pose in time and judge and screen out suitable key frames for redundancy detection, and then send the current user pose to the server through the communication module. A large amount of time and resource optimization is carried out in other threads, and at the same time, the invalid information uploaded by the user in large quantities is blocked by temporarily saving the key frame in the stack, and at the same time, the length of the pose is sent to the server in advance through the communication module using the ZeroMQ technology, so as to ensure the performance of the redundancy detection module.

[0022] (2) Redundant information detection and control module and key frame data reduction module, on the basis of realizing multi-user crowdsourcing mapping, unnecessary resource utilization is reduced by reducing redundant information. In the process of multi-user crowdsourcing mapping, the server can more efficiently establish a map by merging the large maps of multiple users. Multiple users can send maps to the server at the same time, and the server will merge them. However, the redundant information must be considered. Although repeated mapping of the same place can produce more accurate mapping results in the optimization process, considering the multi-user crowdsourcing form, a large number of users repeatedly map the same place, generating a large amount of repeated information, which is more harmful than beneficial for the long-term operation of the entire system. The application can effectively detect redundant information, and the information value brought by redundant information is not high, so eliminating it can greatly reduce the resource consumption of the server. This module is the main innovation point of the application. The purpose of the prior art is to merge the maps of multiple users, and it does not care about redundant information. However, in the process of gradually expanding the system size, there will be a lot of redundant information uploaded by multiple users. Therefore, compared with the existing system, reducing the upload of redundant information can greatly reduce the resource occupation of the entire system. The user pose estimation module is used to accurately determine whether the current key frame information of the user needs to be uploaded, the data structure of the view cone is established to solve the problem of redundant judgment, and the results are fed back to the user in a timely and efficient manner. The elimination of redundant information can greatly reduce the resource consumption (computing resources, storage resources and network resources) of the server. This step is the basis of the system function and lays the foundation for system cooperation.

[0023] (3) The existing multi-user mapping technology only accepts all the information uploaded by the user, establishes a map, and cannot effectively detect and remove redundancy. After removing the redundant information between multiple users through the above steps, a visual SLAM system that consumes a small amount of resources and quickly completes the specified area map construction can be established. In an extreme case (in a large scene, multiple users, long-term deployment), such as hundreds of users (N) exploring the same road, the traffic can be reduced to At the same time, since the process of visual SLAM map establishment at the user end is complete, the accuracy is also guaranteed. The application is verified on simulation data (the ground truth of the map point cloud can be obtained), and the map coverage and map error are verified, both of which only have slight changes and are within an acceptable range. On the EuRoC dataset, after verification, the application can also reduce a large amount of traffic. Therefore, under the condition of tolerable accuracy reduction, the application greatly reduces the utilization of resources.

[0024] (4) Thanks to the matching technology, the application can simultaneously determine the loop, i.e. the user returns to the position previously arrived at, to reduce the system error. Compared with the prior art, due to the introduction of the redundant information detection module, the control module and the key frame data reduction module, some key frames do not have enough map points to detect loop and map merging. The application marks these special key frames processed to avoid system error and error. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 For an edge-assisted crowd-sourced distributed visual SLAM system framework;

[0026] Figure 2 For a view cone diagram, the left drawing is a three-dimensional view cone diagram, and the right drawing is a view cone overlap diagram. DETAILED DESCRIPTION

[0027] The application will be described in detail below in combination with the drawings and embodiments.

[0028] As Figure 1 shown, a user end of an edge-assisted crowd-sourced distributed visual SLAM system based on the application mainly includes a user pose estimation module, a key frame data reduction module and a communication module; and a server end mainly includes a mapping module, a redundant information detection and control module, a multi-map merging module and a communication module.

[0029] User pose estimation module:

[0030] On the user side, whenever a new frame (2D image) is captured, it is used as visual input. Simultaneously, due to the higher frequency of IMU (In-Music Detection and Measurement) input, the closest IMU information is selected as the IMU input. During initialization, two consecutive frames with more than 100 feature points are selected and matched. The fundamental matrix, homography matrix, and their score ratio are calculated. Based on the ratio, one model is selected to recover rotation and translation. The first frame is then used as the world coordinate system, and the triangulated 3D points become the map points. After initialization, the pose calculation process involves matching the information from the previous frame or a reference keyframe with the current frame. Based on the matching, the motion model is adjusted accordingly to optimize the camera pose. Due to resource limitations on the user side, a server is needed as a central node to assist in mapping; that is, the server stores the map from the user side. Therefore, the system needs to transmit map information, i.e., the map information of keyframes, to the server. When deciding which frames should be selected as keyframes, the following factors are considered: the most important factor is the keyframe interval, which should be large enough to avoid cross-frame redundancy, but small enough to ensure successful tracking; the second factor is the number of matched map points, which should be large enough to ensure tracking performance, but small enough to avoid excessive redundancy between keyframes. The goal here is to maintain successful tracking using the fewest possible keyframes. In this module, after generating keyframes, they are not immediately sent to the server, but are temporarily stored. The pose is then sent to the server through the user communication module. After waiting for control information from the server, the keyframe data pruning module judges and filters the data before sending the filtered keyframes to the server.

[0031] User communication module:

[0032] In practical implementation, the necessary environment for running visual SLAM needs to be configured. In addition, a suitable version of ZeroMQ needs to be configured. During runtime, the correct server IP address and port number are specified in the user-side communication module to send user poses, receive control information from the server, and send user-side keyframes.

[0033] Redundancy Information Detection and Control Module:

[0034] Establish a view frustum data structure to identify redundant information, such as... Figure 2 As shown, based on visual sensor information (such as vertical and horizontal field of view), a three-dimensional polyhedron is constructed with each user-uploaded pose as its vertex direction. This three-dimensional polyhedron serves as the sampling point region, and samples are uniformly collected. The overlap range, i.e., the view frustum, is calculated according to a ratio. The interior of the view frustum includes uniformly sampled points distributed within it, used in subsequent steps to determine view frustum overlap.

[0035] Whenever a new key frame is generated at the user end, the communication module of the user end sends its pose to the server end quickly using ZeroMQ, and a corresponding view cone is established at the server end. At the same time, the overlap of the view cone with the view cones of the existing key frames on the server is calculated, i.e. the view cone is uniformly sampled as a sampling point region, and if any two sampling points of two view cones are within a certain distance, they are considered to coincide. The overlap range of the view cone with all the poses established on the server is calculated according to the proportion of the coinciding sampling points. A hyperparameter is set to determine the level of overlap, which is used to decide whether the user needs to deliver the key frame to the server and the information to be delivered. If the overlap proportion is large, the control information of "no transmission" is returned; if the overlap proportion is small, the control information of "complete transmission" is returned; and if the overlap proportion is between the two, the control information of "partial overlap" is returned. After the server completes the calculation, if it is "partial overlap", the overlapping part is additionally counted in the form of points and sent to the user end through the communication module.

[0036] Key frame data reduction module:

[0037] After the user end receives these points with indicative significance, it uses the KD tree to find whether there are map points around the points. If there are, the map points are deleted. After traversing all the map points, the user sends the non-redundant information to the server. After the user end receives the control information, if it is "no transmission", the corresponding key frame is discarded; if it is "complete transmission", the complete key frame is directly transmitted; and if it is "partial overlap", the points with indicative significance (i.e. the points of the overlapping part) are extracted, and the KD tree is used to find whether there are map points around the points. If there are, the map points are deleted. After traversing all the points with indicative significance, the user sends the modified key frame to the server. Essentially, since the complete key frame map information is too heavy, direct transmission wastes a lot of network bandwidth, computing resources and storage resources, therefore, a small pose is used to represent the position of the key frame at the user end, and a view cone is established at the server end to simulate the user's view angle, so as to achieve the purpose of detecting redundancy.

[0038] Server communication module:

[0039] In the implementation of this module, the server end also needs to be configured with the necessary environment for running visual SLAM, in addition to configuring a suitable version of ZeroMQ. First, the receiving module of the server end is started to assign the correct ID number to the user, and then the server end will receive the information (pose and key frame) delivered by the user end and process it.

[0040] Mapping module:

[0041] When the server receives the key frame uploaded by the user, the key frame is added to the map of the user, the map includes all information of the key frame uploaded by the user, and G2O is used for optimization to ensure the accuracy of the map.

[0042] The multi-map merging module comprises:

[0043] With more users joining the system, the maps received by the server are from multiple users, and obviously, the coordinates are not aligned and cannot be merged into a complete map. By identifying the same position through the description information of the key frame, the key frames are merged together by using the position, and the repeated data of the position is removed, and in this way, the merged map of all users is obtained. In view of the problem caused by the redundancy information detection module of the application, the key frame processed by the module is marked to avoid system error and system error.

[0044] Although the specific implementation method of the application is described above, those skilled in the art should understand that these are only examples, and various changes or modifications can be made to these embodiments without departing from the principles and implementation of the application, therefore, the protection scope of the application is defined by the appended claims.

Claims

1. An edge-assisted crowd-sourced distributed visual SLAM system, comprising: The application relates to a user terminal and a server terminal. The user terminal comprises a user pose estimation module, a key frame data reduction module and a communication module; the server terminal comprises a mapping module, a redundant information detection and control module, a multi-map merging module and a communication module. The user pose estimation module is used for estimating the current pose of a user, i.e. obtaining the current position and attitude of the user and determining a key frame; the current frame is taken as an initial pose based on a previous frame, the pose of the current frame of the user is further optimized based on the matching results of two frames of feature points, a key frame is determined, the pose of the current frame of the user and the key frame are stored by using a stack, the pose of the user is sent to the server terminal by the communication module, and control information is judged and fed back by the redundant information detection and control module in the server terminal; The key frame data reduction module is used for further processing of original key frames in the user terminal; after the user pose estimation module sends the pose to the server, the server feeds back control information, the user terminal receives the feedback by using the communication module, and the feedback information is used to delete the data in the original key frame in combination with KD tree acceleration, so that a key frame part capable of providing new data for the server terminal is obtained and is sent to the server terminal by the communication module; The user communication module is used for sending the data of the user terminal and receiving the data of the server terminal; ZeroMQ is used for writing, mainly for sending the pose of the user, receiving the feedback of the server and sending the data after the user deletes the redundancies; The mapping module is used for combining the key frames uploaded by the user into a map, the map comprises all the information of each key frame uploaded by each independent user, each user independently establishes a map in the server terminal, and the map is called a "small map" for distinction; each key frame uploaded by the user is inserted into the small map, the redundant key frames and the three-dimensional mapping of feature points are removed, and optimization is carried out, so that the small map of the user is obtained; The redundant information detection and control module is used for detecting the redundancy of the key frames to be uploaded by the user terminal according to the pose uploaded by the user in the server terminal and feeding back to the user terminal; the pose uploaded by the user is simulated as the user's visual angle, i.e. the range observed by the key frame of the user, the redundancy is judged by using the area coverage of the sampling points of the view cone through the establishment of the data structure of the view cone, in addition to the necessary "non-transmission", "complete transmission" and "partial transmission" instructions, if it is "partial transmission", the repeated sampling points detected are sent to the user communication module in the form of a point set, the key frame data reduction module of the user terminal processes the key frame according to the feedback information and then uploads the key frame to the server through the communication module of the user terminal; The data structure of the view cone is as follows: a three-dimensional polyhedron is established by using the CGAL library and taking each pose uploaded by the user as a vertex direction, the three-dimensional polyhedron is uniformly sampled as a sampling point area, and the overlapping range is calculated according to the proportion. ​ A multi-map merging module is used to realize multi-user mapping. The module realizes merging of multiple user sub-maps from the mapping module to obtain a complete map of the mapping area. The same position is identified through the description information of the key frame. The sub-maps are identified and merged together by using the position, and the repeated data of the position is removed. In this way, the merged map of all users is obtained. Finally, the complete map of the entire mapping area is obtained and stored on the server side. A server communication module is used to accept the data of the user side and send the data of the server side. The server communication module receives the pose uploaded by the user side and sends the control information of the server side. The server communication module receives the key frame uploaded by the user side.

2. An edge-assisted crowdsourced distributed visual SLAM method based on, The implementation steps are as follows: Step 1: The data collected by the visual sensor and the IMU are used as input. The feature points in the picture output by the visual sensor are extracted, and the pose of the current frame, i.e., the pose of the sensor, is calculated. At the same time, the ordinary frames that can summarize some information of the ordinary frames are selected as key frames from the captured frames. The pose of the key frame is sent to the server, and the complete information of the key frame is retained on the user device. Step 2: On the server, a view cone is established using the pose uploaded by the user to simulate the user's view range. The latest view cone is compared with all view cones to determine the overlap. The overlapping sampling point information is sent to the user side as control information. On the server, a view cone is established using the pose uploaded by the user and the camera parameters. The view cone is used to simulate the user's view range at the pose. The newly established view cone is compared with all view cones established by all poses of all users on the server to determine the overlap. Sampling points are designed uniformly inside the view cone. If the sampling points of two view cones are within a certain range, it is determined that the sampling points overlap. The overlap range of the two view cones is determined according to the number of overlapping sampling points. The "no transmission", "complete transmission", and "partial transmission" commands are given. If it is "partial transmission", the related information of the overlapping sampling points, i.e., the three-dimensional positions of each three-dimensional point, is additionally sent to the user side as control information. Step 3: The user side receives the control information from the server side to reduce the key frame data. It is determined whether to transmit and process the key frame. The processed key frame is sent to the server. The user side receives the control information from the server side. The key frame to which the control information belongs is found in the temporary stack. It is determined whether to transmit the key frame. If it is "partial transmission", the data of the key frame is reduced. The three-dimensional mapping of the map point, i.e., the feature point, which is close enough to the sampling point in the control information is found. The information related to the map point in the key frame is deleted. The processed key frame is sent to the server through the communication module. If it is "complete transmission", the entire necessary information of the key frame is sent to the server through the communication module. If it is "no transmission", the key frame is discarded. Step 4: The server receives the key frame uploaded by the user, combines the key frame uploaded by the user into a map, the map includes all information of each key frame uploaded by the user independently, each user will establish a map independently on the server side, in order to distinguish, called "small map"; insert each key frame uploaded by the user into the small map, eliminate redundant key frames and map points, that is, three-dimensional mapping of feature points, and optimize to obtain the small map of the user; Step 5: When a new user joins, a new small map is established for the user, when the new user overlaps with the previous user map, a merged map is obtained through multi-map merging, thus completing the SLAM task and obtaining the merged complete map; When a new user joins, a new small map is established for the user on the server side, the key frames uploaded by the user thereafter are stored on the corresponding small map, isolated from other users, when the new user overlaps with the previous user map, that is, passes through the same area and is detected, a merged map is obtained through multi-map merging, the same for each user, thus completing the SLAM task and obtaining the merged complete map containing multiple user uploaded key frames.

Images