Mapping method for mobile robots based on visual SLAM

Abstract

A mapping method for a mobile robot based on visual SLAM is disclosed, which includes: initialization to generate an initial key frame and an initial pose corresponding to the initial key frame; obtaining an RGB image of a target environment and a depth image corresponding to the RGB image through an RGB-D camera module and converting the depth image into a depth point cloud, wherein each pixel of the depth point cloud has an attribute value of multipath interference; generating a key frame sequence based on SLAM visual tracking; and generating a target map based on the corrected depth point clouds and poses of all key frames in the key frame sequence. The mapping method improves the accuracy of map construction and positioning by fusing poses and correcting the depth point cloud based on inter-frame pose transformation.

Description

Technical Field

[0001] The present application relates to the field of visual SLAM, and more specifically, to a mapping method for a mobile robot based on visual SLAM, a positioning method for a mobile robot based on visual SLAM, a mapping device for a mobile robot based on visual SLAM, a positioning device for a mobile robot based on visual SLAM, and a robot. Background Art

[0002] SLAM technology (Simultaneous Localization And Mapping), that is, simultaneous positioning and mapping technology, means that in the absence of prior information, a subject equipped with specific sensors builds a map of the target environment during movement, and simultaneously estimates its own position and posture to achieve autonomous positioning.

[0003] According to sensor classification, SLAM mainly includes laser SLAM and visual SLAM. When the sensor is mainly laser radar, it is called laser SLAM; when the sensor is mainly an image acquisition device (for example, a camera), it is called visual SLAM. Laser SLAM technology is relatively mature, but laser radar has high cost, high installation requirements, low life, limited detection range, and lack of semantic information; while cameras have low cost, diversified installation methods, no sensor detection distance limit, and can extract semantic information. Visual SLAM is a mainstream direction for future research.

[0004] In recent years, visual SLAM technology has been widely used in mobile terminal devices such as robots, self-driving cars, and drones to achieve functions such as map construction, navigation positioning, and path planning. Among them, depth cameras are commonly used visual sensors in mobile terminal devices that apply visual SLAM technology. They can be used to measure the distance from the depth camera lens to the object being photographed, that is, depth data.

[0005] Depth cameras can be divided into passive depth cameras (e.g., passive binocular depth cameras) and active depth cameras (e.g., structured light depth cameras, TOF depth cameras). Compared with using passive binocular vision to measure depth data, using TOF to measure depth data is relatively easy to implement, especially i-TOF depth cameras (i.e., TOF cameras based on the phase difference principle) that use the indirect TOF (indirect-ToF: i-TOF) principle to measure depth data. They are widely used due to their advantages in image pixels, application costs, etc.

[0006] The i-TOF depth camera can be used to collect RGB images and depth point clouds of the target environment, and then use the RGB images and depth point clouds for synchronous map construction and positioning to achieve navigation and obstacle avoidance of robots (for example, sweeping robots). However, there are some technical problems in the practical application of the visual SLAM solution based on the i-TOF depth camera.

[0007] First, the ordinary i-TOF depth camera has a small field of view and a small overlap between frames, which is not conducive to pose estimation. In addition, if the i-TOF depth camera is used as a single pose sensor, it is difficult to adapt to a variety of different scenarios. The reason is that the data obtained from a single sensor is not enough to achieve accurate pose estimation, resulting in low accuracy of map construction and positioning.

[0008] Secondly, the i-TOF depth camera is easily affected by multipath interference (MPI) when working, which makes the obtained depth point cloud have large errors, resulting in reduced accuracy of map construction and positioning.

[0009] Specifically, due to the complex diffuse reflection and specular reflection in the target environment, after the i-TOF depth camera projects the detection laser to the target object in the target environment, points on the target object that are at different distances from the lens of the i-TOF depth camera may reflect light to the i-TOF depth camera at the same time, resulting in a deviation between the depth measurement value and the actual value, that is, there is an error in the depth value of each pixel in the depth point cloud. Moreover, in scenes with corners, the light returning to the i-TOF depth camera may have been reflected once or multiple times. In other words, the corresponding depth values ​​of the points on the target object that simultaneously reflect light to the i-TOF depth camera may be inconsistent.

[0010] Accordingly, an optimized vision-SLAM-based mapping and positioning solution is expected. Summary of the invention

[0011] One advantage of the present application is that it provides a composition method for a mobile robot based on visual SLAM, a positioning method for a mobile robot based on visual SLAM, a composition device for a mobile robot based on visual SLAM, a positioning device for a mobile robot based on visual SLAM, and a robot, wherein the composition method improves the accuracy of map construction and positioning by fusing poses and correcting depth point clouds based on inter-frame pose transformations.

[0012] Another advantage of the present application is that it provides a composition method for a mobile robot based on visual SLAM, a positioning method for a mobile robot based on visual SLAM, a composition device for a mobile robot based on visual SLAM, a positioning device for a mobile robot based on visual SLAM, and a robot, wherein attribute statistics are performed on the multipath interference of the collected depth point cloud to quantify the multipath interference of each pixel in the depth point cloud.

[0013] In order to achieve at least one of the above advantages or other advantages and purposes, according to one aspect of the present application, a mapping method for a mobile robot based on visual SLAM is provided, which includes: step 1: initialization to generate an initial key frame and an initial pose corresponding to the initial key frame;

[0014] Step 2: Obtain an RGB image of the target environment and a depth image corresponding to the RGB image through an RGB-D camera module and convert the depth image into a depth point cloud, wherein each pixel point of the depth point cloud has an attribute value of multipath interference;

[0015] Step 3: Generate a key frame sequence based on SLAM visual tracking, wherein step 3 includes:

[0016] Step 31: based on the feature points of the RGB image of the current frame, the feature points of the RGB image of the previous frame adjacent to the current frame, and the pose data provided by the inertial sensor corresponding to the current frame and the previous frame respectively, obtaining the estimated pose corresponding to the current frame;

[0017] Step 32: inputting the estimated poses of the current frame and S frames of image frames continuous with the current frame into a sliding window for nonlinear optimization to generate the pose of the current frame;

[0018] Step 33: generating an inter-frame pose transformation matrix based on the pose of the current frame and the pose of the previous frame;

[0019] Step 34: Based on the inter-frame pose transformation matrix and the depth point cloud of the previous frame, correct the pixel points whose attribute values ​​of multipath interference in the depth point cloud of the current frame are higher than a preset threshold, so as to obtain a corrected depth point cloud of the current frame;

[0020] Step 35: iteratively executing steps 31 to 34 to obtain the key frame sequence, each key frame in the key frame sequence having a corrected depth point cloud; and

[0021] Step 4: Generate a target map based on the corrected depth point clouds and poses of all key frames in the key frame sequence.

[0022] In the mapping method for a mobile robot based on visual SLAM according to the present application, the inertial sensor includes an inertial measurement unit and a wheel encoder, and the pose data includes first pose data obtained by the inertial measurement unit and second pose data obtained by the wheel encoder; wherein step 31: based on the feature points of the RGB image of the current frame and the feature points of the RGB image of the previous frame adjacent to the current frame, and the pose data provided by the inertial sensor corresponding to the current frame and the previous frame, respectively, obtaining the estimated pose corresponding to the current frame, including:

[0023] Based on the following loss function, the fusion pose corresponding to the current frame is obtained, wherein the loss function is C(x_k)=cost_imu+cost_wheel+cost_camera;

[0024] Among them, cost_imu represents the integral loss function of the first pose data corresponding to the current frame and the previous frame respectively with the estimated pose as the estimated quantity; cost_wheel represents the integral loss function of the second pose data corresponding to the current frame and the previous frame respectively with the estimated pose as the estimated quantity; cost_camera is the reprojection error generated based on the feature points of the RGB image of the current frame.

[0025] In the mapping method for a mobile robot based on visual SLAM according to the present application, step 32: inputting the estimated poses of the current frame and the S-frame image frames continuous with the current frame into a sliding window for nonlinear optimization to generate the pose of the current frame, including:

[0026] Inputting the estimated poses of the current frame and S frames of image frames continuous with the current frame into the optimization function described below, and calculating the marginalization matrix and displacement vector of the optimization function as the pose of the current frame;

[0027] Wherein, the optimization function is C(x_i)=∑(cost_imu+cost_wheel+cost_camera).

[0028] In the mapping method for a mobile robot based on visual SLAM according to the present application, step 34: based on the inter-frame pose transformation matrix and the depth point cloud of the previous frame, correcting the pixel points whose attribute values ​​of multipath interference in the depth point cloud of the current frame are higher than a preset threshold, so as to obtain the corrected depth point cloud of the current frame, including:

[0029] Filter out the pixel points to be replaced whose attribute values ​​of multipath interference are higher than a preset threshold in the depth point cloud of the current frame;

[0030] Converting the pixel points corresponding to the pixel points to be replaced in the depth point cloud of the previous frame into replacement pixel points through the inter-frame pose transformation matrix; and

[0031] The pixel points to be replaced in the depth point cloud of the current frame are replaced with the replacement pixel points to generate a corrected depth point cloud of the current frame.

[0032] In the mapping method for a mobile robot based on visual SLAM according to the present application, step 4: generating a target map based on the corrected depth point clouds and poses of all key frames in the key frame sequence, including:

[0033] Generate a probability grid map based on the corrected depth point clouds and poses of all key frames in the key frame sequence;

[0034] Generate a point cloud map based on the corrected depth point clouds of all key frames in the key frame sequence;

[0035] The point cloud map and the probability grid map are fused to generate the target map.

[0036] In the mapping method for a mobile robot based on visual SLAM according to the present application, step 3 further comprises: performing back-end optimization on the key frame sequence and / or the key frame;

[0037] The backend optimization includes at least one of the following steps:

[0038] Constructing co-viewing constraints of the feature points based on the feature points of the RGB image of the key frame and a bag-of-words model;

[0039] Based on ICP matching, constructing pose constraints between consecutive key frames; and

[0040] When a closed loop occurs, global nonlinear optimization is performed using the bundle adjustment method.

[0041] In the visual SLAM-based mapping method for a mobile robot according to the present application, the attribute value of the multipath interference is the confidence level of the multipath interference.

[0042] According to another aspect of the present application, a positioning method for a mobile robot based on visual SLAM is provided, comprising:

[0043] Step 110: generating the target map based on the mapping method for a mobile robot based on visual SLAM;

[0044] Step 120: Based on the local RGB image and depth point cloud of the target environment collected by the mobile robot, a local map is generated using the mapping method for the mobile robot based on visual SLAM;

[0045] Step 130: Determine the relocation initial pose of the mobile robot based on the nonlinear matching between the local map and the target map;

[0046] Step 140: constructing a nonlinear optimization function based on the reprojection error of the feature points of the RGB image of the current frame on the point cloud map, the matching error between the corrected depth point cloud of the current frame and the probability grid map, and the inertial pose error determined based on the pose data provided by the inertial sensor corresponding to the current frame and the previous frame, respectively, and obtaining the estimated pose of the current frame through the nonlinear optimization function; and

[0047] Step 150: Input the estimated poses of the current frame and S image frames continuous with the current frame into a sliding window for nonlinear optimization to generate the pose of the current frame.

[0048] In the positioning method for a mobile robot based on visual SLAM according to the present application, the step 140: constructing a nonlinear optimization function based on the reprojection error of the feature points of the RGB image of the current frame on the point cloud map, the matching error between the corrected depth point cloud of the current frame and the probability grid map, and the inertial pose error determined based on the pose data provided by the inertial sensor corresponding to the current frame and the previous frame, respectively, and obtaining the estimated pose of the current frame through the nonlinear optimization function, including:

[0049] Based on the following probability matching loss function, the estimated pose of the current frame is obtained, wherein the loss function is C(xp_k)=cost_imu+cost_wheel+cost_camera+cost_point;

[0050] Among them, cost_imu represents the integral loss function of the first pose data corresponding to the current frame and the previous frame respectively with the estimated pose as the estimated quantity; cost_wheel represents the integral function of the second pose data corresponding to the current frame and the previous frame respectively with the estimated pose as the estimated quantity; cost_camera is the reprojection error of the feature points of the RGB image of the current frame on the point cloud map; cost_point is based on the matching error between the corrected depth point cloud of the current frame and the probability grid map.

[0051] In the positioning method for a mobile robot based on visual SLAM according to the present application, the step 150: inputting the estimated poses of the current frame and the S-frame image frames continuous with the current frame into a sliding window for nonlinear optimization to generate the pose of the current frame, including: inputting the estimated poses of the current frame and the S-frame image frames continuous with the current frame into the optimization function described below, and calculating the marginalization matrix and displacement vector of the optimization function as the pose of the current frame;

[0052] Wherein, the optimization function is C(xp_k)=Σ(cost_imu+cost_wheel+cost_camera+cost_point).

[0053] According to another aspect of the present application, a mapping device for a mobile robot based on visual SLAM is provided, comprising:

[0054] An initialization unit, used to perform step 1: initialization to generate an initial key frame and an initial pose corresponding to the initial key frame;

[0055] A data acquisition unit, configured to execute step 2: acquiring an RGB image of a target environment and a depth image corresponding to the RGB image through an RGB-D camera module and converting the depth image into a depth point cloud, wherein each pixel point of the depth point cloud has an attribute value of multipath interference;

[0056] The tracking unit is used to execute step 3: generating a key frame sequence based on SLAM visual tracking, wherein step 3 includes:

[0057] Step 31: based on the feature points of the RGB image of the current frame, the feature points of the RGB image of the previous frame adjacent to the current frame, and the pose data provided by the inertial sensor corresponding to the current frame and the previous frame respectively, obtaining the estimated pose corresponding to the current frame;

[0058] Step 32: inputting the estimated poses of the current frame and S frames of image frames continuous with the current frame into a sliding window for nonlinear optimization to generate the pose of the current frame;

[0059] Step 33: generating an inter-frame pose transformation matrix based on the pose of the current frame and the pose of the previous frame;

[0060] Step 34: Based on the inter-frame pose transformation matrix and the depth point cloud of the previous frame, correct the pixel points whose attribute values ​​of multipath interference in the depth point cloud of the current frame are higher than a preset threshold, so as to obtain a corrected depth point cloud of the current frame;

[0061] Step 35: iteratively executing steps 31 to 34 to obtain the key frame sequence, each key frame in the key frame sequence having a corrected depth point cloud; and

[0062] The map generation unit is used to execute step 4: generate a target map based on the corrected depth point clouds and poses of all key frames in the key frame sequence.

[0063] According to another aspect of the present application, a positioning device for a mobile robot based on visual SLAM is provided, comprising:

[0064] A map construction unit, configured to execute step 110: generating the target map based on the mapping method for a mobile robot based on visual SLAM;

[0065] A local map construction unit, configured to execute step 120: based on the local RGB image and the depth point cloud of the target environment collected by the mobile robot, generate a local map using the mapping method for the mobile robot based on visual SLAM;

[0066] A relocation initial pose generating unit, configured to execute step 130: determining a relocation initial pose of the mobile robot based on a nonlinear match between the local map and the target map;

[0067] A pose estimation unit is used to perform step 140: construct a nonlinear optimization function based on the reprojection error of the feature points of the RGB image of the current frame on the point cloud map, the matching error between the corrected depth point cloud of the current frame and the probability grid map, and the inertial pose error determined based on the pose data provided by the inertial sensor corresponding to the current frame and the previous frame, respectively, and obtain the estimated pose of the current frame through the nonlinear optimization function; and

[0068] The nonlinear optimization unit is used to execute step S150, input the estimated poses of the current frame and S frame image frames continuous with the current frame into a sliding window for nonlinear optimization to generate the pose of the current frame.

[0069] According to another aspect of the present application, there is provided a robot comprising:

[0070] Robot body;

[0071] An RGB-D camera module installed on the robot body;

[0072] An inertial sensor mounted on the robot body; and

[0073] The device is communicatively connected to the RGB-D camera module and the inertial sensor processing device, wherein the data processing device is used to execute the visual SLAM-based mapping method for a mobile robot, or the visual SLAM-based positioning method for a mobile robot.

[0074] Further objectives and advantages of the present application will be fully reflected through understanding of the following description and drawings.

[0075] These and other objects, features and advantages of the present application are fully reflected in the following detailed description, drawings and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 The figure illustrates a flow chart of a mapping method for a mobile robot based on visual SLAM according to an embodiment of the present application.

[0077] Figure 2 The figure illustrates a flowchart of generating a key frame sequence based on SLAM visual tracking in a composition method for a mobile robot based on visual SLAM according to an embodiment of the present application.

[0078] Figure 3 The diagram illustrates a flowchart of a positioning method for a mobile robot based on visual SLAM according to an embodiment of the present application, in which, based on the inter-frame pose transformation matrix and the depth point cloud of the previous frame, pixel points whose attribute values ​​of multipath interference are higher than a preset threshold in the depth point cloud of the current frame are corrected to obtain a corrected depth point cloud of the current frame.

[0079] Figure 4 The figure illustrates a flow chart of a positioning method for a mobile robot based on visual SLAM according to an embodiment of the present application.

[0080] Figure 5 The figure illustrates a block diagram of a mapping device for a mobile robot based on visual SLAM according to an embodiment of the present application.

[0081] Figure 6 The figure illustrates a block diagram of a positioning device for a mobile robot based on visual SLAM according to an embodiment of the present application.

[0082] Figure 7 A block diagram of a robot according to an embodiment of the present application is illustrated. DETAILED DESCRIPTION

[0083] The following description is used to disclose the present application so that those skilled in the art can implement the present application. The embodiments described below are only examples, and those skilled in the art can think of other obvious variations. The basic principles of the present application defined in the following description can be applied to other embodiments, variations, improvements, equivalent solutions, and other technical solutions that do not deviate from the spirit and scope of the present application.

[0084] Application Overview

[0085] As mentioned above, there are some technical problems in the practical application of the visual SLAM solution based on i-TOF depth camera. First, the ordinary i-TOF depth camera has a small field of view and a small overlap area between frames, which is not conducive to pose estimation; and a single sensor is difficult to adapt to a variety of different scenes. The data obtained from a single sensor is not enough to achieve accurate pose estimation, resulting in low accuracy of map construction and positioning. Secondly, the i-TOF depth camera is easily affected by multipath interference (MPI), resulting in errors in the depth values ​​of each pixel in the obtained depth point cloud.

[0086] For the problem of low accuracy of map construction and positioning caused by low accuracy of pose estimation, on the one hand, wide-angle i-TOF depth cameras can be used to expand the field of view and increase the overlap area between frames to facilitate the matching of corresponding feature points between key frame images, thereby improving the accuracy of pose estimation. On the other hand, multiple sensors can be used to obtain pose data, and by fusing pose data from different sensors, the reliability of pose data can be improved, further improving the accuracy of map construction and positioning. For example, pose data from inertial sensors can be fused, and pose data can be nonlinearly optimized to improve the accuracy of map construction and positioning.

[0087] Regarding the problem of low accuracy of map construction and positioning caused by multipath interference. The inventors of this application try to improve the accuracy of map construction and positioning by starting with the processing of depth point cloud data. Specifically, due to the influence of multipath interference, there may be parts with large errors in the acquired depth point cloud. Then, we can first identify which pixels in the depth point cloud have large errors, and then use a specific data correction scheme to correct these pixels with large errors to improve the accuracy of map construction and positioning.

[0088] In the technical solution of the present application, the multipath interference of the collected depth point cloud can first be attributed to statistics to quantify the multipath interference of each pixel in the depth point cloud. Here, the attribute value of multipath interference reflects the degree to which the depth point cloud is affected by multipath interference. The higher the attribute value of multipath interference, the greater the degree of multipath interference; conversely, the lower the attribute value of multipath interference, the lower the degree of multipath interference. For example, the attribute value of multipath interference can be implemented as the confidence of multipath interference, that is, if the confidence of multipath is higher, the greater the degree of multipath interference; conversely, the lower the confidence of multipath interference, the lower the degree of multipath interference. Correspondingly, the part of the depth point cloud to be corrected can be identified by the attribute value of multipath interference.

[0089] During the depth point cloud correction process, since the data between two adjacent frames of depth point clouds are correlated, in the technical solution of the present application, the inventor of the present application attempts to correct the depth point cloud part to be corrected in the depth point cloud of the current frame by generating an inter-frame pose transformation matrix through the poses of adjacent key frames.

[0090] Based on this, the present application proposes a composition method for a mobile robot based on visual SLAM, which includes: step 1: initialization to generate an initial key frame and an initial pose corresponding to the initial key frame; step 2: through an RGB-D camera module, an RGB image of the target environment and a depth image corresponding to the RGB image are obtained and the depth image is converted into a depth point cloud, and each pixel point of the depth point cloud has an attribute value of multipath interference; step 3: based on SLAM visual tracking, a key frame sequence is generated, wherein step 3 includes: step 31: based on the feature points of the RGB image of the current frame and the feature points of the RGB image of the previous frame adjacent to the current frame, and the pose data provided by the inertial sensor corresponding to the current frame and the previous frame, respectively, the estimated pose corresponding to the current frame is obtained. Step 32: input the estimated poses of the current frame and the S-frame image frames continuous with the current frame into a sliding window for nonlinear optimization to generate the pose of the current frame; Step 33: based on the pose of the current frame and the pose of the previous frame, generate an inter-frame pose transformation matrix; Step 34: based on the inter-frame pose transformation matrix and the depth point cloud of the previous frame, correct the pixel points whose attribute values ​​of multipath interference in the depth point cloud of the current frame are higher than a preset threshold to obtain the corrected depth point cloud of the current frame; Step 35: iteratively execute steps 31 to 34 to obtain the key frame sequence, each key frame in the key frame sequence has a corrected depth point cloud; and Step 4: based on the corrected depth point clouds and poses of all key frames in the key frame sequence, generate a target map. Based on this, the present application also proposes an optical lens, which includes the optical system as described above, and a lens barrel for accommodating the optical system therein.

[0091] Based on this, the present application also proposes a positioning method for a mobile robot based on visual SLAM, which includes: step 110: generating the target map based on the composition method for the mobile robot based on visual SLAM; step 120: generating a local map based on the local RGB image and depth point cloud of the target environment collected by the mobile robot using the composition method for the mobile robot based on visual SLAM; step 130: determining the repositioning initial posture of the mobile robot based on the nonlinear matching between the local map and the target map; step 1 40: construct a nonlinear optimization function based on the reprojection error of the feature points of the RGB image of the current frame on the point cloud map, the matching error between the corrected depth point cloud of the current frame and the probability grid map, and the inertial pose error determined based on the pose data provided by the inertial sensor corresponding to the current frame and the previous frame, respectively, and obtain the estimated pose of the current frame through the nonlinear optimization function; and step 150: input the estimated poses of the current frame and the S-frame image frames continuous with the current frame into a sliding window for nonlinear optimization to generate the pose of the current frame.

[0092] Based on this, the present application also proposes a composition device for a mobile robot based on visual SLAM, which includes: an initialization unit, used to execute step 1: initialization to generate an initial key frame and an initial pose corresponding to the initial key frame; a data acquisition unit, used to execute step 2: through an RGB-D camera module, obtain an RGB image of the target environment and a depth image corresponding to the RGB image and convert the depth image into a depth point cloud, each pixel of the depth point cloud has an attribute value of multipath interference; a tracking unit, used to execute step 3: based on SLAM visual tracking, generate a key frame sequence, wherein step 3 includes: step 31: based on the feature points of the RGB image of the current frame and the feature points of the RGB image of the previous frame adjacent to the current frame, and the pose data provided by the inertial sensor corresponding to the current frame and the previous frame, respectively, obtain Obtain the estimated pose corresponding to the current frame; Step 32: Input the estimated poses of the current frame and the S-frame image frames continuous with the current frame into a sliding window for nonlinear optimization to generate the pose of the current frame; Step 33: Generate an inter-frame pose transformation matrix based on the pose of the current frame and the pose of the previous frame; Step 34: Based on the inter-frame pose transformation matrix and the depth point cloud of the previous frame, correct the pixel points whose attribute values ​​of multipath interference in the depth point cloud of the current frame are higher than a preset threshold to obtain the corrected depth point cloud of the current frame; Step 35: Iteratively execute steps 31 to 34 to obtain the key frame sequence, each key frame in the key frame sequence has a corrected depth point cloud; and a map generation unit, used to execute step 4: generate a target map based on the corrected depth point clouds and poses of all key frames in the key frame sequence.

[0093] Based on this, the present application also proposes a positioning device for a mobile robot based on visual SLAM, which includes: a map construction unit, used to execute step 110: based on the composition method for a mobile robot based on visual SLAM, generate the target map; a local map construction unit, used to execute step 120: based on the local RGB image and depth point cloud of the target environment collected by the mobile robot, use the composition method for a mobile robot based on visual SLAM to generate a local map; a repositioning initial pose generation unit, used to execute step 130: based on the nonlinear matching between the local map and the target map, determine the repositioning of the mobile robot An initial pose; a pose estimation unit, used to execute step 140: construct a nonlinear optimization function based on the reprojection error of the feature points of the RGB image of the current frame on the point cloud map, the matching error between the corrected depth point cloud of the current frame and the probability grid map, and the inertial pose error determined based on the pose data provided by the inertial sensor corresponding to the current frame and the previous frame, respectively, and obtain the estimated pose of the current frame through the nonlinear optimization function; and a nonlinear optimization unit, used to execute step S150, input the estimated poses of the current frame and the S frame image frames continuous with the current frame into a sliding window for nonlinear optimization to generate the pose of the current frame.

[0094] Based on this, the present application also proposes a robot, which includes: a robot body; an RGB-D camera module installed on the robot body; an inertial sensor installed on the robot body; and a data processing device communicatively connected to the RGB-D camera module and the inertial sensor, wherein the data processing device is used to execute the composition method for a mobile robot based on visual SLAM, or the positioning method for a mobile robot based on visual SLAM.

[0095] After introducing the basic principles of the present application, various non-limiting embodiments of the present application will be described in detail with reference to the accompanying drawings.

[0096] The following is a basic introduction to the relevant terms involved in this application.

[0097] Visual SLAM: refers to an entity equipped with a visual sensor, such as a robot, vehicle, drone, etc., which builds a map of the target environment while exploring the target environment without prior information, and estimates its own position and posture to achieve autonomous positioning.

[0098] Key frame: Among all the frames in the video, the video frame that meets the preset conditions has information that is useful for SLAM. Using key frames, the position and posture of the subject equipped with the visual sensor can be effectively estimated and the redundancy of information can be reduced.

[0099] Position and attitude: position and attitude. The position can be expressed by translation in the x, y, and z directions in the coordinate system, and the attitude can be expressed by rotation in the x, y, and z directions in the coordinate system.

[0100] Point cloud: refers to the collection of sampling points of the target object obtained in the same spatial reference system, which can express the spatial distribution and surface characteristics of the target object, and has three-dimensional coordinate information, depth information and other information. Point cloud can be obtained through depth image conversion.

[0101] Inertial sensors: can be used to measure acceleration, rotation, and multiple degrees of freedom. In visual SLAM applications, inertial sensors can be used to obtain posture data.

[0102] Bag-of-words model: It is a feature extraction method that can represent documents as feature vectors. In the application of visual SLAM, the bag-of-words model can be used for feature extraction to quickly find similar images.

[0103] ICP: Iterative Closest Point algorithm is a point cloud matching algorithm that transforms point clouds based on the pose relationship between two sets of point clouds, calculates the error between the closest points in the two sets of point clouds after transformation, and iterates continuously until the error is less than a threshold or the maximum number of iterations is reached.

[0104] Bundle Adjustment (BA) refers to the process of extracting the optimal 3D model and camera parameters (intrinsic and extrinsic parameters) from visual reconstruction, and finally converging the bundles of lightrays reflected from each feature point to the optical center of the camera after making the best adjustment to the camera posture and spatial position of the feature points.

[0105] Exemplary Patterning Methods

[0106] The composition method based on visual SLAM can be applied to mobile terminal devices such as robots, self-driving cars, and drones. For the sake of ease of explanation and understanding, the composition method of this embodiment is explained below using a mobile robot as an example.

[0107] Figure 1 The figure illustrates a flow chart of a mapping method for a mobile robot based on visual SLAM according to an embodiment of the present application. Figure 2 The figure illustrates a flowchart of generating a key frame sequence based on SLAM visual tracking in a composition method for a mobile robot based on visual SLAM according to an embodiment of the present application.

[0108] like Figure 1 and Figure 2As shown, according to the composition method for a mobile robot based on visual SLAM according to an embodiment of the present application, the method includes: step 1: initialization to generate an initial key frame and an initial pose corresponding to the initial key frame; step 2: through an RGB-D camera module, an RGB image of the target environment and a depth image corresponding to the RGB image are acquired and the depth image is converted into a depth point cloud, wherein each pixel point of the depth point cloud has an attribute value of multipath interference; step 3: based on SLAM visual tracking, a key frame sequence is generated, wherein step 3 includes: step 31: based on the feature points of the RGB image of the current frame and the feature points of the RGB image of the previous frame adjacent to the current frame, and the pose data provided by the inertial sensor corresponding to the current frame and the previous frame, respectively, the pose data corresponding to the current frame is obtained. Estimate pose; Step 32: Input the estimated poses of the current frame and the S-frame image frames continuous with the current frame into a sliding window for nonlinear optimization to generate the pose of the current frame; Step 33: Generate an inter-frame pose transformation matrix based on the pose of the current frame and the pose of the previous frame; Step 34: Based on the inter-frame pose transformation matrix and the depth point cloud of the previous frame, correct the pixel points whose attribute values ​​of multipath interference in the depth point cloud of the current frame are higher than a preset threshold to obtain the corrected depth point cloud of the current frame; Step 35: Iteratively execute steps 31 to 34 to obtain the key frame sequence, each key frame in the key frame sequence has a corrected depth point cloud; and Step 4: Generate a target map based on the corrected depth point clouds and poses of all key frames in the key frame sequence.

[0109] In step 2, an RGB image of the target environment and a depth image corresponding to the RGB image are obtained through an RGB-D camera module, and the depth image is converted into a depth point cloud, wherein each pixel of the depth point cloud has an attribute value of multipath interference. In the example of the present application, the attribute value of the multipath interference is the confidence of the multipath interference, which reflects the degree to which the depth point cloud is interfered by multipath. The higher the attribute value of the multipath interference, the greater the degree of multipath interference; conversely, the lower the attribute value of the multipath interference, the lower the degree of multipath interference.

[0110] It is worth mentioning that in the embodiment of the present application, in order to avoid the influence of small field of view angle on pose estimation, preferably, the RGB-D camera module is a wide-angle i-TOF camera module (horizontal field of view angle is not less than 120 degrees), which expands the field of view angle and increases the overlapping area between frames to facilitate the matching of corresponding feature points between key frame images, thereby improving the accuracy of pose estimation.

[0111] Of course, in other examples of the present application, other suitable camera modules such as wide-angle camera modules and panoramic camera modules may be selected, and multiple camera modules may be selected to expand the field of view, which is not limited to the present application.

[0112] As mentioned above, the data obtained from a single sensor is not sufficient to achieve accurate estimation of posture, and multiple sensors can be used to obtain posture data. By fusing posture data from different sensors, the reliability of posture data can be improved. In a specific example of the present application, the posture data from the inertial sensor is fused, and the posture data is nonlinearly optimized to improve the accuracy of map construction and positioning. The inertial sensor includes an inertial measurement unit and a wheel encoder, and the posture data includes the first posture data obtained by the inertial measurement unit and the second posture data obtained by the wheel encoder. Of course, in other examples of the present application, the posture data of other sensors, such as a GPS sensor, can be fused, and this is not limited to the present application.

[0113] Specifically, step 31: based on the feature points of the RGB image of the current frame, the feature points of the RGB image of the previous frame adjacent to the current frame, and the pose data provided by the inertial sensor corresponding to the current frame and the previous frame, obtain the estimated pose corresponding to the current frame, including: based on the following loss function, obtain the fused pose corresponding to the current frame, wherein the loss function is C(x_k)=cost_imu+cost_wheel+cost_camera; wherein cost_imu represents the integral loss function of the first pose data corresponding to the current frame and the previous frame, respectively, with the estimated pose as the estimated amount; cost_wheel represents the integral loss function of the second pose data corresponding to the current frame and the previous frame, respectively, with the estimated pose as the estimated amount; cost_camera is the reprojection error generated based on the feature points of the RGB image of the current frame.

[0114] Accordingly, in a specific example of the present application, step 32: inputting the estimated pose of the current frame and the S-frame image frames continuous with the current frame into a sliding window for nonlinear optimization to generate the pose of the current frame, including: inputting the estimated pose of the current frame and the S-frame image frames continuous with the current frame into the optimization function described below, and calculating the marginalization matrix and displacement vector of the optimization function as the pose of the current frame; wherein the optimization function is C(x_i)=∑(cost_imu+cost_wheel+cost_camera).

[0115] Figure 3The diagram illustrates a flowchart of a positioning method for a mobile robot based on visual SLAM according to an embodiment of the present application, in which, based on the inter-frame pose transformation matrix and the depth point cloud of the previous frame, pixel points whose attribute values ​​of multipath interference are higher than a preset threshold in the depth point cloud of the current frame are corrected to obtain a corrected depth point cloud of the current frame.

[0116] like Figure 3 As shown, step 34: based on the inter-frame pose transformation matrix and the depth point cloud of the previous frame, the pixel points whose attribute values ​​of multipath interference in the depth point cloud of the current frame are higher than a preset threshold are corrected to obtain the corrected depth point cloud of the current frame, including: screening out the pixel points to be replaced whose attribute values ​​of multipath interference in the depth point cloud of the current frame are higher than the preset threshold; converting the pixel points corresponding to the pixel points to be replaced in the depth point cloud of the previous frame into replacement pixel points through the inter-frame pose transformation matrix; and replacing the pixel points to be replaced in the depth point cloud of the current frame with the replacement pixel points to generate the corrected depth point cloud of the current frame.

[0117] It is worth mentioning that in order to obtain key frames with higher point cloud accuracy, the initial key frames in which more than 90% of the pixels with attribute values ​​of multipath interference in the depth point cloud are lower than the preset threshold can be screened out as pre-selected frames, and based on the pose of the pre-selected current frame and the pose of the pre-selected previous frame, the pre-selected inter-frame pose transformation matrix is ​​generated, and based on the pre-selected inter-frame pose transformation matrix and the depth point cloud of the pre-selected previous frame, the pixels with attribute values ​​of multipath interference in the depth point cloud of the pre-selected current frame higher than the preset threshold are corrected to obtain the corrected depth point cloud of the pre-selected current frame.

[0118] In particular, not only can the pixel point replacement method be used, but also the pixel points whose attribute values ​​of multipath interference in the depth point cloud of the current frame are higher than a preset threshold value can be corrected based on the inter-frame pose transformation matrix and the depth point cloud of the previous frame through the inter-frame pose transformation matrix. For example, in other examples of the present application, the pixel points whose attribute values ​​of multipath interference in the depth point cloud of the current frame are higher than a preset threshold value can be corrected through the inter-frame pose transformation matrix and the depth point cloud of the previous frame by weighted methods or other methods, which is not limited to the present application.

[0119] By correcting the depth point cloud, the error of the depth point cloud caused by multipath interference can be reduced, and the accuracy of map construction and positioning can be improved.

[0120] In step 3: in the process of generating a key frame sequence based on SLAM visual tracking, the data obtained for completing map construction and positioning inevitably have accumulated errors, especially as the scope of the target map expands, the accumulated errors will increasingly affect the accuracy of the data. According to the actual scenario and actual needs, after generating the key frame sequence, the key frame sequence and / or the key frame can be back-end optimized.

[0121] Accordingly, the composition method for a mobile robot based on visual SLAM further includes: performing back-end optimization on the key frame sequence and / or the key frame; wherein the back-end optimization includes at least one of the following steps: constructing a co-viewing constraint of the feature points based on the feature points of the RGB image of the key frame and a bag-of-words model; constructing pose constraints between consecutive key frames based on ICP matching; and performing global nonlinear optimization using a bundle adjustment method when a closed loop is generated.

[0122] In a specific example of the present application, step 4: generating a target map based on the corrected depth point clouds and poses of all key frames in the key frame sequence, including: generating a probability grid map based on the corrected depth point clouds and poses of all key frames in the key frame sequence; generating a point cloud map based on the corrected depth point clouds of all key frames in the key frame sequence; fusing the point cloud map and the probability grid map to generate the target map.

[0123] Exemplary Positioning Methods

[0124] According to another aspect of the present application, a positioning method for a mobile robot based on optical vision SLAM is also provided. Figure 4 The figure illustrates a flow chart of a positioning method for a mobile robot based on visual SLAM according to an embodiment of the present application.

[0125] like Figure 4As shown, according to the positioning method for a mobile robot based on visual SLAM according to an embodiment of the present application, it includes: step 110: generating the target map based on the composition method for a mobile robot based on visual SLAM as described; step 120: generating a local map based on the local RGB image and depth point cloud of the target environment collected by the mobile robot with the composition method for a mobile robot based on visual SLAM; step 130: determining the initial pose for repositioning of the mobile robot based on the nonlinear matching between the local map and the target map; step 140: construct a nonlinear optimization function based on the reprojection error of the feature points of the RGB image of the current frame on the point cloud map, the matching error between the corrected depth point cloud of the current frame and the probability grid map, and the inertial pose error determined based on the pose data provided by the inertial sensor corresponding to the current frame and the previous frame, respectively, and obtain the estimated pose of the current frame through the nonlinear optimization function; and step 150: input the estimated poses of the current frame and the S-frame image frames continuous with the current frame into a sliding window for nonlinear optimization to generate the pose of the current frame.

[0126] Specifically, the step 140: constructing a nonlinear optimization function based on the reprojection error of the feature points of the RGB image of the current frame on the point cloud map, the matching error between the corrected depth point cloud of the current frame and the probability grid map, and the inertial pose error determined based on the pose data provided by the inertial sensor corresponding to the current frame and the previous frame, respectively, and obtaining the estimated pose of the current frame through the nonlinear optimization function, including: obtaining the estimated pose of the current frame based on the following probability matching loss function, wherein the loss function is C(xp_k)=cost_imu+cost_wheel+cost_ca mera+cost_point; wherein C(xp_k) represents the estimated pose; cost_imu represents the integral loss function of the first pose data corresponding to the current frame and the previous frame respectively with the estimated pose as the estimated quantity; cost_wheel represents the integral function of the second pose data corresponding to the current frame and the previous frame respectively with the estimated pose as the estimated quantity; cost_camera is the reprojection error of the feature points of the RGB image of the current frame on the point cloud map; cost_point is based on the matching error between the corrected depth point cloud of the current frame and the probability grid map.

[0127] Correspondingly, the step 150: inputting the estimated poses of the current frame and the S-frame image frames continuous with the current frame into a sliding window for nonlinear optimization to generate the pose of the current frame, including: inputting the estimated poses of the current frame and the S-frame image frames continuous with the current frame into the optimization function described below, and calculating the marginalization matrix and displacement vector of the optimization function as the pose of the current frame; wherein the optimization function is C(xp_k)=Σ(cost_imu+cost_wheel+cost_camera+cost_point).

[0128] Exemplary Patterning Devices

[0129] According to another aspect of the present application, a mapping device 10 for a mobile robot based on visual SLAM is also provided. Figure 5 The figure illustrates a block diagram of a mapping device for a mobile robot based on visual SLAM according to an embodiment of the present application.

[0130] like Figure 5 As shown, according to the visual SLAM-based mapping device 10 for a mobile robot according to an embodiment of the present application, it includes: an initialization unit 11, used to execute step 1: initialization to generate an initial key frame and an initial pose corresponding to the initial key frame; a data acquisition unit 12, used to execute step 2: through an RGB-D camera module, an RGB image of the target environment and a depth image corresponding to the RGB image are acquired and the depth image is converted into a depth point cloud, wherein each pixel point of the depth point cloud has an attribute value of multipath interference; a tracking unit 13, used to execute step 3: based on SLAM visual tracking, a key frame sequence is generated, wherein step 3 includes: step 31: based on the feature points of the RGB image of the current frame and the feature points of the RGB image of the previous frame adjacent to the current frame, and the pose data provided by the inertial sensor corresponding to the current frame and the previous frame, respectively. According to the present invention, an estimated pose corresponding to the current frame is obtained; step 32: the estimated poses of the current frame and the S-frame image frames continuous with the current frame are input into a sliding window for nonlinear optimization to generate the pose of the current frame; step 33: based on the pose of the current frame and the pose of the previous frame, an inter-frame pose transformation matrix is ​​generated; step 34: based on the inter-frame pose transformation matrix and the depth point cloud of the previous frame, the pixel points whose attribute values ​​of multipath interference in the depth point cloud of the current frame are higher than a preset threshold are corrected to obtain the corrected depth point cloud of the current frame; step 35: iteratively execute steps 31 to 34 to obtain the key frame sequence, each key frame in the key frame sequence has a corrected depth point cloud; and a map generation unit 14 is used to execute step 4: generate a target map based on the corrected depth point clouds and poses of all key frames in the key frame sequence.

[0131] Exemplary Positioning Devices

[0132] According to another aspect of the present application, a positioning device 20 for a mobile robot based on visual SLAM is also provided. Figure 6 The figure illustrates a block diagram of a positioning device for a mobile robot based on visual SLAM according to an embodiment of the present application.

[0133] like Figure 6 As shown, according to the positioning device 20 for a mobile robot based on visual SLAM according to an embodiment of the present application, it includes: a map construction unit 21, used to execute step 110: based on the composition method for a mobile robot based on visual SLAM, generate the target map; a local map construction unit 22, used to execute step 120: based on the local RGB image and depth point cloud of the target environment collected by the mobile robot, generate a local map with the composition method for a mobile robot based on visual SLAM; a repositioning initial pose generation unit 23, used to execute step 130: based on the nonlinear matching between the local map and the target map, determine the repositioning position of the mobile robot; Positioning the initial posture; a posture estimation unit 24, used to execute step 140: constructing a nonlinear optimization function based on the reprojection error of the feature points of the RGB image of the current frame on the point cloud map, the matching error between the corrected depth point cloud of the current frame and the probability grid map, and the inertial posture error determined based on the posture data provided by the inertial sensor corresponding to the current frame and the previous frame, respectively, and obtaining the estimated posture of the current frame through the nonlinear optimization function; and a nonlinear optimization unit 25, used to execute step S150, inputting the estimated postures of the current frame and the S-frame image frames continuous with the current frame into a sliding window for nonlinear optimization to generate the posture of the current frame.

[0134] Example Robot

[0135] According to another aspect of the present application, a robot 80 is also provided. Figure 7 FIG. 1 is a block diagram of a robot according to an embodiment of the present application. Figure 7 As shown, the robot 80 according to the embodiment of the present application includes: a robot body 81; an RGB-D camera module 82 installed on the robot body; an inertial sensor 83 installed on the robot body; and a data processing device 84 communicatively connected to the RGB-D camera module and the inertial sensor, wherein the data processing device is used to execute the visual SLAM-based composition method for a mobile robot, or the visual SLAM-based positioning method for a mobile robot.

Claims

1. A mapping method for a mobile robot based on visual SLAM, characterized in that: include: Step 1: Initialization to generate an initial key frame and an initial pose corresponding to the initial key frame; Step 2: Obtain an RGB image of the target environment and a depth image corresponding to the RGB image through an RGB-D camera module and convert the depth image into a depth point cloud, wherein each pixel point of the depth point cloud has an attribute value of multipath interference; Step 3: Generate a key frame sequence based on SLAM visual tracking, wherein step 3 includes: Step 31: based on the feature points of the RGB image of the current frame, the feature points of the RGB image of the previous frame adjacent to the current frame, and the pose data provided by the inertial sensor corresponding to the current frame and the previous frame respectively, obtaining the estimated pose corresponding to the current frame; Step 32: inputting the estimated poses of the current frame and S frames of image frames continuous with the current frame into a sliding window for nonlinear optimization to generate the pose of the current frame; Step 33: generating an inter-frame pose transformation matrix based on the pose of the current frame and the pose of the previous frame; Step 34: Based on the inter-frame pose transformation matrix and the depth point cloud of the previous frame, correct the pixel points whose attribute values ​​of multipath interference in the depth point cloud of the current frame are higher than a preset threshold, so as to obtain a corrected depth point cloud of the current frame; Step 35: iteratively executing steps 31 to 34 to obtain the key frame sequence, each key frame in the key frame sequence having a corrected depth point cloud; and Step 4: Generate a target map based on the corrected depth point clouds and poses of all key frames in the key frame sequence, wherein step 4 includes: Generate a probability grid map based on the corrected depth point clouds and poses of all key frames in the key frame sequence; Generate a point cloud map based on the corrected depth point clouds of all key frames in the key frame sequence; The point cloud map and the probability grid map are fused to generate the target map.

2. The mapping method for a mobile robot based on visual SLAM according to claim 1, wherein: The inertial sensor comprises an inertial measurement unit and a wheel encoder, and the posture data comprises first posture data obtained by the inertial measurement unit and second posture data obtained by the wheel encoder; Wherein, step 31: based on the feature points of the RGB image of the current frame and the feature points of the RGB image of the previous frame adjacent to the current frame, and the pose data provided by the inertial sensor corresponding to the current frame and the previous frame respectively, obtaining the estimated pose corresponding to the current frame, comprises: Based on the following loss function, the fusion pose corresponding to the current frame is obtained, wherein the loss function is C(x_k)=cost_imu+cost_wheel+cost_camera; Among them, cost_imu represents the integral loss function of the first pose data corresponding to the current frame and the previous frame respectively with the estimated pose as the estimated quantity; cost_wheel represents the integral loss function of the second pose data corresponding to the current frame and the previous frame respectively with the estimated pose as the estimated quantity; cost_camera is the reprojection error generated based on the feature points of the RGB image of the current frame.

3. The mapping method for a mobile robot based on visual SLAM according to claim 2, wherein: Step 32: inputting the estimated poses of the current frame and S frames of image frames continuous with the current frame into a sliding window for nonlinear optimization to generate the pose of the current frame, including: Inputting the estimated poses of the current frame and S frames of image frames continuous with the current frame into the following optimization function, and calculating the marginalization matrix and displacement vector of the optimization function as the pose of the current frame; Wherein, the optimization function is C(x_i)=∑(cost_imu+cost_wheel+cost_camera).

4. The mapping method for a mobile robot based on visual SLAM according to claim 3, wherein: Step 34: Based on the inter-frame pose transformation matrix and the depth point cloud of the previous frame, correcting the pixel points whose attribute values ​​of multipath interference in the depth point cloud of the current frame are higher than a preset threshold, so as to obtain the corrected depth point cloud of the current frame, including: Filter out the pixel points to be replaced whose attribute values ​​of multipath interference are higher than a preset threshold in the depth point cloud of the current frame; Converting the pixel points corresponding to the pixel points to be replaced in the depth point cloud of the previous frame into replacement pixel points through the inter-frame pose transformation matrix; and The pixel points to be replaced in the depth point cloud of the current frame are replaced with the replacement pixel points to generate a corrected depth point cloud of the current frame.

5. The mapping method for a mobile robot based on visual SLAM according to claim 1, wherein: Step 3, further comprising: performing back-end optimization on the key frame sequence and / or the key frame; The backend optimization includes at least one of the following steps: Constructing co-viewing constraints of the feature points based on the feature points of the RGB image of the key frame and a bag-of-words model; Based on ICP matching, constructing pose constraints between consecutive key frames; and When a closed loop occurs, global nonlinear optimization is performed using the bundle adjustment method.

6. The mapping method for a mobile robot based on visual SLAM according to claim 1, wherein: The attribute value of the multipath interference is the confidence level of the multipath interference.

7. A positioning method for a mobile robot based on visual SLAM, characterized in that: include: Step 110: Generate the target map based on the visual SLAM-based mapping method for a mobile robot according to any one of claims 1 to 6; Step 120: Based on the local RGB image and depth point cloud of the target environment collected by the mobile robot, a local map is generated using the visual SLAM-based mapping method for a mobile robot according to any one of claims 1 to 6; Step 130: Determine the relocation initial pose of the mobile robot based on the nonlinear matching between the local map and the target map; Step 140: constructing a nonlinear optimization function based on the reprojection error of the feature points of the RGB image of the current frame on the point cloud map, the matching error between the corrected depth point cloud of the current frame and the probability grid map, and the inertial pose error determined based on the pose data provided by the inertial sensor corresponding to the current frame and the previous frame, respectively, and obtaining the estimated pose of the current frame through the nonlinear optimization function; as well as Step 150: Input the estimated poses of the current frame and S image frames continuous with the current frame into a sliding window for nonlinear optimization to generate the pose of the current frame.

8. A mapping device for a mobile robot based on visual SLAM, characterized in that: include: An initialization unit, used to perform step 1: initialization to generate an initial key frame and an initial pose corresponding to the initial key frame; A data acquisition unit, configured to execute step 2: acquiring an RGB image of a target environment and a depth image corresponding to the RGB image through an RGB-D camera module and converting the depth image into a depth point cloud, wherein each pixel point of the depth point cloud has an attribute value of multipath interference; The tracking unit is used to execute step 3: generating a key frame sequence based on SLAM visual tracking, wherein step 3 includes: Step 31: based on the feature points of the RGB image of the current frame, the feature points of the RGB image of the previous frame adjacent to the current frame, and the pose data provided by the inertial sensor corresponding to the current frame and the previous frame respectively, obtaining the estimated pose corresponding to the current frame; Step 32: inputting the estimated poses of the current frame and S frames of image frames continuous with the current frame into a sliding window for nonlinear optimization to generate the pose of the current frame; Step 33: generating an inter-frame pose transformation matrix based on the pose of the current frame and the pose of the previous frame; Step 34: Based on the inter-frame pose transformation matrix and the depth point cloud of the previous frame, correct the pixel points whose attribute values ​​of multipath interference in the depth point cloud of the current frame are higher than a preset threshold, so as to obtain a corrected depth point cloud of the current frame; Step 35: iteratively executing steps 31 to 34 to obtain the key frame sequence, each key frame in the key frame sequence having a corrected depth point cloud; and A map generation unit is used to execute step 4: generate a target map based on the corrected depth point clouds and poses of all key frames in the key frame sequence, wherein step 4 includes: generating a probability grid map based on the corrected depth point clouds and poses of all key frames in the key frame sequence; generating a point cloud map based on the corrected depth point clouds of all key frames in the key frame sequence; fusing the point cloud map and the probability grid map to generate the target map.

9. A positioning device for a mobile robot based on visual SLAM, characterized in that: include: A map construction unit, configured to execute step 110: generating the target map based on the visual SLAM-based mapping method for a mobile robot according to any one of claims 1 to 6; A local map construction unit, configured to execute step 120: based on the local RGB image and the depth point cloud of the target environment collected by the mobile robot, generate a local map using the visual SLAM-based mapping method for a mobile robot according to any one of claims 1 to 6; A relocation initial pose generating unit, configured to execute step 130: determining a relocation initial pose of the mobile robot based on a nonlinear match between the local map and the target map; A pose estimation unit is used to perform step 140: construct a nonlinear optimization function based on the reprojection error of the feature points of the RGB image of the current frame on the point cloud map, the matching error between the corrected depth point cloud of the current frame and the probability grid map, and the inertial pose error determined based on the pose data provided by the inertial sensor corresponding to the current frame and the previous frame, respectively, and obtain the estimated pose of the current frame through the nonlinear optimization function; as well as The nonlinear optimization unit is used to execute step S150, input the estimated poses of the current frame and S frame image frames continuous with the current frame into a sliding window for nonlinear optimization to generate the pose of the current frame.

10. A robot, characterized in that: include: Robot body; An RGB-D camera module installed on the robot body; An inertial sensor mounted on the robot body; as well as A data processing device communicatively connected to the RGB-D camera module and the inertial sensor, wherein the data processing device is used to execute the visual SLAM-based mapping method for a mobile robot as described in any one of claims 1 to 6, or the visual SLAM-based positioning method for a mobile robot as described in claim 7.

Images