Indoor semantic map construction method and device based on SLAM (Simultaneous Localization and Mapping) and storage medium

Abstract

The invention discloses an indoor semantic map construction method and device based on SLAM and a storage medium, and belongs to the technical field of environmental awareness, and the method comprises the steps: in the moving process of a robot, obtaining single-line laser radar data, odometer data and an RGB-D image; performing front-end map matching by using odometer data and single-line laser radar data to obtain a current pose of the robot; screening the key frame, and selecting the RGB-D image corresponding to the key frame according to the timestamp of the key frame; loopback detection and graph optimization are carried out in the moving process of the robot so as to update the current pose of the key frame; for each key frame, performing semantic segmentation on the RGB-D image by using the trained YOLOV8 segmentation model, outputting point cloud data with semantic information, and endowing the point cloud data with set color information; and converting the output point cloud data with the color and semantic information to a map coordinate system, and performing superposition to generate a semantic map. The method is low in calculation complexity, convenient to deploy and high in precision.

Description

Technical Field

[0001] The present invention belongs to the technical field of environmental perception, and in particular relates to a method, a device and a storage medium for constructing an indoor semantic map based on SLAM. Background Art

[0002] A semantic map is a map representation with enhanced semantic information. In addition to spatial information, it also contains information describing the categories and attributes of objects in the environment. This type of map is often used in fields such as robotics, autonomous driving, and augmented reality to help the system understand environmental interactions more comprehensively.

[0003] The current methods for generating semantic maps are: 1. Based on point cloud segmentation and classification, multi-line laser radar or RGB-D camera is usually used to collect point cloud data, and semantic labels are added through matching, segmentation, classification and other methods to generate 3D semantic maps. The main existing technologies include clustering, feature-based classification and convolutional upscaling network;

[0004] However, the current method is not conducive to accurate segmentation due to the limited density of laser points and the lack of color information when it is based on point cloud segmentation and classification. In addition, the three-dimensional point cloud matching algorithm has a large amount of data and high computing resource usage.

[0005] 2. Based on deep learning for image and retrograde segmentation and target detection, classify and label different objects in the environment. Common models include RCN (Fully Convolutional Network), Mask R-CNN, DeepLab, etc.

[0006] However, this method can only recognize and segment single-frame images within the camera's field of view, while the camera poses under multiple different perspectives are difficult to calculate, and it is impossible to construct a global semantic map.

[0007] 3. Based on the graph structure and topological map method, the environment is represented as nodes and edges. Nodes represent the semantic information of specific locations, and edges represent the spatial relationship between adjacent locations.

[0008] However, currently this method can only simply describe the connection relationship between key points, does not contain geometric information, cannot build a global semantic map, and is prone to missing important data in the environment. Summary of the invention

[0009] In view of the deficiencies in the prior art, the present invention provides a SLAM-based indoor semantic map construction method, device and storage medium, which have low computational complexity, convenient deployment and high precision.

[0010] The present invention provides the following technical solutions:

[0011] In a first aspect, a method for constructing an indoor semantic map based on SLAM is provided, comprising:

[0012] During the robot's movement, single-line lidar data, odometer data, and RGB-D images are acquired;

[0013] Use odometer data and single-line lidar data to perform front-end map matching to obtain the robot's current position and posture;

[0014] During the robot movement, the key frames are filtered according to the changes in displacement and angle, and the corresponding RGB-D images are selected according to the timestamps of the key frames;

[0015] Perform loop detection and graph optimization during robot movement to update the current pose of the key frame;

[0016] For each key frame, the trained YOLOV8 segmentation model is used to perform semantic segmentation on the RGB-D image, output point cloud data with semantic information, and assign the set color information;

[0017] The output point cloud data with color and semantic information is converted to the map coordinate system and overlaid to generate a semantic map.

[0018] Optionally, the odometer data and the single-line laser radar data are used to perform front-end map matching to obtain the current position and posture of the robot. The specific process is as follows:

[0019] Use the odometer data to preliminarily predict the robot's current position and transform it into the coordinate system of the single-line laser radar;

[0020] Perform voxel filtering on the acquired single-line lidar data;

[0021] The initial predicted pose converted to the single-line laser radar coordinate system is used as the initial pose, the point cloud data after voxel filtering is matched with the front-end map, and the initial pose is iteratively adjusted through the optimization algorithm to obtain the current pose of the robot; the objective function of iteratively adjusting the initial pose through the optimization algorithm is:

[0022]

[0023] Among them, K is the number of current point clouds, M smooth is a smoothing function using bicubic interpolation, h k is the point cloud data after voxel filtering, T ε To h k All points are transformed into the pose in the map coordinate system.

[0024] Optionally, the key frames are screened according to the displacement and angle changes, and the corresponding RGB-D images are selected according to the timestamps of the key frames, specifically:

[0025] When the displacement change and angle change compared with the previous key frame are both greater than the set threshold, the current frame is a key frame;

[0026] When a new key frame is generated, an RGB-D image with the smallest absolute value of the difference between the timestamp of the latest key frame and the timestamp of the latest key frame is found within the specified frame timestamp associated with the latest key frame, and it is used as the RGB-D image corresponding to the latest key frame.

[0027] Optionally, loop detection and graph optimization are performed during the robot movement to update the current position and posture of the key frame, specifically:

[0028] The key frame when the loop closure detection is successful is used as the closed-loop frame, and based on all the obtained closed-loop frames, all key frames are optimized to update the current poses of all key frames;

[0029] When updating the current pose of all key frames, the update is done by minimizing the cost function, as follows:

[0030]

[0031] Among them, T i is the i-th key frame pose, q j is the pose of the jth closed-loop frame, s(T i ,q j ) is based on T i and q j The relative pose of i and j is predicted computationally, Z ij is the relative position of i and j observed during the actual loop closure.

[0032] Optionally, for each key frame, the trained YOLOV8 segmentation model is used to perform semantic segmentation on the RGB-D image, output point cloud data with semantic information, and assign set color information, specifically: the RGB-D image includes an RGB image and a depth image;

[0033] Input the RGB image corresponding to the key frame into the YOLOV8 model for semantic segmentation, and output the category of the specified object in the image and the corresponding segmentation mask;

[0034] Using the depth image and the segmentation mask, all points in the mask are converted into a three-dimensional point cloud in the depth image, and the specified color is set for the generated three-dimensional point cloud; the formula for converting to a three-dimensional point cloud is:

[0035] x cam =zc ·(u mask -u0)·d x / f

[0036] y cam =z c ·(v mask -v0)·d y / f

[0037] z cam =z c

[0038] Among them, x ca, ,y cam and z cam To convert the coordinates of the point cloud to the RGB-D camera coordinate system, u mask and v mask is the pixel corresponding to the midpoint of the segmentation mask in the depth image, z c is the depth value corresponding to the pixel, u0, v0, d x and d y are the internal parameters of the RGB-D camera, and f is the focal length of the depth camera of the RGB-D camera.

[0039] Optionally, the method further includes: performing outlier filtering on the three-dimensional point cloud after the specified color, specifically:

[0040] For each point in the generated 3D point cloud, calculate the average distance between the point and all the points in its area.

[0041]

[0042] Where N(p) is the set of points in the domain of point p; q is any point in the domain of point p;

[0043] Determine the average distance Is it greater than the set value Threshold? If so, it is an outlier point removed; otherwise, it is not an outlier point retained;

[0044]

[0045] in, is the average distance of all points in the generated 3D point cloud, σ is the standard deviation of the 3D point cloud, and k is the adjustment parameter.

[0046] Optionally, when training the YOLOV8 segmentation model, the collected images need to be preprocessed and annotated; the preprocessing of the collected images includes rotating the collected images and adding noise.

[0047] Optionally, the output point cloud data with color and semantic information is converted into a map coordinate system, and the specific process is as follows:

[0048] The output point cloud data with color and semantic information is converted from the camera coordinate system to the robot coordinate system using the camera extrinsic parameters, and the point cloud data is converted from the robot coordinate system to the map coordinate system using the robot keyframe pose. The specific formula is:

[0049] P map =T ro b ot / map ·T cam / ro b ot ·P cam

[0050]

[0051] Among them, P map is the point cloud coordinate in the map coordinate system, P cam is the output point cloud coordinates with color and semantic information, T robot / map is the robot's key frame pose matrix, T cam / robot is the camera extrinsic matrix; R robot / map and t robot / map are the rotation matrix and translation vector of the robot in the map coordinate system, R cam / robot and t cam / robot are the rotation matrix and translation vector of the RGB-D camera in the robot coordinate system, respectively.

[0052] In a second aspect, a computer device is provided, comprising a processor and a memory; wherein, when the processor executes a computer program stored in the memory, the steps of the SLAM-based indoor semantic map construction method described in any one of the first aspects are implemented.

[0053] In a third aspect, a computer-readable storage medium is provided for storing a computer program; when the computer program is executed by a processor, the steps of the SLAM-based indoor semantic map construction method described in any one of the first aspects are implemented.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] The present invention calculates the real-time pose through the SLAM algorithm of a single-line laser radar with loop closure, accumulates key frames, synchronously obtains the depth data and RGB data in the most recent time, uses yolov8 to segment the RGB image, converts the effective area into a three-dimensional point cloud through the depth data, and stores the segmented three-dimensional in the map through the real-time pose calculated by SLAM and the camera external parameters to generate a global three-dimensional semantic map. The present invention only calculates the single-line laser radar data, which has low computational complexity and high precision compared to visual feature matching. It does not need to use a special pan-tilt or environment to add labels to calculate external parameters, and is easy to deploy; through loop detection, the accumulated error of laser matching can be eliminated, and the overall point cloud accuracy is higher; the fusion of laser, odometer, and RGB-D camera to reconstruct the three-dimensional point cloud not only has the advantages of low computational complexity and high precision of single-line laser SLAM, but also has visual-related color and semantic information. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is an overall flow chart of the indoor semantic map construction method based on SLAM of the present invention;

[0057] Figure 2 is a schematic diagram of matching the first frame of laser data and the map in the example given in Embodiment 2 of the present invention;

[0058] Figure 3 It is the grid map after the key frame is inserted in the example given in Embodiment 2 of the present invention;

[0059] Figure 4 is a schematic diagram of the structure of the pre-trained objects in the segmented image in the example given in Embodiment 2 of the present invention;

[0060] Figure 5 is a schematic structural diagram of outputting a point cloud of a specified object in an example given in Embodiment 2 of the present invention;

[0061] Figure 6 It is a map containing semantic information generated in the example given in Embodiment 2 of the present invention. DETAILED DESCRIPTION

[0062] The present invention will be further described below in conjunction with the accompanying drawings. The following examples are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the scope of protection of the present invention. It should be noted that the term "comprising" and any variation thereof in the specification and claims of the present invention are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0063] Example 1

[0064] like Figure 1 As shown, a method for constructing an indoor semantic map based on SLAM is provided, comprising the following steps:

[0065] Step S1: During the movement of the robot, single-line lidar data, odometer data, and RGB-D images are obtained.

[0066] The sensor data of lidar, odometer, RGB-D camera, etc. are prepared, synchronously input into the processor, and released to the outside through ROS messages. The internal parameters of the depth camera are calibrated in advance, and the external parameters between the depth camera and lidar and between the lidar and odometer need to be obtained (using the calibration algorithm or obtaining from the structural parameters). When the system program receives the first laser point, the initial point (0, 0) is set and the drawing of this coordinate system is started; the RGB-D image includes the RGB image and the depth image.

[0067] Step S2: Use the odometer data and single-line lidar data to perform front-end map matching to obtain the current position of the robot.

[0068] The specific process is:

[0069] Step S21: Use the odometer data to preliminarily predict the robot's current position and posture, and convert it to the coordinate system of the single-line laser radar.

[0070] Specifically, perform odometer pose prediction:

[0071] x t+1 =x t +v t cos(θ t )Δt

[0072] y t+1 =y t +v t sin(θ t )Δt

[0073] θ t+1 =θ t +ωΔt

[0074] Where (x t+1 ,y t+1 ,θ t+1 ) is the predicted pose of the odometer, (x t ,y t ,θ t ) is the odometer pose of the previous frame (v t ,ω) are the linear velocity and angular velocity respectively, and Δt is the time interval between two frames.

[0075] Convert to radar coordinate system:

[0076] x′ lidar =x t+1 +(x lidar cos(θ t+1 )-y lidar sin(θ t+1 ))

[0077] y′ lidar =y t+1 +(x lidar sin(θ t+1 )+y lidar cos(θ t+1 ))

[0078] θ lidat =θ t+1

[0079] In the formula (x′ lidar ,y′ lidar ,θ lidat ) is the current position of the laser radar, (x lidar ,y lidar ) is the laser radar position of the previous frame.

[0080] Step S22: performing voxel filtering on the acquired single-line laser radar data.

[0081] The principle of voxel filtering is to divide the current point cloud into voxels with fixed side lengths, and use the centroid of the voxel to replace all points in the voxel, so as to reduce the amount of calculation in subsequent matching and balance the weights of each point. The specific formula is:

[0082] Suppose that in the point cloud in three-dimensional space, point p i ={x i ,y i ,z i} is a point, and the voxel size is defined as {Δx, Δy, Δz}. For each point p i ={x i ,y i ,z i}The index of the point is calculated by the following formula:

[0083]

[0084] in To round down

[0085] Perform the above voxel division on all points in the point cloud and calculate the corresponding center of mass C

[0086]

[0087] At this point, all points in the point cloud will be replaced by the centroid C = (Cx ,C y ,C z ).

[0088] Step S23: Taking the initially predicted posture converted to the single-line laser radar coordinate system as the initial posture, matching the point cloud data after voxel filtering with the front-end map, and iteratively adjusting the initial posture through the optimization algorithm to obtain the current posture of the robot; the objective function of iteratively adjusting the initial posture through the optimization algorithm is:

[0089]

[0090] Among them, K is the number of current point clouds, M smooth is a smoothing function using bicubic interpolation, h k is the point cloud data after voxel filtering, T ε To h k All points of M are transformed into the pose in the map coordinate system. smooth With reference to the prior art, the optimization algorithm refers to the prior art, such as the ICP algorithm.

[0091] Step S3: During the movement of the robot, the key frames are screened according to the changes in displacement and angle, and the corresponding RGB-D images are selected according to the timestamps of the key frames.

[0092] When the displacement change and angle change compared with the previous key frame are both greater than the set threshold, the current frame is a key frame;

[0093] When a new key frame is generated, an RGB-D image with the smallest absolute value of the difference between the timestamp of the latest key frame and the timestamp of the latest key frame is found within the specified frame timestamp associated with the latest key frame, and it is used as the RGB-D image corresponding to the latest key frame.

[0094] Specifically, after the robot moves a certain distance, it decides which frames to set as key frames and which frames to discard based on the distance and angle screening method, in order to improve the efficiency and accuracy of the overall system.

[0095] Assume that the series of poses calculated above are {T1, T2, T3…T n}, for two consecutive poses T i ={x i ,y i ,θ i} and T j ={x j ,y j ,θ j}, the displacement judgment calculation formula is as follows

[0096]

[0097] The angle determination formula is as follows:

[0098] Δθ ij =|θ j -θ i |

[0099] Set the displacement threshold d threshold and the angle threshold θ threshold , when d ij >d threshold Or Δθ ij >θ threshold Save the frame as a keyframe, otherwise discard it.

[0100] Each time a new keyframe is generated, the RGB-D data of the nearest neighbor in time is found. The method formula is as follows: Calculate the absolute value of the difference between the timestamps of the most recent 30 frames and the timestamp of the latest keyframe time delta , select the frame of RGB data and Depth data with the smallest difference.

[0101] time delta =|time keyframe -time currentframe |

[0102] time in the formula delta Indicates the timestamp difference, time keyframe Indicates the key frame timestamp, time currentframe Indicates the timestamp of the current image. At this point, all the steps of front-end matching are completed, and the key frame has the timestamp, laser point cloud, pose, RGB image and depth map information, and the key frame information is passed to the back-end for subsequent processing.

[0103] Step S4: Perform loop detection and graph optimization during the robot's movement to update the current pose of the key frame.

[0104] Specifically, the key frame when the loop detection is successful is used as the closed-loop frame, and based on all the closed-loop frames obtained, all the key frames are optimized to update the current poses of all the key frames; when updating the current poses of all the key frames, they are updated by minimizing the cost function, and the formula is as follows:

[0105]

[0106] Among them, T i is the i-th key frame pose, q j is the pose of the jth closed-loop frame, s(T i ,q j ) is based on T i and q j The relative pose of i and j is predicted computationally, Zij is the relative position of i and j observed during the actual loop closure.

[0107] Perform loop detection and graph optimization in separate threads or nodes. Since front-end matching will inevitably produce cumulative errors, which will lead to inaccurate subsequent map generation, it is necessary to eliminate the cumulative errors by building graph optimization through loop detection. During loop detection, the robot detects whether it has returned to the place it has passed before. When the loop detection is successful, the system will associate the current position with the previous key frame position to form a closed loop.

[0108] The loop detection method can use the map matching method in step S2. If convergence occurs, the detection is considered successful, a closed loop is formed, and graph optimization is performed.

[0109] The SLAM system constructs the robot trajectory and environment into a graph, in which nodes represent the robot's posture and edges represent the constraints between postures. In this system, the postures of the key frames generated by the front-end matching are used as nodes, and edges represent the relationship between adjacent key frames.

[0110] Step S5: For each key frame, use the trained YOLOV8 segmentation model to perform semantic segmentation on the RGB-D image, output point cloud data with semantic information, and assign the set color information.

[0111] Specifically, the RGB image corresponding to the key frame is input into the YOLOV8 model for semantic segmentation, and the category of the specified object in the image and the corresponding segmentation mask are output; using the depth image and the segmentation mask, all points in the mask are converted into a three-dimensional point cloud in the depth image, and the specified color is set for these point clouds. The formula for converting to a three-dimensional point cloud is:

[0112] x cam =z c ·(u mask -u0)·d x / f

[0113] y cam =z c ·(v mask -v0)·d y / f

[0114] z cam =z c

[0115] Among them, x cam ,y cam and z cam To convert the coordinates of the point cloud to the RGB-D camera coordinate system, u mask and v mask is the pixel corresponding to the midpoint of the segmentation mask in the depth image, zc is the depth value corresponding to the pixel, u0, v0, d x and d y are the internal parameters of the RGB-D camera, and f is the focal length of the depth camera of the RGB-D camera.

[0116] Furthermore, since the error during segmentation may cause the point cloud corresponding to the mask to have noise points of other objects, the 3D point cloud after the specified color is filtered for outliers. Specifically, for each point in the generated 3D point cloud, the average distance between the point and all the points in its area is calculated.

[0117]

[0118] Where N(p) is the set of points in the domain of point p; q is any point in the domain of point p;

[0119] Determine the average distance Is it greater than the set value Threshold? If so, it is an outlier point removed; otherwise, it is not an outlier point retained;

[0120]

[0121] in, is the average distance of all points in the generated 3D point cloud, σ is the standard deviation of the 3D point cloud, and k is an adjustment parameter, which is generally 1 or 2.

[0122] In some other embodiments, since the segmentation edge has noise and may bring in other objects, the segmentation edge is corroded and then all points in the mask are converted into a three-dimensional point cloud in the corresponding depth image.

[0123] Furthermore, the RGB image information of the key frame after image optimization is input into the trained YOLO segmentation model to obtain a segmentation mask with semantic information. The collected data needs to be preprocessed and annotated before training. The collected images are first enhanced to improve the generalization ability of the model. This application uses rotation and noise enhancement to enhance the image.

[0124] Specifically, let the original image coordinates be (x, y), the rotation angle be θ, and the rotated coordinates be (x′, y′) which can be calculated by the following formula:

[0125] x′=x*cosθ-y*sinθ

[0126] y′=x*sinθ+y*cosθ

[0127] The rotated empty space needs to be filled.

[0128] Assume that image I(x,y) is the original image and I′(x,y) is the image with added noise. The formula is:

[0129] I′(x,y)=I(x,y)+N(0,σ 2 )

[0130] In the formula, N(0,σ 2 ) means the mean is 0 and the variance is σ 2 The enhanced image and the original image are normalized, that is, the bounding box of the object to be segmented is marked in the image, and the annotated text and image datasets are passed into the YOLOV8 pre-trained model for training and output of the segmentation model.

[0131] Step S6: Convert the output point cloud data with color and semantic information into a map coordinate system and overlay them to generate a semantic map.

[0132] The specific process is: use the camera external parameters to convert the output point cloud data with color and semantic information from the camera coordinate system to the robot coordinate system, and use the robot key frame pose to convert the point cloud data from the robot coordinate system to the map coordinate system. The specific formula is:

[0133] P map =T ro b ot / map ·T cam / ro b ot ·P cam

[0134]

[0135] Among them, P map is the point cloud coordinate in the map coordinate system, P cam is the output point cloud coordinates with color and semantic information, T robot / map is the robot's key frame pose matrix, T cam / robot is the camera extrinsic matrix; R robot / map and t robot / map are the rotation matrix and translation vector of the robot in the map coordinate system, R cam / robot and t cam / robot are the rotation matrix and translation vector of the RGB-D camera in the robot coordinate system, respectively.

[0136] All keyframe point clouds converted to the map coordinate system are superimposed to generate a point cloud map containing semantic information. Then, the point cloud is subjected to voxel filtering in step 2 to generate a semantic map. The optimized slam map and point cloud map are saved as files (pcd or ply format) for subsequent loading and viewing.

[0137] Example 2

[0138] Provide a specific example. In the actual test, the turtlrbot chassis, Wanji 716 radar, i5 processor, ubuntu22.04, and ros2 humble version environment are used to build a three-dimensional point cloud in the test room.

[0139] 1. Sensor data preparation

[0140] According to the structure of the machine, the rotation matrix of the robot's depth camera coordinate system relative to the center coordinate system of the chassis is obtained as follows: The translation matrix is When running the chassis-related ros driver, the lidar, odometer, and depth camera data can be read normally.

[0141] 2. Laser matching

[0142] Control the robot movement, start building the map and save the keyframes. Take the previous frame pose as the starting point. After the laser point cloud is filtered by voxels, calculate the pose predicted by the odometer. Use the predicted pose as the initial pose, match the point cloud and the map, and output the current pose, such as Figure 2 shown.

[0143] 3. Keyframe saving

[0144] Calculate the pose difference of the previous key frame of the current frame, and use the motion filter to determine whether to insert it as a key frame. The displacement threshold used by the motion filter in this example is 0.5m, and the rotation angle threshold is 5°. If the conditions are met, insert a new key frame id = 1, 2, 3..., if not, return to the step to continue matching the next frame. The chassis moves to the rotation matrix of Translation Matrix , 120 keyframes have been inserted. The grid map now displays as follows: Figure 3 shown.

[0145] 4. Save the depth map and RGB map synchronously

[0146] Find the most recent RGB data and depth data corresponding to the 199th keyframe timestamp, and record the keyframe pose, point cloud, color image, and depth image information.

[0147] 5. Loop Detection and Graph Optimization

[0148] The loop detection method can use the map matching method in step S2. If convergence occurs, the detection is considered successful, a closed loop is formed, and graph optimization is performed. The SLAM system constructs the robot trajectory and environment into a graph, where nodes represent the robot's posture and edges represent the constraints between postures. In this system, the posture of the key frame generated by the front-end matching is used as a node, and the edge represents the relationship between adjacent key frames. The latest key frame is matched with the key frame in the history.

[0149] 6. Yolo segmentation keyframe point cloud

[0150] Use the trained yolov8 model to segment the pre-trained objects in the image, such as Figure 4 The images of the chair and the table are segmented as shown in the figure. The segmented images are used to generate a 3D point cloud based on the camera internal parameters, and the point cloud of the specified object is output, as shown in the figure. Figure 5 shown.

[0151] 7. Generate semantic maps

[0152] Convert the point cloud from the camera coordinate system to the map coordinate system to generate a map containing semantic information, such as Figure 6 As shown, the generated semantic map and raster map are saved to the hard disk for subsequent use.

[0153] Example 3

[0154] The present invention provides a computer device, comprising a processor and a memory; wherein the processor implements the steps of the above-mentioned SLAM-based indoor semantic map construction method when executing a computer program stored in the memory.

[0155] For more specific processes of the above method, please refer to the corresponding contents disclosed in the aforementioned embodiments, which will not be repeated here.

[0156] Example 4

[0157] The present invention provides a computer-readable storage medium for storing a computer program; when the computer program is executed by a processor, the steps of the above-mentioned SLAM-based indoor semantic map construction method are implemented.

[0158] For more specific processes of the above method, please refer to the corresponding contents disclosed in the aforementioned embodiments, which will not be repeated here.

[0159] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device and storage medium disclosed in the embodiment, since they correspond to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0160] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention or some parts of the embodiments.

[0161] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A method for constructing an indoor semantic map based on SLAM, characterized in that: include: During the robot's movement, single-line lidar data, odometer data, and RGB-D images are acquired; Use odometer data and single-line lidar data to perform front-end map matching to obtain the robot's current position and posture; During the robot movement, the key frames are filtered according to the changes in displacement and angle, and the corresponding RGB-D images are selected according to the timestamps of the key frames; Perform loop detection and graph optimization during robot movement to update the current pose of the key frame; For each key frame, the trained YOLOV8 segmentation model is used to perform semantic segmentation on the RGB-D image, output point cloud data with semantic information, and assign the set color information; The output point cloud data with color and semantic information is converted to the map coordinate system and overlaid to generate a semantic map.

2. The method for constructing an indoor semantic map based on SLAM according to claim 1, characterized in that: The odometer data and single-line laser radar data are used to perform front-end map matching to obtain the current position of the robot. The specific process is as follows: Use the odometer data to preliminarily predict the robot's current position and transform it into the coordinate system of the single-line laser radar; Perform voxel filtering on the acquired single-line lidar data; The initial predicted pose converted to the single-line laser radar coordinate system is used as the initial pose, the point cloud data after voxel filtering is matched with the front-end map, and the initial pose is iteratively adjusted through the optimization algorithm to obtain the current pose of the robot; the objective function of iteratively adjusting the initial pose through the optimization algorithm is: Among them, K is the number of current point clouds, M smooth is a smoothing function using bicubic interpolation, h k is the point cloud data after voxel filtering, T ε To h k All points are transformed into the pose in the map coordinate system.

3. The method for constructing an indoor semantic map based on SLAM according to claim 1, characterized in that: The key frames are screened according to the displacement and angle changes, and the corresponding RGB-D images are selected according to the timestamps of the key frames, specifically: When the displacement change and angle change compared with the previous key frame are both greater than the set threshold, the current frame is a key frame; When a new key frame is generated, an RGB-D image with the smallest absolute value of the difference between the timestamp of the latest key frame and the timestamp of the latest key frame is found within the specified frame timestamp associated with the latest key frame, and it is used as the RGB-D image corresponding to the latest key frame.

4. The method for constructing an indoor semantic map based on SLAM according to claim 1, characterized in that: The loop detection and graph optimization are performed during the robot movement to update the current position of the key frame, specifically: The key frame when the loop closure detection is successful is used as the closed-loop frame, and based on all the obtained closed-loop frames, all key frames are optimized to update the current poses of all key frames; When updating the current pose of all key frames, the update is done by minimizing the cost function, as follows: Among them, T i is the i-th key frame pose, q j is the pose of the jth closed-loop frame, s(T i ,q j ) is based on T i and q j The relative pose of i and j is predicted computationally, Z ij is the relative position of i and j observed during the actual loop closure.

5. The method for constructing an indoor semantic map based on SLAM according to claim 1, characterized in that: For each key frame, the trained YOLOV8 segmentation model is used to perform semantic segmentation on the RGB-D image, output point cloud data with semantic information, and assign set color information, specifically: the RGB-D image includes an RGB image and a depth image; Input the RGB image corresponding to the key frame into the YOLOV8 model for semantic segmentation, and output the category of the specified object in the image and the corresponding segmentation mask; Using the depth image and the segmentation mask, all points in the mask are converted into a three-dimensional point cloud in the depth image, and the specified color is set for the generated three-dimensional point cloud; the formula for converting to a three-dimensional point cloud is: x cam =z c ·(u mask -u0)·d x / f y cam =z c ·(v mask -v0)·d y / f With cam =from c Among them, x cam ,y cam and z cam To convert the coordinates of the point cloud to the RGB-D camera coordinate system, u mask and v mask is the pixel corresponding to the midpoint of the segmentation mask in the depth image, z c is the depth value corresponding to the pixel, u0, v0, d x and d y are the internal parameters of the RGB-D camera, and f is the focal length of the depth camera of the RGB-D camera.

6. The method for constructing an indoor semantic map based on SLAM according to claim 5, characterized in that: Also includes: Perform outlier filtering on the 3D point cloud after specifying the color, specifically: For each point in the generated 3D point cloud, calculate the average distance between the point and all the points in its area. Where N(p) is the set of points in the domain of point p; q is any point in the domain of point p; Determine the average distance Is it greater than the set value Threshold? If so, it is an outlier point removed; otherwise, it is not an outlier point retained; in, is the average distance of all points in the generated 3D point cloud, σ is the standard deviation of the 3D point cloud, and k is the adjustment parameter.

7. The method for constructing an indoor semantic map based on SLAM according to claim 1, characterized in that: When training the YOLOV8 segmentation model, the collected images need to be preprocessed and annotated; the preprocessing of the collected images includes rotating the collected images and adding noise.

8. The method for constructing an indoor semantic map based on SLAM according to claim 1, characterized in that: The specific process of converting the output point cloud data with color and semantic information into the map coordinate system is as follows: The output point cloud data with color and semantic information is converted from the camera coordinate system to the robot coordinate system using the camera extrinsic parameters, and the point cloud data is converted from the robot coordinate system to the map coordinate system using the robot keyframe pose. The specific formula is: P map =T robot / map ·T cam / robot ·P cam Among them, P map is the point cloud coordinate in the map coordinate system, P cam is the output point cloud coordinates with color and semantic information, T robot / map is the robot's key frame pose matrix, T cam / robot is the camera extrinsic matrix; R robot / map and t robot / map are the rotation matrix and translation vector of the robot in the map coordinate system, R cam / robot and t cam / robot are the rotation matrix and translation vector of the RGB-D camera in the robot coordinate system, respectively.

9. A computer device, characterized in that: It comprises a processor and a memory; wherein, when the processor executes the computer program stored in the memory, the steps of the SLAM-based indoor semantic map construction method described in any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium, characterized in that: Used to store computer programs; when the computer programs are executed by the processor, the steps of the SLAM-based indoor semantic map construction method according to any one of claims 1 to 8 are implemented.

Images