A semantic map construction and navigation method for mobile robots

Abstract

The present invention discloses a method for constructing and navigating a semantic map of a mobile robot. The present invention comprises the following steps: first, constructing a two-dimensional occupancy grid map, collecting environmental information through a laser radar, an inertial measurement unit and an RGBD camera deployed on the robot, and completing the map construction of the indoor environment; then, performing target detection and coordinate conversion on the house number of the room through deep learning; then, based on the two-dimensional occupancy grid map and the position information of the house number of the room, performing room semantic segmentation to form a two-dimensional grid semantic map; finally, based on the two-dimensional occupancy grid map and the semantic map, completing the semantic parsing of the starting point and the target point, performing A* global path planning based on key points, and navigating the robot to the target point. The present invention constructs a semantic map of the room in an indoor environment, provides rich semantic information for robot navigation and human-computer interaction, and the A* path planning based on key points effectively avoids the problem of the robot being too close to obstacles.

Description

Technical Field

[0001] The present invention belongs to the technical field of robots and relates to a map construction method for a mobile robot, and in particular to a semantic map construction and navigation method for a mobile robot. Background Art

[0002] With the vigorous development of robotics technology, service robots are gradually entering people's lives, improving and improving people's quality of life in human-machine coexistence environments such as hotels, office buildings, shopping malls, and schools. Especially under the requirements of reducing contact and keeping distance during the COVID-19 pandemic, it is becoming increasingly important for robots to replace manual labor in tasks such as delivery and handover.

[0003] With the development of robotics, higher requirements are placed on robots' understanding of the environment and human-computer interaction. Semantic maps are an enhanced representation of the environment that contains both geometric information and high-level qualitative features. Modern robots navigate in their environment by calculating their own poses in metric maps. Most semantic mapping methods add semantic information to metric semantic maps. Semantic maps contain high-level features that model human concepts about places, objects, shapes, and even the associations between all these objects, while metric maps retain all the geometric features that robots should understand in order to achieve safe navigation in the surrounding environment. In the application of building service robots, rooms are crucial as environmental semantics, so room semantic segmentation based on occupancy grid maps has become an important part of the semantic map of service robots.

[0004] Room segmentation methods mainly include those based on morphology, distance transformation, Voronoi diagram, and spectral clustering. However, simply relying on two-dimensional grid maps for image analysis and processing is difficult to achieve high accuracy and cannot effectively utilize the robot's own multi-sensor data. The path planned by the traditional A* path planning algorithm is too close to obstacles, especially when the robot passes through the room door, it is too close to one side of the door frame, which reduces the safety of the robot. Summary of the invention

[0005] In order to solve the problems and defects existing in the background technology, the purpose of the present invention is to provide a mobile robot mapping and navigation method, including the construction of a room-based semantic map and a key point-based navigation method.

[0006] The technical solution of the present invention is as follows:

[0007] Step 1: The laser radar, inertial measurement unit and RGBD camera deployed on the mobile robot collect indoor environment information, obtain environmental point cloud information, odometer information and environmental image respectively, construct the indoor environment map according to the environmental point cloud information and odometer information, and obtain a two-dimensional occupancy grid map;

[0008] Step 2: Input all images containing house numbers in the environment image into the house number recognition neural network to recognize the house numbers, obtain the corresponding room house numbers, and then convert the coordinates of each room house number to obtain the location information of each room house number;

[0009] Step 3: Perform room semantic segmentation on the two-dimensional occupancy grid map according to the house numbers and corresponding location information of each room to obtain a room-based two-dimensional occupancy grid semantic map;

[0010] Step 4: According to the starting point and the target point and the two-dimensional occupancy grid semantic map based on the room, the robot is subjected to key point-based global path planning to obtain the planned path. The robot navigates from the starting point to the target point according to the planned path.

[0011] The house number recognition neural network is a trained YOLOv3 network obtained by training the YOLOv3 network using a building room house number data set.

[0012] The step 3 is specifically as follows:

[0013] 3.1) Binarize the two-dimensional occupancy grid map to obtain a binary map, determine the door frame area image of the corresponding room according to the house number and the corresponding position information of each room, use the minimum distance method to perform room door frame detection on the door frame area image of each room, obtain the left and right door frame positions of each room, and then determine the corresponding door inside and outside key points of each room according to the left and right door frame positions of each room;

[0014] 3.2) Determine the dividing line between each room and corridor in the two-dimensional occupancy grid map according to the left and right door frame positions of each room;

[0015] 3.3) According to the house number and corresponding location information of each room as well as the key points inside and outside the door, the connected area detection method is used to perform room segmentation on the two-dimensional occupancy grid map containing the dividing lines between each room and the corridor, and the semantic map of each room and the corridor map are obtained, thereby obtaining a room-based two-dimensional occupancy grid semantic map.

[0016] The step 4 is specifically as follows:

[0017] Firstly, the starting point and the target point are semantically parsed according to the room-based two-dimensional occupancy grid semantic map to obtain the key points of each path. Then, according to the key points of each path, global path planning based on the key points is performed on the room-based two-dimensional occupancy grid semantic map to obtain the planned path. Finally, the robot navigates from the starting point to the target point according to the planned path.

[0018] In step 4, each key point of the path includes a starting point, a target point, a key point inside and outside the door of the room corresponding to the starting point, and a key point inside and outside the door of the room corresponding to the target point.

[0019] In step 4, the A* global path planning method is used to perform global path planning on the robot based on key points.

[0020] The beneficial effects of the present invention are:

[0021] (1) Using deep learning methods to identify room numbers in real time, it is possible to effectively obtain the information of each room number and generate room location information and room number information.

[0022] (2) Based on the two-dimensional occupancy grid map and the room location and house number information, the image is segmented into rooms to obtain a room semantic map based on the room door information, providing rich semantic information for subsequent navigation.

[0023] (3) The A* global path planning based on key points can effectively prevent the robot from getting too close to obstacles during navigation and efficiently perform navigation tasks in indoor building office environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a schematic diagram of the overall process of the present invention.

[0025] Figure 2 It is a schematic diagram of the mapping process of the present invention.

[0026] Figure 3 It is a schematic diagram of the room segmentation process of the present invention.

[0027] Figure 4 It is a schematic diagram of the key point-based path planning process of the present invention.

[0028] Figure 5 It is a schematic diagram of the path planning result based on key points of the present invention. DETAILED DESCRIPTION

[0029] The present invention is further described in detail below in conjunction with the preferred embodiments of the present invention. However, this should not be understood as the scope of the above subject matter of the present invention being limited to the following embodiments, and all technologies realized based on the content of the present invention belong to the scope of the present invention.

[0030] The principles and features of the present invention are described below in conjunction with the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0031] like Figure 1 As shown, the present invention comprises the following steps:

[0032] Step 1: The process diagram of the two-dimensional occupancy grid semantic map of the room is as follows Figure 2 As shown in the figure, the laser radar, inertial measurement unit IMU and RGBD camera deployed on the mobile robot collect indoor environment information respectively, and obtain environmental point cloud information, odometer information and environmental image respectively. Among them, the RGBD camera adopts PrimeSense Carmine 1.09ShortRange somatosensory depth camera. The Zhang Zhengyou calibration method is used to calibrate the camera intrinsic parameters to obtain the camera's intrinsic parameters and distortion parameters, thereby obtaining the transformation matrix from the pixel coordinate system to the camera coordinate system; the eye-outside-hand method is used to calibrate the external parameters to obtain the transformation matrix from the camera coordinate system to the robot coordinate system, including the rotation matrix R and the translation matrix T. In the process of mapping, the visual recognition system is turned on at the door of each room to identify the room number, and the normalized coordinates (u, v, 1) of the recognized result in the pixel coordinate system are analyzed in real time. The coordinates are converted into the coordinates (x c ,y c ,z c ), and then transform the coordinates into the robot coordinate system according to the camera external parameter calibration results (x w ,y w ,z w ), and then converted into coordinates in the map coordinate system (x m ,y m ,0) and recorded as a YAML file. According to the environmental point cloud information and odometer information, the indoor environment map is constructed using the Gmapping algorithm based on 2D laser radar to obtain a two-dimensional occupancy grid map;

[0033] Step 2: Input the images containing house numbers in the environment images captured by the RGBD camera into the house number recognition neural network to recognize the house numbers, obtain the corresponding room house numbers, and then convert the coordinates of the house number positions in the pixel coordinate system into the coordinates in the robot coordinate system to obtain the position information of the house numbers in each room;

[0034] The house number recognition neural network is a trained YOLOv3 network obtained by training the YOLOv3 network using the building room house number dataset. The building room house number dataset consists of pre-collected and pre-processed images containing house numbers and corresponding house number labels.

[0035] Step 3: The flow chart of room semantic segmentation is as follows Figure 3 As shown, the room semantic segmentation is performed on the two-dimensional occupancy grid map according to the house numbers of each room and the corresponding location information to obtain a two-dimensional occupancy grid semantic map based on the room;

[0036] Step 3 is as follows:

[0037] 3.1) Binarize the two-dimensional occupancy grid map to obtain a binary map, where white represents passable areas and black represents obstacles (inaccessible areas). Determine the door frame area image of the corresponding room based on the house number and the corresponding position information of each room. In the specific implementation, the image of the rectangular area of ​​±1.25m of the house number position in the two-dimensional occupancy grid map is used as the door frame area image of the corresponding room. The door frame of each room is detected using the minimum distance method to obtain the left and right door frame positions of each room. Then, the key points inside and outside the door corresponding to each room are determined based on the left and right door frame positions of each room. In the specific implementation, the two positions on the perpendicular midline of the line connecting the left and right door frame positions of each room in the two-dimensional occupancy grid map (i.e. the position of the top door frame) at a distance of 0.4m from the current line are recorded as the door inner key point door_in and the door outer key point door_out;

[0038] 3.2) Determine the dividing lines between each room and the corridor in the two-dimensional occupancy grid map according to the left and right door frame positions of each room; specifically, connect all left and right door frame points in the two-dimensional occupancy grid map to obtain the dividing lines between each room and the corridor, which are used to separate the room and the corridor.

[0039] 3.3) According to the house number of each room and the corresponding location information and the key points inside and outside the door, the two-dimensional occupancy grid map containing the dividing lines between each room and the corridor is segmented using the connected area detection method to obtain the semantic map of each room and the corridor map. Different grayscale values ​​are used for the semantic map of each room and the corridor map, including but not limited to using pgm format files for saving. The semantic information of the two-dimensional occupancy grid semantic map based on the room is recorded and saved using a YAML file. The semantic information of the two-dimensional occupancy grid semantic map based on the room includes the room name information, room location information, and key point location information inside and outside the door of each room. Thus, a two-dimensional occupancy grid semantic map based on the room is obtained.

[0040] Step 4: The flowchart of global path planning based on key points is as follows Figure 4 As shown, according to the starting point and the target point and the two-dimensional occupancy grid semantic map based on the room, the robot is subjected to global path planning based on key points to obtain a planned path, and the robot navigates from the starting point to the target point according to the planned path.

[0041] Step 4 is as follows:

[0042] Firstly, the starting point and the target point are semantically parsed according to the room-based two-dimensional occupancy grid semantic map to obtain the key points of each path. Then, according to the key points of each path, global path planning based on the key points is performed on the room-based two-dimensional occupancy grid semantic map to obtain the planned path. Finally, the robot navigates from the starting point to the target point according to the planned path.

[0043] In step 4, each path key point includes a starting point, a target point, a key point inside and outside the door of the room corresponding to the starting point, and a key point inside and outside the door of the room corresponding to the target point.

[0044] In step 4, the A* global path planning method is used to perform global path planning for the robot based on key points.

[0045] Specifically, before path planning, the saved two-dimensional occupancy grid semantic map is read, and the robot is judged to be in a room or not in a room based on the real-time position detection of the robot, and the position of the target point is judged based on the target point information. If the robot's real-time position is in a room, when performing A* global path planning, the door_in and door_out key points of the room are inserted into the path in sequence; if the target point is in a room, the door_out and door_in key points of the room are inserted into the path in sequence. The planned path example is Figure 5 shown.

Claims

1. A method for constructing and navigating a semantic map of a mobile robot, characterized in that: The following steps are involved: Step 1: The laser radar, inertial measurement unit and RGBD camera deployed on the mobile robot collect indoor environment information, obtain environmental point cloud information, odometer information and environmental image respectively, construct the indoor environment map according to the environmental point cloud information and odometer information, and obtain a two-dimensional occupancy grid map; Step 2: Input all images containing house numbers in the environment image into the house number recognition neural network to recognize the house numbers, obtain the corresponding room house numbers, and then convert the coordinates of each room house number to obtain the location information of each room house number; Step 3: Perform room semantic segmentation on the two-dimensional occupancy grid map according to the house numbers and corresponding location information of each room to obtain a room-based two-dimensional occupancy grid semantic map; Step 4: According to the starting point and the target point and the two-dimensional occupancy grid semantic map based on the room, the robot is subjected to key point-based global path planning to obtain the planned path. The robot navigates from the starting point to the target point according to the planned path. The step 3 is specifically as follows: 3.1) Binarize the two-dimensional occupancy grid map to obtain a binary map, determine the door frame area image of the corresponding room according to the house number and corresponding position information of each room, use the minimum distance method to perform room door frame detection on the door frame area image of each room, obtain the left and right door frame positions of each room, and then determine the corresponding door inside and outside key points of each room according to the left and right door frame positions of each room; 3.2) Determine the dividing line between each room and corridor in the two-dimensional occupancy grid map according to the left and right door frame positions of each room; 3.3) According to the house number and corresponding location information of each room as well as the key points inside and outside the door, the connected area detection method is used to perform room segmentation on the two-dimensional occupancy grid map containing the dividing lines between each room and the corridor, and the semantic map of each room and the corridor map are obtained, thereby obtaining a room-based two-dimensional occupancy grid semantic map.

2. A method for constructing and navigating a semantic map of a mobile robot according to claim 1, characterized in that: The house number recognition neural network is a trained YOLOv3 network obtained by training the YOLOv3 network using a building room house number data set.

3. A method for constructing and navigating a semantic map of a mobile robot according to claim 1, characterized in that: The step 4 is specifically as follows: Firstly, the starting point and the target point are semantically parsed according to the room-based two-dimensional occupancy grid semantic map to obtain the key points of each path. Then, according to the key points of each path, global path planning based on the key points is performed on the room-based two-dimensional occupancy grid semantic map to obtain the planned path. Finally, the robot navigates from the starting point to the target point according to the planned path.

4. A method for constructing and navigating a semantic map of a mobile robot according to claim 3, characterized in that: In step 4, each key point of the path includes a starting point, a target point, a key point inside and outside the door of the room corresponding to the starting point, and a key point inside and outside the door of the room corresponding to the target point.

5. A method for constructing and navigating a semantic map of a mobile robot according to claim 1, characterized in that: In step 4, the A* global path planning method is used to perform global path planning on the robot based on key points.

Images