A robot semi-autonomous control method and system for complex environment

Abstract

The application discloses a kind of robot semi-autonomous control method and system for complex environment, method includes: obtaining the pose information of robot, three-dimensional point cloud data and depth image data;According to pose information, the depth image is fused with the laser radar data, obtain fused laser radar data, and according to fused laser radar data, obtain global three-dimensional occupancy dense map;Point cloud segmentation is carried out based on the three-dimensional point cloud data, and image is fused to obtain obstacle height estimation;Based on depth convolutional neural network, semantic segmentation is carried out on front-view color image, and the point cloud data corresponding to pixel is combined, to realize the identification of ground road by threshold discrimination to region height data obtained by semantic segmentation;According to current position information, a collision-free navigation path is formed with the three-dimensional map with semantic information constructed, for global path planning;On this basis, local path planning is carried out, and local path planning is used for dynamic obstacle avoidance in navigation process.

Description

Technical Field

[0001] This invention relates to a semi-autonomous control method and system for robots in complex environments, belonging to the field of logistics robot technology. Background Technology

[0002] Automated Guided Vehicle (AGV) is a type of wheeled mobile robot that is equipped with automatic guidance devices at the work site, can travel along a prescribed guidance path, and has safety protection and various transfer functions.

[0003] Currently, port logistics robots generally have low levels of intelligence and weak environmental perception capabilities. Technological bottlenecks are becoming increasingly apparent. Traditional programmable and remote-controlled robots, due to issues such as fixed programs and long response times, struggle to respond effectively to rapidly changing environments.

[0004] Secondly, the robot's operating environment is a typical unstructured scenario. The outdoor scene is vast and the environment is complex. Currently, the robot's environmental recognition ability is seriously insufficient, requiring human intervention. Wireless remote control is ineffective, and the personal safety of the operator cannot be guaranteed. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a semi-autonomous control method and system for port logistics robots in complex environments, which can realize synchronous positioning and mapping and autonomous navigation of port logistics robots.

[0006] To achieve the above objectives, the present invention is implemented using the following technical solution:

[0007] In a first aspect, the present invention provides a semi-autonomous control method for robots in complex environments, comprising the following steps:

[0008] Acquire the robot's pose information, 3D point cloud data, and depth image data, and calibrate the 3D point cloud data and depth image data;

[0009] Based on the pose information, the depth image is fused with the 3D point cloud data to obtain fused LiDAR data. Based on the fused LiDAR data, the RTABMAP algorithm is used to obtain a global 3D dense map.

[0010] Point cloud segmentation is performed based on the three-dimensional point cloud data, and the data is fused with the depth image to obtain obstacle height estimation, thus obtaining three-dimensional measurement data;

[0011] Semantic segmentation of the forward-looking color image is performed based on a deep convolutional neural network. Combined with the point cloud data corresponding to the pixels, the height data of the semantically segmented region is thresholded to achieve the recognition of the ground road and obtain scene recognition data.

[0012] The three-dimensional measurement data and the scene recognition data are added to the global three-dimensional occupancy dense map to obtain a global three-dimensional occupancy dense map with semantic information;

[0013] Autonomous navigation based on a known map is achieved using a globally 3D dense map with semantic information.

[0014] Furthermore, methods for acquiring 3D point cloud data and depth image data, and calibrating the 3D point cloud data and depth image data, include:

[0015] Three-dimensional point cloud data is acquired using a three-dimensional lidar, and depth images are obtained using a laser camera.

[0016] Obtain the pixel coordinates of the laser camera and the coordinates of the 3D lidar;

[0017] The camera intrinsic parameter matrix is ​​obtained by calibrating using a multi-triangle calibration method based on the pixel coordinates of the laser camera and the coordinates of the 3D lidar.

[0018] Furthermore, methods for acquiring pose data include:

[0019] The rotational speed of the wheels is obtained, and integral calculations are performed to obtain the position information of the logistics robot. At the same time, the difference in wheel speed is used to obtain the posture information of the logistics robot, and finally the posture data of the logistics robot is obtained.

[0020] Furthermore, the method for obtaining 3D measurement data by segmenting the point cloud based on the 3D point cloud data and fusing it with the depth image to obtain obstacle height estimation includes:

[0021] The region growing method is used to segment the point cloud, complete the static obstacle identification of the environmental point cloud, and obtain the obstacle region;

[0022] Based on the obstacle regions segmented from the point cloud data, the corresponding regions on the depth image are found and the ROI regions are extracted.

[0023] The ROI region is magnified, and the point cloud data corresponding to the obstacle region found in the depth image is used to calculate the coordinates of the lowest and highest points of the obstacle in the camera coordinates through the inverse transformation of the camera intrinsic parameter matrix, so as to obtain the height information of the obstacle.

[0024] Furthermore, methods for obtaining scene recognition data include: performing semantic segmentation on the forward-looking color image based on a deep convolutional neural network, combining the point cloud data corresponding to the pixels, and thresholding the height data of the semantically segmented regions.

[0025] Semantic segmentation of depth image data is performed based on deep convolutional neural networks. Combined with the 3D point cloud data of LiDAR corresponding to each pixel, threshold discrimination is applied to the height data of the semantically segmented region to achieve the recognition of ground roads.

[0026] Using ground roads as the initial safe travel area, other areas at the same distance from the safe travel area are identified. Based on the test results, a screening model is established to remove various types of areas from the safe travel area or issue hazard warnings.

[0027] Furthermore, methods for autonomous navigation based on known maps include:

[0028] Obtain current location information and target point;

[0029] Based on the global 3D dense map with semantic information, a collision-free navigation path is formed using a global path planning algorithm for global path planning;

[0030] Based on global path planning, a D* heuristic path search algorithm is used for local path planning; the local path planning is used for dynamic obstacle avoidance during navigation.

[0031] The D* algorithm is used to search for the optimal travel path, which is continuously updated during the journey until the navigation ends.

[0032] In a second aspect, the present invention provides a semi-autonomous robot control system, comprising:

[0033] Logistics robots;

[0034] A drive unit is mounted on the chassis of the logistics robot and connected to the tires of the logistics robot, used to drive the logistics robot to run;

[0035] A wheeled odometer, connected to the drive unit, is used to detect the position and posture information of the logistics robot.

[0036] A three-dimensional lidar sensor is installed on the logistics robot and connected to the core processor via a USB-to-serial port. It is used to scan the port environment and obtain three-dimensional lidar point cloud data.

[0037] A visible light camera, mounted on the logistics robot, is connected to the core processor via a USB-to-serial port and is used to collect depth image data of the port environment.

[0038] The core processor, installed on the logistics robot, is connected to the drive unit, wheeled odometer, 3D LiDAR sensor, and visible light camera. It is used to construct a global 3D dense map based on the pose information of the logistics robot, 3D LiDAR point cloud data, and depth image data, and to perform path planning and automatic navigation based on the global 3D dense map.

[0039] Furthermore, the core processor includes the following modules:

[0040] Input module: used to acquire the robot's pose information, 3D point cloud data and depth image data, and to calibrate the 3D point cloud data and depth image data;

[0041] Map generation module: used to fuse the depth image with the 3D point cloud data according to the pose information to obtain fused LiDAR data, and use the RTABMAP algorithm to obtain a global 3D dense map based on the fused LiDAR data.

[0042] Height estimation module: used to perform point cloud segmentation based on the 3D point cloud data, and fuse it with the depth image to obtain obstacle height estimation, thus obtaining 3D measurement data;

[0043] Scene recognition module: used to perform semantic segmentation on the front-looking color image based on a deep convolutional neural network, and combine the point cloud data corresponding to the pixels to perform threshold discrimination on the height data of the semantically segmented region to obtain scene recognition data;

[0044] Semantic map module: used to add the 3D measurement data and the scene recognition data to the global 3D occupancy dense map to obtain a global 3D occupancy dense map with semantic information;

[0045] Navigation module: Used for autonomous navigation based on a known map, using a globally 3D dense map with semantic information.

[0046] Furthermore, the height estimation module performs point cloud segmentation based on the 3D point cloud data and fuses it with the depth image to obtain an obstacle height estimate. The method for obtaining 3D measurement data includes:

[0047] The region growing method is used to segment the point cloud, complete the static obstacle identification of the environmental point cloud, and obtain the obstacle region;

[0048] Based on the obstacle regions segmented from the point cloud data, the corresponding regions on the depth image are found and the ROI regions are extracted.

[0049] The ROI region is magnified, and the point cloud data corresponding to the obstacle region found in the depth image is used to calculate the coordinates of the lowest and highest points of the obstacle in the camera coordinates through the inverse transformation of the camera intrinsic parameter matrix, so as to obtain the height information of the obstacle.

[0050] Furthermore, the scene recognition module performs semantic segmentation on the forward-looking color image based on a deep convolutional neural network, and combines the point cloud data corresponding to the pixels to perform threshold discrimination on the height data of the semantically segmented region, thereby obtaining scene recognition data. The method includes:

[0051] Semantic segmentation of depth image data is performed based on deep convolutional neural networks. Combined with the 3D point cloud data of LiDAR corresponding to each pixel, threshold discrimination is applied to the height data of the semantically segmented region to achieve the recognition of ground roads.

[0052] Using ground roads as the initial safe travel area, other areas at the same distance from the safe travel area are identified. Based on the test results, a screening model is established to remove various types of areas from the safe travel area or issue hazard warnings.

[0053] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0054] 1. This invention addresses the shortcomings of current port logistics robots, including low intelligence levels, weak environmental perception capabilities, and insufficient local autonomous navigation. Taking the complex working environments of port parks, such as cargo loading, unloading, storage, and transfer sites, as the target scenario, this invention proposes introducing Simultaneous Localization and Mapping (SLAM) and autonomous navigation technologies into port logistics robots. Multi-line LiDAR sensors, visible light cameras, and wheeled odometers are used to build the robot's spatial positioning and perception capabilities. Research is conducted on multi-sensor information registration, calibration, fusion, and positioning and navigation algorithms, and system software development is undertaken. A semi-autonomous control system is developed to improve the intelligence level and local autonomous navigation capabilities of port logistics robots, providing a suitable solution for upgrading existing port logistics robots.

[0055] 2. Research on registration and calibration technology between 3D point cloud data from LiDAR and depth image data from visible light cameras. This includes synchronous acquisition and control of 3D point cloud data and depth image data, calibration of 3D point cloud data and depth image data, and precise coordinate mapping. A parameterized model is established based on the scanning parameters of a multi-line LiDAR and the relative positional relationship between the visible light camera and the multi-line LiDAR. Based on sensor data calibration, a registration feature dataset consisting of corresponding feature point pairs is extracted from a customized structured scene. The model parameters are solved, and a registration mapping function is established. The registration algorithm between the 3D point cloud data output by the multi-line LiDAR and the pixels of the visible light depth image is studied, and a calibration and registration program module applicable to combat equipment is developed. This lays the foundation for further intelligent obstacle recognition, dynamic intelligent identification of safe movement areas, and hazard warning.

[0056] 3. Research on Dynamic Intelligent Recognition and Hazard Warning Technology for Safe Travel Zones. This research focuses on scene recognition and 3D measurement technology based on intelligent image semantic segmentation. It dynamically detects parameters such as height, width, and distance of obstacles, doorways, and passageways within the travel zone of port logistics robots. Based on the robot's structural features and pre-defined warning parameters, it intelligently identifies safe travel zones, providing the robot with an intuitive auxiliary judgment tool. Simultaneously, research is being conducted on dynamic recognition and warning technologies for obstacles and hazardous areas.

[0057] 4. Autonomous Navigation and Obstacle Avoidance Technology for Port Logistics Robots. Based on the construction of a global map, this study investigates the technologies for autonomous obstacle avoidance / crossing discrimination, path planning, and navigation control of logistics robots. This enables the logistics robot to intelligently plan its travel path and dynamically avoid obstacles according to the target location selected by the operator, achieving autonomous navigation. Attached Figure Description

[0058] Figure 1 This is a flowchart of the SLAM module and navigation module of the present invention;

[0059] Figure 2 A flowchart for constructing a 3D map with semantic information for this invention;

[0060] Figure 3 This is a flowchart of the calibration and registration algorithm between lidar 3D point cloud data and visible light image data. Detailed Implementation

[0061] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0062] Example 1:

[0063] This embodiment provides a semi-autonomous control method for robots in complex environments, including:

[0064] Acquire the robot's pose information, 3D point cloud data, and depth image data, and calibrate the 3D point cloud data and depth image data;

[0065] Based on the pose information, the depth image is fused with the 3D point cloud data to obtain fused LiDAR data. Based on the fused LiDAR data, the RTABMAP algorithm is used to obtain a global 3D dense map.

[0066] Point cloud segmentation is performed based on the three-dimensional point cloud data, and the data is fused with the depth image to obtain obstacle height estimation, thus obtaining three-dimensional measurement data;

[0067] Semantic segmentation of the front-view color image is performed based on a deep convolutional neural network. The point cloud data corresponding to the pixels is combined with the height data of the semantically segmented region for threshold discrimination to obtain scene recognition data.

[0068] The three-dimensional measurement data and the scene recognition data are added to the global three-dimensional occupancy dense map to obtain a global three-dimensional occupancy dense map with semantic information;

[0069] Autonomous navigation based on a known map is achieved using a globally 3D dense map with semantic information.

[0070] Specifically, the following steps are included:

[0071] Step 1: In a customized structured scene, calibrate the 3D point cloud data acquired by the LiDAR and the visible light image obtained by the camera. A multi-triangle calibration method is used. Let the pixel coordinates of the LiDAR be (u,v) and the coordinates of the 3D LiDAR be (x,y,z). The coordinate relationship between the two is as follows:

[0072]

[0073] In the formula: f u and f v These represent the effective focal lengths in the horizontal and vertical directions, respectively. u0 and v0 are the image center point coordinates; these four parameters are camera intrinsic parameters. R and T are the rotation and translation matrices between the camera and the LiDAR, respectively. ij This is the joint transformation matrix of intrinsic and extrinsic parameters.

[0074] There are 12 unknown parameters. To obtain these 12 parameters, at least 6 sets of equations must be obtained to solve for the unknowns. The calibration and registration algorithm flow is as follows: Figure 3 As shown.

[0075] Three points are randomly selected from the scanned point cloud set. It is determined whether these three points are collinear. If they are collinear, they are reselected. If they are not collinear, the plane equation corresponding to these three points is calculated. The vertex coordinates of the triangle in the two-dimensional image are further calculated, and the parameter values ​​in the transformation matrix are calculated based on the least squares method.

[0076] Step 2: Place the logistics robot in a port park environment with an unknown map, and control the robot's movement to build the map. The specific algorithm process is as follows:

[0077] Step 2.1: Obtain the wheel rotation speed from the wheel odometer connected to the drive device, perform integration calculation to obtain the position information of the logistics robot, and at the same time use the difference in wheel speed to obtain the posture information of the logistics robot, and finally obtain the posture data of the logistics robot.

[0078] Step 2.2: Obtain 3D point cloud data from the port environment scanned by the 3D lidar sensor;

[0079] Step 2.3: Obtain RGB image data and depth image data from the port environment captured by the visible light camera;

[0080] Step 2.4: The core processor uses the RTABMAP algorithm to obtain a global 3D dense map based on the pose data of the logistics robot provided by the wheel odometer, the 3D point cloud data scanned by the LiDAR, and the depth image data collected by the visible light camera.

[0081] Step 3: Based on the obtained 3D point cloud data from the LiDAR, perform point cloud segmentation and fuse it with the depth image from the visible light camera to obtain an obstacle height estimate, thus completing the 3D measurement.

[0082] For 3D point cloud segmentation, to meet the requirements of static obstacle recognition and real-time navigation, a region growing method is adopted to segment the point cloud and complete the static obstacle recognition of the environmental point cloud. This algorithm is based on the comparison of the angles between point normals, merging adjacent points that satisfy the smoothness constraint together and outputting them as a cluster of points, where each cluster of points is considered to belong to the same plane.

[0083] Based on the obstacle regions segmented from the point cloud data, the corresponding regions on the image are found and the Regions of Interest (ROIs) are extracted. The ROIs are then magnified. Using the point cloud data corresponding to the more accurate obstacle regions found on the image, the coordinates of the lowest and highest points of the obstacle in the camera coordinate system are calculated through the inverse transformation of the camera intrinsic parameter matrix to obtain the height information of the obstacle.

[0084] Step 4: Based on the RGB image data from the visible light camera obtained above, perform image semantic segmentation and safe travel area identification. This step uses a deep convolutional neural network (CNN) to perform semantic segmentation on the forward-looking color image data obtained from the visible light camera. Combined with the 3D point cloud data of the LiDAR corresponding to each pixel, a threshold judgment is applied to the height data of the semantically segmented areas to identify the ground roads, using the ground roads as the initial selected safe travel areas. Further, other areas at the same distance from the safe travel areas are identified, such as narrow doorways, passages, and obstacles. Based on test results, a screening model is established to remove such areas from the safe travel areas or issue a hazard warning.

[0085] Step 5: Add the 3D measurement data obtained in Step 3 and the scene recognition data obtained in Step 4 to the global map to obtain a global 3D occupancy dense map with semantic information.

[0086] Step 6: Based on a globally 3D dense map with semantic information, autonomous navigation based on a known map can be achieved. The specific algorithm process is as follows:

[0087] Step 6.1: Based on the global 3D dense map with semantic information and the fused LiDAR 3D point cloud data and depth image data, the adaptive Monte Carlo localization (AMCL) method is used to locate the current position of the logistics robot.

[0088] Step 6.2: Select the target point on the control interface. Based on the current position information, the constructed 3D dense map with semantic information, the robot pose information provided by the wheel odometry, the 3D point cloud data scanned by the LiDAR, and the depth image data collected by the visible light camera, the robot uses a global path planning algorithm, including but not limited to the A* algorithm, to guide the robot to the target position and form a collision-free navigation path for global path planning. To improve the search efficiency of the algorithm, after the path is generated, it is checked whether there is a line of sight between two points separated by the path. If there is, the intermediate point between the two points can be deleted to form a collision-free navigation path, thereby performing global path planning.

[0089] Step 6.3: Based on the global path planning in Step 6.2, the 3D point cloud data from the LiDAR scan, and the depth image data acquired by the visible light camera, a local path planning algorithm using the D* heuristic is employed. This local path planning is used for dynamic obstacle avoidance during navigation. Based on the identified safe travel area, the D* algorithm is used to search for the optimal travel path, which is continuously updated during the journey, ultimately achieving autonomous navigation for the logistics robot.

[0090] This invention helps improve the ease of operation of port logistics robots, enhances their ability to cope with complex environments such as cargo loading, unloading, storage, and transfer sites in port parks, improves the execution efficiency of port logistics tasks, and further reduces overall logistics costs. It has a very positive significance for ensuring the production safety of port logistics and improving port throughput efficiency.

[0091] Research on dynamic intelligent identification and hazard warning technology for safe travel areas. This research focuses on scene recognition and 3D measurement technology based on intelligent image semantic segmentation to dynamically detect parameters such as height, width, and distance of obstacles, doorways, and passageways within the travel area of ​​a port logistics robot. Based on the robot's structural features and pre-defined warning parameters, it intelligently identifies safe travel areas, providing the robot with an intuitive auxiliary identification tool. Simultaneously, research is being conducted on dynamic identification and warning technologies for obstacles and hazardous areas.

[0092] Autonomous navigation and obstacle avoidance technology for port logistics robots within line of sight. Based on the construction of a local map, this study investigates the implementation technology of autonomous obstacle avoidance / crossing judgment, path planning, and navigation control within line of sight. This enables the robot to intelligently plan its travel path and intelligently avoid obstacles according to the target position selected in the forward view image, achieving autonomous movement.

[0093] Example 2:

[0094] This embodiment provides a semi-autonomous control system for a logistics robot, including:

[0095] Logistics robots;

[0096] A drive unit is mounted on the chassis of the logistics robot and connected to the tires of the logistics robot, used to drive the logistics robot to run;

[0097] A wheeled odometer, connected to the drive unit of the logistics robot, is used to detect the position and posture information of the logistics robot.

[0098] A 3D LiDAR sensor is mounted on the logistics robot and connected to the core processor via a USB-to-serial adapter. The 3D LiDAR sensor scans the port environment to obtain 3D LiDAR point cloud data, which is used for subsequent map building and autonomous navigation; specifically, the 3D LiDAR sensor is a LeiShen 16-line LiDAR sensor.

[0099] A visible light camera, installed on the logistics robot, is connected to the core processor via a USB-to-serial port. It is used to collect depth image data and RGB image data of the port environment. The RGB images can be used for target detection to obtain semantic information, and the depth images can be used to obtain the target's attitude angle and distance information and perform calibration, registration, and information fusion with the 3D LiDAR point cloud data. Specifically, the visible light camera is a Kinect V2 camera.

[0100] The core processor, mounted on the logistics robot, is connected to the drive unit, wheeled odometer, 3D LiDAR sensor, and visible light camera. The core processor constructs a global 3D dense map based on the robot's pose information, 3D LiDAR point cloud data, depth image data, and RGB image data. Based on this map, it performs point cloud segmentation using region growing to complete 3D measurement and semantic segmentation using a deep convolutional neural network to complete scene recognition, adding semantic information to the global 3D dense map. A global path planning algorithm, including but not limited to the A* algorithm, is used for global path planning, and a D* heuristic path search algorithm is used for local path planning. The generated control signals control the logistics robot for autonomous navigation via the drive unit. Specifically, the core processor uses an NVIDIA TX2 high-performance embedded development board, running Ubuntu 18.04 and ROS Melodic operating systems. The core processor includes the following modules:

[0101] Input module: used to acquire the robot's pose information, 3D point cloud data and depth image data, and to calibrate the 3D point cloud data and depth image data;

[0102] Map generation module: used to fuse the depth image with the 3D point cloud data according to the pose information to obtain fused LiDAR data, and use the RTABMAP algorithm to obtain a global 3D dense map based on the fused LiDAR data.

[0103] Height estimation module: used to perform point cloud segmentation based on the 3D point cloud data, and fuse it with the depth image to obtain obstacle height estimation, thus obtaining 3D measurement data;

[0104] Scene recognition module: used to perform semantic segmentation on the front-looking color image based on a deep convolutional neural network, and combine the point cloud data corresponding to the pixels to perform threshold discrimination on the height data of the semantically segmented region to obtain scene recognition data;

[0105] Semantic map module: used to add the 3D measurement data and the scene recognition data to the global 3D occupancy dense map to obtain a global 3D occupancy dense map with semantic information;

[0106] Navigation module: Used for autonomous navigation based on a known map, using a globally 3D dense map with semantic information.

[0107] This embodiment also provides a semi-autonomous control method for a logistics robot, based on the aforementioned system, including the following steps:

[0108] Step 1: In a customized structured scene (i.e., system-based), calibrate the 3D point cloud data acquired by the LiDAR and the visible light image obtained by the camera. A multi-triangle calibration method is used. Let the pixel coordinates of the LiDAR be (u,v) and the coordinates of the 3D LiDAR be (x,y,z). The coordinate relationship between the two is as follows:

[0109]

[0110] In the formula: f u and f v These represent the effective focal lengths in the horizontal and vertical directions, respectively. u0 and v0 are the image center point coordinates; these four parameters are camera intrinsic parameters. R and T are the rotation and translation matrices between the camera and the LiDAR, respectively. ij This is the joint transformation matrix of intrinsic and extrinsic parameters.

[0111] There are 12 unknown parameters. To obtain these 12 parameters, at least 6 sets of equations must be obtained to solve for the unknowns. The algorithm flow for calibration and registration of LiDAR 3D point clouds and visible light images is as follows: Figure 3 As shown:

[0112] Three points are randomly selected from the scanned point cloud set. It is determined whether these three points are collinear. If they are collinear, they are reselected. If they are not collinear, the plane equation corresponding to these three points is calculated. The vertex coordinates of the triangle in the two-dimensional image are further calculated, and the parameter values ​​in the transformation matrix are calculated based on the least squares method.

[0113] Step 2: Place the logistics robot in a port park environment with an unknown map, and control the robot's movement to build the map. The specific algorithm process is as follows:

[0114] Step 2.1: Obtain the wheel rotation speed from the wheel odometer connected to the drive device, perform integration calculation to obtain the position information of the logistics robot, and at the same time use the difference in wheel speed to obtain the posture information of the logistics robot, and finally obtain the posture data of the logistics robot.

[0115] Step 2.2: Obtain 3D point cloud data from the port environment scanned by the 3D lidar sensor;

[0116] Step 2.3: Obtain RGB image data and depth image data from the port environment captured by the visible light camera;

[0117] Step 2.4: The core processor uses the RTABMAP algorithm to obtain a global 3D dense map based on the pose data of the logistics robot provided by the wheel odometer, the 3D point cloud data scanned by the LiDAR, and the depth image data collected by the visible light camera.

[0118] Step 3: Based on the obtained 3D point cloud data from the LiDAR, perform point cloud segmentation and fuse it with the depth image from the visible light camera to obtain an obstacle height estimate, thus completing the 3D measurement.

[0119] For 3D point cloud segmentation, to meet the requirements of static obstacle recognition and real-time navigation, a region growing method is adopted to segment the point cloud and complete the static obstacle recognition of the environmental point cloud. This algorithm is based on the comparison of the angles between point normals, merging adjacent points that satisfy the smoothness constraint together and outputting them as a cluster of points, where each cluster of points is considered to belong to the same plane.

[0120] Based on the obstacle regions segmented from the point cloud data, the corresponding regions on the image are found and the Regions of Interest (ROIs) are extracted. The ROIs are then magnified. Using the point cloud data corresponding to the more accurate obstacle regions found on the image, the coordinates of the lowest and highest points of the obstacle in the camera coordinate system are calculated through the inverse transformation of the camera intrinsic parameter matrix to obtain the height information of the obstacle.

[0121] Step 4: Based on the RGB image data from the visible light camera obtained above, perform image semantic segmentation and safe travel area identification. This step uses a deep convolutional neural network (CNN) to perform semantic segmentation on the forward-looking color image data obtained from the visible light camera. Combined with the 3D point cloud data of the LiDAR corresponding to each pixel, a threshold judgment is applied to the height data of the semantically segmented areas to identify the ground roads, using the ground roads as the initial selected safe travel areas. Further, other areas at the same distance from the safe travel areas are identified, such as narrow doorways, passages, and obstacles. Based on test results, a screening model is established to remove such areas from the safe travel areas or issue a hazard warning.

[0122] Step 5: Add the 3D measurement data obtained in Step 3 and the scene recognition data obtained in Step 4 to the global map to obtain a global 3D occupancy dense map with semantic information.

[0123] Step 6: Based on a globally 3D dense map with semantic information, autonomous navigation based on a known map can be achieved. The specific algorithm process is as follows:

[0124] Step 6.1: Based on the global 3D dense map with semantic information and the fused LiDAR 3D point cloud data and depth image data, the adaptive Monte Carlo localization (AMCL) method is used to locate the current position of the logistics robot.

[0125] Step 6.2: Select the target point on the control interface. Based on the current position information, the constructed 3D dense map with semantic information, the robot pose information provided by the wheel odometry, the 3D point cloud data scanned by the LiDAR, and the depth image data collected by the visible light camera, the robot uses a global path planning algorithm, including but not limited to the A* algorithm, to guide the robot to the target position and form a collision-free navigation path for global path planning. To improve the search efficiency of the algorithm, after the path is generated, it is checked whether there is a line of sight between two points separated by the path. If there is, the intermediate point between the two points can be deleted to form a collision-free navigation path, thereby performing global path planning.

[0126] Step 6.3: Based on the global path planning in Step 6.2, the 3D point cloud data from the LiDAR scan, and the depth image data acquired by the visible light camera, a local path planning algorithm using the D* heuristic is employed. This local path planning is used for dynamic obstacle avoidance during navigation. Based on the identified safe travel area, the D* algorithm is used to search for the optimal travel path, which is continuously updated during the journey, ultimately achieving autonomous navigation for the logistics robot.

[0127] This invention helps improve the ease of operation of port logistics robots, enhances their ability to cope with complex environments such as cargo loading, unloading, storage, and transfer sites in port parks, improves the execution efficiency of port logistics tasks, and further reduces overall logistics costs. It has a very positive significance for ensuring the production safety of port logistics and improving port throughput efficiency.

[0128] This research focuses on synchronous map building and localization technology for port logistics robots operating in complex environments such as cargo loading, unloading, storage, and transshipment sites in port parks. Based on the latest research results in SLAM technology, the research emphasizes the specific implementation technologies and software and hardware system engineering design for real-time scene detection, map building, local localization, and global localization of port logistics robots in complex and open working environments such as cargo loading, unloading, storage, and transshipment sites in port parks, as well as their ability to enter dynamic and unknown scenarios.

[0129] Research on registration technology between 3D point clouds and visible light images from lidar. This includes synchronous acquisition and control of point cloud data and visible light images, calibration of point cloud data and image data, and precise coordinate mapping. A parametric model is established based on multi-line lidar scanning parameters and the relative positional relationship between the visible light camera and lidar. Based on sensor data calibration, a registration feature dataset consisting of corresponding feature point pairs is extracted from customized structured scenes. Model parameters are solved, and a registration mapping function is established. The research also investigates registration algorithms between 3D point cloud data output by lidar and pixels in visible light images, and develops calibration and registration program modules applicable to combat equipment. This lays the foundation for further intelligent obstacle recognition, dynamic intelligent identification of safe movement areas, and hazard warning.

[0130] Research on dynamic intelligent identification and hazard warning technology for safe travel areas. This research focuses on scene recognition and 3D measurement technology based on intelligent image semantic segmentation to dynamically detect parameters such as height, width, and distance of obstacles, doorways, and passageways within the travel area of ​​a port logistics robot. Based on the robot's structural features and pre-defined warning parameters, it intelligently identifies safe travel areas, providing the robot with an intuitive auxiliary identification tool. Simultaneously, research is being conducted on dynamic identification and warning technologies for obstacles and hazardous areas.

[0131] Autonomous navigation and obstacle avoidance technology for port logistics robots within line of sight. Based on the construction of a local map, this study investigates the implementation technology of autonomous obstacle avoidance / crossing judgment, path planning, and navigation control within line of sight. This enables the robot to intelligently plan its travel path and intelligently avoid obstacles according to the target position selected in the forward view image, achieving autonomous movement.

[0132] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0133] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0134] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0135] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0136] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for semi-autonomous control of a robot in complex environments, characterized in that, The method comprises the following steps: Obtain the pose information, three-dimensional point cloud data and depth image data of the robot, and calibrate the three-dimensional point cloud data and the depth image data; Fuse the depth image and the three-dimensional point cloud data according to the pose information to obtain fused lidar data, and use an RTABMAP algorithm to obtain a global three-dimensional occupancy dense map according to the fused lidar data; Segment the point cloud based on the three-dimensional point cloud data, fuse the segmented point cloud with the depth image to obtain obstacle height estimation, and obtain three-dimensional measurement data; Perform semantic segmentation on a front-view color image based on a depth convolutional neural network, perform threshold discrimination on region height data obtained through semantic segmentation in combination with point cloud data corresponding to pixels, to realize ground road recognition, and obtain scene recognition data; Add the three-dimensional measurement data and the scene recognition data to the global three-dimensional occupancy dense map to obtain a global three-dimensional occupancy dense map with semantic information; Perform autonomous navigation based on a known map based on the global three-dimensional occupancy dense map with semantic information; The method for segmenting the point cloud based on the three-dimensional point cloud data and fusing the segmented point cloud with the depth image to obtain obstacle height estimation and obtain three-dimensional measurement data comprises: Segment the point cloud by using a region growing method to complete static obstacle recognition of environment point cloud and obtain an obstacle region; Find a corresponding region on the depth image according to the obstacle region segmented from the point cloud data and extract an ROI region; Magnify the ROI region, find the point cloud data corresponding to the obstacle region on the depth image, and calculate the coordinates of the lowest and highest parts of the obstacle in the camera coordinates through inverse transformation of the camera intrinsic matrix, to obtain height information of the obstacle; The method for performing semantic segmentation on a front-view color image based on a depth convolutional neural network, performing threshold discrimination on region height data obtained through semantic segmentation in combination with point cloud data corresponding to pixels, to obtain scene recognition data comprises: Perform semantic segmentation on the depth image data based on a depth convolutional neural network, perform threshold discrimination on region height data obtained through semantic segmentation in combination with laser radar three-dimensional point cloud data corresponding to pixels, to realize ground road recognition; Take the ground road as a preliminary selected safe travel region, identify other regions on the same distance section of the safe travel region, and establish a screening model in combination with test conditions to eliminate or make a danger warning for various regions from the safe travel region.

2. The method of robot semi-autonomous control according to claim 1, wherein, The method for calibrating the three-dimensional point cloud data and the depth image data comprises: Collect the three-dimensional point cloud data through a three-dimensional laser radar and obtain the depth image through a laser camera; Obtain pixel coordinates of the laser camera and three-dimensional laser radar coordinates; Calibrate to obtain a camera intrinsic matrix by using a multi-triangle calibration method according to the pixel coordinates of the laser camera and the three-dimensional laser radar coordinates.

3. The method of robot semi-autonomous control of claim 1, wherein, The method for obtaining the pose data comprises: The rotation speed of the wheel is obtained, integral operation is performed, the position information of the logistics robot is obtained, the attitude information of the logistics robot is obtained by using the difference between the wheel speeds, and finally the position and posture data of the logistics robot are obtained.

4. The robot semi-autonomous control method of claim 1, wherein, The method for autonomous navigation based on a known map comprises: obtaining current position information and a target point; forming a collision-free navigation path using a global path planning algorithm based on the global three-dimensional occupancy dense map with semantic information, for global path planning; based on the global path planning, performing local path planning using a D* heuristic path search algorithm; the local path planning is used for dynamic obstacle avoidance in the navigation process; using the D* algorithm to search for an optimal travel path, and continuously updating the travel path during the travel process until the navigation ends.

5. A robotic semi-autonomous control system, characterized by, The method comprises: a logistics robot; a driving device arranged on the chassis of the logistics robot and connected to the tires of the logistics robot, for driving the logistics robot to run; a wheel odometer connected to the driving device, for detecting the position and posture information of the logistics robot; a three-dimensional laser radar sensor arranged on the logistics robot and connected to a core processor through a USB-to-serial port, for scanning the port environment to obtain three-dimensional laser radar point cloud data; a visible light camera arranged on the logistics robot and connected to the core processor through a USB-to-serial port, for collecting depth image data of the port environment; a core processor arranged on the logistics robot and connected to the driving device, the wheel odometer, the three-dimensional laser radar sensor, and the visible light camera, for constructing a global three-dimensional occupancy dense map based on the position and posture information of the logistics robot, the three-dimensional laser radar point cloud data, and the depth image data, and performing path planning and automatic navigation based on the global three-dimensional occupancy dense map; the core processor is configured to execute the method of claim 1.

6. The robotic semi-autonomous control system of claim 5, wherein, The core processor comprises the following modules: an input module for obtaining the position and posture information of the robot, three-dimensional point cloud data, and depth image data, and calibrating the three-dimensional point cloud data and the depth image data; a map generation module for fusing the depth image and the three-dimensional point cloud data based on the position and posture information to obtain fused laser radar data, and using an RTABMAP algorithm to obtain a global three-dimensional occupancy dense map based on the fused laser radar data; a height estimation module for performing point cloud segmentation based on the three-dimensional point cloud data, and fusing the depth image to obtain obstacle height estimation, thereby obtaining three-dimensional measurement data; a scene recognition module for performing semantic segmentation on a front-view color image based on a depth convolutional neural network, and combining pixel corresponding point cloud data to perform threshold discrimination on region height data obtained by semantic segmentation, thereby obtaining scene recognition data; a semantic map module for adding the three-dimensional measurement data and the scene recognition data to the global three-dimensional occupancy dense map, thereby obtaining a global three-dimensional occupancy dense map with semantic information. Navigation module: for autonomous navigation based on known maps based on global three-dimensional occupancy dense map with semantic information.

7. The robotic semi-autonomous control system of claim 6, wherein, The height estimation module performs point cloud segmentation based on the three-dimensional point cloud data, and fuses the depth image to obtain obstacle height estimation, and a method for obtaining three-dimensional measurement data comprises: Taking the region growing method to segment the point cloud, completing the static obstacle recognition of the environment point cloud, and obtaining the obstacle region; According to the obstacle region segmented from the point cloud data, finding the corresponding region on the depth image and extracting the ROI region; The ROI region is enlarged, the point cloud data corresponding to the obstacle region found on the depth image is calculated through the inverse transformation of the camera intrinsic matrix, the coordinates of the lowest and highest parts of the obstacle in the camera coordinates are calculated, and the height information of the obstacle is obtained.

8. The robotic semi-autonomous control system of claim 6, wherein, The scene recognition module performs semantic segmentation on the front-view color image based on a deep convolutional neural network, combines the point cloud data corresponding to the pixels, performs threshold discrimination on the region height data obtained by semantic segmentation, and obtains scene recognition data. Based on the deep convolutional neural network, the depth image data is subjected to semantic segmentation, the three-dimensional point cloud data corresponding to the pixels is combined, the region height data obtained by semantic segmentation is subjected to threshold discrimination, and the ground road is recognized. Taking the ground road as the initial selected safe travel area, other regions on the same distance cross section are identified, a screening model is established combined with the test situation, and various regions are excluded from the safe travel area or a danger warning is made.

Images