Robotic exploration system

Abstract

The application provides a kind of detection robot system, comprising: ROS master control;Laser radar is connected to the ROS master control, and the space is scanned by 360 degrees laser ranging, and the two-dimensional plane point cloud map of the space is generated and transmitted to the ROS master control for viewing;Depth camera is connected to the ROS master control, and ORB-SLAM2 open source algorithm is used as the basic framework, and the visual front end is constructed based on ORB feature points, and the visual back end is constructed based on the nonlinear optimization BA mode of visual SLAM system, and three-dimensional point cloud map reconstruction is carried out, and is transmitted to the ROS master control for viewing.The system has high integration, can be combined in two-dimensional and three-dimensional, real simulation real environment, realize high-precision detection.

Description

Technical Field

[0001] This invention belongs to the field of mobile detection robot application technology, and particularly relates to a detection robot system. Background Technology

[0002] Exploration robots are intelligent mechanisms with perception, decision-making, and movement capabilities. This technology involves artificial intelligence, automatic control, information processing, image processing, and pattern recognition. With the development of computers and artificial intelligence, exploration robots have found wide application not only in industrial manufacturing but also in military, civilian, rescue, marine exploration, and satellite exploration fields. Exploration robot research has become a hot topic.

[0003] As exploration robots develop, the requirements for their system integration are becoming increasingly stringent, necessitating the integration of various modules into a unified platform design. Furthermore, many factors are uncontrollable during robot task execution, such as varying natural lighting conditions at different times of day and the potential for background scattering of specific colors of light, altering the object's color. These factors all affect the recognition results, thus requiring exploration robots to possess high-precision detection capabilities in the real environment. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention proposes a detection robot system with high system integration, capable of combining two-dimensional and three-dimensional approaches to realistically simulate the real environment and achieve high-precision detection.

[0005] To achieve the above objectives, the present invention provides a detection robot system, comprising:

[0006] ROS main controller,

[0007] The lidar is connected to the ROS main controller to perform 360-degree omnidirectional laser ranging scan of the space, generate a two-dimensional planar point cloud map of the space, and transmit it to the ROS main controller for viewing.

[0008] A depth camera is connected to the ROS main controller. Using the ORB-SLAM2 open-source algorithm as the basic framework, a visual front-end is built based on ORB feature points, and a visual back-end is built based on a visual SLAM system using a nonlinear optimization BA method. The system then reconstructs a 3D point cloud map and transmits it to the ROS main controller for viewing.

[0009] Optionally, reconstructing a 3D point cloud map by obtaining target keyframe information through the ORB feature points includes:

[0010] The initial keyframe information, including camera pose and timestamp, is obtained by using the ORB feature points according to the given rules.

[0011] The initial keyframe information is inserted during map construction for filtering to obtain the target keyframe information required for dense 3D reconstruction.

[0012] Based on the target keyframe information, the pose, RGB image, depth image and timestamp information are matched.

[0013] Optionally, a TSDF map can be used to perform 3D point cloud fusion on the point cloud dataset based on the target keyframe information to reconstruct a 3D point cloud map.

[0014] Optionally, the MarchingCube algorithm is used to extract isosurfaces from the TSDF map, and triangular patches are reconstructed from the 3D point cloud map.

[0015] Optionally, statistical filtering can be applied to the reconstructed 3D point cloud map to optimize the point cloud quality.

[0016] Optionally, the statistical filtering of the reconstructed 3D point cloud map includes:

[0017] By performing statistical analysis on the neighborhood of any point in the three-dimensional point cloud map space, a statistical histogram representing the neighborhood features is drawn to identify bad points.

[0018] The histogram is cropped based on the characteristic differences between the bad points and normal points, and the point cloud is statistically filtered.

[0019] Optionally, the ORB feature points include improved FAST keypoints and BRIEF descriptors;

[0020] The FAST keypoints compare the gray values ​​of the pixels around the center point with the gray value of the center point, and use the non-maximum suppression method to find the optimal target bounding box.

[0021] The BRIEF descriptor uses a binary description vector to describe the pixels surrounding the keypoint.

[0022] Optionally, an image pyramid is constructed, and the FAST keypoint detection is performed at each layer of the image pyramid to meet the requirement of scale invariance;

[0023] The principal directions are constructed using the gray-scale centroid method to achieve rotational invariance.

[0024] Optionally, the lidar is a TOF ranging radar.

[0025] Optionally, the ROS host is based on an Ubuntu system. The ROS host acquires the two-dimensional planar point cloud map and / or the three-dimensional point cloud map, and views the two-dimensional planar point cloud map and / or the three-dimensional point cloud map through an Ubuntu computer or virtual machine.

[0026] As can be seen from the above solutions, the advantages of the present invention are:

[0027] The exploration robot system provided by this invention, based on ROS dynamic 3D reconstruction, mainly includes a ROS main controller, a LiDAR, and a depth camera. The ROS main controller adopts a point-to-point, distributed framework, coupling various programs and peripherals together through nodes and topics, resulting in high integration and facilitating modular upgrades and modifications. Simultaneously, precise 2D spatial modeling using a TOF LiDAR ensures map quality; 3D modeling is performed by scanning the object to be explored using the depth camera, realistically simulating the 3D environment. The combination of these two methods achieves high-precision exploration. This system has high integration, combining 2D and 3D aspects to realistically simulate the real environment and achieve high-precision exploration, with a wide application range. Attached Figure Description

[0028] Figure 1 A diagram showing the overall architecture of the detection robot system;

[0029] Figure 2 ROS master node diagram;

[0030] Figure 3 The image shows a 2D planar point cloud map rendering of a simulated scene.

[0031] Figure 4 The image shows a 3D point cloud map rendering of a simulated scene.

[0032] in:

[0033] 1-ROS main controller;

[0034] 11-ROS master node;

[0035] 12-ROS slave nodes;

[0036] 2- LiDAR;

[0037] 3-Depth camera. Detailed Implementation

[0038] To make the above features and effects of the present invention clearer and easier to understand, specific embodiments are described below, and detailed descriptions are provided in conjunction with the accompanying drawings.

[0039] Figure 1 This is an overall architecture diagram of the exploration robot system shown in an embodiment of the present invention.

[0040] A detection robot system, which mainly includes:

[0041] ROS main controller 1,

[0042] LiDAR 2, connected to the ROS main controller 1, performs a 360-degree omnidirectional laser ranging scan of the surrounding space, generating a two-dimensional planar point cloud map of the space, and transmits it to the ROS main controller for viewing; and

[0043] Depth camera 3 is connected to the ROS main controller 1. It uses the ORB-SLAM2 open-source algorithm as the basic framework, constructs a visual front-end based on ORB feature points, constructs a visual back-end based on a visual SLAM system using nonlinear optimization BA, reconstructs a 3D point cloud map, and transmits it to the ROS main controller 1 for viewing.

[0044] In this embodiment, the detection robot system includes a ROS main controller, an N10 LiDAR, and a depth camera. An integrated platform is built through the ROS main controller, and two-dimensional spatial modeling is performed by the N10 LiDAR and three-dimensional modeling is performed by scanning the object to be detected by the depth camera. The combination of the two achieves the purpose of high-precision detection.

[0045] The specific design of each part is as follows:

[0046] (1) In this embodiment, ROS is selected as the main controller for the detection robot. The originally loosely connected modules such as lidar, camera, and robotic arm are coupled and integrated into a unified platform, and a communication framework is provided. Figure 2 The diagram shows the ROS node graph. ROS is a point-to-point, distributed framework consisting of a master node 11 and multiple slave nodes 12. Slave nodes can communicate with each other and simultaneously communicate with the master node. Nodes couple various programs and peripherals together, facilitating modular upgrades and modifications, and allowing robot-based perception, decision-making, and control algorithms to operate more effectively.

[0047] Specifically, this embodiment selects NVIDIA's Jetson TX1 as the ROS controller, using the Ubuntu system as the platform. The Jetson TX1 is NVIDIA's second-generation embedded platform developer kit. Although only the size of a credit card, the Jetson TX1's GPU boasts Teraflops of floating-point performance, perfectly suited to the flexible and compact design of the exploration robot, representing a significant improvement over the Jetson TK1. It features a built-in NVIDIA Maxwell GPU with 256 CUDA cores, a 64-bit ARMA57 CPU, 4GB of LPD DDR4 memory, Bluetooth, an 802.11ac Wi-Fi module, and a Gigabit Ethernet card. It provides interfaces such as USB 3.0 Type-A, USB 2.0, HDMI, PCI-Ex4, and SATA, meeting various design requirements.

[0048] Furthermore, MoveIt was chosen as the robot's operating platform. MoveIt is an easy-to-use integrated development platform comprised of a series of mobile operation function packages, such as motion planning, operation control, 3D perception, kinematics, control, and navigation algorithms. It provides users with a user-friendly GUI, allowing them to see the robot's form in real time and perform a series of interface-based operations. It can be applied to industrial, commercial, R&D, and other fields, and is one of the top three most used function packages in the ROS community. When using traditional robotic arms for motion planning, a series of programming steps are required. For example, moving the robot from point A to point B requires using the robotic arm's vision calibrator to perform vision calibration from point A to point B. At each point, the robotic arm needs to know whether to move in a straight line or a curve. After vision calibration, the vision calibrator and the robotic arm's controller are used to verify that each point has been reached. Using MoveIt in ROS for motion planning saves a lot of visual calibration work. Simply specify the target position for the robotic arm, and it will automatically plan a path based on the current scene. If there are obstacles, the robotic arm will plan a path to avoid them. MoveIt will further refine the planned path to better avoid obstacles. MoveIt has three core modules: kinematics, path planning, and collision detection. To facilitate user experience, three interfaces are also provided: RVizplugin, a visual interface that allows simple mouse drag-and-drop control of the robotic arm; Action, a long-term continuous feedback communication mechanism in ROS besides topics and services, implemented through topics; and C++ API and Python API, allowing control of the robotic arm via C++ or Python programming, providing great convenience for developers.

[0049] (2) Specifically, for lidar, this embodiment selects a TOF lidar, namely the N10 lidar. The N10 lidar has an adjustable scanning frequency of 6-12 Hz, corresponding to an angular resolution of 0.48°-0.96°. The N10 uses TOF ranging technology, with a high-speed laser ranging sampling capability of 4500 times per second, enabling 360-degree omnidirectional laser ranging scanning within a 25-meter radius, and generating a two-dimensional planar point cloud map of the surrounding space. While TOF ranging lidar is expensive, it also improves the performance of the lidar, offering long measurement distances, high measurement accuracy, strong resistance to light, and good environmental adaptability. It is suitable for mobile platforms with large activity spaces and high movement speeds, such as service robots, AGVs, and low-speed logistics vehicles. Using TOF ranging lidar when constructing maps of large-area environments can ensure the quality of the two-dimensional planar point cloud map.

[0050] In this embodiment, the N10 LiDAR is connected to the ROS main controller. The N10 LiDAR performs a 360-degree omnidirectional laser ranging scan of the surrounding space, generating a two-dimensional planar point cloud map, which is then transmitted to the ROS main controller. The constructed two-dimensional planar point cloud map can be viewed on an Ubuntu computer or virtual machine. Specifically, after compiling the radar SDK, the radar is started to acquire radar data and sends it to the corresponding Cartographer node. To ensure that mapping is performed after the robot's turning calibration, IMU data also needs to be integrated for analysis, ultimately resulting in the two-dimensional planar point cloud map constructed by Cartographer. Figure 3 The system displays a 2D point cloud map of a simulated scene. In addition, a 2D mapping module has been added, which can quickly extract information about the surrounding environment, analyze various potential factors, and formulate corresponding plans to issue the next instructions.

[0051] (3) Among them, the depth camera can simultaneously acquire color and depth information of the environment to reconstruct a 3D map. In this embodiment, the AstroPro depth camera can be selected. Its 3D imaging method is monocular structured light, which is suitable for indoor scenes at close and medium distances and has the advantages of high accuracy and low power consumption. A variety of functions can be realized based on this camera, such as face recognition, gesture recognition, human body tracking, 3D measurement, environmental perception and 3D map reconstruction. It can be applied to various scenarios such as living room entertainment, security monitoring, measurement, 3D scanning, motion sensing interaction, and commercial LCD displays.

[0052] In addition, for the reconstruction of 3D point cloud maps by the depth camera, the depth camera is connected to the ROS main controller. The ORB-SLAM2 open source algorithm is used as the basic framework. A visual front-end is built based on ORB feature points, and a visual back-end is built based on a visual SLAM system with nonlinear optimization BA method. The 3D point cloud map is reconstructed and transmitted to the ROS main controller for viewing. Figure 4 The image displayed is a 3D point cloud map rendering of a simulated scene.

[0053] The ORB-SLAM2 framework is a visual SLAM system with a front-end based on "OrientedFAST" keypoints and BRIEF descriptors for feature point extraction and matching, and a back-end based on nonlinear optimization (BA) for visual odometry. The system consists of three threads: Tracking, Local Mapping, and Loop Closure Detection. ORB feature points, composed of improved FAST keypoints and BRIEF descriptors, possess rotation and scale invariance. Compared to SIFT and SURF feature points, ORB feature points offer advantages such as lower computation time (less than 33ms (30FPS) on a CPU) and higher real-time performance. FAST keypoints compare the gray values ​​of pixels surrounding the center point with the center point's gray value and use non-maximum suppression to avoid excessive density of corner points within a certain area. FAST keypoints meet the real-time requirements of the visual front-end. Furthermore, in the process of constructing a 3D point cloud map, the requirement for scale invariance can be met by building a 3D image pyramid and performing FAST keypoint detection at each layer of the pyramid; the gray-scale centroid rule can be used to construct the principal orientation, thereby achieving rotation invariance. The BRIEF descriptor uses binary description vectors to describe the pixels surrounding the keypoints, exhibiting high specificity and being suitable for feature matching and appearance-based scene recognition for loop closure detection.

[0054] Specifically, in the process of 3D point cloud map reconstruction, ORB feature points are used to achieve tracking, local mapping, and loop closure functions. Based on given rules, initial keyframe information, including camera pose and timestamp, is obtained through ORB feature points. Then, during map construction, the initial keyframe information is inserted and filtered to obtain the target keyframe information required for dense 3D reconstruction. Afterward, based on the target keyframe information, pose, RGB image, depth image, and timestamp information are matched to achieve 3D point cloud reconstruction.

[0055] Furthermore, in traditional point cloud fusion and reconstruction, point cloud data is mostly calculated first using depth maps and RGB maps, and then fused with pose information. However, this method suffers from high computational complexity, density explosion caused by point cloud accumulation, and cumulative errors in pose calculation. To avoid these problems, this embodiment uses a TSDF map to reconstruct the point cloud dataset based on keyframe information. By pre-constructing a three-dimensional solid space, controllable point cloud density is achieved, reducing unnecessary redundant calculations. TSDF is a voxel-based reconstruction method, consisting of a three-dimensional cube composed of three-dimensional voxels. TSDF can run on a GPU, with each voxel updated in parallel by the GPU, resulting in fast update speeds that meet real-time reconstruction requirements. Each voxel stores a value from -1 to 1; negative values ​​indicate the voxel is inside the object, and positive values ​​indicate the voxel is further from the object's surface. Therefore, after point cloud data is fused using a TSDF map, the process of extracting the surface is equivalent to extracting the zero isosurface.

[0056] Furthermore, since point cloud data is discretely represented in 3D space, it is difficult to intuitively reflect the texture of the environment. Therefore, while preserving texture details, the MarchingCube algorithm is used to extract isosurfaces from the TSDF map and reconstruct triangular patches from the 3D point cloud map to reproduce a realistic, textured, and spatially continuous 3D environment. The basic idea of ​​the MarchingCube algorithm is to process voxels in the scalar field one by one, separate the voxels that intersect with the isosurfaces, and use interpolation to calculate the intersection points of the isosurfaces with the cube edges. Based on the relative position of each vertex of the cube with the isosurface, the intersection points of the isosurfaces with the cube edges are connected in a certain way to generate isosurfaces, which serve as an approximate representation of the isosurfaces within the cube. The values ​​of each of the 8 vertices of each voxel have two states: above and below the isosurface, resulting in a total of 256 states. Considering rotational and emission symmetries, there are a total of 15 basic modes.

[0057] Furthermore, during the data acquisition and reconstruction process, although the ORB descriptor in the ORB-SLAM2 framework avoids the impact of noise and outliers on pose estimation, the data acquired by the sensors themselves contains noise and outliers, inevitably introducing these so-called 'bad points' into the reconstruction results. To address the aforementioned quality issues of the reconstructed point cloud, this embodiment further performs statistical filtering on the reconstructed 3D environment point cloud to optimize its quality. Specifically, statistical analysis is performed on the neighborhood of any point in the 3D point cloud map space to draw a statistical histogram representing the neighborhood features, identifying bad points; then, the histogram is cropped based on the feature differences between bad points and normal points, thus performing statistical filtering on the point cloud.

[0058] In summary, the exploration robot system provided by this invention, based on ROS dynamic 3D reconstruction, mainly includes a ROS main controller, a lidar, and a depth camera. The ROS main controller adopts a point-to-point, distributed framework, coupling various programs and peripherals together through nodes and topics, resulting in high integration and facilitating modular upgrades and modifications. Simultaneously, precise 2D spatial modeling using a TOF lidar ensures map quality; 3D modeling is performed by scanning the object to be explored using the depth camera, realistically simulating the 3D environment. The combination of these two methods achieves high-precision exploration. This system has a high degree of integration, combining 2D and 3D aspects to realistically simulate the real environment and achieve high-precision exploration. It can be applied in multiple fields, including military, civilian, rescue, marine exploration, and satellite exploration, with a wide range of applications.

[0059] Furthermore, the exploration robot provided in this embodiment can be equipped with various sensors, such as temperature sensors, humidity sensors, infrared sensors, and smoke detectors, to monitor the internal conditions of the mine in real time during missions, fulfilling functions such as temperature detection, humidity detection, infrared detection, and smoke detection, and possessing voice recognition capabilities. To facilitate operation in complex terrain, the exploration robot adopts a tracked carrier design, greatly enhancing its load-bearing capacity and reducing the difficulty of mine rescue. In addition, a low-power, high-brightness lighting system can be added to significantly increase visibility in dark conditions and extend battery life, thus expanding the exploration range. Simultaneously, the exploration robot can employ advanced wireless image transmission and audio monitoring systems, as well as mobile wireless systems. The wireless image transmission and audio monitoring system transmits scanned images to a remote platform in real time, achieving low-distortion and low-noise audio, promptly assisting surface personnel in understanding underground conditions; the enhanced mobile wireless system strengthens its anti-interference capabilities, preventing loss of control underground and making movement more convenient and rapid. Furthermore, the exploration robot can be designed with high-temperature resistant, waterproof, and fire-resistant materials to ensure its normal operation in field mines and other similar environments.

[0060] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0061] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A detection robot system, characterized in that, include: The ROS master controller adopts a point-to-point, distributed framework, which couples various programs and peripherals together through multiple nodes; The lidar is connected to the ROS main controller to perform 360-degree omnidirectional laser ranging scan of the space, generate a two-dimensional planar point cloud map of the space, and transmit it to the ROS main controller for viewing. The depth camera is a monocular structured light depth camera connected to the ROS main controller. It uses the ORB-SLAM2 open-source algorithm as the basic framework, builds a visual front-end based on ORB feature points, builds a visual back-end based on a visual SLAM system using nonlinear optimization BA, performs 3D point cloud map reconstruction, and transmits the data to the ROS main controller for viewing. The process of reconstructing a 3D point cloud map by obtaining target keyframe information through the ORB feature points includes: The initial keyframe information, including camera pose and timestamp, is obtained by using the ORB feature points according to the given rules. The initial keyframe information is inserted during map construction for filtering to obtain the target keyframe information required for dense 3D reconstruction. Based on the target keyframe information, the pose, RGB image, depth image and timestamp information are matched; The TSDF map is used to perform 3D point cloud fusion on the point cloud dataset based on the target keyframe information to reconstruct a 3D point cloud map. The isosurfaces are extracted from the TSDF map using the Moving Cube algorithm, and triangular patches are reconstructed from the 3D point cloud map. After the 3D point cloud map is reconstructed, statistical filtering is performed on the reconstructed 3D point cloud map to optimize the point cloud quality, including: By performing statistical analysis on the neighborhood of any point in the three-dimensional point cloud map space, a statistical histogram representing the neighborhood features is drawn to identify bad points. The histogram is cropped based on the characteristic differences between the bad points and the normal points, and the point cloud is statistically filtered. The ORB feature points include improved FAST keypoints and BRIEF descriptors. The improved FAST keypoints compare the gray values ​​of pixels surrounding the center point with the gray value of the center point, and a non-maximum suppression method is used to find the optimal target bounding box. The BRIEF descriptors use binary description vectors to describe the pixels surrounding the keypoints.

2. The system according to claim 1, characterized in that, An image pyramid is constructed, and the FAST keypoint detection is performed at each layer of the image pyramid to meet the requirement of scale invariance. The principal directions are constructed using the gray-scale centroid method to achieve rotational invariance.

3. The system according to claim 1, characterized in that, The lidar is a TOF ranging radar.

4. The system according to claim 1, characterized in that, The ROS host is based on the Ubuntu system. The ROS host acquires the two-dimensional planar point cloud map and / or the three-dimensional point cloud map, and can view the two-dimensional planar point cloud map and / or the three-dimensional point cloud map through an Ubuntu computer or virtual machine.

Images