A method for determining driving regions based on point cloud and image fusion using transform

Abstract

This invention discloses a method for determining driving areas based on transform-based point cloud and image fusion, relating to the field of robot perception algorithms. The algorithm takes an RGB image, a depth image, and a set of point cloud data as input. Based on a collected ground obstacle dataset, a Segformer neural network is trained, and the Transformer-based Segformer neural network is used to perform semantic segmentation prediction of pixels in the RGB image. This invention uses a Segformer image semantic segmentation neural network to convert ground information from the RGB image into point cloud information, compensating for the problem of missing black obstacle point clouds leading to the inability to determine whether an area is an obstacle or ground, thus improving the computation speed of the actual results and the accuracy of ground and obstacle recognition. By superimposing two cost maps, the advantages and disadvantages of image and point cloud judgment outputs are combined, resulting in a more accurate judgment of obstacle and ground information, further improving the accuracy of ground and obstacle recognition.

Description

Technical Field

[0001] This invention relates to the field of robot perception algorithms, and in particular to a method for determining driving areas based on point cloud and image fusion using transform. Background Technology

[0002] With the continuous development of robotics technology, autonomous navigation has become an important field. Autonomous navigation refers to the ability of a robot to autonomously plan its path, avoid obstacles, and navigate to achieve its predetermined goals without human intervention. This technology has enabled the widespread and in-depth application of robots in many fields such as intelligent manufacturing, smart healthcare, autonomous driving, and smart homes.

[0003] Key technologies for autonomous robot navigation include robot sensor perception, map building, and path planning. Sensors are the robot's "eyes," helping it perceive its surroundings. Map building involves constructing a digital map from the robot's perceived environmental information, enabling it to plan its path. Path planning involves finding the shortest or optimal path on the map to reach the intended goal.

[0004] The robot's perception module is a crucial component for achieving intelligent and autonomous robot development. It helps the robot perceive surrounding information in unknown and complex environments, enabling accurate navigation and decision-making. The perception module primarily relies on sensors, chips, and algorithms to realize its functions. Sensors are essential tools for acquiring information about the surrounding environment, including radar, lidar, cameras, and ultrasonic sensors. These sensors provide three-dimensional information about the environment, offering data support for subsequent navigation and decision-making. Chips are the core components for data processing and computation, and their performance directly affects the overall performance of the robot's autonomous navigation perception module. Algorithms are the key to achieving autonomous navigation.

[0005] The working principle of a robot's autonomous navigation perception module is mainly to acquire information about the surrounding environment through sensors, and then process and calculate it through chips and algorithms to generate the navigation information required by the robot. For example, environmental information acquired by a LiDAR sensor is used to generate a 2D grid map or a 3D point cloud map for the robot, and then the optimal path is calculated through a path planning algorithm, thereby enabling the robot's autonomous navigation.

[0006] During the construction of a 2D raster cost map by the perception module, a problem arises where the point cloud for identifying black obstacles by the RGBD camera is missing. Without the point cloud of this area, it's impossible to determine whether the area is an obstacle or the ground in the 3D coordinate system. Traditionally, for safety reasons, this area is assumed to be an obstacle. This causes passable areas to be calculated as impassable areas in the 2D raster map, thus reducing the drivable area in the map or in real-time navigation.

[0007] In summary, the existing technology has the following problems:

[0008] In the perception module of the aforementioned robot autonomous navigation, the stereo camera's recognition of black objects suffers from several issues. Due to the limited number of feature points and light absorption of black objects, the stereo camera may encounter a problem where the depth image of the black area is null (no depth data) and cannot be converted into a 3D point cloud of that area. Summary of the Invention

[0009] This invention proposes a method for determining driving areas based on transform-based point cloud and image fusion. It utilizes RGB images to fuse point cloud ground data to determine whether the area is feasible, thus solving existing problems.

[0010] To achieve the above objectives, the present invention adopts the following technical solution:

[0011] A method for determining driving regions based on point cloud and image fusion using transform includes the following steps:

[0012] S1. The input to the entire algorithm is an RGB image, a depth image, and a set of point cloud data.

[0013] S2. Based on the collected ground obstacle dataset, train the Segformer neural network, and use the Transformer-based Segformer neural network to perform semantic segmentation prediction of pixels in the RGB image to obtain a grayscale image 1 with binary classification of ground and obstacles, where the pixel value of the obstacle is 1 and the pixel value of the ground is 0.

[0014] S3. Based on the grayscale image 1 obtained in S2, the black obstacle part has no value in the depth image, so point cloud information cannot be obtained. Therefore, it cannot be mapped to the 3D point cloud using pixels. However, the pixels on the ground can be mapped. Taking advantage of this feature, the Transformer neural network judges the coordinates (x1, y1) of the RGB image pixels on the ground and performs coordinate transformation: from the depth image coordinates (x1, y1, d1) to the point cloud coordinates (x2, y2, z2) (Formula 1); by cropping points at a certain height, it is mapped to a 2D costmap. On the costmap, the pixel values ​​on the ground are marked as 0, and the pixel values ​​on the non-ground are marked as 1. This costmap is called the first costmap obtained by the deep neural network.

[0015] Formula 1:

[0016] x2 = z2(x1 - cx) / fx

[0017] y2=z2(y1-cy) / fy

[0018] z2=d1

[0019] Where x2, y2, z2 represent the transformed 3D point cloud coordinates, d1 represents the depth, x1, y1 represent the coordinates of the point in the RGB image, cx, cy are the coordinates of the optical center on the image plane, and fx, fy are the focal lengths of the two axes.

[0020] S4. The original 3D point cloud data is cropped to a certain height and mapped onto a 2D costmap, denoted as the costmap for the second point cloud judgment. Points on the ground below a certain height 'a' are 0, and points on obstacles above a certain height 'a' are 1. There are two cases here: for normal obstacles, there are points in the 3D point cloud, so the value mapped to the 2D costmap is 1. However, black obstacles do not have 3D points, so the value mapped to the point cloud height is 0 in the 2D costmap. We need to get the value of 1 for this black obstacle to satisfy the costmap of the actual obstacle part during navigation.

[0021] S5. Overlay the second 2D costmap onto the first one, perform a union judgment on the pixels with an obstacle value of 1. If there is an obstacle pixel with a pixel value of 1 in either costmap, it is judged as an obstacle and mapped to the third merged costmap, which is recorded as the final output costmap.

[0022] The depth image is represented in the digital image by an N*M matrix, where the data of each point in the matrix represents the distance of that point from the camera; the camera is a stereo camera: the imaging principle of this sensor is to calculate the depth information of the point by triangulation of the feature points of the two cameras, to obtain a depth image, and then to calculate the 3D position information of the point relative to the camera (x2, y2, z2) using the depth image, to obtain 3D point cloud data;

[0023] The Transformer model consists of an encoder and a decoder, each containing multiple layers. In the encoder, the input sequence passes through multiple layers of self-attention and feedforward neural networks to obtain the encoded representation. In the decoder, the target sequence also passes through multiple layers of self-attention and feedforward neural networks, and an encoder-decoder attention mechanism is introduced to obtain information about the input sequence.

[0024] Segformer consists of two parts: 1) a multi-layer Transformer encoder; 2) a lightweight All-MLP decoder.

[0025] Preferably, the point cloud is a set of points located at x2, y2, z2 in a three-dimensional spatial coordinate system.

[0026] Preferably, the RGB image is represented in the digital image by an N*M*3 matrix. The data of each point in the matrix represents the color value at that location. There are a total of 3 layers, representing red, green and blue respectively, which together constitute the color of the digital image.

[0027] Preferably, the costmap is an n*m two-dimensional matrix, where each pixel represents the location information on the map, where 0 represents the robot's drivable area and 1 represents the obstacle area.

[0028] The beneficial effects of this invention are as follows:

[0029] 1. For depth-deficient areas in the depth image generated by the binocular camera, they are considered as obstacles and marked as obstacles in the 2D cost map for use by the navigation module. This invention can identify black obstacle areas and ground areas, avoiding direct judgment of obstacles based on point clouds and depth deficiencies, thus improving the accuracy of the algorithm in identifying black objects and the ground.

[0030] 2. For the depth-deficient areas in the depth image generated by the binocular camera, they are considered drivable and marked as drivable areas in the 2D cost map for use by the navigation module. This invention can identify black obstacle areas and ground areas, avoiding direct judgment of obstacles based on point clouds and depth deficiencies, thus improving the security, recognition accuracy and detection speed of the perception algorithm.

[0031] In summary, this invention provides a driving area determination method based on transform-based point cloud and image fusion. It utilizes RGB images to fuse point cloud ground data to determine whether the area is feasible. This technology has advantages such as high accuracy, fast detection speed, and strong robustness. By leveraging these advantages, obstacle and ground detection can be achieved for different special scenarios. Attached Figure Description

[0032] Figure 1 This is a flowchart of the overall algorithm of the present invention.

[0033] Figure 2 This is a point cloud diagram of the present invention.

[0034] Figure 3 This is a schematic diagram of an RGB image of the present invention.

[0035] Figure 4 This is a schematic diagram of the depth image of the present invention.

[0036] Figure 5 This is a schematic diagram showing the position of the binocular camera in this invention.

[0037] Figure 6 This is a schematic diagram of the Transformer model of the present invention.

[0038] Figure 7 This is a schematic diagram of grayscale image 1 of the present invention. Detailed Implementation

[0039] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0040] Reference Figures 1-7 A method for determining driving regions based on point cloud and image fusion using transform includes the following steps:

[0041] S1. The input to the entire algorithm is an RGB image, a depth image, and a set of point cloud data.

[0042] S2. Based on the collected ground obstacle dataset, train a Segformer neural network. Use the Transformer-based Segformer neural network to perform semantic segmentation prediction on the RGB image pixels, resulting in a binary grayscale image 1 of ground and obstacles. The pixel value of the obstacle is 1, and the pixel value of the ground is 0 (e.g., Figure 7 (as shown)

[0043] S3. Based on the grayscale image 1 obtained in S2, the black obstacle part has no value in the depth image, so point cloud information cannot be obtained. Therefore, it cannot be mapped to the 3D point cloud using pixels. However, the pixels on the ground can be mapped. Taking advantage of this feature, the Transformer neural network judges the coordinates (x1, y1) of the RGB image pixels on the ground and performs coordinate transformation: from the depth image coordinates (x1, y1, d1) to the point cloud coordinates (x2, y2, z2) (Formula 1); by cropping points at a certain height, it is mapped to a 2D costmap. On the costmap, the pixel values ​​on the ground are marked as 0, and the pixel values ​​on the non-ground are marked as 1. This costmap is called the first costmap obtained by the deep neural network.

[0044] Formula 1:

[0045] x2 = z2(x1 - cx) / fx

[0046] y2=z2(y1-cy) / fy

[0047] z2=d1

[0048] Where x2, y2, z2 represent the transformed 3D point cloud coordinates, d1 represents the depth, x1, y1 represent the coordinates of the point in the RGB image, cx, cy are the coordinates of the optical center on the image plane, and fx, fy are the focal lengths of the two axes.

[0049] S4. The original 3D point cloud data is cropped to a certain height and mapped onto a 2D costmap, denoted as the costmap for the second point cloud judgment. Points on the ground below a certain height 'a' are 0, and points on obstacles above a certain height 'a' are 1. There are two cases here: for normal obstacles, there are points in the 3D point cloud, so the value mapped to the 2D costmap is 1. However, black obstacles do not have 3D points, so the value mapped to the point cloud height is 0 in the 2D costmap. We need to get the value of 1 for this black obstacle to satisfy the costmap of the actual obstacle part during navigation.

[0050] S5. Overlay the second 2D costmap onto the first one, perform a union judgment on the pixels with an obstacle value of 1. If there is an obstacle pixel with a pixel value of 1 in either costmap, it is judged as an obstacle and mapped to the third merged costmap, which is recorded as the final output costmap.

[0051] This invention combines the advantages and disadvantages of image and point cloud analysis by overlaying two cost maps, thereby more accurately judging obstacle and ground information and improving the accuracy of ground and obstacle recognition.

[0052] like Figure 2 As shown, a point cloud is a collection of points located at x2, y2, z2 in a three-dimensional coordinate system.

[0053] like Figure 3 As shown, an RGB image is represented by an N*M*3 matrix in a digital image. The data at each point in the matrix represents the color value at that location. There are three layers in total, representing red, green, and blue respectively. Together, they form the color of the digital image.

[0054] like Figure 4 As shown, the depth image is represented in the digital image by an N*M matrix, where the data of each point in the matrix represents the distance of that point from the camera.

[0055] The cost map is an n*m two-dimensional matrix, where each pixel represents the location information on the map, where 0 represents the robot's drivable area and 1 represents the obstacle area.

[0056] like Figure 5 As shown, the binocular camera: The imaging principle of this sensor is to calculate the depth information of the point by triangulation of the feature points of the two cameras, and obtain a depth image. Then, the 3D position information of the point relative to the camera (x2, y2, z2) is calculated from the depth image to obtain 3D point cloud data.

[0057] like Figure 6 As shown, the Transformer neural network is a deep learning model for processing sequential data. It was initially proposed for natural language processing tasks. Compared with traditional recurrent neural networks (RNNs) and convolutional neural networks (CNNs), the Transformer introduces a self-attention mechanism, which can process input sequences in parallel, thereby accelerating the training speed.

[0058] The Transformer model consists of an encoder and a decoder, each containing multiple layers. In the encoder, the input sequence passes through multiple layers of self-attention and feedforward neural networks to obtain the encoded representation. In the decoder, the target sequence also passes through multiple layers of self-attention and feedforward neural networks, and an encoder-decoder attention mechanism is introduced to capture information from the input sequence. The core of the Transformer is the self-attention mechanism, which allows the model to consider the dependencies between different positions in the input sequence when computing the representation. This mechanism enables the model to better capture long-distance dependencies, thus achieving excellent results when processing sequence data.

[0059] The Segformer consists of two parts: 1) a multi-layer Transformer encoder; 2) a lightweight All-MLP (Multi-Layer Perceptron) decoder. The Segformer used in this invention is a semantic segmentation neural network belonging to Transformer (used to segment the category of each pixel in an RGB image). The Segformer takes into account efficiency, accuracy, and robustness. The authors redesigned the encoder and decoder. The main design points include: 1. A novel position-encoding-free multi-layer transformer encoder; 2. A lightweight All-MLP (Multi-Layer Perceptron) decoder design.

[0060] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for determining driving regions based on point cloud and image fusion using transform. The feature is that it includes the following steps: S1. The input to the entire algorithm is an RGB image, a depth image, and a set of point cloud data. S2. Based on the collected ground obstacle dataset, train the Segformer neural network, and use the Transformer-based Segformer neural network to perform semantic segmentation prediction of pixels in the RGB image to obtain a grayscale image 1 with binary classification of ground and obstacles, where the pixel value of the obstacle is 1 and the pixel value of the ground is 0. S3. Based on the grayscale image 1 obtained in S2, the black obstacle part cannot be mapped to the 3D point cloud because it has no value in the depth image. However, the pixels on the ground can be mapped. Taking advantage of this feature, the Transformer neural network determines the coordinates (x1, y1) of the RGB image pixels on the ground and performs coordinate transformation: using formula 1, the coordinates (x1, y1, d1) of the depth image are transformed to the coordinates (x2, y2, z2) of the point cloud. By cropping points at a certain height, they are mapped to a 2D costmap. On the costmap, the pixel values ​​of the ground are marked as 0, and the pixel values ​​of the non-ground pixels are marked as 1. This costmap is called the first costmap obtained by the deep neural network. Formula 1: x2 = z2(x1 - cx) / fx y2=z2(y1-cy) / fy z2=d1 Where x2, y2, z2 represent the transformed 3D point cloud coordinates, d1 represents the depth, x1, y1 represent the coordinates of the point in the RGB image, cx, cy are the coordinates of the optical center on the image plane, and fx, fy are the focal lengths of the two axes. S4. The original 3D point cloud data is cropped to a certain height and mapped onto a 2D costmap, denoted as the costmap for the second point cloud judgment. Points on the ground below a certain height 'a' are 0, and points on obstacles above a certain height 'a' are 1. There are two cases here: for normal obstacles, there are points in the 3D point cloud, so the value mapped to the 2D costmap is 1. However, black obstacles do not have 3D points, so the value mapped to the point cloud height is 0 in the 2D costmap. We need to get the value of 1 for this black obstacle to satisfy the costmap of the actual obstacle part during navigation. S5. Overlay the second 2D costmap onto the first one, perform a union judgment on the pixels with an obstacle value of 1. If there is an obstacle pixel with a pixel value of 1 in either costmap, it is judged as an obstacle and mapped to the third merged costmap, which is recorded as the final output costmap. The depth image is represented in the digital image by an N*M matrix, where the data of each point in the matrix represents the distance of that point from the camera; the camera is a stereo camera: the imaging principle of this sensor is to calculate the depth information of the point by triangulation of the feature points of the two cameras, to obtain a depth image, and then to calculate the 3D position information of the point relative to the camera (x2, y2, z2) using the depth image, to obtain 3D point cloud data; The Transformer model consists of an encoder and a decoder, each containing multiple layers. In the encoder, the input sequence passes through multiple layers of self-attention and feedforward neural networks to obtain the encoded representation. In the decoder, the target sequence also passes through multiple layers of self-attention and feedforward neural networks, and an encoder-decoder attention mechanism is introduced to obtain information about the input sequence. Segformer consists of two parts: 1) a multi-layer Transformer encoder; 2) a lightweight All-MLP decoder.

2. The method for determining driving regions based on point cloud and image fusion according to claim 1, characterized in that, The point cloud is a set of points located at x2, y2, z2 in a three-dimensional spatial coordinate system.

3. The method for determining driving regions based on point cloud and image fusion according to claim 1, characterized in that, The RGB image is represented in digital image using an N*M*3 matrix. The data of each point in the matrix represents the color value at that location. There are a total of 3 layers, representing red, green, and blue respectively. Together, they form the color of the digital image.

4. The method for determining driving regions based on point cloud and image fusion according to claim 1, characterized in that, The costmap is an n*m two-dimensional matrix, where each pixel represents the location information on the map, where 0 represents the robot's drivable area and 1 represents the obstacle area.

Images