A legged robot terrain semantic segmentation method

Abstract

The application discloses a terrain semantic segmentation method for a foot-type robot. Firstly, the foot-type robot is used to shoot pictures of terrains and pedestrians in different weather and time periods, Kalman filtering is used to remove the unclear pictures caused by shaking of the foot-type robot during movement, and different terrains are labeled in the pictures as a training set; secondly, a convolution semantic segmentation network is constructed, a loss function is designed according to a classification loss, a positioning loss and a segmentation loss, and the semantic segmentation network is trained on a high-performance server using the training set of terrains; finally, a neural network with the same structure as that on the high-performance server is constructed on the foot-type robot through a TensorRT inference engine, and the trained weight is imported into the neural network, so that the terrain can be quickly and accurately segmented by the quadruped robot. The application solves the problems that the collected images are unclear due to serious shaking of the foot-type robot during movement, and that the semantic segmentation speed is not good due to low performance of the processor on the foot-type robot, so that the real-time requirement cannot be met.

Description

Technical Field

[0001] This invention relates to the field of environmental perception, specifically to a method for terrain semantic segmentation of a legged robot. Background Technology

[0002] With the rapid development of robotics technology, robots have begun to be widely used in people's production and daily life. Legged robots are flexible in movement and have good terrain adaptability, enabling them to walk in a wide range of environments and terrains. They have a wide range of applications and are considered the best mobile platform for scientific exploration, emergency search and rescue, material transportation, reconnaissance and patrol operations.

[0003] Environmental perception is fundamental to the localization, navigation, path planning, and motion control of legged robots, and the ability to understand the environment is a key factor in achieving autonomous operation of legged robots. To support the autonomous operation of legged robots in dynamic environments, it is necessary to accurately segment different terrains within the environment.

[0004] Semantic understanding of terrain in the environment will strongly support path planning and gait design for legged robots. Therefore, it is necessary to quickly and accurately classify different types of terrain, such as flat ground, slopes, and steps, to achieve fast and high-precision semantic segmentation of different terrains in the environment.

[0005] Papandreou et al. proposed a semantic segmentation network based on an encoder-decoder structure, which uses a deep convolutional neural network for encoding and a conditional random field for decoding. It introduces a spatial pyramid module with depthwise separable convolutions to improve the running efficiency of the semantic segmentation algorithm. However, it suffers from problems such as loss of segmentation details and limited correlation between semantic information between upper and lower layers, resulting in low segmentation edge accuracy. It also suffers from problems such as redundant data, resulting in low terrain segmentation speed.

[0006] While some progress has been made in road segmentation, indoor and layout semantic segmentation, the severe shaking of legged robots during movement leads to blurry images. In outdoor environments, the boundaries between different terrains such as slopes, steps, and potholes are often unclear, making it difficult for existing semantic segmentation methods to accurately segment these terrains. Furthermore, the low processor performance of legged robots results in slow semantic segmentation speeds that fail to meet real-time requirements. Therefore, constructing convolutional semantic segmentation networks to accurately identify terrain types and obtain precise segmentations of different terrains in real time remains a challenge.

[0007] Based on this, the present invention provides a method for terrain semantic segmentation of a legged robot. Summary of the Invention

[0008] The purpose of this invention is to provide a terrain semantic segmentation method for legged robots. By collecting terrain and pedestrian data under different weather conditions and at different times using a legged robot as a training set, a convolutional semantic segmentation network is trained. The trained network is then transferred to the legged robot to improve the terrain semantic segmentation capability of the legged robot and support the legged robot to operate autonomously in dynamic environments.

[0009] This invention utilizes principles such as convolutional semantic segmentation networks and TensorRT inference engine acceleration to design a terrain semantic segmentation technology for legged robots, enabling legged robots to quickly and accurately segment different terrains and improve their environmental understanding capabilities.

[0010] This invention proposes an environmental perception technology for legged robots, comprising the following steps:

[0011] Step 1: Collect different terrain data, label the data, deblur the blurry images in the data, and construct a training dataset.

[0012] Step 1.1: Use a legged robot to capture terrain images under different weather conditions and at different times. Due to severe shaking during the robot's movement, the captured images are often blurry. Therefore, Kalman filtering is used to remove noise and obtain clearer images.

[0013] Step 1.2: Label the obtained image data. The specific labeling method is as follows: Select the area to be labeled in the image, label the different terrain types (slope, steps, flat land), and divide the labeled images into training set and test set at a ratio of 4:1.

[0014] After obtaining the deep learning training dataset, a convolutional semantic segmentation network is constructed.

[0015] Step 2: Train a convolutional semantic segmentation network on a server using the dataset obtained in Step 1. Fully leverage the network's rapid representation and recognition capabilities. Evaluate the network's semantic segmentation performance by defining a loss function. Use an end-to-end training method to obtain the trained neural network parameters. Implement semantic segmentation for different terrains on a high-performance server.

[0016] Step 2.1: Design the loss function f of the convolutional semantic segmentation network, taking into account classification loss, localization loss and masking loss, and form the loss function f in a weighted sum manner to improve the training effect;

[0017] Step 2.2: Train the convolutional semantic segmentation network using the training dataset established in Step 1. The specific steps are as follows: Use the YOLACT network model to extract specific semantic information of different terrains. Assume that the multi-scale features of the entire terrain are obtained, calculate the probability of all types of terrain, select the terrain type with the highest probability as the identified terrain type, and realize the semantic segmentation of different terrains.

[0018] After training the convolutional semantic segmentation network on the server in step 2, considering the significant difference between the processor performance of the legged robot and the server performance, the accuracy and recognition speed of the trained neural network on the legged robot are significantly lower than those on the server, making it difficult to apply in practice. To solve this problem, the generated network is converted into a lightweight network using the TensorRT inference engine, and then the converted network is transferred to the quadruped robot, enabling the legged robot to quickly and accurately segment different terrains.

[0019] Step 3: Transfer the network obtained in Step 2 to the legged robot, and use the image data collected by the depth camera on the legged robot as the input of the neural network to perform real-time semantic segmentation of different terrains.

[0020] Step 3.1: Install the appropriate version of the TensorRT inference engine based on the model of the legged robot's internal processor and GPU processor.

[0021] Step 3.2: Convert the neural network weight file trained in Step 2 into a weight file that can be used by TensorRT. The specific steps are as follows: On the processor of the legged robot, build a one-dimensional array of the neural network weight file trained in Step 2 according to the order of different network layers. The parameters of each node in each layer of the network are stored in the array according to the order in the original network. Different arrays are built for different layers of the network and written in the file in hexadecimal form.

[0022] Step 3.3: Use the TensorRT inference engine to build a neural network with the exact same number of layers, neurons, and structure as the neural network built in Step 2. Use the weights obtained in Step 2.1 to build a convolutional semantic segmentation network on the legged robot.

[0023] Step 3.4: Use the image data obtained by the depth camera on the legged robot as the input of the neural network, and output the real-time segmented terrain semantic information.

[0024] Based on a convolutional semantic segmentation network and combined with the TensorRT inference engine, accurate and fast semantic segmentation of different terrains can be achieved on legged robots. Unlike existing technologies, this invention deblurs the real image data collected on the legged robot and, combined with the TensorRT inference engine, significantly improves the accuracy and speed of terrain segmentation, thus realizing semantic segmentation of terrain on legged robots. Attached Figure Description

[0025] The accompanying drawings, which form part of this invention, are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation of this invention.

[0026] Figure 1 This is the overall system flowchart of the present invention;

[0027] Figure 2 This is the overall system framework diagram of the present invention;

[0028] Figure 3 This is a schematic diagram of the legged robot in this invention dividing a sloping terrain in the environment;

[0029] Figure 4 This is a schematic diagram of the legged robot in this invention dividing different terrains in the environment;

[0030] Figure 5 This is a schematic diagram of the legged robot in the present invention dividing the terrain in a mountainous environment; Detailed Implementation

[0031] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. It should be noted that the aspects described below with reference to the accompanying drawings and specific embodiments are merely exemplary and should not be construed as limiting the scope of protection of the present invention in any way.

[0032] The server used for training in this invention is equipped with an Intel(R) Xeon(R) CPU E5-2620v4 @ 2.10GHz × 16, a quad-core NVIDIA GeForce GTX 2080 graphics card, 128GB of memory, and runs on Ubuntu 16.04. The training is implemented using Python 3.6 and the PyTorch deep learning framework. The training hyperparameters are: batch size of 2, number of training generations of 100, number of iterations of 1600, and adaptive moment estimation as the gradient optimization method.

[0033] The legged robot used in this invention is the unitree A1 quadruped robot, equipped with an NVIDIA TX2 processor and a Realsensed 455 depth camera. This camera supports depth image acquisition, with a maximum frame rate of 60 FPS and a maximum resolution of 1080*1920.

[0034] The advantage of this method is that it can accurately and quickly segment different terrains on legged robots, with high robustness, terrain segmentation accuracy ≥70%, and recognition speed ≥30FPS.

[0035] This method uses Kalman filtering to remove the jitter in the acquired images caused by the legged robot's walking motion. The specific method is as follows:

[0036] First, establish the state equation and observation equation for the image.

[0037] Equations of state:

[0038] X(i,j)=CX(i,j-1)+DW(i,j)

[0039] Observation equation:

[0040] Y(i,j)=AX(i,j)+V(i,j)

[0041] Where X is the input image matrix, X(i,j) is a point in the input image matrix with x-coordinate i and y-coordinate j, A is the output matrix, and C is the state transition matrix, using the identity matrix. T represents the matrix transpose operation, W(i,j) and V(i,j) are the process noise and observation noise, respectively, and the matrix is ​​constructed using Gaussian noise.

[0042] Deblurring process:

[0043]

[0044]

[0045] Parameter update process:

[0046]

[0047]

[0048]

[0049] in, The optimal predicted value, Let u(i,j) be the optimal filter value, u(i,j) be the system input, A be the output matrix, and C be the transition matrix. E = [1, 0, 0, 0] TR and Q are the covariance matrices of the noise during the deblurring process and the observation noise, respectively. It is a prediction matrix. K is the filter variance matrix, K(i,j) is the Kalman filter gain, and Y(i,j) is the output image.

[0050] First, the input image is modeled using a state equation. Then, a prediction formula is used for forecasting. Finally, the parameters in the prediction formula are iterated using a parameter update formula. If the parameters in the prediction formula remain unchanged between two calculations, the final predicted result is output as the deblurred image.

[0051] YOLACT (You Only Look At CoefficienTs) is a real-time instance segmentation and detection network that uses ResNet101 as its backbone network. It has 101 convolutional layers, including convolutional layers with 3*3 kernels, convolutional layers with 1*1 kernels, and 3*3 max pooling layers. Finally, it connects to a fully connected layer with a 1000-dimensional output and uses Softmax for classification to obtain the final terrain classification result.

[0052] The loss function of the convolutional semantic segmentation network constructed in this invention is defined as follows:

[0053]

[0054] v = (v x ,v y ,v w ,v h )

[0055]

[0056] L cls =-log p u

[0057]

[0058]

[0059] L mask =BCE(M,M gt )

[0060] in For the total loss, L cls For classification loss, L box For bounding box regression loss, L mask Let p be the masking loss, u be the detected class, and v be the true bounding box regression target tuple for class v, where v x v is the x-coordinate of the true bounded box. yv is the ordinate of the true bounded box. w v is the width in pixels of the true bounded box. h t is the height in pixels of the true bounded box. u For the predicted tuples of the true class u, The x-coordinate of the prediction box. The vertical coordinate of the prediction box. This represents the width of the predicted bounding box in pixels. Here, is the height in pixels of the predicted bounding box, log represents the logarithm, and M is the assembled mask. is the ground truth mask, and BCE is the binary cross-entropy.

[0061] The YOLACT neural network is considered to have converged when the total number of iterations reaches 100.

[0062] To simplify the explanation of this method, the above figures and text are described as a series of steps. However, it should be understood and understood that these methods are not restricted by the order of operations, because some actions may occur in different orders when implemented according to one or more steps, but those skilled in the art can understand the principle behind the actions.

[0063] Although the method has been described step by step through illustrative specific embodiments of the invention to enable those skilled in the art to understand it, the invention is not limited to the scope of the specific embodiments. Those skilled in the art can make various modifications or variations within the scope of the claims, as long as the variations are within the spirit and scope of the invention as defined and determined by the appended claims.

Claims

1. A method for terrain semantic segmentation of a legged robot, characterized in that, include: Step 1: Collect different terrain data, label the data, deblur the blurry images in the data, and build a training dataset; Step 2: Train a convolutional semantic segmentation network on the server using the dataset, evaluate the semantic segmentation performance of the network by defining a loss function, and obtain the trained neural network parameters using an "end-to-end" training method; implement semantic segmentation for different terrains on the server. Step 2 specifically includes the following steps: Step 2.1: Design the loss function f of the convolutional semantic segmentation neural network. Consider the classification loss, localization loss and mask loss, and form the loss function f in a weighted sum manner to improve the training effect. The types of hyperparameters used in training are 2, the batch size is 2, the number of training generations is 100, the number of iterations is 1600, and the gradient optimization method is adaptive moment estimation. Step 2.2: Train the convolutional semantic segmentation network using the training dataset established in Step 1. The specific steps are as follows: Use the YOLACT network model with ResNet101 as the backbone network. There are 101 convolutional layers, including convolutional layers with 3*3 kernels, convolutional layers with 1*1 kernels, and 3*3 max pooling layers. Finally, connect them into a fully connected layer with a 1000-dimensional output and use the flexible maximum transfer function for classification. Extract specific semantic information of different terrains through the YOLACT network, obtain multi-scale features of all terrains, calculate the probability of all types of terrain, select the terrain type with the highest probability as the identified terrain type, and achieve semantic segmentation of different terrains. Step 3: Transfer the network obtained in Step 2 to the legged robot, and use the image data collected by the depth camera on the legged robot as the input of the neural network to perform semantic segmentation of different terrains and dynamic pedestrian recognition in real time. Step 3 specifically includes the following steps: Step 3.1: Install the TensorRT inference engine; Step 3.2: Convert the neural network weight file trained in Step 2 into a weight file that can be used by TensorRT. The specific steps are as follows: On the processor of the legged robot, build a one-dimensional array of the neural network weight file trained in Step 2 according to the order of different network layers. The parameters of each node in each layer of the network are stored in the array according to the order in the original network. Different arrays are built for different layers of the network and written in the file in hexadecimal form. Step 3.3: Use the TensorRT inference engine to build a neural network with the exact same number of layers, neurons, and structure as the neural network built in Step 2. Use the weights obtained in Step 2.1 to build a convolutional semantic segmentation network on the legged robot. Step 3.4: Use the image data obtained by the depth camera on the legged robot as the input of the neural network, and output the real-time segmented terrain semantic information.

2. The method for semantic segmentation of terrain in a legged robot according to claim 1, characterized in that: Step 1 specifically includes the following steps: Step 1.1: Use a legged robot to capture terrain images under different weather conditions and time periods to obtain terrain images under different lighting conditions; use Kalman filtering to remove data noise, specifically as follows: Establish state equations and observation equations for the image: Equations of state: X(i,j)=CX(i,j-1)+DW(i,j) Observation equation: Y(i,j)=AX(i,j)+V(i,j) Where X is the input image matrix, X(i,j) is a point in the input image matrix with x-coordinate i and y-coordinate j, A is the output matrix, and C is the state transition matrix, using the identity matrix. T represents the matrix transpose operation, W(i,j) and V(i,j) are the process noise and observation noise, respectively, and the matrix is ​​constructed using Gaussian noise; Deblurring process: Parameter update process: in, The optimal predicted value, Let u(i,j) be the optimal filter value, u(i,j) be the system input, A be the output matrix, and C be the transition matrix. E = [1, 0, 0, 0] T R and Q are the covariance matrices of the noise during the deblurring process and the observation noise, respectively. It is a prediction matrix. Here, K(i,j) is the Kalman filter gain, and Y(i,j) is the output image; First, the input image is modeled using a state equation. Then, a prediction formula is used to make a prediction. Next, the parameters in the prediction formula are iterated using a parameter update formula. When the parameters in the prediction formula remain unchanged between two calculations, the final prediction result is output as the image deblurring result, resulting in clear image data. Step 1.2: Label the obtained image data. The specific labeling method is as follows: Select the area to be labeled in the image, label different terrain types, and divide the labeled images into training set and test set.

Images