Structural terrain semantic segmentation image optimization method based on gradient data

Abstract

The invention discloses a structural terrain semantic segmentation image optimization method based on gradient data. The method comprises the following steps: converting depth data (Depth) into gradient data with image pixel gray value change based on a Sobel operator, normalizing the gradient data, and displaying the normalized gradient data into a normal map by using red, green and blue (RGB) data; the normal chartlet and red green blue (RGB) data are fused according to a pixel point matrix Max method, recoding of an input image is achieved, and normal direction information with color representation and image information fused with a basic RGB image are output; and optimizing the edge of the semantic segmentation result of the MobileNetV3 lightweight network based on the Canny edge detection data and the gradient data of the normal map. Compared with the prior art, the method optimizes the edge of the semantic image, and gives consideration to both speed and precision. The semantic segmentation edge optimization method provided by the invention is simple and efficient, and is suitable for semantic segmentation of most structural terrains.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and robot perception, and specifically provides a structural terrain semantic segmentation image optimization method based on gradient data. Background Art

[0002] With the diversification of robot structures, their ability to traverse diverse terrains has gradually improved, leading to increasingly demanding requirements for robots' perception of their surroundings. Accurately identifying the terrain ahead and accurately traversing it is one of the current perception challenges for robots. Especially for robots that need to plan paths, the incompleteness of terrain recognition directly affects whether the robot can correctly utilize terrain information for further planning. Therefore, terrain recognition based on semantic segmentation is becoming a new trend. However, for most semantic segmentation methods, the segmentation results are missing information around the boundary lines, resulting in the terrain ahead being identified but not fully utilized. Therefore, post-processing and optimization of semantic segmentation can better segment the entire terrain framework, allowing the robot to better utilize terrain information for further planning.

[0003] Chinese patent publication number CN1 15690429A discloses a terrain semantic segmentation method for a legged robot, which effectively prevents the problem of blurred images caused by the shaking of the quadruped robot during movement. However, the model of this method only solves the problem of image collection under the jitter problem, but does not solve the problem of incomplete semantic segmentation. Chinese Patent Publication No. CN116883586A discloses a method, system, and product for constructing a terrain semantic map based on a binocular camera. This method uses a binocular camera to achieve three-dimensional reconstruction of semantic information, improving the accuracy of the three-dimensional environment map. However, this method only removes pixels of dynamic objects and does not address the accuracy of semantic segmentation. Chinese Patent Publication No. CN116452929A discloses a method and system for constructing a terrain semantic map in a field environment. This method uses a lidar to realize a terrain semantic map and improve the positioning accuracy of an unmanned vehicle. However, the lidar used in this method is expensive, and the method of only removing keyframes of dynamic objects is not applicable to most situations. Chinese Patent Publication No. CN117693768A discloses a method and device for optimizing a semantic segmentation model. This method performs semantic segmentation based on annotated images and unannotated images separately, and then uses the results of the latter to optimize the results of the former. This method optimizes the semantic segmentation effect of the annotated images, but is not real-time and cannot be directly used on a robot. In summary, the current application methods of semantic segmentation are mature, but for most robots, it is not only necessary to identify the type of terrain, but also to have a complete framework for identifying the terrain, and at the same time, it must have the ability to run in real time on the robot's host. Summary of the Invention

[0004] The purpose of the present invention is to provide a structural terrain semantic segmentation image optimization method based on gradient data, which can optimize the semantic segmentation results of structural terrain and can be deployed on a computer platform, so as to quickly and efficiently perform accurate semantic segmentation of the structural terrain in front.

[0005] The technical solution to achieve the purpose of the present invention is:

[0006] A method for optimizing structured terrain semantic segmentation images based on gradient data, characterized in that the strategy includes the following steps: S1. Calculate the gradient value of each pixel of the depth image based on the Sobel operator; S2, normalize the gradient value to give RGB information and derive the normal map based on the color space; S3, the normal map based on the color space and the RGB image are fused using the Max method to construct an improved structural terrain image; S4. Build a terrain detection model based on the improved structural terrain image; S5, use the TensorRT framework to optimize and accelerate the S4 model; S6, softmax post-processing based on the image processing results of the terrain detection model; S7, perform Canny edge calculation based on normal data; S8, perform edge optimization on the S6 reasoning result based on boundary point data and gradient data; S9. Model evaluation.

[0007] Furthermore, in step S1, a depth map containing depth information is obtained by a depth camera, and a gradient calculation is performed on the depth map according to the grayscale difference between each pixel in the depth map, based on the Sobel operator in the horizontal and vertical directions: The depth image is a two-dimensional matrix. Here, Depth represents the depth map matrix. The Sobel operator is convolved with the depth map Depth to calculate the horizontal partial derivative G. x and the vertical partial derivative G y : Get the gradient data:

[0008] Furthermore, in step S2, the horizontal partial derivative G calculated in step 2 isx and the vertical partial derivative G y , stack it with a full 1 matrix to form a three-dimensional matrix Nor. The normal vector modulus of Nor along the third dimension is Then, the Nor normalization process is mapped to the image space. Based on the color coding, the brightness value of the pixel ranges from [0, 255], and its interval length is defined as C. The normalized normal vector component ranges from [-1, 1], and its interval length is proportion = 2: Gets a normal image based on the color space.

[0009] Furthermore, in step S3, both the Normal image matrix and the RGB image matrix are three-dimensional matrices, and the two are Max-valued along the third dimension. Assume that the vector corresponding to a certain position (μ, υ) on the matrix on the Normal matrix is ​​l Normal (μ, υ) = (l1, l2, l3), the corresponding vector on the RGB matrix is ​​L RGB (μ,υ)=(L1, L2, L3), take the Max value for it, l Merge (μ,υ)=Max(l Normal , L RGB )=Max{(l1, L1), (l2, L2), (l3, L3)} Will l Merge (μ, v) is assigned to the third dimension of the new matrix Merge, and its first two dimensions are the same as the Normal matrix and the RGB matrix, thereby obtaining the Merge image matrix of the improved structural terrain image.

[0010] Furthermore, in step S4, the improved structural terrain image is annotated using image annotation software, and the annotated file is converted into a VOC format annotation file, and the training set, validation set, and test set are divided. Finally, a lightweight network (MobileNetV3-large) is used to build a terrain detection model based on the improved structural terrain image. Data set loading and a series of data enhancement processes are set. The hyperparameter learning rate is set to the range of lr = [0.001, 0.0001], the momentum is set to the range of momentum = [0.8, 0.9], and the weight decay weight_decay is set to 0.001. The model is trained from the initial state for 500 to 1000 rounds to obtain a terrain detection model based on the improved structural terrain image.

[0011] Furthermore, in step S5, the TensorRT framework is used to optimize and accelerate the model, and the Pytorch model is converted into a TensorRT model, where the precision can be N, where NNN∈[FP32, FP16, INT8].

[0012] Furthermore, in step S6, the terrain detection model infers and predicts the resulting image data of the improved structural terrain image, including the classification label of a certain pixel point and its corresponding confidence value. The original confidence value is normalized by the softmax method. Assuming that there are K classifications, the K-dimensional original vector z is converted into a K-dimensional real vector Softmax(z). Then, Where e is the base of natural logarithms and i represents the index of the element in the original vector z. After Softmax processing, the probability distribution for each category is obtained. Based on the probability distribution, further threshold processing is performed to select the categories with a probability greater than or equal to 50% and less than or equal to 70% among the pixels of each category as the fuzzy classification pixel set.

[0013] Furthermore, in step S7, based on the Canny edge calculation algorithm and the normal data based on the color space obtained in step S2, Gaussian filtering is performed, and then non-maximum suppression is directly applied to refine the edge. Finally, the double threshold algorithm is used to connect and track the edge to obtain the pixel coordinate set I of the boundary point. edge =(μ,υ).

[0014] Furthermore, in step S8, based on the gradient data obtained in step S2 and the pixel coordinates of the boundary points obtained in step S7, the fuzzy classification pixel point set obtained in step S6 is subjected to secondary classification processing. In the first step, each pixel point in the fuzzy classification pixel point set is compared with the boundary point coordinates to determine whether it is a boundary point. If it is a boundary point, its classification category is retained; in the second step, it is determined whether there are boundary points around the fuzzy classification pixel point set. If there are boundary points, they are key feature points. It is determined whether the key feature point is within the boundary. If it is within the boundary, the category of the point is changed to The boundary category is selected, and if it is outside the boundary, its category is changed to the category outside the boundary; the third step is to align the coordinates of the fuzzy classification pixel point set with the coordinates of the gradient value set, and determine whether its gradient value exceeds the threshold k = [100,,110]. If it exceeds, the point is a boundary point and its category is retained; the fourth step is to calculate the difference between the gradient value of the fuzzy classification pixel point set and the gradient value of the surrounding points. If one of the difference values ​​exceeds the threshold d = [20,30], the point is a key feature point, and its category is determined according to the second step, thereby realizing edge optimization based on the improved structural terrain semantic segmentation image.

[0015] Furthermore, in step S9, considering that the proposed method is performed on the results obtained by semantic segmentation, in order to verify the accuracy of the method, pixel accuracy (pixAcc) and mean intersection over union (mIoU) are used for evaluation.

[0016] Compared with the existing semantic segmentation methods for structural terrain, this invention has the following significant advantages:

[0017] (1) Compared with traditional RGB images, the improved structural terrain image based on gradient data proposed in this invention reduces the interference of background images behind the terrain, better displays the characteristics and structural properties of the structural terrain, reduces the number of data sets required for model training, and improves the efficiency of semantic model training and the accuracy of the model in identifying structural terrain.

[0018] (2) The semantic image edge optimization algorithm based on gradient data and boundary point coordinates proposed in the present invention further refines the boundaries of traditional semantic segmentation images, solves the problems of blurred and inaccurate boundaries, and significantly optimizes the boundaries after semantic segmentation, making the boundaries clearer and more accurate, thereby significantly improving the accuracy and precision of the semantic segmentation results.

[0019] (3) The present invention proposes a structural terrain semantic segmentation image optimization method based on gradient data, which takes into account the low computing power of general computer CPUs and GPUs. It uses TensorRT to optimize and accelerate the semantic segmentation of structural terrain. Without affecting the segmentation results, it greatly improves the frame rate during semantic segmentation and has a more reliable effect when deployed on a real machine. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 To implement an image optimization method for structural terrain semantic segmentation based on gradient data;

[0021] Figure 2 for Figure 1 Schematic diagram of image data, (a) is RGB image, (b) is depth map image, (c) is normal data based on color space, and (d) is improved structural terrain image;

[0022] Figure 3 Schematic diagram of annotation, (a) is RGB image annotation, (b) is normal map annotation, and (c) is improved structural terrain image annotation;

[0023] Figure 4 Schematic diagram of the optimization method;

[0024] Figure 5 The optimization effect diagram is shown in Figure 1. (a) is the image before optimization, and (b) is the image after optimization.

[0025] Figure 6 Schematic diagram of running this model on a general host. DETAILED DESCRIPTION

[0026] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0027] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0028] like Figure 1 FIG. 1 is a flow chart of a method for optimizing a structured terrain semantic segmentation image based on gradient data according to the present invention, wherein the method comprises the following steps: S1. Calculate the gradient value of each pixel of the depth image based on the Sobel operator; S2, normalize the gradient value to give RGB information and derive the normal map based on the color space; S3, the normal map based on the color space and the RGB image are fused using the Max method to construct an improved structural terrain image; S4. Build a terrain detection model based on the improved structural terrain image; S5, use the TensorRT framework to optimize and accelerate the S4 model; S6, softmax post-processing based on the image processing results of the terrain detection model; S7, perform Canny edge calculation based on normal data; S8, perform edge optimization on the S6 reasoning result based on boundary point data and gradient data; S9. Model evaluation.

[0029] like Figure 2 As shown in (a) and (b), in step S1, a depth map containing depth information is obtained by a depth camera, and a gradient calculation is performed on the depth map according to the grayscale difference between each pixel in the depth map, based on the Sobel operator in the horizontal and vertical directions: The depth image is a two-dimensional matrix. Here, Depth represents the depth map matrix. The Sobel operator is convolved with the depth map Depth to calculate the horizontal partial derivative G. x and the vertical partial derivative G y : Get the gradient data:

[0030] like Figure 2 As shown in (c), in step S2, the horizontal partial derivative G calculated in step S1 is x and the vertical partial derivative G y , stack it with a full 1 matrix to form a three-dimensional matrix Nor. The normal vector modulus of Nor along the third dimension is Then, the Nor normalization process is mapped to the image space. Based on the color coding, the brightness value of the pixel ranges from [0, 255], and its interval length is defined as C. The normalized normal vector component ranges from [-1, 1], and its interval length is proportion = 2: Gets a normal image based on the color space.

[0031] like Figure 2 As shown in (d), in step S3, the Normal matrix and the RGB matrix are both three-dimensional matrices. Max values ​​are taken along the third dimension. Assume that the vector corresponding to a position (μ, u) on the matrix on the Normal matrix is ​​l Normal (μ, υ) = (l1, l2, l3), the corresponding vector on the RGB matrix is ​​L RGB (μ,υ)=(L1, L2, L3), take the Max value for it, l Merge (μ,υ)=Max(l Normal , L RGB )=Max{(l1, L1), (l2, L2), (l3, L3)} Will l Merge (μ, v) is assigned to the third dimension of the new matrix Merge, and its first two dimensions are the same as the Normal matrix and the RGB matrix, thereby obtaining the Merge image matrix of the improved structural terrain image.

[0032] like Figure 3The figure shows a diagram of annotation. In step S4, the improved structural terrain image is annotated using image annotation software. The annotated file is then converted to a VOC format annotation file and randomly divided into a training set, a validation set, and a test set in a ratio of 4:4:1. Finally, a lightweight network (MobileNetV3-large) is used to build a terrain detection model based on the improved structural terrain image. The hyperparameter learning rate is set to the range of lr = [0.001, 0.0001], the momentum is set to the range of momentum = [0.8, 0.9], and the weight decay weight_decay is set to 0.001. The model is trained from the initial state for 500 to 1000 rounds to obtain a terrain detection model based on the improved structural terrain image.

[0033] In step S5, the TensorRT framework is used to optimize and accelerate the model, and the Pytorch model is converted into a TensorRT model, where the accuracy can be NNN, NNN∈[FP32, FP16, INT8].

[0034] In step S6, the terrain detection model infers and predicts the resulting image data of the improved structural terrain image, including the classification label of a certain pixel point and its corresponding value. The original output value is normalized using the softmax method to calculate the probability distribution. Assuming that there are K types of classification, the K-dimensional original vector z is converted into a K-dimensional real vector Softmax(z). Then, Where e is the base of natural logarithms and i represents the index of the element in the original vector z. After Softmax processing, the probability distribution for each category is obtained. Based on the probability distribution, further threshold processing is performed to select the categories with a probability greater than or equal to 50% and less than or equal to 70% among the pixels of each category as the fuzzy classification pixel set.

[0035] In step S7, based on the Canny edge calculation algorithm and the normal data based on the color space obtained in step S2, Gaussian filtering is performed, and then non-maximum suppression is directly applied to refine the edge. Finally, the double threshold algorithm is used to connect and track the edge to obtain the pixel coordinate set I of the boundary point. edge =(μ,υ).

[0036] like Figure 4As shown in FIG, a schematic diagram of the optimization method, wherein p is a certain pixel point, and the color and gradient values ​​around the point are marked. In step S8, based on the gradient data obtained in step S2 and the pixel coordinates of the boundary points obtained in step S7, the fuzzy classified pixel point set obtained in step S6 is subjected to secondary classification processing. In the first step, each pixel in the fuzzy classified pixel set is compared with the coordinates of the boundary points to determine whether it is a boundary point. If it is a boundary point, its classification category is retained. In the second step, whether there are boundary points around the fuzzy classified pixel set is determined. If there are boundary points, the point is a key feature point. The key feature point is determined to be within the boundary. If it is within the boundary, the point's category is changed to the within-boundary category; if it is outside the boundary, the point's category is changed to the outside-boundary category. In the third step, the coordinates of the fuzzy classified pixel set are aligned with the coordinates of the gradient value set to determine whether the gradient value exceeds the threshold k = [100, 110]. If so, the point is a boundary point and its category is retained. In the fourth step, the difference between the gradient value of the fuzzy classified pixel set and the gradient value of the surrounding points is calculated. If any of the difference values ​​exceeds the threshold d = [20, 30], the point is a key feature point and its category is determined according to the second step. This method achieves edge optimization based on improved structural terrain semantic segmentation images.

[0037] like Figure 5 As shown in FIG, it is an optimization effect diagram. In the step S8, considering that the proposed method is performed on the results obtained by semantic segmentation, in order to verify the accuracy of the method, pixel accuracy (pixAcc) and mean intersection over union (mIoU) are used for evaluation. The data in the table shows that using the improved structural terrain map as training images and performing edge optimization on the output results improves pixAcc by 2.3% and mIoU by 12.2% compared to using only RGB images, but the FPS decreases by 13.9%. Although the overall speed decreases slightly, the decrease is not significant and remains above 30 FPS. The improvements in pixAcc and mIoU are relatively significant, indicating a strong optimization effect. This comparison demonstrates that deploying our method on any network model offers superior accuracy to classic terrain detection methods.

[0038] The above-mentioned tool damage image target detection model has a compact structure. Compared with the classic target detection network, it has higher prediction accuracy and faster speed. At the same time, it takes into account the problem of insufficient computing power of the edge computing platform, thereby expanding the use scenarios of the model and providing a reliable and advantageous solution for target detection tasks that require high precision and low latency.

[0039] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for optimizing structured terrain semantic segmentation images based on gradient data, characterized in that: The strategy includes the following steps: S1. Calculate the gradient value of each pixel of the depth image based on the Sobel operator; S2, normalize the gradient value to give RGB information and derive the normal map based on the color space; S3, the normal map based on the color space and the RGB image are fused using the Max method to construct an improved structural terrain image; S4. Build a terrain detection model based on the improved structural terrain image; S5, use the TensorRT framework to optimize and accelerate the S4 model; S6, softmax post-processing based on the image processing results of the terrain detection model; S7, perform Canny edge calculation based on normal data; S8, edge optimization of the reasoning result of S6 based on boundary point data and gradient data; S9. Model evaluation.

2. The method for optimizing structured terrain semantic segmentation images based on gradient data according to claim 1, characterized in that: In step S1, a depth map containing depth information is obtained through a depth camera, and a gradient is calculated based on the grayscale difference between each pixel of the depth image, based on the Sobel operator in the horizontal and vertical directions: and The depth image is a two-dimensional matrix. Here, Depth represents the depth map matrix. The Sobel operator is convolved with the depth map Depth to calculate the horizontal partial derivative G. x and the vertical partial derivative G y : Get the gradient data:

3. The method for optimizing structured terrain semantic segmentation images based on gradient data according to claim 1, characterized in that: In step S2, the horizontal partial derivative G calculated in step 2 x and the vertical partial derivative G y , stack it with a full 1 matrix to form a three-dimensional matrix Nor, where the normal vector modulus of Nor along the third dimension is Then, the Nor normalization is mapped to the image space. Based on color coding, the brightness value of the pixel ranges from [0, 255], and its interval length is defined as C. The normalized normal vector component ranges from [-1, 1], and its interval length is proportion = 2: Get the normal image based on the color space.

4. The method for optimizing structured terrain semantic segmentation images based on gradient data according to claim 1, characterized in that: In step S3, both the Normal image matrix and the RGB image matrix are three-dimensional matrices, and the two are Max-valued along the third dimension. Assume that the vector corresponding to a certain position (μ, υ) on the matrix on the Normal matrix is ​​l Normal (μ, υ) = (l1, l2, l3), the corresponding vector on the RGB matrix is ​​L RGB (μ,υ)=(L1,L2,L3), and take the Max value for it. L Merge (μ,υ)=Max(l Normal ,L RcB )=Max{(l1,L1),(l2,L2),(l3,L3)} Will l Merge (μ, υ) is assigned to the third dimension of the new matrix Merge, and its first two dimensions are the same as the Normal matrix and the RGB matrix, so as to obtain the Merge image matrix of the improved structural terrain image.

5. The method for optimizing structured terrain semantic segmentation images based on gradient data according to claim 1, characterized in that: In the step S4, the improved structural terrain image is annotated using image annotation software, and then the annotation file is converted into a annotation file in VOC format, and the training set, validation set and test set are divided. Finally, a lightweight network (MobileNetV3-large) is used to build a terrain detection model based on the improved structural terrain image, set data set loading and a series of data enhancement processing, set the hyperparameter learning rate value range to lr=[0.001, 0.0001], set the momentum value range to momentum=[0.8, 0.9], set weight decay weight_decay=0.001, and train the model from the initial state for 500 to 1000 rounds to obtain a terrain detection model based on the improved structural terrain image.

6. The method for optimizing structured terrain semantic segmentation images based on gradient data according to claim 1, characterized in that: In step S5, the TensorRT framework is used to optimize and accelerate the model, and the Pytorch model is converted into a TensorRT model, where the precision can be N, where NNN∈[FP32, FP16, INT8].

7. The method for optimizing structured terrain semantic segmentation images based on gradient data according to claim 1, characterized in that: In step S6, the terrain detection model infers and predicts the result image data of the improved structural terrain image, including the classification label of a certain pixel point and its corresponding confidence value. The original confidence value is normalized by probability distribution using the softmax method. Assuming that there are K types of classification, the K-dimensional original vector z is converted into a K-dimensional real vector Softmax(z), then, Among them, e is the base of natural logarithm, i represents the element index in the original vector z. After Softmax processing, the probability distribution for each classification is obtained. Based on the probability distribution, further threshold processing is performed to take the classifications with probabilities greater than or equal to 50% and less than or equal to 70% in the pixels of each classification as the fuzzy classification pixel set.

8. The method for optimizing structured terrain semantic segmentation images based on gradient data according to claim 1, characterized in that: In step S7, based on the Canny edge calculation algorithm and the normal data based on the color space obtained in step S2, Gaussian filtering is performed, and then non-maximum suppression is directly applied to refine the edge, and finally the double threshold algorithm is used to connect and track the edge to obtain the pixel coordinate set I of the boundary point. edge =(μ, v).

9. The method for optimizing structured terrain semantic segmentation images based on gradient data according to claim 1, characterized in that: In step S8, based on the gradient data obtained in step S2 and the pixel coordinates of the boundary points obtained in step S7, the fuzzy classified pixel point set obtained in step S6 is subjected to secondary classification processing. In the first step, each pixel point in the fuzzy classified pixel point set is compared with the boundary point coordinates to determine whether it is a boundary point. If it is a boundary point, its classification category is retained; The second step is to determine whether there are boundary points around the fuzzy classification pixel set. If there are boundary points, they are key feature points. Determine whether the key feature points are within the boundary. If they are within the boundary, change the category of the point to the category within the boundary. If they are outside the boundary, change their category to the category outside the boundary. The third step is to align the coordinates of the fuzzy classification pixel set with the coordinates of the gradient value set, and determine whether their gradient values ​​exceed the threshold value k = [100, 110]. If they exceed, the point is a boundary point and its category is retained. The fourth step is to calculate the difference between the gradient value of the fuzzy classified pixel point set and the gradient value of the surrounding points. If one of the difference values ​​exceeds the threshold d = [20, 30], the point is a key feature point, and its category is determined according to the second step to achieve edge optimization based on the improved structural terrain semantic segmentation image.

10. The method for optimizing structured terrain semantic segmentation images based on gradient data according to claim 1, characterized in that: In step S9, considering that the proposed method is performed on the results obtained by semantic segmentation, in order to verify the accuracy of the method, pixel accuracy (pixAcc) and mean intersection over union (mIoU) are used for evaluation.

Images