A gradient data-based structural terrain semantic segmentation image optimization method

Abstract

The application discloses a structure terrain semantic segmentation image optimization method based on gradient data. The method comprises the following steps: converting depth data (Depth) into gradient data with image pixel gray value change based on a Sobel operator, and normalizing the gradient data to show a normal map with red, green and blue (RGB) data; fusing the normal map and the red, green and blue (RGB) data according to a pixel point matrix Max method, re-encoding the input image, and outputting image information with color representation of normal direction information and basic RGB image fusion; and based on the Canny edge detection data and the gradient data of the normal map, optimizing the edges of the semantic segmentation result of the MobileNetV3 lightweight network. Compared with the prior art, the method optimizes the edges of the semantic image, and simultaneously considers speed and accuracy. The semantic segmentation edge optimization method is simple and efficient, and is suitable for semantic segmentation of most structure terrains.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision and robot perception, specifically a method for optimizing structural terrain semantic segmentation images based on gradient data. Background Technology

[0002] With the diversification of robot structures, robots' ability to traverse various terrains has been gradually enhanced. Consequently, the demands on robots' environmental perception capabilities have increased significantly. Accurately identifying and traversing the terrain ahead is one of the current challenges in robot perception. This is especially true for robots that require path planning; the completeness of terrain recognition directly impacts their ability to correctly utilize terrain information for subsequent planning. Therefore, terrain recognition based on semantic segmentation is gradually becoming a new trend. However, most semantic segmentation methods exhibit gaps around the boundary lines, resulting in the inability to fully utilize the terrain even after it has been identified. Therefore, post-processing and optimization of semantic segmentation are crucial for better segmenting the entire terrain framework, enabling robots to better utilize terrain information for subsequent planning.

[0003] Chinese Patent Publication No. CN1 15690429A discloses a terrain semantic segmentation method for a quadruped robot, which effectively prevents the problem of blurry images caused by the shaking of the quadruped robot during movement. However, this method only solves the image collection problem under jitter, and does not solve the problem of incomplete semantic segmentation. Chinese patent publication number CN116883586A discloses a method, system and product for constructing terrain semantic maps based on binocular cameras. This method uses binocular cameras to realize the three-dimensional reconstruction of semantic information and improves the accuracy of the three-dimensional environment map. However, this method only removes the pixels of dynamic objects and does not process the accuracy of semantic segmentation. Chinese patent publication number CN116452929A discloses a method and system for constructing terrain semantic maps in the wild environment. This method uses LiDAR to realize terrain semantic maps and improves the positioning accuracy of unmanned vehicles. However, the LiDAR used in this method is expensive, and the method of only removing keyframes of dynamic objects is not suitable for most situations. Chinese patent publication number CN117693768A discloses a method and device for optimizing semantic segmentation models. This method performs semantic segmentation based on labeled images and unlabeled images respectively, and then uses the result of the latter to optimize the result of the former. This method optimizes the semantic segmentation effect of labeled images, but it is not real-time and cannot be directly used on robots. In summary, current methods for semantic segmentation are mature, but for most robots, it is not only necessary to identify the type of terrain, but also to have a complete framework for identifying that terrain, and at the same time, it is necessary to have the ability to run in real time on the robot's host. Summary of the Invention

[0004] The purpose of this invention is to provide an image optimization method for semantic segmentation of structured terrain based on gradient data. This method can optimize the semantic segmentation results of structured terrain and can be deployed on a computer platform, thereby enabling fast and efficient accurate semantic segmentation of the structured terrain in front of the image.

[0005] The technical solution to achieve the objective of this invention is as follows:

[0006] A method for optimizing structured terrain semantic segmentation images based on gradient data, characterized in that the strategy includes the following steps: S1. Calculate the gradient value of each pixel in the depth image based on the Sobel operator; S2. Gradient value normalization assigns RGB information and exports a color space-based normal map; S3. The color space-based normal map and RGB image are fused using the Max method to construct an improved structured terrain image; S4. Build a terrain detection model based on improved structured terrain images; S5. Use the TensorRT framework to optimize and accelerate the model in S4; S6. Softmax post-processing based on image processing results of terrain detection model; S7. Perform Canny edge calculation based on normal data; S8. Perform edge optimization on the inference results of S6 based on boundary point data and gradient data; S9. Model Evaluation.

[0007] Furthermore, in step S1, a depth map containing depth information is acquired using a depth camera. Gradient calculation is performed on the depth map based on the grayscale differences between pixels, using the Sobel operator in both the horizontal and vertical directions. The depth image is a two-dimensional matrix, here denoted as Depth. The Sobel operator is convolved with the depth image Depth to calculate the horizontal partial derivative G. x and the vertical partial derivative G y : Obtain gradient data:

[0008] Furthermore, in step S2, the horizontal partial derivative G calculated in step 2...x and the vertical partial derivative G y Stack this matrix with a matrix of all 1s to form a three-dimensional matrix Nor. The normal vector of Nor along the third dimension has a magnitude of . Then, the Nor normalization process is mapped to the image space. Based on color encoding, the brightness value of the pixel ranges from [0, 255], and its interval length is defined as C. The normalized normal vector component ranges from [-1, 1], and its interval length is proportion = 2. Obtain a normal image based on the color space.

[0009] Furthermore, in step S3, both the Normal image matrix and the RGB image matrix are three-dimensional matrices. Max values ​​are taken along the third dimension of both matrices. It is assumed that a certain position (μ, υ) on the matrix corresponds to a vector l on the Normal matrix. Normal (μ, υ) = (l1, l2, l3), and the corresponding vector on the RGB matrix is ​​L. RGB Given (μ, υ) = (L1, L2, L3), take its maximum value. l Merge (μ, υ) = Max(l) Normal L RGB )=Max{(l1, L1), (l2, L2), (l3, L3)} l Merge (μ, v) is assigned to the third dimension of the new matrix Merge, whose first two dimensions are the same as the Normal matrix and the RGB matrix, resulting in the Merge image matrix of the improved structured terrain image.

[0010] Furthermore, in step S4, the improved structural terrain image is annotated using image annotation software, and then the annotation file is converted into a VOC format annotation file, dividing it into training, validation, and test sets. Finally, a lightweight network (MobileNetV3-large) is used to build a terrain detection model based on the improved structural terrain image. Dataset loading and a series of data augmentation processes are set. The hyperparameter learning rate is set to a range of lr = [0.001, 0.0001], the momentum is set to a range of momentum = [0.8, 0.9], and the weight decay is set to weight_decay = 0.001. The model is trained for 500 to 1000 epochs from its initial state to obtain the terrain detection model based on the improved structural terrain image.

[0011] Furthermore, in step S5, the TensorRT framework is used to optimize and accelerate the model, converting the PyTorch model into a TensorRT model, where the accuracy can be N, where N ∈ [FP32, FP16, INT8].

[0012] Furthermore, in step S6, the terrain detection model infers and predicts the improved structural terrain image, resulting in image data containing the classification label of a pixel and its corresponding confidence value. The original confidence value is normalized using the softmax method. Assuming there are K classifications, the K-dimensional original vector z is transformed into a K-dimensional real vector Softmax(z). Where e is the base of the natural logarithm, and i represents the index of an element in the original vector z. After Softmax processing, a probability distribution for each category is obtained. Based on the probability distribution, a thresholding process is further performed, and the category with a probability greater than or equal to 50% and less than or equal to 70% is selected as the fuzzy classification pixel set.

[0013] Furthermore, in step S7, based on the Canny edge calculation algorithm and the color space-based normal data obtained in step S2, Gaussian filtering is performed, then non-maximum suppression is directly applied to refine the edges, and finally, a double threshold algorithm is used for edge connection and edge tracking to obtain the pixel coordinate set I of the boundary points. edge = (μ, υ).

[0014] Furthermore, in step S8, based on the gradient data obtained in step S2 and the pixel coordinates of the boundary points obtained in step S7, a secondary classification process is performed on the fuzzy classification pixel set obtained in step S6. First, the coordinates of each pixel in the fuzzy classification pixel set are compared with those of the boundary points to determine if it is a boundary point; if it is, its classification category is retained. Second, it is determined whether there are boundary points around the fuzzy classification pixel set; if there are, they are key feature points. It is then determined whether the key feature point is within the boundary; if it is, its category is changed. The first step is to classify the points within the boundary. If a point is outside the boundary, its category is changed to the category outside the boundary. The second step is to align the coordinates of the fuzzy classification pixel set with the coordinates of the gradient value set and determine whether its gradient value exceeds the threshold k = [100, 110]. If it does, the point is a boundary point and its category is retained. The third step is to align the coordinates of the fuzzy classification pixel set with the gradient values ​​of the surrounding points. If one of the differences exceeds the threshold d = [20, 30], the point is a key feature point and its category is determined according to the second step, thus realizing edge optimization of the image based on the improved structural terrain semantic segmentation.

[0015] Furthermore, in step S9, considering that the proposed method is performed on the results obtained from semantic segmentation, the pixel accuracy (pixAcc) and average intersection-union ratio (mIoU) are used for evaluation to verify the accuracy of the method.

[0016] Compared to existing structural terrain semantic segmentation methods, the significant advantages of this invention are:

[0017] (1) The improved structured terrain image based on gradient data proposed in this invention reduces the interference of background images behind the terrain compared with traditional RGB images, and can better display the features and structural characteristics of the structured terrain. It reduces the amount of dataset required for model training, while improving the efficiency of semantic model training and the accuracy of model recognition of structured terrain.

[0018] (2) The semantic image edge optimization algorithm based on gradient data and boundary point coordinates proposed in this invention further refines the boundary to address the problems of blurred and inaccurate boundaries in traditional semantic segmentation images. It significantly optimizes the boundary after semantic segmentation, making the boundary clearer and more accurate, and significantly improves the accuracy and precision of semantic segmentation results.

[0019] (3) The proposed method for optimizing semantic segmentation of structured terrain based on gradient data takes into account the limited computing power of general computer CPUs and GPUs. It utilizes TensorRT to optimize and accelerate the semantic segmentation of structured terrain, significantly improving the frame rate during semantic segmentation without affecting the segmentation results. It has a more reliable effect when deployed on a real machine. Attached Figure Description

[0020] Figure 1 To implement an image optimization method for semantic segmentation of structured terrain based on gradient data;

[0021] Figure 2 for Figure 1 Schematic diagram of image data: (a) is an RGB image, (b) is a depth map image, (c) is color space-based normal data, and (d) is an improved structured terrain image;

[0022] Figure 3 The following are illustrations of the annotations: (a) RGB image annotation, (b) normal map annotation, and (c) improved structural terrain image annotation.

[0023] Figure 4 This is a schematic diagram of the optimization method;

[0024] Figure 5 To optimize the results, (a) is the image before optimization, and (b) is the image after optimization;

[0025] Figure 6 This is a schematic diagram of how the model runs on a typical host computer. Detailed Implementation

[0026] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0028] like Figure 1 The diagram shown is a flowchart illustrating a method for optimizing structural terrain semantic segmentation images based on gradient data, as described in this invention. The method includes the following steps: S1. Calculate the gradient value of each pixel in the depth image based on the Sobel operator; S2. Gradient value normalization assigns RGB information and exports a color space-based normal map; S3. The color space-based normal map and RGB image are fused using the Max method to construct an improved structured terrain image; S4. Build a terrain detection model based on improved structured terrain images; S5. Use the TensorRT framework to optimize and accelerate the model in S4; S6. Softmax post-processing based on image processing results of terrain detection model; S7. Perform Canny edge calculation based on normal data; S8. Perform edge optimization on the inference results of S6 based on boundary point data and gradient data; S9. Model Evaluation.

[0029] like Figure 2 As shown in (a)(b), in step S1, a depth map containing depth information is acquired by a depth camera, and gradient calculation is performed on it based on the gray-level difference between each pixel in the depth image, using the Sobel operator in the horizontal and vertical directions: The depth image is a two-dimensional matrix, here denoted as Depth. The Sobel operator is convolved with the depth image Depth to calculate the horizontal partial derivative G. x and the vertical partial derivative G y : Obtain gradient data:

[0030] like Figure 2 As shown in (c), in step S2, the horizontal partial derivative G calculated in step S1... x and the vertical partial derivative G y Stack this matrix with a matrix of all 1s to form a three-dimensional matrix Nor. The normal vector of Nor along the third dimension has a magnitude of . Then, the Nor normalization process is mapped to the image space. Based on color encoding, the brightness value of the pixel ranges from [0, 255], and its interval length is defined as C. The normalized normal vector component ranges from [-1, 1], and its interval length is proportion = 2. Obtain a normal image based on the color space.

[0031] like Figure 2 As shown in (d), in step S3, both the Normal matrix and the RGB matrix are three-dimensional matrices. Max values ​​are taken along the third dimension of both matrices. Assume that a certain position (μ, u) on the matrix corresponds to the vector l on the Normal matrix. Normal (μ, υ) = (l1, l2, l3), and the corresponding vector on the RGB matrix is ​​L. RGB Given (μ, υ) = (L1, L2, L3), take its maximum value. l Merge (μ, υ) = Max(l) Normal L RGB )=Max{(l1, L1), (l2, L2), (l3, L3)} l Merge (μ, v) is assigned to the third dimension of the new matrix Merge, whose first two dimensions are the same as the Normal matrix and the RGB matrix, resulting in the Merge image matrix of the improved structured terrain image.

[0032] like Figure 3As shown in the diagram, in step S4, the improved structural terrain image is annotated using image annotation software, and then the annotation file is converted into a VOC format annotation file. The annotation file is then randomly divided into a training set, a validation set, and a test set in a 4:4:1 ratio. Finally, a lightweight network (MobileNetV3-large) is used to build a terrain detection model based on the improved structural terrain image. The hyperparameter learning rate is set to lr = [0.001, 0.0001], the momentum is set to momentum = [0.8, 0.9], and the weight decay is set to weight_decay = 0.001. The model is trained for 500-1000 epochs from its initial state to obtain the terrain detection model based on the improved structural terrain image.

[0033] In step S5, the TensorRT framework is used to optimize and accelerate the model, converting the PyTorch model into a TensorRT model, where the accuracy can be NNN, NNN∈[FP32, FP16, INT8].

[0034] In step S6, the terrain detection model infers and predicts the improved structured terrain image, resulting in image data containing the classification label of a pixel and its corresponding value. The original output value is normalized using the softmax method to calculate the probability distribution. Assuming there are K classifications, the K-dimensional original vector z is transformed into a K-dimensional real vector Softmax(z). Where e is the base of the natural logarithm, and i represents the index of an element in the original vector z. After Softmax processing, a probability distribution for each category is obtained. Based on the probability distribution, a thresholding process is further performed, and the category with a probability greater than or equal to 50% and less than or equal to 70% is selected as the fuzzy classification pixel set.

[0035] In step S7, based on the Canny edge calculation algorithm and the color space-based normal data obtained in step S2, Gaussian filtering is performed, then non-maximum suppression is directly applied to refine the edges, and finally, a double threshold algorithm is used for edge connection and edge tracking to obtain the pixel coordinate set I of the boundary points. edge = (μ, υ).

[0036] like Figure 4As shown, this is a schematic diagram of the optimization method, where p is a certain pixel point, and the colors and gradient values ​​around the point have been marked. In step S8, based on the gradient data obtained in step S2 and the pixel coordinates of the boundary points obtained in step S7, the fuzzy classification pixel point set obtained in step S6 is subjected to secondary classification processing. The first step involves comparing the coordinates of each pixel in the fuzzy classification pixel set with those of the boundary points to determine if it is a boundary point. If it is, its classification is retained. The second step involves checking if any boundary points exist around the fuzzy classification pixel set. If boundary points exist, they are considered key feature points. The key feature points are then checked to see if they are within the boundary. If they are, their classification is changed to the "within the boundary" category; otherwise, they are changed to the "outside the boundary" category. The third step involves aligning the coordinates of the fuzzy classification pixel set with the coordinates of the gradient value set and checking if the gradient value exceeds the threshold k = [100, 110]. If it does, the point is considered a boundary point, and its classification is retained. The fourth step involves calculating the difference between the gradient value of the fuzzy classification pixel set and the gradient values ​​of the surrounding points. If any difference exceeds the threshold d = [20, 30], the point is considered a key feature point, and its classification is determined according to the second step. This achieves edge optimization based on improved structural terrain semantic segmentation of images.

[0037] like Figure 5 As shown in the figure, in step S8, considering that the proposed method is based on the results obtained from semantic segmentation, the pixel accuracy (pixAcc) and average intersection-union ratio (mIoU) are used to evaluate the accuracy of the method in order to verify its accuracy. The data in the table shows that using the improved structural topographic map as the training image and performing edge optimization on the output results improves pixAcc by 2.3% and mIoU by 12.2% compared to using only RGB images, but reduces FPS by 13.9%. Although the overall speed decreases somewhat, the decrease is not significant and it remains above 30 frames per second. Relatively speaking, the improvements in pixAcc and mIoU are relatively large, indicating good optimization. The above comparison shows that deploying the method of this invention on any network model has a greater advantage in accuracy than classical terrain detection methods.

[0038] The aforementioned target detection model for tool damage images has a compact structure and, compared to classic target detection networks, offers higher prediction accuracy and faster speed. It also addresses the issue of insufficient computing power on edge computing platforms, thereby expanding the model's application scenarios and providing a reliable and advantageous solution for target detection tasks requiring high accuracy and low latency.

[0039] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for optimizing structured terrain semantic segmentation images based on gradient data, characterized in that, The method includes the following steps: S1. Calculate the gradient value of each pixel in the depth image based on the Sobel operator; S2. Gradient value normalization assigns RGB information and exports a color space-based normal map; S3. The color space-based normal map and RGB image are fused using the Max method to construct an improved structured terrain image; S4. Build a terrain detection model based on improved structured terrain images; S5. Use the TensorRT framework to optimize and accelerate the model in S4; S6. Softmax post-processing based on image processing results of terrain detection model; S7. Perform Canny edge calculation based on normal data; S8. Perform edge optimization on the inference results of S6 based on boundary point data and gradient data; S9. Model Evaluation; In step S8, based on the gradient data obtained in step S2 and the pixel coordinates of the boundary points obtained in step S7, a secondary classification process is performed on the fuzzy classification pixel set obtained in step S6. First, the coordinates of each pixel in the fuzzy classification pixel set are compared with those of the boundary points to determine if it is a boundary point; if it is, its classification category is retained. Second, it is determined whether there are boundary points around the fuzzy classification pixel set; if there are, they are key feature points. It is then determined whether the key feature point is within the boundary; if it is within the boundary, its category is changed to the "within the boundary" category; if it is outside the boundary, its category is changed to the "outside the boundary" category. Third, the coordinates of the fuzzy classification pixel set are aligned with the coordinates of the gradient value set, and it is determined whether the gradient value exceeds a threshold. If the value exceeds a threshold, the point is considered a boundary point, and its category is retained. The fourth step involves calculating the difference between the gradient values ​​of the fuzzy classification pixel set and the gradient values ​​of surrounding points. If any of these differences exceeds a threshold... If the point is identified as a key feature point, its category is determined according to the second step, thus achieving edge optimization based on the improved structural terrain semantic segmentation image.

2. The image optimization method for structured terrain semantic segmentation based on gradient data as described in claim 1, characterized in that, In step S1, a depth map containing depth information is acquired using a depth camera. Gradient calculation is performed on the depth map based on the grayscale differences between pixels, using the Sobel operator in both the horizontal and vertical directions. and A depth image is a two-dimensional matrix, and here we take... Represents the depth map matrix, combining the Sobel operator with the depth map. Perform convolution calculations and calculate the horizontal partial derivatives. and vertical partial derivatives : Obtain gradient data: 。 3. The image optimization method for structured terrain semantic segmentation based on gradient data as described in claim 1, characterized in that, In step S2, the horizontal partial derivative calculated in step S1 and vertical partial derivatives Stack it with a matrix of all 1s to form a three-dimensional matrix. , The magnitude of the normal vector along the third dimension is: Then Normalization is mapped to the image space, and based on color encoding, the brightness value of a pixel ranges from [value missing]. Its interval length is defined as The normalized normal vector components take values ​​within the range of... Its interval length is : Obtain a normal image based on the color space.

4. The image optimization method for structured terrain semantic segmentation based on gradient data as described in claim 1, characterized in that, In step S3, both the Normal image matrix and the RGB image matrix are three-dimensional matrices, and they are processed along the third dimension. Take a value, assuming a certain position on the matrix. exist The corresponding vector on the matrix is ,exist The corresponding vector on the matrix is to conduct Values, Will Assign a new matrix The third dimension, its first two dimensions and Matrix and With the same matrix, an improved structured terrain image is obtained. Image matrix.

5. The image optimization method for structured terrain semantic segmentation based on gradient data as described in claim 1, characterized in that, In step S4, the improved structured terrain image is annotated using image annotation software, and then the annotation file is converted into a VOC format annotation file. Training, validation, and test sets are then defined. Finally, a lightweight network is used to build a terrain detection model based on the improved structured terrain image. Dataset loading and a series of data augmentation processes are configured, and the hyperparameter learning rate is set to a range of values. Set the momentum value range to Set weight decay The model is trained for 500 to 1000 rounds from its initial state to obtain a terrain detection model based on improved structured terrain images.

6. The image optimization method for structured terrain semantic segmentation based on gradient data as described in claim 1, characterized in that, In step S5, the TensorRT framework is used to optimize and accelerate the model, converting the PyTorch model into a TensorRT model, where the accuracy is N, and... .

7. The image optimization method for structured terrain semantic segmentation based on gradient data as described in claim 1, characterized in that, In step S6, the terrain detection model infers and predicts the improved structural terrain image, and the resulting image data includes the classification label of a pixel and its corresponding confidence value. The method normalizes the probability distribution of the original confidence values. Assuming there are K classifications, it transforms the K-dimensional original vector z into a K-dimensional real vector. ,but, in, It is the base of the natural logarithm. This represents the index of an element in the original vector z, after... After processing, a probability distribution for each category is obtained. Based on the probability distribution, a thresholding process is further performed, and the category with a probability greater than or equal to 50% and less than or equal to 70% is taken as the fuzzy classification pixel set.

8. The image optimization method for structured terrain semantic segmentation based on gradient data as described in claim 1, characterized in that, In step S7, based on the Canny edge calculation algorithm and the color space-based normal data obtained in step S2, Gaussian filtering is performed, then non-maximum suppression is directly applied to refine the edges, and finally, a double threshold algorithm is used for edge connection and edge tracking to obtain the set of pixel coordinates of the boundary points. .

9. The image optimization method for structured terrain semantic segmentation based on gradient data as described in claim 1, characterized in that, In step S9, considering that the proposed method is performed on the results obtained from semantic segmentation, the pixel accuracy (pixAcc) and average intersection-union ratio (mIoU) are used for evaluation to verify the accuracy of the method.

Images