A robot grass boundary recognition and positioning method and a mowing robot

Abstract

The application relates to a robot grassland boundary recognition and positioning method, which comprises the following steps: S1, acquiring an RGB image and a depth image of the grassland through an RGB-D camera installed on the robot, calibrating the internal parameters and external parameters of the camera; S2, performing semantic segmentation on the RGB image to obtain a grassland area and a non-grassland area, and searching a boundary contour to obtain a grassland boundary point set 1; S3, performing HSV threshold segmentation on the RGB image, determining grassland class HSV upper and lower thresholds, and searching a boundary contour to obtain a grassland boundary point set 2; S4, performing boundary fusion and filtering on the grassland boundary point set 1 and the grassland boundary point set 2 to obtain an effective grassland boundary point set; and S5, performing coordinate transformation on the grassland boundary points based on the depth image and the internal parameters and external parameters of the camera to obtain relative coordinates of the effective grassland boundary point set. The above technical scheme can accurately extract and position the grassland boundary in the image, and has wide adaptability to different grassland types and light conditions.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of lawn mowers, in particular to a robot lawn boundary recognition and positioning method and a lawn mowing robot. BACKGROUND

[0002] The existing lawn mowing robots need to obtain a target lawn boundary global map before mowing. The target lawn boundary global map is mainly obtained by manually pushing the lawn mower along the lawn boundary to establish a map. Or an electronic fence is used, which needs to be manually buried and is complicated to operate, has poor adaptability, and the wire is prone to aging, requiring regular inspection and maintenance, and has high cost.

[0003] In recent years, technical solutions based on visual sensors have gradually appeared to obtain the boundary of the lawn. By using computer vision technology to recognize the boundary of the lawn, combined with the positioning data of the lawn mower, the boundary global map of the target lawn can be easily established. The difficulty of this problem mainly lies in the diversification of the lawn scene and the accuracy of the lawn boundary recognition. The current method of realizing lawn boundary recognition based on vision has the following shortcomings:

[0004] 1. The method of segmenting by predefining the color features of the lawn has poor adaptability and is sensitive to light conditions, and cannot be applied to different types of lawn environments with large seasonal changes;

[0005] 2. The method of recognizing the lawn by the texture features of the lawn has a large amount of calculation and requires high hardware. And the reference texture features of the lawn need to be extracted and stored in advance, and have poor generalization;

[0006] 3. The semantic segmentation method based on deep learning also has the problems of relying on training data and inaccurate boundary segmentation, which affects the boundary positioning accuracy. SUMMARY

[0007] In view of the shortcomings of the prior art, the present application provides a robot lawn boundary recognition and positioning method based on visual technology, which can accurately extract and position the boundary of the lawn in the image, and has wide adaptability to different types of lawns, light conditions, etc.

[0008] The robot lawn boundary recognition and positioning method of the present application comprises the following steps:

[0009] S1. Obtain the RGB image and depth image of the lawn through the RGB-D camera installed on the robot, and calibrate the internal and external parameters of the camera;

[0010] S2. Perform semantic segmentation on the RGB image to obtain the grassland area and non-grassland area, and search for the boundary contour to obtain the grassland boundary point set 1;

[0011] S3, performing HSV threshold segmentation on the RGB image, determining the HSV upper and lower thresholds of the grassland class, and searching the boundary contour to obtain a grassland boundary point set 2;

[0012] S4, performing boundary fusion and filtering on the grassland boundary point set 1 and the grassland boundary point set 2 to obtain an effective grassland boundary point set;

[0013] S5, obtaining the depth of the boundary point pixel based on the depth image, converting the effective grassland boundary point set from the image coordinate system to the camera coordinate system using the pinhole camera model, and obtaining the relative coordinates of the effective grassland boundary point set according to the intrinsic and extrinsic parameters of the camera.

[0014] Further, the step S2 comprises the following steps:

[0015] S2-1, constructing a semantic segmentation network model;

[0016] S2-2, selecting only images containing the grassland class and corresponding semantic annotations from the public semantic segmentation dataset to initialize the semantic segmentation network model, and obtaining the parameters of the semantic segmentation network model;

[0017] S2-3, loading the obtained parameters of the semantic segmentation network model into the semantic segmentation network model, obtaining a number of actual grassland scene boundary images using an RGB-D camera and self-semantic annotation, simultaneously inputting the number of actual grassland scene boundary images and the corresponding semantic annotations into the semantic segmentation network model for training, and fine-tuning the semantic segmentation network model;

[0018] S2-4, inputting the RGB image into the trained semantic segmentation neural network model, segmenting the grassland region and the non-grassland region in the RGB image, and then using a function to search the boundary contour in the RGB image and filter out the contours with too small length to obtain the grassland boundary point set 1.

[0019] Further, the semantic segmentation network model in step S2-1 is Bisenet-v2.

[0020] Further, the public semantic segmentation dataset in step S2-2 includes ADE20K and TAS500.

[0021] Further, the function in step S2-4 is the findcontour() function in the opencv library.

[0022] Further, the step S3 comprises the following steps:

[0023] S3-1, grid sampling and statistics of the RGB image, dividing it into a plurality of small grids, and calculating the mean and standard deviation of the H, S and V components of each small grid;

[0024] S3-2, each small grid is clustered using the K-means++ algorithm, and the center, standard deviation average, and class point number of each class are calculated;

[0025] S3-3, convert the RGB image of the grass region and the non-grass region segmented in step 2 to the HSV color space, and calculate the average value of the H, S, and V components of the grass region as the HSV prior reference value of the grassland;

[0026] S3-4, compare the nearest distance of each class to the grassland HSV prior reference value, select the minimum cluster number K with the smallest nearest distance as the best cluster number, select the class closest to the grassland HSV prior reference value under the best cluster number as the recognized grassland class, and save the HSV statistical mean h mean , s mean , v mean and standard deviation h std , s std , v std of the recognized grassland class;

[0027] S3-5, calculate the grassland class threshold value:

[0028] Let the lower threshold [ H lower , S lower , V lower ] and the upper threshold [ H upper , S upper , V upper ];

[0029] Then the upper and lower thresholds of H, S, and V are:

[0030] ;

[0031] ;

[0032] ;

[0033] ;

[0034] ;

[0035] ;

[0036] where k1 and k2 are proportional coefficients, the standard deviation is set to be within a certain range, s std is not greater than 40, since the color of the grassland is mainly green and yellow, the additional restrictions H lower is not less than 10, S lower is not less than 10, since the brightness component is independent in the HSV color space, a certain range of brightness V is ensured, which has strong adaptability to different lighting conditions, and the additional restriction V upperNo less than 150;

[0037] S3-6, according to the obtained grassland class H, S, V upper and lower threshold value segmentation RGB image, morphological inflation, close operation, morphological kernel selection square, eliminate hole noise, reduce connected region, statistics and remove small connected domain, then use the function to search the boundary contour in the RGB image, obtain the grassland boundary point set 2.

[0038] Further, the function in the step S3-6 is findcontour() function in opencv library.

[0039] Further, the step S4 comprises the following steps: if the number of contours of the grassland boundary point set 2 is greater than 10, it is judged that the HSV dynamic threshold segmentation is invalid, and the grassland boundary point set 1 is selected as the effective grassland boundary point; otherwise, the grassland boundary point set 1 is traversed, for each point p1 on the contour, it is searched whether the 20 pixel distance neighborhood contains the boundary point in the grassland boundary point set 2, if yes, the point p2 in the grassland boundary point set 2 which is the nearest point to p1 in the Manhattan pixel distance is replaced as the effective grassland boundary point, otherwise, the point p1 is reserved as the effective grassland boundary point, and the effective grassland boundary point set is obtained.

[0040] Further, it further comprises a step S6, and the step S6 comprises the following steps: according to the GPS, positioning data is acquired, the effective grassland boundary point set is converted from the camera coordinate system to the world coordinate system in combination with the camera pose, the too high and too far boundary points in the world coordinate system are filtered out, and the grassland boundary global map is established.

[0041] A mowing robot, which applies any one of the robot grassland boundary recognition and positioning methods.

[0042] Compared with the prior art, the beneficial effects of the present application are as follows: the HSV threshold segmentation and semantic segmentation are used to obtain the grassland boundary point set respectively, then the boundary fusion and filtering are used to optimize the boundary points recognized by the semantic segmentation, so that the extracted grassland boundary is more accurate and smooth, some non-grassland class contours affected by light shadow and color features are eliminated, the image frame pixels are removed, the final effective grassland boundary point set is obtained, the generalization is stronger for different types, different seasons and different light conditions of the grassland environment, then according to the obtained depth image, the camera internal parameter and the external parameter, the coordinate transformation is carried out on the effective grassland boundary point set, the final effective grassland boundary point coordinate is obtained, and the boundary recognition precision is improved. BRIEF DESCRIPTION OF DRAWINGS

[0043] The drawings described herein are used to provide further understanding of the present application, and in the drawings:

[0044] Figure 1 The flow chart of the present application;

[0045] Figure 2RGB image of the grassland for the embodiment;

[0046] Figure 3 For Figure 2 Image after semantic segmentation;

[0047] Figure 4 For Figure 3 Contour extraction image after boundary contour search;

[0048] Figure 5 For Figure 2 Image after grid sampling statistics;

[0049] Figure 6 For Figure 5 Image after grid clustering;

[0050] Figure 7 For Figure 6 Image of grassland HSV upper and lower threshold segmentation;

[0051] Figure 8 For Figure 7 Image after morphological processing;

[0052] Figure 9 For Figure 8 Image after small connected domain elimination;

[0053] Figure 10 For Figure 9 Contour extraction image after boundary contour search;

[0054] Figure 11 For Figure 4 And Figure 10 Image after boundary fusion and filtering;

[0055] Figure 12 For Figure 11 Grassland boundary point coordinate transformation result overhead view.

[0056] Referring to Figures 1-12 One of the embodiments, a robot grassland boundary recognition and positioning method, comprising the following steps:

[0057] S1, through the RGB-D camera installed on the robot, the RGB image, the depth image of the grassland, the calibration of the camera's internal and external parameters are obtained;

[0058] S2, the RGB image is subjected to semantic segmentation, the grassland area and the non-grassland area are obtained, and the boundary contour is searched to obtain the grassland boundary point set 1;

[0059] S3, the RGB image is subjected to HSV threshold segmentation, the grassland class HSV upper and lower threshold values are determined, and the boundary contour is searched to obtain the grassland boundary point set 2;

[0060] S4, boundary fusion and filtering of the meadow boundary point set 1 and the meadow boundary point set 2 to obtain an effective meadow boundary point set;

[0061] S5, obtaining the depth of the boundary point pixel pair based on the depth image, converting the effective meadow boundary point set from the image coordinate system to the camera coordinate system using the pinhole camera model, and obtaining the relative coordinates of the effective meadow boundary point set according to the intrinsic and extrinsic parameters of the camera.

[0062] Further, the step S2 comprises the following steps:

[0063] S2-1, constructing a semantic segmentation network model, which can adopt a network model with better real-time performance such as Bisenet-v2;

[0064] S2-2, filtering out only images containing the grass category and corresponding semantic annotations from the public semantic segmentation dataset (such as ADE20K, TAS500) to initialize the semantic segmentation network model, and obtaining the parameters of the semantic segmentation network model;

[0065] S2-3, loading the obtained parameters of the semantic segmentation network model into the semantic segmentation network model, obtaining a number of actual meadow scene (such as a court, a park, a courtyard) boundary images using an RGB-D camera and self-semantic annotation, simultaneously sending the number of actual meadow scene boundary images and corresponding semantic annotations into the semantic segmentation network model for training, and fine-tuning the semantic segmentation network model;

[0066] S2-4, inputting the RGB image into the trained semantic segmentation neural network model, segmenting the grass region and the non-grass region in the RGB image, and then using the findcontour() function in the opencv library to search for the boundary contour in the RGB image, filter out the contours with too small length, and obtain the meadow boundary point set 1.

[0067] Further, the step S3 comprises the following steps:

[0068] S3-1, grid sampling statistics of the RGB image, dividing the image into square small grids, taking the image size of 640*480 as an example, setting the grid side length to 20 pixels, and the length is too small, which will increase the calculation cost, and too large will result in inaccurate statistical results. The average and standard deviation of the H, S, and V components of each small grid are calculated.

[0069] S3-2, clustering each small grid using the K-means++ algorithm, taking the cluster number K as 2, 3, and 4 respectively, and calculating the center, standard deviation average, and class point number of each class.

[0070] S3-3, convert the RGB image of the grass region and the non-grass region segmented in step 2 to HSV color space, and count the average values of the H, S, and V components of the grass region as the HSV prior reference value of the grassland;

[0071] S3-4, compare the closest_dis of each class to the grassland HSV prior reference value, select the cluster number K with the smallest closest_dis as the optimal cluster number, select the class closest to the grassland HSV prior reference value under the optimal cluster number as the recognized grassland class, and save the HSV statistical mean h mean 、s mean 、v mean and standard deviation h std 、s std 、v std of the recognized grassland class;

[0072] S3-5, calculate the grassland class threshold:

[0073] Let the lower threshold be [ H lower , S lower , V lower ] and the upper threshold be [ H upper , S upper , V upper ];

[0074] Then the upper and lower thresholds of H, S, and V are:

[0075] ;

[0076] ;

[0077] ;

[0078] ;

[0079] ;

[0080] ;

[0081] where k1 and k2 are proportional coefficients, the standard deviation is set to be within the range of s std not greater than 40, since the color of the grassland is mainly green and yellow, and the additional restrictions are H lower not less than 10, S lower not less than 10, since the brightness component is independent in the HSV color space, and a certain brightness V range is ensured, which is suitable for different lighting conditions, and the additional restrictions are V upper not less than 150;

[0082] S3-6, according to the obtained grassland class H, S, V upper and lower threshold value segmentation of RGB image, morphological dilation, close operation, morphological kernel selection square, eliminate hole noise, reduce connected region, statistics and remove small connected domain, then use the findcontour() function in the opencv library to search the boundary contour in the RGB image, and obtain the grassland boundary point set 2.

[0083] Further, the step S4 includes the following steps: if the number of contours of the grassland boundary point set 2 is greater than 10, it is judged that the HSV dynamic threshold segmentation is invalid, and the grassland boundary point set 1 is selected as the effective grassland boundary point; otherwise, the grassland boundary point set 1 is traversed, and for each point p1 on the contour, it is searched whether the 20-pixel-distance neighborhood contains the boundary point in the grassland boundary point set 2, if yes, the point p2 in the grassland boundary point set 2 which is the nearest point to p1 in the Manhattan pixel distance is replaced as the effective grassland boundary point, otherwise, the point p1 is kept as the effective grassland boundary point, and the effective grassland boundary point set is obtained. In this way, the boundary points of semantic segmentation recognition can be optimized, the result is more accurate, the smooth boundary contour is obtained, and some non-grassland class contours affected by light shadow and color characteristics are eliminated. The image border pixels are removed, and the final effective grassland boundary point is obtained.

[0084] Further, it further includes a step S6, which includes obtaining positioning data according to GPS, combining the camera pose, converting the effective grassland boundary point set from the camera coordinate system to the world coordinate system, filtering out the too-high and too-far boundary points in the world coordinate system, and establishing a grassland boundary global map.

[0085] A mowing robot, which applies any one of the above robot grassland boundary recognition and positioning methods.

[0086] Although the embodiments of the present application have been shown and described, it can be understood by those skilled in the art that various changes, modifications, replacements and variations can be made to the embodiments without departing from the principles and spirits of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A method for identifying and locating grass boundaries using a robot, characterized in that: It includes the following steps: S1. Acquire RGB and depth images of the grassland using the RGB-D camera installed on the robot, and calibrate the camera's intrinsic and extrinsic parameters; S2. Perform semantic segmentation on the RGB image to obtain grassland and non-grassland regions, and search for boundary contours to obtain grassland boundary point set 1; S3. Perform HSV thresholding on the RGB image to determine the upper and lower HSV thresholds for the grassland class, and search for the boundary contour to obtain the grassland boundary point set 2; Step S3 includes the following steps: S3-1. Sampling statistics of the RGB image grid are divided into several small grids, and the mean and standard deviation of the H, S, and V components of each small grid are calculated. S3-2. Cluster each small cell using the K-means++ algorithm, and statistically calculate the center, standard deviation, mean, and number of points in each class. S3-3. Convert the RGB image of the grass and non-grass regions segmented in step 2 to the HSV color space, and calculate the average value of the H, S, and V components of the grass region as the HSV prior reference value of the grass. S3-4. Compare the nearest distances to the prior HSV reference values ​​of the grassland for each category, select the cluster number K with the smallest nearest distance as the optimal number of clusters, select the class with the optimal number of clusters that is closest to the prior HSV reference value of the grassland as the grassland class to be identified, and save the HSV statistical mean h of the identified grassland class. mean s mean v mean and standard deviation h std s std v std ; S3-5, Calculate the threshold for grassland types: Let the lower bound of the threshold be [H] lower S lower V lower ], upper bound [ H upper S upper V upper ]; The upper and lower thresholds for H, S, and V are: ； ； ； ； ； ； Where k1 and k2 are proportionality coefficients, the standard deviation is set with a range limit, and s std No more than 40, with additional restrictions H due to the predominantly green and yellow color of the grassland. lower Not less than 10, S lower Not less than 10. Because the luminance components in the HSV color space are independent, a certain luminance V range is guaranteed, which makes it highly adaptable to different lighting conditions. Additional limitation V. upper No less than 150; S3-6. Based on the obtained upper and lower thresholds of grassland class H, S, and V, the RGB image is segmented, and morphological dilation and closing operations are performed. A square morphological kernel is selected to eliminate hole noise and reduce connected regions. Small connected regions are counted and removed. Then, the boundary contour in the RGB image is searched using a function to obtain the grassland boundary point set 2. S4. Perform boundary fusion and filtering on grassland boundary point set 1 and grassland boundary point set 2 to obtain an effective grassland boundary point set; S5. Obtain the depth corresponding to the boundary point pixels based on the depth image, and use the pinhole camera model to transform the effective grassland boundary point set from the image coordinate system to the camera coordinate system. Based on the camera's intrinsic and extrinsic parameters, obtain the relative coordinates of the effective grassland boundary point set.

2. The method for robot grassland boundary recognition and localization according to claim 1, characterized in that: Step S2 includes the following steps: S2-1. Construct a semantic segmentation network model; S2-2. Select images containing only the grass category and their corresponding semantic annotations from the publicly available semantic segmentation dataset to initialize the semantic segmentation network model and obtain the parameters of the semantic segmentation network model. S2-3. Load the obtained parameters of the semantic segmentation network model into the semantic segmentation network model. Use an RGB-D camera to acquire several images of the boundary of the actual grassland scene and perform semantic annotation. Simultaneously feed the images of the boundary of the actual grassland scene and the corresponding semantic annotations into the semantic segmentation network model for training and fine-tuning the semantic segmentation network model. S2-4. Input the RGB image into the trained semantic segmentation neural network model to segment the grassland region and non-grass region in the RGB image. Then, use a function to search for boundary contours in the RGB image, filter out contours that are too short, and obtain the grassland boundary point set 1.

3. The method for robot grassland boundary recognition and localization according to claim 2, characterized in that: The semantic segmentation network model in step S2-1 is Bisenet-v2.

4. The method for robot grass boundary recognition and localization according to claim 2, characterized in that: The semantic segmentation datasets disclosed in step S2-2 include ADE20K and TAS500.

5. The method for robot grassland boundary recognition and localization according to claim 2, characterized in that: The function in step S2-4 is the findcontour() function from the OpenCV library.

6. The method for robot grass boundary recognition and localization according to claim 1, characterized in that: The function in steps S3-6 is the findcontour() function from the OpenCV library.

7. The method for robot grassland boundary recognition and localization according to claim 1, characterized in that: Step S4 includes the following steps: If the number of contours in the grassland boundary point set 2 is greater than 10, then the HSV dynamic threshold segmentation is determined to be invalid, and the grassland boundary point set 1 is selected as the effective grassland boundary point; otherwise, the grassland boundary point set 1 is traversed, and for each point p1 on the contour, a 20-pixel distance neighborhood is searched to see if it contains a boundary point in the grassland boundary point set 2. If it does, it is replaced with the point p2 in the grassland boundary point set 2 that has the closest Manhattan pixel distance to p1 as the effective grassland boundary point; otherwise, the point p1 is retained as the effective grassland boundary point, and the effective grassland boundary point set is obtained.

8. The method for robot grass boundary recognition and localization according to claim 1, characterized in that: It also includes step S6, which involves acquiring positioning data based on GPS, combining it with camera pose, transforming the effective grassland boundary point set from the camera coordinate system to the world coordinate system, filtering out boundary points that are too high or too far in the world coordinate system, and establishing a global map of the grassland boundary.

9. A lawnmower robot, characterized in that: The method for identifying and locating a robot grassland boundary is described in any one of claims 1-8.

Images