A Method for Discriminating Inside and Outside of Rows of Trellis Orchard Robots

Abstract

The present invention is a method for distinguishing inside and outside rows of a trellis orchard robot. The method establishes a judgment mechanism for switching navigation algorithms inside and outside rows of an orchard. If the GNSS signal outside the trellis orchard row is good, a GNSS navigation algorithm is used. If the GNSS signal inside the trellis orchard row is weak, a navigation algorithm based on vision or laser radar is used. When the robot moves from outside the trellis orchard to inside the trellis orchard, a corresponding switch is performed to improve the recognition rate of inside and outside rows in a structured orchard scene. The present invention uses semantic segmentation in deep learning to perform only two classifications, and at the same time uses front and rear RGB cameras to detect whether the row boundary has been reached. The category definition is clear, the explanatory power is high, and the accuracy rate is high. It can be distinguished in different trellis orchards under different light intensities, and it is more universal and robust. It can stably and reliably segment rows in trellis orchards, provide information inside and outside rows of trellis orchards, and provide a good switching mechanism for trellis navigation algorithms.

Description

Technical Field

[0001] The present invention relates to the technical field of fruit transportation robot navigation, and particularly to a method for discriminating inside and outside rows of a pergola orchard robot. Background Art

[0002] The prior art for identifying inside and outside rows of orchards mainly focuses on open orchard and farmland scenarios. In open environments with good GNSS signals, most use real-time GNSS information for positioning and navigation to make control decisions. At the same time, the similarity of open scenarios is small, and the increase or decrease in the area of the orchard region or the number of fruit trees can be identified by using a camera to determine the end of the row.

[0003] However, the above methods are difficult to apply in structured pergola orchards and are subject to many limitations. Because the GNSS signal is weak, and the layout of pergola orchards is relatively regular with many similar scenarios, it is impossible to achieve good identification and classification. Existing pergola identification algorithms mainly focus on segmenting and identifying roads and fruit trees. However, due to the influence of road weeds and uneven distribution of fruit tree planting, the recognition rate of inside and outside rows is relatively low. Summary of the Invention

[0004] The purpose of the present invention is to overcome the problems existing in the prior art and provide a method for discriminating inside and outside rows of a pergola orchard robot, establishing a judgment mechanism for switching the navigation algorithm inside and outside the orchard rows. For example, when the GNSS signal is good outside the pergola orchard row, the GNSS-based navigation algorithm is used; when the GNSS signal is weak inside the pergola orchard row, the vision or lidar-based navigation algorithm is used. When the robot moves from outside the pergola orchard to inside the pergola orchard, the corresponding switch is made to improve the recognition rate of inside and outside rows in the structured orchard scenario.

[0005] To achieve the above technical purposes and obtain the above technical effects, the present invention is realized through the following technical solutions:

[0006] A method for discriminating inside and outside rows of a pergola orchard robot, the method comprising the following steps:

[0007] Step 1) Install an RGB camera in front of and behind the robot chassis vehicle, place it in the center of the pergola orchard row, and obtain a frame of real-time RGB image from the front and rear RGB cameras;

[0008] Step 2) Perform real-time image slicing on the front and rear frame RGB images based on the ROI region;

[0009] Step 3) Send the sliced ROI region image into the trained UNet model, use the trained weights for semantic segmentation, and the output is the mask map of the current frame;

[0010] Step 4) Calculate the area S of the pixel values of the blank region segmented in the current mask map;

[0011] Step 5) Set the prior threshold K. The threshold K is the applicable value for each row in the pergola orchard obtained through multiple experiments. When the pixel value area S of the blank area >= the threshold K, it is defined as outside the pergola orchard row, i.e., class = outline. Conversely, when the pixel value area S of the blank area < the threshold K, it is defined as inside the pergola orchard row, i.e., class = inline;

[0012] Step 6) If it is determined to be inside the pergola orchard row, directly output the information inline inside the pergola orchard row. If it is determined to be outside the pergola orchard row, start accumulating the count i. When the accumulated number of consecutive frames i >= 5 times, output the information outline outside the pergola orchard row. If the number of consecutive frames < 5 times, output the information inline inside the pergola orchard row to eliminate the error value;

[0013] Step 7) Compare the output information of the front and rear RGB cameras. If the front RGB camera outputs outline information and the rear RGB camera outputs outline information, it is determined to be outside the pergola orchard row. If the front RGB camera outputs inline information and the rear RGB camera outputs inline information, it is determined to be inside the pergola orchard row. If the front RGB camera outputs inline information and the rear RGB camera outputs outline information, or the front RGB camera outputs outline information and the rear RGB camera outputs inline information, it is determined to be at the boundary of the pergola orchard row, and the navigation algorithm inside and outside the pergola orchard row makes corresponding switches;

[0014] Step 8) Repeat the above process starting from Step 1).

[0015] Further, in the above Step 3), the training process of the UNet model is as follows:

[0016] Step 3.1) Make an image classification sample set using the method of taking frames at intervals;

[0017] Step 3.2) Slice and crop, set the ROI area, and only focus on the sky area of the current row in the travel detection method;

[0018] Step 3.3) Sample annotation. The sample data set is divided into two categories, namely the sky and the background. The foreground is the segmented blank sky part, and the background is the rest except the sky;

[0019] Step 3.4) Data augmentation. By performing one or more combinations of operations such as flipping the image, randomly splicing, adding, and brightness adjustment, increase the data diversity and improve the generalization ability of the model;

[0020] Step 3.5) Use the UNet segmentation network for classification and segmentation to obtain the corresponding mask map and the optimal model weights.

[0021] Further, in the said step 1), the positions of the front and rear RGB cameras are on the central axis of the robot chassis vehicle, and the RGB cameras are installed on the corresponding camera fixing plates. The camera fixing plates are installed on the frame of the robot chassis vehicle through damping hinges, and the angles θ and the field of view angles α of the RGB cameras are adjusted through the damping hinges.

[0022] Further, in the said step 2), use the front RGB camera and the rear RGB camera to respectively obtain a frame of color image in the due front and due rear of the current fruit row, and perform image slicing based on the ROI region on both of these two frames of images. The resolution of the sliced ROI region is 320x320.

[0023] Further, in the said step 5), the threshold K is preferably 30000.

[0024] The beneficial effects of the present invention are as follows:

[0025] 1. By means of semantic segmentation in deep learning, the present invention only performs binary classification, and at the same time detects whether the row boundary is reached through the front and rear RGB cameras. The category definition is clear, the interpretability is high, and the accuracy is high. It can be discriminated in different light intensities and different trellis orchards, and has greater universality and robustness.

[0026] 2. It can be stably and reliably segmented between the rows of the trellis orchard, provide information inside and outside the rows of the trellis orchard, and can provide a good switching mechanism for the navigation algorithm of the trellis type. Description of the Drawings

[0027] Figure 1 It is the flow chart of the method for discriminating inside and outside the rows of the trellis orchard robot of the present invention;

[0028] Figure 2 It is the schematic diagram of the positions of inside the row, outside the row, and the row boundary of the present invention;

[0029] Figure 3 It is the flow chart of the training of the UNet model of the present invention;

[0030] Figure 4 It is the schematic diagram of the installation positions of the front and rear RGB cameras of the present invention;

[0031] Figure 5 It is the schematic diagram of the real-time image slicing of the front and rear frame RGB images based on the ROI region of the present invention;

[0032] Figure 6 It is the process of the present invention to output the mask map after sending the sliced ROI region image into the UNet model;

[0033] Figure 7 Schematic diagram for the present invention to judge inside and outside the rows of a pergola orchard using the threshold K: (a) is inside the row, and (b) is outside the row;

[0034] Figure 8 Sample tagging diagram of the present invention: (a) is the original image, (b) is the tagging image, and (c) is the tagging visualization image;

[0035] Figure 9 Schematic diagram for the present invention to perform data augmentation operation on an image: (a) is the original image, (b) is flipping, (c) is salt-and-pepper noise, (d) is Gaussian noise + salt-and-pepper noise, and (e) is salt-and-pepper noise + brightness adjustment. Detailed implementation manners

[0036] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0037] As Figure 1 shown, a method for discriminating inside and outside the rows of a pergola orchard robot includes the following steps:

[0038] Step 1) Install an RGB camera in front of and behind the robot chassis vehicle, place it in the center of the rows of the pergola orchard, and obtain a frame of real-time RGB image from the front and rear RGB cameras;

[0039] Step 2) Perform real-time image slicing on the front and rear frames of RGB images based on the ROI region, as Figure 5 shown;

[0040] Step 3) Feed the sliced ROI region image into the trained UNet model, as Figure 6 shown, use the trained weights for semantic segmentation, and the output is the mask image of the current frame;

[0041] Step 4) Calculate the area S of the pixel values of the blank region segmented in the current mask image;

[0042] Step 5) Set a prior threshold K. The threshold K is the applicable value for each row in the pergola orchard obtained through multiple experiments. As Figure 7 shown, when the area S of the pixel values of the blank region >= the threshold K, as Figure 7 shown in (b), in this embodiment, the threshold K is 30000, Figure 7 the area S of the pixel values of the blank region in (b) is 33400, which is greater than 30000, and it is defined as outside the rows of the pergola orchard, that is, class = outline. On the contrary, when the area S of the pixel values of the blank region < the threshold K, as Figure 7 shown in (a), Figure 7 the area S of the pixel values of the blank region in (a) is 18590, which is less than 30000, and it is defined as inside the rows of the pergola orchard, that is, class = inline;

[0043] Step 6): If it is determined that it is within the pergola orchard row, directly output the information inline for within the pergola orchard row. If it is determined that it is outside the pergola orchard row, start accumulating the count i. When the accumulated number of consecutive frames i >= 5 times, output the information outline for outside the pergola orchard row. If the number of consecutive frames is < 5 times, output the information inline for within the pergola orchard row, thus eliminating the error value;

[0044] Step 7): Compare the output information of the front and rear RGB cameras. As Figure 2 and Figure 4 shown, if the chassis robot is in position 1 or position 5, and the front RGB camera (i.e., the No. 1 RGB camera in the figure) outputs the outline information and the rear RGB camera (i.e., the No. 2 RGB camera in the figure) outputs the outline information, it is determined to be outside the pergola orchard row; if the chassis robot is in position 3, and the front RGB camera outputs the inline information and the rear RGB camera outputs the inline information, it is determined to be within the pergola orchard row; if the chassis robot is in position 2 or position 4, and the front RGB camera outputs the inline information and the rear RGB camera outputs the outline information or the front RGB camera outputs the outline information and the rear RGB camera outputs the inline information, it is determined to be at the boundary of the pergola orchard row, and the navigation algorithm inside and outside the pergola orchard row makes corresponding switches;

[0045] Step 8): Repeat the above process starting from Step 1).

[0046] In the said Step 3), as Figure 3 shown, the training process of the UNet model is as follows:

[0047] Step 3.1): Make an image classification sample set using the method of taking frames at intervals;

[0048] Step 3.2): Slice and crop, set the ROI area, and in the travel detection method, only need to focus on the sky area of the current row;

[0049] Step 3.3): Sample annotation. The sample data set is divided into two categories, namely sky and background. The foreground is the segmented blank sky part, and the background is the rest except the sky, as Figure 8 shown;

[0050] Step 3.4): Data augmentation. By performing one or more combinations of operations such as flipping the image, randomly splicing, adding, and brightness adjustment on the image, increase the data diversity and improve the generalization ability of the model. As Figure 9 shown, in this embodiment, the image is respectively flipped, salt and pepper noise is added, Gaussian noise + salt and pepper noise, and salt and pepper noise + brightness adjustment operations are performed to increase the data diversity;

[0051] Step 3.5) Use the UNet segmentation network for classification and segmentation to obtain the corresponding mask map and the optimal model weights.

[0052] In the said step 1), as Figure 4 shown, the positions of the front and rear RGB cameras are on the central axis of the robot chassis vehicle, and the RGB cameras are installed on the corresponding camera fixing plates. The camera fixing plates are installed on the frame of the robot chassis vehicle through damping hinges. The angle θ of the RGB camera is adjusted through the damping hinges. In this embodiment, the angle of the RGB camera is 25°, and the field of view angle is α.

[0053] In the said step 2), use the front RGB camera and the rear RGB camera to respectively obtain a frame of color image in the direct front and direct rear of the current fruit row, and perform image slicing based on the ROI region on both of these two frames of images. The resolution of the sliced ROI region is 320x320.

[0054] In the said step 5), the threshold K is preferably 30000.

[0055] The above are only the preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for discriminating inside and outside of rows of pergola orchard robots, characterized in that, The method includes the following steps: Step 1) Install an RGB camera in front of and behind the robot chassis vehicle, place it in the center of the pergola orchard row, and obtain a frame of real-time RGB image from the front and rear RGB cameras; Step 2) Perform real-time image slicing on the front and rear frame RGB images based on the ROI region; Step 3) Send the sliced ROI region image into the trained UNet model, perform semantic segmentation using the trained weights, and the output is the mask map of the current frame; Step 4) Calculate the area S of the pixel values of the blank region segmented in the current mask map; Step 5) Set a prior threshold K. The threshold K is the applicable value for each row in the pergola orchard obtained through multiple experiments. When the area S of the pixel values of the blank region >= threshold K, it is defined as outside the pergola orchard row, that is, class = outline. On the contrary, when the area S of the pixel values of the blank region < threshold K, it is defined as inside the pergola orchard row, that is, class = inline; Step 6) If it is determined to be inside the pergola orchard row, directly output the information inline inside the pergola orchard row. If it is determined to be outside the pergola orchard row, start accumulating the count i. When the accumulated number of consecutive frames i >= 5 times, output the information outline outside the pergola orchard row. If the number of consecutive frames < 5 times, output the information inline inside the pergola orchard row to exclude error values; Step 7) Compare the output information of the front and rear RGB cameras. If the front RGB camera outputs outline information and the rear RGB camera outputs outline information, it is determined to be outside the pergola orchard row. If the front RGB camera outputs inline information and the rear RGB camera outputs inline information, it is determined to be inside the pergola orchard row. If the front RGB camera outputs inline information and the rear RGB camera outputs outline information, or the front RGB camera outputs outline information and the rear RGB camera outputs inline information, it is determined to be at the boundary of the pergola orchard row, and the in-row and out-of-row navigation algorithms of the pergola orchard are switched accordingly; Step 8) Repeat the above process starting from Step 1).

2. The method for discriminating inside and outside the rows of the pergola orchard robot according to claim 1, wherein In the above Step 3), the training process of the UNet model is as follows: Step 3.1) Make an image classification sample set using the method of taking frames at intervals; Step 3.2) Slice and crop, set the ROI region, and only need to focus on the sky region of the current row in the travel detection method; Step 3.3) Sample annotation. The sample data set is divided into two categories, namely sky and background. The foreground is the segmented blank sky part, and the background is the rest except the sky; Step 3.4) Data augmentation. Increase data diversity and improve the generalization ability of the model by performing one or more combinations of operations such as flipping the image, randomly splicing, adding, and brightness adjustment; Step 3.5) Use the UNet segmentation network for classification and segmentation to obtain the corresponding mask map and the optimal model weights.

3. The method for discriminating inside and outside the rows of a pergola orchard robot according to claim 1, characterized in that, In the step 1), the positions of the front and rear RGB cameras are on the central axis of the robot chassis vehicle, and the RGB cameras are installed on the corresponding camera fixing plates. The camera fixing plates are installed on the frame of the robot chassis vehicle through damping hinges, and the angles θ and the field of view angles α of the RGB cameras are adjusted through the damping hinges.

4. The method for discriminating inside and outside of the rows of the pergola orchard robot according to claim 3, wherein In the step 2), a frame of color image in the due front and due rear of the current fruit row is respectively obtained by using the front RGB camera and the rear RGB camera, and image slicing based on the ROI region is performed on both of these two frames of images. The resolution of the sliced ROI region is 320x320.

5. The method for discriminating inside and outside the rows of a pergola orchard robot according to claim 1, characterized in that, In the step 5), the threshold K is preferably 30000.

Images