Method for autonomous navigation between rows of fruit trees in vineyard

Abstract

The invention provides a method for autonomous navigation between rows of fruit trees in a vineyard, and the method mainly comprises two modules: a motion control module based on barrier information on two sides, and a structured orchard row tail recognition algorithm module based on a local image sequence. During working, the mobile robot firstly realizes inter-row walking through the motion control module based on obstacle information on two sides, and then realizes row tail identification through the structured orchard row tail identification algorithm module based on the local image sequence, so that the robot stops moving in time. According to the invention, the mobile robot can safely and autonomously walk between fruit tree rows, and the recognition rate of the row tail in a structured orchard scene is improved.

Description

Technical Field

[0001] The present invention relates to the field of smart agricultural technology, and in particular to a method for autonomous navigation between fruit tree rows in a vineyard. Background Art

[0002] In the vineyard scenario, the autonomous walking of mobile robots between rows of fruit trees is generally divided into two stages: one is to generate motion control strategies based on environmental information, and the other is to identify the end of the row of fruit trees and stop working in time. In terms of motion control between rows of fruit trees, the existing technology extracts navigation lines through visual semantic segmentation, image morphology and other schemes to guide the robot's movement. However, in the vineyard scenario, the front and back backgrounds are similar, and the road surface morphology is complex and different. Traditional schemes cannot effectively control the movement of robots between rows. In terms of row end recognition, the existing technology for orchard row end recognition is more focused on open orchards and farmland scenes. The open environment has good GPS signals, and most of them use real-time GPS information for positioning and navigation to make control decisions. At the same time, the similarity of open scenes is relatively small, and cameras can be used to identify the increase or decrease in the area of ​​the orchard area or the number of fruit trees to identify the end of the row. However, it is difficult to apply it to structured trellis orchards, and there are many restrictions because the GPS signal is weak. At the same time, the trellis orchard is arranged in a regular manner, and there are many similar scenes, which makes it difficult to achieve good recognition and classification. The existing trellis recognition algorithm is more focused on the road surface and fruit tree segmentation and recognition. However, due to the influence of weeds on the road surface and the uneven distribution of fruit tree planting, the recognition rate at the end of the row is relatively low.

[0003] Therefore, how to provide a method for autonomous navigation between rows of fruit trees in a vineyard to solve the problems existing in the existing technology is of great significance for its application. Summary of the invention

[0004] In view of this, the purpose of this application is to provide a method for autonomous navigation between rows of fruit trees in a vineyard to solve the problem.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] A method for autonomous navigation between rows of fruit trees in a vineyard comprises the following steps:

[0007] S3.1: Acquire a frame of image in real time through the camera;

[0008] S3.2: Determine whether the end of the row has been reached by using a structured orchard row end recognition algorithm module based on a local image sequence. If the end of the row has been reached, the mobile robot stops moving; if the end of the row has not been reached, proceed to the next step;

[0009] S3.3: Outputting motion control strategy through motion control algorithm module based on obstacle information on both sides;

[0010] S3.4: The mobile robot receives the motion control strategy and executes a right turn or a left turn;

[0011] S3.5: Loop from S3.1 until the end of the line is reached and the movement stops.

[0012] Preferably, the motion control algorithm module based on obstacle information on both sides comprises the following steps:

[0013] S1.1: Detect all obstacles (fruit trees and support racks) using object detection algorithms;

[0014] S1.2: Divide the obstacles into two categories, left and right, based on the dividing line in the middle of the image and extract feature points for each category;

[0015] S1.3: Find the feature points L and R closest to the center line among the obstacle feature points on the left and right sides respectively;

[0016] S1.4: Translate the prior oblique line to the key point R on the right, and select a point G on the oblique line with the same y coordinate value as the feature point L;

[0017] S1.5: Calculate the distance l1 from point G to the midline, and the distance l2 from point L to the midline, and calculate the ratio k of l1 and l2. If k>1, turn right; if k<1, turn left.

[0018] Preferably, the structured orchard row end recognition algorithm module based on local image sequence comprises the following steps:

[0019] S2.1: Prepare an image classification sample set of local image slices, and perform image classification model training to obtain the optimal model parameters;

[0020] S2.2: Obtain an image from a video stream, generate a local slice sequence on top of the image, input the slice sequence into an image classification model, and obtain a classification result;

[0021] Preferably, the local slice sequence is generated on the top of the image, and the image slice is used as a basic unit, and a frame is drawn on the original image according to a certain degree of overlap, and a slice sequence is generated after stacking, and the parameters in the image classification model are the optimal model parameters obtained in S2.1;

[0022] S2.3: Combining multidimensional classification results in slice sequence, time, and space dimensions;

[0023] S2.4: Obtain the category with the most occurrences in each dimension in sequence and time order, and the final result is the result of the end-of-row classification at the corresponding spatial position;

[0024] S2.5 performs fuzzy matching on the spatial coding of the above classification results according to the coding sequence designed from the inside of the line to the end of the line, and determines that it has reached the end of the line if the conditions are met.

[0025] Preferably, the image classification sample set includes three categories: the background category is coded as 0, the sky category is coded as 1, and the leaves category is coded as 2.

[0026] Preferably, the category with the most acquisition times is calculated by the following formula:

[0027] y = argmax(bincount(x))

[0028] Where x is the classification result, y is the final result, bincount is the number of times the maximum value of the 0-sequence appears in the statistical array, and argmax is the subscript of the maximum value in the statistical array.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] 1. Compared with the traditional visual navigation method, this algorithm can achieve stable acquisition of deflection direction in vineyard scenes with complex environments, fruit trees and road surfaces of various shapes.

[0031] 2. Compared with the traditional method of extracting navigation lines for inter-row driving, the algorithm of the present invention is faster and can meet the requirements of real-time navigation.

[0032] 3. Compared with the image classification between lines and at the end of lines at the picture level, the image classification of the present invention is based on local image sequences, with clear category definition, high interpretability and high accuracy.

[0033] 4. Comprehensive decision-making is made from three dimensions: sequence, time and space, which improves the robustness and robustness of line end recognition.

[0034] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application so that it can be implemented in accordance with the contents of the specification, and to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the following is a detailed description of the preferred embodiments of the present application in conjunction with the accompanying drawings as follows.

[0035] Based on the detailed description of the specific embodiments of the present application in combination with the accompanying drawings below, those skilled in the art will become more aware of the above and other objects, advantages and features of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings without creative work. In all drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, each element or part is not necessarily drawn according to the actual scale.

[0037] Figure 1 This is an overall flow chart of an embodiment of the present application;

[0038] Figure 2 It is the flow chart of motion control algorithm based on obstacle information on both sides;

[0039] Figure 3 It is the test result diagram;

[0040] Figure 4 It is a schematic diagram of the middle dividing line;

[0041] Figure 5 Schematic diagram for extracting the nearest feature points on both sides;

[0042] Figure 6 A schematic diagram for obtaining feature point G;

[0043] Figure 7 is a schematic diagram of distance l1 and distance l2;

[0044] Figure 8 It is the flow chart of the structured orchard row end recognition algorithm based on local image sequence;

[0045] Fig. 9 This is a flowchart of the local slice sequence reasoning of an embodiment of the present application;

[0046] Fig.10 It is the standard spatial position encoding of the embodiment of the present application. DETAILED DESCRIPTION

[0047] To make the purpose, technical scheme and advantages of the embodiment of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiment of the present application. Obviously, the described embodiment is a part of the embodiment of the present application, rather than all of the embodiments. In the following description, specific details such as specific configuration and components are provided only to help fully understand the embodiments of the present application. Therefore, it should be clear to those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. In addition, for clarity and brevity, the description of known functions and structures is omitted in the embodiment.

[0048] In addition, the present application may repeat reference numerals and / or letters in different examples. This repetition is for the purpose of simplicity and clarity, and does not in itself indicate the relationship between the various embodiments and / or settings discussed.

[0049] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, B exists alone, and A and B exist at the same time. The term " / and" in this article describes another type of association object relationship, indicating that there can be two relationships. For example, A / and B can mean: A exists alone, and A and B exist alone. In addition, the character " / " in this article generally indicates that the previous and next associated objects are in an "or" relationship.

[0050] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusions.

[0051] See also Figure 1-10 The present invention provides a technical solution for a method for autonomous navigation between rows of fruit trees in a vineyard: In response to the above needs, an embodiment of the present invention provides a method for autonomous navigation between rows of fruit trees in a vineyard. The method is mainly divided into two stages: one is a motion control algorithm based on obstacle information on both sides, and the other is a structured orchard row end recognition algorithm based on a local image sequence.

[0052] like Figure 1 , the method of the present invention mainly comprises the following steps:

[0053] 1. Output a frame of image

[0054] 2. The structured orchard row end recognition algorithm module based on the local image sequence determines whether the robot has reached the end of the row. If it has, the robot stops moving; if it has not, it proceeds to the next step.

[0055] 3. The motion control module outputs the motion control strategy based on the obstacle information on both sides.

[0056] 4. The mobile robot receives the motion control strategy and executes a right turn or a left turn.

[0057] 5. Repeat step 1 until the recognition reaches the end of the line

[0058] like Figure 2 , the motion control algorithm module based on obstacle information on both sides has the following steps:

[0059] like Figure 3 As shown, all obstacles (fruit trees and support frames) are detected using the target detection algorithm;

[0060] like Figure 4 As shown in the figure, the obstacles are divided into two categories, left and right, according to the dividing line in the middle of the image, and feature points are extracted respectively. For the obstacle on the left, the lower right corner point of the envelope is used as the feature point, and for the obstacle on the right, the lower left corner point of the envelope is used as the feature point;

[0061] like Figure 5 As shown, find the feature points L and R closest to the center line among the obstacle feature points on the left and right sides respectively;

[0062] like Figure 6 As shown, the prior oblique line is translated to the key point R on the right, and a point G with the same y coordinate value as the feature point L is taken on the oblique line;

[0063] like Figure 7 As shown, the distance I1 from point G to the center line and the distance I2 from point L to the center line are calculated, and the ratio k of I1 and I2 is calculated. If k>1, the mobile robot needs to turn right, and if k<1, the mobile robot needs to turn left.

[0064] like Figure 8 As shown in FIG. 1 , the structured orchard row end recognition algorithm module based on local image sequence has the following steps:

[0065] Create an image classification sample set for local image slices and train the image classification model to obtain the optimal model parameters. The model training process includes the following steps:

[0066] Slice cropping. Draw a frame at the top of the image with a size of 96×96 pixels to crop. Once the first row is cropped, the image cropping task is completed. According to this rule, traverse all images, execute the cropping task, and generate an image sample set.

[0067] Sample annotation. The sample set has three categories: background, sky, and leaves. Specifically, in an image slice, when the sky accounts for more than half, the slice is considered as the sky category; when the leaves account for more than half, the slice is considered as the leaves category; the rest are considered as the background category.

[0068] Model training. In this embodiment, Mobilenet is selected as the image classification model, the number of output layers of the fully connected layer is changed to 3, the model parameters are iteratively updated using the above-mentioned annotated data set, and Focal Loss is used as the loss function of this task. When the loss value is the smallest, the optimal model parameters are saved. The loss function is defined as follows:

[0069] FL(P t )=-α t (1-Pt ) γ log(P t )

[0070] Among them, P t is the predicted probability, α t is the category weight factor, and γ is the sample difficulty weight adjustment factor.

[0071] Get the image from the video stream, generate a local slice sequence on top of the image, input the slice sequence into the image classification model, and get the classification result, such as Fig. 9 As shown, the specific steps are as follows:

[0072] First, obtain the motion parameters of the car at the corresponding time. When the car starts to move from the unloading point outside the orchard row, use the generated path points to plan the path to the predefined path points outside the row, start the end-of-row recognition algorithm, and switch the visual navigation algorithm within the row according to the weight ratio. At this time, the end-of-row recognition algorithm is always turned on to determine whether it has reached the end of the row. When the car reaches the end of the row during its forward movement, the car slows down until it stops. When the car is stationary, the end-of-row recognition operation is not performed. When the car is returning, the end-of-row recognition algorithm is turned on at the same time, which is used as the basis for switching the navigation algorithms inside and outside the orchard row. The movement process of the car is repeated in this way;

[0073] A rectangular frame of 96×96 pixels with a step size of 64 is used to crop the image. Fig. 9 As shown in a;

[0074] The above cropped slice sequence is stacked into a four-dimensional tensor with the following formula:

[0075] Tensor=Stack([x1,x2,...,x n ])

[0076] Among them, xi is the i-th picture, Stack is the stacking function, and Tensor is the multidimensional tensor.

[0077] Load the optimal model parameters obtained through training. The classification result is a sequence of 012 codes, such as Fig. 9 As shown in d.

[0078] Composing multi-dimensional classification results in slice sequence, time and space dimensions;

[0079] According to the sequence and time order, obtain the category with the most occurrences in each dimension, and use the final result as the end-of-row classification result in the spatial position. The specific steps are as follows:

[0080] The category with the most acquisitions in the sequence dimension is calculated using the following formula:

[0081] y = argmax(bincount(x))

[0082] Where x is the classification result, y is the final result, bincount is the number of times the maximum value of the 0-sequence appears in the statistical array, and argmax is the subscript of the maximum value in the statistical array.

[0083] The category with the highest number of acquisitions in the time dimension.

[0084] In the spatial dimension, 1 decimeter is used as the basic grid, and the maximum classification value within each grid is counted.

[0085] According to the designed coding sequence from the beginning of the line to the end of the line, the spatial coding of the above classification results is fuzzy matched, and it is determined to have reached the end of the line if the conditions are met. The specific steps are as follows:

[0086] The classification result list within the two-meter length range was intercepted, and a total of twenty valid data were obtained;

[0087] According to the visual features from the beginning of the line to the end of the line, the design standard is coded as 2 for the first ten and 1 for the last ten, such as Fig.10 As shown;

[0088] The obtained classification result list is fuzzy matched with the designed standard code. When the front and back matching rate reaches more than 60%, it can be regarded as reaching the end of the line.

[0089] Of course, the present invention may have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.

[0090] The above description is only the preferred embodiment of the present invention, which does not limit the protection scope of the present invention. For those skilled in the art, the present invention can be modified and varied in various ways. Within the spirit and principle of the present invention, any change, modification, replacement, integration and parameter change of these embodiments by conventional substitution or capable of achieving the same function without departing from the principle and spirit of the present invention shall fall within the protection scope of the present invention.

Claims

1. A method for autonomous navigation between rows of fruit trees in a vineyard, characterized by: The following steps are involved: S3.1: Acquire a frame of image in real time through the camera; S3.2: Determine whether the end of the row has been reached by using a structured orchard row end recognition algorithm module based on a local image sequence. If the end of the row has been reached, the mobile robot stops moving; if the end of the row has not been reached, proceed to the next step; S3.3: Outputting motion control strategy through motion control algorithm module based on obstacle information on both sides; S3.4: The mobile robot receives the motion control strategy and executes a right turn or a left turn; S3.5: Loop from S3.1 until the end of the line is reached and the movement stops.

2. A method for autonomous navigation between rows of fruit trees in a vineyard as claimed in claim 1, characterized in that: The motion control algorithm module based on obstacle information on both sides includes the following steps: S1.1: Detect all obstacles (fruit trees and support racks) using object detection algorithms; S1.2: Divide the obstacles into two categories, left and right, based on the dividing line in the middle of the image and extract feature points for each category; S1.3: Find the feature points L and R closest to the center line among the obstacle feature points on the left and right sides respectively; S1.4: Translate the prior oblique line to the key point R on the right, and select a point G on the oblique line with the same y coordinate value as the feature point L; S1.5: Calculate the distance l1 from point G to the midline, and the distance l2 from point L to the midline, and calculate the ratio k of l1 and l2. If k>1, you need to turn right. If k<1, you need to turn left.

3. A method for autonomous navigation between rows of fruit trees in a vineyard as claimed in claim 2, characterized in that: The structured orchard row end recognition algorithm module based on local image sequence comprises the following steps: S2.1: Prepare an image classification sample set of local image slices, and perform image classification model training to obtain the optimal model parameters; S2.2: Obtain an image from a video stream, generate a local slice sequence on top of the image, input the slice sequence into an image classification model, and obtain a classification result; The local slice sequence is generated on the top of the image, and the image slice is used as the basic unit. The frame is drawn on the original image according to a certain degree of overlap, and the slice sequence is generated after stacking. The parameters in the image classification model are the optimal model parameters obtained in S2.1; S2.3: Combining multidimensional classification results in slice sequence, time, and space dimensions; S2.4: Obtain the category with the most occurrences in each dimension in sequence and time order, and the final result is the result of the end-of-row classification at the corresponding spatial position; S2.5 performs fuzzy matching on the spatial coding of the above classification results according to the coding sequence designed from the inside of the line to the end of the line, and determines that it has reached the end of the line if the conditions are met.

4. A method for autonomous navigation between rows of fruit trees in a vineyard as claimed in claim 3, characterized in that: The image classification sample set includes three categories: the background category is coded as 0, the sky category is coded as 1, and the leaf category is coded as 2.

5. The method for autonomous navigation between rows of fruit trees in a vineyard as claimed in claim 3, characterized in that: The category with the most acquisition times is calculated by the following formula: y = argmax(bincount(x)) Where x is the classification result, y is the final result, bincount is the number of times the maximum value of the 0-sequence appears in the statistical array, and argmax is the subscript of the maximum value in the statistical array.

Images