A machine vision-based orchard inter-row road deviation identification method

Abstract

The application discloses a kind of orchard inter-row road deviation identification method based on machine vision, belong to the technical field of agricultural machinery automation, comprising: carrying out orchard inter-row road segmentation;The image after segmentation is binarized, and road image processing coordinate system is established;In the segmented image of orchard inter-row road after binarization processing, select n rows, and obtain the y coordinate value of the left and right boundary points of the selected row in the orchard inter-row road;The center point of the selected n rows is calculated, and the average position of the road center point is calculated;The average position of the road center point calculated is added to the list;Judge the length of list D_List is equal to.Based on the segmented orchard inter-row road map, the road row is selected, and the road edge and road center point coordinate data of the selected road row are extracted, the Cox-Stuart trend detection algorithm is fused, and the path deviation problem of orchard operation mobile platform caused by orchard operation mobile platform equipment failure, GNSS signal loss and other reasons is solved.

Description

TECHNICAL FIELD

[0001] The application discloses a fruit orchard inter-row road deviation identification method based on machine vision and belongs to the technical field of agricultural machinery automation. BACKGROUND

[0002] The fruit industry is a labor-intensive industry, and a large amount of labor is needed for spraying and picking in the orchard, and the low labor productivity and labor shortage seriously restrict the development of the fruit industry. In order to promote the sound development of the fruit industry, intelligent agricultural machinery is the inevitable way of future development. Positioning, autonomous navigation and obstacle avoidance technology are important components of intelligent agricultural machinery:

[0003] The Chinese patent document CN113280820A discloses a fruit orchard visual navigation path extraction method and system based on a neural network, which proposes a fruit orchard visual navigation path extraction method based on a neural network. The fruit orchard path is extracted by using a semantic segmentation method, and the left and right boundary points of the path edge are scanned. The fruit orchard visual navigation path is extracted by using the weighted average value of the left and right boundary points. The Chinese patent document CN114413892A discloses a novel fruit orchard robot combined navigation method, which proposes a fruit orchard robot combined navigation method combining inertial navigation and GPS navigation. The Chinese patent document CN111983637A discloses a fruit orchard inter-row path extraction method based on a laser radar, and the Chinese patent document CN111539473A discloses a fruit orchard inter-row navigation line extraction method based on a 3D laser radar, both of which propose a path extraction method based on a laser radar by detecting the trunk of a fruit tree. The Chinese patent document CN112425344A discloses a fruit orchard self-propelled weeding machine and an autonomous navigation and obstacle avoidance method thereof, which proposes a fruit orchard self-propelled weeding machine autonomous navigation and obstacle avoidance method combining the first information collected by an image sensor and the second information collected by a mechanical contact type obstacle sensing module. In the existing research, the GNSS / INS combined navigation method is mostly used for navigation. This method has good navigation effect under the condition that the GNSS signal is good, but in actual orchard applications, the GNSS signal is lost due to environmental interference, which affects the navigation effect. The Chinese patent document CN110948490A discloses a fruit orchard inspection robot with autonomous navigation, which includes a walking mechanism, a control module, a positioning module, a wireless communication module, a laser radar and a spherical camera. The positioning module includes an RTK unit and a ROS data processing system, and the RTK unit includes a first RTK antenna and a second RTK antenna.

[0004] However, existing tracked orchard mobile platforms employ dual-motor drive, which may experience issues such as one track malfunctioning and stopping, or GNSS signal loss during operation, potentially leading to deviations from the autonomous navigation path between orchard rows. Therefore, this invention focuses on solving the technical problem of how to utilize computer vision technology to achieve low-cost, high-precision intelligent control of orchard robots to ensure they do not deviate from their designated paths. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention discloses a machine vision-based method for identifying road offsets between rows in orchards. This method assists orchard robots in using RTK for path navigation by segmenting and detecting roads using machine vision, and by analyzing and determining whether road offsets exist using segmented road edge information and the Cox-Stuart trend detection method.

[0006] The detailed technical solution of this invention is as follows:

[0007] A machine vision-based method for recognizing row offsets in orchards, characterized by comprising:

[0008] Step S1: The orchard robot acquires images of the orchard roads between rows through a camera while moving between rows, and uses a deep learning semantic segmentation method to segment the orchard roads between rows.

[0009] The camera is installed directly in front of the orchard robot and is tilted downwards at an angle to the horizontal plane.

[0010] The semantic segmentation method involves acquiring images of orchard inter-row roads to construct an orchard inter-row road dataset, training a semantic segmentation algorithm based on transfer learning on the orchard inter-row road dataset to obtain an orchard inter-row road segmentation model, and then using the trained orchard inter-row road segmentation model to segment the orchard inter-row roads. The semantic segmentation algorithm based on transfer learning includes, but is not limited to, the FCN algorithm, the PSPNet algorithm, the Deeplabv3 algorithm, and the Deeplabv3plus algorithm.

[0011] Step S2: Binarize the segmented image and establish a road image processing coordinate system;

[0012] In the binarized orchard road segmentation image, the pixel value of the orchard road is 255, the pixel value of the non-road image part is 0, the image row size is R, and the column size is C.

[0013] The coordinate system for road image processing is established as follows: the origin is the top left corner of the segmented image, the positive x-axis is the direction from top to bottom along the left side of the image, and the positive y-axis is the direction from left to right along the top of the image.

[0014] Step S3: Select n rows in the segmented image of the orchard inter-row roads after binarization, and obtain the y-coordinate values ​​of the left and right boundary points of the orchard inter-row roads in the selected rows;

[0015] The method for selecting rows in the segmented image of the orchard inter-row roads after binarization is to select a row from the bottom edge of the image as the first row, and then select a row at fixed intervals m upwards, for a total of n rows.

[0016] The method for obtaining the y-coordinate value of the left boundary point of the orchard inter-row road is to scan the image pixel values ​​from left to right. The first pixel with a value of 255 is the left boundary point of the orchard inter-row road, and its y-coordinate value is recorded. , where i is the row number in the selected n rows;

[0017] The y-coordinate value of the right boundary point of the orchard road is obtained by scanning the image pixel values ​​from right to left. The first pixel with a value of 255 is the right boundary point of the orchard road, and its y-coordinate value is recorded. , where i is the row number in the selected n rows;

[0018] Step S4: Calculate the center point of each of the selected n rows of roads. And calculate the average position of the road center point. ;

[0019] The method for calculating the center point of the selected road in the row is as follows: (I);

[0020] The method for calculating the average position of the road center point is as follows: (II);

[0021] Step S5: Calculate the average position of the road center point. Add to list middle;

[0022] The average position of the road center point The method to add to list D_List is to Add it to the end of D_List. The data in D_List is ordered, with the data added first appearing first and the data added later appearing last.

[0023] Step S6: Determine the length of list D_List:

[0024] If the length is less than N, return S1;

[0025] If it equals N, continue the calculation downwards;

[0026] If the number is greater than N, then delete the first number in list D_List (i.e., the first number added), and denote the N numbers in list D_List as N. ;

[0027] Step S7: If N is even, then divide the data in list D_List into N / 2 groups. If N is odd, then the data in list D_List will be divided into... Group, ;

[0028] Step S8: Calculate the difference S between the two values ​​in each group. If N is even, record the number of groups N / 2 where the difference between the two values ​​is positive. Number of negative values If N is odd, record The number of pairs of values ​​with a positive difference in the group n pos The number of negative values ​​n reg ;

[0029] The method for calculating the difference S between the two values ​​of each group is as follows: (III);

[0030] Step S9: Utilize Calculate the P-value:

[0031] ,

[0032] In formula (IV), ;

[0033] Step S10: Using the calculated... And the trend of P-value calculation:

[0034] like Greater than Furthermore, when the p-value is less than 0.05, the trend is 'inc' (i.e., an increasing trend).

[0035] like Less than If the p-value is less than 0.05, the trend is 'dec' (i.e., a downward trend).

[0036] Otherwise, the trend is 'no_trend' (i.e., no trend);

[0037] Step S11: Select the left offset boundary value L of the image. v Image right offset boundary value R v And calculate whether the road is off-center;

[0038] The left offset boundary value L of the image v The range of values ​​is Image right offset boundary value R v The range of values ​​is , ;

[0039] The method for determining road deviation is as follows:

[0040] If the trend is 'inc' and Or the trend is 'dec' and If so, the road is deviated;

[0041] Otherwise, the road will not have any offset.

[0042] The technical advantages of this invention are:

[0043] To address the issue of path deviation during autonomous navigation between rows in orchards caused by tracked mobile platforms experiencing track malfunctions such as one track stopping or GNSS signal loss, this paper proposes an artificial intelligence-based method. Specifically, a semantic segmentation model based on transfer learning is used to identify the paths between orchard rows. Based on the segmented orchard road map, road rows are selected, and the coordinates of the road edges and center points of the selected road rows are extracted. By integrating the Cox-Stuart trend detection algorithm, a method for identifying orchard road deviation is proposed. This method, to some extent, solves the problems of path deviation and tree collisions caused by equipment failure and GNSS signal loss in orchard mobile platforms. Attached Figure Description

[0044] Figure 1 This is a schematic diagram illustrating the determination of path deviation between rows in an orchard using the method of the present invention.

[0045] exist Figure 1 In the image, 1 represents the orchard road between rows; 2 represents the selected orchard road after segmentation; 3 represents the right boundary point of the orchard road in the selected image row; 4 represents the left boundary point of the orchard road in the selected image row; and 5 represents the center point of the orchard road in the selected image row.

[0046] Figure 2 This is a schematic diagram showing the result of determining the path deviation of orchard inter-row roads using the method of the present invention. The determined trend is rightward deviation.

[0047] Figure 3 This is a schematic diagram showing the result of determining the path deviation of orchard inter-row roads using the method of this invention. The determined trend is leftward deviation.

[0048] Figure 4 This is a flowchart of the path offset recognition process of the present invention. Detailed Implementation

[0049] The present invention will now be described in detail with reference to the embodiments and accompanying drawings, but is not limited thereto.

[0050] Example 1

[0051] like Figure 4 As shown, a machine vision-based method for recognizing row offsets in orchards includes:

[0052] Step S1: The orchard robot acquires images of the orchard roads between rows through a camera while moving between rows, and uses a deep learning semantic segmentation method to segment the orchard roads between rows.

[0053] The camera is installed directly in front of the orchard robot and is tilted downwards at an angle to the horizontal plane.

[0054] The semantic segmentation method involves acquiring images of orchard inter-row roads to construct an orchard inter-row road dataset, training a semantic segmentation algorithm based on transfer learning on the orchard inter-row road dataset to obtain an orchard inter-row road segmentation model, and then using the trained orchard inter-row road segmentation model to segment the orchard inter-row roads. The semantic segmentation algorithm based on transfer learning includes, but is not limited to, the FCN algorithm, the PSPNet algorithm, the Deeplabv3 algorithm, and the Deeplabv3plus algorithm.

[0055] Step S2: Binarize the segmented image and establish a road image processing coordinate system;

[0056] In the binarized orchard road segmentation image, the pixel value of the orchard road is 255, the pixel value of the non-road image part is 0, the image row size is R=544, and the column size is C=960;

[0057] The coordinate system for road image processing is established as follows: the origin is the top left corner of the segmented image, the positive x-axis is the direction from top to bottom along the left side of the image, and the positive y-axis is the direction from left to right along the top of the image.

[0058] Step S3: Select n rows (n=24) from the binarized orchard inter-row road segmentation image and obtain the y-coordinate values ​​of the left and right boundary points of the orchard inter-row roads in the selected rows;

[0059] The method for selecting rows of the orchard road segmentation image after binarization is to select a row from the bottom edge of the image as the first row, and then select a row upwards at a fixed interval of m=20, for a total of n=24 rows.

[0060] The method for obtaining the y-coordinate value of the left boundary point of the orchard inter-row road is to scan the image pixel values ​​from left to right. The first pixel with a value of 255 is the left boundary point of the orchard inter-row road, and its y-coordinate value is recorded. , where i is the row number in the selected n=24 rows;

[0061] The y-coordinate value of the right boundary point of the orchard road is obtained by scanning the image pixel values ​​from right to left. The first pixel with a value of 255 is the right boundary point of the orchard road, and its y-coordinate value is recorded. , where i is the row number in the selected n=24 rows;

[0062] Step S4: Calculate the center point of each of the selected n=24 road rows. And calculate the average position of the road center point. ;

[0063] The method for calculating the center point of the selected road in the row is as follows: (I);

[0064] The method for calculating the average position of the road center point is as follows: (II);

[0065] Step S5: Calculate the average position of the road center point. Add to list middle;

[0066] The average position of the road center point The method to add to list D_List is to Add it to the end of D_List. The data in D_List is ordered, with the data added first appearing first and the data added later appearing last.

[0067] Step S6: Determine the length of list D_List:

[0068] If the length is less than N=12, then return S1;

[0069] If the result is N=12, continue the calculation downwards;

[0070] If the number is greater than N=12, then delete the first number in list D_List (i.e., the first number added), and denote the N=12 numbers in list D_List as... ;

[0071] Step S7: If N is an even number (12), then divide the data in list D_List into N / 2 = 6 groups. ,Right now ;

[0072] Step S8: Calculate the difference S between the two values ​​in each group of numbers, and record the number of groups of numbers with a positive difference. Number of negative values ;

[0073] The method for calculating the difference S between the two values ​​of each group is as follows: ;

[0074] Step S9: Utilize Calculate the P-value:

[0075] ,

[0076] In formula (IV), ;

[0077] Step S10: Using the calculated... And the trend of P-value calculation:

[0078] like Greater than Furthermore, when the p-value is less than 0.05, the trend is 'inc' (i.e., an increasing trend).

[0079] like Less than If the p-value is less than 0.05, the trend is 'dec' (i.e., a downward trend).

[0080] Otherwise, the trend is 'no_trend' (i.e., no trend);

[0081] Step S11: Select the left offset boundary value L of the image. v =262, Image right offset boundary value R v =282, and calculate whether the road is off-center;

[0082] The left offset boundary value L of the image v The range of values ​​is Image right offset boundary value R v The range of values ​​is , ;

[0083] The method for determining road deviation is as follows:

[0084] If the trend is 'inc' and Or the trend is 'dec' and If so, the road is deviated;

[0085] Otherwise, the road will not have any offset.

[0086] In this embodiment, as Figure 1As shown, the D_List values ​​calculated using the orchard row road offset identification method are [270.7, 270.65, 270.52, 270.42, 270.42, 270.67, 270.83, 270.54, 270.25, 270.25, 270.25, 270.08], n pos For 1, n reg The value is 5, the p-value is 0.2187, and D... arv The value is 270.08, and the trend shows no deviation.

[0087] In this embodiment, as Figure 2 As shown, the D_List values ​​calculated using the orchard row road offset identification method are [269.44, 270.29, 264.99, 270.54, 270.67, 270.33, 270.50, 270.88, 272.87, 277.65, 281.97, 286.59],n pos For 6, n reg The value is 0, the p-value is 0.03125, and D... arv The value is 286.59, and the trend is a rightward deviation;

[0088] In this embodiment, as Figure 3 As shown, the D_List values ​​calculated using the orchard row road offset identification method are [271.38, 270.63, 271.29, 270.75, 269.86, 269.06, 267.27, 264.24, 261.06, 257.89, 253.89, 248.78],n pos =0, n reg The value is 6, the p-value is 0.03125, and D arv The value is 248.78, and the trend is to the left.

Claims

1. A machine vision-based method for recognizing row offsets in orchards, characterized in that, include: Step S1: The orchard robot acquires images of the orchard roads between rows through a camera while moving between rows, and uses a deep learning semantic segmentation method to segment the orchard roads between rows. Step S2: Binarize the segmented image and establish a road image processing coordinate system; Step S3: Select n rows in the segmented image of the orchard inter-row roads after binarization, and obtain the y-coordinate values ​​of the left and right boundary points of the orchard inter-row roads in the selected rows; The method for selecting rows in the segmented image of the orchard inter-row roads after binarization is to select a row from the bottom edge of the image as the first row, and then select a row at fixed intervals m upwards, for a total of n rows. The method for obtaining the y-coordinate value of the left boundary point of the orchard inter-row road is to scan the image pixel values ​​from left to right. The first pixel with a value of 255 is the left boundary point of the orchard inter-row road, and its y-coordinate value is recorded. , where i is the row number in the selected n rows; The y-coordinate value of the right boundary point of the orchard road is obtained by scanning the image pixel values ​​from right to left. The first pixel with a value of 255 is the right boundary point of the orchard road, and its y-coordinate value is recorded. , where i is the row number in the selected n rows; Step S4: Calculate the center point of each of the selected n rows of roads. And calculate the average position of the road center point. ; The method for calculating the center point of the selected road in the row is as follows: (I); The method for calculating the average position of the road center point is as follows: (II); Step S5: Calculate the average position of the road center point. Add to list middle; The average position of the road center point The method to add to list D_List is to Add to the very end of D_List; Step S6: Determine the length of list D_List: If the length is less than N, return S1; If it equals N, continue the calculation downwards; If the number is greater than N, then delete the first number from list D_List, and denote the N numbers in list D_List as N. ; Step S7: If N is even, then divide the data in list D_List into N / 2 groups. If N is odd, then the data in list D_List will be divided into... Group, ; Step S8: Calculate the difference S between the two values ​​in each group. If N is even, record the number of groups N / 2 where the difference between the two values ​​is positive. Number of negative values If N is odd, record The number of pairs of values ​​with a positive difference in the group n pos The number of negative values ​​n reg ; The method for calculating the difference S between the two values ​​of each group is as follows: (III); Step S9: Utilize Calculate the P-value: ， In formula (IV), ; Step S10: Using the calculated... And the trend of P-value calculation: like Greater than Furthermore, when the p-value is less than 0.05, the trend is 'inc', which means an increasing trend; like Less than Furthermore, if the p-value is less than 0.05, the trend is 'dec', which indicates a downward trend. Otherwise, the trend is 'no_trend', meaning there is no trend.

2. The orchard row offset recognition method based on machine vision according to claim 1, characterized in that, In step S1, the camera is installed directly in front of the orchard robot and tilted downwards at an angle to the horizontal plane; The semantic segmentation method involves acquiring images of orchard inter-row roads to construct an orchard inter-row road dataset, training a semantic segmentation algorithm based on transfer learning on the orchard inter-row road dataset to obtain an orchard inter-row road segmentation model, and then using the trained orchard inter-row road segmentation model to segment the orchard inter-row roads. The semantic segmentation algorithm based on transfer learning includes the FCN algorithm, PSPNet algorithm, Deeplabv3 algorithm, and Deeplabv3plus algorithm.

3. The orchard row offset recognition method based on machine vision according to claim 1, characterized in that, In step S2, the pixel value of the orchard inter-row road segmentation image after binarization is 255, the pixel value of the non-road image part is 0, the row size of the image is R, and the column size is C. The coordinate system for road image processing is established as follows: the origin is the top left corner of the segmented image, the positive x-axis is the direction from top to bottom along the left side of the image, and the positive y-axis is the direction from left to right along the top of the image.

4. The orchard row offset recognition method based on machine vision according to claim 1, characterized in that, The identification method further includes step S11, Step S11: Select the left offset boundary value L of the image. v Image right offset boundary value R v And calculate whether the road is off-center; The left offset boundary value L of the image v The range of values ​​is Image right offset boundary value R v The range of values ​​is , ; The method for determining road deviation is as follows: If the trend is 'inc' and Or the trend is 'dec' and If so, the road is deviated; Otherwise, the road will not have any offset.

Images