An automatic profiling tea picking method based on RGB-D camera

Abstract

The present invention discloses an automatic profiling tea picking method based on an RGB-D camera. The method comprises the following steps: scanning a tea bed surface with an RGB-D camera, extracting a tea bed surface depth image containing depth information and RGB image information, and fitting a preliminary harvesting cutting depth benchmark of a cutter using the tea bed surface depth information; then, estimating the growth density using the lateral variance of the bed surface depth information, and performing segmentation of the tea bed surface depth image using a variable threshold k-means algorithm to extract the fresh leaf pixel ratio; finally, an innovative estimation model for the optimal cutter profiling cutting depth benchmark is proposed by integrating the preliminary harvesting cutting depth benchmark of the cutter, the growth density, and the fresh leaf pixel ratio parameters; inputting a disturbance measurement signal of an uneven ridge into a linear active disturbance rejection control system, and using the filtered disturbance measurement signal as a known disturbance to improve the design of an extended state observer. The profiling cutting depth benchmark is more accurate, and the control system has high robustness and dynamic response performance.

Description

Technical Field

[0001] The present invention belongs to the technical field of agricultural machinery, and relates to the combined application of image processing technology and computer control technology in the field of tea picking machinery, and in particular to an automatic profiling tea picking method based on RGB-D camera scanning. Background Art

[0002] In recent years, my country's tea production has rapidly expanded, with both production area and output increasing. However, my country's tea industry remains large but lacks strength and refinement, resulting in low levels of technological advancement and mechanization in tea products, hindering their continued healthy development. Manual plucking, currently the predominant method for plucking fresh tea leaves before initial processing, is time-consuming, labor-intensive, and inefficient. This method also results in uneven quality due to differences in the pickers' strength, skill, and experience. Therefore, it is only suitable for small batches of premium teas requiring only one bud and one leaf, or one bud and two leaves. Existing tea plucking machines on the market are primarily used for bulk tea harvesting with lower requirements for fresh leaves. These single-person portable plucking machines harvest young tea leaves from the tops of tea trees using a handheld machine. While this improves tea collection efficiency compared to manual plucking, manual plucking is generally expensive, increasing production costs. Furthermore, the limited time available for plucking and the low efficiency of manual plucking significantly limit tea production.

[0003] To improve tea picking efficiency, some automated profiling tea plucking equipment is currently being implemented. For example, patent CN113039936A uses ultrasonic ranging for automated tea picking. However, since ultrasonic rangefinders can only measure point-to-point, they are significantly affected by the spacing between tea leaves, making them ineffective for harvesting sparse tea leaves. Others, such as 2D-LiDAR-based profiling tea plucking machines, use 2D-LiDAR point cloud fitting to determine a cutting depth benchmark for profiling tea leaves. However, since 2D-LiDAR captures limited information about the tea leaf surface, the estimated profiling cutting depth benchmark is significantly affected by leaf density and fresh leaf growth, resulting in significant deviations.

[0004] Tea is primarily grown in hilly and mountainous areas, where uneven ridges and furrows create significant disturbances. Control systems must overcome these disturbances to achieve optimal harvesting results. Traditional PID controllers and their improved algorithms are prone to overshoot or under-response when faced with disturbances. While active disturbance rejection controllers (ADRCs) are capable of estimating the total system disturbance, the extended state observer (ESO) lacks accuracy when faced with rapidly changing disturbances. While mathematical models can compensate for these disturbances, they cannot model the unevenness of the tea fields, reducing the DRC's performance.

[0005] In response to the above problems, there is an urgent need for an automatic shape-profiling harvesting method that can adapt to the growth of different fresh tea leaves and has strong anti-interference control capabilities to improve the efficiency and quality of bulk tea shape-profiling harvesting. Summary of the Invention

[0006] The purpose of the present invention is to provide an automatic profiling tea picking method based on an RGB-D camera in response to the deficiencies in the prior art.

[0007] The specific steps of the present invention are as follows:

[0008] Step 1: Fix the gantry support beam on the frame of the tea picking mechanism, above the tea ridge; fix the guide rail on the gantry support beam, and fix the RGB-D camera, accelerometer 1 and multiple profiling harvesting devices equidistantly arranged along the guide rail on the gantry support beam; the profiling harvesting device includes a self-propelled mechanism, a direct-acting member and a micro-element profiling harvesting terminal, and the direct-acting member and the micro-element profiling harvesting terminal are driven by the self-propelled mechanism to move along the gantry support beam; the micro-element profiling harvesting terminal includes a harvesting frame and a cutting knife; the harvesting frame is driven to rise and fall by the direct-acting member; the cutting knife is installed at the bottom of the harvesting frame; and an accelerometer 2 is fixed on the harvesting frame.

[0009] Step 2: The direct-acting parts of each profiling harvesting device automatically adjust the micro-element profiling harvesting end to the initial height position. The RGB-D camera on the gantry support beam scans the width of the tea field below. The controller determines the number of profiling harvesting devices required above the tea field based on the width of the tea field and the width of the micro-element profiling harvesting end fed back by the RGB-D camera, and then controls the self-propelled mechanisms of the corresponding number of profiling harvesting devices in the middle to move along the gantry support beam to above the tea field.

[0010] Step 3: During the harvesting process, the RGB-D camera is used to scan the tea stalk surface to obtain a depth image.

[0011] Step 4: Extract the depth information of each cutter and the depth information of the canopy surface of the area to be mined in front of each cutter from the depth image. Radius filtering is used to remove outliers, resulting in a depth information image of each cutter and a depth information image of the canopy surface of the area to be mined in front of each cutter. Accelerometer 2 is used to obtain the vertical acceleration of the cutter. A Kalman filter algorithm is used, using the cutter depth information as the measurement data, and the vertical acceleration state of the cutter is combined to recursively correct the cutter depth information image. Simultaneously, accelerometer 1 is used to obtain the vertical acceleration of the RGB-D camera. A Kalman filter algorithm is used, using the canopy surface depth information of the area to be mined as the measurement data, and the vertical acceleration state of the RGB-D camera is combined to recursively correct the canopy surface depth information image of the area to be mined. Then, a RANSAC algorithm is used to fit a horizontal line to the canopy surface depth information of each canopy surface to serve as the preliminary cutting depth reference s.

[0012] Step 5: Extract the RGB information of all tea leaf surfaces in front of the cutting blade from the depth image to obtain an RGB information image of the tea leaf surface. Pixels in the tea leaf surface RGB information image whose depth information falls within a preset range above are directly classified as fresh leaves, while pixels in a preset range below are directly classified as background. For the remaining image pixels, select the color component with the largest difference between fresh leaves and background as the segmentation basis. Use the variable threshold K-means clustering algorithm to segment the pixels into background and fresh leaves clusters. Then calculate the proportion P of fresh leaf pixels.

[0013] Step 6: Obtain tea leaf depth information images of all leaf surfaces in front of the cutting blades based on the leaf depth information of each area to be harvested, and extract the average value Var of the vertical depth variance of fresh tea leaves from the tea leaf depth information images.

[0014] Step 7: Establish a BP neural network model to optimize the best cutter profiling harvesting cutting depth benchmark S, and perform training and verification.

[0015] Step 8. During the harvesting process, the preliminary harvesting cutting depth reference s of the area to be harvested in front of each cutter is fitted in real time to obtain the fresh leaf pixel ratio P on the tea stalk surface and the average value of the vertical depth variance Var, which are input into the trained BP neural network model. The optimal cutter profiling harvesting cutting depth reference S of the area to be harvested in front of each cutter is output in real time, and the position information of each cutter is obtained in real time through the cutter depth information image corrected by the Kalman filter algorithm.

[0016] Step 9: Based on the position information of each cutter and the optimal cutter profiling cutting depth reference S of the area to be harvested in front of each cutter output by the BP neural network model, the linear active disturbance rejection control method is used to obtain the actual control value of the cutter profiling cutting depth.

[0017] Step 10: The direct-acting parts of each profiling harvesting device automatically adjust the cutting depth of the cutter at the end of the micro-element profiling harvesting according to the actual control value u in real time, so as to realize automatic profiling tea picking on the canopy surface of the area to be harvested in front of each cutter.

[0018] Preferably, the self-propelled mechanism includes a rotary drive member, a travel block and a roller; the roller is hinged at the bottom of the travel block and forms a rolling friction pair with the guide rail; the roller is driven by the rotary drive member; the base of the linear member is fixed to the travel block.

[0019] Preferably, limit blocks are fixed on both ends of the guide rail, and reflective strips are fixed on the side baffles of the harvesting frame.

[0020] Preferably, the remaining image pixels in step 5 are segmented into two cluster categories, background and fresh leaves, by a variable threshold K-means clustering algorithm, and the proportion P of fresh leaf pixels is calculated. The specific steps are as follows:

[0021] (1) In the remaining image pixel samples, the initial value of the color space distance threshold is set, and the maximum number of iterations of the K-means clustering algorithm is set; then two pixel samples are randomly selected, and the pixel with the smaller value of the segmentation basis color component is used as the initial background cluster center, and the other pixel is used as the initial fresh leaf cluster center;

[0022] (2) Segment the remaining image pixels into two cluster categories, background and fresh leaves, using the K-means clustering algorithm. Then, determine whether the color space Euclidean distance between the background cluster center and the fresh leaf cluster center is greater than the color space distance threshold. If the color space Euclidean distance between the background cluster center and the fresh leaf cluster center is greater than the color space distance threshold, the segmentation is completed. Otherwise, execute step (3).

[0023] (3) Determine whether the number of iterations is greater than the set maximum number of iterations. If so, reduce the color space distance threshold and return to step (2). Otherwise, use the pixel with the maximum value of the segmentation-based color component in the cluster category with the largest degree of dispersion as the new fresh leaf cluster center, and use the pixel with the minimum value of the segmentation-based color component in the cluster category with the largest degree of dispersion as the new background cluster center, and return to step (2).

[0024] More preferably, when pixel segmentation is performed on the next tea shed surface RGB information image, the current color space distance threshold is used as the initial value of the color space distance threshold of the K-means clustering algorithm.

[0025] Preferably, the specific steps of step six are as follows:

[0026] (1) According to the depth information of the tea shed surface in each area to be mined, the tea shed surface depth information image of the tea shed surface in front of all the cutters is obtained. The tea shed surface depth information image is divided into m areas with equal areas, and the vertical depth variance of the point cloud in each area is calculated.

[0027]

[0028] Where n is the number of pixels in the region; x i is the vertical depth of the i-th pixel; It is the average vertical depth of the pixels in the area.

[0029] (2) Calculate the average value of the vertical distance depth variance of the point cloud in m regions

[0030]

[0031] In the formula, Var(x j ) is the vertical depth variance of the point cloud in the jth region.

[0032] Preferably, the specific steps of step seven are as follows:

[0033] (1) A number of different tea shed surfaces were selected. For each tea shed surface, the vertical depth variance average value Var, the fresh leaf pixel ratio P, and the preliminary harvesting cutting depth reference s of the area to be harvested in front of each cutter were obtained respectively. The optimal cutter profiling harvesting cutting depth reference S of the area to be harvested in front of each cutter was manually determined, thereby establishing a data set containing the vertical depth variance average value Var, the fresh leaf pixel ratio P, the preliminary harvesting cutting depth reference s of the area to be harvested in front of each cutter, and the optimal cutter profiling harvesting cutting depth reference S of the area to be harvested in front of each cutter.

[0034] (2) A BP neural network model was constructed, with the vertical depth variance mean value Var, the fresh leaf pixel ratio P, and the preliminary harvesting cutting depth reference s of the area to be harvested in front of each cutter as input, and the optimal cutter profiling harvesting cutting depth reference S of the area to be harvested in front of each cutter as output; the data set was divided into a test set and a validation set, and the BP neural network model was trained and validated.

[0035] Preferably, the specific steps of using the linear active disturbance rejection control method to obtain the actual control value of the cutting depth of the cutter profiling harvesting are as follows:

[0036] (1) The difference h between the cutter position and the optimal cutter profiling harvesting cutting depth benchmark of the area to be harvested in front of the cutter output by the BP neural network model and the rate of change of the difference v are used as state variables to establish the dynamic control equation of the control object:

[0037]

[0038] In the formula, u is the actual control quantity output, b is the control quantity magnification factor, f(t) is the total disturbance, is the derivative of the difference h, is the derivative of the rate of change of the difference v.

[0039] (2) Since the dynamic model of the controlled object is a second-order model, the state space of the third-order extended state observer is established as follows:

[0040]

[0041] Where z1 is the estimated value of the difference h, z2 is the estimated value of the difference change rate v, z3 is the estimated value of the total disturbance f(t), b0 is the estimated value of the control amplification factor b, and fh is the derivative of the interference signal of the RGB-D camera measured by the accelerometer in the total disturbance due to the unevenness of the ridge. is the derivative of the estimated value z1, is the derivative of the estimated value z2, is the derivative of the estimated value z3; the difference h is the input of the extended state observer, and fh is input in the extended state observer; β 01 , β 02 , β 03 Gain for estimating the error of the extended state observer.

[0042] (3) Design the linear active disturbance rejection system controller as follows:

[0043] First, establish the following formula:

[0044]

[0045] Where r1 is the target value of the difference h, which is set to 0; r2 is the target value of the difference change rate v, which is set to 0; e1 is the difference between the target value r1 and the estimated value z1, and e2 is the difference between the target value r2 and the estimated value z2;

[0046] Then, design the preliminary output value of the controller control quantity:

[0047] u0=k p *e1+k d *e2

[0048] Where k p is the proportional adjustment coefficient, k d is the differential adjustment coefficient.

[0049] Finally, the actual control quantity of the linear active disturbance rejection system controller is designed:

[0050]

[0051] More preferably, a dead zone is set between the actual controlled variable u and the controlled object.

[0052] The present invention has the following beneficial effects:

[0053] (1) The present invention designs multiple micro-element profiling harvesting terminals, each of which can move horizontally and vertically. The tea bed surface is profiled by vertical movement without the need to control the cutting knife posture. The tea bed surface can be completely covered by the micro-element terminal through horizontal movement, which is suitable for profiling harvesting of tea beds of different widths and improves the profiling harvesting efficiency.

[0054] (2) The present invention uses an RGB-D camera to scan the tea shed surface, extracts the tea shed surface depth image (including depth information and RGB image information), and fits the initial harvesting cutting depth benchmark of the cutter through the tea shed surface depth information; then, the lateral variance of the shed surface depth information is used to estimate the growth density, and the tea shed surface depth image is segmented using the variable threshold K-means algorithm to extract the fresh leaf pixel ratio; finally, an innovative estimation model is proposed to integrate the initial harvesting cutting depth benchmark of the cutter, growth density and fresh leaf pixel ratio parameters to establish the optimal cutter profiling harvesting cutting depth benchmark, which is more advantageous than simply relying on RGB camera images or 3D-LiDAR point clouds to estimate the profiling harvesting cutting depth benchmark.

[0055] (3) During the profiling harvesting process, it is hoped that the cutter will cut the fresh tea leaves along the optimal cutter profiling harvesting cutting depth benchmark. However, under the disturbance conditions of uneven ridges and furrows in the tea fields in hilly and mountainous areas, the disturbance of the uneven ridges and furrows is transmitted to the end cutter through the tea picking device, causing the cutter to shake, and the quality of the fresh leaf profiling harvesting is seriously affected. The present invention proposes inputting the disturbance measurement signal of the uneven ridges and furrows into the linear anti-disturbance control system, using the filtered interference measurement signal as the known disturbance, and improving the design of the extended state observer. The introduction of the measured value of the known disturbance improves the estimation accuracy and convergence speed of the extended state observer for the rapidly changing total disturbance, thereby improving the robustness and dynamic response performance of the control system. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 This is a schematic diagram of the contour harvesting devices used in the device of the present invention being arranged equidistantly before contouring.

[0057] Figure 2 It is a schematic diagram of the contour harvesting device in the present invention.

[0058] Figure 3 This is a schematic diagram of the positions of the various contoured harvesting devices in the device used in the present invention after being allocated according to the width of the tea field.

[0059] Figure 4 This is a schematic diagram of the RGB-D camera in the present invention scanning the tea shed surface.

[0060] Figure 5 This is a schematic diagram of Kalman filter fusion in the present invention.

[0061] Figure 6 This is a schematic diagram of the positions of the preliminary harvesting cutting depth reference and the optimal cutter profiling harvesting cutting depth reference in the present invention.

[0062] Figure 7 This is a flowchart of the present invention for extracting decision information from the depth image obtained by scanning the tea shed surface with an RGB-D camera.

[0063] Figure 8This is a block diagram of the cutting depth control system of the cutter in the present invention. DETAILED DESCRIPTION

[0064] The present invention will be further described below with reference to the accompanying drawings.

[0065] An automatic profiling tea picking method based on RGB-D camera, the specific steps are as follows:

[0066] Step 1: Figure 1 As shown, the gantry support crossbeam 1-1 is fixedly mounted on the frame of the tea picking travel mechanism (the tea picking travel mechanism is provided with a travel wheel 1-4 hinged to the bottom of the frame and a motor for driving the travel wheel to rotate), and is located above the tea ridge; the guide rail 1-6 is fixed on the gantry support crossbeam 1-1, and the RGB-D camera 1-9, the accelerometer 1-8 and a plurality of profiling harvesting devices equidistantly arranged along the guide rail 1-6 are fixed on the gantry support crossbeam; the profiling harvesting device includes a self-propelled mechanism, a direct-acting member (for example, a linear module 1-3 driven by a servo motor 2-1) and a micro-element profiling harvesting terminal 1-2, and the direct-acting member and the micro-element profiling harvesting terminal 1-2 are driven by the self-propelled mechanism to move along the gantry support crossbeam; The micro-element profiling harvesting terminal 1-2 includes a harvesting frame and a cutting knife 2-5; the harvesting frame is driven to rise and fall by a direct-acting member; the cutting knife 2-5 is installed at the bottom of the harvesting frame and is driven by its own power source; an accelerometer 2-10 is also fixed on the harvesting frame; preferably, the self-propelled mechanism includes a rotating drive member (such as a motor), a walking block 2-2 and a roller; the roller is hinged at the bottom of the walking block 2-2 and forms a rolling friction pair with the guide rail 1-6; the roller is driven by a rotating drive member; the base of the direct-acting member is fixed to the walking block 2-2; preferably, limit blocks 1-5 are fixed at both ends of the guide rail 1-6, and reflective strips 2-4 are fixed on the side baffle 2-3 of the harvesting frame to facilitate the RGB-D camera 1-9 to locate the cutting knife position.

[0067] Step 2: The direct-acting parts of each profiling harvesting device automatically adjust the micro-element profiling harvesting end 1-2 to the initial height position ( Figure 1 As shown), the RGB-D camera 1-9 on the gantry support beam 1-1 scans the tea ridge below ( Figure 1 The width of the tea ridges (numbered 1-7) is shown in the figure. The controller determines the number of profiling harvesting devices required above the tea ridges based on the width of the tea ridges fed back by the RGB-D cameras 1-9 and the width of the micro-element profiling harvesting terminals 1-2, and then controls the self-propelled mechanisms of the corresponding number of profiling harvesting devices in the middle to move along the crossbeam of the gantry support to the top of the tea ridges. In this way, the number of required micro-element profiling harvesting terminals can be calculated according to the width of the tea ridges to adapt to tea ridges of different widths. In order to avoid collisions between adjacent profiling harvesting devices, adjacent profiling harvesting devices can be set at intervals of 0.5 to 1 cm. Figure 3 The following are the components of each contoured harvesting device: Figure 1The tea leaves are evenly spaced and adjusted to the corresponding tea ridges on the gantry support beam 1-1, so as to avoid the problem of some tea leaves not being harvested and causing waste.

[0068] Step 3: When automatically harvesting large quantities of tea by profiling, it is hoped that the cutter can cut along the roots of the fresh tea leaves and cut the fresh tea leaves completely. Therefore, the roots of the fresh tea leaves are used as the optimal cutting depth reference position for the cutter profiling. Since the sensor cannot directly sense the position of the profiling cutting depth reference during the actual profiling harvesting process, the present invention obtains the optimal profiling cutting depth reference position by algorithmically processing the sensor perception information, and obtains a depth image (including depth information and RGB information) by scanning the tea shed surface with an RGB-D camera. Figure 4 As shown (only five cutters are shown in the figure, which does not represent the actual situation), the RGB-D camera has a depression angle of 45°, number 4-1 indicates the ridge, and number 4-2 indicates the tea shed surface in the area to be harvested in front of the cutters.

[0069] Step 4: Assuming that the length and growth density of fresh leaves on a row of tea leaves within the scanning range of the RGB-D camera are the same, the depth information of each cutting knife and the depth information of the tea leaf surface in front of each cutting knife are extracted from the depth image. The outliers are removed through the radius filtering method to obtain the depth information image of each cutting knife and the depth information image of the tea leaf surface in front of each cutting knife. In order to resist natural light when acquiring depth information, the outdoor RGB-D camera adopts the method of time of flight measurement (ToF) to acquire depth information. The sampling frequency of the depth information image is generally below 50Hz, and the measurement accuracy is generally 2 to 3cm, which cannot meet the measurement accuracy requirements of contour harvesting. The sampling frequencies of accelerometer 1-8 and accelerometer 2 1-10 are several times that of the RGB-D camera. Therefore, during driving, the present application obtains the vertical acceleration of the cutter through accelerometer 2 1-10, adopts the Kalman filter algorithm, uses the cutter depth information as the measurement data, and combines the vertical acceleration state of the cutter to recursively correct the cutter depth information image. At the same time, the vertical acceleration of the RGB-D camera is obtained through accelerometer 1-8, adopts the Kalman filter algorithm, uses the depth information of the canopy surface of the to-be-mined area as the measurement data, and combines the vertical acceleration state of the RGB-D camera to recursively correct the depth information image of the canopy surface of the to-be-mined area. Figure 5 As shown, Figure 5 k represents the time. Then, the RANSAC algorithm is used to fit a horizontal straight line to the depth information of the tea leaf surface of each area to be harvested as the preliminary harvesting cutting depth benchmark s. At this time, when the tea leaf surface growth density is large, the estimated preliminary harvesting cutting depth benchmark s value is likely to be too high, otherwise it is likely to be too low. In addition, when the fresh tea leaves are long, the estimated preliminary harvesting cutting depth benchmark s value is likely to be too high, otherwise it is likely to be too low. Figure 6The figure shows the cutting depth benchmark for fresh tea leaves. Figure 6 The middle number 6-1 indicates the position of the preliminary harvesting cutting depth reference s, and the number 6-2 indicates the position of the optimal cutter profiling harvesting cutting depth reference. There is a large deviation between the two.

[0070] Step 5, extract the RGB information of the tea canopy surface in front of all cutters from the depth image to obtain the RGB information image of the tea canopy surface. Under natural conditions, the fresh leaves on the tea canopy surface block each other, and the old leaves are close to the fresh leaves in color. The length of the fresh leaves cannot be directly obtained, but from a statistical point of view, there is a positive correlation between the fresh leaf pixel ratio and the fresh leaf length after image segmentation. The present invention uses the fresh leaf pixel ratio to express the fresh leaf length. According to the spatial distribution characteristics of the fresh leaves, the pixels whose depth information in the tea canopy surface RGB information image is located in the upper preset range are directly classified as fresh leaves, and the pixels in the lower preset range are directly classified as background. The remaining image pixels select the color component (such as G color component) with a large difference between fresh leaves and background as the segmentation basis, and are segmented into two clustering categories of background and fresh leaves by the variable threshold K-means clustering algorithm. Then, the ratio P of fresh leaf pixels is calculated, as shown in FIG. Figure 7 The figure shows the flow chart of the variable threshold K-means clustering algorithm. The specific steps are as follows:

[0071] (1) In the remaining image pixel samples, the initial value of the color space distance threshold is set, and the maximum number of iterations of the K-means clustering algorithm is set; then two pixel samples are randomly selected, and the pixel with the smaller value of the segmentation basis color component is used as the initial background cluster center, and the other pixel is used as the initial fresh leaf cluster center;

[0072] (2) Segment the remaining image pixels into two cluster categories, background and fresh leaves, using the K-means clustering algorithm. Then, determine whether the color space Euclidean distance between the background cluster center and the fresh leaf cluster center is greater than the color space distance threshold. If the color space Euclidean distance between the background cluster center and the fresh leaf cluster center is greater than the color space distance threshold, the segmentation is completed. Otherwise, execute step (3).

[0073] (3) Determine whether the number of iterations is greater than the set maximum number of iterations. If so, reduce the color space distance threshold and return to step (2). Otherwise, use the pixel with the maximum value of the segmentation-based color component in the cluster category with the largest degree of dispersion as the new fresh leaf cluster center, and use the pixel with the minimum value of the segmentation-based color component in the cluster category with the largest degree of dispersion as the new background cluster center, and return to step (2).

[0074] Among them, when performing pixel segmentation on the next tea shed surface RGB information image, the current color space distance threshold is used as the initial value of the color space distance threshold of the K-means clustering algorithm to improve the segmentation efficiency.

[0075] Step 6. Based on the depth information of the tea bed surface in each area to be harvested, the tea bed surface depth information image of all the tea bed surfaces in front of the cutting knife is obtained, and the variance average of the vertical depth of the fresh tea leaves (the projection of the distance from the RGB-D camera in the vertical direction) that characterizes the growth density of fresh tea leaves is extracted from the tea bed surface depth information image. When the growth density of fresh tea leaves is high, the distance between the leaves is small, and vice versa. The large distance between the leaves makes it easy for the laser beam of the RGB-D camera to pass through the fresh leaf layer to reach the bottom. At this time, the depth information is sparsely distributed. On the contrary, it is easily reflected by the surface fresh leaves, and the depth information is densely distributed. Therefore, it is proposed to divide the tea bed surface depth information image into several areas, and express the growth density of fresh tea leaves by the average value of the vertical depth variance of the point cloud of each area. This method has strong real-time performance and low computational complexity. The specific steps are as follows:

[0076] (1) According to the depth information of the tea shed surface in each area to be harvested, the tea shed surface depth information image of the tea shed surface in front of all the cutters is obtained. The tea shed surface depth information image is divided into m areas with equal areas, and the vertical direction (Z-axis direction) depth variance of the point cloud (all pixels in the area) in each area is calculated.

[0077]

[0078] Where n is the number of pixels in the region; x i is the vertical depth of the i-th pixel, in meters; It is the average vertical depth of the pixels in the area, in meters.

[0079] (2) Calculate the average vertical depth variance of the point cloud in m regions

[0080]

[0081] In the formula, Var(x j ) is the vertical depth variance of the point cloud in the jth region, in m 2 .

[0082] Step 7: Establish a BP neural network model to optimize the best cutter profiling harvesting cutting depth benchmark S, and perform training and verification. The specific steps are as follows:

[0083] (1) Select multiple different tea shed surfaces, and for each tea shed surface, obtain the vertical depth variance average Var, fresh leaf pixel ratio P, and the preliminary harvesting cutting depth reference s of the area to be harvested in front of each cutter, and manually determine the optimal cutter profiling harvesting cutting depth reference S of the area to be harvested in front of each cutter, thereby establishing a data set containing the vertical depth variance average Var, fresh leaf pixel ratio P, the preliminary harvesting cutting depth reference s of the area to be harvested in front of each cutter, and the optimal cutter profiling harvesting cutting depth reference S of the area to be harvested in front of each cutter. Among them, when selecting tea shed surfaces, four types of tea shed surfaces should be covered as much as possible: dense long leaf type, dense short leaf type, sparse long leaf type, and sparse short leaf type. The tea shed surfaces can be divided into dense long leaf type, dense short leaf type, sparse long leaf type, and sparse short leaf type according to the fresh leaf pixel ratio P and the vertical depth variance average Var. The Var boundary between dense and sparse, and the P boundary between long leaf and short leaf are pre-set.

[0084] (2) A BP neural network model was constructed, with the vertical depth variance mean value Var, the fresh leaf pixel ratio P, and the preliminary harvesting cutting depth reference s of the area to be harvested in front of each cutter as input, and the optimal cutter profiling harvesting cutting depth reference S of the area to be harvested in front of each cutter as output; the data set was divided into a test set and a validation set with a ratio of 7:3, and the BP neural network model was trained and validated.

[0085] Step 8. During the actual profiling harvesting process, the preliminary harvesting cutting depth reference s of the area to be harvested in front of each cutter is fitted in real time to obtain the fresh leaf pixel ratio P on the tea stalk surface and the average value of the vertical depth variance Var, which are input into the trained BP neural network model. The optimal cutter profiling harvesting cutting depth reference S of the area to be harvested in front of each cutter is output in real time, and the position information of each cutter is obtained in real time through the cutter depth information image corrected by the Kalman filter algorithm.

[0086] Step nine, based on the position information of each cutter and the optimal cutter profiling harvesting cutting depth reference S of the area to be harvested in front of each cutter output by the BP neural network model, a linear self-anti-disturbance control method is used to obtain the actual control value of the cutter profiling harvesting cutting depth. In the actual profiling harvesting process, assuming that the cutter cutting depth is H, it is hoped that the cutter can harvest fresh tea leaves along the position of the optimal cutter profiling harvesting cutting depth reference S, so the control target is HS=0, but it is affected by model uncertainty, sensor measurement noise and uneven ridge and furrow fluctuations, especially the interference of uneven ridge and furrow fluctuations, which brings great challenges to the high-precision and fast-response control of the cutter. Therefore, the present invention adopts a linear self-anti-disturbance control method to provide decision information for the control of the profiling harvesting cutting depth of each cutter, and introduces the measured value of the known disturbance to improve the estimation accuracy and convergence speed of the expanded state observer for rapidly changing disturbances. Figure 8As shown in FIG, the specific steps of obtaining the actual control value of the cutting depth of the cutter profiling harvesting by the linear active disturbance rejection control method are as follows:

[0087] (1) The difference h between the cutter position and the optimal cutter profiling harvesting cutting depth benchmark of the area to be harvested in front of the cutter output by the BP neural network model and the rate of change of the difference v are used as state variables to establish the dynamic control equation of the control object:

[0088]

[0089] In the formula, u is the actual control quantity output, b is the control quantity magnification factor, f(t) is the total disturbance, is the derivative of the difference h, is the derivative of the rate of change of the difference v.

[0090] (2) Since the dynamic model of the controlled object is a second-order model, the state space of the third-order extended state observer is established as follows:

[0091]

[0092] Where z1 is the estimated value of the difference h, z2 is the estimated value of the difference change rate v, z3 is the estimated value of the total disturbance f(t), b0 is the estimated value of the control amount amplification factor b, and fh is the derivative (with respect to time) of the interference signal of the RGB-D camera 1-9 measured by the accelerometer 1-8 due to the disturbance of the ridge. is the derivative of the estimated value z1, is the derivative of the estimated value z2, is the derivative of the estimated value z3; the difference h is the input of the extended state observer, and fh is input into the extended state observer. Inputting the differential of the known disturbance part (fh) can improve the accuracy and convergence speed of the total disturbance estimate z3; since the accelerometer measurement signal contains a lot of noise, the known disturbance signal is low-pass filtered and then differentiated before being input into the extended state observer; β 01 , β 02 , β 03 is the gain of the estimated error of the extended state observer, which is calculated according to the pole configuration requirements of the extended state observer. For the convenience of parameter debugging, β is generally taken 01 =3w0, w0 is the observation frequency of the extended state observer. The larger the value of w0 is, the smaller the steady-state error of the system control is, but the greater the observation noise is.

[0093] (3) Design the linear active disturbance rejection system controller as follows:

[0094] First, establish the following formula:

[0095]

[0096] Where r1 is the target value of the difference h, which is set to 0; r2 is the target value of the difference change rate v, which is set to 0; e1 is the difference between the target value r1 and the estimated value z1, and e2 is the difference between the target value r2 and the estimated value z2;

[0097] Then, design the preliminary output value of the controller control quantity:

[0098] u0=k p *e1+k d *e2

[0099] Where k p is the proportional adjustment coefficient, k d is the differential adjustment coefficient. According to the transfer function pole configuration relationship between the input and output of the linear active disturbance rejection system controller, k p =2ξw c , w c is the control frequency, ξ is the damping coefficient, and its value is 0.707, at which the response speed is the fastest and there is no overshoot. Among them, the observation frequency w0 of the extended state observer is generally 3~5w c .

[0100] Finally, the actual control quantity of the linear active disturbance rejection system controller is designed:

[0101]

[0102] A dead zone is set between the actual control quantity u and the controlled object to avoid frequent vibration of the cutter caused by a very small control quantity.

[0103] Step 10. The direct-acting parts of each profiling harvesting device automatically adjust the cutting depth of the cutter at the micro-element profiling harvesting end 1-2 in real time according to the actual control amount u, so as to realize automatic profiling tea picking on the canopy surface of the area to be harvested in front of each cutter, so that each cutter can cut along the root of the fresh tea leaves as much as possible.

Claims

1. An automatic profiling tea picking method based on an RGB-D camera, characterized by: The specific steps are as follows: Step 1: Fix the gantry support crossbeam on the frame of the tea picking travel mechanism, located above the tea ridge; fix the guide rail on the gantry support crossbeam, and fix the RGB-D camera, accelerometer 1 and multiple profiling harvesting devices equidistantly arranged along the guide rail on the gantry support crossbeam; the profiling harvesting device includes a self-propelled mechanism, a linear motion member and a micro-element profiling harvesting terminal, and the linear motion member and the micro-element profiling harvesting terminal are driven by the self-propelled mechanism to move along the gantry support crossbeam; the micro-element profiling harvesting terminal includes a harvesting frame and a cutting knife; the harvesting frame is driven to rise and fall by the linear motion member; the cutting knife is installed at the bottom of the harvesting frame; the harvesting frame is fixed with accelerometer 2; Step 2: The direct-acting parts of each profiling harvesting device automatically adjust the micro-element profiling harvesting end to the initial height position. The RGB-D camera on the gantry support beam scans the width of the tea field below. The controller determines the number of profiling harvesting devices required above the tea field based on the width of the tea field and the width of the micro-element profiling harvesting end fed back by the RGB-D camera. Then, the controller controls the self-propelled mechanisms of the corresponding number of profiling harvesting devices in the middle to move along the gantry support beam to above the tea field. Step 3: During the harvesting process, the RGB-D camera is used to scan the tea stalk surface to obtain a depth image; Step 4: Extract the depth information of each cutter and the depth information of the canopy surface of the area to be mined in front of each cutter from the depth image, and remove outliers through the radius filtering method to obtain the depth information image of each cutter and the depth information image of the canopy surface of the area to be mined in front of each cutter; obtain the vertical acceleration of the cutter through accelerometer 2, adopt the Kalman filter algorithm, use the cutter depth information as the measurement data, and recursively correct the cutter depth information image in combination with the vertical acceleration state of the cutter; at the same time, obtain the vertical acceleration of the RGB-D camera through accelerometer 1, adopt the Kalman filter algorithm, use the depth information of the canopy surface of the area to be mined as the measurement data, and recursively correct the depth information image of the canopy surface of the area to be mined in combination with the vertical acceleration state of the RGB-D camera; then, use the RANSAC algorithm to fit a horizontal straight line to the canopy surface depth information of each area to be mined as the preliminary harvesting cutting depth reference s; Step 5: Extract the RGB information of all tea leaves in front of the cutting blade from the depth image to obtain an RGB information image of the tea leaf surface. Pixels in the RGB information image of the tea leaf surface whose depth information is within a preset range above are directly classified as fresh leaves, and pixels in a preset range below are directly classified as background. The remaining image pixels are segmented using the color component with a large difference between fresh leaves and background as the basis for segmentation using a variable threshold K-means clustering algorithm into two cluster categories: background and fresh leaves. Then, the proportion P of fresh leaf pixels is calculated. Step 6: Obtain tea leaf depth information images of all tea leaf surfaces in front of the cutting blades based on the tea leaf depth information of each area to be harvested, and extract the average value Var of the vertical depth variance of fresh tea leaves from the tea leaf depth information images; Step 7: Establish a BP neural network model to optimize the best cutter profiling harvesting cutting depth benchmark S, and perform training and verification; Step 8: During the harvesting process, the preliminary harvesting cutting depth reference s of the area to be harvested in front of each cutter is fitted in real time to obtain the fresh leaf pixel ratio P and the vertical depth variance average Var of the tea stalk, which are input into the trained BP neural network model. The optimal cutter profiling harvesting cutting depth reference S of the area to be harvested in front of each cutter is output in real time, and the position information of each cutter is obtained in real time through the cutter depth information image corrected by the Kalman filter algorithm; Step 9: Based on the position information of each cutter and the optimal cutter profiling cutting depth reference S of the area to be harvested in front of each cutter output by the BP neural network model, a linear active disturbance rejection control method is used to obtain the actual control value of the cutter profiling cutting depth; Step 10: The direct-acting parts of each profiling harvesting device automatically adjust the cutting depth of the cutter at the end of the micro-element profiling harvesting according to the actual control value u in real time, so as to realize automatic profiling tea picking on the canopy surface of the area to be harvested in front of each cutter.

2. The automatic profiling tea picking method based on an RGB-D camera according to claim 1, characterized in that: The self-propelled mechanism includes a rotary drive member, a travel block and a roller; the roller is hinged at the bottom of the travel block and forms a rolling friction pair with the guide rail; the roller is driven by the rotary drive member; the base of the linear member is fixed to the travel block.

3. The automatic profiling tea picking method based on an RGB-D camera according to claim 1, characterized in that: Both ends of the guide rail are fixed with limit blocks, and the side baffles of the harvesting frame are fixed with reflective strips.

4. The automatic profiling tea picking method based on an RGB-D camera according to claim 1, characterized in that: In step 5, the remaining image pixels are segmented into two cluster categories, background and fresh leaves, using the variable threshold K-means clustering algorithm, and the proportion P of fresh leaf pixels is calculated. The specific steps are as follows: (1) In the remaining image pixel samples, the initial value of the color space distance threshold is set, and the maximum number of iterations of the K-means clustering algorithm is set; then two pixel samples are randomly selected, and the pixel with the smaller value of the segmentation basis color component is used as the initial background cluster center, and the other pixel is used as the initial fresh leaf cluster center; (2) Segment the remaining image pixels into two cluster categories, background and fresh leaves, using the K-means clustering algorithm. Then, determine whether the color space Euclidean distance between the background cluster center and the fresh leaf cluster center is greater than the color space distance threshold. If the color space Euclidean distance between the background cluster center and the fresh leaf cluster center is greater than the color space distance threshold, the segmentation is completed. Otherwise, execute step (3). (3) Determine whether the number of iterations is greater than the set maximum number of iterations. If it is greater than the maximum number of iterations, reduce the color space distance threshold and return to step (2). Otherwise, take the pixel with the maximum value of the segmentation-based color component in the cluster category with the largest degree of dispersion as the new fresh leaf cluster center, and take the pixel with the minimum value of the segmentation-based color component in the cluster category with the largest degree of dispersion as the new background cluster center, and return to step (2).

5. The automatic profiling tea picking method based on RGB-D camera according to claim 4, characterized in that: When performing pixel segmentation on the next tea shed surface RGB information image, the current color space distance threshold is used as the initial value of the color space distance threshold of the K-means clustering algorithm.

6. The automatic profiling tea picking method based on an RGB-D camera according to claim 1, characterized in that: Step 6 The specific steps are as follows: (1) According to the depth information of the tea shed surface in each area to be mined, the tea shed surface depth information image of the tea shed surface in front of all the cutters is obtained. The tea shed surface depth information image is divided into m areas with equal areas, and the vertical depth variance of the point cloud in each area is calculated. Where n is the number of pixels in the region; x i is the vertical depth of the i-th pixel; is the average vertical depth of the pixels in the area; (2) Calculate the average vertical depth variance of the point cloud in m regions In the formula, Var(x j ) is the vertical depth variance of the point cloud in the jth region.

7. The automatic profiling tea picking method based on an RGB-D camera according to claim 1, characterized in that: Step 7 The specific steps are as follows: (1) Select multiple different tea shed surfaces, and for each tea shed surface, obtain the vertical depth variance average value Var, the fresh leaf pixel ratio P, and the preliminary harvesting cutting depth reference s of the area to be harvested in front of each cutter, and manually determine the optimal cutter profiling harvesting cutting depth reference S of the area to be harvested in front of each cutter, thereby establishing a data set containing the vertical depth variance average value Var, the fresh leaf pixel ratio P, the preliminary harvesting cutting depth reference s of the area to be harvested in front of each cutter, and the optimal cutter profiling harvesting cutting depth reference S of the area to be harvested in front of each cutter; (2) A BP neural network model was constructed, with the vertical depth variance mean value Var, the fresh leaf pixel ratio P, and the preliminary harvesting cutting depth reference s of the area to be harvested in front of each cutter as input, and the optimal cutter profiling harvesting cutting depth reference S of the area to be harvested in front of each cutter as output; the data set was divided into a test set and a validation set, and the BP neural network model was trained and validated.

8. The automatic profiling tea picking method based on an RGB-D camera according to claim 1, characterized in that: The specific steps for obtaining the actual control value of the cutting depth of the cutter using the linear active disturbance rejection control method are as follows: (1) The difference h between the cutter position and the optimal cutter profiling harvesting cutting depth benchmark of the area to be harvested in front of the cutter output by the BP neural network model and the rate of change of the difference v are used as state variables to establish the dynamic control equation of the control object: In the formula, u is the actual control quantity output, b is the control quantity magnification factor, f(t) is the total disturbance, is the derivative of the difference h, is the derivative of the rate of change of the difference v; (2) Since the dynamic model of the controlled object is a second-order model, the state space of the third-order extended state observer is established as follows: Where z1 is the estimated value of the difference h, z2 is the estimated value of the difference change rate v, z3 is the estimated value of the total disturbance f(t), b0 is the estimated value of the control amplification factor b, and fh is the derivative of the interference signal of the RGB-D camera measured by the accelerometer in the total disturbance due to the unevenness of the ridge. is the derivative of the estimated value z1, is the derivative of the estimated value z2, is the derivative of the estimated value z3; the difference h is the input of the extended state observer, and fh is input in the extended state observer; β 01 , β 02 , β 03 The gain of the error estimation for the extended state observer; (3) Design the linear active disturbance rejection system controller as follows: First, establish the following formula: Where r1 is the target value of the difference h, which is set to 0; r2 is the target value of the difference change rate v, which is set to 0; e1 is the difference between the target value r1 and the estimated value z1, and e2 is the difference between the target value r2 and the estimated value z2; Then, design the preliminary output value of the controller control quantity: u0=k p *e1+k d *e2 Where k p is the proportional adjustment coefficient, k d is the differential adjustment coefficient; Finally, the actual control quantity of the linear active disturbance rejection system controller is designed:

9. The automatic profiling tea picking method based on RGB-D camera according to claim 8, characterized in that: A dead zone is set between the actual controlled variable u and the controlled object.

Images