Feedback control based vibration type blueberry harvesting machine control system and method

Abstract

The application belongs to the field of automatic picking, and provides a vibration type blueberry picking machine control system and method based on feedback control, which combines machine vision and blueberry picking technology. A camera is installed above a conveying belt to collect conveying belt images in real time, machine vision uses high-precision algorithms to accurately identify blueberries and calculate the proportion of impurities, so as to adjust the conveying belt speed and further reserve fan processing impurity time. The camera is installed above a blueberry harvesting tray to collect blueberry images in the harvesting tray in real time, and the proportion of unripe fruits is determined through a convolutional neural network. When the proportion is too large, the amplitude and frequency of the vibration picking device are reduced to reduce the unripe fruit drop rate. In addition, the system also has remote monitoring and fault self-diagnosis functions, which are convenient for managers to monitor the working conditions in real time.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of automatic picking, and particularly relates to a feedback control-based vibration type blueberry picking machine control system and method. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] With the rapid development of blueberry planting industry, blueberry picking has become a time-consuming and labor-intensive link in blueberry production. Due to the characteristics of small fruit size, high yield and strong seasonality, the traditional manual picking method is not only inefficient but also costly. In recent years, some blueberry picking machines have emerged, but most of them have low intelligence, low picking efficiency and high fruit damage rate. Therefore, it is particularly important to develop an efficient and intelligent blueberry picking intelligent control system.

[0004] For the structure of the existing fruit picking machine, with reference to the patent with the application number 202211227359.3 and the name of a towable lifting V-shaped vibration type small berry picking machine, it discloses that the picking part is mainly composed of a gantry frame, a shrub guiding device, a vibration picking device, a conveying device and the like. The main mechanism for generating vibration is the vibration picking device, which includes an inertial vibration power device, a vibration shaft assembly and a vibration disc assembly; the picking machine relies heavily on fixed frequency operation, resulting in unstable picking performance on different shaped fruit trees. SUMMARY

[0005] In order to solve the above problems, the present application provides a feedback control-based vibration type blueberry picking machine control system and method. The present application uses multi-information fusion to optimize the parameters of each part of the blueberry picking machine. The sensor captures the specific conditions of the blueberry picking machine during operation and feeds back to the decision system. The decision system precisely controls the motor to make the blueberry picking machine reach a certain position.

[0006] According to some embodiments, the first aspect of the present application provides a feedback control-based vibration type blueberry picking machine control method, which adopts the following technical scheme:

[0007] A feedback control-based vibration type blueberry picking machine control method, comprising:

[0008] Obtaining the image of the fruit tree for preprocessing, fitting the crown curve based on the preprocessed image of the fruit tree and judging the height and angle of the fruit tree, and adjusting the harvesting angle of the picking machine according to the height and angle of the fruit tree;

[0009] According to the fixed voltage of the picking machine, the initial values of the amplitude and frequency of the vibration picking device in the picking machine are determined, so as to vibrate and pick the blueberries.

[0010] Obtaining the blueberry image in the fruit collection device and preprocessing, adjusting the amplitude and frequency of the vibration picking device in the harvester according to the proportion of unripe fruits in the preprocessed blueberry image;

[0011] Obtaining the blueberry image in the conveying device of the harvester and preprocessing, adjusting the running speed of the conveying belt according to the impurity content in the preprocessed blueberry image of the conveying device;

[0012] After obtaining the image of the harvested fruit tree and preprocessing, adjusting the initial value of the amplitude and frequency of the vibration picking device when harvesting the next fruit tree according to the proportion of mature fruits in the preprocessed image of the harvested fruit tree.

[0013] Further, the fruit tree image is obtained and preprocessed, the crown curve is fitted based on the preprocessed fruit tree image, and the height and angle of the fruit tree are determined, specifically:

[0014] Obtaining the fruit tree image and performing gray scale processing to generate a gray scale image;

[0015] Using Hough transformation to fit the real center line of the fruit tree in the gray scale image to obtain the fitted center line of the fruit tree;

[0016] Determining the angle of the fruit tree based on the slope of the fitted center line of the fruit tree and the slope of the real center line of the fruit tree;

[0017] Calculating the lateral position deviation of the fruit tree based on the data of the real center line of the fruit tree, and taking the maximum value of the lateral position deviation of the fruit tree as the height of the fruit.

[0018] Further, adjusting the harvesting angle of the harvester according to the height and angle of the fruit tree, specifically:

[0019] Adjusting the picking frame in the harvester to be raised or lowered based on the height of the fruit tree, so that it matches the height of the blueberry plant; wherein the picking frame is a V-shaped frame composed of two vibration picking devices;

[0020] According to the angle of the fruit tree, changing the inclination angle of the vibration picking device at the fixed position to match the growth direction of the blueberry fruit.

[0021] Further, adjusting the amplitude and frequency of the vibration picking device in the harvester according to the proportion of unripe fruits in the preprocessed blueberry image, specifically:

[0022] Using a pre-trained image target recognition classification model to determine the proportion of unripe fruits in the preprocessed blueberry image;

[0023] If the proportion of unripe fruits is greater than a set threshold, the amplitude and frequency of the vibration picking device in the harvester are reduced to reduce the unripe fruit drop rate.

[0024] Further, the proportion of unripe fruits in the preprocessed blueberry image is determined using a pre-trained image target recognition classification model, specifically:

[0025] The target detection network based on deep learning is used as the image target recognition classification model to recognize and classify the processed blueberry image.

[0026] According to the recognition and classification results, the number of mature classification results and unripe classification results is determined, and the proportion of unripe classification results in all recognition and classification results is calculated to obtain the proportion of unripe fruits.

[0027] Further, the running speed of the conveyor belt is adjusted according to the high and low of the impurity content in the preprocessed conveyor belt blueberry image, specifically:

[0028] The proportion of impurities in the preprocessed conveyor belt blueberry image is segmented and recognized based on a deep learning algorithm.

[0029] If the proportion of impurities is greater than a set impurity proportion threshold, the running speed of the conveyor belt is reduced.

[0030] Further, the proportion of impurities in the preprocessed conveyor belt blueberry image is segmented and recognized based on a deep learning algorithm, specifically:

[0031] The preprocessed conveyor belt blueberry image is feature extracted to obtain a blueberry mask image and an impurity mask image.

[0032] The number of pixels is calculated based on the blueberry mask image and the impurity mask image to obtain the area of the impurity mask image and the area of the blueberry mask image.

[0033] The proportion of the impurity mask image is obtained based on the area of the impurity mask image divided by the sum of the area of the impurity mask image and the area of the blueberry mask image, that is, the proportion of impurities in the preprocessed conveyor belt blueberry image.

[0034] According to some embodiments, the second aspect of the present application provides a feedback control based vibration type blueberry harvester control system, which adopts the following technical scheme:

[0035] A feedback control based vibration type blueberry harvester control system, comprising:

[0036] The harvesting angle adjustment module is configured to obtain a fruit tree image for preprocessing, fit a tree crown curve based on the preprocessed fruit tree image, and determine the height and angle of the fruit tree, and adjust the harvesting angle of the harvester according to the height and angle of the fruit tree.

[0037] The vibration picking device setting module is configured to determine initial values of the amplitude and frequency of the vibration picking device in the harvester according to a fixed voltage of the harvester, so as to vibrate and pick blueberries;

[0038] The vibration picking device dynamic adjustment module is configured to acquire and pre-process blueberry images in the fruit collecting device, and adjust the amplitude and frequency of the vibration picking device in the harvester according to the proportion of unripe fruits in the pre-processed blueberry images.

[0039] The conveying device dynamic adjustment module is configured to acquire and pre-process blueberry images of the conveying device in the harvester, and adjust the running speed of the conveying device according to the impurity content in the pre-processed blueberry images of the conveying device.

[0040] The vibration picking device refreshing module is configured to acquire and pre-process images of a picked fruit tree, and adjust the initial values of the amplitude and frequency of the vibration picking device when picking the next fruit tree according to the proportion of ripe fruits in the pre-processed images of the picked fruit tree.

[0041] Further, the harvesting angle adjustment module acquires images of each fruit tree during picking by the blueberry harvester by using a camera arranged on the outer side of the blueberry harvester.

[0042] Further, the vibration picking device dynamic adjustment module acquires blueberry images in the fruit collecting device by using a camera arranged above the fruit collecting device.

[0043] The conveying device dynamic adjustment module acquires blueberry images of the conveying device by using a camera arranged above the conveying device.

[0044] Compared with the prior art, the present application has the following beneficial effects:

[0045] The present application detects the number of picked blueberries in the fruit collecting device in real time to determine the picking efficiency of the vibration picking device, determines the proportion of unripe fruits, and when the proportion is greater than a threshold value, the amplitude and frequency of the vibration picking device are reduced to reduce the shedding rate of unripe fruits. The number of pulses of the motor is used to accurately control the amplitude and frequency of the vibration picking device, thereby improving the picking efficiency. In addition, the picked fruit tree is detected in real time, and when the proportion of ripe fruits on the picked fruit tree exceeds a certain threshold value, the initial values of the amplitude and frequency of the vibration picking device are increased before picking the next fruit tree, thereby improving the net recovery rate of fruits.

[0046] The present application uses multi-information fusion in the blueberry harvester to optimize the parameters of each part of the blueberry harvester, uses sensors to capture the specific conditions of the harvester during operation, and feeds back to the decision system, and the decision system accurately controls the motor to make the harvester reach a certain position.

[0047] The application adopts a camera to collect blueberry images on a conveying device in real time, identifies the blueberry images and calculates the impurity proportion through a deep learning segmentation algorithm, sets a threshold value, if the impurity proportion exceeds the threshold value, reduces the conveying device speed, increases the time for the fan of the cleaning device to remove impurities, and additionally slightly reduces the amplitude and rotating speed of the vibration picking device. BRIEF DESCRIPTION OF DRAWINGS

[0048] The drawings accompanying the specification of this application form a part thereof, serve to further provide a further understanding of the application, and together with the description of the exemplary embodiments of the application, serve to explain the application, and do not constitute an improper limitation of the application.

[0049] Figure 1 is a feedback control-based vibration type blueberry harvester control method flow chart in embodiment one of the application;

[0050] Figure 2 is a hough transformation fitting blueberry tree angle schematic diagram in embodiment one of the application;

[0051] Figure 3 is an image target recognition classification model blueberry recognition effect schematic diagram in embodiment one of the application;

[0052] Figure 4 is a whole mechanism schematic diagram of the blueberry harvester in embodiment two of the application;

[0053] Figure 5 is a signal processing process of the execution model in embodiment two of the application. DETAILED DESCRIPTION

[0054] The application will be further described below in conjunction with the drawings and embodiments.

[0055] It should be noted that the following detailed description is all exemplary, and is intended to provide further description of the application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as generally understood by those skilled in the art to which the application belongs.

[0056] It should be noted that the terms used herein are only for the purpose of describing specific embodiments, and are not intended to limit the exemplary embodiments according to the application. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form, and in addition, it should be understood that when the terms "comprise" and / or "include" are used in the specification, they indicate the presence of a feature, step, operation, device, component and / or combination thereof.

[0057] The embodiments in the application and the features in the embodiments can be combined with each other without conflict.

[0058] Embodiment one

[0059] As Figure 1 shown, the embodiment provides a feedback control-based vibration blueberry harvester control method, comprising:

[0060] Obtaining the fruit tree image for preprocessing, fitting the crown curve based on the preprocessed fruit tree image and determining the height and angle of the fruit tree, and adjusting the harvesting angle of the harvester according to the height and angle of the fruit tree;

[0061] Determining the initial values of the amplitude and frequency of the vibration picking device in the harvester according to the fixed voltage of the harvester, so as to vibrate and pick the blueberries;

[0062] Obtaining the blueberry image in the fruit collecting device and preprocessing, and adjusting the amplitude and frequency of the vibration picking device in the harvester according to the proportion of unripe fruits in the preprocessed blueberry image;

[0063] Obtaining the blueberry image of the conveying device in the harvester and preprocessing, and adjusting the running speed of the conveying device according to the impurity content in the preprocessed blueberry image of the conveying device;

[0064] After obtaining the harvested fruit tree image and preprocessing, adjusting the initial values of the amplitude and frequency of the vibration picking device when picking the next fruit tree according to the proportion of ripe fruits in the preprocessed harvested fruit tree image.

[0065] The preprocessing of the fruit tree image, the fitting of the crown curve based on the preprocessed fruit tree image and the determination of the height and angle of the fruit tree are as follows:

[0066] Obtaining the fruit tree image for gray scale processing to generate a gray scale image;

[0067] Fitting the real center line of the fruit tree in the gray scale image using Hough transformation to obtain the fitted center line of the fruit tree;

[0068] Determining the angle of the fruit tree based on the slope of the fitted center line of the fruit tree and the slope of the real center line of the fruit tree;

[0069] Calculating the lateral position deviation of the fruit tree according to the data of the real center line of the fruit tree, and taking the maximum value of the lateral position deviation of the fruit tree as the height of the fruit.

[0070] The classic Hough transformation is used to fit the center line of the blueberry branch, and the evaluation indexes are the center fitting time , angle deviation and lateral position deviation , the calculation method is as shown in Figure 2 , and the specific calculation formula is:

[0071] ;

[0072] ;

[0073] ;

[0074] In the formula: represents the number of real center lines of the blueberry tree image; represents the number of real center lines of the blueberry tree image; represents the total number of center lines of the blueberry tree image; represents the number of images; represents the fitting time of the real center line of the blueberry tree image; represents the slope of the real center line of the blueberry tree image; represents the slope of the fitted center line of the blueberry tree image; represents the lateral position deviation of the blueberry tree image; represents the lateral position deviation of the blueberry tree image; represents the lateral position deviation of the blueberry tree image. It can be understood that the above angle deviation is the angle of the fruit tree.

[0075] According to the height and angle of the fruit tree, the harvesting angle of the harvester is adjusted, specifically:

[0076] Based on the height of the fruit tree, the picking frame in the harvester is adjusted to be raised or lowered to match the height of the blueberry plant; wherein the picking frame is a V-shaped frame composed of two vibrating picking devices;

[0077] According to the angle of the fruit tree, the inclination angle of the vibrating picking device is changed by adjusting the upper fixed position, so that the inclination angle of the vibrating picking device matches the growth direction of the blueberry fruit.

[0078] After obtaining the height and angle related parameters of the fruit tree, the harvester starts the up and down lifting motor of the lifting system to raise or lower the V-shaped frame-picking frame composed of two vibrating picking devices. By adjusting the height of the picking frame, it matches the height of the blueberry plant. This can reduce the loss of fruit and improve the picking efficiency.

[0079] The picking frame of the harvester is V-shaped, and the bottom is equipped with an adjusting rudder to transmit angle information and adjust the picking angle. The angle of the vibrating picking device is adjusted to match the growth direction of the blueberry fruit.

[0080] The picking frame of the harvester is V-shaped, and the bottom is equipped with an adjusting rudder to transmit angle information and adjust the picking angle. The angle of the vibrating picking device is adjusted to match the growth direction of the blueberry fruit.

[0081] The amplitude and frequency of the vibrating picking device in the harvester are adjusted according to the proportion of unripe fruits in the pre-processed blueberry image, specifically:

[0082] The proportion of unripe fruits in the pre-processed blueberry image is determined by using a pre-trained image target recognition classification model; ​

[0083] If the proportion of unripe fruits is greater than the set threshold, the amplitude and frequency of the vibration picking device in the harvester are reduced to reduce the unripe fruit drop rate.

[0084] The proportion of unripe fruits in the pre-processed blueberry image is determined using a pre-trained image target recognition classification model, and the model recognition effect is as shown in Figure 3 The specific process is as follows:

[0085] 1. Image preprocessing

[0086] A large number of blueberry image samples are obtained to ensure that the blueberry images meet the requirements of the model input, and preprocessing is performed, which usually includes adjusting the image size, normalizing the pixel value, etc.

[0087] Adjust the image size: adjust the image to the required size of 640x640 pixels for model input.

[0088] Normalization: scale the pixel value to between 0 and 1, or the model's training method adjusts the use of [0, 255] to [-1, 1] scaling.

[0089] After preprocessing, the blueberry image sample dataset is obtained.

[0090] 2. Model selection or fine-tuning

[0091] The target detection network (YOLO) of deep learning is used as the image target recognition classification model.

[0092] The blueberry image sample dataset is labeled using the mature blueberry label and the unripe blueberry label, and the training set, test set, and validation set are divided according to the labeled blueberry image sample dataset;

[0093] The model is trained using the training set to fine-tune the parameters of the image target recognition classification model to adapt to the current blueberry image classification, and the validation set and test set are used for verification and testing, and finally the trained image target recognition classification model is obtained to distinguish between mature and unripe blueberries.

[0094] 3. Target detection classification

[0095] If the number of blueberries in the image is large and unevenly distributed, use the target detection model YOLO to locate each blueberry and classify its maturity.

[0096] 4. Result calculation

[0097] The proportion of unripe blueberries is calculated according to the model output.

[0098] For each detected blueberry, record its classification result (mature or immature) and calculate the proportion of immature blueberries among all detected blueberries.

[0099] Adjust the running speed of the conveyor based on the impurity ratio in the pre-processed conveyor blueberry image. Specifically:

[0100] Segment and identify the impurity ratio in the pre-processed conveyor belt blueberry image based on deep learning algorithms.

[0101] If the impurity ratio is greater than the set threshold value, reduce the running speed of the conveyor belt.

[0102] Segment and identify the impurity ratio in the pre-processed conveyor belt blueberry image based on deep learning algorithms. Specifically:

[0103] Feature extraction is performed on the pre-processed conveyor blueberry image to obtain a blueberry mask image and a foreign matter mask image.

[0104] Based on the blueberry mask image and the foreign matter mask image, the number of pixels is calculated to obtain the area of the foreign matter mask image and the area of the blueberry mask image.

[0105] Based on the area of the foreign matter mask image divided by the sum of the area of the foreign matter mask image and the area of the blueberry mask image, the proportion of the foreign matter mask image is obtained, which is the impurity ratio in the pre-processed conveyor blueberry image.

[0106] 1. Data Preparation

[0107] Collect data: Collect conveyor images containing blueberries and impurities. These images should be representative, covering different lighting conditions, blueberry distribution, and impurity types.

[0108] Label data: Label the images to distinguish between blueberries and impurities. This usually involves creating polygon or rectangular boxes to mark each object.

[0109] Data augmentation: To increase the generalization ability of the model, you can perform transformations such as rotation, scaling, and flipping on the images.

[0110] Preprocessing: Adjust the image size, normalize the pixel values, and possibly perform other enhancement or filtering operations.

[0111] 2. Model Selection and Training

[0112] Select model: For image segmentation tasks, select the YOLO deep learning model.

[0113] Train the model: Train the model using the labeled dataset. This may require adjusting the model architecture, hyperparameters, and loss function to achieve optimal performance.

[0114] Validation Model: Evaluate the model's performance on the validation set to ensure it can accurately segment blueberries and impurities.

[0115] 3. Inference

[0116] Load Model: Load the trained model during the inference phase.

[0117] Preprocess Input Image: Apply the same preprocessing steps to the input image as during training.

[0118] Perform Segmentation: Use the model to segment the input image, resulting in masks for blueberries and impurities.

[0119] 4. Result Calculation

[0120] Calculate Impurity Area: Calculate the total area of the impurity region based on the segmentation results. This can be done by counting the number of pixels in the impurity mask.

[0121] Calculate Total Area: Similarly, calculate the total area of the entire image (or the total area of blueberries and impurities if only the relative proportion is needed).

[0122] Calculate Impurity Proportion: Divide the impurity area by the total area to obtain the proportion of impurities.

[0123] First, the blueberry image on the conveyor belt needs to be preprocessed. Preprocessing steps may include image denoising, grayscale conversion, binarization, morphological processing, etc., to improve image quality and segmentation effectiveness. These steps help reduce noise and interference in the image, making blueberries and impurities more clearly distinguishable.

[0124] Deep Learning Model Selection and Training: Use a deep learning model for image segmentation. The model has strong feature extraction and segmentation capabilities, allowing it to accurately identify blueberries and impurities. During model training, a large amount of labeled data is required. The labeled data should include accurate location and category information of blueberries and impurities. Through training, the model can learn the features of blueberries and impurities and accurately segment them in new images. It should be understood that the deep learning model used for image segmentation here can use any existing technology that can achieve image segmentation.

[0125] Image Segmentation and Impurity Recognition: After the model is trained, it can be used to segment the preprocessed conveyor belt blueberry image. The segmentation result will include a mask image of blueberries and impurities, where each pixel is assigned a class label (blueberry or impurity). By counting the number of pixels in the impurity mask image, the total area of impurities can be obtained. At the same time, the total area of the entire image can also be calculated. The proportion of impurities can be calculated by dividing the impurity area by the total area.

[0126] Result verification and optimization: Finally, the segmentation results need to be verified and optimized. The verification step includes checking the accuracy, completeness and consistency of the segmentation results, etc. If errors or deficiencies are found in the segmentation results, the model can be further optimized and adjusted to improve the segmentation effect and the accuracy of the impurity ratio.

[0127] Embodiment two

[0128] The embodiment provides a feedback control-based vibration type blueberry harvesting machine control system, which comprises:

[0129] The harvesting angle adjustment module is configured to acquire a fruit tree image for preprocessing, fit a tree crown curve based on the preprocessed fruit tree image, and determine the height and angle of the fruit tree, and adjust the harvesting angle of the harvesting machine according to the height and angle of the fruit tree.

[0130] The vibration picking device setting module is configured to determine the initial values of the amplitude and frequency of the vibration picking device in the harvesting machine according to the fixed voltage of the harvesting machine, so as to vibrate and pick blueberries.

[0131] The vibration picking device dynamic adjustment module is configured to acquire a blueberry image in the fruit collecting device and perform preprocessing, and adjust the amplitude and frequency of the vibration picking device in the harvesting machine according to the proportion of unripe fruits in the preprocessed blueberry image.

[0132] The conveying device dynamic adjustment module is configured to acquire a blueberry image of the conveying device in the harvesting machine and perform preprocessing, and adjust the running speed of the conveying device according to the impurity content in the preprocessed blueberry image of the conveying device.

[0133] The vibration picking device refreshing module is configured to acquire a harvested fruit tree image after harvesting and perform preprocessing, and adjust the initial values of the amplitude and frequency of the vibration picking device when picking the next fruit tree according to the proportion of ripe fruits in the preprocessed harvested fruit tree image.

[0134] The harvesting angle adjustment module uses a camera arranged on the outer side of the blueberry harvesting machine to acquire an image of each fruit tree during the picking process of the blueberry harvesting machine.

[0135] The vibration picking device dynamic adjustment module uses a camera arranged above the fruit collecting device to acquire a blueberry image in the fruit collecting device.

[0136] The conveying device dynamic adjustment module uses a camera arranged above the conveying device to acquire a blueberry image of the conveying device.

[0137] It should be noted that the harvester control system described in this embodiment is a blueberry harvester with the structure disclosed in the patent with the application number 202211227359.3 and the name of a towable and liftable V-shaped vibration type small berry harvester. The overall structure of the harvester structure is mainly composed of a gantry frame. The overall power supply device is a lithium battery that supplies power to the servo motor. The PLC, single-chip microcomputer and other control equipment are powered through an inverter. In addition, when the machine is working, the brushing turning speed, beating efficiency, conveying speed and other parameters can be adjusted in a small range according to the changes of external environmental physical parameters.

[0138] The overall machine is as shown in Figure 4 The working principle is as follows:

[0139] During blueberry harvesting, the "riding ridge" method is used to move at a constant speed along the soil ridge where blueberries are planted. Blueberry plants enter the brushing and beating area along the center of the gantry frame. The lithium battery supplies power to the stepping motor, which drives the vibration picking device and the vibration shaft assembly on the vibration disc assembly to rotate and reciprocate. The vibration prongs on the vibration disc drive the shrub branches to shake, and the fruit is picked. After picking, the blueberry fruits fall into the horizontal conveying device, and the conveying device and the lifting running system enter the fruit collecting basket to complete the picking task on the blueberry machine. Blueberry plants are shrubs, and the vibration picking device produces vibration effect on multiple main branches and side branches during vibration. During vibration, the motion of the fruit and the stem is simplified as a swinging motion.

[0140] In order to realize the rapid picking of blueberries, during the development of the intelligent control system of the blueberry harvester, the working process of the control system is as follows:

[0141] (1) Control the blueberry harvester to move at a constant speed;

[0142] (2) The infrared sensor installed on the blueberry harvester will detect the position of the fruit tree, so that after the harvester stops moving at the ground picking point, the image of the fruit tree is collected by the external camera, the image is processed by gray scale to generate a gray scale image, and then the Hough change and Canny edge detection algorithm are used to fit the straight line in the fruit tree image, the crown curve is fitted and its height and angle are judged, and the related data are obtained. Adjust the posture angle of the blueberry harvester through the hydraulic device to adjust the posture of the blueberry harvester;

[0143] (3) Input a fixed voltage to drive the motor to output a fixed frequency and amplitude, and the vibration picking device vibrates the fruit;

[0144] (4) The vibration picking device vibrates the fruit, and the fruit is knocked off the branches and falls into the fruit collecting device at the lower end. A camera fixed above the fruit collecting device collects the image of the blueberries in the fruit collecting device in real time, and the proportion of unripe fruit is determined. When the proportion is too large, the amplitude and frequency of the vibration picking device are reduced to reduce the falling rate of unripe fruit. The amplitude and speed of the vibration picking device are automatically adjusted according to the artificial neural network, and the mechanical harvesting action is completed according to the set time;

[0145] (5) The harvested blueberries are placed in the conveying device, and the fan of the cleaning device can clean and remove impurities such as leaves and branches of blueberries on the conveying device. At the same time, a fixed camera is placed above the conveying device to collect blueberry images on the conveying device in real time, and the impurity content is determined. When the impurity content is too high, the input voltage of the conveying device driving motor is reduced, the conveying device running speed is reduced, and the fan processing time of the conveying device blueberries in the cleaning device is increased to remove impurities.

[0146] (6) The cleaned blueberries are conveyed to the fruit collecting basket. After completing the work of one tree, the external camera acquires images of the fruit tree that has been harvested, and uses detection algorithms for real-time detection. When the proportion of mature fruit on the post-harvest fruit tree is too high, the amplitude and frequency of the vibration picking device are increased at the initial value during the next tree operation to improve the net recovery rate of fruit.

[0147] 1. System architecture

[0148] The control system is composed of hardware and software parts. The hardware part includes machine vision devices, intelligent sensing modules, controllers, and the structure of the existing patent disclosed picking machine mentioned above, etc. The software part includes image processing algorithms, control system software, multi-information fusion technology, artificial neural network technology, etc.

[0149] 2. Machine vision device

[0150] The fruit tree image is collected by an external camera, and the fruit tree image is processed to generate a gray image. Next, the Hough change and Canny edge detection algorithm is used to fit the straight line in the fruit tree image, the tree crown curve is fitted and its height and angle are judged, and the related data is obtained. The harvesting angle of the picking machine is adjusted through the hydraulic device. After completing the work of one tree, the external camera acquires images of the fruit tree that has been harvested, and uses detection algorithms for real-time detection. When the proportion of mature fruit on the post-harvest fruit tree is too high, the amplitude and frequency of the vibration picking device are increased at the initial value during the next tree operation to improve the net recovery rate of fruit.

[0151] In addition, a camera is installed above the fruit collecting device to collect blueberry images in real time, to determine the proportion of unripe fruits, and when the proportion is too large, the amplitude and frequency of the vibration picking device are reduced to reduce the falling rate of unripe fruits. According to the artificial neural network, the amplitude and rotational speed of the vibration picking device are automatically adjusted to complete the mechanical harvesting action according to the set time.

[0152] The camera on the conveying device collects and judges the blueberry impurities on the conveying device in real time. The blueberry image on the conveying device is segmented and recognized by a deep learning algorithm, and the impurity proportion of the conveying device is judged. If the proportion is relatively high, the speed of the conveying device is reduced to give the cleaning device fan sufficient time to remove impurities, thereby improving the impurity removal rate.

[0153] 3. Execution module

[0154] As shown in Figure 5 , the execution module collects the motor output signal of the vibration picking device and performs noise reduction processing to obtain a noise-reduced signal of the collected amplitude; a speed signal of the vibration picking device is collected; the noise-reduced signal and the speed signal are integrated to obtain the real-time signal of the vibration picking device;

[0155] The obtained signal is processed to obtain a real-time accurate noise-reduced signal. A pre-set magneto-electric Hall sensor is used to collect an analog signal, and an analog-to-digital conversion is performed on the obtained analog signal to obtain a rotational speed signal of the motor.

[0156] The analog signal is discretely processed to obtain a discrete signal of the analog signal; the analog signal is sampled to obtain a sampling signal of the discrete signal; the sampling signal is encoded to obtain the encoded signal, and the encoded signal is determined as the rotational speed signal. The above signals are normalized to obtain a normalized rotational speed signal; the normalized noise-reduced signal, the rotational speed signal, and the pre-set integration weight are used to generate the real-time signal of the device. According to the real-time signal and the pre-set signal threshold, a signal difference value is generated. The signal difference value is obtained by subtracting the threshold value from the real-time signal. The cumulative error and the prediction error change rate are generated using the signal difference value. The adjustment value of the harvester is generated according to the cumulative error.

[0157] The preset adjustment algorithm and the system difference value are used to generate the adjustment difference value of the harvester, and the harvester is adjusted according to the adjustment difference value. The preset adjustment function is:

[0158] ;

[0159] In the formula, is the given adjustment value of the system, is the proportional coefficient, is the integral coefficient, is the differential coefficient, is the accumulated error of the harvesting device, is the error rate of the prediction device.

[0160] 4. Conveyor device

[0161] The camera collects conveyor blueberry image data on the conveying device, sets a threshold value. Determine the proportion of unripe fruits, when the proportion is too large, then reduce the motor voltage input, and then reduce the speed of the conveying device, increase the fan processing time of the conveying device in the cleaning device, and then remove impurities. Therefore, the conveying device speed data should be collected in real time, and the conveying device driving motor output data should be filtered. The exponential moving average filter (EMA) is used to filter the data signal, which can ensure data smoothing while responding faster to actual force changes.

[0162] The calculation formula of EMA is as follows:

[0163] ;

[0164] Among them, represents the filtered value, represents the EMA filtered data at the last moment, represents the current moment data. According to the response speed and noise suppression evaluation of the moving average filter under different smoothing factors , the performance of the tactile sensor on the end effector is determined to determine the best value.

[0165] In this embodiment, when the worker needs to control the blueberry picking machine, the remote control device can send a control signal. When the controller receives the control signal, the worker operates the remote control device to control the controller to move the blueberry picking machine on the ground. At the same time, when the blueberry picking machine fails, the controller will detect the corresponding function of each module, and finally feedback the detection data to the remote control device, so that the worker can timely maintenance and debugging, to improve the working efficiency of the blueberry picking machine.

[0166] Although the specific embodiments of the present application are described above in combination with the drawings, it is not a limitation on the scope of protection of the present application. Those skilled in the art should understand that various modifications or changes made by those skilled in the art on the basis of the technical solutions of the present application without creative labor are still within the scope of protection of the present application.

Claims

1. A control method of a vibration type blueberry harvester based on feedback control, characterized by, The method comprises the following steps: Obtaining a fruit tree image for preprocessing, fitting a tree crown curve based on the preprocessed fruit tree image, and determining the height and angle of the fruit tree, and adjusting the harvesting angle of the harvester according to the height and angle of the fruit tree; The method comprises the following steps: The classic Hough transform is used to fit the centerline of the blueberry branch, and the evaluation indexes are the center fitting time , the angular deviation , and the longitudinal position deviation , and the specific calculation formula is: ； ； ； Where: Indicates the first True center line; Indicates the total number of center lines in the blueberry tree image: Indicates the number of images; Indicates the The fitting time of the true center line, in ms; Indicates the image The slope of the true center line of the blueberry tree; Indicates the image The slope of the fitted center line of the blueberry tree; Indicates the image The vertical position deviation of the blueberry tree is in pixels. The above angular deviation is used as the angle of the fruit tree. The maximum vertical position deviation of the fruit tree is used as the height of the fruit tree. According to the fixed voltage of the harvester, the initial values of the amplitude and frequency of the vibration picking device in the harvester are determined, so as to vibrate and pick blueberries; Obtaining a blueberry image in the fruit collecting device and preprocessing, adjusting the amplitude and frequency of the vibration picking device in the harvester according to the proportion of unripe fruits in the preprocessed blueberry image; Obtaining a blueberry image of the conveying device in the harvester and preprocessing, adjusting the running speed of the conveying belt according to the impurity content in the preprocessed blueberry image of the conveying device, comprising: Segmenting and identifying the impurity content in the preprocessed blueberry image of the conveying device based on a deep learning algorithm; If the impurity content is greater than the set impurity content threshold, the running speed of the conveying belt is reduced; After obtaining a harvested fruit tree image and preprocessing, the initial values of the amplitude and frequency of the vibration picking device are adjusted according to the proportion of ripe fruits in the preprocessed harvested fruit tree image when picking the next fruit tree.

2. The feedback control-based vibration-type blueberry harvester control method of claim 1, wherein, According to the height and angle of the fruit tree, the harvesting angle of the harvester is adjusted, specifically: Based on the height of the fruit tree, the picking frame in the harvester is raised or lowered to match the height of the blueberry plant; wherein the picking frame is a V-shaped frame composed of two vibration picking devices; According to the angle of the fruit tree, the inclination angle of the vibration picking device is adjusted by changing the upper fixed position, so that the inclination angle of the vibration picking device matches the growth direction of the blueberry fruit.

3. The feedback control-based vibration-type blueberry harvesting machine control method of claim 1, wherein, The method for adjusting the amplitude and frequency of the vibration picking device in the harvester according to the proportion of unripe fruits in the preprocessed blueberry image, specifically: Using a pre-trained image target recognition classification model to determine the proportion of unripe fruits in the preprocessed blueberry image; If the proportion of unripe fruits is greater than the set threshold, the amplitude and frequency of the vibration picking device in the harvester are reduced to reduce the shedding rate of unripe fruits.

4. The feedback control-based vibration-type blueberry harvesting machine control method of claim 3, wherein, Using a pre-trained image target recognition classification model to determine the proportion of unripe fruits in the preprocessed blueberry image, specifically: Using a deep learning target detection network as the image target recognition classification model to identify and classify the processed blueberry image; According to the identification and classification results, the number of mature classification results and unripe classification results is determined, and the proportion of unripe classification results in all identification and classification results is calculated to obtain the proportion of unripe fruits.

5. The feedback control-based vibration-type blueberry harvester control method of claim 1, wherein, The method for segmenting and identifying the impurity content in the preprocessed blueberry image of the conveying device based on a deep learning algorithm, specifically: Feature extraction is performed on the preprocessed blueberry image of the conveying device to obtain a blueberry mask image and a impurity mask image; Based on the blueberry mask image and the impurity mask image, the number of pixels is calculated to obtain the area of the impurity mask image and the area of the blueberry mask image; The proportion of the impurity mask image is obtained based on the area of the impurity mask image divided by the sum of the area of the impurity mask image and the area of the blueberry mask image, that is, the proportion of the impurities in the pre-processed conveyor blueberry image.

6. A feedback control based vibration type blueberry harvester control system characterized by, Comprise: The harvesting angle adjustment module is configured to acquire a fruit tree image for preprocessing, fit a tree crown curve based on the pre-processed fruit tree image, and determine the height and angle of the fruit tree, and adjust the harvesting angle of the harvester according to the height and angle of the fruit tree. The acquisition of the fruit tree image for preprocessing, the fitting of the tree crown curve based on the pre-processed fruit tree image, and the determination of the height and angle of the fruit tree comprise: The classical Hough transform is used to fit the centerline of the blueberry branch, and the evaluation indexes are the center fitting time , the angular deviation and the longitudinal position deviation , and the specific calculation formula is: ； ； ； In the formula: represents the number of real center lines of blueberry trees in the image; represents the number of real center lines of blueberry trees in the image; represents the total number of image center lines of blueberry trees: represents the number of images; represents the fitting time of the th real center line of blueberry trees, in ms; represents the slope of the th real center line of blueberry trees in the image; represents the slope of the th fitted center line of blueberry trees in the image; represents the longitudinal position deviation of the th blueberry tree in the image, in pixels, wherein the above angular deviation is the angle of the fruit tree, and the maximum value of the longitudinal position deviation of the fruit tree is the height of the fruit tree; The vibration picking device setting module is configured to determine the initial values of the amplitude and frequency of the vibration picking device in the harvester according to the fixed voltage of the harvester, so as to vibrate and pick blueberries; The vibration picking device dynamic adjustment module is configured to acquire a blueberry image in the fruit collecting device and pre-process it, and adjust the amplitude and frequency of the vibration picking device in the harvester according to the proportion of unripe fruits in the pre-processed blueberry image; The conveyor dynamic adjustment module is configured to acquire a conveyor blueberry image in the harvester and pre-process it, and adjust the running speed of the conveyor according to the impurity content in the pre-processed conveyor blueberry image, comprising: Based on the deep learning algorithm, the proportion of impurities in the pre-processed conveyor blueberry image is segmented and identified; If the proportion of impurities is greater than the set impurity proportion threshold, the running speed of the conveyor belt is reduced; The vibration picking device refreshing module is configured to acquire a harvested fruit tree image and pre-process it, and adjust the initial values of the amplitude and frequency of the vibration picking device when picking the next fruit tree according to the proportion of ripe fruits in the pre-processed harvested fruit tree image.

7. The feedback control based vibration type blueberry harvester control system as claimed in claim 6, wherein, In the harvesting angle adjustment module, a camera arranged on the outer side of the blueberry harvester acquires an image of each fruit tree during the picking process of the blueberry harvester.

8. The feedback control based vibration type blueberry harvester control system as claimed in claim 6, wherein, In the vibration picking device dynamic adjustment module, a camera arranged above the fruit collecting device acquires a blueberry image in the fruit collecting device; In the conveyor dynamic adjustment module, a camera arranged above the conveyor acquires a conveyor blueberry image.

Images