Self-adaptive regulation and control method of potato picking harvester based on deep learning

Abstract

The invention relates to the technical field of potato harvesting, in particular to a potato picking harvester self-adaptive regulation and control method based on deep learning, which comprises the following steps: periodically collecting and picking harvester front earth surface images; dividing a region of interest; identifying the number of potatoes and the thickness of the potato stream within the region of interest; the advancing speed of the picking harvester, the rotating speed of a conveying chain and the number of potatoes to be picked are collected; identifying whether the potato flow thickness is within an allowable variation range; and if not, inputting the forward speed, the rotating speed of the conveying chain and the number of the potatoes into the neural network, and outputting the forward speed and the rotating speed of the conveying chain at the next moment. Based on the deep learning algorithm, the neural network model and the PID controller, various key indexes in the working environment can be autonomously learned, the advancing speed of the potato picking machine and the rotating speed of the conveying chain are judged and calculated according to the thickness of the potato flow in front of the picking machine and the number of potatoes to be picked, and the harvesting efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of potato harvesting, and in particular to an adaptive control method of a potato picking harvester based on deep learning. Background Art

[0002] The planting area of ​​potatoes is increasing as their application value is gradually explored, but large-scale planting makes potato harvesting a problem. Compared with potato combine harvesters, potato pickers are more adaptable and flexible. However, potato pickers still have the problem of high potato damage rate, so potato pickers are generally used to harvest starch potatoes, and commercial potatoes on the market are still picked up manually. Traditional manual harvesting is time-consuming and labor-intensive, with low work efficiency and high labor intensity, but the level of intelligence of mechanized picking and harvesting is low, and it is impossible to perform adaptive operations according to different working environments.

[0003] In the actual harvesting environment, when the moving speed of the potato picking harvester increases while the speed of the potato picking harvester conveyor chain remains unchanged, potatoes will accumulate, causing the working efficiency of the potato picking harvester to decrease; when the moving speed of the potato picking harvester slows down while the speed of the potato picking harvester conveyor chain remains unchanged, it will cause a waste of power resources of the potato picking harvester. Moreover, the potato picking harvester needs to work in a complex and changeable working environment, which will be affected by a combination of factors such as soil conditions (clay and sandy loam), terrain conditions (plains, hills and mountains) and plot yield, resulting in different working environments for the potato picking harvester, which will in turn have a significant impact on the working efficiency of the potato picking harvester. However, there is no adaptive control method for potato picking harvesters at present, resulting in a large number of low working efficiency or waste of potato picking harvesters. Summary of the invention

[0004] The present invention aims to solve the above problems and provides an adaptive control method for a potato picking harvester based on deep learning. The technical solution adopted is as follows: A deep learning-based adaptive control method for a potato harvester comprises the following steps: S1. A depth camera is installed at the front of the harvester, and the depth camera periodically collects the surface image of the front of the harvester; S2. The area near the pile shovel of the harvester in the surface image is divided into the second region of interest, and the area in front of the pile shovel to be harvested is divided into the first region of interest; S3. Using a deep learning algorithm to identify the number of potatoes N0 in the first region of interest, and using a depth camera to identify the thickness of the potato flow in the second region of interest; S4. The adaptive system of the harvester collects the forward speed V0 and the conveyor chain speed V of the harvester in the same cycle as the depth cameraR0 and the number of potatoes to be picked up, N0; S5. Identify whether the potato flow thickness is within a preset allowable variation range; S6. If the potato flow thickness is within the preset allowable range, maintain the current forward speed V0 and conveyor chain speed V R0 If the number of potatoes N0 is not within the preset allowable range, the forward speed V0 and the conveyor chain speed V R0 The first BP neural network calculates and outputs the forward speed V1 and the conveyor chain speed V at the next moment. R1 ; S7. After the adjustment is completed, the depth camera collects the potato flow thickness of the next cycle and repeats steps S5 and S6.

[0005] Based on the above scheme, the deep learning algorithm described in step S2 is the PD-net algorithm, including a backbone extraction network and a detection head network. The backbone extraction network includes two branches. The first branch uses a 3*3 convolutional layer to extract conventional features, and the second branch uses a full-dimensional convolutional layer. The first branch and the second branch perform feature fusion at the fourth, sixth and eighth layers respectively, and then input them into the residual block for calculation and finally input them into the detection head.

[0006] Preferably, the forward speed V1 and the conveyor chain speed V output by the first BP neural network are R1 The signals are respectively output to the drive devices of the conveyor chain and the picker travel mechanism after passing through the PID controller.

[0007] Based on the above scheme, the conveyor chain speed V output by the first BP neural network R1 The first PID controller is input, and the system input, system output and system error of the first PID controller are input into the second BP neural network, and the second BP neural network calculates and outputs the adjusted integral adjustment coefficient, proportional adjustment coefficient and differential adjustment coefficient of the first PID controller; the forward speed V1 output by the first BP neural network is input into the second PID controller, and the system input, system output and system error of the second PID controller are input into the third BP neural network, and the third BP neural network calculates and outputs the adjusted integral adjustment coefficient, proportional adjustment coefficient and differential adjustment coefficient of the second PID controller.

[0008] Preferably, the number of potatoes to be picked up N0, the forward speed V0, and the conveyor chain speed V collected by the depth camera are R0 The outlier value is used to determine whether there is an error, and the forward speed V1 and the conveyor chain speed V output by the first BP neural network are R1 Determine whether there is an error through outliers.

[0009] Based on the above scheme, the initial data set of the adaptive system contains 200 sets of data samples, and the data set is randomly divided into training set and validation set in a ratio of 8:2; for the correct number of potatoes N0, forward speed V0, and conveyor chain speed V R0 , forward speed V1 and conveyor chain speed V R1 , recorded as a group of data samples. When the data sample records are full of 10 groups, the data set is updated in a circular covering manner; the updated data set is input into the first BP neural network for training, and the weight of the first BP neural network is updated.

[0010] On the basis of the above scheme, in the initial stage of adaptation, the allowable variation range of the preset potato flow thickness is increased to a times the rated allowable variation range. Each time the data set is updated, the allowable variation range of the potato flow thickness is reduced proportionally until the allowable variation range of the potato flow thickness is reduced to the rated allowable variation range after the data set is updated 20 times, and the initial data set is completely covered.

[0011] The beneficial effects of the present invention are: 1. A simple and practical potato recognition model PD-net is proposed. While meeting the recognition accuracy requirements, the model abandons the complex network structure, reduces the waste of computing resources, and effectively improves the real-time performance of the calculation; 2. A systematic adaptive control method is proposed based on deep learning algorithm, neural network model and PID controller. It can autonomously learn various key indicators in the working environment, and thus judge and calculate the forward speed of the potato picker and the speed of the conveyor chain according to the thickness of the potato flow in front of the picker and the number of potatoes to be picked up. The data set and neural network weights are continuously learned and updated, so that the control system can meet the actual needs of the harvesting environment and improve work quality and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 : Adaptive control flow chart of the present invention; Figure 2 : Schematic diagram of the structure of the picking harvester of the present invention; Figure 3 : Schematic diagram of the location area division of the region of interest of the present invention; Figure 4 : Network model diagram of the PD-net algorithm of the present invention; Figure 5 : The residual block calculation structure model diagram in the PD-net algorithm of the present invention; Figure 6 : The convolutional layer calculation structure model diagram in the PD-net algorithm of the present invention; Figure 7: The calculation structure model diagram of the residual bottleneck module in the PD-net algorithm of the present invention; Figure 8 : Detection head calculation structure model diagram in the PD-net algorithm of the present invention; Fig. 9 : The full-dimensional convolutional layer structure model diagram in the PD-net algorithm of the present invention; Fig.10 : The present invention adopts Yolo v11n algorithm to identify the experimental effect diagram; Fig.11 : The present invention adopts PD-net algorithm to identify the experimental effect diagram; Fig.12 : Schematic diagram of the adaptive system of the present invention; Fig.13 : The first neural network structure diagram of the present invention; Fig.14 : The second and third neural network structure diagram of the present invention.

[0013] Explanation of the reference numerals: 1-frame, 2-travel mechanism, 3-pile shovel, 4-primary conveyor chain, 5-secondary conveyor chain, 6-depth camera. DETAILED DESCRIPTION

[0014] The present invention will be further described below in conjunction with the accompanying drawings and embodiments: In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be directly connected or indirectly connected through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0015] In the description of the present invention, it should be understood that the terms "center", "length", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first" and "second" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0016] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may include that the first and second features are in direct contact, or may include that the first and second features are not in direct contact but are in contact through another feature between them. Moreover, a first feature being "above", "above" and "above" a second feature includes that the first feature is directly above and obliquely above the second feature, or simply indicates that the first feature is higher in level than the second feature. A first feature being "below", "below" and "below" a second feature includes that the first feature is directly below and obliquely below the second feature, or simply indicates that the first feature is lower in level than the second feature.

[0017] One of the implementation structures of the picking harvester adopted in this scheme is as follows Figure 2 As shown, it includes a frame 1, which is moved by a traveling mechanism 2. A linked primary conveyor chain 4 and a secondary conveyor chain 5 are arranged on the frame 1. A pile shovel 3 is arranged at the bottom front end of the frame 1. The pile shovel 3 scoops up potatoes on the surface and piles them to the primary conveyor chain 4. The primary conveyor chain 4 transfers the potatoes to the secondary conveyor chain 5 and performs subsequent harvesting operations. There is a synergistic relationship between the conveying speeds of the primary conveyor chain 4 and the secondary conveyor chain 5. The speed of the secondary conveyor chain 5 can be adjusted accordingly according to the speed of the primary conveyor chain 4. A depth camera 6 is arranged at the front of the frame 1.

[0018] like Figure 1 As shown, a deep learning-based adaptive control method for a potato harvester comprises the following steps: S1. The depth camera 6 periodically collects the surface image of the front of the harvester, and the period may be 1s; S2. The area near the harvester shovel 3 in the surface image is divided into the second area of ​​interest, and the area to be harvested in front of the shovel 3 is divided into the first area of ​​interest, such as Figure 3 As shown; S3. Using a deep learning algorithm to identify the number of potatoes N0 in the first region of interest, and identifying the thickness of the potato flow in the second region of interest through a depth camera 6; S4. The adaptive system of the harvester collects the forward speed V0 and the conveyor chain speed V of the harvester in the same cycle as the depth camera 6. R0 and the number of potatoes to be picked up, N0; S5. Identify whether the potato flow thickness is within a preset allowable variation range; S6. If the potato flow thickness is within the preset allowable range, maintain the current forward speed V0 and conveyor chain speed V R0 If the number of potatoes N0 is not within the preset allowable range, the forward speed V0 and the conveyor chain speed V R0The first BP neural network calculates and outputs the forward speed V1 and the conveyor chain speed V at the next moment. R1 ; S7. After the adjustment is completed, the depth camera 6 collects the potato flow thickness of the next cycle, and repeats steps S5 and S6, thereby completing the adaptive adjustment of the working efficiency of the potato picking harvester.

[0019] like Figures 4 to 9 As shown, the deep learning algorithm described in step S2 is the PD-net algorithm. The PD-net algorithm is based on yolo v11n, but abandons the complex network structure of the yolo v11n algorithm. The PD-net algorithm includes a backbone extraction network and a detection head network. The backbone extraction network includes two branches. The first branch uses a 3*3 convolutional layer to extract conventional features, and the second branch uses a full-dimensional convolutional layer. The first branch and the second branch perform feature fusion in the fourth, sixth and eighth layers respectively, and then input into the residual block for calculation, and finally input into the detection head.

[0020] In traditional convolutional neural networks, each convolution layer usually uses a fixed, static convolution kernel to extract features. In contrast, the full-dimensional convolution layer introduces a dynamic, multi-dimensional attention mechanism, which comprehensively improves the design of the convolution kernel. The principles are as follows: 1. Multi-dimensional dynamic attention mechanism: The core innovation of the full-dimensional convolution layer is its multi-dimensional dynamic attention mechanism. Traditional dynamic convolution usually only achieves dynamicity in one dimension, the number of convolution kernels, that is, by weighted combination of multiple convolution kernels to adapt to different input features. The full-dimensional convolution layer further expands this concept. It not only dynamically adjusts the number of convolution kernels, but also involves the other three dimensions of the convolution kernel: spatial size, number of input channels, and number of output channels. This means that the full-dimensional convolution layer can adapt to the characteristics of the input data more finely, thereby improving the effect of feature extraction; 2. Parallel strategy: The full-dimensional convolution layer adopts a parallel strategy to learn attention in different dimensions at the same time. This strategy allows the network to be more efficient in processing the features of each dimension while ensuring the complementarity and synergy between the dimensions.

[0021] The running results of the PD-net algorithm and the yolo v11n algorithm were experimentally compared in the following environment: running environment: Intel (R) Core (TM) i5-9300H, 2.4GHz processor, 8GB running memory, 64GB storage memory, Nvidia Geforce GTX 1650 GPU (4GB memory), the environment used is Pytorch1.8.1, Cuda10.1, Python3.9.

[0022] The results of the comparative test are as follows Fig.10 , Fig.11 As shown in Table 1.

[0023] Table 1 Comparison test data of yolo v11n algorithm and PD-net algorithm

[0024] By comparison, we can see that Yolo v11n is 0.008 and 0.003 higher than PD-net in terms of precision and recall, respectively, with small differences. In terms of floating-point operations, PD-net is 1.8GFLOPs lower than Yolo v11n, and PD-net shows a smaller amount of operations. In terms of model size, PD-net is 3458KB higher than Yolo v11n, indicating that PD-net's storage requirements are greater than Yolo v11n. In terms of frame rate, PD-net is 13f / s higher than Yolo v11n, showing higher real-time performance. The data shows that the PD-net model has less operations, better real-time performance, and faster running speed.

[0025] like Fig.12 As shown, the adaptive system included in this scheme includes a learning system and a control system. The learning system includes training the first BP neural network to fit the functional relationship of the key indicators of the current working environment. The control system is used to regulate the forward speed V1 and the conveyor chain speed V1 of each output. R1 .

[0026] The learning system also includes the judgment of data, the number of potatoes to be picked up N0 and the forward speed V0 and the conveyor chain speed V collected by the depth camera 6 R0 The outlier value is used to determine whether there is an error, and the forward speed V1 and the conveyor chain speed V output by the first BP neural network are R1 Determine whether there is an error through outliers.

[0027] The initial data set of the learning system in the adaptive system contains 200 sets of data samples, and the data set is randomly divided into a training set and a validation set in a ratio of 8:2; for the correct number of potatoes N0, forward speed V0, and conveyor chain speed V R0 , forward speed V1 and conveyor chain speed V R1 , recorded as a group of data samples. When the data sample records are full of 10 groups, the data set is updated in a circular covering manner; the updated data set is input into the first BP neural network for training, and the weight of the first BP neural network is updated.

[0028] In the initial stage of adaptation and even in the early stage, there may be too few successful adjustment samples, which cannot trigger the conditions for updating the data set. Therefore, in the initial stage of adaptation, the allowable variation range of the preset potato flow thickness is increased to a times the rated allowable variation range to generate more successful samples. Each time the data set is updated, the allowable variation range of the potato flow thickness is reduced proportionally until the allowable variation range of the potato flow thickness is reduced to the rated allowable variation range after the data set is updated 20 times, and the initial data set is completely covered.

[0029] In the control system part, the forward speed V1 and the conveyor chain speed V output by the first BP neural network are R1 After being output to the driving device of the conveyor chain and the picker travel mechanism 2 by the PID controller, specifically, the conveyor chain speed V R1 After being input into the first PID controller, the first PID controller calculates and outputs the result, and controls the conveyor chain hydraulic proportional reversing valve to regulate the conveyor chain hydraulic motor, thereby adjusting the running speed of the conveyor chain; after the forward speed V1 is input into the second PID controller, the second PID controller calculates and outputs the result, and controls the picking harvester hydraulic proportional reversing valve to regulate the driving hydraulic motor, thereby adjusting the travel speed of the travel mechanism 2.

[0030] In order to continuously update and optimize the parameters of the PID controller, the second BP neural network and the third BP neural network are introduced. The conveyor chain speed V output by the first BP neural network is R1 Input the first PID controller, the system input r(k) and system output y out (k) and the system error e(k) are input into the second BP neural network, and the integral adjustment coefficient, proportional adjustment coefficient and differential adjustment coefficient of the adjusted first PID controller are calculated and output by the second BP neural network; the forward speed V1 output by the first BP neural network is input into the second PID controller, and the system input r(k), system output y out The integral adjustment coefficient, proportional adjustment coefficient and differential adjustment coefficient of the adjusted second PID controller are calculated and output by the third BP neural network.

[0031] The structure diagram of the first BP neural network is as follows Fig.13 As shown; the structure diagram of the second BP neural network and the third BP neural network is as shown Fig.14 As shown, the two have the same structure but may have different weights.

[0032] The present invention is described above by way of examples, but the present invention is not limited to the above specific embodiments, and any changes or modifications made based on the present invention belong to the scope of protection claimed by the present invention.

Claims

1. A deep learning-based adaptive control method for a potato harvester, characterized in that: The following steps are involved: S1. A depth camera (6) is installed at the front of the harvester, and the depth camera (6) periodically collects surface images of the front of the harvester; S2. The area near the pile shovel (3) of the picking harvester in the surface image is divided into the second area of ​​interest, and the area to be harvested in front of the pile shovel (3) is divided into the first area of ​​interest; S3. Using a deep learning algorithm to identify the number of potatoes N0 in the first region of interest, and using a depth camera (6) to identify the thickness of the potato flow in the second region of interest; S4. The adaptive system of the harvester and the depth camera (6) collect the forward speed V0 and the conveyor chain speed V of the harvester in the same cycle R0 and the number of potatoes to be picked up, N0; S5. Identify whether the potato flow thickness is within a preset allowable variation range; S6. If the potato flow thickness is within the preset allowable range, maintain the current forward speed V0 and conveyor chain speed V R0 If the number of potatoes N0 is not within the preset allowable range, the forward speed V0 and the conveyor chain speed V R0 The first BP neural network calculates and outputs the forward speed V1 and the conveyor chain speed V at the next moment. R1 ; S7. After the adjustment is completed, the depth camera (6) collects the potato flow thickness of the next cycle and repeats steps S5 and S6.

2. The method for adaptively controlling a potato harvester based on deep learning according to claim 1, characterized in that: The deep learning algorithm described in step S2 is the PD-net algorithm, including a backbone extraction network and a detection head network. The backbone extraction network includes two branches. The first branch uses a 3*3 convolutional layer to extract conventional features, and the second branch uses a full-dimensional convolutional layer. The first branch and the second branch perform feature fusion at the fourth, sixth and eighth layers respectively, and then input them into the residual block for calculation and finally input them into the detection head.

3. The method for adaptively controlling a potato harvester based on deep learning according to claim 1, characterized in that: The forward speed V1 and the conveyor chain speed V output by the first BP neural network R1 The signals are respectively outputted to the drive devices of the conveyor chain and the picker travel mechanism (2) after passing through the PID controller.

4. The method for adaptively controlling a potato harvester based on deep learning according to claim 3, characterized in that: The conveyor chain speed V output by the first BP neural network R1 The first PID controller is input, and the system input, system output and system error of the first PID controller are input into the second BP neural network, and the second BP neural network calculates and outputs the adjusted integral adjustment coefficient, proportional adjustment coefficient and differential adjustment coefficient of the first PID controller; the forward speed V1 output by the first BP neural network is input into the second PID controller, and the system input, system output and system error of the second PID controller are input into the third BP neural network, and the third BP neural network calculates and outputs the adjusted integral adjustment coefficient, proportional adjustment coefficient and differential adjustment coefficient of the second PID controller.

5. The method for adaptively controlling a potato harvester based on deep learning according to claim 1, characterized in that: The number of potatoes to be picked up N0, the forward speed V0, and the conveyor chain speed V collected by the depth camera (6) are R0 The outlier value is used to determine whether there is an error, and the forward speed V1 and the conveyor chain speed V output by the first BP neural network are R1 Determine whether there is an error through outliers.

6. The method for adaptively controlling a potato harvester based on deep learning according to claim 5, characterized in that: The initial data set of the adaptive system contains 200 sets of data samples, which are randomly divided into training set and validation set in a ratio of 8:

2. R0 , forward speed V1 and conveyor chain speed V R1 , recorded as a group of data samples. When the data sample records are full of 10 groups, the data set is updated in a circular covering manner; the updated data set is input into the first BP neural network for training, and the weight of the first BP neural network is updated.

7. The method for adaptively controlling a potato harvester based on deep learning according to claim 6, characterized in that: In the initial stage of adaptation, the preset allowable variation range of potato flow thickness is increased to a times the rated allowable variation range. Each time the data set is updated, the allowable variation range of potato flow thickness is reduced proportionally until the allowable variation range of potato flow thickness is reduced to the rated allowable variation range after the data set is updated 20 times, and the initial data set is completely covered.

Images