Harvester-grain transport vehicle dual-mode switching collaborative grain unloading system and method based on harvester unloading port identification and tracking

By introducing unloading port identification and tracking technology into the harvester-grain transport vehicle system, and utilizing binocular cameras and YOLOv8 and Kernel Correlation Filter algorithms, dual-mode switching and coordinated unloading of the harvester and grain transport vehicle are achieved. This solves the dependence on positioning accuracy in existing technologies, improves operational accuracy and efficiency, and reduces energy consumption.

CN118584953BActive Publication Date: 2024-11-22NANJING AGRICULTURAL UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410624883.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-05-20
Publication Date
2024-11-22
Estimated Expiration
2044-05-20

AI Technical Summary

Technical Problem

The existing harvester-grain truck collaborative unloading system relies heavily on positioning accuracy, resulting in insufficient operational reliability and efficiency. Furthermore, the traditional model is inadequate in terms of flexibility and energy consumption.

Method used

A vision-based unloading port identification and tracking technology is adopted. By using a binocular camera and embedded device to identify the unloading port of the harvester, and combining YOLOv8 and Kernel Correlation Filter algorithms, dual-mode switching and coordinated unloading of the harvester and the grain transport vehicle are realized, reducing the dependence on positioning accuracy and improving operation accuracy and efficiency.

Benefits of technology

It improves the operational precision and efficiency of harvester-grain truck coordinated unloading, reduces energy consumption, minimizes grain loss and operational errors, and enhances the stability and flexibility of the system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118584953B_ABST
    Figure CN118584953B_ABST
Patent Text Reader

Abstract

The present invention provides a harvester-grain transport vehicle dual-mode switching collaborative grain unloading system and method based on harvester grain unloading port identification and tracking. First, the harvester automatic driving system sends a dispatch instruction to the grain transport vehicle automatic driving system according to the predicted time when the harvester grain bin is full and the time taken by the grain transport vehicle to reach the current position; after the grain transport vehicle moves to the position, the grain unloading port real-time identification and tracking system integrates the improved YOLOv8 algorithm and the KCF kernel correlation filter algorithm to locate and track the grain unloading port, and sends the relative position relationship data to the grain transport vehicle automatic driving system. The grain transport vehicle automatic driving system analyzes and calculates the optimal position relationship between the harvester grain unloading port and the grain transport vehicle grain bin, and uses this as the optimization target to control the grain transport vehicle and the harvester to work collaboratively. The present invention helps to reduce manual participation, reduce the labor intensity of grain unloading operations, improve harvesting operation efficiency, and solve the problems of high energy consumption, low overall efficiency, and heavy reliance on the positioning accuracy of the harvester and grain transport vehicle in the collaborative grain unloading process.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned agricultural machinery, and in particular relates to a harvester-grain transport vehicle collaborative grain unloading system and method based on harvester grain unloading port identification and tracking. Background Art

[0002] As unmanned farms transform from concept to reality, the development of combined harvesting operations towards intelligence and unmanned operation is an inevitable trend in the development of harvesting operations. Unmanned combined harvesting operations integrate a complete set of intelligent systems into traditional combined harvesting machinery, which can realize unmanned autonomous operations and achieve precision in the production process of harvesting operations, thereby more effectively improving production efficiency; harvesting is the final link of unmanned operations, and unloading grain is an important link to ensure that crop grains are stored in the warehouse.

[0003] In recent years, agricultural machinery unmanned technology has made important progress, and the coordinated unloading of grain by harvesters and grain transporters is also moving towards unmanned operation. At present, the harvesting and unloading process mainly relies on the traditional unloading mode, but there have also been some explorations of coordinated unloading. At present, the research on coordinated unloading mainly focuses on the fixed-point unloading mode and the master-slave real-time collaborative mode of harvesting and unloading.

[0004] The fixed-point unloading mode is that the harvester plans a fixed unloading point in advance, and the unmanned grain transport vehicle arrives at the fixed unloading point in advance according to the established route planning to wait for the unmanned harvester; after the unmanned harvester arrives at the fixed unloading point, it opens the unloading barrel, aligns the unloading port with the grain transport vehicle's granary, and starts unloading. After unloading, the harvester continues to harvest, and the grain transport vehicle transfers the grain to the grain storage point. This mode is conducive to improving the efficiency of unloading, but this mode requires setting up unloading points at fixed locations, which limits the flexibility of operation. If the harvester and grain transport vehicle need to operate at different locations, it is necessary to set up unloading points multiple times, which increases the operating cost and time cost; in the fixed-point unloading mode, the harvester and grain transport vehicle need to wait at a fixed location, which increases the non-operating time and affects the overall efficiency.

[0005] The traditional mode of harvesting and unloading grain is that the grain truck always maintains a relative position with the harvester in operation. When the grain truck's granary is full, the grain truck transfers the grain to the grain storage point, then looks for the harvester to continue harvesting and unloading. The advantage of this mode is that the harvester can harvest without stopping, which improves the operation efficiency and reduces the non-operating time of the harvester. However, in this mode, the harvester and the grain truck always maintain a relative position relationship throughout the entire operation process, and the grain truck is always working, which greatly increases energy consumption. Moreover, the existing harvester-grain truck collaborative unloading is only based on geographic location information and relative location information, and is heavily dependent on the positioning accuracy of the harvester and the grain truck. However, the positioning system may be affected by factors such as signal interference, weather or equipment failure, resulting in inaccurate positioning, which in turn leads to insufficient reliability and stability of collaborative operations.

[0006] Therefore, how to improve the unmanned level of grain unloading operations, solve the problem that the existing collaborative unloading operations of harvesters and grain transporters are heavily dependent on the positioning accuracy of harvesters and grain transporters, and improve the robustness and efficiency of the harvester-grain transporter collaborative unloading system are issues that need to be urgently addressed in unmanned harvesting operations. Summary of the invention

[0007] In view of the deficiencies in the prior art, the present invention provides a harvester-grain transport vehicle dual-mode switching collaborative grain unloading system and method based on harvester grain unloading port identification and tracking, including dual modes of fixed-point unloading and harvesting and unloading that can be switched at any time. When the grain transport vehicle arrives at the position specified by the harvester, the grain unloading port identification system located on the grain transport vehicle identifies the grain unloading port of the harvester and starts a master-slave real-time collaborative mode of harvesting and unloading based on a visual solution. This collaborative grain unloading process reduces manual participation, reduces the labor intensity of the grain unloading process, improves the harvesting operation efficiency, and solves the problems of high energy consumption, low overall efficiency, and heavy dependence on the positioning accuracy of the harvester and the grain transport vehicle in the collaborative grain unloading process. It has higher operating accuracy and can effectively avoid the problem of grain loss during the unloading process.

[0008] The present invention achieves the above technical objectives through the following technical means.

[0009] A harvester-grain transport vehicle dual-mode switching collaborative grain unloading method based on harvester grain unloading port identification and tracking includes the following processes:

[0010] Step 1: The harvester's automatic driving system plans the harvesting route, the harvester starts harvesting, and the grain transporter waits for dispatching instructions at a preset fixed position in the field;

[0011] Step 2: Based on the geographic location information of the harvester and the grain truck, the harvester automatic driving system sends a dispatch instruction to the grain truck automatic driving system through the harvester-grain truck communication system according to the predicted time when the harvester's grain bin is full and the time it takes for the grain truck to reach the harvester's current location;

[0012] Step 3: The grain truck automatic driving system plans the optimal path according to the target position information contained in the dispatching instruction, controls the grain truck to drive to the position specified by the harvester, and then switches to the master-slave real-time collaborative mode of harvesting and unloading grain based on the visual solution; the grain unloading port real-time identification and tracking system uses image acquisition hardware to identify the position information and depth information of the harvester's grain unloading port, and uses embedded devices to integrate relevant algorithms to achieve target detection, positioning, and tracking of the grain unloading port, obtain the relative position relationship between the grain unloading port and the image acquisition hardware, and transmit it to the grain truck automatic driving system;

[0013] The image acquisition hardware is installed on the grain truck. Based on the positional relationship between the image acquisition hardware and the grain truck's granary, the grain truck's automatic driving system obtains the positional relationship between the harvester's grain unloading port and the grain truck's granary through coordinate transformation. Then, according to the structural dimensions of the harvester and the grain truck, the optimal positional relationship between the harvester's grain unloading port and the grain truck's granary is determined, and this is used as the optimization target to control the coordinated grain unloading operation of the grain truck and the harvester.

[0014] Step 4: After completing the coordinated grain unloading, the grain transport truck returns to its original position and transfers the grain in the granary to the grain storage area.

[0015] Furthermore, in step 3, the specific process of target detection, positioning, and tracking at the grain transport vehicle unloading port in the master-slave real-time collaborative mode of harvesting and unloading grain based on the visual solution is as follows:

[0016] S3.1: After the grain truck arrives at the location designated by the harvester, the automatic driving system of the grain truck controls the grain truck to switch from the fixed-point unloading mode to the master-slave real-time collaborative mode of harvesting and unloading based on the vision solution;

[0017] S3.2: The real-time identification and tracking system of the grain unloading port uses a binocular camera and embedded devices to identify the grain unloading port of the harvester;

[0018] S3.2.1: Dataset establishment: Establish a harvester unloading port dataset for training the YOLOv8 model;

[0019] S3.2.2: Identify algorithm design;

[0020] Based on the YOLOv8 model, the recognition model training of the harvester unloading port is carried out. In the YOLOv8 model, a lightweight depth-separable convolution module is used to replace the 3x3 convolution layer. In addition, the number of convolution blocks in the Darknet-53 network in the YOLOv8 model is reduced, and the number of layers and channels of the network is reduced to reduce the complexity of the model. The embedded device uses the improved YOLOv8 model to recognize the image information of the harvester unloading port collected by the binocular camera.

[0021] S3.3: The embedded device obtains the position information of the harvester's grain unloading port based on the depth information provided by the binocular camera, and integrates the YOLOv8 algorithm and the Kernel Correlation Filter kernel correlation filter algorithm to achieve target positioning and tracking of the harvester's grain unloading port. The grain truck's automatic driving system then controls the grain truck and the harvester to coordinate the grain unloading operation.

[0022] Furthermore, the specific process of S3.3 is as follows:

[0023] S3.3.1: Perform binocular camera calibration;

[0024] S3.3.2: Target tracking algorithm design;

[0025] The kernel correlation filter algorithm is used to transform the harvester unloading port tracking problem into a classification problem, that is, to classify the foreground target and the background, find the position of the object in the current frame, and predict the position of the object in the next frame.

[0026] The target area is sampled cyclically and a sample set is obtained. The objective function is established by solving the ridge regression problem:

[0027] f(x i )=w T x i (1)

[0028] minw∑i(f(xi)-yi) 2 +λw 2 (2)

[0029] Formula (1) is a linear classifier, where f(x i ) is the regression function, w is the column vector representing the weight coefficient, x i is the sample, T represents the transposition; Formula (2) is the ridge regression equation, where y i is the sample label, λ is the regularization coefficient, which is used to control the structural complexity of the system;

[0030] By introducing a nonlinear mapping function, the low-dimensional nonlinear inseparable problem is mapped to a high-dimensional space, thereby transforming it into a linear solution, as shown in formula (3). Substituting formula (3) into formula (2) in the kernel space and performing discrete Fourier transform, as shown in formula (4):

[0031]

[0032]

[0033] In the formula, is a nonlinear mapping function, α is the transformation form of the solution to the ridge regression problem, is the discrete Fourier transform of vector α, α i Indicates the weight coefficient of sample i mapped from low dimension to high dimension, is the discrete Fourier transform of y, where y represents the sample regression value, is the first row element of the kernel function matrix;

[0034] The tracking result is obtained by performing correlation detection on the input image. The sample response is:

[0035]

[0036] Where z is the sample matrix, is the response value, and its maximum position is the position with the strongest correlation, which is the predicted target position;

[0037] S3.3.3: The real-time identification and tracking system for the grain unloading port obtains the relative position relationship between the harvester's grain unloading port and the binocular camera based on the tracking results, and transmits it to the automatic driving system of the grain transport vehicle. After the automatic driving system of the grain transport vehicle analyzes and processes the received data, it controls the grain transport vehicle and the harvester to coordinate the grain unloading operation.

[0038] A system for realizing the above-mentioned harvester-grain transport vehicle dual-mode switching collaborative grain unloading method based on harvester grain unloading port identification and tracking, comprising a grain unloading port real-time identification and tracking system, a harvester automatic driving system, a grain transport vehicle automatic driving system, and a harvester-grain transport vehicle communication system;

[0039] The real-time identification and tracking system for grain unloading ports includes image acquisition hardware, embedded devices, and communication modules installed on the grain transport vehicle, which are used to identify, locate, and track the grain unloading port of the harvester, calculate the relative position relationship between the grain unloading port of the harvester and the image acquisition hardware, and transmit it to the automatic driving system of the grain transport vehicle;

[0040] The harvester automatic driving system and the grain transporter automatic driving system are located in the harvester cab and on the grain transporter, respectively, and are used to realize the autonomous navigation and motion control of the harvester and the grain transporter, respectively. Both systems include an electric steering wheel, a navigation controller, an angle sensor, an on-board display terminal, a positioning board, a differential reference station, and a positioning antenna. The harvester automatic driving system also includes a grain bin internal status monitor.

[0041] The harvester-grain transporter communication system is used to realize the dispatch information communication between the harvester and the grain transporter, and complete the coordinated dispatch of the harvester and the grain transporter, including a harvester communication module and a grain transporter communication module. The harvester communication module and the grain transporter communication module are connected through the data transmission unit signal.

[0042] Furthermore, the binocular camera is used to obtain position information and depth information of the harvester's grain unloading port; the embedded device runs the YOLOv8 algorithm to perform target detection and position calculation based on the information obtained by the binocular camera, and realizes real-time tracking of the grain unloading port by fusing the YOLOv8 algorithm and the Kernel Correlation Filter kernel correlation filter algorithm, and obtains the relative position relationship between the harvester's grain unloading port and the binocular camera.

[0043] Furthermore, the on-board display terminal on the harvester uses the positioning board, differential reference station, and positioning antenna to obtain the positioning information of the harvester, and carries out path planning and unloading point prediction according to the field conditions; the electric steering wheel on the harvester maintains the path tracking accuracy according to the instructions of the navigation controller, and obtains steering information through the angle sensor to form a navigation closed loop to realize automatic navigation of the harvester; the internal status monitor of the grain silo is used to detect the loading status of the harvester grain silo and feed back to the on-board display terminal on the harvester, and the on-board display terminal predicts the filling time of the grain silo and the time taken by the grain truck to reach the unloading point, and accordingly sends a dispatch instruction to the grain truck to be dispatched through the harvester-grain truck communication system.

[0044] Furthermore, the on-board display terminal on the grain transport vehicle utilizes the positioning board, the differential reference station and the positioning antenna to obtain the positioning information of the grain transport vehicle, and plans the optimal path according to the target position information contained in the dispatching instruction sent by the harvester automatic driving system, and controls the grain transport vehicle to reach the target position according to the optimal path; at the same time, the on-board display terminal on the grain transport vehicle is also used to realize the switching between the fixed-point unloading mode and the master-slave real-time collaborative mode of harvesting and unloading grain based on the visual solution. In the master-slave real-time collaborative mode of harvesting and unloading grain based on the visual solution, the grain transport vehicle and the harvester are controlled to coordinate the unloading operation according to the relative position relationship between the unloading port of the harvester and the grain transport vehicle obtained by the real-time identification and tracking system of the unloading port.

[0045] The present invention has the following beneficial effects:

[0046] (1) For the first time, a method for collaborative unloading of grain between a harvester and a grain transporter based on identification and tracking of the harvester unloading port was proposed. By identifying and tracking the harvester unloading port, the operation accuracy of unmanned collaborative unloading of grain between the harvester and the grain transporter was significantly improved.

[0047] (2) Based on the YOLOV8 algorithm, the grain unloading port can be quickly identified, the target position can be determined more accurately, the tracking accuracy and stability can be improved, the target can be lost, the waiting time can be reduced, and the efficiency of collaborative work can be improved; the YOLOV8 algorithm is designed to be lightweight, the computational complexity and memory usage of the algorithm can be reduced, and the grain unloading port identification system can be ensured to run stably for a long time, so as to realize a mobile grain unloading port identification system, improve the stability, reliability, practicality and flexibility of the system, and make it more suitable for running on embedded devices.

[0048] (3) The improved KCF algorithm can capture the target position and motion trajectory more accurately, improve the accuracy and stability of tracking, and provide a more reliable solution for real-time target tracking; the improved KCF algorithm can realize the target positioning and tracking of the grain transport vehicle to the grain harvester unloading port based on the depth information provided by the binocular camera; the present invention combines the traditional target tracking algorithm with the depth perception technology to achieve a higher level of target tracking performance. This innovative method provides new ideas and methods for the field of machine vision, expands the application scope of target tracking technology, and is of great significance for promoting the development of intelligent agriculture, automated harvesting and other fields.

[0049] (4) It has high operational safety. The present invention uses a lightweight YOLOV8 algorithm to make the grain unloading process between the harvester and the grain truck more efficient and smooth, saving the tracking time of the grain unloading port and reducing the possibility of operational errors, thereby reducing the risk of accidents. The improved KCF algorithm can improve the motion control accuracy of the grain truck, thereby ensuring the safety of unmanned collaborative grain unloading operations. By tracking the target, it can avoid collisions between the grain truck and other objects, thereby reducing accidents during the operation.

[0050] (4) It has high operating efficiency. The grain transport vehicle of the present invention does not move synchronously with the harvester throughout the entire operation process. Instead, it starts to identify and track the grain unloading port after the harvester predicts the time when the granary is full and the time it takes for the grain transport vehicle to reach the current position and sends a scheduling instruction. It cooperates with the harvester to achieve collaborative grain unloading operations, which can effectively save energy, reduce the labor intensity of the grain unloading process, and solve the problems of high energy consumption and low overall efficiency in the traditional collaborative grain unloading process. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a flow chart of the harvester-grain transporter dual-mode switching collaborative grain unloading method based on harvester grain unloading port identification and tracking;

[0052] Figure 2 This is a binocular camera imaging model diagram;

[0053] Figure 3 This is a schematic diagram of the harvester-grain transporter collaborative communication system. DETAILED DESCRIPTION

[0054] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments, but the scope of protection of the present invention is not limited thereto. It should be noted that: unless otherwise defined, the technical terms or scientific terms used herein should be the usual meanings understood by people with ordinary skills in the field to which the present invention belongs. In the description used in the patent application specification and claims of the present invention, it should be emphasized that, unless otherwise clearly specified and limited, words such as "one", "an" or "the" do not indicate a quantitative limitation, but indicate that there is at least one instance. Similarly, similar words such as "include" or "comprise" indicate the listed elements and possible other elements; elements or objects include the elements or objects listed after "include" or "comprise" and their equivalents, but do not exclude other elements or objects with the same function.

[0055] The harvester-grain transport vehicle dual-mode switching collaborative grain unloading system based on harvester grain unloading port identification and tracking described in the present invention includes a grain unloading port real-time identification and tracking system, a harvester automatic driving system, a grain transport vehicle automatic driving system, and a harvester-grain transport vehicle communication system.

[0056] The real-time identification and tracking system for grain unloading ports is installed on the grain transport vehicle, and is used to detect and identify the grain unloading ports, calculate the relative position relationship between the harvester and the grain transport vehicle, and transmit the position relationship information to the automatic driving system of the grain transport vehicle, and use the automatic driving system of the grain transport vehicle to adjust the posture of the grain transport vehicle, so that the harvester and the grain transport vehicle always maintain a suitable relative position for unloading grain. The real-time identification system for grain unloading ports includes image acquisition hardware installed on the grain transport vehicle (a binocular camera is selected in this embodiment), an embedded device, and a communication module. The binocular camera is used to obtain the position information and depth information of the harvester grain unloading port; the embedded device runs the YOLOv8 algorithm based on the information obtained by the binocular camera to perform target detection and position calculation, and realizes real-time tracking of the grain unloading port by fusing the YOLOv8 algorithm and the Kernel Correlation Filter kernel correlation filter algorithm, thereby obtaining the relative position relationship between the harvester grain unloading port and the binocular camera; the embedded device runs the ROS system and communicates with the automatic driving system of the grain transport vehicle through the communication module, and transmits the calculated position relationship information to the automatic driving system of the grain transport vehicle, so that the automatic driving system of the grain transport vehicle adjusts the posture of the grain transport vehicle. Through the synergy of the real-time identification system of the grain unloading port and the automatic driving system of the grain transporter, efficient collaborative unloading operations between the harvester and the grain transporter can be achieved, improving the accuracy and intelligence of unloading operations in the harvesting and unloading mode.

[0057] The harvester automatic driving system is installed in the harvester cab to realize the harvester's autonomous navigation and motion control. The harvester automatic driving system includes an electric steering wheel, a navigation controller, an angle sensor, an on-board display terminal, a positioning board, a differential reference station, a positioning antenna, and a grain silo internal status monitor. The on-board display terminal uses the positioning board, the differential reference station, and the positioning antenna to obtain the harvester's positioning information, and carries out path planning and grain unloading point prediction according to the field conditions; the electric steering wheel is responsible for maintaining the path tracking accuracy according to the instructions of the navigation controller, and obtains steering information through the angle sensor to form a navigation closed loop to realize the harvester's automatic navigation; the grain silo internal status monitor is responsible for detecting the grain silo loading status and feeding it back to the harvester automatic driving system. During unmanned harvesting, the harvester automatic driving system is responsible for completing the harvesting path planning and the internal status monitoring of the harvester's grain silo. According to the predicted grain silo filling time and the time taken for the grain truck to reach the harvester's predicted grain unloading position, the dispatching instructions are sent to the grain truck to be dispatched through the harvester-grain truck communication system.

[0058] The automatic driving system of the grain truck is installed on the grain truck to realize the autonomous navigation and motion control of the grain truck. The automatic driving system of the grain truck includes an electric steering wheel, a navigation controller, an angle sensor, an on-board display terminal, a positioning board, a differential reference station, and a positioning antenna. The on-board display terminal uses the positioning board, the differential reference station, and the positioning antenna to obtain the positioning information of the grain truck, and plans the optimal path according to the target position information contained in the dispatching instruction sent by the harvester automatic driving system; the electric steering wheel is responsible for maintaining the path tracking accuracy according to the navigation controller instruction, and obtains the steering information through the angle sensor to form a navigation closed loop to realize the automatic navigation of the grain truck. The automatic driving system of the grain truck plans the optimal path according to the target position information contained in the dispatching instruction sent by the harvester. After reaching the target position according to the optimal path, it switches to the master-slave real-time collaborative mode of harvesting and unloading grain based on the visual solution, and adjusts the position of the grain truck according to the relative position relationship between the unloading port and the grain truck obtained by the real-time identification and tracking system of the unloading port, ensuring that the grain truck and the harvester always maintain a suitable relative position for unloading grain, thereby realizing the harvester harvesting and unloading grain and completing the efficient transfer of grain.

[0059] The harvester-grain transporter communication system is used to realize the dispatch information communication between the harvester and the grain transporter, and complete the coordinated dispatch of the harvester and the grain transporter. It includes a harvester communication module installed on the harvester and a grain transporter communication module installed on the grain transporter. The harvester communication module and the grain transporter communication module are connected through 4G DTU. The harvester communication module and the grain transporter communication module both include components such as a data transmission unit (DTU), a processor, and an antenna, which are used to realize the sending and receiving of data, and to parse and process the received data. During the grain unloading operation, the harvester communication module is responsible for transmitting the dispatch instructions and the position information data of the harvester to the grain transporter communication module through the 4G DTU, so that the grain transporter automatic driving system can adjust the posture of the grain transporter to make it reach the specified position. At the same time, the grain transporter communication module is used to transmit the position information and status information of the grain transporter to the harvester communication module through the 4G DTU, so as to realize the adjustment and control of the grain unloading operation by the harvester automatic driving system. Through the synergy of the harvester-grain truck communication system, real-time information transmission and data exchange between the harvester and the grain truck can be achieved, thereby ensuring the accuracy and efficiency of grain unloading operations and improving the level of automation and intelligence in agricultural production.

[0060] Reference Figure 1 The harvester-grain transport vehicle dual-mode switching collaborative grain unloading method based on harvester grain unloading port identification and tracking of the present invention specifically includes the following steps:

[0061] S1: The harvester and the grain transporter arrive at the area to be harvested. The harvester plans the harvesting route and starts the harvesting operation. The grain transporter waits for dispatching instructions at a preset fixed position in the field.

[0062] S2: The harvester automatic driving system is based on the geographic location information of the harvester and the grain truck. According to the predicted time for the harvester's grain bin to be filled and the time it takes for the grain truck to reach the harvester's current location, the harvester-grain truck communication system sends a dispatch instruction to the grain truck automatic driving system of the grain truck to be dispatched.

[0063] S3: After receiving the dispatch instruction, the automatic driving system of the grain transporter plans the optimal path according to the target position information contained in the dispatch instruction, controls the grain transporter to drive to the position specified by the harvester according to the optimal path, and then switches to the master-slave real-time collaborative mode of harvesting and unloading grain based on the visual solution; the real-time identification and tracking system of the grain unloading port uses a binocular camera to identify the position information and depth information of the grain unloading port of the harvester, and then the embedded device uses the information obtained by the binocular camera to fuse the YOLOv8 algorithm and Kernel Correlation The kernel correlation filter algorithm is used to realize target detection, positioning and tracking of the grain unloading port, obtain the relative position relationship between the grain unloading port and the binocular camera and transmit it to the automatic driving system of the grain transport vehicle; since the binocular camera is installed on the grain transport vehicle, there is a certain position relationship between the binocular camera and the grain transport vehicle granary. The automatic driving system of the grain transport vehicle obtains the position relationship between the grain unloading port of the harvester and the grain transport vehicle granary through coordinate transformation, and determines the optimal position relationship between the grain unloading port of the harvester and the grain transport vehicle granary according to the structural dimensions of the harvester and the grain transport vehicle. With the optimal position relationship as the optimization goal, the grain transport vehicle and the harvester are controlled to work in coordination, realizing the operation mode of harvesting and unloading grain, so that the grain in the grain bin of the harvester is efficiently transferred to the grain bin of the grain transport vehicle with low loss, and the coordinated unloading of grain is completed; the specific control process is as follows:

[0064] S3.1: After the grain truck arrives at the location designated by the harvester, the automatic driving system of the grain truck controls the grain truck to switch from the fixed-point unloading mode to the master-slave real-time collaborative mode of harvesting and unloading based on the vision solution.

[0065] S3.2: When the grain transport vehicle arrives at the location designated by the harvester, the grain unloading port real-time identification and tracking system located on the grain transport vehicle identifies the grain unloading port of the harvester;

[0066] S3.2.1: Dataset establishment;

[0067] Establish a harvester unloading port dataset for training the YOLOv8 model;

[0068] S3.2.2: Identify algorithm design;

[0069] The recognition model training of the harvester grain unloading port is carried out based on the YOLOv8 model. In the YOLOv8 model, a large number of 3x3 convolutional layers are used, which makes the network relatively deep and has more parameters. In order to reduce the number of parameters and the amount of calculation and make the algorithm more suitable for deployment on the embedded development board, the YOLOv8 model is improved and the lightweight Depthwise SeparableConvolution module is used to replace the 3x3 convolutional layer. The improved lightweight YOLOv8 model can reduce the complexity of the model while maintaining performance; in addition, by reducing the number of convolution blocks in the Darknet-53 network in the YOLOv8 model, the number of layers and channels of the network are reduced, and the complexity of the model is further reduced; using the improved YOLOv8 model to recognize the image information of the harvester grain unloading port collected by the binocular camera can improve the detection effect of the YOLOv8 algorithm on small targets (grain unloading port) and achieve a balance between model lightweight and performance.

[0070] S3.3: The real-time identification and tracking system of the grain unloading port obtains the location information of the harvester's grain unloading port based on the depth information provided by the binocular camera, and integrates the YOLOv8 algorithm and the Kernel Correlation Filter kernel correlation filter algorithm to achieve target positioning and tracking of the harvester's grain unloading port;

[0071] S3.3.1: Reference Figure 2 The imaging model diagram of the binocular camera shown in the figure is used to calibrate the binocular camera;

[0072] Step 1: Prepare a chessboard, stick it on the wall, and use a ruler to measure the real physical length of the black and white squares; Step 2: Call the binocular camera and take a series of chessboard images from different angles; Step 3: Use the left eye image data set to calibrate the left eye camera, and obtain the left eye internal reference matrix K1 and the left eye distortion coefficient vector D1; Step 4: Use the right eye image data set to calibrate the right eye camera, and obtain the right eye internal reference matrix K2 and the right eye distortion coefficient vector D2; Step 5: Take the parameters K1, K2, D1, and D2 measured by the left and right eyes as input, and then use the chessboard images with one-to-one correspondence between the left and right eyes at the same time, call the stereoCalibrate function, and output the left and right eye rotation matrix R and translation vector T; Step 6: Based on the measured K1, K2, D1, D2, R, and T, perform binocular vision image correction; After the above steps 1-6, the binocular camera is calibrated, so that the binocular camera can obtain the position information of the harvester unloading port according to the depth information;

[0073] Figure 2 In, O L , O R is the center of the left and right apertures, b is the baseline, f is the focal length, P LFor the left image, P R For the right purpose, u L and u R are the coordinates of the imaging plane, and P represents any point in space.

[0074] S3.3.2: Target tracking algorithm design;

[0075] The discriminative target tracking algorithm (Kernel Correlation Filter) is used to transform the harvester unloading port tracking problem into a classification problem, that is, to classify the foreground target and the background, find the position of the object in the current frame, and predict the position of the object in the next frame.

[0076] The target area is sampled cyclically and a sample set is obtained. The objective function is established by solving the ridge regression problem:

[0077] f(x i )=w T x i (1)

[0078] min w ∑ i (f(x i )-y i ) 2 +λw 2 (2)

[0079] Formula (1) is a linear classifier, where f(x i ) is the regression function, w is the column vector representing the weight coefficient, x i is the sample, T represents the transposition; Formula (2) is the ridge regression equation, where y i is the sample label, λ is the regularization coefficient, which is used to control the structural complexity of the system;

[0080] By introducing a nonlinear mapping function, the low-dimensional nonlinear inseparable problem is mapped to a high-dimensional space, thereby transforming it into a linear solution, as shown in formula (3). Substituting formula (3) into formula (2) in the kernel space and performing discrete Fourier transform, as shown in formula (4):

[0081]

[0082]

[0083] In the formula, is a nonlinear mapping function, α is the transformation form of the solution to the ridge regression problem, is the discrete Fourier transform of vector α, α i Indicates the weight coefficient of sample i mapped from low dimension to high dimension, is the discrete Fourier transform of y, where y represents the sample regression value, is the first row element of the kernel function matrix;

[0084] Therefore, this filter can be used to perform correlation detection on the input image to obtain the tracking result. The sample response is:

[0085]

[0086] Where z is the sample matrix, is the response value, and its maximum position is the position with the strongest correlation, which is the predicted target position; Represents the elements corresponding to the samples and the sample matrix.

[0087] S3.3.3: The grain unloading port position detected by the YOLOv8 algorithm is used as the initial position, and the Kernel Correlation Filter (KCF) kernel correlation filter algorithm is used to locate and track the grain unloading port in real time. The KCF algorithm realizes the positioning of the target by tracking the feature information of the target between consecutive frames, and integrates the YOLOv8 algorithm and the Kernel Correlation Filter kernel correlation filter algorithm to realize the target positioning and tracking of the harvester grain unloading port. The fused algorithm can still keep tracking when the target moves or is blocked, ensuring the accuracy and stability of tracking. At this time, the grain unloading port real-time identification and tracking system has obtained the relative position relationship between the harvester grain unloading port and the binocular camera and transmitted it to the grain transport vehicle automatic driving system. Since the binocular camera is installed on the grain transport vehicle, there is a certain position relationship between the binocular camera and the grain transport vehicle granary. The grain transport vehicle automatic driving system can obtain the position relationship between the harvester grain unloading port and the grain transport vehicle granary through coordinate transformation, and then control the grain transport vehicle to keep the above position relationship, so that the grain in the harvester granary is efficiently and low-loss transferred to the grain transport vehicle granary, completing the coordinated grain unloading.

[0088] S4: After completing the coordinated grain unloading, the grain transport vehicle returns to its original position and transfers the grain in the grain bin of the grain transport vehicle to the grain storage area; at this point, a cycle of coordinated grain unloading is completed.

[0089] The embodiments are preferred implementations of the present invention, but the present invention is not limited to the above-mentioned implementations. Any obvious improvements, substitutions or modifications that can be made by those skilled in the art without departing from the essential content of the present invention belong to the protection scope of the present invention.

Claims

1. A harvester-grain transport vehicle dual-mode switching collaborative grain unloading method based on harvester grain unloading port identification and tracking, characterized in that: The process includes the following: Step 1: The harvester's automatic driving system plans the harvesting route, the harvester starts harvesting, and the grain transporter waits for dispatching instructions at a preset fixed position in the field; Step 2: Based on the geographic location information of the harvester and the grain truck, the harvester automatic driving system sends a dispatch instruction to the grain truck automatic driving system through the harvester-grain truck communication system according to the predicted time when the harvester's grain bin is full and the time it takes for the grain truck to reach the harvester's current location; Step 3: The grain truck’s automatic driving system plans the optimal path based on the target location information contained in the dispatching instruction, controls the grain truck to drive to the location specified by the harvester, and then switches to the master-slave real-time collaborative mode of harvesting and unloading grain based on the visual solution; The real-time identification and tracking system of the grain unloading port uses image acquisition hardware to identify the position and depth information of the harvester's grain unloading port, and uses embedded devices to integrate relevant algorithms to achieve target detection, positioning, and tracking of the harvester's grain unloading port, obtain the relative position relationship between the harvester's grain unloading port and the image acquisition hardware, and transmit it to the automatic driving system of the grain transport vehicle; The image acquisition hardware is installed on the grain truck. Based on the positional relationship between the image acquisition hardware and the grain truck's granary, the grain truck's automatic driving system obtains the positional relationship between the harvester's grain unloading port and the grain truck's granary through coordinate transformation. Then, according to the structural dimensions of the harvester and the grain truck, the optimal positional relationship between the harvester's grain unloading port and the grain truck's granary is determined, and this is used as the optimization target to control the coordinated grain unloading operation of the grain truck and the harvester. Step 4: After completing the coordinated unloading of grain, the grain transport vehicle returns to its original position and transfers the grain in the granary to the grain storage area; In step 3, the specific process of target detection, positioning, and tracking of the grain transport vehicle unloading port in the master-slave real-time collaborative mode of harvesting and unloading grain based on the visual solution is as follows: S3.1: After the grain truck arrives at the location designated by the harvester, the automatic driving system of the grain truck controls the grain truck to switch from the fixed-point unloading mode to the master-slave real-time collaborative mode of harvesting and unloading based on the vision solution; S3.2: The real-time identification and tracking system of the grain unloading port uses a binocular camera and embedded devices to identify the grain unloading port of the harvester; S3.2.1: Dataset establishment: Establish a harvester unloading port dataset for training the YOLOv8 model; S3.2.2: Identify algorithm design; Based on the YOLOv8 model, the recognition model training of the harvester unloading port was carried out. In the YOLOv8 model, the lightweight depth-separable convolution module was used to replace the 3x3 convolution layer. In addition, the number of convolution blocks in the Darknet-53 network in the YOLOv8 model was reduced, and the number of layers and channels of the network was reduced to reduce the complexity of the model. The embedded device uses the improved YOLOv8 model to recognize the image information of the harvester unloading port collected by the binocular camera; S3.3: The embedded device obtains the location information of the harvester's grain unloading port based on the depth information provided by the binocular camera, integrates the YOLOv8 algorithm and the Kernel Correlation Filter algorithm, and realizes the target positioning and tracking of the harvester's grain unloading port. Then the grain transport vehicle's automatic driving system controls the grain transport vehicle and the harvester to coordinate the grain unloading operation. The specific process of S3.3 is as follows: S3.3.1: Perform binocular camera calibration; S3.3.2: Target tracking algorithm design; The kernel correlation filter algorithm is used to transform the harvester unloading port tracking problem into a classification problem, that is, to classify the foreground target and the background, find the position of the object in the current frame, and predict the position of the object in the next frame. The target area is sampled cyclically and a sample set is obtained. The objective function is established by solving the ridge regression problem: f(x i )=w T x i (1) minwΣi(f(xi)-yi) 2 +λw 2 (2) Formula (1) is a linear classifier, where f(x i ) is the regression function, w is the column vector representing the weight coefficient, x i is the sample, T represents the transposition; Formula (2) is the ridge regression equation, where y i is the sample label, λ is the regularization coefficient, which is used to control the structural complexity of the system; By introducing a nonlinear mapping function, the low-dimensional nonlinear inseparable problem is mapped to a high-dimensional space, thereby transforming it into a linear solution, as shown in formula (3). Substituting formula (3) into formula (2) in the kernel space and performing discrete Fourier transform, as shown in formula (4): In the formula, is a nonlinear mapping function, α is the transformation form of the solution to the ridge regression problem, is the discrete Fourier transform of vector α, α i Indicates the weight coefficient of sample i mapped from low dimension to high dimension, is the discrete Fourier transform of y, y represents the sample regression value, is the first row element of the kernel function matrix; The tracking result is obtained by performing correlation detection on the input image. The sample response is: Where z is the sample matrix, is the response value, and its maximum position is the position with the strongest correlation, which is the predicted target position; Represents the elements corresponding to the samples and the sample matrix; S3.3.3: The real-time identification and tracking system for the grain unloading port obtains the relative position relationship between the harvester's grain unloading port and the binocular camera based on the tracking results, and transmits it to the automatic driving system of the grain transport vehicle. After the automatic driving system of the grain transport vehicle analyzes and processes the received data, it controls the grain transport vehicle and the harvester to coordinate the grain unloading operation.

2. A system for realizing the harvester-grain transport vehicle dual-mode switching collaborative grain unloading method based on harvester grain unloading port identification and tracking as described in claim 1, characterized in that: Including real-time identification and tracking system for grain unloading port, automatic driving system for harvester, automatic driving system for grain transporter, and communication system between harvester and grain transporter; The real-time identification and tracking system for grain unloading ports includes image acquisition hardware, embedded devices, and communication modules installed on the grain transport vehicle, which are used to identify, locate, and track the grain unloading port of the harvester, calculate the relative position relationship between the grain unloading port of the harvester and the image acquisition hardware, and transmit it to the automatic driving system of the grain transport vehicle; The harvester automatic driving system and the grain transporter automatic driving system are located in the harvester cab and on the grain transporter, respectively, and are used to realize the autonomous navigation and motion control of the harvester and the grain transporter, respectively. Both systems include an electric steering wheel, a navigation controller, an angle sensor, an on-board display terminal, a positioning board, a differential reference station, and a positioning antenna. The harvester automatic driving system also includes a grain bin internal status monitor. The harvester-grain transporter communication system is used to realize the dispatch information communication between the harvester and the grain transporter, and complete the coordinated dispatch of the harvester and the grain transporter, including a harvester communication module and a grain transporter communication module. The harvester communication module and the grain transporter communication module are connected through the data transmission unit signal.

3. The system according to claim 2, characterized in that The image acquisition hardware is a binocular camera, which is used to obtain the position information and depth information of the harvester's grain unloading port; The embedded device runs the YOLOv8 algorithm for target detection and position calculation based on the information obtained by the binocular camera. By integrating the YOLOv8 algorithm and the Kernel Correlation Filter algorithm, real-time tracking of the grain unloading port is achieved, and the relative position relationship between the harvester unloading port and the binocular camera is obtained.

4. The system according to claim 2, characterized in that The on-board display terminal on the harvester uses a positioning board, a differential reference station, and a positioning antenna to obtain the harvester's positioning information, and carries out path planning and unloading point prediction according to the field conditions; the electric steering wheel on the harvester maintains path tracking accuracy according to the instructions of the navigation controller, and obtains steering information through an angle sensor to form a navigation closed loop to achieve automatic navigation of the harvester; the grain silo internal status monitor is used to detect the loading status of the harvester's grain silo and feed it back to the on-board display terminal on the harvester, and the on-board display terminal predicts the time it takes for the grain silo to be filled and the time it takes for the grain truck to reach the unloading point, and accordingly sends a dispatch instruction to the grain truck to be dispatched through the harvester-grain truck communication system.

5. The system according to claim 2, characterized in that The on-board display terminal on the grain transport vehicle utilizes the positioning board, the differential reference station and the positioning antenna to obtain the positioning information of the grain transport vehicle, and plans the optimal path according to the target position information contained in the dispatching instruction sent by the automatic driving system of the harvester, and controls the grain transport vehicle to reach the target position according to the optimal path; at the same time, the on-board display terminal on the grain transport vehicle is also used to realize the switching between the fixed-point unloading mode and the master-slave real-time collaborative mode of harvesting and unloading grain based on the visual solution. In the master-slave real-time collaborative mode of harvesting and unloading grain based on the visual solution, the grain transport vehicle and the harvester are controlled to coordinate the unloading operation according to the relative position relationship between the unloading port of the harvester and the grain transport vehicle obtained by the real-time identification and tracking system of the unloading port.