Automatic pine nut picking method and system based on unmanned aerial vehicle and deep learning

Abstract

The invention relates to the technical field of automatic picking, in particular to an automatic pine nut picking method and system based on an unmanned aerial vehicle and deep learning, the unmanned aerial vehicle is controlled to fly over a pine nut forest according to a preset track, and image data of the pine nut forest and flight parameter data of the unmanned aerial vehicle are obtained; extracting pine nut regions by using a deep learning image segmentation algorithm to obtain pine nut target data; calculating a prediction frame parameter of the pine nut target according to the pine nut target data; calculating priorities of different pine nut targets by using a machine learning method based on the prediction frame parameters, and determining a picking sequence; calculating three-dimensional position coordinates of the pine nut target according to the pine nut target data and the flight parameter data; based on the three-dimensional position coordinates and the picking sequence, generating a flight path of the unmanned aerial vehicle by using a path planning algorithm; a mechanical arm on the unmanned aerial vehicle is activated, and pine nut picking operation is executed; the picked pine nuts are stored in a collecting device of the unmanned aerial vehicle; and a picking task completion report is generated, so that automatic picking is realized.

Description

Technical Field

[0001] The present invention relates to the field of automatic picking technology, and more specifically, to an automatic pine nut picking method and system based on drones and deep learning. Background Art

[0002] In recent years, with the acceleration of agricultural modernization, automated harvesting technology has received widespread attention and application in the field of fruit harvesting. Among them, pine nuts, as an important economic crop, have always been a hot topic of research in the automation of their harvesting process. Traditional pine nut picking methods mainly rely on manual labor, which is not only labor-intensive and inefficient, but also has safety hazards and is difficult to meet the needs of modern agricultural production.

[0003] To solve these problems, researchers began to try to apply drone technology to pine nut picking. Early drone picking systems were mainly operated by remote control. Although this improved picking efficiency and safety to a certain extent, it still required professional operators to control the entire process, making it difficult to achieve true automation. With the development of computer vision and machine learning technology, some semi-automated pine nut picking systems began to appear. These systems can initially identify the location of pine nuts through image recognition technology, but their recognition accuracy and reliability are still low in complex natural environments.

[0004] At present, the closest existing technology is the drone harvesting system combined with deep learning algorithms. Such systems can use deep learning models to analyze images and identify the location and maturity of pine nuts. However, these systems still have some significant technical problems: first, their pine nut recognition accuracy is not ideal under complex lighting conditions and background environments; second, the existing systems are not smart enough in multi-target priority sorting and path planning, which often leads to low picking efficiency; third, these systems lack effective obstacle avoidance mechanisms and are prone to collisions in dense pine forests; in addition, most of the existing systems are single-machine operations and lack the ability to coordinate multiple machines, making it difficult to achieve large-scale and efficient harvesting; finally, these systems generally lack adaptive optimization capabilities and find it difficult to dynamically adjust working parameters according to actual picking conditions. Summary of the invention

[0005] The present invention aims to solve the above technical problems and provide a method and system for automatic pine nut picking based on drones and deep learning. The method of the present invention realizes efficient, accurate and safe automatic pine nut picking by innovatively combining multimodal image acquisition, deep learning target detection, intelligent path planning, multi-machine collaborative operation and adaptive optimization.

[0006] The present invention provides an automatic pine nut picking method based on drones and deep learning, comprising:

[0007] The acquisition steps include:

[0008] Control the UAV to fly over the pine forest according to a preset trajectory to obtain image data of the pine forest and flight parameter data of the UAV;

[0009] The image data includes aerial images and infrared images;

[0010] The flight parameter data includes the three-dimensional position coordinates, flight attitude and photo shooting parameters of the drone;

[0011] Processing steps include:

[0012] Based on the image data, a deep learning image segmentation algorithm is used to extract the pine nut area to obtain pine nut target data;

[0013] Calculating the prediction box parameters of the pine nut target according to the pine nut target data;

[0014] Based on the prediction frame parameters, a machine learning method is used to calculate the priorities of different pine nut targets and determine the picking order;

[0015] Calculating the three-dimensional position coordinates of the pine nut target according to the pine nut target data and the flight parameter data;

[0016] Based on the three-dimensional position coordinates and the picking order, a flight path of the drone is generated using a path planning algorithm;

[0017] The picking steps include:

[0018] Control the drone to move to the target pine nut position according to the flight path;

[0019] Activate the robotic arm on the drone to perform pine nut picking operations;

[0020] The harvested pine nuts are stored in the drone’s collection device;

[0021] Output steps include:

[0022] Generate a picking task completion report, including picking quantity, picking efficiency and picking quality data.

[0023] Preferably, the obtaining step specifically includes:

[0024] An infrared light source is configured on the drone to simultaneously capture an infrared image when acquiring the image data;

[0025] Controlling the UAV to cruise along a serpentine route and fly at a constant speed;

[0026] At each preset time interval, the aerial image, the infrared image and the flight parameter data are simultaneously acquired.

[0027] Preferably, calculating the prediction box parameters of the pine nut target in the processing step includes:

[0028] Scanning and identifying the image data using a trained pine nut detection model to obtain a pine nut region;

[0029] Segment the pine nut region using the trained pine nut instance segmentation model to obtain the pixel position and area of ​​the pine nut;

[0030] Calculate the actual position and size of the pine nuts based on the pixel position and area in combination with the flight parameter data;

[0031] Generates prediction box parameters including top-left corner coordinates, width, and height.

[0032] Preferably, calculating the priorities of different pine nut targets in the processing step includes:

[0033] Set the standard prediction box area C;

[0034] Calculate the prediction box area S of each pine nut target;

[0035] According to the formula Calculate the priority P of each pine nut target, where e is the base of the natural logarithm.

[0036] Preferably, the processing step further comprises:

[0037] Build a reliability model based on historical picking data;

[0038] Calculating the picking reliability of the target in each prediction frame using the reliability model;

[0039] According to the preset reliability threshold, mark the prediction box with low reliability;

[0040] When generating the flight path, high-reliability prediction boxes are prioritized.

[0041] Preferably, the picking step comprises:

[0042] A plurality of mechanical arms are arranged on the drone, and each mechanical arm is provided with a plurality of pine nut collecting heads;

[0043] The picking area is divided into several sub-areas according to the distribution of pine nuts;

[0044] Control different robotic arms to perform picking operations in different sub-areas at the same time;

[0045] When picking in a sub-area is completed, the robot arm allocation is dynamically adjusted to ensure optimal utilization of resources.

[0046] Preferably, the step of avoiding obstacles is also included:

[0047] Use LiDAR to detect obstacles around the drone in real time;

[0048] Calculate the area of ​​the obstacle and the distance to the drone;

[0049] When an obstacle is detected, determine whether the preset obstacle avoidance conditions are met;

[0050] If the obstacle avoidance conditions are met, control the drone to adjust the flight altitude or change the flight path to avoid obstacles.

[0051] As a preferred embodiment, it also includes a multi-machine coordination step:

[0052] Deploy multiple drones to perform harvesting tasks simultaneously;

[0053] The central control system monitors the location, power and mission progress of each drone in real time;

[0054] Dynamically allocate picking areas and tasks based on the status of each drone;

[0055] When a drone is detected to have a battery level lower than a preset threshold, it will be marked as a low-power drone;

[0056] Control low-power drones to stay away from other drones and reduce their flight altitude;

[0057] Adjust the position and height of other drones to ensure that the overall picking efficiency is maximized.

[0058] Preferably, the method further includes an adaptive adjustment step:

[0059] Real-time monitoring of the success rate and efficiency of the picking process;

[0060] Dynamically adjust the prediction box parameter calculation method, priority evaluation criteria, and path planning strategy based on monitoring results;

[0061] Continuously optimize the deep learning model during the picking process to improve the accuracy of pine nut identification;

[0062] Automatically adjust flight parameters and picking strategies according to different time periods and weather conditions.

[0063] The automatic pine nut picking system based on drone and deep learning for executing the method comprises:

[0064] An image acquisition module is used to control the UAV to fly over the pine forest and acquire image data and flight parameter data;

[0065] Deep learning processing modules for:

[0066] extracting pine nut targets from the image data using a deep learning algorithm;

[0067] Calculate the prediction box parameters of the pine nut target;

[0068] Assess picking priorities for different pine nut targets;

[0069] A three-dimensional positioning module, used for calculating the three-dimensional position coordinates of the pine nut target according to the image data and the flight parameter data;

[0070] A path planning module, for generating a flight path of the drone based on the three-dimensional position coordinates and picking priorities;

[0071] A robotic arm control module, used to control multiple robotic arms on the drone to perform picking operations;

[0072] Obstacle avoidance module, used to detect and avoid obstacles during flight;

[0073] Multi-machine collaboration module, used to distribute tasks and coordinate actions among multiple drones;

[0074] Adaptive optimization module, used to adjust system parameters and strategies based on real-time picking data;

[0075] The central control module is used to coordinate the work of each module and generate task reports.

[0076] The beneficial effects of the present invention are mainly reflected in the following aspects:

[0077] First, the present invention greatly improves the accuracy and robustness of pine nut identification by combining multimodal data collection methods of aerial images and infrared images. This method can effectively overcome the influence of complex lighting conditions and background interference, allowing the system to maintain a high recognition rate in various environments.

[0078] Secondly, the invention introduces a target detection and priority evaluation mechanism based on deep learning, which can not only accurately locate pine nuts, but also intelligently evaluate the picking value of each pine nut. This mechanism ensures that the system always prioritizes the most valuable targets, significantly improving picking efficiency and quality.

[0079] Furthermore, the intelligent path planning algorithm of the present invention fully considers the spatial distribution and priority of multiple pine nut targets and can generate the optimal picking path. This not only reduces the ineffective flight time of the drone, but also maximizes the picking efficiency of a single flight.

[0080] In addition, the multi-machine coordination mechanism of the present invention realizes the coordinated work of multiple drones, significantly improving the efficiency of large-scale harvesting. Through dynamic task allocation and resource optimization, the system can reasonably allocate work according to the status of each drone to ensure the maximization of overall efficiency.

[0081] Finally, the adaptive optimization mechanism of the present invention enables the system to dynamically adjust operating parameters based on real-time picking data. This self-optimization capability enables the system to continuously improve performance and adapt to different environmental conditions and pine nut growth conditions.

[0082] In general, the method and system provided by the present invention have significant advantages in improving picking efficiency, reducing labor intensity, ensuring operation safety, etc. It not only solves many problems existing in the prior art, but also provides innovative solutions for the field of agricultural automation, and is expected to promote technological progress and improve production efficiency in the entire industry. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0084] Please refer to Figure 1 The present invention provides an automatic pine nut picking method and system based on drones and deep learning. The method collects temperature, environmental parameters and vibration data during the concrete pouring process, builds an intelligent monitoring model, and realizes real-time monitoring and early warning of the concrete pouring quality.

[0085] The present invention provides an automatic pine nut picking method and system based on drones and deep learning. The method realizes automatic identification, positioning and picking of pine nuts through intelligent drone technology and advanced deep learning algorithms, greatly improving picking efficiency and accuracy.

[0086] First, the method of the present invention includes an acquisition step, a processing step, a picking step and an output step. In the acquisition step, the drone flies over the pine nut forest according to a preset trajectory, and simultaneously acquires image data and flight parameter data. Preferably, the image data includes aerial images and infrared images, and this dual-modal image acquisition method can improve the accuracy of pine nut identification. The flight parameter data includes the three-dimensional position coordinates, flight attitude and photo shooting parameters of the drone, which provide a basis for subsequent precise positioning and picking.

[0087] In the processing step, the method of the present invention first uses a deep learning image segmentation algorithm to extract the pine nut area and obtain pine nut target data. The key to this step is the use of an efficient deep learning model, such as Mask R-CNN or U-Net, to ensure accurate identification of pine nuts in a complex natural environment. Next, the prediction box parameters are calculated based on the pine nut target data, which generally include the coordinates, width, and height of the bounding box.

[0088] The prediction box parameters can be calculated using the following formula:

[0089]

[0090] Among them, x and y are the coordinates of the upper left corner of the prediction box, w and h are the width and height of the prediction box respectively. c ,y c ) is the coordinate of the center point of the pine nut, (x 1 ,y 1 ) and (x 2 ,y 2 ) are the upper left corner and lower right corner coordinates of the pine nut boundary, respectively. In one embodiment of the present invention, a machine learning method is used to calculate the priorities of different pine nut targets to determine the best picking order. Priority calculation can take into account multiple factors, such as the maturity, size, and location of the pine nuts. For example, the following formula can be used to calculate the priority:

[0091] P=w 1 ·S+w 2 ·M+w 3 D,

[0092] Among them, P is the priority score, S is the normalized value of pine nut size, M is the maturity assessment value, D is the distance factor, and w 1 、w 2 and w 3 is the weight coefficient, which can be adjusted according to actual needs.

[0093] After determining the picking order, the method uses the flight parameter data and image data to calculate the three-dimensional position coordinates of the pine nut target. This step usually involves complex visual SLAM (simultaneous localization and mapping) technology. Preferably, the following formula can be used for coordinate conversion:

[0094]

[0095] Among them, (X, Y, Z) is the world coordinate of the pine nut, (x, y) is the image coordinate, f is the focal length of the camera, R is the rotation matrix, and T is the translation vector. These two parameters can be obtained from the attitude data of the drone. Based on the calculated three-dimensional position coordinates and picking order, the present invention uses a path planning algorithm to generate the flight path of the drone. Preferably, an improved A" algorithm or RRT (fast random exploration tree) algorithm can be used to implement path planning to ensure that the drone can reach the target position efficiently and avoid possible obstacles. In the picking step, the drone moves to the target pine nut position according to the planned path, and activates the robotic arm to perform the picking operation. In one embodiment of the present invention, the motion control of the robotic arm adopts an inverse kinematics algorithm, and the Jacobian matrix method can be used to solve the joint angles:

[0096] Δθ=J -1 Δx,

[0097] Among them, Δθ is the change of joint angle, J -1 is the inverse of the Jacobian matrix, and Δx is the position change of the end effector.

[0098] Finally, in the output step, the system generates a picking task completion report, including picking quantity, picking efficiency and picking quality data. These data can be used for subsequent system optimization and efficiency analysis.

[0099] Furthermore, the present invention also includes some innovative designs in the acquisition step. For example, an infrared light source is configured on the drone to capture infrared images while acquiring image data. This approach can improve the recognition rate of pine nuts under different lighting conditions, especially on cloudy days or when there are many shades of trees. At the same time, the drone is controlled to cruise along a serpentine route and fly at a constant speed. This flying method can ensure a comprehensive and uniform scan of the entire pine nut forest.

[0100] Preferably, the flight speed of the drone can be set within the range of 3-5 m / s, which can ensure the clarity of image acquisition and improve work efficiency. Aerial images, infrared images and flight parameter data are simultaneously acquired at each preset time interval (e.g., every 0.5 seconds). This synchronous acquisition method provides a reliable basis for subsequent data processing and analysis.

[0101] The method of the present invention also includes a detailed pine nut target prediction box parameter calculation process in the processing step. First, the image data is scanned and identified using a trained pine nut detection model to obtain a pine nut area. The detection model here can use advanced target detection algorithms such as YOLOv5 or Faster R-CNN, and the accuracy of the model can usually reach more than 95%.

[0102] Next, the trained pine nut instance segmentation model is used to segment the pine nut region to obtain the pixel position and area of ​​the pine nut. The instance segmentation model can use algorithms such as Mask R-CNN or DeepLab, which can accurately segment each independent pine nut instance.

[0103] Based on the pixel position and area, combined with the flight parameter data, the actual position and size of the pine nuts are calculated. This step involves the conversion from the image coordinate system to the world coordinate system, which usually requires considering the camera's intrinsic and extrinsic parameters. Finally, the predicted box parameters including the upper left corner coordinates, width, and height are generated.

[0104] This method of the present invention achieves accurate positioning and parameter calculation of pine nuts through the fine processing of multiple steps, laying a solid foundation for subsequent picking operations. At the same time, this processing method based on deep learning has strong adaptability and scalability, and can adapt to different environmental conditions and pine nut varieties. In a preferred embodiment of the present invention, the processing step also includes an innovative pine nut target priority calculation method. This method sets a standard prediction box area C, calculates the prediction box area S of each pine nut target, and then uses the following formula to calculate the priority P of each pine nut target:

[0105]

[0106] Where e is the base of the natural logarithm, which is approximately equal to 2.718. The design of this formula takes into account the deviation of the pine nut size from the standard size, so that pine nuts that are closer to the standard size are given higher priority. Preferably, the standard prediction box area C can be set to 100 square pixels, which usually corresponds to the pine nut size with the best maturity.

[0107] A significant advantage of this method is that it can automatically adjust the picking order and give priority to picking the most suitable pine nuts, thereby improving picking efficiency and quality. For example, when S is equal to C, the value of P is 1, indicating that it fully meets the standard size; when S deviates from C, the value of P decreases as the deviation increases. This mechanism ensures that the pine nuts in the best condition are given priority during the picking process.

[0108] Furthermore, the method of the present invention also introduces a reliability model based on historical picking data in the processing step. The purpose of this model is to improve the success rate of picking operations and reduce unnecessary attempts. Specifically, the process of establishing the reliability model is as follows:

[0109] First, the system records the results of each picking operation, including the number of successful pickings and the number of failed pickings. Then, using this historical data, the picking reliability R of each target in the prediction box can be calculated using the following formula:

[0110]

[0111] Among them, N s is the number of successful pickings, N f is the number of failures, α and β are smoothing factors, which can usually be set to 1. This formula is essentially a Bayesian estimate, which can give reasonable estimates even when the amount of data is small.

[0112] Preferably, a reliability threshold can be set, such as 0.7. When the reliability R of a prediction box is lower than this threshold, the system will mark it as a low-reliability prediction box. When generating a flight path, the system will give priority to high-reliability prediction boxes, thereby improving the overall picking efficiency.

[0113] The method of the present invention adopts an innovative design of multi-robot collaborative operation in the picking step. Multiple robotic arms are configured on the drone, and each robotic arm is provided with multiple pine nut collection heads. This design significantly improves the parallelism and efficiency of picking. According to the distribution of pine nuts, the system will divide the picking area into multiple sub-areas and control different robotic arms to perform picking operations in different sub-areas at the same time.

[0114] For example, in one embodiment, the drone can be equipped with 4 robotic arms, each with 3 collection heads. This configuration enables the drone to handle 12 picking points at the same time, greatly improving work efficiency. When the pine nut picking in a sub-area is completed, the system will dynamically adjust the allocation of the robotic arms to ensure optimal utilization of resources. This dynamic adjustment strategy can be based on the following formula:

[0115]

[0116] Among them, E i is the efficiency index of the i-th robot arm, N i is the number of pine nuts picked, T i is the working time, w i is a weight factor (which can be set according to the performance or position of the robot arm). The system will give priority to assigning new picking tasks to the robot arm with a higher efficiency index, thereby achieving the optimal allocation of resources.

[0117] The method of the present invention also includes an intelligent obstacle avoidance step, which is essential to ensure the safe operation of the drone in the complex pine forest environment. In this step, the system uses the laser radar to detect obstacles around the drone in real time, calculate the area of ​​the obstacle and the distance to the drone. When an obstacle is detected, the system will determine whether the preset obstacle avoidance conditions are met.

[0118] Preferably, the obstacle avoidance condition can be set as follows: when the area of ​​the obstacle is greater than 100 square centimeters and the distance from the drone is less than 2 meters, the obstacle avoidance operation is triggered. This setting takes into account the general size of pine branches and leaves and the operating space of the drone. When the obstacle avoidance condition is met, the system will control the drone to adjust the flight altitude or change the flight path to avoid the obstacle.

[0119] The specific obstacle avoidance strategy can be determined based on the following formula:

[0120] ΔH=k·(2-d),

[0121] Among them, ΔH is the height that needs to be adjusted, d is the distance between the obstacle and the drone (in meters), and k is the adjustment coefficient, which can be set to 0.5. This formula ensures that the drone can smoothly avoid obstacles while staying close to the target pine nuts.

[0122] Through these innovative designs and algorithms, the method of the present invention realizes an efficient, safe and accurate automatic pine nut picking process, greatly improves the picking efficiency, reduces labor costs, and also reduces interference with the environment. This method is not only suitable for pine nut picking, but can also be extended to other types of fruit picking, and has broad application prospects. The method of the present invention further introduces an innovative mechanism of multi-machine collaborative work, which greatly improves the overall picking efficiency and the robustness of the system. In this mechanism, multiple drones are deployed simultaneously to perform picking tasks, and the position, power and task progress of each drone are monitored in real time through a central control system. This centralized management method makes the entire picking process more orderly and efficient.

[0123] Preferably, the central control system adopts a distributed computing architecture that can quickly process large amounts of real-time data. The system dynamically allocates picking areas and tasks based on the status of each drone. This dynamic allocation strategy can be expressed by the following formula:

[0124]

[0125] Among them, S i is the picking area assigned to the i-th drone, E i is the efficiency coefficient of the UAV, B i is the remaining battery power percentage, n is the total number of drones, A total is the total picking area. This formula ensures that task allocation takes into account both the efficiency and the remaining power of the drone, thereby achieving the optimal allocation of resources. A special feature of the present invention is the introduction of the concept of low-power drones. When the system detects that the power of a drone is lower than a preset threshold (for example, 30%), it will be marked as a low-power drone. For low-power drones, the system will adopt a special management strategy: control it away from other drones and lower its flight altitude. This strategy can not only extend the working time of low-power drones, but also reduce the risk of collision with other drones. At the same time, the system will adjust the position and altitude of other drones accordingly to ensure the maximization of the overall picking efficiency. This adjustment can be achieved through the following formula:

[0126]

[0127] Among them, H j is the new flight altitude of the jth UAV, H baseis the reference height, ΔH is the maximum height adjustment, d ij is the distance between the jth UAV and the low-power UAV i, d max is the preset maximum impact distance. This formula ensures that other drones closer to the low-power drone will appropriately increase their flight altitude, thus forming a coordinated three-dimensional working space.

[0128] The method of the present invention also includes an adaptive adjustment mechanism, which enables the entire system to continuously optimize its performance. Specifically, the system monitors the success rate and efficiency of the picking process in real time and dynamically adjusts multiple key parameters based on the monitoring results.

[0129] For example, the prediction box parameter calculation method can be optimized according to the actual picking situation. Assuming that the system finds that the current prediction box is often slightly larger than the actual pine nut size, resulting in reduced picking accuracy, the prediction box size can be adjusted using the following formula:

[0130]

[0131] Among them, W new and W old are the prediction box widths before and after adjustment, α is the learning rate (usually set between 0.01 and 0.1), E size is the size error, E max is the maximum acceptable error. The height adjustment can be achieved using a similar formula.

[0132] Priority evaluation criteria and path planning strategies can also be adaptively adjusted in a similar way. This continuous optimization mechanism ensures that the system can adapt to different environmental conditions and pine nut growth conditions and always maintain an efficient working state.

[0133] In addition, the method of the present invention continuously optimizes the deep learning model during the picking process to improve the accuracy of pine nut identification. This can be achieved by online learning, that is, the results of each picking are used as new training samples to update the model. The model can be updated using the stochastic gradient descent method, and its weight update formula is as follows:

[0134]

[0135] Among them, w t and w t+1 are the current and updated model weights, respectively, η is the learning rate, is the gradient of the loss function.

[0136] Finally, the invention also automatically adjusts flight parameters and picking strategies according to different time periods and weather conditions. For example, in low light conditions, the system increases the exposure time for image acquisition; in strong wind conditions, the system reduces the flight speed and increases the stability control of the robot arm. These adjustments can be achieved through a fuzzy logic-based control system to ensure efficient picking operations in all environments.

[0137] In general, the pine nut automatic picking system based on drone and deep learning provided by the present invention is a highly intelligent and automated solution. The system includes an image acquisition module 1, a deep learning processing module 2, a three-dimensional positioning module 3, a path planning module 4, a mechanical arm control module 5, an obstacle avoidance module 6, a multi-machine collaboration module 7, an adaptive optimization module 8 and a central control module 9. These modules work closely together to achieve an efficient, accurate and safe pine nut picking process.

[0138] The image acquisition module 1 is responsible for controlling the drone to fly over the pine nut forest and obtain high-quality image data and flight parameter data. The deep learning processing module 2 uses advanced algorithms to extract pine nut targets from the image data, calculate the prediction box parameters, and evaluate the picking priority of different pine nut targets. The three-dimensional positioning module 3 combines the image data and flight parameter data to accurately calculate the spatial coordinates of each pine nut target.

[0139] The path planning module 4 generates the optimal flight path of the drone based on the three-dimensional position coordinates and picking priority. The robotic arm control module 5 is responsible for accurately controlling the multiple robotic arms on the drone to perform picking operations. The obstacle avoidance module 6 detects the surrounding environment in real time to ensure that the drone can safely avoid various obstacles.

[0140] The multi-machine collaboration module 7 and the adaptive optimization module 8 are the two major innovations of this system. The former realizes the task allocation and coordinated actions among multiple drones, greatly improving the overall work efficiency; the latter ensures that the system always maintains the best working state through real-time data analysis and parameter adjustment.

[0141] The central control module 9 acts as the "brain" of the entire system, coordinating the work of each module and generating detailed task reports. These reports not only record the picking quantity and quality, but also include system performance analysis and optimization suggestions, providing valuable data support for subsequent system upgrades.

[0142] Through this modular design, the system of the present invention has extremely strong flexibility and scalability. It can not only efficiently complete the pine nut picking task, but also be applicable to other types of fruit picking through simple adjustments. This versatility makes the system have broad application prospects in the field of modern agricultural automation and is expected to make important contributions to the improvement of agricultural production efficiency.

[0143] It should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modification, replacement, and improvement made within the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. The automatic pine nut picking method based on drone and deep learning is characterized by: include: The acquisition steps include: Control the UAV to fly over the pine forest according to a preset trajectory to obtain image data of the pine forest and flight parameter data of the UAV; The image data includes aerial images and infrared images; The flight parameter data includes the three-dimensional position coordinates, flight attitude and photo shooting parameters of the drone; Processing steps include: Based on the image data, a deep learning image segmentation algorithm is used to extract the pine nut area to obtain pine nut target data; Calculating the prediction box parameters of the pine nut target according to the pine nut target data; Based on the prediction frame parameters, a machine learning method is used to calculate the priorities of different pine nut targets and determine the picking order; Calculating the three-dimensional position coordinates of the pine nut target according to the pine nut target data and the flight parameter data; Based on the three-dimensional position coordinates and the picking order, a flight path of the drone is generated using a path planning algorithm; The picking steps include: Control the drone to move to the target pine nut position according to the flight path; Activate the robotic arm on the drone to perform pine nut picking operations; The harvested pine nuts are stored in the drone’s collection device; Output steps include: Generate a picking task completion report, including picking quantity, picking efficiency and picking quality data.

2. The automatic pine nut picking method based on drone and deep learning according to claim 1 is characterized in that: The acquisition step specifically includes: An infrared light source is configured on the drone to simultaneously capture an infrared image when acquiring the image data; Controlling the UAV to cruise along a serpentine route and fly at a constant speed; At each preset time interval, the aerial image, the infrared image and the flight parameter data are simultaneously acquired.

3. The automatic pine nut picking method based on drone and deep learning according to claim 1 is characterized in that: The prediction box parameters of the pine nut target are calculated in the processing step including: Scanning and identifying the image data using a trained pine nut detection model to obtain a pine nut region; Segment the pine nut region using the trained pine nut instance segmentation model to obtain the pixel position and area of ​​the pine nut; Calculate the actual position and size of the pine nuts based on the pixel position and area in combination with the flight parameter data; Generates prediction box parameters including top-left corner coordinates, width, and height.

4. The automatic pine nut picking method based on drone and deep learning according to claim 1 is characterized in that: The priority of calculating different pine nut targets in the processing step includes: Set the standard prediction box area C; Calculate the predicted box area S of each pine nut target; According to the formula Calculate the priority P of each pine nut target, where e is the base of the natural logarithm.

5. The automatic pine nut picking method based on drone and deep learning according to claim 1 is characterized in that: The processing steps also include: Build a reliability model based on historical picking data; Calculating the picking reliability of the target in each prediction frame using the reliability model; According to the preset reliability threshold, mark the prediction box with low reliability; When generating the flight path, high-reliability prediction boxes are prioritized.

6. The automatic pine nut picking method based on drone and deep learning according to claim 1 is characterized in that: The picking steps include: A plurality of mechanical arms are arranged on the drone, and each mechanical arm is provided with a plurality of pine nut collecting heads; The picking area is divided into several sub-areas according to the distribution of pine nuts; Control different robotic arms to perform picking operations in different sub-areas at the same time; When picking in a sub-area is completed, the robot arm allocation is dynamically adjusted to ensure optimal utilization of resources.

7. The automatic pine nut picking method based on drone and deep learning according to claim 1 is characterized in that: It also includes obstacle avoidance steps: Use LiDAR to detect obstacles around the drone in real time; Calculate the area of ​​the obstacle and the distance to the drone; When an obstacle is detected, determine whether the preset obstacle avoidance conditions are met; If the obstacle avoidance conditions are met, control the drone to adjust the flight altitude or change the flight path to avoid obstacles.

8. The automatic pine nut picking method based on drone and deep learning according to claim 1 is characterized in that: It also includes multi-machine collaboration steps: Deploy multiple drones to perform harvesting tasks simultaneously; The central control system monitors the location, power and mission progress of each drone in real time; Dynamically allocate picking areas and tasks based on the status of each drone; When a drone is detected to have a battery level lower than a preset threshold, it will be marked as a low-power drone; Control low-power drones to stay away from other drones and reduce their flight altitude; Adjust the position and height of other drones to ensure that the overall picking efficiency is maximized.

9. The automatic pine nut picking method based on drone and deep learning according to claim 1 is characterized in that: It also includes an adaptive adjustment step: Real-time monitoring of the success rate and efficiency of the picking process; Dynamically adjust the prediction box parameter calculation method, priority evaluation criteria, and path planning strategy based on monitoring results; Continuously optimize the deep learning model during the picking process to improve the accuracy of pine nut identification; Automatically adjust flight parameters and picking strategies according to different time periods and weather conditions.

10. An automatic pine nut picking system based on drone and deep learning that implements the method according to any one of claims 1 to 3, characterized in that: include: An image acquisition module is used to control the UAV to fly over the pine forest and acquire image data and flight parameter data; Deep learning processing module for: extracting pine nut targets from the image data using a deep learning algorithm; Calculate the prediction box parameters of the pine nut target; Assess picking priorities for different pine nut targets; A three-dimensional positioning module, used for calculating the three-dimensional position coordinates of the pine nut target according to the image data and the flight parameter data; A path planning module, for generating a flight path of the drone based on the three-dimensional position coordinates and picking priorities; A robotic arm control module, used to control multiple robotic arms on the drone to perform picking operations; Obstacle avoidance module, used to detect and avoid obstacles during flight; Multi-machine collaboration module, used to distribute tasks and coordinate actions among multiple drones; Adaptive optimization module, used to adjust system parameters and strategies based on real-time picking data; The central control module is used to coordinate the work of each module and generate task reports.

Images