Air-ground cooperative fruit target picking decision method, system and device
By collecting image information from the air and ground above the orchard, the detection path and harvesting strategy are determined, solving the problem that existing fruit harvesters cannot accurately harvest areas with different maturity and yield, and realizing efficient automated harvesting of target fruits.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- INTELLIGENT EQUIPMENT RESEARCH CENTER BEIJING ACADEMY OF AGRICULTURE AND FORESTRY SCIENCES
- Filing Date
- 2023-03-14
- Publication Date
- 2026-05-12
AI Technical Summary
Existing fruit harvesters cannot accurately harvest fruits at different stages of maturity, resulting in low harvest quality. Furthermore, they cannot differentiate harvesting tasks for different yield areas in the orchard, limiting the improvement in harvesting efficiency.
Based on the first image information collected over the target orchard, the ground unmanned vehicle is used to determine the target detection path. The ground unmanned vehicle equipped with a visual sensor is then controlled to collect the second image information within the orchard. Combining the first and second image information, the yield distribution and maturity of the target fruits are determined. Based on the yield distribution and maturity of the target fruits, as well as the terrain information of the orchard, a harvesting strategy is determined.
It enables precise and efficient automated harvesting of fruits with different yield distributions and different maturity levels, improving harvesting quality and efficiency.
Smart Images
Figure CN116958032B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of agricultural informatization technology, and in particular to a fruit target harvesting decision-making method, system and device based on air-ground cooperation. BACKGROUND
[0002] In recent years, various fruit harvesting machines have appeared on the market, such as handheld fruit harvesting machines and tree-shaped wrapping fruit harvesting machines. The appearance of these fruit harvesting machines can improve the efficiency of fruit harvesting and alleviate the problem of labor shortage.
[0003] However, the existing fruit harvesting machines cannot achieve accurate harvesting of fruit targets with different maturity, and the harvesting quality is low. Moreover, they cannot achieve differentiated allocation of harvesting tasks in different yield areas in the orchard, and the improvement of harvesting efficiency is limited. SUMMARY
[0004] In view of the problems existing in the prior art, the present application provides a fruit target harvesting decision-making method, system and device based on air-ground cooperation.
[0005] In a first aspect, the present application provides a fruit target harvesting decision-making method based on air-ground cooperation, comprising:
[0006] determining a target detection path based on first image information collected in the air above a target orchard;
[0007] controlling a ground unmanned vehicle carrying a visual sensor to collect second image information in the target orchard based on the target detection path;
[0008] determining the yield distribution and maturity of fruit targets in the target orchard based on the second image information and the first image information;
[0009] determining a harvesting strategy for fruit targets in the target orchard based on the yield distribution and maturity of the fruit targets, and the topographic information of the target orchard, wherein the topographic information of the target orchard is determined based on the first image information.
[0010] Optionally, according to the fruit target harvesting decision-making method based on air-ground cooperation provided by the present application, the determination of the harvesting strategy for fruit targets in the target orchard based on the yield distribution and maturity of the fruit targets, and the topographic information of the target orchard comprises:
[0011] determining the harvesting batch of fruit targets in each preset area in the target orchard, the number of harvesting vehicles required for harvesting fruit targets in each preset area, and the operation path of each harvesting vehicle for harvesting fruit targets based on the yield distribution and maturity of the fruit targets, and the topographic information of the target orchard.
[0012] Optionally, according to the air-ground coordinated fruit target harvesting decision-making method provided by the present invention, the step of determining the target detection path based on the first image information collected over the target orchard includes:
[0013] Based on the first image information, the location information of each fruit tree in the target orchard and the terrain information of the target orchard are determined;
[0014] Based on the location information of each fruit tree and the terrain information of the target orchard, the target detection path is determined.
[0015] Optionally, according to the air-ground coordinated fruit target harvesting decision-making method provided by the present invention, the method further includes:
[0016] Based on the harvesting strategy for the target fruit types in the target orchard, harvesting vehicles are dispatched to carry out harvesting operations in the target orchard.
[0017] Secondly, the present invention also provides an air-ground coordinated fruit target harvesting decision system, comprising:
[0018] Unmanned aerial vehicles equipped with visual sensors, unmanned ground vehicles equipped with visual sensors, and ground station management platforms;
[0019] The drone is used to collect first image information over the target orchard based on the shooting instructions sent by the ground station management platform, and to send the first image information to the ground station management platform;
[0020] The ground unmanned vehicle is used to collect second image information in the target orchard based on the target detection path determined by the ground station management platform, and send the second image information to the ground station management platform;
[0021] The ground station management platform is used to determine the target detection path based on the first image information and send the target detection path to the ground unmanned vehicle. It is also used to determine the yield distribution and maturity of fruit targets in the target orchard based on the second image information and the first image information, and to determine the harvesting strategy of fruit targets in the target orchard based on the yield distribution and maturity of the fruit targets and the terrain information of the target orchard, wherein the terrain information of the target orchard is determined based on the first image information.
[0022] Optionally, according to the air-ground collaborative fruit target harvesting decision-making system provided by the present invention, the ground station management platform is specifically used for:
[0023] Based on the yield distribution and maturity of the target fruits, as well as the terrain information of the target orchard, the harvesting batches of the target fruits in each preset area of the target orchard, the number of harvesting vehicles required to harvest the target fruits in each preset area, and the operation path of each harvesting vehicle for harvesting the target fruits are determined.
[0024] Optionally, according to the air-ground coordinated fruit target harvesting decision system provided by the present invention, the ground station management platform is further specifically used for:
[0025] Based on the first image information, the location information of each fruit tree in the target orchard and the terrain information of the target orchard are determined;
[0026] Based on the location information of each fruit tree and the terrain information of the target orchard, the target detection path is determined.
[0027] Optionally, according to the air-ground coordinated fruit target harvesting decision system provided by the present invention, the system further includes a harvesting vehicle controller and a harvesting vehicle;
[0028] The harvester controller is used to dispatch the harvester to carry out harvesting operations in the target orchard based on the harvesting strategy of the target fruit species determined by the ground station management platform.
[0029] Thirdly, the present invention also provides an air-ground coordinated fruit target harvesting decision-making device, comprising:
[0030] The first determining module is used to determine the target detection path based on the first image information collected over the target orchard;
[0031] The image acquisition module is used to control a ground unmanned vehicle equipped with a visual sensor to acquire second image information in the target orchard based on the target detection path;
[0032] The second determining module is used to determine the yield distribution and maturity of the target fruit in the target orchard based on the second image information and the first image information.
[0033] The third determining module is used to determine the harvesting strategy for the fruit targets in the target orchard based on the yield distribution and maturity of the fruit targets, as well as the terrain information of the target orchard. The terrain information of the target orchard is determined based on the first image information.
[0034] Fourthly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the air-ground coordinated fruit target harvesting decision method as described in the first aspect.
[0035] The air-ground collaborative fruit target harvesting decision-making method, system, and apparatus provided by this invention first determines the target detection path for a ground-based unmanned vehicle based on first image information collected over the target orchard. Then, the ground-based unmanned vehicle equipped with a visual sensor is controlled to collect second image information within the target orchard based on the target detection path. Next, based on the second and first image information, the yield distribution and maturity of the fruit targets within the target orchard are determined. Finally, based on the yield distribution and maturity of the fruit targets, as well as the terrain information of the target orchard, a harvesting strategy for the fruit targets within the target orchard is determined. According to this harvesting strategy, accurate, efficient, and automated harvesting of fruit targets with different yield distributions and different maturity levels can be achieved. Attached Figure Description
[0036] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0037] Figure 1 This is a flowchart illustrating the air-ground coordinated fruit target harvesting decision-making method provided by the present invention;
[0038] Figure 2 This is a schematic diagram of the air-ground coordinated fruit target harvesting decision system provided by the present invention;
[0039] Figure 3 This is a schematic diagram of image information acquisition provided by the present invention;
[0040] Figure 4 This is a schematic diagram of the process for harvesting camellia oleifera fruit using the air-ground coordinated fruit target harvesting decision system provided by the present invention;
[0041] Figure 5 This is a schematic diagram of the structure of the air-ground coordinated fruit target harvesting decision device provided by the present invention;
[0042] Figure 6 This is a schematic diagram of the physical structure of the electronic device provided by the present invention. Detailed Implementation
[0043] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0044] It should be noted that in the description of this invention, the terms "first," "second," etc., are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, the first object can be one or more.
[0045] The following description, in conjunction with the accompanying drawings, provides an exemplary introduction to the air-ground coordinated fruit target harvesting decision-making method, system, and apparatus provided by the present invention.
[0046] Figure 1 This is a flowchart illustrating the air-ground coordinated fruit target harvesting decision-making method provided by the present invention, as shown below. Figure 1 As shown, the method includes:
[0047] Step 100: Determine the target detection path based on the first image information collected over the target orchard;
[0048] Step 110: Control the ground unmanned vehicle equipped with a visual sensor to collect second image information in the target orchard based on the target detection path;
[0049] Step 120: Based on the second image information and the first image information, determine the yield distribution and maturity of the target fruits in the target orchard;
[0050] Step 130: Based on the yield distribution and maturity of the target fruits, and the topographic information of the target orchard, determine the harvesting strategy for the target fruits in the target orchard. The topographic information of the target orchard is determined based on the first image information.
[0051] It should be noted that the executing entity of the air-ground coordinated fruit target harvesting decision-making method provided in this embodiment of the invention can be an electronic device, a component in the electronic device, an integrated circuit, or a chip. The electronic device can be a mobile electronic device or a non-mobile electronic device. For example, a mobile electronic device can be a mobile phone, tablet computer, laptop computer, PDA, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc., while a non-mobile electronic device can be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This embodiment of the invention does not specifically limit the specific implementation of these devices.
[0052] The following example, using a computer executing the air-ground coordinated fruit target harvesting decision-making method provided by this invention, illustrates the technical solution of this invention in detail.
[0053] Specifically, to overcome the shortcomings of existing fruit harvesters, such as the inability to accurately harvest fruits at different maturity levels, low harvesting quality, and the inability to differentiate harvesting tasks for different yield areas in orchards, resulting in limited improvement in harvesting efficiency, this invention first determines the target detection path for a ground-based unmanned vehicle based on first image information collected from the air above the target orchard. Then, the ground-based unmanned vehicle, equipped with a visual sensor, is controlled to collect second image information within the target orchard based on this target detection path. Next, based on the second and first image information, the yield distribution and maturity of the fruits within the target orchard are determined. Finally, based on the yield distribution, maturity, and terrain information of the fruits, a harvesting strategy for the fruits within the target orchard is determined. According to this harvesting strategy, accurate, efficient, and automated harvesting of fruits with different yield distributions and maturity levels can be achieved.
[0054] It should be noted that, in the embodiments of the present invention, the target fruit in the target orchard can be any common harvestable fruit or dried fruit, such as apples, pears, peaches, camellia oleifera fruit and walnuts, etc., and the embodiments of the present invention do not make specific limitations on this.
[0055] Optionally, in this embodiment of the invention, before determining the target detection path based on the first image information, a drone equipped with a visual sensor can be controlled to collect the first image information over the target orchard.
[0056] Optionally, the first image information can be processed and analyzed to determine the target detection path for the ground unmanned vehicle.
[0057] Optionally, after determining the target detection path, a ground-based unmanned vehicle equipped with a visual sensor can be controlled to collect second image information within the target orchard based on the target detection path.
[0058] It should be noted that since the first image information was collected from above the target orchard, it includes the distribution information of fruit targets in the canopy above the fruit trees. In order to obtain complete distribution information of fruit targets, a ground-based unmanned vehicle collects second image information in the target orchard along the target detection path. The second image information includes the distribution information of fruit targets in the middle and lower canopies of the fruit trees. This can overcome the defect of incomplete data collection caused by the field of view of the visual sensor.
[0059] Optionally, in this embodiment of the invention, the second image information and the first image information can be fused using image stitching technology to generate a complete three-dimensional map, and then the yield distribution and maturity of the target fruit in the target orchard can be determined based on the three-dimensional map.
[0060] Optionally, in this embodiment of the invention, the topographic information of the target orchard (e.g., the slope between rows of fruit trees and the distribution of obstacles, as well as the canopy height of the rows of fruit trees) can also be determined based on the first image information.
[0061] Optionally, in this embodiment of the invention, the harvesting strategy for the fruit in the target orchard can be determined based on the yield distribution and maturity of the fruit targets, as well as the topographic information of the target orchard.
[0062] It is understood that, since the harvesting strategy of the present invention is determined based on the yield distribution and maturity of the target fruit, the subsequent harvesting of the target fruit in the target orchard according to the harvesting strategy can achieve accurate, efficient and automated harvesting of fruit with different yield distributions and different maturity levels.
[0063] The air-ground collaborative fruit target harvesting decision-making method provided by this invention first determines the target detection path for a ground-based unmanned vehicle based on first image information collected over the target orchard. Then, it controls the ground-based unmanned vehicle equipped with a visual sensor to collect second image information within the target orchard based on the target detection path. Next, based on the second and first image information, it determines the yield distribution and maturity of the fruit targets within the target orchard. Finally, based on the yield distribution and maturity of the fruit targets, as well as the terrain information of the target orchard, it determines the harvesting strategy for the fruit targets within the target orchard. According to this harvesting strategy, accurate, efficient, and automated harvesting of fruit targets with different yield distributions and different maturity levels can be achieved.
[0064] Optionally, determining the harvesting strategy for the fruit targets within the target orchard based on the yield distribution and maturity of the target fruit, as well as the topographic information of the target orchard, includes:
[0065] Based on the yield distribution and maturity of the target fruits, as well as the terrain information of the target orchard, the harvesting batches of the target fruits in each preset area of the target orchard, the number of harvesting vehicles required to harvest the target fruits in each preset area, and the operation path of each harvesting vehicle for harvesting the target fruits are determined.
[0066] Specifically, in this embodiment of the invention, after determining the yield distribution and maturity of the fruit targets in the target orchard based on the second image information and the first image information, the harvesting batches of the fruit targets in each preset area of the target orchard, the number of harvesting vehicles required to harvest the fruit targets in each preset area, and the operation path of each harvesting vehicle for harvesting the fruit targets can be determined based on the yield distribution and maturity of the fruit targets and the terrain information of the target orchard.
[0067] For example, for fruit targets in the target orchard whose maturity is greater than the first preset value, it can be determined that the first batch of fruit targets in that area will be harvested; for fruit targets in the target orchard whose maturity is greater than the second preset value but less than the first preset value, it can be determined that the second batch of fruit targets in that area will be harvested; for fruit targets in the target orchard whose maturity is less than the second preset value, it can be determined that the third batch of fruit targets in that area will be harvested.
[0068] For example, for areas within the target orchard where the yield distribution of fruit is greater than the fourth preset value, it can be determined that 10 harvesting vehicles are needed to harvest the fruit in that area; for areas within the target orchard where the yield distribution of fruit is less than the fourth preset value, it can be determined that 5 harvesting vehicles are needed to harvest the fruit in that area.
[0069] It is understood that by harvesting fruits in batches based on their maturity, the embodiments of the present invention can improve the quality of fruit harvesting and increase economic benefits. Moreover, by scheduling harvesting vehicles according to the yield of fruits in different areas of the target orchard, the embodiments of the present invention can reduce the harvesting time and improve the harvesting efficiency.
[0070] Optionally, the operation path for each harvesting vehicle can be determined based on the terrain information of the target orchard.
[0071] For example, based on the slope between fruit tree rows, the distribution of obstacles, and the canopy height of the fruit tree rows, the operation path for each harvesting vehicle to harvest the target fruit can be determined.
[0072] This invention, through its embodiments, determines the harvesting batches of fruit targets in each preset area of the target orchard, the number of harvesting vehicles required to harvest fruit targets in each preset area, and the operating path of each harvesting vehicle, based on the yield distribution and maturity of the fruit targets and the terrain information of the target orchard. This enables precise and efficient automated harvesting of fruit targets with different yield distributions and different maturity levels.
[0073] Optionally, determining the target detection path based on the first image information collected over the target orchard includes:
[0074] Based on the first image information, the location information of each fruit tree in the target orchard and the terrain information of the target orchard are determined;
[0075] Based on the location information of each fruit tree and the terrain information of the target orchard, the target detection path is determined.
[0076] Specifically, in this embodiment of the invention, after acquiring the first image information collected above the target orchard, the location information of each fruit tree in the target orchard and the terrain information of the target orchard can be determined based on the first image information. Then, based on the location information of each fruit tree and the terrain information of the target orchard, the target detection path can be determined for the ground unmanned vehicle.
[0077] Optionally, in this embodiment of the invention, based on the first image information, a target detection algorithm (e.g., YOLOv3) can be used to identify and locate the fruit trees and the fruit targets in the canopy above the fruit trees in the target orchard, thereby obtaining the location information of each fruit tree in the target orchard and the location information of the fruit targets in the canopy above each fruit tree. It should be noted that the location information of the fruit targets in the middle and lower canopy of the fruit trees can be obtained based on the second image information.
[0078] Optionally, in this embodiment of the invention, based on the first image information, the edge of the crop row of fruit trees in the target orchard can be obtained using an edge detection algorithm. Then, based on a preset threshold, the data of the fruit tree canopy is filtered out. After performing plane fitting on the filtered data, the slope between adjacent crop rows is calculated, and obstacles between crop rows are identified. Finally, the optimal detection path (target detection path) is planned for the ground unmanned vehicle by combining the location information of the fruit trees, the crop row edge information, and the obstacle information.
[0079] Optionally, the method further includes:
[0080] Based on the harvesting strategy for the target fruit types in the target orchard, harvesting vehicles are dispatched to carry out harvesting operations in the target orchard.
[0081] Specifically, in this embodiment of the invention, after determining the harvesting strategy for the target fruit in the target orchard, harvesting vehicles can be dispatched to carry out harvesting operations in the target orchard based on the harvesting strategy.
[0082] For example, based on this harvesting strategy, the first batch of harvesting tasks is sent to the harvesting vehicle. After the current batch of harvesting tasks is completed, the location information of the fruit targets in the target orchard is updated. When the next batch of fruit targets is harvested, a new task is sent to the harvesting vehicle again. This process is repeated until all batches of fruit targets are harvested.
[0083] Optionally, the harvesting vehicle in this embodiment of the invention can be an unmanned harvesting vehicle.
[0084] The air-ground collaborative fruit target harvesting decision-making method provided by this invention first determines the target detection path for a ground-based unmanned vehicle based on first image information collected over the target orchard. Then, it controls the ground-based unmanned vehicle equipped with a visual sensor to collect second image information within the target orchard based on the target detection path. Next, based on the second and first image information, it determines the yield distribution and maturity of the fruit targets within the target orchard. Finally, based on the yield distribution and maturity of the fruit targets, as well as the terrain information of the target orchard, it determines the harvesting strategy for the fruit targets within the target orchard. According to this harvesting strategy, accurate, efficient, and automated harvesting of fruit targets with different yield distributions and different maturity levels can be achieved.
[0085] The air-ground coordinated fruit target harvesting decision system provided by the present invention will be described below. The air-ground coordinated fruit target harvesting decision system described below can be referred to in correspondence with the air-ground coordinated fruit target harvesting decision method described above.
[0086] Figure 2 This is a schematic diagram of the air-ground coordinated fruit target harvesting decision system provided by the present invention, as shown below. Figure 2As shown, the system includes: a drone equipped with a visual sensor, a ground-based unmanned vehicle equipped with a visual sensor, and a ground station management platform; wherein:
[0087] The drone is used to collect first image information over the target orchard based on the shooting instructions sent by the ground station management platform, and to send the first image information to the ground station management platform;
[0088] The ground unmanned vehicle is used to collect second image information in the target orchard based on the target detection path determined by the ground station management platform, and send the second image information to the ground station management platform;
[0089] The ground station management platform is used to determine the target detection path based on the first image information and send the target detection path to the ground unmanned vehicle. It is also used to determine the yield distribution and maturity of fruit targets in the target orchard based on the second image information and the first image information, and to determine the harvesting strategy of fruit targets in the target orchard based on the yield distribution and maturity of the fruit targets and the terrain information of the target orchard, wherein the terrain information of the target orchard is determined based on the first image information.
[0090] Specifically, to overcome the shortcomings of existing fruit harvesters, such as their inability to accurately harvest fruits at different maturity levels, low harvesting quality, and limited efficiency improvement due to their inability to differentiate harvesting tasks for different yield areas within the orchard, this invention utilizes a drone equipped with a visual sensor to collect first image information from above the target orchard and transmit this information to a ground station management platform. Based on this first image information, the ground station management platform determines a target detection path for a ground-based unmanned vehicle. Then, the ground-based unmanned vehicle, also equipped with a visual sensor, collects second image information within the target orchard based on this target detection path and transmits it to the ground station management platform. Furthermore, based on the second and first image information, the ground station management platform determines the yield distribution and maturity of the fruits within the target orchard. Finally, based on the yield distribution, maturity, and terrain information of the fruits, a harvesting strategy is determined for the fruits within the target orchard. This harvesting strategy enables precise, efficient, and automated harvesting of fruits with different yield distributions and maturity levels.
[0091] It should be noted that during the process of the ground-based unmanned vehicle collecting second image information in the target orchard based on the target detection path, the detection height of the visual sensor mounted on the ground-based unmanned vehicle can be adjusted according to the canopy height of the crop row and the terrain information to ensure that the fruit tree canopy is always within the field of view of the visual sensor.
[0092] It is understood that, through the collaboration of a ground station management platform, a drone equipped with a visual sensor, and a ground unmanned vehicle equipped with a visual sensor, the embodiments of the present invention can accurately and efficiently complete the decision-making for the harvesting of fruit targets.
[0093] Optionally, the drone equipped with a visual sensor may include a drone flight platform, a first control module, and a first visual detection module, wherein:
[0094] The drone flight platform consists of a drone and a bracket responsible for fixing the corresponding components. The bracket is fixed to the bottom of the drone and is equipped with a first visual detection module and a first control module.
[0095] The first control module includes a control unit and a communication unit. The control unit is used to control the UAV's flight attitude and detection path, and the communication unit is used for the UAV to communicate with the ground station management platform, including receiving operation instructions and transmitting image information.
[0096] The first visual detection module includes a visual sensor and a gyroscope. The visual sensor is responsible for collecting image information within the field of view, and the gyroscope is responsible for monitoring the attitude information of the visual sensor in real time. Then, the ground station management platform can complete the preliminary target orchard map drawing based on the UAV's flight path, the image data of the visual sensor, and the attitude information obtained by the gyroscope. Based on the drawn target orchard map, the platform can determine the target detection path for the ground unmanned vehicle.
[0097] Optionally, the unmanned ground vehicle equipped with visual sensors may include a ground motion platform, a second control module, and a second visual detection module, wherein:
[0098] The ground motion platform is responsible for carrying the entire ground unmanned vehicle. The motion platform is equipped with a gyroscope to collect the position and posture information of the ground unmanned vehicle in real time, and the motion platform is also equipped with a bracket to fix the visual sensor.
[0099] The second control module includes a communication unit and a control unit. The control unit is responsible for tracking and controlling the navigation path of the ground unmanned vehicle, and the communication unit is responsible for data transmission and receiving of task instructions between the ground unmanned vehicle and the ground station management platform.
[0100] The second visual detection module has the same composition and function as the first visual detection module of the UAV, and will not be described again here.
[0101] Optionally, in this embodiment of the invention, the ground station management platform can receive instructions from the user to make harvesting decisions for fruit targets in the target orchard. After receiving the instructions from the user, the ground station management platform can send a shooting instruction to the drone to take aerial photos of the target orchard. After receiving the shooting instruction, the drone will collect first image information in the air above the target orchard and send the first image information to the ground station management platform so that the ground station management platform can determine the target detection path for the ground drone based on the first image information. Then, based on the first image information and the second image information collected by the ground drone along the target detection path, the yield distribution and maturity of the fruit targets in the target orchard can be determined. Based on the yield distribution and maturity of the fruit targets and the terrain information of the target orchard, the harvesting strategy for the fruit targets in the target orchard can be determined.
[0102] Figure 3 This is a schematic diagram of image information acquisition provided by the present invention, such as... Figure 3 As shown, the RTK+ZED camera on the drone can capture a top view of the fruit tree, and the RTK+ZED camera on the ground unmanned vehicle can capture a side view of the fruit tree. Furthermore, the RTK+ZED camera on the ground unmanned vehicle can capture multiple side views of the fruit tree from multiple angles, so that the ground management platform can more accurately determine the yield distribution and maturity of the target fruit in the target orchard based on the top view and multiple side views of the fruit tree.
[0103] The air-ground collaborative fruit target harvesting decision system provided by this invention uses a drone equipped with a visual sensor to collect first image information over a target orchard and sends this first image information to a ground station management platform. Based on this first image information, the ground station management platform determines a target detection path for a ground-based unmanned vehicle. Then, the ground-based unmanned vehicle, also equipped with a visual sensor, collects second image information within the target orchard based on this target detection path and sends it to the ground station management platform. Further, based on the second and first image information, the ground station management platform determines the yield distribution and maturity of the fruit targets within the target orchard. Finally, based on the yield distribution, maturity, and terrain information of the fruit targets, a harvesting strategy is determined for the fruit targets within the target orchard. According to this harvesting strategy, accurate, efficient, and automated harvesting of fruit targets with different yield distributions and maturity levels can be achieved.
[0104] Optionally, the ground station management platform is specifically used for:
[0105] Based on the yield distribution and maturity of the target fruits, as well as the terrain information of the target orchard, the harvesting batches of the target fruits in each preset area of the target orchard, the number of harvesting vehicles required to harvest the target fruits in each preset area, and the operation path of each harvesting vehicle for harvesting the target fruits are determined.
[0106] Specifically, in this embodiment of the invention, after the ground station management platform determines the yield distribution and maturity of fruit targets in the target orchard based on the second image information and the first image information, it can determine the harvesting batch of fruit targets in each preset area of the target orchard, the number of harvesting vehicles required to harvest fruit targets in each preset area, and the operation path of each harvesting vehicle for harvesting the fruit targets, based on the yield distribution and maturity of the fruit targets and the terrain information of the target orchard.
[0107] Optionally, the ground station management platform is also specifically used for:
[0108] Based on the first image information, the location information of each fruit tree in the target orchard and the terrain information of the target orchard are determined;
[0109] Based on the location information of each fruit tree and the terrain information of the target orchard, the target detection path is determined.
[0110] Specifically, in this embodiment of the invention, after the ground station management platform receives the first image information collected by the UAV over the target orchard, it can determine the location information of each fruit tree in the target orchard and the terrain information of the target orchard based on the first image information, and then determine the target detection path for the ground unmanned vehicle based on the location information of each fruit tree and the terrain information of the target orchard.
[0111] Optionally, the system further includes a harvester controller and a harvester;
[0112] The harvester controller is used to dispatch the harvester to carry out harvesting operations in the target orchard based on the harvesting strategy of the target fruit species determined by the ground station management platform.
[0113] Specifically, the air-ground collaborative fruit target harvesting decision system provided in this embodiment of the invention may further include a harvesting vehicle controller and a harvesting vehicle, wherein the harvesting vehicle controller may dispatch harvesting vehicles to carry out harvesting operations in the target orchard based on the harvesting strategy of the target fruit targets in the target orchard determined by the ground station management platform.
[0114] For example, after the ground management platform determines the harvesting strategy for the target fruit in the target orchard, it sends the harvesting strategy to the harvesting vehicle controller. Then, based on the harvesting strategy, the harvesting vehicle controller sends the first batch of harvesting tasks to the harvesting vehicle. After the current batch of harvesting tasks is completed, the ground management platform updates the location information of the target fruit in the target orchard. When the next batch of fruit is harvested, a new task is sent to the harvesting vehicle again. This process is repeated until all batches of fruit are harvested.
[0115] Optionally, the harvesting vehicle in this embodiment of the invention can be an unmanned harvesting vehicle.
[0116] Taking the harvesting of camellia oleifera fruit as an example, Figure 4 This is a schematic diagram of the harvesting process of camellia oleifera fruit using the air-ground coordinated fruit target harvesting decision system provided by the present invention, as shown in the figure. Figure 4 As shown, it includes:
[0117] Step 400: The drone equipped with a visual sensor collects first image information over the camellia orchard and can send the first image information to the ground station management platform.
[0118] Step 410: The ground station management platform processes the first image information and plans the target detection path for the ground unmanned vehicle;
[0119] Step 420: The ground unmanned vehicle equipped with a visual sensor collects second image information in the camellia orchard based on the target detection path, and can send the second image information to the ground station management platform.
[0120] Step 430: The ground station management platform generates a camellia fruit harvesting strategy based on the first image information and the second image information;
[0121] Step 440: The unmanned harvesting vehicle performs harvesting operations based on the harvesting strategy.
[0122] Furthermore, the ground station management platform can determine whether the unmanned harvesting vehicle has harvested all batches of camellia oleifera fruit. If it is determined that not all batches of camellia oleifera fruit have been harvested, the unmanned harvesting vehicle will continue to be dispatched to carry out harvesting operations based on the harvesting strategy until the unmanned harvesting vehicle has harvested all batches of camellia oleifera fruit.
[0123] It should be noted that the ground station management platform can obtain the specific location of the camellia trees in the entire park, the location of the camellia fruits in the canopy above the trees, and their maturity information based on the first image information; the ground station management platform can obtain the location of the camellia fruits in the middle and lower canopies of the camellia trees in the entire park, and their maturity information based on the second image information.
[0124] Therefore, the ground station management platform can obtain the distribution and maturity of camellia fruits in the entire camellia oil garden based on the first and second image information.
[0125] It should be noted that since immature camellia fruits are green and mature camellia fruits are greenish-brown, the ground station management platform can convert the first image information and the second image information into an RGB color model based on the different colors of camellia fruits at different stages of maturity, and then determine the ratio of the area of the camellia fruit image to the area of the original image. Furthermore, based on the size of the ratio and the preset threshold, the maturity of the camellia fruits is classified.
[0126] Optionally, the ground station management platform can generate a three-dimensional harvesting map based on the camellia oleifera garden data in the first and second image information. This map includes the specific location information of the camellia oleifera fruits, their maturity status, the yield status of each area, the canopy height of the crop rows, and the topographic map of the camellia oleifera garden. Then, based on the maturity status of the camellia oleifera fruits, the harvesting batches and the harvesting time of different batches are planned. Finally, based on the specific yield status of different areas in each harvesting batch, the corresponding number of unmanned harvesting vehicles are allocated to each batch. This process is repeated until all batches of camellia oleifera fruits are harvested.
[0127] Optionally, in this embodiment of the invention, after generating a three-dimensional harvesting map, the ground station management platform can divide the camellia oleifera garden into regions according to crop rows, and allocate a corresponding number of harvesting vehicles to each region based on the number of camellia oleifera fruits in each region.
[0128] Optionally, the ground station management platform can statistically analyze the yield distribution and maturity of camellia fruits within the camellia oleifera garden. Then, based on the yield and maturity of camellia fruits in each area, as well as previously obtained terrain information, it can plan the harvesting batches of camellia fruits in the garden, the number of harvesting vehicles to be dispatched to each area, and the operating routes of the harvesting vehicles. After the planning is completed, the ground station management platform sends the first batch of harvesting tasks to the harvesting vehicle controller. After the current batch of camellia fruits is harvested, the ground station management platform updates the location information of the camellia fruits in the garden. When the next batch of camellia fruits is harvested, a new task is sent to the unmanned harvesting vehicle. This cycle is repeated until all batches of camellia fruits are harvested.
[0129] The air-ground collaborative fruit target harvesting decision system provided by this invention uses a drone equipped with a visual sensor to collect first image information over a target orchard and sends this first image information to a ground station management platform. Based on this first image information, the ground station management platform determines a target detection path for a ground-based unmanned vehicle. Then, the ground-based unmanned vehicle, also equipped with a visual sensor, collects second image information within the target orchard based on this target detection path and sends it to the ground station management platform. Further, based on the second and first image information, the ground station management platform determines the yield distribution and maturity of the fruit targets within the target orchard. Finally, based on the yield distribution, maturity, and terrain information of the fruit targets, a harvesting strategy is determined for the fruit targets within the target orchard. According to this harvesting strategy, accurate, efficient, and automated harvesting of fruit targets with different yield distributions and maturity levels can be achieved.
[0130] The air-ground coordinated fruit target harvesting decision device provided by the present invention will be described below. The air-ground coordinated fruit target harvesting decision device described below and the air-ground coordinated fruit target harvesting decision method described above can be referred to in correspondence.
[0131] Figure 5 This is a schematic diagram of the air-ground coordinated fruit target harvesting decision-making device provided by the present invention, as shown below. Figure 5 As shown, the device includes: a first determining module 510, an image acquisition module 520, a second determining module 530, and a third determining module 540; wherein:
[0132] The first determining module 510 is used to determine the target detection path based on the first image information collected over the target orchard;
[0133] The image acquisition module 520 is used to control a ground unmanned vehicle equipped with a vision sensor to acquire second image information in the target orchard based on the target detection path;
[0134] The second determining module 530 is used to determine the yield distribution and maturity of the target fruit in the target orchard based on the second image information and the first image information.
[0135] The third determining module 540 is used to determine the harvesting strategy for the fruit targets in the target orchard based on the yield distribution and maturity of the fruit targets, as well as the terrain information of the target orchard. The terrain information of the target orchard is determined based on the first image information.
[0136] The air-ground collaborative fruit target harvesting decision device provided by this invention first determines the target detection path for a ground-based unmanned vehicle based on first image information collected above the target orchard. Then, it controls the ground-based unmanned vehicle equipped with a visual sensor to collect second image information within the target orchard based on the target detection path. Then, based on the second image information and the first image information, it determines the yield distribution and maturity of the fruit targets within the target orchard. Finally, based on the yield distribution and maturity of the fruit targets, as well as the terrain information of the target orchard, it determines the harvesting strategy for the fruit targets within the target orchard. According to the harvesting strategy, accurate, efficient and automated harvesting of fruit targets with different yield distributions and different maturity levels can be achieved.
[0137] Optionally, the third determining module 540 is specifically used for:
[0138] Based on the yield distribution and maturity of the target fruits, as well as the terrain information of the target orchard, the harvesting batches of the target fruits in each preset area of the target orchard, the number of harvesting vehicles required to harvest the target fruits in each preset area, and the operation path of each harvesting vehicle for harvesting the target fruits are determined.
[0139] Optionally, the first determining module 510 is specifically used for:
[0140] Based on the first image information, the location information of each fruit tree in the target orchard and the terrain information of the target orchard are determined;
[0141] Based on the location information of each fruit tree and the terrain information of the target orchard, the target detection path is determined.
[0142] Optionally, the apparatus further includes a harvesting module, the harvesting module being used for:
[0143] Based on the harvesting strategy for the target fruit types in the target orchard, harvesting vehicles are dispatched to carry out harvesting operations in the target orchard.
[0144] The air-ground collaborative fruit target harvesting decision device provided by this invention first determines the target detection path for a ground-based unmanned vehicle based on first image information collected above the target orchard. Then, it controls the ground-based unmanned vehicle equipped with a visual sensor to collect second image information within the target orchard based on the target detection path. Then, based on the second image information and the first image information, it determines the yield distribution and maturity of the fruit targets within the target orchard. Finally, based on the yield distribution and maturity of the fruit targets, as well as the terrain information of the target orchard, it determines the harvesting strategy for the fruit targets within the target orchard. According to the harvesting strategy, accurate, efficient and automated harvesting of fruit targets with different yield distributions and different maturity levels can be achieved.
[0145] It should be noted that the air-ground coordinated fruit target harvesting decision device provided in the embodiments of the present invention can realize all the method steps implemented in the air-ground coordinated fruit target harvesting decision method embodiments and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiments and the beneficial effects will not be described in detail.
[0146] Figure 6 This is a schematic diagram of the physical structure of the electronic device provided by the present invention, such as... Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute the air-ground coordinated fruit target harvesting decision-making method provided by the above methods, which includes:
[0147] Based on the first image information collected over the target orchard, the target detection path is determined;
[0148] Controlling a ground-based unmanned vehicle equipped with a visual sensor, based on the target detection path, to collect second image information within the target orchard;
[0149] Based on the second image information and the first image information, the yield distribution and maturity of the target fruits in the target orchard are determined;
[0150] Based on the yield distribution and maturity of the target fruits, as well as the topographic information of the target orchard, a harvesting strategy for the target fruits within the target orchard is determined. The topographic information of the target orchard is determined based on the first image information.
[0151] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0152] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer is able to execute the air-ground coordinated fruit target harvesting decision-making method provided by the above methods, the method comprising:
[0153] Based on the first image information collected over the target orchard, the target detection path is determined;
[0154] Controlling a ground-based unmanned vehicle equipped with a visual sensor, based on the target detection path, to collect second image information within the target orchard;
[0155] Based on the second image information and the first image information, the yield distribution and maturity of the target fruits in the target orchard are determined;
[0156] Based on the yield distribution and maturity of the target fruits, as well as the topographic information of the target orchard, a harvesting strategy for the target fruits within the target orchard is determined. The topographic information of the target orchard is determined based on the first image information.
[0157] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described air-ground coordinated fruit target harvesting decision-making methods, the method comprising:
[0158] Based on the first image information collected over the target orchard, the target detection path is determined;
[0159] Controlling a ground-based unmanned vehicle equipped with a visual sensor, based on the target detection path, to collect second image information within the target orchard;
[0160] Based on the second image information and the first image information, the yield distribution and maturity of the target fruits in the target orchard are determined;
[0161] Based on the yield distribution and maturity of the target fruits, as well as the topographic information of the target orchard, a harvesting strategy for the target fruits within the target orchard is determined. The topographic information of the target orchard is determined based on the first image information.
[0162] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0163] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0164] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for making decisions on the harvesting of fruit using a combination of air and ground-based methods, characterized in that, include: Based on the first image information collected over the target orchard, the target detection path is determined; Controlling an unmanned ground vehicle equipped with a visual sensor, based on the target detection path, to collect second image information within the target orchard; Based on the second image information and the first image information, the yield distribution and maturity of the target fruits in the target orchard are determined; Based on the yield distribution and maturity of the target fruits, as well as the topographic information of the target orchard, a harvesting strategy for the target fruits in the target orchard is determined. The topographic information of the target orchard is determined based on the first image information. The method of determining the harvesting strategy for the target fruits within the target orchard based on the yield distribution and maturity of the target fruits, as well as the topographic information of the target orchard, includes: Based on the yield distribution and maturity of the target fruits, and the topographic information of the target orchard, the harvesting batches of the target fruits in each preset area within the target orchard, the number of harvesting vehicles required to harvest the target fruits in each preset area, and the operating routes of each harvesting vehicle for harvesting the target fruits are determined, including: The first batch of fruit targets in the target orchard with a maturity greater than a first preset value is harvested; the second batch of fruit targets in the target orchard with a maturity greater than a second preset value but less than the first preset value is harvested; and the third batch of fruit targets in the target orchard with a maturity less than the second preset value is harvested. The process of determining the target detection path based on the first image information collected over the target orchard includes: Based on the first image information, the location information of each fruit tree in the target orchard and the terrain information of the target orchard are determined. Based on the location information of each fruit tree and the terrain information of the target orchard, the target detection path is determined, including: Based on the first image information, the edge information of the crop rows of fruit trees in the target orchard is obtained by using an edge detection algorithm. Then, the data of the fruit tree canopy is filtered out based on a preset threshold. The slope between adjacent crop rows is calculated after plane fitting of the filtered data, and obstacles between crop rows are identified. The target detection path is planned for the ground unmanned vehicle by combining the location information of each fruit tree, the crop row edge information and the obstacle information. The first image information includes the distribution information of fruit targets in the canopy above the fruit tree, and the second image information includes the distribution information of fruit targets in the middle and lower canopies of the fruit tree. The yield distribution and maturity of the fruit targets are determined by fusing the second image information and the first image information through image stitching technology to generate a complete three-dimensional map. The terrain information is determined based on the first image.
2. The air-ground coordinated fruit target harvesting decision-making method according to claim 1, characterized in that, The method further includes: Based on the harvesting strategy for the target fruit types in the target orchard, harvesting vehicles are dispatched to carry out harvesting operations in the target orchard.
3. A fruit harvesting decision-making system with air-ground coordination, characterized in that, include: Unmanned aerial vehicles equipped with visual sensors, unmanned ground vehicles equipped with visual sensors, and ground station management platforms; The drone is used to collect first image information over the target orchard based on the shooting instructions sent by the ground station management platform, and to send the first image information to the ground station management platform; The ground unmanned vehicle is used to collect second image information in the target orchard based on the target detection path determined by the ground station management platform, and send the second image information to the ground station management platform; The ground station management platform is used to determine the target detection path based on the first image information and send the target detection path to the ground unmanned vehicle. It is also used to determine the yield distribution and maturity of fruit targets in the target orchard based on the second image information and the first image information, and to determine the harvesting strategy of fruit targets in the target orchard based on the yield distribution and maturity of the fruit targets and the terrain information of the target orchard. The terrain information of the target orchard is determined based on the first image information. The method of determining the harvesting strategy for the target fruits within the target orchard based on the yield distribution and maturity of the target fruits, as well as the topographic information of the target orchard, includes: Based on the yield distribution and maturity of the target fruits, and the topographic information of the target orchard, the harvesting batches of the target fruits in each preset area within the target orchard, the number of harvesting vehicles required to harvest the target fruits in each preset area, and the operating routes of each harvesting vehicle for harvesting the target fruits are determined, including: The first batch of fruit targets in the target orchard with a maturity greater than a first preset value is harvested; the second batch of fruit targets in the target orchard with a maturity greater than a second preset value but less than the first preset value is harvested; and the third batch of fruit targets in the target orchard with a maturity less than the second preset value is harvested. The process of determining the target detection path based on the first image information collected over the target orchard includes: Based on the first image information, the location information of each fruit tree in the target orchard and the terrain information of the target orchard are determined. Based on the location information of each fruit tree and the terrain information of the target orchard, the target detection path is determined, including: Based on the first image information, the edge information of the crop rows of fruit trees in the target orchard is obtained by using an edge detection algorithm. Then, the data of the fruit tree canopy is filtered out based on a preset threshold. The slope between adjacent crop rows is calculated after plane fitting of the filtered data, and obstacles between crop rows are identified. The target detection path is planned for the ground unmanned vehicle by combining the location information of each fruit tree, the crop row edge information and the obstacle information. The first image information includes the distribution information of fruit targets in the canopy above the fruit tree, and the second image information includes the distribution information of fruit targets in the middle and lower canopies of the fruit tree. The yield distribution and maturity of the fruit targets are determined by fusing the second image information and the first image information through image stitching technology to generate a complete three-dimensional map. The terrain information is determined based on the first image.
4. The air-ground coordinated fruit target harvesting decision system according to claim 3, characterized in that, The system also includes a harvester controller and a harvester; The harvester controller is used to dispatch the harvester to carry out harvesting operations in the target orchard based on the harvesting strategy of the target fruit species determined by the ground station management platform.
5. A decision-making device for fruit harvesting using a combination of air and ground-based methods, characterized in that, include: The first determining module is used to determine the target detection path based on the first image information collected over the target orchard; The image acquisition module is used to control a ground unmanned vehicle equipped with a visual sensor to acquire second image information in the target orchard based on the target detection path; The second determining module is used to determine the yield distribution and maturity of the target fruit in the target orchard based on the second image information and the first image information. The third determining module is used to determine the harvesting strategy for the fruit targets in the target orchard based on the yield distribution and maturity of the fruit targets, as well as the terrain information of the target orchard. The terrain information of the target orchard is determined based on the first image information. The method of determining the harvesting strategy for the target fruits within the target orchard based on the yield distribution and maturity of the target fruits, as well as the topographic information of the target orchard, includes: Based on the yield distribution and maturity of the target fruits, and the topographic information of the target orchard, the harvesting batches of the target fruits in each preset area within the target orchard, the number of harvesting vehicles required to harvest the target fruits in each preset area, and the operating routes of each harvesting vehicle for harvesting the target fruits are determined, including: The first batch of fruit targets in the target orchard with a maturity greater than a first preset value is harvested; the second batch of fruit targets in the target orchard with a maturity greater than a second preset value but less than the first preset value is harvested; and the third batch of fruit targets in the target orchard with a maturity less than the second preset value is harvested. The process of determining the target detection path based on the first image information collected over the target orchard includes: Based on the first image information, the location information of each fruit tree in the target orchard and the terrain information of the target orchard are determined. Based on the location information of each fruit tree and the terrain information of the target orchard, the target detection path is determined, including: Based on the first image information, the edge information of the crop rows of fruit trees in the target orchard is obtained by using an edge detection algorithm. Then, the data of the fruit tree canopy is filtered out based on a preset threshold. The slope between adjacent crop rows is calculated after plane fitting of the filtered data, and obstacles between crop rows are identified. The target detection path is planned for the ground unmanned vehicle by combining the location information of each fruit tree, the crop row edge information and the obstacle information. The first image information includes the distribution information of fruit targets in the canopy above the fruit tree, and the second image information includes the distribution information of fruit targets in the middle and lower canopies of the fruit tree. The yield distribution and maturity of the fruit targets are determined by fusing the second image information and the first image information through image stitching technology to generate a complete three-dimensional map. The terrain information is determined based on the first image.
6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the air-ground coordinated fruit target harvesting decision method as described in any one of claims 1 to 2.