Air-ground cooperative picking system and method

Abstract

The invention provides an air-ground cooperative picking system and method. The air-ground collaborative picking system comprises an unmanned aerial vehicle and an intelligent agricultural machine, the unmanned aerial vehicle collects a first aerial image of an orchard, and a two-dimensional aerial map is constructed; the intelligent agricultural machine obtains a two-dimensional radar grid map of the orchard and sends the two-dimensional radar grid map to the unmanned aerial vehicle; the unmanned aerial vehicle splices the two-dimensional aerial photography map and the two-dimensional radar grid map into a navigation planning map; the unmanned aerial vehicle performs maturity analysis on the navigation planning map by using a first identification model to obtain a to-be-picked task list; the unmanned aerial vehicle sends the to-be-picked task list to the intelligent agricultural machine; the intelligent agricultural machine determines a task path according to the to-be-picked task list and uses a second recognition model to recognize the to-be-picked target based on image data obtained on the task path so as to perform picking operation, and the first recognition model is the same as the second recognition model. The system can improve the fruit harvesting accuracy.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an air-ground collaborative picking system and method. Background Art

[0002] Orchards refer to agricultural land specifically used for growing fruit trees. They are used to produce fruit. Among them, large orchards can better meet market demand and enhance market competitiveness due to their large scale and high yield.

[0003] The picking of fruits in large orchards mostly relies on manual picking, but manual picking is time-consuming and labor-intensive. In contrast, the use of machine-assisted harvesting can achieve less or even no work, thus improving picking efficiency.

[0004] However, current machine-assisted fruit harvesting is not very accurate. Summary of the invention

[0005] The main purpose of the embodiments of the present application is to propose an air-ground collaborative picking system and method, aiming to improve the accuracy of harvesting fruits.

[0006] To achieve the above objectives, the present application proposes an air-ground collaborative harvesting system, which includes a drone and an intelligent agricultural machine, including:

[0007] The drone collected the first aerial images of the orchard and constructed a two-dimensional aerial map;

[0008] The intelligent agricultural machinery obtains a 2D radar grid map of the orchard and sends the 2D radar grid map to the drone;

[0009] The drone stitches the 2D aerial map and the 2D radar grid map into a navigation planning map;

[0010] The drone uses the first recognition model to perform maturity analysis on the navigation planning map and obtains a list of tasks to be picked;

[0011] The drone sends the list of tasks to be harvested to the smart agricultural machinery;

[0012] The intelligent agricultural machinery determines the task path according to the list of tasks to be picked, and uses the second recognition model to identify the targets to be picked based on the image data obtained on the task path to perform the picking operation. The first recognition model is the same as the second recognition model.

[0013] Optionally, before the UAV uses the first recognition model to perform maturity analysis on the navigation planning map to obtain a list of tasks to be picked, the system further includes:

[0014] The drone collected a second aerial image of the orchard;

[0015] The drone obtains the annotation of the second aerial image to obtain an annotated image;

[0016] The drone uses the labeled images to train the initial model and obtain the first recognition model;

[0017] The drone sends the first recognition model to the intelligent agricultural machine so that the intelligent agricultural machine determines the second recognition model.

[0018] Optionally, the drone uses the labeled image to train the initial model to obtain a first recognition model, including:

[0019] The drone performs data augmentation on the annotated image to obtain augmented data;

[0020] The drone uses the augmented data to train the initial model and obtain the first recognition model.

[0021] Optionally, the drone stitches the two-dimensional aerial map and the two-dimensional radar grid map into a navigation planning map, including:

[0022] The UAV determines the common feature points of the two-dimensional aerial map and the two-dimensional radar grid map;

[0023] The UAV aligns the two-dimensional aerial map and the two-dimensional radar grid map according to common feature points to obtain a navigation planning map.

[0024] Optionally, based on the image data acquired on the task path, a second recognition model is used to identify the target to be picked so as to perform a picking operation, including:

[0025] When the intelligent agricultural machinery moves to the area to be picked based on the task path, it collects image data and three-dimensional point cloud data, and the image data includes two-dimensional images;

[0026] The intelligent agricultural machinery uses the second recognition model to recognize the two-dimensional image and obtains the target to be picked;

[0027] The intelligent agricultural machinery maps the target to be picked into the three-dimensional point cloud data to obtain the spatial position information of the target to be picked;

[0028] Intelligent agricultural machinery controls the robotic arm to pick the objects to be picked based on the spatial position information.

[0029] Optionally, the intelligent agricultural machine determines a task path according to the list of tasks to be harvested, including:

[0030] The intelligent agricultural machinery is positioned according to the real-time dynamic difference algorithm to obtain the first position;

[0031] The intelligent agricultural machinery registers the first pose according to the normal distribution transformation algorithm to obtain the second pose;

[0032] The intelligent agricultural machinery performs path planning based on the second posture and the list of tasks to be picked to obtain the task path.

[0033] Optionally, the intelligent agricultural machine obtains a two-dimensional radar grid map of the orchard, including:

[0034] The intelligent agricultural machinery obtains the two-dimensional point cloud grid map of the orchard and obtains the control points in the two-dimensional point cloud grid map;

[0035] Intelligent agricultural machinery obtains the location information of control points;

[0036] The intelligent agricultural machinery calibrates the two-dimensional point cloud grid map according to the position information of the control points to obtain a two-dimensional radar grid map.

[0037] Optionally, the smart agricultural machinery and the drone are aligned using the same satellite timestamp.

[0038] To achieve the above objectives, the present application proposes, on the other hand, a method for collaborative picking between air and ground, comprising:

[0039] Collect the first aerial image of the orchard and construct a two-dimensional aerial map;

[0040] Obtain the two-dimensional radar grid map sent by intelligent agricultural machinery;

[0041] Splice the two-dimensional aerial map and the two-dimensional radar grid map into a navigation planning map;

[0042] Use the first recognition model to perform maturity analysis on the navigation planning map to obtain a list of tasks to be picked;

[0043] The list of tasks to be picked is sent to the intelligent agricultural machine, so that the intelligent agricultural machine determines the task path according to the list of tasks to be picked, and uses the second recognition model on the task path to identify the target to be picked to perform the picking operation, and the first recognition model is the same as the second recognition model.

[0044] To achieve the above objectives, the present application proposes, on the other hand, a method for collaborative picking between air and ground, comprising:

[0045] Obtain a 2D radar grid map of the orchard and send it to the drone;

[0046] Receive a list of pending picking tasks sent by the drone, where the list of pending picking tasks is a mosaic map of a two-dimensional aerial map of the orchard and a two-dimensional radar grid map of the drone, obtained by analyzing the first recognition model;

[0047] Determine the task path based on the list of tasks to be picked;

[0048] Based on the image data acquired on the task path, the second recognition model is used to recognize the target to be picked so as to perform the picking operation, and the first recognition model is the same as the second recognition model.

[0049] To achieve the above-mentioned purpose, the present application proposes, on the other hand, an air-ground collaborative picking device, comprising:

[0050] A collection unit, used for collecting a first aerial image of the orchard and constructing a two-dimensional aerial map;

[0051] An acquisition unit, used for acquiring a two-dimensional radar grid map sent by the intelligent agricultural machinery;

[0052] A splicing unit, used for splicing the two-dimensional aerial map and the two-dimensional radar grid map into a navigation planning map;

[0053] A planning unit, configured to perform maturity analysis on the navigation planning map using the first recognition model to obtain a list of tasks to be picked;

[0054] The sending unit is used to send the list of tasks to be picked to the intelligent agricultural machinery, so that the intelligent agricultural machinery determines the task path according to the list of tasks to be picked, and uses the second recognition model on the task path to identify the targets to be picked to perform the picking operation, and the first recognition model is the same as the second recognition model.

[0055] To achieve the above-mentioned purpose, the present application proposes, on the other hand, an air-ground collaborative picking device, comprising:

[0056] An acquisition unit, used for acquiring a two-dimensional radar grid map of the orchard and sending it to the UAV;

[0057] A receiving unit is used to receive a list of pending picking tasks sent by the drone, where the list of pending picking tasks is a mosaic map of a two-dimensional aerial map of the orchard and a two-dimensional radar grid map of the drone, obtained by analyzing using a first recognition model;

[0058] A determination unit, used to determine a task path according to a list of tasks to be picked;

[0059] The picking unit is used to identify the target to be picked based on the image data obtained on the task path using the second recognition model to perform the picking operation, and the first recognition model is the same as the second recognition model.

[0060] To achieve the above-mentioned purpose, the present application proposes an electronic device on the other hand, which includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned open-air collaborative picking method when executing the computer program.

[0061] To achieve the above objectives, the present application proposes, on the other hand, a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned open-air collaborative picking method is implemented.

[0062] The air-ground collaborative picking system of the present application includes a drone and an intelligent agricultural machinery, the drone collects a first aerial image of an orchard and constructs a two-dimensional aerial map; the intelligent agricultural machinery obtains a two-dimensional radar grid map of the orchard, and sends the two-dimensional radar grid map to the drone; the drone splices the two-dimensional aerial map and the two-dimensional radar grid map into a navigation planning map; the drone uses a first recognition model to perform a maturity analysis on the navigation planning map to obtain a list of tasks to be picked; the drone sends the list of tasks to be picked to the intelligent agricultural machinery; the intelligent agricultural machinery determines a task path according to the list of tasks to be picked, and uses a second recognition model to identify the target to be picked based on the image data obtained on the task path to perform a picking operation, and the first recognition model is the same as the second recognition model.

[0063] The system uses drones to stitch together aerial images of orchards and radar grid maps of agricultural machinery, and then uses a recognition model to identify tasks to be picked from the stitched map. The agricultural machinery can generate a picking path based on the tasks to be picked sent by the drone, and based on the images taken on the picking path, use the same recognition model as the drone to identify the targets to be picked for picking. The stitched map obtains more comprehensive orchard information, making the recognition results of the recognition model more accurate. The agricultural machinery uses the same model as the drone for recognition, further improving the accuracy of picking fruits. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is a system architecture diagram provided by an embodiment of the present application;

[0065] Figure 2 It is a schematic diagram of the interaction process of an air-ground collaborative picking system provided by an embodiment of the present application;

[0066] Figure 3 It is an interactive schematic diagram of an air-ground collaborative picking system provided by an embodiment of the present application;

[0067] Figure 4 It is a block diagram provided by an embodiment of the present application;

[0068] Figure 5 It is a data augmentation schematic diagram provided in an embodiment of the present application;

[0069] Figure 6 It is a schematic diagram of a generation process of a navigation planning map provided in an embodiment of the present application;

[0070] Figure 7 It is another schematic diagram of the generation process of the navigation planning map provided in the embodiment of the present application;

[0071] Figure 8 This is a schematic diagram of the output results of a drone usage maturity identification model provided by an embodiment of the present application;

[0072] Fig. 9 It is a schematic diagram of the recognition result of the intelligent agricultural machinery using the maturity recognition model to recognize the image taken by the camera provided by the embodiment of the present application;

[0073] Fig.10 It is a visualization diagram of the joint calibration coordinates of the camera and the laser radar provided in the embodiment of the present application;

[0074] Fig.11 It is a schematic diagram of a joint calibration process provided by an embodiment of the present application;

[0075] Fig.12 It is a schematic diagram of a scene of laser point cloud and image fusion provided in an embodiment of the present application;

[0076] Fig.13 It is a flow chart of an open-ground collaborative picking method provided in an embodiment of the present application;

[0077] Fig.14 It is a flowchart of another air-ground collaborative picking method provided by an embodiment of the present application;

[0078] Fig.15 It is a structural schematic diagram of an air-ground collaborative picking device provided in an embodiment of the present application;

[0079] Fig.16 is a structural schematic diagram of another air-ground collaborative picking device provided in an embodiment of the present application;

[0080] Fig.17 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0081] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "corresponding to" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0082] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0083] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0084] In addition, in order to better illustrate the present application, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present application can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present application.

[0085] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0086] The lychee picking process currently relies mainly on manual operation. Although it is common and traditional, it has many disadvantages. First of all, manual picking is time-consuming and labor-intensive, and its efficiency is extremely low in large orchards. In contrast, the use of agricultural machinery to assist harvesting can achieve less or even no people working. An automated picking machine can harvest about 400 kilograms a day, and can operate continuously at high intensity around the clock, greatly improving the picking efficiency. Especially in the face of natural disasters such as typhoons in the southern coastal areas, it can ensure the rapid harvest of ripe fruits. However, the existing agricultural machinery-assisted methods have a high fruit loss rate, and mechanical picking equipment can easily cause physical damage to the fruit, with a damage rate of up to 15%-20%, which is much higher than the 5%-10% of manual picking. This leads to damage to the surface of the lychee and a decline in internal quality, reducing the market value and affecting the storage and transportation quality.

[0087] The air-ground collaborative picking system in the embodiment of the present application includes: a drone collects a first aerial image of an orchard and constructs a two-dimensional aerial map; the intelligent agricultural machinery obtains a two-dimensional radar grid map of the orchard and sends the two-dimensional radar grid map to the drone; the drone splices the two-dimensional aerial map and the two-dimensional radar grid map into a navigation planning map; the drone uses a first recognition model to perform a maturity analysis on the navigation planning map to obtain a list of tasks to be picked; the drone sends the list of tasks to be picked to the intelligent agricultural machinery; the intelligent agricultural machinery determines a task path according to the list of tasks to be picked, and uses a second recognition model to identify the target to be picked based on the image data obtained on the task path to perform a picking operation, and the first recognition model is the same as the second recognition model.

[0088] The system uses drones to stitch together aerial images of orchards and radar grid maps of agricultural machinery, and then uses a recognition model to identify tasks to be picked from the stitched map. The agricultural machinery can generate a picking path based on the tasks to be picked sent by the drone, and based on the images taken on the picking path, use the same recognition model as the drone to identify the targets to be picked for picking. The stitched map obtains more comprehensive orchard information, making the recognition results of the recognition model more accurate. The agricultural machinery uses the same model as the drone for recognition, further improving the accuracy of picking fruits.

[0089] System architecture and scenario description of the application of the embodiments of the present disclosure

[0090] Figure 1 1 is a system architecture diagram of the air-ground collaborative harvesting method according to an embodiment of the present application, which includes a drone 110 and an intelligent agricultural machine 120.

[0091] Drone 110 is responsible for collecting aerial images of the orchard, which can be used to construct a two-dimensional aerial map to provide a macro perspective of the orchard.

[0092] The intelligent agricultural machinery 120 collects a two-dimensional radar grid map of the orchard and sends it to the drone 110. The drone 110 stitches the collected aerial images with the two-dimensional radar grid map to form a comprehensive navigation planning map. The drone 110 uses the first recognition model to analyze the navigation planning map, identify fruits with suitable maturity, and generate a list of tasks to be picked. The drone 110 sends the list of tasks to be picked to the intelligent agricultural machinery 120. The intelligent agricultural machinery 120 determines the task path based on the list of tasks to be picked sent by the drone, and based on the image data on the task path, uses the same second recognition model as the drone to identify the target to be picked, and performs the picking operation.

[0093] The air-ground collaborative picking method provided in the embodiment of the present application will be described below in conjunction with the accompanying drawings. The executor of the following prompt word processing method takes the terminal device as an example, and can be specifically implemented by the terminal device by running the various computer programs mentioned above; of course, based on the understanding of the following text, it is not difficult to see that the air-ground collaborative picking method provided in the embodiment of the present application can also be implemented collaboratively by the terminal device and the server.

[0094] See also Figure 2 ,like Figure 2 The figure shows an interactive flow diagram of an air-ground collaborative picking system provided by this embodiment, the system comprising:

[0095] Step 201: The drone collects a first aerial image of the orchard and constructs a two-dimensional aerial map.

[0096] The first aerial image may refer to an image of the ground taken from the air by a photographic device carried by a drone, an airplane or other aircraft. In the embodiment of the present application, the first aerial image may be an image obtained by a drone carrying a high-resolution camera photographing the orchard at a predetermined flight altitude and path.

[0097] A 2D aerial map is a flat map that uses aerial images to show the two-dimensional features of the earth's surface, that is, it displays geographic information in a flat form and does not include elevation data of the terrain. This map can show the layout of an orchard, including the distribution of fruit trees, roads, boundaries and other information.

[0098] The first aerial image can be processed to generate an orthoimage, which is a two-dimensional image with a vertical perspective that can eliminate the perspective difference caused by terrain undulations. The two-dimensional image processed based on multiple aerial images can be fused together to form a continuous and seamless map, which can be called a two-dimensional aerial map.

[0099] Step 202: The intelligent agricultural machine obtains a two-dimensional radar grid map of the orchard and sends the two-dimensional radar grid map to the drone.

[0100] A two-dimensional radar grid map can be a map constructed using radar data. It divides the environment into small grid cells and assigns a specific value to each grid to indicate the state of the area, such as whether it is occupied. Intelligent agricultural machinery equipment can be equipped with sensors such as laser radar (LiDAR), which can obtain three-dimensional point cloud data of the orchard. These point cloud data need to be processed and projected onto a two-dimensional plane to construct a two-dimensional map. The continuous two-dimensional map space is converted into discrete grid cells, each of which contains information about the area, thereby forming a grid map. This grid map can be called a two-dimensional radar grid map. Intelligent agricultural machinery can transmit the two-dimensional radar grid map to the drone through a network or a local area network. The order of step 202 and step 201 is not fixed.

[0101] In a feasible implementation manner, the intelligent agricultural machine obtains a two-dimensional radar grid map of an orchard, including:

[0102] The intelligent agricultural machinery obtains the two-dimensional point cloud grid map of the orchard and obtains the control points in the two-dimensional point cloud grid map;

[0103] Intelligent agricultural machinery obtains the location information of control points;

[0104] The intelligent agricultural machinery calibrates the two-dimensional point cloud grid map according to the position information of the control points to obtain a two-dimensional radar grid map.

[0105] In this implementation, the two-dimensional point cloud grid map is a data representation form that projects three-dimensional point cloud data onto a two-dimensional plane, and it characterizes the attributes of the map by dividing the space into uniform grids and storing specific values ​​in each grid. The three-dimensional point cloud data can be obtained by intelligent agricultural machinery through radar scanning of an orchard.

[0106] Control points can refer to points with obvious features and known coordinates on the map, and can be specific feature points in the orchard, such as road intersections, river bends, building corners, etc. The control points can be obtained by parsing and identifying objects based on aerial images and selecting some of the objects with obvious features and easy to locate, or they can be manually selected from a two-dimensional point cloud grid map.

[0107] The location information of the control points can be the two-dimensional coordinate information of each control point, which can include the two-dimensional coordinates (X, Y) of each control point, and can also include height information (Z). The location information of the control points can be determined by field investigation and measurement to determine the precise location of the control points. For example, using a surveying device such as a total station or GPS to obtain the coordinates of the control points. The location information of the control points can also be determined by airborne laser radar scanning of intelligent agricultural machinery.

[0108] The 2D point cloud grid map can be calibrated by building a coordinate transformation model based on the positions of the control points on the original image and the reference image. This model can be an affine transformation, a polynomial transformation, or other more complex geometric transformation model that maps the original coordinate system to the reference coordinate system. The coordinate transformation model is applied to the entire 2D point cloud grid map. This process can adjust the coordinates of each grid cell to ensure that the feature points on the map match the actual positions of the control points. Using the known positions of the control points and the corresponding feature points on the map, the map is corrected. The correction process can include rotating, translating, and scaling the map to minimize the error between the control points.

[0109] In a feasible implementation, the smart agricultural machinery and the UAV are aligned using the same satellite timestamp.

[0110] In this implementation, a common time base can be selected, usually satellite time (such as GPS time), and all sensor data on smart agricultural machinery and drones will be aligned with this time base. When the sensor data on smart agricultural machinery and drones are collected, the timestamp of each data point is recorded at the same time. If the timestamps of the sensors are not based on the same time base, or there is a deviation, these timestamps need to be calibrated.

[0111] Step 203: The UAV stitches the two-dimensional aerial map and the two-dimensional radar grid map into a navigation planning map.

[0112] The navigation planning map may refer to a map that has integrated multiple data sources and is ready for further analysis and processing. The navigation planning map in the embodiment of the present application may refer to the result after the splicing and fusion of the two-dimensional aerial map and the two-dimensional radar grid map. The two types of data may be pre-processed to ensure that they are suitable for splicing in quality, and the pre-processing may include correcting, denoising and enhancing the image, and ensuring that all data are in the same coordinate system. Then, the image may be directly registered using image registration technology to determine the geometric transformation parameters required for aligning the two-dimensional aerial image with the two-dimensional radar grid map, such as rotation, translation and scaling. After image registration, the color and details of the two-dimensional aerial image may be fused with the terrain information of the two-dimensional radar grid map in the overlapping area. After fusion, these data may be integrated into a single data structure to form a complete navigation planning map.

[0113] In a feasible implementation manner, before the UAV uses the first recognition model to perform maturity analysis on the navigation planning map to obtain a list of tasks to be picked, the system further includes:

[0114] The drone collected a second aerial image of the orchard;

[0115] The drone obtains the annotation of the second aerial image to obtain an annotated image;

[0116] The drone uses the labeled images to train the initial model and obtain the first recognition model;

[0117] The drone sends the first recognition model to the intelligent agricultural machine so that the intelligent agricultural machine determines the second recognition model.

[0118] In this embodiment, the second aerial image may refer to an image of the ground taken from the air by a photographic device carried by a drone, an airplane or other aircraft. The second aerial image here may be an image obtained by a drone carrying a high-resolution camera photographing the orchard at a predetermined flight altitude and path.

[0119] Annotated images can refer to images that have additional information added to the original image manually or automatically. Here, it can be an image in which the maturity of the fruit is annotated on the second aerial image manually or automatically. In annotated images, the target object is usually marked, and a bounding box can be used to locate the position of the target in the image. These boxes usually include the coordinates of the upper left and lower right corners of the target. Each annotated target is accompanied by a category label that tells the model what the target is. For example, in an aerial image of an orchard, an annotated image can label ripe apples as "ripe fruit."

[0120] The initial model can be a deep learning model, which can be trained based on the annotated images. The model that meets the required recognition accuracy can be called the first recognition model. The annotated images can be preprocessed before training, which can include adjusting the image size, normalizing pixel values, and performing data enhancement, such as random rotation or scaling. The data set can also be divided into a training set, a validation set, and a test set, where the training set is used to train the model, the validation set is used to adjust the model parameters, and the test set is used to finally evaluate the model performance.

[0121] After the drone uses the labeled images to train the first recognition model, this model is used to identify the maturity of crops in the orchard. Since smart agricultural machinery also needs to perform similar recognition tasks, such as identifying and locating ripe fruits on the ground for picking, the drone will send its trained first recognition model to the smart agricultural machinery, which can use the first recognition model as the second recognition model to identify fruits. Throughout the agricultural production process, using the same model can ensure that the recognition tasks from the air to the ground are highly consistent, reducing errors caused by model differences.

[0122] In a feasible implementation, the drone uses the labeled image to train the initial model to obtain a first recognition model, including:

[0123] The drone performs data augmentation on the annotated image to obtain augmented data;

[0124] The drone uses the augmented data to train the initial model and obtain the first recognition model.

[0125] In this implementation, data augmentation can be to increase the size and diversity of a data set by transforming existing data or generating new data. Augmented data can refer to a new, transformed training data set generated by data augmentation technology. Although these data are derived from the original data set, they are modified through a series of operations so that the model can learn the characteristics of the data from different angles and conditions. Here, data augmentation can be performed on annotated images, and operations such as rotation, scaling, cropping, and color adjustment can be performed on the original images. Different shooting conditions and perspectives can be simulated, and changes in the real world can also be simulated.

[0126] After data augmentation, the drone will use the augmented data to train the initial model. The initial model can be a pre-trained deep learning model. By training on augmented data, the model can learn richer and more varied features, which helps the model make more accurate recognition and judgment when facing complex situations in the real world.

[0127] In a feasible implementation, the drone stitches the two-dimensional aerial map and the two-dimensional radar grid map into a navigation planning map, including:

[0128] The UAV determines the common feature points of the two-dimensional aerial map and the two-dimensional radar grid map;

[0129] The UAV aligns the two-dimensional aerial map and the two-dimensional radar grid map according to common feature points to obtain a navigation planning map.

[0130] In this embodiment, the common feature points may refer to significant points or landmarks that exist in two or more data sources and can be identified and matched. These significant points may be obvious landmarks, terrain features, or any points that can be clearly identified in both data. For example, image processing technology may be used to identify features such as roads, rivers, and buildings in an aerial image, and then the corresponding echo signals may be found in a radar grid map.

[0131] After determining the common feature points, the 2D aerial map and the 2D radar grid map can be spatially matched and calibrated. During the alignment process, the drone can use algorithms, such as feature point matching algorithms, to calculate the geometric transformation relationship between the maps, including translation, rotation, and scaling. These transformation parameters are used to accurately match the feature points on the aerial map with the feature points on the radar grid map. And based on the coordinate transformation, the two maps are aligned in the same coordinate system and merged into a navigation planning map. Using common feature points to stitch different types of map data can generate more accurate, comprehensive, and practical maps.

[0132] Step 204: The UAV uses the first recognition model to perform maturity analysis on the navigation planning map to obtain a list of tasks to be picked.

[0133] The first recognition model can refer to an algorithmic model used to identify specific features or objects from image data in the field of machine learning and image recognition. In the field of agriculture, such a model can be used to identify key information such as crop maturity, pests and diseases, and crop types. The first recognition model can be an image recognition model based on deep learning, which can extract features from images and classify them. In agriculture, the first recognition model can be used to identify different growth stages of crops, such as by analyzing the color, shape, and size of crops to determine their maturity. The first recognition model can be a neural network structure that can process image data and recognize patterns in images. For example, a neural network model can be defined to achieve image recognition by stacking multiple convolutional layers and fully connected layers.

[0134] The task list to be picked may be a task list indicating the fruits to be picked, i.e., a list of tasks including individual fruits to be picked, and may include the location information of each fruit to be picked in the navigation planning map. In some cases, the task list to be picked may also prioritize the picking operations based on factors such as the maturity of the crop, weather conditions, and market demand. The task list to be picked may also be updated in real time to reflect the latest image data and weather changes to ensure the timeliness and accuracy of the picking operations.

[0135] Step 205: The drone sends the list of tasks to be harvested to the intelligent agricultural machinery.

[0136] Drones can transmit the list of tasks to be harvested to smart agricultural machinery through wireless networks. The transmission process may involve technologies such as low-power wide area network (Long Range Radio, LoRa) or narrowband Internet of Things (Narrow Band Internet of Things, NB-IoT) to ensure stable communication of equipment over long distances and large areas.

[0137] Step 206: The intelligent agricultural machine determines a task path according to the task list to be picked, and uses a second recognition model to identify the target to be picked based on the image data obtained on the task path to perform a picking operation. The first recognition model is the same as the second recognition model.

[0138] A mission path can refer to the path followed by smart agricultural machinery when performing a specific task, such as crop picking. This path is optimized based on a series of factors and objectives to ensure the efficiency and effectiveness of the operation. For example, the distribution of crops is the basis for path planning, and agricultural machinery needs to cover all areas where mature crops are located. At the same time, terrain conditions such as slope and soil type will also affect the movement speed and path selection of agricultural machinery, so they need to be considered in path planning. The characteristics of the agricultural machinery itself, such as size and turning radius, are also key factors in determining the best path it can take. One of the goals of path planning is to improve work efficiency, which can minimize the moving distance and time of agricultural machinery. In addition, path planning can also avoid unnecessary repeated driving and crop damage, while avoiding obstacles such as trees and buildings.

[0139] Image data can refer to visual information represented in a digital format, which consists of a series of pixels, each of which contains a specific color and brightness value. Cameras installed on agricultural machinery can capture surrounding visual information as image data and call image recognition technology to analyze the image.

[0140] The second recognition model may refer to an algorithm model used to recognize specific features or objects from image data in the field of machine learning and image recognition. The second recognition model can be used to identify different growth stages of crops, such as determining their maturity by analyzing the color, shape and size of crops. That is, the image data can be input into the second recognition model so that the second recognition model outputs a recognition result. The parameters of the first recognition model and the second recognition model are the same.

[0141] In smart agriculture, the target to be picked can refer to crops that are identified as ripe and suitable for picking through image recognition technology. The target to be picked here can be the target fruit suitable for picking identified by the second recognition model. The second recognition model can directly output the target to be picked, or the second recognition model can output the maturity of each fruit, and then determine the target fruit that meets the requirements as the target to be picked based on the maturity.

[0142] In a feasible implementation, based on the image data acquired on the task path, a second recognition model is used to identify the target to be picked so as to perform a picking operation, including:

[0143] When the intelligent agricultural machinery moves to the area to be picked based on the task path, it collects image data and three-dimensional point cloud data, and the image data includes two-dimensional images;

[0144] The intelligent agricultural machinery uses the second recognition model to recognize the two-dimensional image and obtains the target to be picked;

[0145] The intelligent agricultural machinery maps the target to be picked into the three-dimensional point cloud data to obtain the spatial position information of the target to be picked;

[0146] Intelligent agricultural machinery controls the robotic arm to pick the objects to be picked based on the spatial position information.

[0147] In this implementation, the area to be picked may refer to a fruit area that has reached suitable picking conditions in agricultural production, and here may refer to the area where each target to be picked in the task path is located.

[0148] The image data may be image information around the intelligent agricultural machinery when the intelligent agricultural machinery moves to the area to be picked. The image data may include a two-dimensional image. A two-dimensional image may refer to an image displayed on a plane. It has only two dimensions, length and width, and no depth.

[0149] 3D point cloud data can be a data structure used to represent objects or scenes in 3D space. It consists of multiple 3D coordinate points (X, Y, Z) obtained by discrete sampling of the actual object or scene surface. Point cloud can be regarded as a discrete representation of the scene surface in a given coordinate system. In addition to the basic 3D coordinates, point cloud data may also contain other attributes such as color, classification value, intensity value and time.

[0150] Spatial position information can refer to data that describes the position of an object or point in three-dimensional space. This information usually includes the coordinates of the object (such as X, Y, and Z coordinates), which define the exact position of the object in space. The spatial position information of the target to be picked can be the depth information of each pixel point of the target to be picked in the image, which can be achieved through technologies such as depth cameras, stereo vision, or time-of-flight (ToF) cameras. With the depth information, the two-dimensional image coordinates of the target to be picked can be converted into three-dimensional camera coordinates. Next, these three-dimensional points are converted from the camera coordinate system to the radar coordinate system, so that the three-dimensional position corresponding to the feature point in the image can be found in the radar point cloud.

[0151] A robotic arm can be a highly flexible automated device that can grasp, move, and place tools or objects by controlling the movement of each joint. Smart agricultural machinery can determine the precise position that the robotic arm needs to reach based on the spatial coordinates and distance indicated by the spatial position information, that is, control the robotic arm to automatically adjust its posture and movement to pick fruits in a gentle and accurate manner.

[0152] In a feasible implementation, the intelligent agricultural machine determines the task path according to the list of tasks to be picked, including:

[0153] The intelligent agricultural machinery is positioned according to the real-time dynamic difference algorithm to obtain the first position;

[0154] The intelligent agricultural machinery registers the first pose according to the normal distribution transformation algorithm to obtain the second pose;

[0155] The intelligent agricultural machinery performs path planning based on the second posture and the list of tasks to be picked to obtain the task path.

[0156] In this implementation, the Real-Time Kinematic (RTK) algorithm can rely on data transmission between the base station and the mobile station. The base station receives satellite signals in real time and sends its observation data (including pseudorange and carrier phase) to the mobile station. After receiving these data, the mobile station eliminates most errors through differential calculation, thereby achieving high-precision positioning.

[0157] The first pose may be the initial position and pose determined by the intelligent agricultural machine before operation. The pose may be the direction of the agricultural machine relative to the ground, usually expressed as a heading, that is, the angle of the head of the intelligent agricultural machine relative to the north direction. The first pose may be determined by a real-time dynamic difference algorithm.

[0158] The Normal Distributions Transform (NDT) algorithm can divide the point cloud data into multiple small voxels (grids) and model the points in each voxel in a normal distribution, thereby achieving accurate registration between point clouds. In practical applications, the NDT algorithm usually involves two point cloud data sets: one is the reference point cloud, and the other is the point cloud to be registered.

[0159] The reference point cloud may be point cloud data that has been previously obtained by the intelligent agricultural machinery in some way (such as preliminary detailed mapping), which represents the precise location of certain feature points in the orchard.

[0160] The point cloud to be registered can be point cloud data obtained by the intelligent agricultural machinery through sensors (such as lidar) at the current moment, which represents the environment currently perceived by the agricultural machinery.

[0161] The NDT algorithm evaluates the degree of match between two point clouds by calculating the matching probability of each point in the point cloud to be registered in the corresponding normal distribution in the reference point cloud. This process is actually constantly adjusting the point cloud to be registered (based on the first pose) to make it match the reference point cloud as much as possible. Through the optimization algorithm, the pose of the point cloud to be registered is adjusted to maximize the match between the two point clouds. This optimized pose can be called the second pose.

[0162] Intelligent agricultural machinery can use path planning algorithms (such as A* algorithm, Dijkstra algorithm or sampling-based methods) to calculate the optimal path from the current second pose to the position of the target to be picked in the list of tasks to be picked. These algorithms can take into account multiple factors, including the terrain of the orchard, the distribution of crops, the location of obstacles, and the movement ability of agricultural machinery. By comprehensively considering these factors, the path planning algorithm can generate an efficient path that avoids obstacles. In addition, path planning can also consider the operating parameters of agricultural machinery, such as operating width, turning radius and operating direction. These parameters will affect the movement trajectory and efficiency of agricultural machinery when performing picking tasks. Intelligent agricultural machinery can optimize the path based on these parameters to ensure that all target areas can be covered during the picking process and to minimize repeated operations and empty driving. The results of path planning can generate a series of task path points, which can constitute the task path to guide the intelligent agricultural machinery to move in the orchard and perform picking tasks.

[0163] The air-ground collaborative picking system of the embodiment of the present application includes a drone and an intelligent agricultural machinery. The drone collects a first aerial image of an orchard and constructs a two-dimensional aerial map; the intelligent agricultural machinery obtains a two-dimensional radar grid map of the orchard and sends the two-dimensional radar grid map to the drone; the drone splices the two-dimensional aerial map and the two-dimensional radar grid map into a navigation planning map; the drone uses a first recognition model to perform a maturity analysis on the navigation planning map to obtain a list of tasks to be picked; the drone sends the list of tasks to be picked to the intelligent agricultural machinery; the intelligent agricultural machinery determines a task path according to the list of tasks to be picked, and uses a second recognition model to identify the target to be picked based on the image data obtained on the task path to perform a picking operation, and the first recognition model is the same as the second recognition model.

[0164] The system uses drones to stitch together aerial images of orchards and radar grid maps of agricultural machinery, and then uses a recognition model to identify tasks to be picked from the stitched map. The agricultural machinery can generate a picking path based on the tasks to be picked sent by the drone, and based on the images taken on the picking path, use the same recognition model as the drone to identify the targets to be picked for picking. The stitched map obtains more comprehensive orchard information, making the recognition results of the recognition model more accurate. The agricultural machinery uses the same model as the drone for recognition, further improving the accuracy of picking fruits.

[0165] The interactive schematic diagram of an air-ground collaborative picking system in the embodiment of the present application can be referred to Figure 3 shown.

[0166] Step 301: The drone collects aerial images of the litchi orchard.

[0167] The aerial image may be an RGB image of a litchi orchard taken by a drone, and the aerial image here may be a pre-taken image.

[0168] Step 302: The drone obtains the labeled aerial images, trains the labeled aerial images, obtains a maturity recognition model, and sends it to the intelligent agricultural machinery.

[0169] The aerial images can be annotated by the UAV based on neural networks or manually. Manual annotation can be done by using image annotation software to annotate the images containing litchi in the drone aerial images. According to the four states of litchi maturity, the area containing litchi in the image area is annotated with a box and g, y, mr, fr, corresponding to the four categories of green, yellow, half red, and full red respectively. The box diagram is shown in the figure below. Figure 4 As shown, Figure 4 In the figure, the thin dashed line frame represents the green category, the thick dashed line frame represents the yellow category, the thin solid line frame represents the half-red category, and the thick solid line frame represents the full red category.

[0170] The maturity recognition model can be obtained by training a pre-trained model based on annotated aerial images. The pre-trained model can use YoLoV8n, which is a model variant in the YOLOv8 series and is called "YOLOv8Nano". It is the lightest version in the YOLOv8 series, focusing on low-memory devices. It has the smallest model size, the fewest parameters, and the fastest speed.

[0171] The data augmentation technology is used to expand the annotated aerial images, such as rotation, scaling, contrast, rain, fog and other five-dimensional transformations. The contrast, rain and fog transformations can reuse the annotated data, thereby increasing the diversity of the data and improving the robustness of the detection and recognition model. Figure 5 As shown in the figure, the original image has different images after three transformations: contrast, rain, and fog.

[0172] Step 303: The drone collects aerial data of the litchi orchard.

[0173] Aerial photography data can be image data currently captured by a drone in real time.

[0174] Step 304: The drone converts the aerial photography data into a two-dimensional map.

[0175] The drone can collect aerial orthophotos of the orchard, and the map stitching software can be used to generate two-dimensional aerial maps offline from the aerial orthophotos.

[0176] Step 305: The intelligent agricultural machinery constructs a two-dimensional grid map of the litchi orchard and sends it to the drone.

[0177] Exemplarily, intelligent agricultural machinery collects orchard laser point cloud data. The three-dimensional point cloud data of the laser point cloud can be projected onto a two-dimensional plane to construct a two-dimensional map. The continuous two-dimensional map space is converted into discrete grid units, each unit containing information about the area, thus forming a two-dimensional grid map.

[0178] Among them, the three-dimensional point cloud data can be integrated with the vehicle-mounted inertial measurement unit (IMU) data, and the three-dimensional grid map can be constructed using the simultaneous laser localization and mapping (SLMA) algorithm. The three-dimensional grid map is then converted into a two-dimensional grid map through operations such as clipping, filtering, and rasterization projection.

[0179] SLAM technology refers to the process of simultaneously positioning a device and building an environmental map when it moves in an unknown environment. SLAM technology can be divided into two main parts: Localization and Mapping. Localization refers to estimating the device's position given a map; Mapping refers to estimating the environmental map given the device's position.

[0180] Step 306: The drone stitches the two-dimensional map and the two-dimensional grid map into a navigation planning map.

[0181] The navigation planning map can contain visual information of the terrain and precise spatial location information, allowing the drone to accurately locate and navigate based on this map.

[0182] Exemplarily, the stitching method can be to use a feature point matching algorithm (such as Scale-Invariant Feature Transform (SIFT) or Oriented FAST and Rotated BRIEF (ORB)) to identify common feature points in the two-dimensional map and the two-dimensional grid map. Apply the RANSAC algorithm to remove mismatched points and improve the stitching accuracy. Use image registration technology (affine transformation) to align the two maps. Use multi-resolution fusion technology to retain high-resolution details so that each grid on the two-dimensional grid map has longitude and latitude information.

[0183] In one example, a schematic diagram of a generation process of a navigation planning map can refer to Figure 6As shown, step 601, obtain the laser point cloud data of the orchard. Step 602, construct a three-dimensional point cloud map, and use the laser SLAM algorithm optimized by multi-sensor fusion to generate a three-dimensional grid map of the orchard scene. Step 603, convert the three-dimensional grid map into a two-dimensional grid map, and perform operations such as clipping, filtering, and rasterization projection on the three-dimensional grid map file through 3D point cloud processing software. Step 604, control point information collection, which can be a control point obtained by manual selection or automatic identification, and then collect the position information of the control point based on RTK positioning technology. Step 605, align the two-dimensional grid map according to the position information of the control point, which can be through the geographic information system software, based on the two-dimensional grid map and the longitude and latitude information corresponding to the control point, and use the alignment tool to associate the longitude and latitude of the position corresponding to the control point in the two-dimensional grid map with the position information of the control point. Step 606, aerial photography of the orchard, can be obtained by drone aerial photography of the orchard at a preset height. Step 607: Convert the aerial orchard image into a two-dimensional aerial map, which can be a map constructed by stitching multiple images using map stitching software. Step 608: Crop the two-dimensional aerial map to match the required area or boundary. Step 609: Stitch the registered two-dimensional laser grid map with the aerial map into a navigation planning map. This navigation planning map is a map that includes the entire orchard information. The precise terrain data of the grid map is combined with the visual information of the aerial map to improve the overall accuracy of the map. The fused map contains not only terrain and landform information, but also detailed visual features of the objects, such as buildings, roads, vegetation, etc. Step 610: Map annotation tool semantic annotation, you can use the map annotation tool to semantically annotate the map, add descriptive labels to different features on the map, such as marking the drivable area and the operating area, and you can also annotate the grid map and the aerial map in detail, so that the map is easier to find, interpret, combine and reuse, and enhance the semantic information of the map by adding metadata. For example, it includes but is not limited to labeling specific elements on the map, such as roads, buildings, natural features, etc., and may describe the attributes of these elements, such as the name of the road, the purpose of the building, etc.

[0184] In one example, another schematic diagram of a generation process of a navigation planning map can be referred to as Figure 7As shown in the figure, the intelligent agricultural machinery builds a 3D grid map of the orchard based on SLAM technology. The 3D grid map is imported into the Cloud Compare software to generate a point cloud map, and then the point cloud map is converted into a 2D point cloud grid map based on the 3D point cloud cutting of Cloud Compare. The 2D point cloud grid map is a two-dimensional projection of the point cloud map, which is used to display the orthophoto view of the terrain. The point cloud map displayed by Cloud Compare can find control points, such as 5, through the top view, and determine the longitude and latitude of the control points and the coordinates of the 2D rectangular coordinate system by detecting the location information of each control point on the spot. The 2D point cloud grid map can be registered by the location information of the control points to obtain the registered point cloud grid map. The drone can take aerial photos of the orchard to obtain aerial images, and merge multiple aerial images into a 2D aerial map. The 2D aerial map can be cut to obtain an aerial map covering the orchard. The registered point cloud grid map can be spliced ​​with the aerial map covering the orchard to obtain a spliced ​​map including grid information and image information.

[0185] Step 307: The UAV calls the maturity recognition model to recognize the navigation planning map, obtains a list of tasks to be harvested, and sends it to the intelligent agricultural machinery.

[0186] The navigation planning map records the latitude and longitude information of ripe lychees to be picked detected by the drone during flight, as well as the corresponding two-dimensional radar coordinate information. The maturity recognition model can identify ripe lychees as targets to be picked from the navigation planning map, and then generate tasks based on the coordinate information of the targets to be picked in the navigation planning map. These tasks can constitute a list of tasks to be picked.

[0187] The output of the drone usage maturity identification model can be referred to Figure 8 As shown, based on each aerial image, the maturity of each fruit in each image can be identified. The maturity corresponding to partially overlapping fruits can be displayed separately or overlapped, and the maturity of each fruit can be viewed separately by selecting the overlapping maturity. The overlapping maturity displayed in the figure is only an example and does not consider the actual value.

[0188] Step 308: The intelligent agricultural machine plans a path according to the list of tasks to be picked and navigates to the area to be picked.

[0189] The planned path can be provided by modeling the road network and optimizing the algorithm to provide real-time driving route planning and select the optimal path. Based on the tasks in the task list to be picked, the planned path can be selected by sequentially selecting the task closest to the intelligent agricultural machinery in the task list to be picked as the first task, or all the tasks in the task list to be picked can be planned together, that is, a total path that does not repeat the path and covers all tasks can be selected.

[0190] The area to be picked may be an area near the locations of the objects to be picked in the list of tasks to be picked.

[0191] For example, the intelligent agricultural machinery can start the satellite positioning program, determine the approximate area of ​​the current agricultural machinery in the fused map based on the high-precision RTK positioning solution, start the laser point cloud NDT positioning program, and solve the precise position of the agricultural machinery in the map through NDT registration.

[0192] Based on the NDT-based agricultural machinery posture, the coordinates of the fruit to be picked transmitted by the drone and the pre-built fusion map, the A* path planning algorithm is used to generate the optimal global path from the current agricultural machinery position to the point to be picked.

[0193] Intelligent agricultural machinery uses the RRT (rapid random tree) obstacle avoidance algorithm to plan and optimize local paths based on the planned global path and real-time dynamic obstacle perception.

[0194] Step 309: The intelligent agricultural machinery uses the maturity recognition model to identify the objects to be harvested.

[0195] When the smart agricultural machine moves to the area to be picked, it can use the camera to take images of fruit trees around the smart agricultural machine, and then use the maturity recognition model to perform image recognition on the fruit tree images to identify fruits that meet the maturity requirements as the targets to be picked. The camera can use a high-resolution industrial camera for close-range lychee fruit detection and recognition.

[0196] For example, the recognition results of the intelligent agricultural machinery using the maturity recognition model to recognize the image taken by the camera can be found in Fig. 9 As shown, based on the fruits shown in the image, each fruit is labeled with maturity, where Fig. 9 The maturity levels are examples only and do not take into account actual values.

[0197] Step 310: The intelligent agricultural machine drives the mechanical arm to pick the target based on the three-dimensional position coordinates of the target.

[0198] The three-dimensional position coordinates can be the X, Y, and Z coordinates of the target to be picked in space. After determining the fruit tree image where the target to be picked is located, the intelligent agricultural machinery can also obtain the surrounding three-dimensional point cloud data through lidar. After determining the longitude and latitude information of the target to be picked based on the fruit tree image data, the longitude and latitude information of the target to be picked can be mapped to the three-dimensional point cloud data, and the precise three-dimensional position coordinates of the target to be picked can be determined.

[0199] The three-dimensional position coordinates of the target to be picked can be calibrated by the camera and the laser radar, and the transformation matrix between the camera and the laser radar coordinate system is calculated by the joint calibration. The formula for rigid body transformation is:

[0200] R*P l +t=P v

[0201] Where P l is the coordinate of the object in the laser radar coordinate system, P v is the coordinate of the object in the camera coordinate system, R is the rotation matrix, and t is the translation vector. Let the lidar and the camera perceive the same three-dimensional point at the same time. These objects have their own coordinates in different sensors. We can form a system of equations through multiple sets of corresponding points to find an approximate optimal solution to minimize the sensor perception error after calibration. The error expression is

[0202]

[0203] i is the statistical count, k is the number of pixels, and the minimum error can be calculated to solve the rigid body transformation between the camera and the lidar, and the image information collected by the camera can be converted to the lidar coordinate system. The visualization of the joint calibration coordinates of the camera and the lidar can be referred to Fig.10 As shown, it is necessary to determine the position and direction of the camera and lidar relative to the base coordinate system. In the figure, Velodyne refers to the Velodyne lidar sensor installed on the agricultural machinery. Camera refers to the camera used in conjunction with the lidar. base_link represents the base coordinate system of the agricultural machinery and is the reference point of other coordinate systems on the agricultural machinery. The joint can be adjusted based on the position and direction relative to the base coordinate system.

[0204] For example, the joint calibration process can refer to Fig.11 As shown, Fig.11 As shown on the left, the image can be divided into multiple columns of pixels according to the direction of the LiDAR scan. The resolution of the pixels can be adjusted according to the size of the identified object. The depth information of each pixel is determined by the depth mean of the point cloud in the pixel area and is recorded as the depth information of the pixel in the RGBD format data. Fig.11 The right side shows the relationship between the radar data and the camera coordinate system of the intelligent agricultural machinery, where the "left_front" coordinate system refers to the coordinate system with the left front of the vehicle as the origin, and is associated with the lidar sensor on the left front of the vehicle. It is used to describe the position information of the point cloud data scanned by the lidar in the left front area of ​​the vehicle. The camera coordinate system can be a coordinate system centered on the camera, which is used to describe the position of objects in the image captured by the camera. The lidar point cloud data can be converted to the camera coordinate system for fusion with the image data, or vice versa, the image data can be converted to the "left_front" coordinate system to utilize the depth information of the lidar. This conversion usually involves rotation and translation operations to ensure that the data is aligned in the correct spatial reference.

[0205] In one example, laser point cloud is fused with image data. Fig.12 A schematic diagram of a laser point cloud and image fusion scene is shown, where the laser radar point cloud data is combined with the actual image. The laser radar point cloud is overlaid on the street view image in the form of points of different colors, which represent the three-dimensional spatial information obtained by the laser radar scan.

[0206] The way to map the target to be picked to the radar data can be to use the maturity recognition model to detect the RGB image collected by the camera on the intelligent agricultural machinery. The camera performs target detection to obtain the position of the target to be picked in the image. Through the joint calibration result of the laser radar and the camera, and the point cloud ID contained in the pixels in the detection frame, the two-dimensional detection frame is mapped to the three-dimensional laser point cloud. In the 3D laser point cloud data, trace back to the three-dimensional point cloud in the detection frame, convert the pixel points in the 2D Bounding Box into a three-dimensional radar point cloud, calculate the centroid, and calculate the geometric center of the three-dimensional point cloud cluster as the 3D coordinates of the litchi. Calculate the pixel centroid G of the target point cloud c :

[0207]

[0208] c x 、c y 、c z is the center of mass space coordinate, x i ,y i 、z i is the spatial coordinate of the pixel point, i is the count number, n is the total number of pixels, and the centroids of multiple point clouds of targets to be picked are calculated as output results. After the picking control system receives the number of targets to be picked and the spatial position information, it can control the robotic arm to perform pruning and picking procedures on the branch positions of the targets to be picked.

[0209] Step 311: The intelligent agricultural machine determines whether the pending picking tasks in the pending picking task list are completed. If yes, the process ends; otherwise, step 308 is executed.

[0210] The smart agricultural machine performs the picking tasks one by one through the robotic arm according to the information in the task list, such as the location and type of the target. After each picking is completed, the smart agricultural machine checks the status of the task to determine whether it has been completed. The picking area can be scanned again to confirm whether the target has been picked. If a task has been completed, it will be removed from the list of tasks to be picked or marked as completed. If the task is not completed, the smart agricultural machine can record this status and prepare to perform the next picking attempt. The smart agricultural machine continues to check the list of tasks to be picked. If all tasks in the list are marked as completed, the smart agricultural machine can end the picking process. If there are still unfinished tasks, the smart agricultural machine will return to step 308 and continue to execute the next task to be picked. The smart agricultural machine will continue to loop step 308, that is, "execute picking tasks", until all tasks are completed. This cycle ensures the continuity and integrity of the picking operation.

[0211] The above describes the entire air-ground collaborative picking system. The following describes the air-ground collaborative picking methods performed by the drone and intelligent agricultural machinery in the air-ground collaborative picking system respectively.

[0212] See also Fig.13 ,like Fig.13 The figure is a flow chart of an air-ground collaborative picking method provided in an embodiment of the present application, which is executed by a drone in an air-ground collaborative picking system, including:

[0213] Step 1301: collect a first aerial image of the orchard and construct a two-dimensional aerial map;

[0214] Step 1302: Obtain a two-dimensional radar grid map sent by the intelligent agricultural machinery;

[0215] Step 1303: splice the two-dimensional aerial map and the two-dimensional radar grid map into a navigation planning map;

[0216] Step 1304: Use the first recognition model to perform maturity analysis on the navigation planning map to obtain a list of tasks to be picked;

[0217] Step 1305: Send the list of tasks to be picked to the intelligent agricultural machine, so that the intelligent agricultural machine determines the task path according to the list of tasks to be picked, and uses the second recognition model to identify the targets to be picked on the task path to perform the picking operation. The first recognition model is the same as the second recognition model.

[0218] Drones can take aerial photos of orchards, collect first aerial images, and build two-dimensional aerial maps based on these aerial images. This map can show the layout of the orchard and the distribution of fruit trees. Then, drones can obtain two-dimensional radar grid maps collected by smart agricultural machinery. This map is generated by radar sensors and provides location and shape information of objects in the orchard, which helps to identify objects in the orchard more accurately.

[0219] The drone can then stitch the 2D aerial map and the 2D radar grid map together to form a comprehensive navigation planning map. This map combines visual and radar data to provide more comprehensive information for maturity analysis. The navigation planning map is analyzed for maturity using a first recognition model, which may be based on a machine learning algorithm that can identify the maturity of the fruit, thereby generating a list of tasks to be picked, listing which fruit trees have ripe fruits and need to be picked.

[0220] Finally, the list of tasks to be picked is sent to the intelligent agricultural machine, which determines the optimal picking task path based on this list. On the execution path, the intelligent agricultural machine uses the second recognition model to identify specific targets to be picked in the surrounding fruit trees for picking. The first recognition model and the second recognition model are the same, that is, they have the same maturity recognition accuracy. The drone can perform Figure 2 or Figure 3 The relevant enforcement measures for UAVs will not be elaborated here.

[0221] See also Fig.14 ,like Fig.14 The figure is a flow chart of another air-ground collaborative picking method provided in an embodiment of the present application, which is executed by an intelligent agricultural machine in an air-ground collaborative picking system, including:

[0222] Step 1401: Obtain a two-dimensional radar grid map of the orchard and send it to the drone;

[0223] Step 1402: receiving a list of pending picking tasks sent by the drone, where the list of pending picking tasks is a mosaic map of a two-dimensional aerial map of the orchard and a two-dimensional radar grid map of the drone, obtained by analyzing the first recognition model;

[0224] Step 1403, determining the task path according to the list of tasks to be picked;

[0225] Step 1404: Based on the image data acquired on the task path, use the second recognition model to recognize the target to be picked so as to perform the picking operation. The first recognition model is the same as the second recognition model.

[0226] Smart agricultural machinery can obtain a 2D radar grid map of the orchard, which shows the location and shape of objects in the orchard in detail. This radar grid map is then sent to the drone, which uses it to stitch together the previously collected 2D aerial map of the orchard to form a comprehensive navigation planning map. This map combines visual and radar data to provide the drone with a comprehensive view of the orchard.

[0227] The drone then uses the first recognition model to perform a maturity analysis on the stitched map. This model is based on a machine learning algorithm and can identify the maturity of the fruit. After the analysis is completed, the drone generates a task list to be picked, which details which fruit trees have ripe fruits and need to be picked.

[0228] After receiving the list of tasks to be picked sent by the drone, the intelligent agricultural machine determines the optimal picking task path based on the list. On the determined task path, the intelligent agricultural machine obtains image data and uses the second recognition model to identify the specific target to be picked. Figure 2 or Figure 3 The relevant means of implementation of intelligent agricultural machinery will not be elaborated here.

[0229] See also Fig.15 ,like Fig.15 The figure shows a schematic diagram of the structure of an air-ground collaborative picking device provided in an embodiment of the present application. The device 1500 includes:

[0230] The acquisition unit 1501 is used to acquire a first aerial image of the orchard and construct a two-dimensional aerial map;

[0231] An acquisition unit 1502 is used to acquire a two-dimensional radar grid map sent by the intelligent agricultural machinery;

[0232] A splicing unit 1503 is used to splice the two-dimensional aerial map and the two-dimensional radar grid map into a navigation planning map;

[0233] The planning unit 1504 is used to perform maturity analysis on the navigation planning map using the first recognition model to obtain a list of tasks to be picked;

[0234] The sending unit 1505 is used to send the list of tasks to be picked to the intelligent agricultural machinery, so that the intelligent agricultural machinery determines the task path according to the list of tasks to be picked, and uses the second recognition model on the task path to identify the targets to be picked to perform the picking operation. The first recognition model is the same as the second recognition model.

[0235] Among them, the air-ground collaborative picking device 1500 can perform Fig.13 The open-air collaborative picking method will not be described here.

[0236] See also Fig.16 ,like Fig.16 FIG. 1 is a schematic diagram of the structure of another air-ground collaborative picking device provided in an embodiment of the present application. The device 1600 includes:

[0237] The acquisition unit 1601 is used to acquire a two-dimensional radar grid map of the orchard and send it to the drone;

[0238] The receiving unit 1602 is used to receive a list of pending picking tasks sent by the drone, where the list of pending picking tasks is a mosaic map of a two-dimensional aerial map of the orchard and a two-dimensional radar grid map of the drone, obtained by analyzing the first recognition model;

[0239] The determination unit 1603 is used to determine the task path according to the list of tasks to be picked;

[0240] The picking unit 1604 is used to identify the target to be picked based on the image data obtained on the task path using the second recognition model to perform a picking operation, and the first recognition model is the same as the second recognition model.

[0241] Among them, the air-ground collaborative picking device 1600 can perform Fig.14 The open-air collaborative picking method will not be described here.

[0242] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned air-ground collaborative picking method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0243] See also Fig.17 , Fig.17 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0244] The processor 1701 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0245] The memory 1702 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1702 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1702, and the processor 1701 calls and executes the air-ground collaborative picking method of the embodiment of this application;

[0246] Input / output interface 1703, used to implement information input and output;

[0247] Communication interface 1704, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0248] A bus 1705 that transmits information between the various components of the device (e.g., the processor 1701, the memory 1702, the input / output interface 1703, and the communication interface 1704);

[0249] The processor 1701 , the memory 1702 , the input / output interface 1703 and the communication interface 1704 are connected to each other in communication within the device via the bus 1705 .

[0250] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned open-air collaborative picking method is implemented.

[0251] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0252] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0253] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0254] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0255] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0256] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0257] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0258] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0259] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0260] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0261] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0262] The preferred embodiments of the present application are described above with reference to the accompanying drawings, but the scope of the rights of the present application is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present application should be within the scope of the rights of the present application.

Claims

1. An air-ground collaborative picking system, characterized in that: The air-ground collaborative harvesting system includes a drone and an intelligent agricultural machine, including: The drone collects a first aerial image of the orchard to construct a two-dimensional aerial map; The intelligent agricultural machine obtains a two-dimensional radar grid map of the orchard, and sends the two-dimensional radar grid map to the drone; The drone splices the two-dimensional aerial map and the two-dimensional radar grid map into a navigation planning map; The drone uses the first recognition model to perform maturity analysis on the navigation planning map to obtain a list of tasks to be picked; The drone sends the list of tasks to be picked to the intelligent agricultural machine; The intelligent agricultural machine determines a task path according to the list of tasks to be picked, and uses a second recognition model to identify the targets to be picked based on the image data obtained on the task path to perform a picking operation, and the first recognition model is the same as the second recognition model.

2. The system according to claim 1, characterized in that Before the UAV uses the first recognition model to perform maturity analysis on the navigation planning map to obtain a list of tasks to be picked, the system further includes: The drone collects a second aerial image of the orchard; The drone obtains annotations of the second aerial image to obtain an annotated image; The drone uses the annotated image to train an initial model to obtain the first recognition model; The drone sends the first recognition model to the intelligent agricultural machine, so that the intelligent agricultural machine determines the second recognition model.

3. The system according to claim 2, characterized in that The drone uses the annotated image to train the initial model to obtain the first recognition model, including: The drone performs data augmentation on the annotated image to obtain augmented data; The drone uses the augmented data to train an initial model to obtain the first recognition model.

4. The system according to claim 1, characterized in that The drone splices the two-dimensional aerial map and the two-dimensional radar grid map into a navigation planning map, including: The unmanned aerial vehicle determines common feature points of the two-dimensional aerial map and the two-dimensional radar grid map; The unmanned aerial vehicle aligns the two-dimensional aerial map and the two-dimensional radar grid map according to the common feature points to obtain a navigation planning map.

5. The system according to claim 1, characterized in that The method of using a second recognition model to identify the target to be picked based on the image data acquired on the task path to perform a picking operation includes: When the intelligent agricultural machine moves to the to-be-picked area based on the task path, the image data and the three-dimensional point cloud data are collected, and the image data includes a two-dimensional image; The intelligent agricultural machine uses a second recognition model to recognize the two-dimensional image to obtain a target to be picked; The intelligent agricultural machine maps the target to be picked to the three-dimensional point cloud data to obtain spatial position information of the target to be picked; The intelligent agricultural machine controls the mechanical arm to pick the target to be picked according to the spatial position information.

6. The system according to claim 1, characterized in that The intelligent agricultural machine determines a task path according to the list of tasks to be picked, including: The intelligent agricultural machine is positioned according to a real-time dynamic difference algorithm to obtain a first position; The intelligent agricultural machine registers the first posture according to a normal distribution transformation algorithm to obtain a second posture; The intelligent agricultural machine performs path planning according to the second posture and the list of tasks to be picked to obtain a task path.

7. The system according to claim 1, characterized in that The intelligent agricultural machine obtains a two-dimensional radar grid map of the orchard, including: The intelligent agricultural machine obtains a two-dimensional point cloud grid map of the orchard, and obtains control points in the two-dimensional point cloud grid map; The intelligent agricultural machine obtains the position information of the control point; The intelligent agricultural machine calibrates the two-dimensional point cloud grid map according to the position information of the control point to obtain a two-dimensional radar grid map.

8. The system according to claim 1, characterized in that The intelligent agricultural machinery and the unmanned aerial vehicle are aligned using the same satellite timestamp.

9. A method for collaborative picking in open space, characterized in that: include: Collect the first aerial image of the orchard and construct a two-dimensional aerial map; Obtain the two-dimensional radar grid map sent by intelligent agricultural machinery; splicing the two-dimensional aerial map and the two-dimensional radar grid map into a navigation planning map; Using the first recognition model to perform maturity analysis on the navigation planning map to obtain a list of tasks to be picked; The list of tasks to be picked is sent to the intelligent agricultural machine, so that the intelligent agricultural machine determines a task path according to the list of tasks to be picked, and uses a second recognition model on the task path to identify the targets to be picked to perform a picking operation, and the first recognition model is the same as the second recognition model.

10. A method for collaborative picking between open space and ground, characterized in that: include: Obtain a 2D radar grid map of the orchard and send it to the drone; Receiving a list of pending picking tasks sent by the drone, wherein the list of pending picking tasks is obtained by analyzing a spliced ​​map of the two-dimensional aerial map of the orchard and the two-dimensional radar grid map by the drone using a first recognition model; Determine the task path according to the list of tasks to be picked; Based on the image data acquired on the task path, a second recognition model is used to recognize the target to be picked so as to perform a picking operation, and the first recognition model is the same as the second recognition model.

Images