Fruit picking method, device, medium, and electronic device

Abstract

Embodiments of the present application disclose a fruit picking method, device, medium and electronic equipment. The method comprises: determining a growth position of a target fruit tree on both sides of a fruit tree row, and determining a picking route of a picking robot based on the growth position; the picking route comprises at least two picking points; controlling the picking robot to move along the picking route, stopping the picking robot when the picking robot reaches the picking points, and determining a fruit density in different height ranges of the target fruit tree; based on the fruit density in different height ranges of the target fruit tree, controlling the picking robot to adjust a mechanical arm to a target height for fruit picking. The technical solution provided by the present application improves the degree of mechanization of fruit picking, improves the efficiency of fruit picking, and reduces the cost of fruit picking.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of fruit picking, in particular to a fruit picking method, device, medium and electronic equipment. BACKGROUND

[0002] The planting area and yield of fruit trees in China have been ranked first in the world for many years, but the degree of mechanization of domestic orchards is still relatively low. Fruit picking, as an important part of orchard planting, still relies on manual labor. Manual picking has problems such as high labor intensity, low picking efficiency, and high picking cost. SUMMARY

[0003] The present application provides a fruit picking method, device, medium and electronic equipment, which can improve the degree of mechanization of fruit picking, improve the efficiency of fruit picking, and reduce the cost of fruit picking.

[0004] According to a first aspect of the present application, a fruit picking method is provided, the method comprising:

[0005] determining the growth positions of the target fruit tree on both sides of the fruit tree row, and determining the picking route of the picking robot based on the growth positions; the picking route comprises at least two picking points;

[0006] controlling the picking robot to move along the picking route, stopping the picking robot when the picking robot reaches the picking point, and determining the fruit density in different height ranges on the target fruit tree;

[0007] controlling the picking robot to adjust the mechanical arm to a target height for fruit picking based on the fruit density in different height ranges on the target fruit tree.

[0008] According to a second aspect of the present application, a fruit picking device is provided, the device comprising:

[0009] a picking route determination module configured to determine the growth positions of the target fruit tree on both sides of the fruit tree row, and determine the picking route of the picking robot based on the growth positions; the picking route comprises at least two picking points;

[0010] a fruit density determination module configured to control the picking robot to move along the picking route, stop the picking robot when the picking robot reaches the picking point, and determine the fruit density in different height ranges on the target fruit tree;

[0011] a target height adjustment module configured to control the picking robot to adjust the mechanical arm to a target height for fruit picking based on the fruit density in different height ranges on the target fruit tree.

[0012] According to a third aspect of the present application, the embodiments of the present application provide a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the fruit picking method according to the embodiments of the present application.

[0013] According to a fourth aspect of the present application, the embodiments of the present application provide an electronic device, which comprises a memory, a processor and a computer program stored in the memory and executable by the processor, and the processor implements the fruit picking method according to the embodiments of the present application when executing the computer program.

[0014] The technical scheme of the present application determines the picking route of the picking robot based on the growth position of the target fruit tree on both sides of the fruit tree row, controls the picking robot to move along the picking route, and controls the picking robot to stop when the picking robot reaches the picking point, thereby providing data support for using robot technology for fruit picking. The embodiments of the present application determine the fruit density in different height ranges of the target fruit tree, control the picking robot to adjust the mechanical arm to the target height for fruit picking based on the fruit density in different height ranges of the target fruit tree, thereby reducing the movement distance of the mechanical arm, shortening the time loss and resource loss of fruit picking, and being conducive to improving the fruit picking efficiency. The embodiments of the present application use the picking robot for fruit picking, thereby reducing the cost of fruit picking.

[0015] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0017] Figure 1A is a flowchart of the fruit picking method according to the first embodiment of the present application;

[0018] Figure 1B is a schematic diagram of the picking route of the picking robot between the rows of fruit trees according to the first embodiment;

[0019] Figure 2A is a flowchart of the fruit picking method according to the second embodiment of the present application;

[0020] Figure 2B is a first structural schematic diagram of a picking robot according to the second embodiment of the present application;

[0021] Figure 2C is a second structural schematic view of a picking robot according to Embodiment Two of the present application;

[0022] Figure 2D is a partial structural schematic view of a picking robot according to Embodiment Two of the present application;

[0023] Figure 3 is a structural schematic view of a fruit picking device according to Embodiment Three of the present application;

[0024] Figure 4 is a structural schematic view of a mechanical arm control system deployed in a picking robot. DETAILED DESCRIPTION

[0025] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative labor should fall within the scope of protection of the present application.

[0026] It should be noted that the terms “first”, “second”, “target” and “candidate” and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms “include” and “have” and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] Embodiment One

[0028] Figure 1A is a flowchart of a fruit picking method according to Embodiment One of the present application. This embodiment can be applicable to the case of picking fruit in a dwarf and densely planted orchard. The method can be executed by a fruit picking device, which is realized in the form of hardware and / or software and can be integrated into an electronic device running this system.

[0029] As shown in Figure 1A , the method comprises:

[0030] S110, determine the growth positions of the target fruit trees on both sides of the fruit tree rows, and determine the picking route of the picking robot based on the growth positions; the picking route includes at least two picking points.

[0031] Wherein, the target fruit trees growing on both sides of the fruit tree rows refer to the fruit trees that need to be picked. The fruit tree rows refer to the passageways between two rows of fruit trees in the orchard. The fruit picking of the target fruit trees is performed by the picking robot.

[0032] The picking route is used to guide the picking robot to complete the fruit picking of the target fruit trees growing on both sides of the fruit and vegetable rows from a certain position in the orchard. The picking path is determined according to the growth positions of the target fruit trees on both sides of the fruit and vegetable rows. Optionally, the growth positions of the target fruit trees on both sides of the fruit and vegetable rows can be determined by image processing technology and radar detection technology.

[0033] Optionally, the picking route includes at least two picking points. The picking points are used for fruit picking of the target fruit trees. The number of picking points in the picking route is not limited here and is determined according to the specific circumstances. It is worth noting that the picking points in the picking route should be able to cover all the target fruit trees in the picking route. Optionally, the number of picking points is determined according to the working range of the picking robot. The working range includes the working radius. Optionally, the movement distance of the picking robot is determined. If the movement distance of the picking robot is the working distance, it can be determined that the picking robot reaches the picking point. Optionally, the picking point is marked in the picking route, and the real-time position of the picking robot is monitored. If the real-time position of the picking robot coincides with the picking point in the picking route, it is determined that the picking robot reaches the picking point. In the case that the picking robot reaches the picking point, the picking robot is instructed to pick fruits.

[0034] S120, control the picking robot to move along the picking route, control the picking robot to stop in the case that the picking robot reaches the picking point, and determine the fruit density in different height ranges of the target fruit trees;

[0035] The picking robot is controlled to move along the picking route, and the real-time position of the picking robot is monitored to determine whether the picking robot reaches the picking point. Specifically, in the case that the unmanned mobile chassis of the picking robot reaches the picking point, it is determined that the picking robot reaches the picking point. In the case that the picking robot reaches the picking point, the picking robot is controlled to stop and pick fruits. The picking robot is installed with a lifting mechanism, the lifting mechanism is installed with a mechanical arm, and the mechanical arm is installed with a gripper for fruit picking. After the picking robot completes the picking task at the current point, the picking robot is controlled to continue moving along the picking route to the next point for picking, until all the points have been picked.

[0036] In the process of fruit picking by the picking robot, the height of the mechanical arm is adjusted by controlling the lifting mechanism to adjust the height so as to control the gripper on the mechanical arm to pick fruits at different heights.

[0037] S130, based on the fruit density in the different height range of the target fruit tree, the picking robot is controlled to adjust the mechanical arm to the target height for fruit picking.

[0038] Wherein, the target height is within the height range of the target fruit tree, and the target height is determined according to the fruit density in the different height range of the target fruit tree. Optionally, the fruit density in the different height range of the target fruit tree is arranged in descending order, and the target height is determined according to the sorting result. Then, the picking robot is controlled to adjust the mechanical arm to the target height for fruit picking.

[0039] It can be understood that the fruit hanging rate is different at different positions of the fruit tree, and based on the fruit density in the different height range of the target fruit tree, the picking robot is controlled to adjust the mechanical arm to the target height for fruit picking, which can reduce the movement distance of the mechanical arm, shorten the time loss and resource loss of fruit picking, and is beneficial to improve the fruit picking efficiency.

[0040] Figure 1B is a schematic diagram of the picking route of the picking robot provided in embodiment one between the rows of fruit trees, referring to Figure 1B If there are at least two rows of fruit trees, after the picking robot completes the picking in the current row of fruit trees, it turns into the next row of fruit trees until all the fruits in the rows of fruit trees in the orchard are picked.

[0041] The technical scheme of the present application determines the picking route of the picking robot based on the growth position of the target fruit tree on both sides of the row of fruit trees, controls the picking robot to move along the picking route, and controls the picking robot to stop when the picking robot reaches the picking point. This provides data support for using robot technology for fruit picking. The embodiments of the present application determine the fruit density in the different height range of the target fruit tree, and based on the fruit density in the different height range of the target fruit tree, the picking robot is controlled to adjust the mechanical arm to the target height for fruit picking, which can reduce the movement distance of the mechanical arm, shorten the time loss and resource loss of fruit picking, and is beneficial to improve the fruit picking efficiency. The embodiments of the present application use the picking robot for fruit picking, which reduces the cost of fruit picking.

[0042] In an optional embodiment, the growth position of the target fruit tree on both sides of the fruit tree row is determined, including: controlling the picking robot to collect image data on both sides of the fruit tree row; inputting the image data into a pre-trained semantic segmentation model, segmenting the target fruit tree growing on both sides of the fruit tree row from the image data by using the semantic segmentation model, and determining the image coordinates of the target fruit tree; determining the point cloud coordinates of the target fruit tree according to a coordinate conversion matrix and the image coordinates of the target fruit tree; projecting the point cloud coordinates of the target fruit tree onto the ground to obtain the growth position of the target fruit tree on both sides of the fruit tree row; wherein the coordinate conversion matrix is obtained by jointly calibrating a first camera and a laser radar; the first camera is used to collect image data on both sides of the fruit tree row, and the laser radar is used to collect point cloud data on both sides of the fruit tree row.

[0043] Optionally, the picking robot is installed with a first camera and a laser radar. The first camera is used to collect image data on both sides of the fruit tree row. The laser radar is used to collect point cloud data on both sides of the fruit tree row. Optionally, the first camera and the laser radar are installed on the unmanned mobile chassis. Optionally, the first camera is a color camera, and the image data collected by the first camera is a color image.

[0044] Before the first camera collects image data and before the laser radar collects point cloud data, the first camera and the laser radar are jointly calibrated to obtain a coordinate conversion matrix. The coordinate conversion matrix is used to convert image coordinates and point cloud coordinates. The image coordinates are two-dimensional coordinates, and the point cloud coordinates are three-dimensional coordinates. The coordinate conversion matrix can realize the conversion between two-dimensional coordinates and three-dimensional coordinates.

[0045] The image data on both sides of the fruit tree row can be used to identify the types of environmental objects on both sides of the fruit tree row. The image data includes type information of the environmental objects. The image data on both sides of the fruit tree row can include sky, road and fruit trees, etc.

[0046] Optionally, the image data is input into a pre-trained semantic segmentation model, and the semantic segmentation model is used to identify the types of environmental objects on both sides of the fruit tree row. The semantic segmentation model is used to segment the target fruit tree growing on both sides of the fruit tree row from the image data. Optionally, the semantic segmentation model is a neural network model based on deep learning technology. For example, the semantic segmentation model is a Unet model or a deeplab model. The semantic segmentation model is obtained by supervised training.

[0047] The semantic segmentation model segments the target fruit tree growing on both sides of the fruit tree row from the image data, and determines the image coordinates of the target fruit tree.

[0048] According to the coordinate conversion matrix and the image coordinates of the target fruit tree, the point cloud coordinates of the target fruit tree are determined. The coordinate conversion matrix is used to project the target fruit tree from a two-dimensional plane to a three-dimensional space. The point cloud coordinates of the target fruit tree are projected onto the ground to obtain the growth position of the target fruit tree on both sides of the fruit tree row. Among them, the target fruit tree segmented from the image data includes the fruit tree trunk and the fruit tree crown. Correspondingly, the image coordinates of the target fruit tree include the trunk coordinates and the crown coordinates. Optionally, according to the coordinate conversion matrix and the trunk coordinates of the target fruit tree, the point cloud coordinates of the fruit tree trunk are determined; the point cloud coordinates of the fruit tree trunk are projected onto the ground to obtain the growth position of the target fruit tree on both sides of the fruit tree row.

[0049] The above technical solution provides a feasible growth position determination scheme for determining the growth position of the target fruit tree on both sides of the fruit tree row, and provides data support for generating a picking route based on the growth position of the target fruit tree.

[0050] In an optional embodiment, the picking route of the picking robot based on the growth position comprises: performing curve fitting on the growth position of the target fruit tree on both sides of the fruit tree row to obtain a left fruit tree row curve and a right fruit tree row curve on both sides of the fruit tree row; determining a center line between the left fruit tree row curve and the right fruit tree row curve, and determining the picking route of the picking robot based on the center line; and determining at least two picking points in the picking route according to the working distance of the picking robot.

[0051] Among them, the growth position of the target fruit tree is obtained by projecting the point cloud coordinates of the fruit tree trunk onto the ground. Optionally, a curve fitting algorithm is used to perform curve fitting on the growth position of the target fruit tree on both sides of the fruit tree row to obtain a left fruit tree row curve and a right fruit tree row curve on both sides of the fruit tree row. Among them, the left fruit tree row curve and the right fruit tree row curve respectively correspond to the projection connecting lines of the left and right rows of fruit tree trunks on the ground. Optionally, the curve fitting algorithm is a Ransac algorithm.

[0052] A center line between the left fruit tree row curve and the right fruit tree row curve is determined, and the distance between a point on the center line and the left fruit tree row curve and the right fruit tree row curve is equal. Optionally, the center line between the left fruit tree row curve and the right fruit tree row curve serves as the picking route of the picking robot.

[0053] Then, at least two picking points are determined in the picking route according to the working distance of the picking robot. In the case that the picking robot reaches the picking point, the picking robot is instructed to pick fruits.

[0054] The above technical solution provides a feasible picking route determination scheme, which can be used to determine the picking route of the picking robot. Technical support is provided for controlling the picking robot to perform a fruit picking task.

[0055] In an optional embodiment, the control of the picking robot to move along the picking route and stop when the picking robot reaches the picking point comprises: acquiring real-time position and real-time attitude of the picking robot; determining yaw angle and lateral offset distance of the picking robot relative to the picking route based on the picking route and the real-time position and real-time attitude of the picking robot; controlling walking driving units and steering driving units of the picking robot to move the picking robot along the picking route based on the yaw angle and the lateral offset distance; and determining whether the picking robot reaches the picking point according to the real-time position of the picking robot, and controlling the picking robot to stop when the picking robot reaches the picking point.

[0056] The real-time position and the real-time attitude are used to determine the real-time state of the picking robot. The real-time position and the real-time attitude are collected by a laser radar, a first camera and an inertial attitude sensor installed on the picking robot.

[0057] The yaw angle and the lateral offset distance of the picking robot relative to the picking route are determined based on the picking route and the real-time position and the real-time attitude of the picking robot. The yaw angle refers to the included angle between the current heading angle of the picking robot and the picking route. The yaw angle is used to quantify the degree of angular deviation of the picking robot. The lateral offset distance refers to the distance between the center coordinate of the chassis of the picking robot and the picking route. The lateral offset distance is used to quantify the degree of positional deviation of the picking robot.

[0058] The angle compensation direction and degree of the picking robot can be determined based on the yaw angle. The position compensation direction and degree of the picking robot can be determined based on the lateral offset distance. Optionally, the walking driving units and the steering driving units of the picking robot are controlled to move the picking robot along the picking route in order to reduce the yaw angle and shorten the lateral offset distance. The walking driving units are used to adjust the wheel speed of the picking robot. The steering driving units are used to adjust the heading angle of the picking robot. Optionally, a tracking control algorithm is used to control the walking driving units and the steering driving units of the picking robot.

[0059] During the movement of the picking robot, whether the picking robot reaches the picking point is determined according to the real-time position of the picking robot, and the picking robot is controlled to stop when the picking robot reaches the picking point, in preparation for fruit picking.

[0060] The technical scheme above determines the yaw angle and lateral offset distance of the picking robot relative to the picking route, controls the walking driving unit and the steering driving unit of the picking robot based on the yaw angle and the lateral offset distance to make the picking robot move along the picking route, so that the picking robot can be ensured not to deviate from the picking route, and technical support is provided for controlling the picking robot to perform the fruit picking task.

[0061] Embodiment Two

[0062] Figure 2A is a flowchart of the fruit picking method according to Embodiment Two of the present application. This embodiment is further optimized on the basis of the above-mentioned embodiments.

[0063] As shown in Figure 2A , the method comprises the following steps.

[0064] S210, determining the growth positions of a target fruit tree on both sides of a fruit tree row, and determining a picking route of a picking robot based on the growth positions; the picking route comprises at least two picking points.

[0065] S220, controlling the picking robot to move along the picking route, and controlling the picking robot to stop when the picking robot reaches the picking points.

[0066] S230, controlling a lifting mechanism of the picking robot to move up and down; wherein a second camera is installed on the lifting mechanism, and the second camera is a depth camera.

[0067] A second camera is installed on the lifting mechanism, and the second camera is a depth camera. The second camera is used to identify and locate the fruits on the target fruit tree.

[0068] Controlling the lifting mechanism of the picking robot to move up and down can drive the second camera installed on the lifting mechanism to move together.

[0069] S240, during the up-and-down movement of the lifting mechanism, controlling the second camera to perform image acquisition to obtain depth images and color images in different height ranges on the target fruit tree.

[0070] During the up-and-down movement of the lifting mechanism, the second camera is controlled to perform image acquisition to obtain depth images and color images in different height ranges on the target fruit tree.

[0071] Optionally, the lifting mechanism is controlled to move up and down multiple times, and the second camera is controlled to perform image acquisition multiple times and from multiple angles. The depth images and color images acquired by the second camera are used to determine the spatial distribution of the fruits on the target fruit tree.

[0072] The color image is mainly used for identifying the fruits on the target fruit tree, and the depth image is mainly used for positioning the fruits on the target fruit tree.

[0073] S250, inputting the color image into a pre-trained target recognition model, and identifying the candidate fruits to be picked in the color image by using the target recognition model to determine the pixel position of the candidate fruits in the color image.

[0074] The target recognition model is used for identifying the candidate fruits to be picked in the color image. The target recognition model is obtained by supervised pre-training. Optionally, the model training samples of the target recognition model are obtained by a depth camera. Illustratively, the depth camera can be an Intel RealSense depth camera, and the image acquisition resolution is 1280 pixels x 720 pixels.

[0075] Optionally, the model training samples are obtained by the depth camera at a range of 10-50 cm at angles of front view, side view, top view and overhead view, and further, the model training samples can also include two directions of sun-facing and sun-backing, and the model training samples need to ensure that the complex background of the orchard is considered. In addition, when collecting the model training samples, attention should also be paid to including the relative positions of various fruits, such as single unobstructed, single obstructed by branches / leaves, multiple fruits obstructing each other, and the like, so as to improve the sample representativeness of the model training samples. Optionally, the model training samples are obtained by random transformation, such as scaling, rotation and flipping, and the like, data enhancement processing.

[0076] Training the target recognition model based on the model training samples can provide the target recognition model with more diverse image features, which are not only highly robust but also have strong discrimination, and can adapt to different orchard scenes. This is conducive to improving the accuracy of the target recognition model.

[0077] The color image is used as the input data of the target recognition model, and the target recognition model identifies the candidate fruits to be picked in the color image to determine the pixel position of the candidate fruits in the color image. The candidate fruits are at least two. The pixel position of the candidate fruits in the color image is the position of the candidate fruits in the two-dimensional plane, and the picking of the fruits needs to further determine the position of the candidate fruits in the three-dimensional space.

[0078] Optionally, the target recognition model is also used for identifying the maturity and diameter of the fruits, and identifying the candidate fruits to be picked in the color image according to the set maturity and diameter. This can realize fruit grading in the harvesting stage.

[0079] S260, registering the depth image and the color image, and extracting depth data of the candidate fruit from the registered depth image based on the pixel position of the candidate fruit in the color image.

[0080] The position of the candidate fruit in the three-dimensional space is included in the depth image. The depth image and the color image are registered to determine the pixel correspondence between the depth image and the color image.

[0081] Based on the pixel position of the candidate fruit in the color image and the pixel correspondence, the pixel position of the candidate fruit in the depth image is determined, and the depth data of the candidate fruit is extracted from the depth image based on the pixel position of the candidate fruit in the depth image. The depth data of the candidate fruit is used to quantify the actual distance between the candidate fruit and the second camera.

[0082] Optionally, the depth data of the candidate fruit is the depth value corresponding to the center pixel of the candidate fruit. In order to ensure the accuracy of the depth value, the depth values corresponding to the auxiliary pixels in the range around the center pixel are averaged, and the average result is taken as the depth data of the candidate fruit. The auxiliary pixels are used to determine the depth data of the candidate fruit.

[0083] Illustratively, the auxiliary pixels in the range around the center pixel can be 40 pixels around the center pixel. The process of determining the depth data of the candidate fruit is described taking the 40 pixels around the center pixel as auxiliary pixels: 1) determine the center pixel (ux, uy) of the candidate fruit; 2) randomly obtain the depth value corresponding to the position near the i-th center pixel, where i is a positive integer between 1 and 40; 3) determine whether the depth value is valid, and if it is valid, add it to the depth array distance_list; 4) after the 40th depth value is added to the depth array distance_list, average the depth values in the depth array distance_list, and take the average result as the depth data of the candidate fruit.

[0084] S270, based on the depth data of the candidate fruit and the pixel position of the candidate fruit in the color image, determine the fruit density in different height ranges on the target fruit tree.

[0085] Optionally, the three-dimensional coordinates of the candidate fruit in the camera coordinate system are calculated based on the depth data of the candidate fruit and the pixel position of the candidate fruit in the color image, and the spatial coordinates of the candidate fruit on the target fruit tree are determined based on the three-dimensional coordinates of the candidate fruit, thereby obtaining the fruit density in different height ranges on the target fruit tree.

[0086] S280、based on the fruit density in different height ranges on the target fruit tree, control the picking robot to adjust the mechanical arm to a target height for fruit picking.

[0087] The technical scheme of the present application installs a depth camera on the lifting mechanism of the picking robot, drives the depth camera to move up and down together by controlling the lifting mechanism to move up and down, and collects images of the target fruit tree, to obtain depth images and color images in different height ranges on the target fruit tree. Based on the depth images and the color images, candidate fruits on the target fruit tree are identified and located, the spatial coordinates of the candidate fruits in the three-dimensional space are determined, and the fruit density in different height ranges on the target fruit tree is obtained. Based on the fruit density in different height ranges on the target fruit tree, the picking robot is controlled to adjust the mechanical arm to a target height for fruit picking. The embodiment of the present application combines image processing technology and robot technology, improves the mechanization degree of fruit picking, improves the fruit picking efficiency, and reduces the fruit picking cost.

[0088] In an optional embodiment, based on the fruit density in different height ranges on the target fruit tree, the picking robot is controlled to adjust the mechanical arm to a target height for fruit picking, comprising: based on the fruit density in different height ranges on the target fruit tree, determining the picking priority of different height ranges on the target fruit tree; determining the target height to be operated based on the picking priority of different height ranges, and controlling the picking robot to adjust the mechanical arm to the target height for fruit picking.

[0089] The picking priority is related to the height range, and is used to determine the fruit picking sequence corresponding to different height ranges. Optionally, the fruit density in different height ranges on the target fruit tree is sorted in descending order, and the picking priority of different height ranges on the target fruit tree is determined according to the sorting result. Optionally, the earlier the position in the sorting result is, the higher the picking priority of the height range is.

[0090] Optionally, different height ranges are obtained by cutting the target fruit tree based on a preset interval. The preset interval is determined based on the actual situation, and its specific value is not limited here. Optionally, the preset interval is determined according to the fruit tree species and the height of the target fruit tree. For example, the preset interval is 0.2 meters.

[0091] According to the determined spatial coordinates of the candidate fruits, a preset interval of 0.2 meters is set, and the fruit density of the candidate fruits in different height intervals within a height range of 0 to 3 meters is calculated. And arrange in order from large to small according to the fruit density. According to the sorting result, determine the picking priority of different height ranges on the target fruit tree. Optionally, the earlier the position in the sorting result, the higher the picking priority of the height range. Optionally, the height interval with the largest fruit density is determined as the target interval, and the picking priority of the target interval is the first priority. The middle position of the height interval is determined as the target height.

[0092] Optionally, according to the picking priority of the height interval, determine the next interval after the target interval, and control the picking robot to pick the candidate fruits in the next interval until all the fruits in all height intervals are picked. After picking is completed, control the mechanical arm of the picking robot to return to the initial position. Optionally, the initial position of the mechanical arm is the middle position of the lifting mechanism.

[0093] Illustratively, the height interval with the largest fruit density is 1.2 meters to 1.4 meters, and the middle position 1.3 meters is determined as the target height. Control the picking robot to adjust the mechanical arm to the target height through the lifting mechanism for fruit picking. Preferentially pick the fruits within the range of 1.2 meters to 1.4 meters.

[0094] The above technical solution determines the picking priority of different height ranges on the target fruit tree based on the fruit density in different height ranges on the target fruit tree, determines the target height to be operated based on the picking priority of different height ranges, and controls the picking robot to adjust the mechanical arm to the target height for fruit picking. The fruits in the height range with high fruit density can be picked preferentially, the movement distance of the mechanical arm can be reduced, the time loss and resource loss of fruit picking can be shortened, and the fruit picking efficiency can be improved.

[0095] In an optional embodiment, the control of the picking robot to adjust the mechanical arm to the target height for fruit picking comprises: controlling the picking robot to adjust the mechanical arm to the target height, and determining the working range of the mechanical arm with the target height as the working center; determining the picking order of the candidate fruits according to the working range and the depth data of the candidate fruits; determining the target fruit from the candidate fruits based on the picking order; planning the picking path for the gripper of the mechanical arm according to the spatial position of the target fruit and the spatial position of the gripper of the mechanical arm; controlling the gripper of the mechanical arm of the picking robot to move to the target fruit based on the picking path; determining the relative distance between the gripper of the mechanical arm and the target fruit according to the depth image collected by the third camera installed on the mechanical arm; determining whether the gripper of the mechanical arm reaches the specified position based on the relative distance, and if the gripper of the mechanical arm reaches the specified position, controlling the gripper of the mechanical arm to pick the target fruit.

[0096] The mechanical arm is installed on the lifting mechanism of the picking robot, and the mechanical arm can be adjusted to the target height by adjusting the height of the lifting mechanism. The working range of the mechanical arm is determined with the target height as the working center. It can be known that the candidate fruits appearing in the working range are the fruits that can be picked by the picking robot. That is, the candidate fruits outside the working range will not be picked by the picking robot. The working range of the picking robot is used to determine the picking order of the candidate fruits, and the candidate fruits outside the working range are not included in this picking, and the picking order of these candidate fruits is not determined.

[0097] For the candidate fruits appearing in the working range, the picking order of the candidate fruits is determined according to the depth data of the candidate fruits. The depth data of the candidate fruits is used to quantify the relative distance between the candidate fruits and the second camera. The depth data of the candidate fruits is actually the depth value of the fruit center. The greater the depth value, the farther the candidate fruit is from the second camera, and the closer the growth position of the candidate fruit is to the outside.

[0098] The picking order of the candidate fruits can be determined based on the depth data of the candidate fruits. Optionally, the picking strategy of first outer layer and then inner layer is adopted, that is, the candidate fruits on the outside of the target fruit tree are picked first, and then the candidate fruits on the inside of the target fruit tree are picked, until all the candidate fruits in the working range are picked.

[0099] The target fruit refers to a candidate fruit to be picked by the picking robot, and the target fruit is within the working range of the picking robot. The gripper in the picking robot is used to grasp the target fruit. Specifically, the gripper of the mechanical arm in the picking robot is controlled to move to the target fruit based on the picking path. The picking path is determined according to the spatial position of the target fruit and the spatial position of the gripper in the mechanical arm. The spatial position of the target fruit is the path endpoint of the picking path, and the spatial position of the gripper in the mechanical arm is the path starting point of the picking path. The spatial position of the target fruit is calculated according to the depth data of the target fruit and the pixel position of the target fruit in the color image.

[0100] In the process of moving the gripper of the mechanical arm in the picking robot to the target fruit, the relative distance between the gripper of the mechanical arm and the target fruit is determined according to the depth image collected by the third camera installed on the mechanical arm. Whether the gripper of the mechanical arm reaches the specified position is determined based on the relative distance. If the gripper of the mechanical arm reaches the specified position, the gripper of the mechanical arm is controlled to pick the target fruit.

[0101] Optionally, after the gripper of the mechanical arm reaches the target fruit, the solenoid valve is powered on to control the gripper to close, the end joint of the mechanical arm is rotated back and forth at least twice, and the tension is measured by the tension sensor arranged on the end joint. When the tension is less than the set tension threshold, it is determined that the target fruit and the fruit stem on the target fruit branch have been separated, which indicates that the target fruit has been picked from the target fruit tree. The tension threshold is used to measure whether the target fruit and the fruit stem are separated. The tension threshold is not limited here and is determined according to the actual situation. Exemplarily, the tension threshold is 2N.

[0102] Optionally, after the target fruit is picked, the mechanical arm drives the gripper to run to the fruit collection port and places the fruit into the pneumatic fruit conveying pipe. The pneumatic peristaltic air bag inside the conveying pipe allows the fruit to slowly flow into the fruit storage box along the pipe, effectively preventing the fruit from colliding or being blocked in the pipeline.

[0103] Optionally, according to the picking order of the candidate fruit, the next fruit after the target fruit in the picking order is determined, and the picking robot is controlled to pick the next fruit until all the candidate fruits within the working range are picked.

[0104] The above technical solution provides a practical fruit picking method by controlling the picking robot to adjust the mechanical arm to the target height for fruit picking, which provides technical support for improving the mechanization degree of fruit picking, improving the fruit picking efficiency, and reducing the fruit picking cost.

[0105] Optionally, the picking robot is a wheeled robot. Figure 2B is a first structural schematic diagram of a picking robot provided according to Embodiment Two of the present application; Figure 2Cis a second structural schematic diagram of a picking robot provided according to Embodiment Two of the present application. The first structural schematic diagram and the second structural schematic diagram correspond to different perspectives.

[0106] Referring to Figure 2B and Figure 2C , the picking robot comprises an unmanned mobile chassis, a navigation module, a lifting mechanism, a mechanical arm, a pneumatic fruit conveying pipe, a fruit storage box and the like. The first camera and the laser radar are installed on the unmanned mobile chassis. The first camera and the laser radar are used to determine a picking route. The navigation module is used to control the picking robot to move along the picking route and to control the picking robot to stop when the picking robot reaches a picking point. The lifting mechanism is connected with the mechanical arm. Specifically, the mechanical arm is installed on the lifting mechanism. The lifting mechanism is used to adjust the height of the mechanical arm. The mechanical arm is used to pick fruits. The first camera is a color camera.

[0107] Figure 2D is a partial structural schematic diagram of a picking robot provided according to Embodiment Two of the present application. As shown in Figure 2D , a second camera is installed on the lifting mechanism. The second camera is a depth camera. The second camera performs image acquisition to obtain depth images and color images in different height ranges of the target fruit tree during the up-and-down movement of the lifting mechanism, so as to identify and locate the fruits on the target fruit tree based on the depth images and the color images.

[0108] A gripper is installed on the mechanical arm and is used to grab fruits from the target fruit tree. A third camera is installed on the mechanical arm close to the gripper. The third camera is a depth camera. The third camera is used to determine the relative distance between the gripper of the mechanical arm and the target fruit during the movement of the gripper of the mechanical arm towards the target fruit, so as to determine whether the gripper of the mechanical arm has grabbed the fruit.

[0109] After the fruits are picked, the mechanical arm drives the gripper to run to a fruit collection port and places the fruits into the pneumatic fruit conveying pipe. The pneumatic fruit conveying pipe has a pneumatic peristaltic air bag inside, so that the fruits slowly flow into the fruit storage box along the pipe, effectively preventing the fruits from colliding or being blocked in the pipe.

[0110] Continuing to refer to Figures 2B to 2D , each of the left and right sides of the picking robot is provided with a set of picking device. The picking device comprises a lifting mechanism, a mechanical arm, a pneumatic fruit conveying pipe, a fruit storage box, a pneumatic fruit conveying pipe, a gripper, a first camera and a second camera and the like. It is worth noting that the picking device configured on the picking robot is not limited to two sets. The number of sets of picking devices that can be configured on the picking robot can be determined according to the size of the chassis of the unmanned mobile chassis. For example, when the chassis of the unmanned mobile chassis is large enough, four sets or even more sets of picking devices can be arranged on the picking robot.

[0111] Optionally, in the process of fruit picking by the picking robot, the plurality of collection devices arranged on the picking robot are controlled to simultaneously pick fruits, so as to further improve the fruit picking efficiency.

[0112] Embodiment Three

[0113] Figure 3 is a structural schematic diagram of the fruit picking device provided in Embodiment Three of the present application. The present embodiment can be applied to the case of picking fruits in a dwarf and dense planting orchard. The device can be implemented by software and / or hardware, and can be integrated into an electronic device.

[0114] As shown in Figure 3 , the device can include:

[0115] The picking route determination module 310 is configured to determine the growth positions of the target fruit tree on both sides of the fruit tree row, and determine the picking route of the picking robot based on the growth positions. The picking route includes at least two picking points.

[0116] The fruit density determination module 320 is configured to control the picking robot to move along the picking route, control the picking robot to stop when the picking robot reaches the picking point, and determine the fruit density in different height ranges of the target fruit tree.

[0117] The target height adjustment module 330 is configured to control the picking robot to adjust the mechanical arm to a target height for fruit picking based on the fruit density in different height ranges of the target fruit tree.

[0118] The technical solution of the present application determines the picking route of the picking robot based on the growth positions of the target fruit tree on both sides of the fruit tree row, controls the picking robot to move along the picking route, and controls the picking robot to stop when the picking robot reaches the picking point, thereby providing data support for using robot technology for fruit picking. The present embodiment determines the fruit density in different height ranges of the target fruit tree, and controls the picking robot to adjust the mechanical arm to a target height for fruit picking based on the fruit density in different height ranges of the target fruit tree, thereby reducing the movement distance of the mechanical arm, shortening the time loss and resource loss of fruit picking, and improving the fruit picking efficiency. The present embodiment uses the picking robot for fruit picking, thereby reducing the fruit picking cost.

[0119] Optionally, the picking route determination module 310 comprises: a first image acquisition submodule configured to control the picking robot to acquire image data of both sides of the fruit tree rows; a target fruit tree determination submodule configured to input the image data into a pre-trained semantic segmentation model, and use the semantic segmentation model to segment target fruit trees growing on both sides of the fruit tree rows from the image data and determine image coordinates of the target fruit trees; a point cloud coordinate determination submodule configured to determine point cloud coordinates of the target fruit trees according to a coordinate conversion matrix and the image coordinates of the target fruit trees; and a growth position determination submodule configured to project the point cloud coordinates of the target fruit trees onto the ground to obtain growth positions of the target fruit trees on both sides of the fruit tree rows; wherein the coordinate conversion matrix is obtained by jointly calibrating a first camera and a laser radar, the first camera is configured to acquire the image data of both sides of the fruit tree rows, and the laser radar is configured to acquire point cloud data of both sides of the fruit tree rows.

[0120] Optionally, the picking route determination module 310 comprises: a curve fitting submodule configured to perform curve fitting on the growth positions of the target fruit trees on both sides of the fruit tree rows to obtain a left fruit tree row curve and a right fruit tree row curve on both sides of the fruit tree rows; a picking route determination submodule configured to determine a center line between the left fruit tree row curve and the right fruit tree row curve, and determine a picking route of the picking robot based on the center line; and a picking point position determination submodule configured to determine at least two picking point positions in the picking route according to a working distance of the picking robot.

[0121] Optionally, the fruit density determination module 320 comprises: a posture position acquisition submodule configured to acquire a real-time position and a real-time posture of the picking robot; an angle and distance determination submodule configured to determine a yaw angle and a lateral offset distance of the picking robot relative to the picking route based on the picking route and the real-time position and the real-time posture of the picking robot; a driving unit control submodule configured to control a walking driving unit and a steering driving unit of the picking robot based on the yaw angle and the lateral offset distance to enable the picking robot to move along the picking route; and a control stopping submodule configured to determine whether the picking robot reaches the picking point position according to the real-time position of the picking robot, and control the picking robot to stop in a case where the picking robot reaches the picking point position.

[0122] Optionally, the fruit density determination module 320 comprises: a movement control submodule for controlling the lifting mechanism of the picking robot to move up and down; wherein the lifting mechanism is provided with a second camera, and the second camera is a depth camera; a second image acquisition submodule for controlling the second camera to acquire images to obtain depth images and color images in different height ranges of the target fruit tree during the movement of the lifting mechanism; a fruit recognition submodule for inputting the color images into a pre-trained target recognition model, and using the target recognition model to recognize candidate fruits to be picked in the color images to determine pixel positions of the candidate fruits in the color images; an image registration submodule for registering the depth images and the color images, and extracting depth data of the candidate fruits from the registered depth images based on the pixel positions of the candidate fruits in the color images; and a fruit density determination submodule for determining the fruit density in different height ranges of the target fruit tree based on the depth data of the candidate fruits and the pixel positions of the candidate fruits in the color images.

[0123] Optionally, the target height adjustment module 330 comprises: a picking priority determination submodule for determining picking priorities of different height ranges of the target fruit tree based on the fruit density in the different height ranges; and a target height determination submodule for determining a target height to be worked based on the picking priorities of the different height ranges, and controlling the picking robot to adjust the mechanical arm to the target height for fruit picking.

[0124] The target height adjustment module 330 comprises: a working range determination submodule for controlling the picking robot to adjust the mechanical arm to the target height, and determining a working range of the mechanical arm with the target height as the working center; a picking sequence determination submodule for determining a picking sequence of the candidate fruits according to the working range and the depth data of the candidate fruits; a spatial position determination submodule for determining a target fruit from the candidate fruits based on the picking sequence; a picking path determination submodule for planning a picking path for the gripper of the mechanical arm according to the spatial position of the target fruit and the spatial position of the gripper in the mechanical arm; a gripper control submodule for controlling the gripper of the mechanical arm of the picking robot to move to the target fruit based on the picking path; a distance determination submodule for determining a relative distance between the gripper of the mechanical arm and the target fruit according to a depth image acquired by a third camera installed on the mechanical arm; and a fruit picking submodule for determining whether the gripper of the mechanical arm reaches a specified position based on the relative distance, and picking the target fruit by controlling the gripper of the mechanical arm to reach the specified position if the gripper of the mechanical arm reaches the specified position.

[0125] The fruit picking device provided by the embodiment can execute the fruit picking method provided by any embodiment of the application, has the corresponding performance module and beneficial effects for executing the fruit picking method.

[0126] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user data comply with relevant laws and regulations and do not violate public order and good customs.

[0127] Embodiment Four

[0128] Figure 4 is a structural schematic diagram of a mechanical arm control system deployed in a picking robot. Referring to Figure 4 , the mechanical arm control system deployed in the picking robot includes an industrial computer, a second camera, a third camera and a motion controller. The second camera and the third camera are in communication connection with the industrial computer through USB data lines; the second camera is a depth camera and is installed on a lifting mechanism. The second camera is used for identifying and positioning fruits on a target fruit tree; the third camera is a depth camera and is installed at a position close to a gripper of the mechanical arm; the third camera is used for determining the relative distance between the gripper of the mechanical arm and the target fruit in the process that the gripper of the mechanical arm moves towards the target fruit, so as to determine whether the gripper of the mechanical arm grasps the fruit.

[0129] Optionally, the picking robot is driven by a motor or by hydraulic pressure. Figure 4 is shown a mechanical arm control system deployed in a picking robot driven by a motor. Referring to Figure 4 , the motion controller is in communication connection with the industrial computer through an EtherCAT bus, and the motion controller is also in communication connection with a series mechanical arm joint motor control system, a mechanical arm end effector control system and a lifting device control system through a CAN bus.

[0130] The series mechanical arm joint motor control system includes a plurality of series mechanical arm joint motor drivers and a plurality of series mechanical arm joint motors. The series mechanical arm joint motor drivers are in communication connection with the series mechanical arm joint motors. The motion controller is in communication connection with the series mechanical arm joint motor drivers through the CAN bus, and the series mechanical arm joint motors can be controlled through the series mechanical arm joint motor drivers.

[0131] The mechanical arm end effector control system includes a mechanical arm end effector electromagnetic valve, a mechanical arm end gripper cylinder and an end tension and pressure sensor. The mechanical arm end effector electromagnetic valve is in communication connection with the motion controller through a digital quantity IO interface, and the motion controller can control the mechanical arm end gripper cylinder through the mechanical arm end effector electromagnetic valve. The end tension and pressure sensor is in communication connection with the motion controller through an analog quantity interface.

[0132] The lifting device control system comprises a mechanical arm overall height lifting motor driver, a mechanical arm overall height lifting motor and a lifting screw grating ruler displacement sensor. The mechanical arm overall height lifting motor driver is in communication connection with the motion controller through a CAN bus. The mechanical arm overall height lifting motor driver is in communication connection with the mechanical arm overall height lifting motor and the lifting screw grating ruler displacement sensor respectively.

[0133] A fruit picking method performed by the mechanical arm control system comprises the following steps: 1) initializing the mechanical arm control system, starting the device driver program, the IO interface service program and the coordinate transformation program; 2) controlling the lifting device control system through the motion controller, controlling the lifting mechanism to run to the highest position, controlling the lifting motor to rotate to drive the lifting mechanism to move uniformly from top to bottom, and continuously collecting multiple images (including color images and depth images) by the second camera installed on the lifting mechanism in the process of the lifting mechanism running from top to bottom, and transmitting the color images and the depth images to the industrial computer; 3) running the pre-trained target recognition model on the industrial computer to identify the candidate fruit to be picked in the color image to determine the pixel position of the candidate fruit in the color image; 4) registering the depth image with the color image, extracting the depth data of the candidate fruit from the registered depth image based on the pixel position of the candidate fruit in the color image, and obtaining the spatial coordinates of the candidate fruit based on the depth data of the candidate fruit and the pixel position of the candidate fruit in the color image.

[0134] 5) After determining the spatial coordinates of the candidate fruits, a preset interval of 0.2 meters is set, the fruit density of the candidate fruits in different height intervals within the range of 0 to 3 meters is calculated, and the candidate fruits are arranged in array A in descending order of fruit density. The height interval with the largest fruit density is selected as the target interval, and the height of the mechanical arm is adjusted to the middle position of the target interval by the lifting mechanism. 6) According to the depth data of the candidate fruits, the relative distance of the candidate fruits to the fruit collection port is determined, and the candidate fruits are stored in array B in descending order of the relative distance, i.e., the picking order is obtained; 7) The spatial coordinates of the candidate fruits are transformed, and the spatial coordinates of the candidate fruits are transformed from the earth coordinate system to the camera coordinate system. According to the working range of the mechanical arm, it is determined whether the candidate fruit is within the working range. If it is outside the working range, the candidate fruit is rejected; 8) The candidate fruits within the working range are taken as the target fruits that can be picked, and the spatial coordinates of the target fruits are sent to the motion planner. The picking path of the mechanical arm is planned by the motion planner, and the picking path of the mechanical arm is sent to the motion controller. The motion controller obtains the angle of each serial mechanical arm joint motor through the inverse kinematics calculation of the mechanical arm, and then sends it to each serial mechanical arm joint motor driver through the CAN bus to control the mechanical arm to move the gripper towards the target fruit. 9) According to the depth image collected by the third camera installed on the mechanical arm, the relative distance between the gripper of the mechanical arm and the target fruit is determined, and it is determined whether the gripper of the mechanical arm reaches the position of the target fruit. 10) After the gripper of the mechanical arm reaches the position of the target fruit, the motion controller controls the mechanical arm end effector electromagnetic valve to be energized to close the gripper, and the mechanical arm end joint rotates back and forth by 180° for 3 times. When the tension measured by the end tension sensor is less than the set tension threshold (for example, 2.0N), it is determined that the target fruit and the fruit stem on the branch have been separated, and the target fruit has been picked from the target fruit tree.

[0135] 11) After the target fruit is picked, the mechanical arm drives the gripper to run to the fruit collection port and places the fruit into the conveying pipe. The conveying pipe has a pneumatic peristaltic air bag inside, which makes the fruit flow slowly along the pipe into the fruit collection basket, effectively preventing the fruit from colliding or being blocked in the pipe. 12) The industrial computer directly starts the next target fruit grabbing according to the spatial coordinates of the next fruit for picking path planning, until the target fruits in the target interval are picked. 13) According to the fruit density of different height intervals, the mechanical arm is adjusted to the middle position of the next interval. The second camera collects the depth image and color image of the target fruit tree in the next interval and transmits them to the industrial computer. Repeat steps 4) to 13), and determine whether all target fruits in all height intervals have been picked according to the depth image and color image collected by the second camera. If all target fruits have been picked, the mechanical arm is commanded to return to the initial position.

[0136] It should be understood that the various forms of flow shown above can be used to reorder, add, or remove steps. For example, the steps recited in the present application can be performed in parallel, in series, or in a different order, as long as the desired results of the present application are achieved, which is not limited herein.

[0137] The foregoing detailed description has not been limited by a particular embodiment thereof. Alternative embodiments, which practice the application, will be apparent to those skilled in the art upon reading this disclosure. For instance, it should be appreciated that those skilled in the art can make modifications to the specific embodiments described herein, or combine, sub-combine, and substitute various features, without departing from the spirit and scope of the application. Accordingly, the scope of the application should be determined by the following claims.

Claims

1. A method of fruit picking, characterized in that, The method comprises: determining the growth position of the target fruit tree on both sides of the fruit tree row, and determining the picking route of the picking robot based on the growth position; the picking route comprises at least two picking points; controlling the picking robot to move along the picking route, stopping the picking robot when the picking robot reaches the picking point, and determining the fruit density in different height ranges of the target fruit tree; based on the fruit density in different height ranges of the target fruit tree, controlling the picking robot to adjust the mechanical arm to the target height for fruit picking; determining the growth position of the target fruit tree on both sides of the fruit tree row, and determining the picking route of the picking robot based on the growth position; the picking route comprises at least two picking points; curve fitting is performed on the growth position of the target fruit tree on both sides of the fruit tree row to obtain left and right fruit tree row curves on both sides of the fruit tree row; determining the center line between the left and right fruit tree row curves, and determining the picking route of the picking robot based on the center line; determining at least two picking points in the picking route according to the working distance of the picking robot; the control of the picking robot to move along the picking route, and the control of the picking robot to stop when the picking robot reaches the picking point, comprises: obtaining the real-time position and real-time attitude of the picking robot; based on the picking route and the real-time position and real-time attitude of the picking robot, determining the yaw angle and lateral offset distance of the picking robot relative to the picking route; based on the yaw angle and the lateral offset distance, controlling the walking drive unit and the steering drive unit of the picking robot to move along the picking route; determining whether the picking robot reaches the picking point according to the real-time position of the picking robot, and controlling the picking robot to stop when the picking robot reaches the picking point; determining the fruit density in different height ranges of the target fruit tree, comprises: controlling the lifting mechanism of the picking robot to move up and down; wherein the second camera is installed on the lifting mechanism, and the second camera is a depth camera; controlling the second camera to collect images to obtain depth images and color images in different height ranges of the target fruit tree in the process of moving the lifting mechanism up and down; inputting the color image into a pre-trained target recognition model, and using the target recognition model to identify the candidate fruit to be picked in the color image to determine the pixel position of the candidate fruit in the color image; registering the depth image and the color image, and extracting the depth data of the candidate fruit from the registered depth image based on the pixel position of the candidate fruit in the color image; based on the depth data of the candidate fruit and the pixel position of the candidate fruit in the color image, determining the fruit density in different height ranges of the target fruit tree; based on the fruit density in different height ranges of the target fruit tree, controlling the picking robot to adjust the mechanical arm to the target height for fruit picking, comprises: determine a picking priority of different height ranges on the target fruit tree based on the fruit density in the different height ranges on the target fruit tree; determine a target height to be operated based on the picking priority of different height ranges, and control the picking robot to adjust a mechanical arm to the target height for fruit picking; controlling the picking robot to adjust the mechanical arm to the target height for fruit picking, comprising: controlling the picking robot to adjust the mechanical arm to the target height, and determining a working range of the mechanical arm with the target height as a working center; determining a picking order of the candidate fruits according to the working range and the depth data of the candidate fruits; determining a target fruit from the candidate fruits based on the picking order; planning a picking path for a gripper of the mechanical arm according to the spatial position of the target fruit and the spatial position of the gripper in the mechanical arm; controlling the gripper of the mechanical arm in the picking robot to move to the target fruit based on the picking path; determining a relative distance between the gripper of the mechanical arm and the target fruit according to a depth image collected by a third camera installed on the mechanical arm; determining whether the gripper of the mechanical arm reaches a specified position based on the relative distance, and if the gripper of the mechanical arm reaches the specified position, controlling the gripper of the mechanical arm to pick the target fruit.

2. The method of claim 1, wherein, The determination of the growth positions of the target fruit trees on both sides of the fruit tree rows comprises: controlling the picking robot to collect image data on both sides of the fruit tree rows; inputting the image data into a pre-trained semantic segmentation model, segmenting the target fruit trees growing on both sides of the fruit tree rows from the image data by using the semantic segmentation model, and determining image coordinates of the target fruit trees; determining point cloud coordinates of the target fruit trees according to a coordinate conversion matrix and the image coordinates of the target fruit trees; projecting the point cloud coordinates of the target fruit trees onto the ground to obtain the growth positions of the target fruit trees on both sides of the fruit tree rows; wherein the coordinate conversion matrix is obtained by jointly calibrating a first camera and a laser radar; the first camera is used to collect the image data on both sides of the fruit tree rows, and the laser radar is used to collect point cloud data on both sides of the fruit tree rows.

3. A fruit picking device, characterized in that The device comprises: a picking route determination module configured to determine growth positions of target fruit trees on both sides of fruit tree rows, and determine a picking route of a picking robot based on the growth positions; the picking route comprises at least two picking points; a fruit density determination module configured to control the picking robot to move along the picking route, stop the picking robot when the picking robot reaches the picking points, and determine fruit density in different height ranges on the target fruit trees; a target height adjustment module configured to control the picking robot to adjust a mechanical arm to a target height for fruit picking based on the fruit density in the different height ranges on the target fruit trees.

4. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by a processor to implement the fruit picking method of any one of claims 1-2.

5. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the fruit picking method of any one of claims 1-2.

Images