Picking robot, fruit positioning method therefor, apparatus thereof, electronic device, and medium

Abstract

A picking robot, a fruit positioning method therefor, an apparatus, an electronic device, and a medium. The robot comprises a robot body, the robot body comprising a processor (101). A plurality of first image acquisition modules (200) and a second image acquisition module (300) are mounted on the robot body. The robot body comprises a plurality of mechanical arms (104), a first image acquisition module (200) is correspondingly mounted around each mechanical arm (104), and each first image acquisition module and the corresponding mechanical arm do not interfere with one another. The second image acquisition module (300) is mounted at a base position of the robot body and is used for determining a base coordinate system. On the basis of acquiring the base coordinate system and a fruit tree image of a sub-area corresponding to each first image acquisition module (200) in an operation area, the processor (101) determines global fruit positioning distribution information in the operation area and controls each mechanical arm (104) to perform cooperative operation. According to the described picking robot, the precision and range of fruit positioning can be improved, and the fruit picking efficiency of the robot is greatly improved.

Description

Picking robot and fruit positioning method, device, electronic equipment and medium thereof

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on October 9, 2023, with application number 202311296030.7 and invention name “Picking robot and its fruit positioning method, device, electronic equipment and medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present invention relates to the field of smart agricultural technology, and in particular to a picking robot and a fruit positioning method, device, electronic equipment and medium thereof. Background Art

[0003] In the context of agricultural labor shortages, robotic harvesting is an urgent need for the development of the fruit and vegetable industry. Researchers have conducted extensive research on fruit information acquisition methods and systems for harvesting robots and have made positive progress.

[0004] Existing robotic harvesting systems typically use a single visual sensor, using deep learning and traditional machine vision techniques to identify and locate fruit and guide the harvesting robot's operations. As the number of tasks continues to increase, harvesting robots that integrate multiple robotic arms and end effectors are gaining increasing attention.

[0005] Multi-arm picking robots have numerous mechanisms, a wide operating range, and are prone to interference between their arms. Pre-emptive knowledge of fruit distribution is crucial for control and planning, significantly impacting picking efficiency. However, existing multi-arm picking robots primarily utilize a single vision sensor, mounted far from the work surface. This long sensing distance results in significant deviations in fruit positioning, resulting in low picking efficiency.

[0006] Summary of the Invention

[0007] In view of the shortcomings of the existing technology, the purpose of the present invention is to provide a robot operation planning method, system and application thereof, the specific scheme is as follows:

[0008] The present invention provides a picking robot and a fruit positioning method, device, electronic equipment and medium thereof, which are used to solve the defects of the multi-arm picking robot in the prior art, such as large fruit positioning deviation and low picking efficiency.

[0009] The present invention provides a picking robot, comprising:

[0010] A robot body, the robot body including a processor; a plurality of first image acquisition modules and a second image acquisition module are mounted on the robot body; the robot body includes a plurality of robotic arms;

[0011] A first image acquisition module is installed around each robotic arm, and each first image acquisition module does not interfere with the corresponding robotic arm; the second image acquisition module is installed at the base of the robot body to determine the base coordinate system;

[0012] The processor is used to determine the global fruit positioning distribution information of the working area based on the fruit tree images of the corresponding sub-areas in the working area captured by each first image acquisition module and the base coordinate system, and determine the local fruit positioning distribution information corresponding to each robotic arm according to the global fruit positioning distribution information to control each robotic arm to perform collaborative work.

[0013] According to a picking robot provided by the present invention, the robot body includes a main frame and a plurality of connecting rods installed on the main frame;

[0014] At least two robotic arms are mounted on each connecting rod, and a first image acquisition module corresponding to each robotic arm is mounted on each connecting rod. Each first image acquisition module is located near an end joint of the corresponding robotic arm.

[0015] According to a picking robot provided by the present invention, the robotic arm is a telescopic robotic arm; each first image acquisition module is installed on the side of the corresponding telescopic robotic arm close to the claw, and the central axis of the shooting angle of each first image acquisition module is consistent with the telescopic direction of the telescopic robotic arm.

[0016] The present invention also provides a fruit positioning method applied to any of the above-mentioned picking robots, comprising:

[0017] Obtaining fruit tree images of corresponding sub-areas in the operation area captured by each first image acquisition module, and inputting each of the fruit tree images into a preset target detection model to obtain two-dimensional bounding box information and mask areas of each fruit in each of the fruit tree images output by the preset target detection model;

[0018] generating a three-dimensional point cloud of each fruit in each fruit tree image using the mask area and corresponding image depth information of each fruit in each fruit tree image, and determining a positioning coordinate point of each fruit in each fruit tree image based on the three-dimensional point cloud and two-dimensional bounding box information of each fruit in each fruit tree image;

[0019] Based on the result of converting the positioning coordinate points of each fruit in each of the fruit tree images into the base coordinate system, the global fruit positioning distribution information of the operation area is determined.

[0020] According to a fruit positioning method provided by the present invention, the positioning coordinate points of each fruit in each fruit tree image are determined based on the three-dimensional point cloud and two-dimensional bounding box information of each fruit in each fruit tree image, including:

[0021] Using a point cloud clustering algorithm, clustering calculation is performed on the three-dimensional point cloud of each fruit in each of the fruit tree images to determine the surface feature points of each fruit in each of the fruit tree images;

[0022] generating a three-dimensional viewing cone and a center line of the viewing cone corresponding to each fruit in each fruit tree image according to the two-dimensional bounding box information of each fruit in each fruit tree image;

[0023] Based on the center line of the viewing cone and the surface feature points corresponding to each fruit in each of the fruit tree images, the positioning coordinate points of each fruit in each of the fruit tree images are determined.

[0024] According to a fruit positioning method provided by the present invention, the positioning coordinate point of each fruit in each fruit tree image is determined based on the center line of the viewing cone and the surface feature points corresponding to each fruit in each fruit tree image, including:

[0025] For each fruit in each of the fruit tree images, construct a sphere with the surface feature point as the sphere center and the target length as the radius; the target length is determined based on the depth value corresponding to the surface feature point;

[0026] Determine two intersection points where the center line of the viewing cone corresponding to each fruit in each of the fruit tree images passes through the corresponding sphere;

[0027] From the two intersection points corresponding to the fruits in each of the fruit tree images, an intersection point with a greater distance from the shooting focus is determined as a positioning coordinate point of each of the fruits in each of the fruit tree images.

[0028] According to a fruit positioning method provided by the present invention, the method determines the global fruit positioning distribution information of the operation area based on the result of converting the positioning coordinate points of each fruit in each fruit tree image into a base coordinate system, including:

[0029] Converting the positioning coordinate points of each fruit in each of the fruit tree images to the base coordinate system to obtain the positioning coordinate points of each fruit in each of the fruit tree images in the base coordinate system;

[0030] Determining pairs of positioning coordinate points between adjacent fruits according to the positioning coordinate points of each fruit in the base coordinate system, and determining the distance between each pair of positioning coordinate points in the base coordinate system;

[0031] Determine the target positioning coordinate point pairs whose distance is less than the target threshold, and remove one positioning coordinate point from each target positioning coordinate point pair;

[0032] According to each positioning coordinate point retained in the base coordinate system, global fruit positioning distribution information of the operation area is generated.

[0033] The present invention also provides a fruit positioning device, comprising:

[0034] an output module configured to obtain images of fruit trees in corresponding sub-areas of the acquisition operation area by each first image acquisition module, and input each of the fruit tree images into a preset target detection model to obtain two-dimensional bounding box information and a mask area of ​​each fruit in each of the fruit tree images output by the preset target detection model;

[0035] a positioning module, configured to generate a three-dimensional point cloud of each fruit in each of the fruit tree images using the mask area and corresponding image depth information of each fruit in each of the fruit tree images, and determine the positioning coordinate points of each fruit in each of the fruit tree images based on the three-dimensional point cloud and two-dimensional bounding box information of each fruit in each of the fruit tree images;

[0036] The processing module is used to determine the global fruit positioning distribution information of the operation area based on the result of converting the positioning coordinate points of each fruit in each fruit tree image into the base coordinate system.

[0037] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-described fruit positioning methods is implemented.

[0038] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which implements any of the above-mentioned fruit positioning methods when executed by a processor.

[0039] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned fruit positioning methods.

[0040] The picking robot and its fruit positioning method, device, electronic device and medium provided by the present invention perform multi-view image acquisition by setting multiple image acquisition modules on the multi-arm picking robot body, and use a processor to uniformly convert the images acquired from various viewpoints into the robot's base coordinate system, synchronously obtain visual information of all picking targets in the working area, and generate global fruit positioning distribution information that matches the range size of the robot's working space, which is conducive to the efficient and coordinated operation of each robotic arm, can achieve accurate detection of fruits at a relatively close distance in the working area, obtain fruit information within a larger range, and improve the accuracy and range of fruit positioning, which is conducive to greatly improving the robot's fruit picking efficiency.

[0041] Figures in the specification

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] FIG1 is a schematic diagram of the structure of a picking robot provided by the present invention;

[0044] FIG2 is a second structural diagram of the picking robot provided by the present invention;

[0045] FIG3 is a third structural diagram of the picking robot provided by the present invention;

[0046] FIG4 is a schematic flow chart of a fruit positioning method provided by the present invention;

[0047] FIG5 is a schematic structural diagram of a fruit positioning device provided by the present invention;

[0048] FIG6 is a schematic diagram of the physical structure of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0049] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0050] In the description of the invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood broadly. For example, they may refer to fixed, detachable, or integral connections; mechanical or electrical connections; direct or indirect connections through an intermediary; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0051] The following describes the harvesting robot and its fruit positioning method, device, electronic device and medium of the present invention in conjunction with Figures 1 to 6.

[0052] FIG1 is a schematic diagram of the structure of a picking robot provided by the present invention. As shown in FIG1 , the picking robot may include:

[0053] The robot body 100 includes a processor 101; a plurality of first image acquisition modules 200 are installed on the robot body 100, including a first image acquisition module 1, a first image acquisition module 2, ..., a first image acquisition module n, and a second image acquisition module 300; the robot body includes a plurality of robotic arms; n represents the number of the first image acquisition modules 200, and n is greater than 1.

[0054] A first image acquisition module 200 is installed around each robotic arm, and each first image acquisition module 200 does not interfere with the corresponding robotic arm; a second image acquisition module 300 is installed at the base position of the robot body 1 for determining the base coordinate system.

[0055] The processor 101 is used to determine the global fruit positioning distribution information of the working area based on the fruit tree images and base coordinate system of the corresponding sub-area in the working area captured by each first image acquisition module 200, and determine the local fruit positioning distribution information corresponding to each robotic arm according to the global fruit positioning distribution information, so as to control each robotic arm to perform collaborative work.

[0056] Specifically, in an embodiment of the present invention, the image acquisition module can specifically adopt a color depth camera based on different ranging principles, such as various types of color depth cameras (also called stereo vision cameras) based on structured light technology, binocular vision stereo matching technology or time of flight, to obtain images of fruit trees in the working area. The fruit tree images can include color images and depth images of the fruit tree working surface.

[0057] The first image acquisition module described in the embodiment of the present invention refers to an image acquisition module for acquiring images of fruit trees in each sub-area of ​​a fruit tree picking operation area.

[0058] The second image acquisition module described in the embodiment of the present invention refers to an image acquisition module used to determine the base coordinate system of the robot and assist in calibrating each first image acquisition module.

[0059] It should be noted that the base coordinate system is also called the robot coordinate system. It is a virtual Cartesian coordinate system based on the robot installation base and used to describe the movement of the robot body.

[0060] The global fruit location distribution information described in the embodiment of the present invention refers to the fruit location distribution on a global scale in the entire picking operation area that is fused and covered by the shooting ranges of all first image acquisition modules.

[0061] The local fruit location distribution information described in the embodiment of the present invention refers to the fruit location distribution corresponding to the sub-area range that each robotic arm is responsible for picking after segmenting the global fruit location distribution information of the entire operation area.

[0062] In an embodiment of the present invention, multiple first image acquisition modules and one second image acquisition module are installed on the robot body, and a first image acquisition module is installed around each robotic arm. Each first image acquisition module can assist the corresponding robotic arm in fruit picking operations.

[0063] Among them, the installation angle of each first image acquisition module can be flexibly adjusted according to the actual specific operation conditions. The principle of adjustment is: to ensure that there is no interference with the robot arm body and the line of sight is not blocked by the robot arm body.

[0064] FIG2 is a second structural diagram of the harvesting robot provided by the present invention. As shown in FIG2 , the robot body 100 further includes a main body frame 102 and a plurality of connecting rods 103 layeredly mounted on the main body frame 102 .

[0065] At least two robotic arms 104 are mounted on each connecting rod 103 . A first image acquisition module 200 corresponding to each robotic arm 104 is mounted on each connecting rod 103 . Each first image acquisition module 200 is located near the end joint of the corresponding robotic arm 104 .

[0066] Optionally, a fruit collection conveyor belt 105 may be installed on the main frame 102 to collect the fruits picked by the robotic arm 104. Depending on the actual operation requirements, the first image acquisition module 200 may be installed on the side of the fruit collection conveyor belt 105 near the end joint of the robotic arm 104.

[0067] It is understandable that the robot body may further include a mobile platform, and the bottom of the main frame of the fuselage is fixedly connected to the mobile platform, such as by bolts, for flexibly moving the robot to the working area.

[0068] Specifically, in an embodiment of the present invention, the main frame of the picking robot body can adopt a door frame structure, and support the installation and fixation of the robotic arm drive mechanism and multiple image acquisition modules by arranging multiple layers of connecting rods.

[0069] Among them, at least two robotic arms are installed on each connecting rod, that is, it can be adaptively adjusted according to the actual size of the fruit tree. For a large operating area, more than two robotic arms can also be installed on each connecting rod. The present invention does not make specific limitations on this.

[0070] Wherein, on each connecting rod, a corresponding first image acquisition module is installed near the end joint of each robotic arm.

[0071] It should be noted that, in the embodiment of the present invention, the robotic arm can be a rectangular coordinate type robotic arm, and each first image acquisition module can also be installed on other devices near the end joint of the robot, and the image acquisition line of sight direction forms a certain angle with the telescopic direction of the telescopic arm.

[0072] Figure 3 is the third structural schematic diagram of the picking robot provided by the present invention. As shown in Figure 3, the robotic arm 104 can adopt a telescopic robotic arm; each first image acquisition module 200 is installed on the side of the corresponding telescopic robotic arm 104 close to the claw, and the central axis of the shooting angle of each first image acquisition module 200 is consistent with the telescopic direction of the telescopic robotic arm.

[0073] The picking robot in the embodiment of the present invention adopts a telescopic robotic arm and is installed in accordance with the central axis of the shooting angle of the first image acquisition module and the telescopic direction of the telescopic robotic arm. This can effectively reduce the impact of the robotic arm on the shooting of the image acquisition module during operation. At the same time, it is convenient for the robotic arm to pick fruits in time without adjusting the picking angle, which is conducive to improving picking efficiency.

[0074] 2 , in an embodiment of the present invention, the second image acquisition module may be installed at a base position on the main frame of the fuselage to determine the base coordinate system of the robot and calibrate each first image acquisition module.

[0075] The harvesting robot of the embodiment of the present invention is manufactured by adopting a main frame structure, which has a simple structure, facilitates the installation of various image acquisition modules and mechanical arms, consumes less materials, saves manufacturing costs, and is conducive to energy conservation and emission reduction.

[0076] In an embodiment of the present invention, a second image acquisition module is installed at the base of the robot body. By measuring the pose matrix of the second image acquisition module, a base coordinate system can be determined. This allows the second image acquisition module to calibrate the coordinate systems of the images from each of the first image acquisition modules, converting them into a unified base coordinate system. This facilitates image fusion processing and allows for the acquisition of global fruit distribution information within the work area.

[0077] In an embodiment of the present invention, each image acquisition module is electrically connected to a processor, and the processor can obtain the fruit tree images of the corresponding sub-area in the operation area captured by each first image acquisition module, as well as the base coordinate system, and can calibrate the coordinate system of the fruit tree images captured by each first image acquisition module, and uniformly convert each fruit tree image into the base coordinate system for representation.

[0078] More specifically, in an embodiment of the present invention, a specific method for calibrating multiple first image acquisition modules includes:

[0079] First, observe the fruit tree images captured by multiple first image acquisition modules (such as stereo vision cameras). By controlling the positions of multiple robotic arms, adjust the observation positions of multiple first image acquisition modules on the fruit trees, select one or several groups of observation positions that can cover a larger field of view, ensure that all fruits in the working space are within the field of view, and avoid the appearance of visual blind spots; secondly, install a second image acquisition module D1 at a fixed position on the robot base, as shown in Figure 2, and manually measure the pose matrix of the second image acquisition module installed at the fixed position and the robot base coordinate system.

[0080] Then, the calibration plate is placed within the field of view of all first image acquisition modules, and the RGB color images of the calibration plate captured by all first image acquisition modules are obtained. The camera external attitude parameter calibration algorithm is run to obtain the relative posture relationship between each first image acquisition module and the calibration plate. And further obtain the posture relationship between each first image acquisition module and the second image acquisition module

[0081] Finally, combined with the pose relationship between the second image acquisition module and the calibration plate The pose transformation matrix between each first image acquisition module A1~An and the base coordinate system can be obtained by the following formula: Right now:

[0082] Where An represents the coordinate system of the nth first image acquisition module; D represents the coordinate system of the second image acquisition module; B represents the base coordinate system; and board represents the calibration board coordinate system.

[0083] Furthermore, in an embodiment of the present invention, the processor can acquire the fruit tree images and base coordinate system of the sub-area corresponding to the viewing angle in the working area based on the acquired first image acquisition modules, and use the depth information of each fruit in the color image and the depth image to perform point cloud projection. Through the above-mentioned calibration relationship, the visual information of all picking targets in the working area in the base coordinate system can be obtained, and the global fruit positioning distribution information of the working area can be determined by calculating the positioning center point of each picking target, so that the local fruit positioning distribution information of each robotic arm in the corresponding working sub-area can be determined according to the global fruit positioning distribution information, and each robotic arm can be controlled to perform collaborative operations to pick the fruits in the working area.

[0084] In an embodiment of the present invention, by using multiple cameras to collect images at set positions, visual information of all picking targets in the working area can be obtained synchronously. After target positioning and coordinate transformation, the coordinates of all fruits are obtained, thereby improving the performance of work planning. Compared with long-distance observation by a single camera at a long distance, it is easier to implement in installation, and close observation helps to improve measurement accuracy. At the same time, compared with the solution of continuously moving a single camera to obtain global information of the fruit, the solution of the embodiment of the present invention does not require repeated movement of the camera, and can synchronously obtain global distribution information of all fruits in the working area, which is more timely. In addition, the detection results of multiple visual units in the embodiment of the present invention can play a role of mutual verification, thereby effectively reducing the occurrence of false detections.

[0085] In some embodiments, after each first image acquisition module completes image acquisition, it can send the captured fruit tree images to an image processing module within a built-in processor, such as a graphics edge computing chip, graphics workstation, or graphics server, via a wired connection, Wi-Fi, or 4G / 5G communication method. This module then uses a series of machine learning-based algorithms, including image recognition, detection, and segmentation, to identify and locate the fruit targets. Because multiple image acquisition modules are used, a single computer cannot simultaneously complete control planning and graphics reasoning tasks. Therefore, multiple graphics reasoning devices need to be configured to form a distributed solution.

[0086] Optionally, in this embodiment, if four stereo vision cameras are used for image acquisition, one to four independent graphics processing computers can be configured to share the graphics computing pressure, depending on the system's real-time requirements. Regarding the allocation of graphics computing resources, the balance between the graphics processing unit (GPU) and the central processing unit (CPU) can be considered to improve the efficiency of graphics computing from multiple sources. The following solutions can be adopted:

[0087] (1) An independent graphics computing unit processes the image information collected by two stereo vision cameras, and a main control computer processes the image information collected by the other two stereo vision cameras.

[0088] (2) Two independent graphics computing units process the image information collected by one stereo vision camera respectively, and one main control computer processes the image information collected by two stereo vision cameras.

[0089] (3) A main control computer processes the image information collected by four stereo vision cameras.

[0090] The above centralized solutions can achieve different application effects, and performance and cost are comprehensively considered in specific applications.

[0091] In this embodiment, by flexibly configuring graphics computing resources and not relying on a single computer for graphics processing, tasks are distributed to multiple edge computing platforms, reducing the pressure on the main control system and helping to save hardware costs.

[0092] The picking robot of the embodiment of the present invention performs multi-perspective image acquisition by setting multiple image acquisition modules on the multi-arm picking robot body, and uses a processor to uniformly convert the images acquired from various perspectives into the robot's base coordinate system, thereby synchronously obtaining visual information of all picking targets in the working area, and generating global fruit positioning distribution information that matches the range size of the robot's working space. This is conducive to the efficient collaborative operation of various robotic arms, and can not only realize accurate detection of fruits at a relatively close distance in the working area and obtain fruit information within a larger range, but also improve the accuracy and range of fruit positioning, which is conducive to greatly improving the robot's fruit picking efficiency.

[0093] Figure 4 is a flow chart of the fruit positioning method provided by the present invention. As shown in Figure 4, it can be understood that the method can be applied to any of the above-mentioned picking robots, and its execution body is the processor in the picking robot. The method includes: step 410, step 420 and step 430.

[0094] Step 410: Obtain fruit tree images of corresponding sub-areas in the operation area captured by each first image acquisition module, and input each fruit tree image into a preset target detection model to obtain two-dimensional bounding box information and mask areas of each fruit in each fruit tree image output by the preset target detection model;

[0095] Step 420, using the mask area and corresponding image depth information of each fruit in each of the fruit tree images, generates a three-dimensional point cloud of each fruit in each fruit tree image, and based on the three-dimensional point cloud and two-dimensional bounding box information of each fruit in each fruit tree image, determines the positioning coordinate points of each fruit in each fruit tree image.

[0096] Step 430 : Based on the result of converting the positioning coordinate points of each fruit in each fruit tree image into the base coordinate system, the global fruit positioning distribution information of the operation area is determined.

[0097] Specifically, the two-dimensional bounding box information described in the embodiment of the present invention refers to two-dimensional frame information that completely surrounds the fruit target, which is output by performing target detection on each fruit in each fruit tree image using a target detection algorithm.

[0098] It should be noted that the preset object detection model can be built based on a deep convolutional neural network model, such as the YOLOv4 network model, which includes a shared encoding network and two decoding networks with different tasks. Among them, the encoding network consists of a backbone network (Backbone) responsible for feature extraction and a neck network (Neck) that collects feature maps at different stages; one decoding network consists of a target detection head network (Detect Head) for predicting the fruit occlusion type and the complete two-dimensional bounding box, and the other decoding network consists of an instance segmentation head network (Segment Head) for segmenting the pixel mask of the visible part of the fruit.

[0099] In an embodiment of the present invention, after the multi-arm picking robot moves to the working site, the robot main control system sends a start command to control the multi-arm mechanism to reach the set observation position. After arriving at the observation position, in step 410, all the first image acquisition modules simultaneously acquire images of fruit trees in the corresponding sub-areas in the working area, including color images and depth images of the fruit tree working surface, and send each fruit tree image to the processor for image processing. After obtaining the fruit tree images acquired by each first image acquisition module, the processor first inputs each acquired fruit tree image into a preset target detection model. The preset target detection model will perform image segmentation on the fruit target on the color image in the fruit tree image to obtain the mask area and two-dimensional bounding box information of each fruit.

[0100] Furthermore, in an embodiment of the present invention, in step 420, the image depth information of each fruit in the depth image of each fruit tree image is used to determine the depth value of the mask pixel area of ​​each fruit. In combination with the known camera internal imaging model parameters, a three-dimensional point cloud of the mask area is calculated and generated to obtain a three-dimensional point cloud of each fruit in each fruit tree image. At the same time, in combination with the two-dimensional bounding box information of each fruit output by the aforementioned preset target detection model, the spatial geometric position relationship between the three-dimensional point cloud and the two-dimensional bounding box information of each fruit in each fruit tree image is calculated to estimate the center of mass position of each fruit in each fruit tree image, thereby obtaining the positioning coordinate point of each fruit in each fruit tree image.

[0101] Furthermore, in an embodiment of the present invention, in step 430, the positioning coordinate points of each fruit in each fruit tree image are converted to the base coordinate system to obtain the positioning coordinate points of all fruits in the working area in the base coordinate system. Based on this conversion result, the fruit positioning results in the overlapping and interlaced fields of view are processed in the base coordinate system, and the same fruit targets in multiple visual acquisition fields of view are eliminated to avoid repetition of the planning process, and finally generate global fruit positioning distribution information of the working area.

[0102] The fruit positioning method of the picking robot in the embodiment of the present invention performs multi-perspective image acquisition by setting multiple image acquisition modules on the multi-arm picking robot body, and uniformly converts the acquired images from each perspective into the robot's base coordinate system, thereby synchronously obtaining visual information of all picking targets in the working area, and generating global fruit positioning distribution information that matches the range size of the robot's working space, ensuring that each robotic arm can work together efficiently, and can achieve accurate detection of fruits at a relatively close distance in the working area and obtain fruit information within a larger range, and can also improve the accuracy and range of fruit positioning, thereby greatly improving the robot's fruit picking efficiency.

[0103] Based on the content of the above embodiment, as an optional embodiment, determining the positioning coordinate points of each fruit in each fruit tree image based on the three-dimensional point cloud and two-dimensional bounding box information of each fruit in each fruit tree image includes:

[0104] The point cloud clustering algorithm is used to perform clustering calculation on the three-dimensional point cloud of each fruit in each fruit tree image, and the surface feature points of each fruit in each fruit tree image are determined.

[0105] According to the two-dimensional bounding box information of each fruit in each fruit tree image, a three-dimensional viewing cone and a center line of the viewing cone corresponding to each fruit in each fruit tree image are generated.

[0106] Based on the center line of the viewing cone and the surface feature points corresponding to each fruit in each fruit tree image, the positioning coordinate point of each fruit in each fruit tree image is determined.

[0107] Specifically, the surface feature points described in the embodiment of the present invention refer to points used to describe the surface features of a fruit.

[0108] In an embodiment of the present invention, after obtaining the three-dimensional point cloud information of each fruit in each fruit tree image, the three-dimensional point cloud of each fruit is clustered and calculated using point cloud clustering and filtering methods to obtain the point cloud centroid, thereby obtaining the surface feature points of each fruit in each fruit tree image.

[0109] Furthermore, in an embodiment of the present invention, based on the two-dimensional bounding box information of each fruit in each fruit tree image, the principles of geometric optics are utilized and combined with the shooting focus of the image to further generate a three-dimensional visual cone on the optical path from the shooting focus to the two-dimensional bounding box of the fruit, as well as a center line of the visual cone passing through the two-dimensional bounding box.

[0110] Furthermore, in an embodiment of the present invention, the positioning coordinate points of each fruit in each fruit tree image are calculated based on the center line of the viewing cone corresponding to each fruit in each fruit tree image and the spatial geometric relationship between the surface feature points.

[0111] The method of the embodiment of the present invention estimates the position of feature points on the surface of the fruit by utilizing image point cloud clustering calculation, and combines the optical cone method to calculate and infer the center of mass of each fruit from the level of geometric imaging principles, thereby realizing the identification and positioning of each fruit. This has good performance for the common scene of fruit being obscured in orchards, can greatly reduce the impact of foreign object occlusion on fruit positioning, and improves the accuracy of the algorithm for estimating the center of mass position of the fruit.

[0112] Based on the content of the above embodiment, as an optional embodiment, determining the positioning coordinate point of each fruit in each fruit tree image based on the center line of the viewing cone and the surface feature points corresponding to each fruit in each fruit tree image includes:

[0113] For each fruit in each fruit tree image, a sphere is constructed with the surface feature point as the center and the target length as the radius; the target length is determined based on the depth value corresponding to the surface feature point.

[0114] Determine the two intersection points where the center line of the viewing cone corresponding to each fruit in each fruit tree image passes through its corresponding sphere.

[0115] From the two intersection points corresponding to the fruits in each fruit tree image, the intersection point with the largest distance from the shooting focus is determined as the positioning coordinate point of each fruit in each fruit tree image.

[0116] Specifically, in the embodiment of the present invention, for each fruit in each fruit tree image, a surface feature point P is constructed. s The target length is a sphere with a center of φ and a radius of φ. The target length can be calculated using the following formula:

[0117] Where Δu represents the side length of the complete fruit two-dimensional bounding box on the U axis of the image plane; z' represents the depth value corresponding to the surface feature point; K u Represents the scale factor of the camera on the U axis; r is the target length, that is, the radius of the fruit.

[0118] Then, after determining r, we can construct a surface feature point Ps A sphere with centered at r and radius r is calculated. The two intersection points of the center line of the viewing cone corresponding to each fruit in each fruit tree image passing through its corresponding sphere are obtained.

[0119] Finally, from the two intersection points corresponding to each fruit in each fruit tree image, the intersection point with the largest distance from the shooting focus is determined as the centroid P of the fruit sphere. o , that is, obtaining the positioning coordinate points of each fruit in each fruit tree image.

[0120] The method of the embodiment of the present invention, by considering the relationship between the surface feature points of the fruit and the center of mass of the fruit sphere in geometric space, solves the center of mass of the fruit sphere according to the position of the surface feature points of the fruit, thereby improving the accuracy of the estimation of the center of mass position of the fruit and improving the accuracy of fruit positioning and identification.

[0121] Based on the content of the above embodiment, as an optional embodiment, based on the result of converting the positioning coordinate points of each fruit in each fruit tree image into the base coordinate system, the global fruit positioning distribution information of the operation area is determined, including:

[0122] The positioning coordinate points of each fruit in each fruit tree image are converted to the base coordinate system to obtain the positioning coordinate points of each fruit in each fruit tree image in the base coordinate system.

[0123] According to the positioning coordinate points of each fruit in the base coordinate system, the positioning coordinate point pairs between adjacent fruits are determined, and the distance between each positioning coordinate point pair in the base coordinate system is determined.

[0124] Determine the target positioning coordinate point pairs whose distance is less than the target threshold, and remove one positioning coordinate point from each target positioning coordinate point pair.

[0125] According to the various positioning coordinate points retained in the base coordinate system, the global fruit positioning distribution information of the operation area is generated.

[0126] Specifically, the positioning coordinate point pair described in the embodiment of the present invention refers to a point pair formed by two adjacent positioning coordinate points among all the fruit positioning coordinate points in the same base coordinate system.

[0127] The target threshold described in the embodiment of the present invention refers to a pre-set distance threshold, which can be used to determine whether two adjacent positioning coordinate points are the same repeated positioning coordinate points. The selection of this threshold can be flexibly adjusted according to actual conditions.

[0128] Furthermore, in the embodiment of the present invention, the three-dimensional positioning coordinate points P of each fruit in the working area under different camera fields of view are obtained. o Then, the camera external pose parameter information determined above is used The positioning coordinate points of each fruit in each fruit tree image are converted to a unified base coordinate system, thereby obtaining the positioning coordinate points P′ of each fruit in the operating area under the base coordinate system. o .

[0129] In order to eliminate the repeated positioning information of the same fruit target under different image acquisition modules, the positioning coordinates of all fruits are screened after transformation.

[0130] Specifically, according to the positioning coordinate points of each fruit in the base coordinate system, the positioning coordinate point pairs between all adjacent fruits are obtained, and the distance between each positioning coordinate point pair in the base coordinate system is calculated. The positioning coordinate points of fruits that are too close to each other are judged and the target threshold D is set. th , determine that the distance is less than the target threshold D th The target positioning coordinate point pairs are obtained, and duplicate targets are eliminated from the target positioning coordinate point pairs, that is, one positioning coordinate point in the target positioning coordinate point pairs is eliminated, so that the global fruit positioning distribution information of the operation area can be generated according to the various positioning coordinate points retained in the base coordinate system, the complete fruit coordinate distribution in the operation area can be obtained, and the precise positioning of all fruits in the large-scale picking operation area can be completed.

[0131] The method of the embodiment of the present invention converts the positioning coordinate points of each fruit in each fruit tree image into the base coordinate system, processes repeated targets in overlapping and interlaced fields of view, and eliminates the same fruit targets in multiple different acquisition fields of view, thereby avoiding repetition of subsequent operation planning processes, improving the accuracy of the robot's global positioning results for all fruits in the picking operation area, and is conducive to improving the picking efficiency of the multi-arm robot.

[0132] The fruit positioning device provided by the present invention is described below. The fruit positioning device described below and the fruit positioning method described above can be referenced to each other.

[0133] FIG5 is a schematic diagram of the structure of the fruit positioning device provided by the present invention. As shown in FIG5 , the device can be applied to any of the above-mentioned picking robots, and the device includes:

[0134] The output module 510 is used to obtain the fruit tree images of the corresponding sub-areas in the operation area captured by each first image acquisition module, and input each of the fruit tree images into a preset target detection model to obtain the two-dimensional bounding box information and mask area of ​​each fruit in each of the fruit tree images output by the preset target detection model.

[0135] The positioning module 520 is used to generate a three-dimensional point cloud of each fruit in each fruit tree image by using the mask area and corresponding image depth information of each fruit in each fruit tree image, and determine the positioning coordinate point of each fruit in each fruit tree image based on the three-dimensional point cloud and two-dimensional bounding box information of each fruit in each fruit tree image.

[0136] The processing module 530 is used to determine the global fruit positioning distribution information of the operation area based on the result of converting the positioning coordinate points of each fruit in each of the fruit tree images into the base coordinate system.

[0137] The fruit positioning device described in this embodiment can be used to execute the above-mentioned fruit positioning method embodiment. Its principles and technical effects are similar and will not be repeated here.

[0138] The fruit positioning device of the picking robot in the embodiment of the present invention performs multi-perspective image acquisition by setting multiple image acquisition modules on the multi-arm picking robot body, and uniformly converts the acquired images from each perspective into the robot's base coordinate system, thereby synchronously obtaining visual information of all picking targets in the working area, and generating global fruit positioning distribution information that matches the range size of the robot's working space, ensuring that each robotic arm can work together efficiently, and can achieve accurate detection of fruits at a relatively close distance in the working area and obtain fruit information within a larger range, and can also improve the accuracy and range of fruit positioning, thereby greatly improving the robot's fruit picking efficiency.

[0139] Figure 6 is a schematic diagram of the physical structure of the electronic device provided by the present invention. As shown in Figure 6, the electronic device may include: a processor (processor) 610, a communication interface (Communications Interface) 620, a memory (memory) 630 and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call the logic instructions in the memory 630 to execute the fruit positioning method provided by the above-mentioned methods, which includes: obtaining the fruit tree images of the corresponding sub-areas in the working area captured by each first image acquisition module, and inputting each of the fruit tree images into a preset target detection model to obtain the two-dimensional bounding box information and mask area of ​​each fruit in each of the fruit tree images output by the preset target detection model; using the mask area and corresponding image depth information of each fruit in each of the fruit tree images to generate a three-dimensional point cloud of each fruit in each of the fruit tree images, and based on the three-dimensional point cloud and two-dimensional bounding box information of each fruit in each of the fruit tree images, determining the positioning coordinate points of each fruit in each of the fruit tree images; based on the result of converting the positioning coordinate points of each fruit in each of the fruit tree images to the base coordinate system, determining the global fruit positioning distribution information of the working area.

[0140] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0141] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the fruit positioning method provided by the above-mentioned methods, which includes: obtaining fruit tree images of corresponding sub-areas in the working area captured by each first image acquisition module, and inputting each of the fruit tree images into a preset target detection model to obtain two-dimensional bounding box information and mask area of ​​each fruit in each of the fruit tree images output by the preset target detection model; using the mask area and corresponding image depth information of each fruit in each of the fruit tree images to generate a three-dimensional point cloud of each fruit in each of the fruit tree images, and based on the three-dimensional point cloud and two-dimensional bounding box information of each fruit in each of the fruit tree images, determining the positioning coordinate points of each fruit in each of the fruit tree images; based on the result of converting the positioning coordinate points of each fruit in each of the fruit tree images to the base coordinate system, determining the global fruit positioning distribution information of the working area.

[0142] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the fruit positioning method provided by the above-mentioned methods, the method comprising: obtaining fruit tree images of corresponding sub-areas in the operating area captured by each first image acquisition module, and inputting each of the fruit tree images into a preset target detection model to obtain two-dimensional bounding box information and mask areas of each fruit in each of the fruit tree images output by the preset target detection model; using the mask areas and corresponding image depth information of each fruit in each of the fruit tree images to generate a three-dimensional point cloud of each fruit in each of the fruit tree images, and based on the three-dimensional point cloud and two-dimensional bounding box information of each fruit in each of the fruit tree images, determining the positioning coordinate points of each fruit in each of the fruit tree images; and determining the global fruit positioning distribution information of the operating area based on the result of converting the positioning coordinate points of each fruit in each of the fruit tree images to the base coordinate system.

[0143] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0144] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0145] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A picking robot, characterized in that: include: A robot body, the robot body comprising a processor; a plurality of first image acquisition modules and a second image acquisition module are mounted on the robot body; the robot body comprises a plurality of mechanical arms; A first image acquisition module is correspondingly installed around each mechanical arm, and each first image acquisition module does not interfere with the corresponding mechanical arm; the second image acquisition module is installed at the base position of the robot body to determine the base coordinate system; The processor is used to determine the global fruit positioning distribution information of the working area based on obtaining the fruit tree images of the corresponding sub-areas in the working area captured by each of the first image acquisition modules and the base coordinate system, and determine the local fruit positioning distribution information corresponding to each robotic arm according to the global fruit positioning distribution information, so as to control each robotic arm to perform collaborative work.

2. The harvesting robot according to claim 1, characterized in that: The robot body includes a main frame and a plurality of connecting rods mounted on the main frame; At least two mechanical arms are installed on each of the connecting rods, and a first image acquisition module corresponding to each mechanical arm is installed on each of the connecting rods. Each of the first image acquisition modules is located near the end joint of the corresponding mechanical arm.

3. The harvesting robot according to claim 1, characterized in that: The robotic arm is a telescopic robotic arm; each of the first image acquisition modules is installed on a side of the corresponding telescopic robotic arm close to the gripper, and the central axis of the shooting angle of each of the first image acquisition modules is consistent with the telescopic direction of the telescopic robotic arm.

4. A fruit positioning method applied to the picking robot according to any one of claims 1 to 3, characterized in that: include: Obtain fruit tree images of corresponding sub-areas in the operation area captured by each first image acquisition module, and input each of the fruit tree images into a preset target detection model to obtain two-dimensional bounding box information and mask areas of each fruit in each of the fruit tree images output by the preset target detection model; Using the mask area and the corresponding image depth information of each fruit in each of the fruit tree images, a three-dimensional point cloud of each fruit in each of the fruit tree images is generated, and based on the three-dimensional point cloud and two-dimensional bounding box information of each fruit in each of the fruit tree images, the three-dimensional point cloud of each fruit in each of the fruit tree images is determined. The location coordinates of each fruit; Based on the result of converting the positioning coordinate points of each fruit in each of the fruit tree images into the base coordinate system, the global fruit positioning distribution information of the operation area is determined.

5. The fruit positioning method according to claim 4, characterized in that: The determining of the positioning coordinate points of each fruit in each of the fruit tree images based on the three-dimensional point cloud and two-dimensional bounding box information of each fruit in each of the fruit tree images comprises: Using a point cloud clustering algorithm, clustering calculation is performed on the three-dimensional point cloud of each fruit in each of the fruit tree images to determine the surface feature points of each fruit in each of the fruit tree images; Generate a three-dimensional viewing cone and a center line of the viewing cone corresponding to each fruit in each fruit tree image according to the two-dimensional bounding box information of each fruit in each fruit tree image; Based on the center line of the viewing cone and the surface feature points corresponding to each fruit in each of the fruit tree images, the positioning coordinate points of each fruit in each of the fruit tree images are determined.

6. The fruit positioning method according to claim 5, characterized in that: The method of determining the positioning coordinate points of each fruit in each of the fruit tree images based on the center line of the viewing cone and the surface feature points corresponding to each fruit in each of the fruit tree images comprises: For each fruit in each of the fruit tree images, construct a sphere with the surface feature point as the sphere center and the target length as the radius; the target length is determined based on the depth value corresponding to the surface feature point; Determine two intersection points where the center line of the viewing cone corresponding to each fruit in each of the fruit tree images passes through the corresponding sphere; From the two intersection points corresponding to each fruit in each of the fruit tree images, determine the intersection point with the largest distance from the shooting focus as the positioning coordinate point of each fruit in each of the fruit tree images.

7. The fruit positioning method according to any one of claims 4 to 6, characterized in that: The method of determining the global fruit positioning distribution information of the operation area based on the result of converting the positioning coordinate points of each fruit in each of the fruit tree images into the base coordinate system includes: Converting the positioning coordinate points of each fruit in each of the fruit tree images to the base coordinate system to obtain the positioning coordinate points of each fruit in each of the fruit tree images in the base coordinate system; Determine the positioning coordinate point pairs between adjacent fruits according to the positioning coordinate points of each fruit in the base coordinate system, and determine the distance between each positioning coordinate point pair in the base coordinate system; Determine the target positioning coordinate point pairs whose distance is less than the target threshold, and remove one positioning coordinate point from each of the target positioning coordinate point pairs; The global fruit positioning distribution information of the operation area is generated according to each positioning coordinate point retained in the base coordinate system.

8. A fruit positioning device, characterized in that: include: An output module is used to obtain the fruit tree images of the corresponding sub-areas in the acquisition operation area of ​​each first image acquisition module, and input each of the fruit tree images into a preset target detection model to obtain the two-dimensional bounding box information and mask area of ​​each fruit in each of the fruit tree images output by the preset target detection model; A positioning module, used to generate a three-dimensional point cloud of each fruit in each of the fruit tree images by using the mask area and corresponding image depth information of each fruit in each of the fruit tree images, and determine the positioning coordinate points of each fruit in each of the fruit tree images based on the three-dimensional point cloud and two-dimensional bounding box information of each fruit in each of the fruit tree images; The processing module is used to determine the global fruit positioning distribution information of the operation area based on the result of converting the positioning coordinate points of each fruit in each of the fruit tree images into the base coordinate system.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, it implements the fruit positioning method as described in any one of claims 4 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the fruit positioning method as described in any one of claims 4 to 7 is implemented.

Images