A real-time fruit counting method based on visual SLAM and object detection

Abstract

The invention discloses a real-time fruit counting method based on visual SLAM and target detection, which uses a Realsense D435i depth camera combined with a SLAM algorithm to obtain the position information and image information of the camera in an orchard at a certain moment; the YOLOV5 target detection algorithm is used to detect the fruit in the image through the acquired color image and depth image, and the fruit pixels are mapped to a three-dimensional space, and the RANSAC algorithm is used to fit the center SC and radius r of each fruit sphere; the fruit sphere is converted to the orchard world coordinate system in combination with the position obtained by the SLAM algorithm, and the three-dimensional position information (x, y, z) and radius information r of the identified fruit center are recorded; and a fruit de-duplication detection algorithm is used to accurately count the number of fruits. The method of the invention has the advantages of low cost, strong robustness and high real-time performance, and can estimate the number of fruits in real time in an orchard environment.

Description

Technical Field

[0001] The invention relates to the technical field of robots, and in particular to a real-time fruit counting method based on visual slam and target detection. Background Art

[0002] Fruit yield estimation is one of the most common tasks in agriculture and one of the most demanding and challenging areas in orchard management. If the number and distribution of fruits in an orchard can be accurately calculated, subsequent tasks such as handling, warehousing and sales can be arranged more reasonably, which is crucial to improving the productivity of the orchard. Some scholars have proposed a method for intelligently detecting the number of fruits in an orchard. For example, patent CN112233121A discloses a fruit yield estimation method based on binocular spatial positioning and intelligent segmentation. It collects images, posture information and object posture information, performs binocular spatial positioning, constructs the coordinates of all objects in the entire orchard scene, segments the fruits in the image, records the position of the identified fruit pixels in the image projected to the three-dimensional scene, and counts the number of fruits. However, this method cannot estimate the yield in real time. The binocular matching algorithm it uses generates point clouds and the maskrcnn instance segmentation method obtains the mask of the fruit. Both algorithms cannot achieve real-time counting and need to collect data for processing, which is inconvenient to use. The method for obtaining the position and posture is RTK-GPS. RTK is affected by the occlusion of fruit trees in the orchard, and its accuracy will decrease, which will directly affect the subsequent fruit counting. Moreover, the equipment it uses is RTK-GPS, binocular cameras, etc., which are costly. Summary of the invention

[0003] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a real-time fruit counting method based on visual slam and target detection, which can estimate the yield in real time, is less affected by fruit tree occlusion, has low cost and is easy to use.

[0004] The purpose of the present invention is achieved through the following technical solutions:

[0005] A real-time fruit counting method based on visual SLAM and target detection comprises the following steps:

[0006] (1) Using the Realsense D435i depth camera combined with the SLAM algorithm to obtain the position and image information of the camera in the orchard at a certain moment;

[0007] (2) The YOLOV5 target detection algorithm is used to detect the fruits in the image through the acquired color image and depth image, the fruit pixels are mapped to three-dimensional space, and the RANSAC algorithm is used to fit the center SC and radius r of each fruit sphere;

[0008] (3) Combined with the pose obtained by the SLAM algorithm, the fruit sphere is transformed into the orchard world coordinate system, and the three-dimensional position information (x, y, z) and radius information r of the identified fruit sphere center are recorded;

[0009] (4) Use a fruit deduplication detection algorithm to accurately count the number of fruits.

[0010] In the step (1), the image information of the Realsense D435i depth camera is saved through the function of the robot operating system ROS, including the color image RGB and the depth image DEPTH; the posture information is synchronously saved through the corresponding inertial sensor data IMU combined with the VINS-RGBD algorithm, including the rotation matrix R and the translation vector t corresponding to each frame of the image.

[0011] The step (2) specifically includes the following steps:

[0012] (2-1) Align the depth image and color image;

[0013] (2-2) Based on the aligned depth image, the pixel of each fruit is mapped to the three-dimensional space in combination with the YOLOV5 target detection algorithm to obtain the three-dimensional point cloud of the fruit;

[0014] (2-3) The fruit three-dimensional point cloud is fitted with a sphere using the RANSAC algorithm to obtain the center SC and radius r of each fruit sphere.

[0015] In the step (2), the fruits in the color image are rapidly detected at the object level, and on this basis, the pixel set of each fruit in the color image is obtained. Because the detection frame of each object, i.e., the fruit, can be obtained by using the YOLOV5 target detection algorithm, compared with the pixel-level segmentation algorithm, the rapid detection at the object level adopted by the present invention can both meet the requirements and have the advantage of high speed.

[0016] The step (3) specifically includes the following steps:

[0017] (3-1) Starting from the initial position of the camera, the camera IMU module (Inertial Measurement Unit) outputs IMU information in real time, including x, y, z axes, angular velocity w, and linear acceleration information;

[0018] (3-2) Using the pre-calibrated external parameter matrix between the IMU and the camera, the VINS-RGBD algorithm is used to solve and save the camera pose of each frame through the received color image, depth image and IMU information;

[0019] (3-3) According to the camera position, the center of the fruit ball is converted from the current camera coordinate SC to the coordinate WC in the orchard world coordinate system.

[0020] In the step (4), a fruit de-duplication detection algorithm is used, and its principle is: because the frame rate of the Realsense D435i depth camera is 30 frames per second, the same fruit will be detected multiple times in different frames, and the background will be mistakenly detected as fruit. Often, it will only appear briefly in a certain frame. Based on this, we assume that the misdetected fruit will only appear once, and the correct fruit will be detected at least twice. Therefore, if a fruit has no overlap with other fruits, it is considered to be a false detection and removed; if a fruit overlaps with other fruits, it is considered to be a repeated detection of fruit, and a valid fruit sphere is retained, and the number of valid fruit spheres is calculated. After the fruit sphere is converted to the world coordinate system, it can be judged whether the fruit is detected repeatedly based on the fruit center and radius; if the distance between the centers of the two fruit spheres is less than the sum of the radii of the two fruits, it is considered to be a repeated fruit, so as to achieve the purpose of de-duplication.

[0021] The fruit deduplication detection algorithm comprises the following steps:

[0022] (4-1) Prioritize the retention of spheres with high confidence and inlier rate, where confidence is the detection confidence of YOLOV5, and inlier rate is calculated by RANSAC; let C = confidence + inlier rate, and sort all fruit spheres in descending order according to the C value;

[0023] (4-2) Select the fruit sphere 1 with the highest C and remove sphere 1 from the input list;

[0024] (4-3) Calculate the distance dist between the center of sphere 1 and the remaining spheres 2 one by one. If the distance dist is less than the sum of the radii of the two spheres, it is considered a duplicate fruit and sphere 2 is removed from the input list.

[0025] (4-4) Sphere 1 that has no repetition with other fruits is considered to be a false detection, and only sphere 1 with repetition is added to the output list;

[0026] (4-5) Repeat steps 2 to 4 until the input list is empty;

[0027] (4-6) After traversing all the fruits, duplicate fruits and misdetected background objects are filtered out, and finally the total number of fruits in the orchard is obtained.

[0028] Compared with the prior art, the present invention has the following advantages and effects:

[0029] (1) The present invention only uses one depth camera and the IMU built into the camera, uses fewer devices, and the equipment price is relatively low, which has the advantage of low cost.

[0030] (2) The present invention improves the running speed through three aspects: depth value acquisition method, deep learning method, and de-duplication algorithm, and has the advantage of real-time performance: a) Depth value acquisition method: Compared with the prior art using the binocular matching method, the present invention uses a depth camera to obtain depth values ​​in a shorter time; b) Deep learning method: The present invention uses the smallest model YOLOV5S of YOLOV5 and compresses it using tensorRT. Using a mobile 1660ti graphics card, it can reach 200 frames per second. The mask-RCNN used in the prior art can only reach 5 frames per second under the TianX graphics card; c) De-duplication algorithm: The present invention uses the fruit point cloud to fit a sphere, and uses the center and radius to deduplicate the fruit, which solves the problem of repeated detection of the same fruit in the orchard and improves the accuracy of fruit yield estimation. Compared with the prior art using all depth points for de-duplication, the amount of calculation is smaller.

[0031] (3) The method of the present invention has the advantages of low cost, strong robustness and high real-time performance, and can estimate the number of fruits in real time in an orchard environment. The method of the present invention can also be extended to applications such as counting other types of fruits, obtaining three-dimensional information, and distributing them in orchards. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 The present invention is a flow chart of real-time fruit counting.

[0033] Figure 2 This is the effect diagram of YOLOV5 detecting pitaya in the present invention.

[0034] Figure 3 This is an example diagram of RANSAC sphere fitting of the present invention.

[0035] Figure 4 This is the reconstruction effect diagram of VINS-RGBD of the present invention.

[0036] Figure 5 This is a sphere diagram of the fruit before de-weighting according to the present invention.

[0037] Figure 6 This is the effect diagram of the fruit after the fruit deduplication detection algorithm of the present invention has been removed DETAILED DESCRIPTION

[0038] In order to facilitate the understanding of the present invention, the present invention will be described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be pointed out that, for those skilled in the art, without departing from the inventive concept, the present invention can also make some variations and improvements, which all belong to the protection scope of the present invention.

[0039] Example 1

[0040] This embodiment takes the real-time counting of dragon fruit as an example. Figure 1 As shown, the real-time fruit counting method based on visual slam and target detection includes the following steps:

[0041] (1) A handheld Realsense D435i depth camera was slowly moved around a row of trees to record video. The Realsense D435i was connected to a computer to obtain image and IMU data. The pose information was synchronously saved through the corresponding inertial sensor data IMU combined with the VINS-RGBD algorithm, including the rotation matrix R and translation vector t corresponding to each frame of the image.

[0042] (2) Use the YOLOV5 object detection algorithm to detect the fruit in the image and map the fruit pixels to three-dimensional space. The specific steps are as follows:

[0043] (2-1) Use YOLOV5S, the smallest model of YOLOV5, to detect dragon fruit and compress it using tensorRT. Using a mobile 1660ti graphics card, it can reach 200 frames per second. Perform object-level rapid detection of fruits in color images and obtain the detection box bbox of each fruit. On this basis, obtain the pixel set of each fruit in the image, such as Figure 2 As shown;

[0044] (2-2) Since the resolution and pose of the depth image and RGB image are different, they need to be aligned first so that the pixels on the depth image and RGB image can maintain a one-to-one correspondence;

[0045] (2-3) According to the aligned depth image, combined with the pitaya bbox detected by the YOLOV5 model, the pixel (u, v) of each fruit is mapped to the three-dimensional space (x, y, z) to obtain the three-dimensional point cloud of the fruit; the coordinates (x, y, z) of the pixel (u, v) of each fruit mapped to the three-dimensional space can be calculated according to formula (1), where (u, v) is the image coordinate of the pixel, I depth is the depth map, c x , c y , f x and f y is the camera internal parameter, z is the depth:

[0046]

[0047] (3) Under the premise of ensuring the accuracy of the algorithm, in order to reduce the amount of calculation, the RANSAC algorithm is used to fit the sphere only for the three-dimensional point cloud of the fruit with a depth less than 0.9 meters to obtain the sphere center S C and radius r, the specific steps are as follows:

[0048] (3-1) Randomly select 4 different points and use the least squares method to calculate the parameters of the sphere;

[0049] (3-2) Calculate the distance between all points of the fruit and the spherical surface. If the distance is less than 10 mm, the point is considered to be an interior point, and the interior point ratio is calculated.

[0050] (3-3) When the steps (3-1) and (3-2) are executed more than 500 times or the inlier rate is greater than 80%, stop and take the ball parameter with the highest inlier rate as the final solution;

[0051] (3-4) Since the radius of pitaya is generally between 35 and 75 mm, if the radius of the sphere obtained in step (3-1) is not within this range, skip and restart step (3-1); for each predicted pitaya, the RANSAC algorithm estimates the corresponding parameters and fits a sphere, such as Figure 3 As shown; using a sphere center S C and radius r represents a pitaya fruit, which makes the parameters concise and helps the step of removing duplicate fruits in the subsequent fruit deduplication detection algorithm;

[0052] (4) Combined with the pose obtained by the SLAM algorithm, the fruit sphere is transferred to the world coordinate system, and the three-dimensional position information (x, y, z) and radius information r of the identified fruit sphere center are recorded. The specific steps are as follows:

[0053] (4-1) Starting from the initial position of the camera, the camera IMU module outputs x, y, z three-axis, angular velocity w and linear acceleration information (i.e., IMU information) in real time;

[0054] (4-2) Using the pre-calibrated external parameter matrix between the IMU and the camera, the VINS-RGBD algorithm can solve and save the camera pose of each frame by receiving color images, depth images, and IMU information, including the rotation matrix R of the world coordinate system and the translation vector t, as follows Figure 4 As shown;

[0055] (4-3) According to the camera pose and formula (2), where s c is the coordinate of the sphere center in the camera coordinate system, Sw is the coordinate of the sphere center in the world coordinate system, R is the rotation matrix, t is the translation vector, and the fruit sphere center SC is transformed from the current relative camera coordinate to the orchard world coordinate system Sw.

[0056] s w =Rs c +t (2)

[0057] (5) The same fruit may be detected multiple times in different frames. After the fruit sphere is converted to the world coordinate system, it can be determined whether the fruit is detected repeatedly based on the fruit center and radius. C , filter out the positions that are identical or within the allowable error range, retain a valid fruit position, and screen the number of all fruit positions, such as Figure 5 and Figure 6 As shown in the figure, compared with the fruit being correctly detected multiple times frame by frame by the YOLOV5 algorithm, the background being misdetected as fruit often only appears briefly in a certain frame. Therefore, based on this phenomenon, we proposed a fruit de-duplication detection algorithm, which only retains fruit spheres that have been detected twice or more. The specific steps are as follows:

[0058] (5-1) Prioritize the spheres with high confidence and inlier rate. The confidence is the detection confidence of YOLOV5, and the inlier rate is calculated by the RANSAC algorithm; let C = confidence + inlier rate, and sort all fruit spheres in descending order according to the C value;

[0059] (5-2) Select sphere 1 with the highest C and remove it from the input list;

[0060] (5-3) Calculate the distance dist between the center of sphere 1 and the remaining spheres 2 one by one. If the distance dist is less than the sum of the radii of the two spheres, it is considered a duplicate fruit and sphere 2 is removed from the input list;

[0061] (5-4) Sphere 1 that has no repetition with other fruits is considered to be a false detection, and only sphere 1 with repetition is added to the output list;

[0062] (5-5) Repeat steps 2 to 4 until the input list is empty;

[0063] (5-6) After traversing all the fruits, duplicate fruits and misdetected background objects are filtered out, and finally the total number of fruits in the orchard is obtained.

[0064] This embodiment uses dragon fruit as the object, with a detection range of 25 meters * 6 meters, and contains two data sets, one for two rows of fruit and the other for three rows of fruit. The whole process calculation speed of this embodiment is 13.06 frames per second, and the calculated number of fruits can be seen in real time, with an accuracy of 93.94%. The accuracy of the real-time fruit counting of the present invention is similar to that of the non-real-time algorithm, and is much higher than the real-time tracking algorithm of the prior art.

[0065] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. All equivalent structural changes made by using the contents of the present invention specification and drawings under the concept of the present invention, or directly / indirectly applied in other related technical fields are included in the patent protection scope of the present invention.

Claims

1. A real-time fruit counting method based on visual slam and target detection, characterized in that The steps include: (1) Use the Realsense D435i depth camera combined with the SLAM algorithm to obtain the position and image information of the camera in the orchard at a certain moment; (2) Using the acquired color image and depth image, the YOLOV5 target detection algorithm is used to detect the fruit in the image, the fruit pixels are mapped to three-dimensional space, and the RANSAC algorithm is used to fit the center SC and radius r of each fruit sphere; (3) Combine the pose obtained by the SLAM algorithm to transform the fruit sphere into the orchard world coordinate system, and record the three-dimensional position information (x, y, z) and radius information r of the identified fruit sphere center; (4) Use a fruit deduplication detection algorithm to accurately count the number of fruits; The fruit deduplication detection algorithm comprises the following steps: (4-1) Prioritize the retention of spheres with high confidence and inlier rate, where confidence is the detection confidence of YOLOV5 and inlier rate is calculated by RANSAC; let C = confidence + inlier rate, and sort all fruit spheres in descending order according to the C value; (4-2) Select the fruit sphere 1 with the highest C and remove sphere 1 from the input list; (4-3) Calculate the distance dist between the center of sphere 1 and the remaining spheres 2 one by one. If the distance dist is less than the sum of the radii of the two spheres, it is considered a duplicate fruit and sphere 2 is removed from the input list; (4-4) Sphere 1 that has no repetition with other fruits is considered to be a false detection, and only sphere 1 with repetition is added to the output list; (4-5) Repeat steps 2 to 4 until the input list is empty; (4-6) After traversing all the fruits, duplicate fruits and misdetected background objects are filtered out, and finally the total number of fruits in the orchard is obtained.

2. The fruit real-time counting method based on visual slam and target detection according to claim 1, characterized in that: In the step (1), the image information of the RealsenseD435i depth camera is saved through the function of the robot operating system ROS, including the color image RGB and the depth image DEPTH; the posture information is synchronously saved through the corresponding inertial sensor data IMU combined with the VINS-RGBD algorithm, including the rotation matrix R and the translation vector t corresponding to each frame of the image.

3. The fruit real-time counting method based on visual SLAM and target detection according to claim 1, characterized in that: The step (2) specifically includes the following steps: (2-1) Align depth image and color image; (2-2) Based on the aligned depth image, the YOLOV5 target detection algorithm is used to map the pixels of each fruit into a three-dimensional space to obtain a three-dimensional point cloud of the fruit; (2-3) The RANSAC algorithm is used to fit the fruit three-dimensional point cloud into a sphere to obtain the center SC and radius r of each fruit sphere.

4. The fruit real-time counting method based on visual slam and target detection according to claim 1, characterized in that: In the step (2), the fruits in the color image are quickly detected at the object level to obtain a pixel set of each fruit in the color image.

5. The real-time fruit counting method based on visual slam and target detection according to claim 1, characterized in that: The step (3) specifically includes the following steps: (3-1) Starting from the initial position of the camera, the camera IMU module outputs IMU information in real time, including x, y, z axes, angular velocity w and linear acceleration information; (3-2) Using the pre-calibrated external parameter matrix between the IMU and the camera, the VINS-RGBD algorithm is used to solve and save the camera pose of each frame through the received color image, depth image and IMU information; (3-3) According to the camera position, the center of the fruit ball is converted from the current camera coordinate SC to the coordinate WC in the orchard world coordinate system.

Images