Visual-based fruit basket tracking method and device for orchard transport vehicle

Abstract

A visual-based fruit basket tracking method and device for orchard transport vehicles, the method comprising the following steps: real-time estimation of the global position and posture of the orchard transport vehicle; during the driving of the orchard transport vehicle, real-time data collected by sensors are used to estimate the global position and posture of the orchard transport vehicle based on a visual-inertial odometry calculation method, so as to obtain the real-time relative position and relative posture of the orchard transport vehicle relative to the starting point; target detection and global position conversion of the fruit basket; the coordinates of the fruit basket recognition frame are obtained based on a target detection algorithm, and the position and posture are converted to obtain the relative position of the fruit basket in the global position relative to the starting point; and path planning for fruit basket tracking, in which the shortest driving distance is taken as the optimization target to plan the shortest path for the transport vehicle to traverse all the fruit baskets once and return to the initial starting point. The application also provides a visual-based fruit basket tracking device for orchard transport vehicles, which uses the method to plan the shortest path for the orchard transport vehicle according to the global position of the fruit basket.

Description

Technical Field

[0001] This invention relates to agricultural robot navigation technology, and in particular to a vision-based method and device for tracking fruit baskets in orchard transport vehicles. Background Technology

[0002] As the world's largest fruit producer, my country's fruit industry has become a vital source of national economic income and farmers' earnings. However, with my country's economic development and urbanization, problems such as labor shortages and rising production costs are becoming increasingly apparent. In the fruit production process, the transportation of fruit and agricultural inputs consumes a significant amount of human and material resources; therefore, the research and development and promotion of orchard transportation machinery are of great importance.

[0003] Currently, most orchard transport machinery in my country is manually operated and lacks autonomous operation capabilities, resulting in relatively low labor productivity. To further improve fruit transport efficiency and reduce manual labor costs, unmanned operation of orchard transport machinery is a current research hotspot, with autonomous driving being a key technology. Global satellite navigation technology is widely used in agricultural machinery autonomous driving and is relatively mature. However, in orchard environments, due to complex terrain and dense tree canopies, satellite navigation suffers from multipath effects and signal blockage, affecting the navigation accuracy of machinery operating in mountainous orchards. Visual navigation, with its advantages of large information acquisition capacity, relatively low cost, and no detection distance limitations, is one of the main research directions in orchard navigation. However, current research on orchard visual navigation is mostly based on orchard harvesting, plant protection, and inspection robots, with relatively few visual navigation technologies specifically designed for fruit transport tasks and orchard transport machinery. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a vision-based method and device for tracking fruit baskets in orchard transport vehicles, addressing the aforementioned problems existing in the prior art.

[0005] To achieve the above objectives, the present invention provides a vision-based method for tracking fruit baskets in orchard transport vehicles, comprising the following steps:

[0006] S100, Real-time global position and attitude estimation of the orchard transport vehicle: During the operation of the orchard transport vehicle, based on the data collected in real time by the sensors, the global position and attitude of the orchard transport vehicle are estimated using the visual inertial odometer method, so as to obtain the real-time relative position and attitude of the orchard transport vehicle relative to the starting point.

[0007] S200, Fruit basket target detection and global position transformation: Based on the target detection algorithm, the coordinate position of the fruit basket recognition bounding box is obtained, and pose transformation is performed to obtain the relative position of the fruit basket with respect to the starting point in the global context; and

[0008] S300, fruit basket tracking path planning, with the goal of minimizing the travel distance, plans the shortest path for the transport vehicle to traverse all fruit baskets in one go and return to the initial origin.

[0009] The above-mentioned vision-based orchard transport vehicle fruit basket tracking method, wherein the fruit basket tracking path planning in step S300 is a shortest path fruit basket traversal method based on modern optimization algorithms.

[0010] The above-mentioned vision-based orchard transport vehicle fruit basket tracking method involves the continuous detection and picking of new fruit baskets as the orchard transport vehicle moves. When the planned number of fruit baskets changes, a modern optimization algorithm is used to replan the orchard transport vehicle's driving path and obtain the shortest path for the current orchard transport vehicle to traverse all fruit baskets and return to the origin.

[0011] The aforementioned vision-based orchard transport vehicle fruit basket tracking method selects the path with the smallest cumulative turning angle if there are several shortest paths with similar distances.

[0012] In the aforementioned vision-based orchard transport vehicle fruit basket tracking method, in step S100, a depth camera is used to acquire in real time the RGB three-channel color image in front of the orchard transport vehicle and the depth information corresponding to each pixel; an inertial measurement unit is used to acquire in real time the acceleration and angular acceleration in three directions in the local coordinate system of the current depth camera.

[0013] In the aforementioned vision-based orchard transport vehicle fruit basket tracking method, the depth camera and the inertial measurement unit are respectively installed at the front of the orchard transport vehicle.

[0014] In the above-mentioned vision-based orchard transport vehicle fruit basket tracking method, in step S200, the fruit baskets within the field of view of the depth camera are detected based on a target detection algorithm to obtain the coordinate position of the fruit basket identification border in the RGB three-channel color image in front of the orchard transport vehicle.

[0015] The aforementioned vision-based orchard transport vehicle fruit basket tracking method involves obtaining the position of the fruit basket in the local coordinate system of the orchard transport vehicle based on the pixel-by-pixel depth information collected by the depth camera and the intrinsic parameters of the depth camera.

[0016] The aforementioned vision-based orchard transport vehicle fruit basket tracking method involves performing pose transformation based on the coordinate position of the fruit basket identification border in the RGB three-channel color image in front of the orchard transport vehicle and the position of the fruit basket in the local coordinate system of the orchard transport vehicle, and the real-time relative position and relative posture of the orchard transport vehicle relative to the starting point, to obtain the relative position of the fruit basket relative to the starting point in the global context.

[0017] To better achieve the above objectives, the present invention also provides a vision-based orchard transport vehicle fruit basket tracking device, wherein the vision-based orchard transport vehicle fruit basket tracking method described above is used to plan the shortest path of the orchard transport vehicle based on the global position of the fruit basket.

[0018] The technical advantages of this invention are as follows:

[0019] This invention can quickly plan the tracking path of fruit baskets for transport vehicles in orchard environments based on sensor data collected by depth cameras and inertial measurement units, using visual inertial odometry, target detection algorithms, and modern optimization algorithms. This further improves the efficiency of fruit transfer and enables unmanned transport operations and efficient material transportation in hilly orchards.

[0020] This invention, while applying a target detection algorithm to identify fruit baskets, combines visual inertial odometry to estimate the current global position and attitude of the orchard transport vehicle in real time. Based on camera intrinsic parameters, image depth, and local-to-global coordinate transformation, the global position of the fruit baskets is derived. During algorithm execution, the orchard transport vehicle can plan the shortest path based on the global position of the fruit baskets. Even if some fruit baskets leave the depth camera's field of view during the transport vehicle's tracking process, it will not affect the orchard transport vehicle's shortest path planning because the identified fruit baskets always remain in the global map. Using modern optimization algorithms for shortest path planning of the orchard transport vehicle is faster and more efficient than exhaustive search or depth-first search algorithms. Whenever the number of planned fruit baskets changes, the modern optimization algorithm will re-plan the orchard transport vehicle's path, dynamically planning the shortest traversal path for the fruit baskets in real time. The path planning effect is not affected by human factors.

[0021] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments, but this is not intended to limit the present invention. Attached Figure Description

[0022] Figure 1 This is a flowchart of a method according to an embodiment of the present invention;

[0023] Figure 2 This is a schematic diagram illustrating the working principle of an embodiment of the present invention.

[0024] Among them, the attached reference numerals

[0025] 1 Orchard transport vehicle

[0026] 2 fruit baskets Detailed Implementation

[0027] The structural and working principles of the present invention will be described in detail below with reference to the accompanying drawings:

[0028] See Figure 1 , Figure 1 This is a flowchart of a method according to an embodiment of the present invention. The present invention provides a vision-based method for tracking fruit baskets 2 in an orchard transport vehicle 1, including real-time global position and pose estimation of the orchard transport vehicle 1, target detection and global position transformation of the fruit baskets 2, and path planning for tracking the fruit baskets 2 based on the shortest path. Specifically, it includes the following steps:

[0029] Step S100: Real-time global position and attitude estimation of orchard transport vehicle 1. During the operation of orchard transport vehicle 1, based on the data collected by the sensors in real time, the global position and attitude of orchard transport vehicle 1 are estimated using the visual inertial odometer method to obtain the real-time relative position and attitude of orchard transport vehicle 1 relative to the starting point.

[0030] Step S200: Target detection and global position transformation of fruit basket 2. Based on the target detection algorithm, the coordinate position of the identified bounding box of fruit basket 2 is obtained, and pose transformation is performed to obtain the relative position of fruit basket 2 with respect to the starting point in the global coordinate system; and

[0031] Step S300: Tracking path planning for fruit basket 2. With the shortest travel distance as the optimization objective, plan the shortest path for the transport vehicle to traverse all fruit baskets 2 in one go and return to the initial origin.

[0032] In step S300, the preferred method for planning the tracking path of the fruit baskets 2 is the shortest path traversal method based on a modern optimization algorithm. As the orchard transport vehicle 1 moves, new fruit baskets 2 are continuously detected and picked up. When the planned number of fruit baskets 2 changes, the modern optimization algorithm is used to replan the orchard transport vehicle 1's driving path, obtaining the shortest path for the current orchard transport vehicle 1 to traverse all fruit baskets 2 and return to the origin. If several shortest paths with similar distances exist, the path with the smallest cumulative turning angle is selected.

[0033] In step S100 of this embodiment, a depth camera is used to acquire in real time an RGB three-channel color image of the front of the orchard transport vehicle 1 and the depth information corresponding to each pixel; an inertial measurement unit is used to acquire in real time the acceleration and angular acceleration in three directions in the local coordinate system of the current depth camera. The depth camera and the inertial measurement unit are preferably installed at the front of the orchard transport vehicle 1.

[0034] In step S200 of this embodiment, target detection is performed on the fruit basket 2 within the field of view of the depth camera based on the target detection algorithm, and the coordinate position of the identified border of the fruit basket 2 in the RGB three-channel color image in front of the orchard transport vehicle 1 is obtained. Based on the pixel-by-pixel depth information collected by the depth camera and the intrinsic parameters of the depth camera, the position of the fruit basket 2 in the local coordinate system of the orchard transport vehicle 1 is obtained. Based on the coordinate position of the identified border of the fruit basket 2 in the RGB three-channel color image in front of the orchard transport vehicle 1 and the position of the fruit basket 2 in the local coordinate system of the orchard transport vehicle 1, a pose transformation is performed with the real-time relative position and relative attitude of the orchard transport vehicle 1 relative to the starting point to obtain the global relative position of the fruit basket 2 relative to the starting point.

[0035] The present invention also provides a vision-based tracking device for fruit baskets 2 in an orchard transport vehicle 1, comprising a depth camera, an inertial measurement unit, and a tracking module. The depth camera and the inertial measurement unit are preferably installed at the front of the orchard transport vehicle 1, respectively. The tracking module can be installed on the vehicle body, in the driver's cab, or on an external server. It receives information collected by the depth camera and the inertial measurement unit via wireless communication and uses the aforementioned vision-based tracking method for fruit baskets 2 in an orchard transport vehicle 1 to plan the shortest path for the orchard transport vehicle 1 based on the global position of the fruit baskets 2.

[0036] See Figure 2 , Figure 2 This is a schematic diagram illustrating the working principle of an embodiment of the present invention. In this embodiment, a depth camera and an inertial measurement unit (IMU) are mounted on the front of the vehicle body. The depth camera acquires real-time RGB three-channel color images of the front of the orchard transport vehicle 1, as well as pixel-by-pixel depth information. The IMU acquires real-time acceleration and angular acceleration in three directions within the local coordinate system of the current depth camera. Before the orchard transport vehicle 1 starts moving, its initial position and attitude are recorded based on the real-time data acquired by the sensors and visual inertial odometry (VIO) methods (such as VINS-Fusion, ORB-SLAM3, etc.). During the movement of the orchard transport vehicle 1, its global position and attitude are estimated based on the real-time data acquired by the sensors and visual inertial odometry, yielding the relative position and attitude of the orchard transport vehicle 1 relative to the starting point.

[0037] While the visual-inertial odometry method estimates the global position and pose of the orchard transport vehicle 1 in real time, it uses a depth camera to capture real-time RGB three-channel color images of the area in front of the orchard transport vehicle 1. Based on object detection algorithms (such as SSD-MobileNet, Faster-RCNN, YOLOv5, etc.), it performs object detection on the fruit baskets 2 within the field of view of the depth camera, obtaining the coordinates of the identified bounding boxes of the fruit baskets 2 in the image. Since each pixel of the color image corresponds to the acquired depth information, the position of the fruit baskets 2 in the local coordinate system of the orchard transport vehicle 1 can be obtained based on the intrinsic parameters of the depth camera. Then, based on the global position and pose estimation of the orchard transport vehicle 1 obtained in the previous step, pose transformation is performed to obtain the relative position of the fruit baskets 2 with respect to the starting point in the global coordinate system. At this point, the global positions of both the fruit baskets 2 and the transport vehicle are known.

[0038] Through the calculations in the previous two steps, the global positions of both fruit basket 2 and orchard transport vehicle 1 are known. Therefore, the distances between fruit basket 2 and fruit basket 2 and between fruit basket 1 can be obtained. Based on the identified global positions of fruit basket 2 and orchard transport vehicle 1, the distances between orchard transport vehicle 1 and fruit basket 2, and between fruit basket 2 and fruit basket 1, are obtained. To obtain the travel path of orchard transport vehicle 1, based on modern optimization algorithms (such as genetic algorithms, ant colony algorithms, tabu search algorithms, etc.), with the shortest travel distance as the optimization objective, the shortest path for the transport vehicle to traverse all fruit baskets 2 and return to the initial origin in one go is planned. During the travel of orchard transport vehicle 1, due to the limitations of the depth camera's detection range and its own field of view, new fruit baskets 2 will be continuously detected. At the same time, when orchard transport vehicle 1 reaches the vicinity of fruit basket 2, the tracked fruit basket 2 will be picked up. Whenever the number of planned fruit baskets 2 changes, the modern optimization algorithm will re-plan the travel path, and finally obtain the shortest path for the current transport vehicle to traverse all fruit baskets 2 and return to the origin. If several shortest paths with similar distances exist, prioritize the path with the smallest cumulative turning angle. Planning the shortest tracking path for the fruit basket 2 can effectively improve the efficiency of fruit transportation.

[0039] This invention discloses a vision-based method and apparatus for tracking fruit baskets 2 in an orchard transport vehicle 1. Based on visual inertial odometry, it estimates the global position and attitude of the transport vehicle in real time. Then, using a target detection algorithm, it detects fruit baskets 2 in real time from a color image and acquires their corresponding depth information. Next, based on the current global position and attitude of the transport vehicle, the depth information of the fruit baskets 2, and the intrinsic parameters of the depth camera, it calculates the global position of the fruit baskets 2. Finally, based on a modern optimization algorithm, with the shortest distance as the optimization objective, it solves for the traversal strategy of the fruit baskets 2, determining the global tracking path of the fruit baskets 2 in the orchard transport vehicle 1. This method can automatically detect fruit baskets 2 within the field of view, quickly planning the shortest tracking path for the fruit baskets 2 in the orchard transport vehicle 1, providing a valuable research reference for the automatic transfer of fruit in the orchard transport vehicle 1.

[0040] This invention, while applying a target detection algorithm to identify fruit baskets, combines visual inertial odometry to estimate the current global position and attitude of the orchard transport vehicle in real time. Based on camera intrinsic parameters, image depth, and local-to-global coordinate transformation, the global position of the fruit baskets is derived. During algorithm execution, the orchard transport vehicle can plan the shortest path based on the global position of the fruit baskets. Even if some fruit baskets leave the depth camera's field of view during the transport vehicle's tracking process, it will not affect the orchard transport vehicle's shortest path planning because the identified fruit baskets always remain in the global map. Using modern optimization algorithms for shortest path planning of the orchard transport vehicle is faster and more efficient than exhaustive search or depth-first search algorithms. Whenever the number of planned fruit baskets changes, the modern optimization algorithm will re-plan the orchard transport vehicle's path, dynamically planning the shortest traversal path for the fruit baskets in real time. The path planning effect is not affected by human factors.

[0041] Of course, the present invention may have other various embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and modifications according to the present invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. A vision-based method for tracking fruit baskets in orchard transport vehicles, characterized in that, Includes the following steps: S100, Real-time global position and attitude estimation of the orchard transport vehicle: During the operation of the orchard transport vehicle, based on the data collected in real time by the sensors, the global position and attitude of the orchard transport vehicle are estimated using the visual inertial odometer method, so as to obtain the real-time relative position and attitude of the orchard transport vehicle relative to the starting point. S200, Fruit basket target detection and global position transformation: Based on the target detection algorithm, the coordinate position of the fruit basket recognition border is obtained, and the pose transformation is performed to obtain the relative position of the fruit basket with respect to the starting point in the global context. as well as S300, Fruit basket tracking path planning, with the shortest travel distance as the optimization objective, plans the shortest path for the transport vehicle to traverse all fruit baskets in one go and return to the initial origin. Fruit basket tracking path planning is a shortest path fruit basket traversal method based on modern optimization algorithms. In step S100, a depth camera is used to acquire in real time RGB three-channel color images of the front of the orchard transport vehicle and the depth information corresponding to each pixel; An inertial measurement unit is used to collect the acceleration and angular acceleration in three directions in the local coordinate system of the current depth camera in real time; In step S200, the target detection algorithm is used to detect the fruit baskets within the field of view of the depth camera to obtain the coordinate position of the fruit basket identification frame in the RGB three-channel color image in front of the orchard transport vehicle; based on the depth information corresponding to each pixel collected by the depth camera and the intrinsic parameters of the depth camera, the position of the fruit basket in the local coordinate system of the orchard transport vehicle is obtained. Based on the coordinates of the fruit basket's frame in the RGB three-channel color image in front of the orchard transport vehicle and the position of the fruit basket in the local coordinate system of the orchard transport vehicle, pose transformation is performed with the real-time relative position and attitude of the orchard transport vehicle relative to the starting point to obtain the relative position of the fruit basket relative to the starting point in the global map; the identified fruit basket always exists in the global map, and the orchard transport vehicle performs shortest path planning based on the global position of the fruit basket. Whenever the planned number of fruit baskets changes, the modern optimization algorithm will replan the orchard transport vehicle's route and dynamically plan the shortest route for traversing the fruit baskets in real time. As the orchard transport vehicle moves along, new fruit baskets are constantly detected and picked up. When the planned number of fruit baskets changes, a modern optimization algorithm is used to replan the orchard transport vehicle's route and obtain the shortest path for the current orchard transport vehicle to traverse all fruit baskets and return to the origin.

2. The vision-based orchard transport vehicle fruit basket tracking method as described in claim 1, characterized in that, If there are several shortest paths with similar distances, choose the path with the smallest cumulative turning angle.

3. The vision-based orchard transport vehicle fruit basket tracking method as described in claim 1, characterized in that, The depth camera and the inertial measurement unit are respectively installed at the front of the orchard transport vehicle.

4. A vision-based orchard transport vehicle fruit basket tracking device, characterized in that, The vision-based orchard transport vehicle fruit basket tracking method described in any one of claims 1-3 is used to plan the shortest path for the orchard transport vehicle based on the global position of the fruit basket.

Images