Automatic blueberry picking system and method based on visual identification and path planning

Abstract

The invention discloses an automatic blueberry picking system and method based on visual identification and path planning, the system comprises a moving assembly, a navigation assembly, a picking assembly and a control assembly, a first camera and a second camera of the navigation assembly are used for obtaining blueberry plant planting positions and park layout images in real time; the control assembly is used for receiving transmission data of the radar, the first camera and the second camera and carrying out picking path planning and charging path planning through spatial modeling and a path planning algorithm. And the third camera is also used for receiving transmission data of the third camera, identifying the color, shape, size and position of the blueberry fruit through a visual identification algorithm, and generating fruit maturity data and spatial displacement data so as to realize autonomous picking. Meanwhile, the picking system is high in automation degree, high in operation efficiency, good in environment adaptability, safe, sanitary and high in expansibility and is also suitable for other small berry crops.

Description

Technical Field

[0001] The present invention relates to the field of picking equipment, and in particular to a blueberry automatic picking system and method based on visual recognition and path planning. Background Art

[0002] Blueberry planting and harvesting are particularly popular in China, but traditional blueberry picking techniques rely on manual harvesting, and the harvesting process is cumbersome, time-consuming and labor-intensive.

[0003] For this reason, some blueberry picking equipment has appeared in the existing designs. However, after investigation, it was found that such equipment requires manual control and is slightly insufficient in terms of picking efficiency and automation. For example, the existing blueberry picking robots are at a semi-automatic level during the blueberry picking process, that is, they still require human intervention and guidance during the picking process and cannot achieve completely autonomous picking.

[0004] Therefore, existing blueberry picking equipment has problems such as low overall picking efficiency, low degree of automation, and reliance on manual management. Summary of the invention

[0005] In order to solve the above problems, the present invention provides an automatic blueberry picking system based on visual recognition and path planning. The blueberry picking robot of the present invention can completely get rid of manual labor and pick blueberry fruits by itself through visual recognition and path planning, etc., without the need for human operation throughout the process, saving time and labor, and being safe and intelligent.

[0006] The technical solution adopted by the present invention to solve the technical problem is: The blueberry automatic picking system based on visual recognition and path planning includes A mobile assembly, including a tracked chassis and an emergency pull bar mounted at the rear thereof; A navigation component includes a radar disposed at the front end of the crawler chassis, a first camera, and a second camera fixed by a telescopic bracket, wherein the radar is used to detect obstacles and impassable road sections, and the first camera and the second camera are used to obtain the planting position of blueberry plants and the layout image of the park in real time, and transmit them to the control component to realize path planning; The picking assembly includes a bionic mechanical arm and a storage box installed on the crawler chassis. The top of the bionic mechanical arm is equipped with a third camera, and the front end is equipped with a comb-tooth picker. The bottom of the comb-tooth picker faces the storage box. The storage box is installed on the crawler chassis through a lifting mechanism. The control component is used to receive the transmission data of the radar, the first camera, and the second camera, and to perform picking path planning and charging path planning through spatial modeling and path planning algorithms. It is also used to receive the transmission data of the third camera and to identify the color, shape, size and position of the blueberry fruit through a visual recognition algorithm, and to generate fruit maturity data and spatial displacement data to achieve autonomous picking.

[0007] As an improvement of the above technical solution, the emergency pull rod is a foldable and telescopic pull rod.

[0008] As an improvement of the above technical solution, an emergency stop button is provided on the crawler chassis.

[0009] As an improvement of the above technical solution, a power supply box is provided on the crawler chassis, and the power supply box is located below the lifting mechanism.

[0010] As an improvement of the above technical solution, a charging indicator light and nighttime clearance lights are provided on the crawler chassis.

[0011] As an improvement of the above technical solution, the bionic robotic arm is a six-axis robotic arm, which includes multiple joints and an upper arm and a lower arm, and can perform various actions such as extension, bending, and rotation in the x-axis, y-axis, and z-axis directions. The comb-tooth picker is installed at the wrist at the front end of the lower arm.

[0012] As an improvement of the above technical solution, the storage box has a built-in weighing sensor, which can monitor the blueberry collection status in real time and transmit the data to the control component through Zigbee communication.

[0013] As an improvement of the above technical solution, the lifting mechanism is a small manual scissors lifting platform.

[0014] As an improvement of the above technical solution, the control component includes a control panel installed on the crawler chassis for data communication and display.

[0015] The present invention also adopts the following technical solutions to solve the technical problems: The automatic blueberry picking method based on visual recognition and path planning includes S1 Path Planning The radar, the first camera and the second camera are used to obtain the planting location of blueberry plants, the layout image of the plantation, and the obstacle image in real time, to perform fusion modeling of the plantation, and to plan the picking path based on the path planning algorithm; The blueberry picking robot is tested for power. When the power is lower than the set value, the charging pile is located and the charging path is planned based on the path planning algorithm. When the power level is higher than the set value, the blueberry picking robot will perform intelligent cruising along the planned picking path; S2 autonomous picking The third camera acquires blueberry fruit images in real time, identifies the color, shape, size and position of the blueberry fruit based on a visual recognition algorithm, generates fruit maturity data and performs target comparison; When the maturity meets the target ratio < 80%, continue cruising; When the maturity meets the target proportion ≥ 80%, the cruise is stopped and spatial displacement data is generated; Based on the spatial displacement data, the bionic robotic arm is controlled to complete the picking of blueberries and reset; S3 Fruit Collection The blueberry harvesting status is monitored in real time through the storage box. When the set harvesting weight is not reached, the cruise is continued and S2 is executed; When the set picking weight is reached, the robot is located at the nearest blueberry collection point and controls the blueberry picking robot to navigate there to perform box changing operations; Continue picking or automatically recharge.

[0016] As an improvement of the above technical solution, in S1, when performing picking path planning based on the path planning algorithm, the route can be set manually, and the blueberry picking robot can be controlled to cruise along the given route.

[0017] As an improvement of the above technical solution, in S1, when the blueberry picking robot encounters an obstacle or an impassable section during the cruising process, it will stop cruising, record and feedback the obstacle data, and generate spatial information to update the cruising route.

[0018] As an improvement of the above technical solution, in S2, when the maturity meets the target proportion of 50%≤c<80%, it is fed back to the control panel for manual selection whether to pick or continue cruising.

[0019] As an improvement of the above technical solution, in S2, after the spatial displacement data is generated, it is analyzed whether the bionic robotic arm described in the spatial displacement data is reachable; when the bionic robotic arm cannot reach, the distance is measured and the robot is cruised again.

[0020] The beneficial effects brought by the present invention are: The present invention obtains the topography of the blueberry picking environment and the characteristics of blueberry plants in real time, and then trains the model through means such as machine learning, so that the robot can achieve autonomous path planning and automatic picking.

[0021] First, path planning of the blueberry picking robot: The blueberry picking robot involved in the present invention can independently realize path planning, including planning of the picking route during operation and planning of the path between the work place and the charging station when charging is needed.

[0022] Second, blueberry picking robot visual recognition: visually identify blueberry fruits, judge their maturity, and classify and pick them according to different needs.

[0023] At the same time, the picking system of the present invention has a high degree of automation, high operating efficiency, good environmental adaptability, safety and hygiene, and strong scalability, and is also applicable to other small berry crops. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0025] Figure 1 This is a schematic diagram of the structure of an automatic blueberry picking system based on visual recognition and path planning in Example 1; Figure 2 This is a schematic diagram of the process of the automatic blueberry picking method based on visual recognition and path planning of Example 2.

[0026] Numbers in the figure: 1. First camera; 2. Radar; 3. Telescopic bracket; 4. Second camera; 5. Third camera; 6. Bionic robotic arm; 7. Comb-tooth picker; 8. Storage box; 9. Lifting platform; 10. Power box; 11. Control panel; 12. Emergency stop button; 13. Charging indicator light and night clearance light; 14. Tracked chassis; 15. Emergency pull rod. DETAILED DESCRIPTION

[0027] The following will be combined with the embodiments of the present invention and the accompanying drawings to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] Example 1 Reference Figure 1 , an automatic blueberry picking system based on visual recognition and path planning, including The mobile assembly includes a crawler chassis 14 and an emergency pull rod 15 installed at the rear thereof. The crawler chassis 14 can adapt to the uneven terrain of the blueberry planting area, is relatively stable when traveling, and has a strong carrying capacity, which can effectively increase the single transportation volume of blueberries during the picking process; The navigation component includes a radar 2, a first camera 1, and a second camera 4 installed and fixed by a telescopic bracket 3, arranged at the front end of the crawler chassis 14. The radar 2 is used to detect obstacles and impassable road sections. The first camera 1 and the second camera 4 are used to obtain the planting position of blueberry plants and the layout image of the park in real time, and transmit them to the control component to realize path planning; The picking assembly includes a bionic mechanical arm 6 and a storage box 8 installed on the crawler chassis 14. The top of the bionic mechanical arm 6 is equipped with a third camera 5, and the front end is equipped with a comb-tooth picker 7. The bottom of the comb-tooth picker 7 faces the storage box 8. The storage box 8 is installed on the crawler chassis 14 through a lifting mechanism. The control component is used to receive the transmission data of the radar 2, the first camera 1, and the second camera 4, and perform picking path planning and charging path planning through spatial modeling and path planning algorithms. It is also used to receive the transmission data of the third camera 5 and identify the color, shape, size and position of the blueberry fruit through a visual recognition algorithm, and generate fruit maturity data and spatial displacement data to achieve autonomous picking.

[0029] The automatic blueberry picking system of this embodiment can perform path planning based on the path planning algorithm for the layout of the blueberry plantation area: according to the complexity of the problem, combined with the data of the first camera 1, the second camera 4, and the radar 2, the Autoware outdoor mapping model is integrated, and the A* algorithm or the Dijkstra algorithm is selected to realize complex path planning and obstacle avoidance. It should be noted that the smoothing processing and dynamic planning method of the generated path should be considered to update the path in real time for path planning, and the tracked chassis 14 is used for field application to move, and the multi-steering bionic robot arm 6 is carried to realize the functions of blueberry transportation, obstacle avoidance, etc.; the bionic robot arm 6 The front-end link blueberry picking tool is equipped with a third camera 5, uses an image recognition algorithm, and is based on the YOLOv5 model. Through customized steps such as data annotation and fine-tuning the model, the maturity of blueberries can be distinguished and identified. The detailed basic steps include environment setting code, data preparation code (for image acquisition and maturity annotation), data conversion code (converting the annotated data into a YOLOv5 recognizable format), modifying the configuration file, training the model, and reasoning evaluation to achieve recognition and positioning of blueberries, and the control component controls the blueberry picking. After the picking is completed, the blueberries are put into the storage box 8. When the storage box 8 is full, the blueberry picking robot can move to a designated location to store the blueberries in a centralized manner.

[0030] In this embodiment, the bionic robot arm 6 is preferably a six-axis robot arm. It has multiple joints, divided into an upper arm and a lower arm, and can flexibly realize multiple movements such as extension, bending, and rotation, as well as a wide working range. The bionic robot arm 6 can move in the x-axis, y-axis, and z-axis directions, and can use rolling, pitching, and other movements for positioning, which is suitable for simulating the complex movements of human arms. After the crawler chassis 14 reaches the navigation position, the robot arm is controlled by the control component to simulate the manual picking process, complete the blueberry picking, and put the picked blueberries into the storage box 8 below.

[0031] Among them, the blueberry picking tool is a comb-tooth picker 7, which can effectively improve the blueberry picking efficiency and protect the blueberry plants from being damaged during the picking process. The tool is connected to the front wrist of the bionic mechanical arm 6 to simulate the process of manual fruit picking. The comb-tooth picker 7 is inserted into the plant cluster, and then pulled up forward and upward to pick the blueberry fruits; at the same time, the comb teeth are bent to a certain arc, so that the blueberry fruits are easier to collect.

[0032] As an improvement of the above embodiment, the emergency pull rod 15 is a foldable telescopic pull rod. At the same time, the crawler chassis 14 is provided with an emergency stop button 12 and a power box 10, the power box 10 is located below the lifting mechanism, and the outer side of the crawler chassis 14 can also be provided with a charging indicator light and a nighttime clearance light 13.

[0033] As an improvement of the above embodiment, the lifting mechanism is a small manual scissors lifting platform 9.

[0034] In this embodiment, the control component includes a processor with a built-in algorithm and a machine learning model, and a control panel 11 connected to the processor in communication, which can realize the following functions by controlling the camera, the crawler chassis 14, the bionic robot arm 6, etc.: 1. Use image recognition algorithms to identify the color, shape, size, maturity and location of blueberries; 2. According to the planting location of blueberry plants and the layout plan of the park, the path planning algorithm is used to plan the path and realize intelligent field patrol; 3. Use the bionic robot arm 6 to control the comb-tooth picking tool to pick blueberries; 4. Wireless communication is used for data collection, data transmission, data feedback, etc. The weighing sensor built into the storage box 8 can be used to monitor the blueberry collection status (such as overflow, weight, etc.) in real time. Zigbee communication can also be used to transmit data to a management platform, such as a mobile phone APP, a computer client, etc. The management personnel can remotely control the data based on the relevant data, and adjust the data and working area of ​​the robots running in the park in a timely manner to improve work efficiency.

[0035] Example 2 Reference Figure 2 The automatic blueberry picking method based on visual recognition and path planning of this embodiment is applied to the automatic blueberry picking system of embodiment 1, including: S1 Path Planning The planting location of blueberry plants, the layout image of the park, and the obstacle image data are obtained in real time through the radar 2, the first camera 1, and the second camera 4, and the planting park is fused modeled, and the picking path is planned based on the path planning algorithm; the fusion modeling of this embodiment can use SLAM to build a three-dimensional map, in which the open source RTAB-Map library integrates vision and laser radar 2 solutions, which can realize real-time positioning and navigation in unknown complex environments; in addition, for simpler environments, Gmapping\hector\karto\cartographer can be used for 2D modeling; The blueberry picking robot is tested for power. When the power is lower than the set value, the charging pile is located, and the charging path is planned based on the path planning algorithm. For example, the charging pile target is set in the Autoware outdoor mapping model, and the A* algorithm or Dijkstra algorithm is selected to implement charging path planning and obstacle avoidance in combination with camera and sensor data. At the same time, the path smoothing and dynamic planning methods are used to generate the path to update the path in real time. When the power level is higher than the set value, the blueberry picking robot will perform intelligent cruising along the planned picking path; For the planned picking path, you can manually adjust and give a route, so when performing intelligent cruising, you can choose to cruise along a given route or a planned route; When the blueberry picking robot encounters an obstacle or an impassable section during the cruising process, it will stop cruising, record and feedback the obstacle data, generate spatial information to the fusion modeling step, and update the cruising route in real time.

[0036] S2 autonomous picking The third camera 5 acquires blueberry fruit images in real time, identifies the color, shape, size and position of the blueberry fruit based on a visual recognition algorithm, generates fruit maturity data and performs target comparison; When the maturity meets the target ratio < 80%, continue cruising; When the maturity meets the target proportion of 50%≤c<80%, it is fed back to the control panel 11 to manually choose whether to pick or continue cruising; When the maturity meets the target proportion ≥ 80%, the cruise is stopped and spatial displacement data is generated; Analyze the spatial displacement data to determine whether the bionic robot arm 6 is reachable. The implementation method is to simulate the movement of the chassis and the robot arm after running the A* algorithm in the path planning algorithm, taking a simple code written in Python as an example, and determine whether the maximum dx and dy values ​​of the movable amount match the position information obtained by positioning; when the bionic robot arm 6 cannot reach, measure the distance and re-cruise, otherwise, based on the spatial displacement data, control the bionic robot arm 6 to complete the picking of blueberries and reset; Specifically, the end of the robot arm is controlled to move to a precise picking position, and the controller at the end of the robot arm changes the angle to control the rotation of the robot arm to complete the picking, and then the robot arm is reset, and the controller at the end of the robot arm changes the angle again to control the rotation of the robot arm to dump the picked blueberries into a storage box; S3 Fruit Collection The blueberry collection status is monitored in real time through the storage box 8. When the set weight of picking is not reached, the cruise is continued and S2 is executed; When the set picking weight is reached, the robot is located at the nearest blueberry collection point and controlled to navigate there to perform box-changing operations. For example, the collection point target is set in the Autoware outdoor mapping model, and the A* algorithm or Dijkstra algorithm is selected to implement recycling path planning and obstacle avoidance in combination with camera and radar 2 data. The box-changing operation is completed manually. Continue picking or automatically recharge.

[0037] It should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. The blueberry automatic picking system based on visual recognition and path planning is characterized by: include A mobile assembly, including a tracked chassis and an emergency pull bar mounted at the rear thereof; A navigation component includes a radar disposed at the front end of the crawler chassis, a first camera, and a second camera fixed by a telescopic bracket, wherein the radar is used to detect obstacles and impassable road sections, and the first camera and the second camera are used to obtain the planting position of blueberry plants and the layout image of the park in real time, and transmit them to the control component to realize path planning; The picking assembly includes a bionic mechanical arm and a storage box installed on the crawler chassis. The top of the bionic mechanical arm is equipped with a third camera, and the front end is equipped with a comb-tooth picker. The bottom of the comb-tooth picker faces the storage box. The storage box is installed on the crawler chassis through a lifting mechanism. The control component is used to receive the transmission data of the radar, the first camera, and the second camera, and to perform picking path planning and charging path planning through spatial modeling and path planning algorithms. It is also used to receive the transmission data of the third camera and to identify the color, shape, size and position of the blueberry fruit through a visual recognition algorithm, and to generate fruit maturity data and spatial displacement data to achieve autonomous picking.

2. The automatic blueberry picking system according to claim 1, characterized in that: A power supply box is provided on the crawler chassis, and the power supply box is located below the lifting mechanism; The crawler chassis is provided with a charging indicator light and a nighttime clearance light.

3. The automatic blueberry picking system according to claim 1, characterized in that: The bionic robotic arm is a six-axis robotic arm, which includes multiple joints and an upper arm and a lower arm, and can perform multiple actions such as extension, bending, and rotation in the x-axis, y-axis, and z-axis directions. The comb-tooth picker is installed at the front wrist of the lower arm.

4. The automatic blueberry picking system according to claim 1, characterized in that: The storage box has a built-in weighing sensor that can monitor the blueberry collection status in real time and transmit the data to the control component via Zigbee communication.

5. The automatic blueberry picking system according to claim 1, characterized in that: The lifting mechanism is a small manual scissor lift platform.

6. The automatic blueberry picking system according to claim 1, characterized in that: The control assembly includes a control panel installed on the crawler chassis for communicating and displaying data.

7. The automatic blueberry picking method based on visual recognition and path planning is characterized by: The method is applied to the automatic blueberry picking system according to any one of claims 1 to 6, comprising: S1 Path Planning The radar, the first camera and the second camera are used to obtain the planting location of blueberry plants, the layout image of the plantation, and the obstacle image in real time, to perform fusion modeling of the plantation, and to plan the picking path based on the path planning algorithm; The blueberry picking robot is tested for power. When the power is lower than the set value, the charging pile is located and the charging path is planned based on the path planning algorithm. When the power level is higher than the set value, the blueberry picking robot will perform intelligent cruising along the planned picking path; S2 autonomous picking The third camera acquires blueberry fruit images in real time, identifies the color, shape, size and position of the blueberry fruit based on a visual recognition algorithm, generates fruit maturity data and performs target comparison; When the maturity meets the target ratio < 80%, continue cruising; When the maturity meets the target proportion ≥ 80%, the cruise is stopped and spatial displacement data is generated; Based on the spatial displacement data, the bionic robotic arm is controlled to complete the picking of blueberries and reset; S3 Fruit Collection The blueberry harvesting status is monitored in real time through the storage box. When the set harvesting weight is not reached, the cruise is continued and S2 is executed; When the set picking weight is reached, the robot is located at the nearest blueberry collection point and controls the blueberry picking robot to navigate there to perform box changing operations; Continue picking or automatically recharge.

Images