A fruit and vegetable picking robot arm based on visual analysis

By using a combination of multiple micro cameras and sensors in the fruit and vegetable picking robot, the accurate identification and picking of bar fruits and vegetables is achieved, solving the problems of high cost and difficult maintenance in the existing technology, and reducing hardware and maintenance costs. Suitable for a variety of fruit and vegetable species.

CN118617433BActive Publication Date: 2024-11-15CHONGQING XINDA ZHISHENG TECH CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410763422.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-06-13
Publication Date
2024-11-15
Estimated Expiration
2044-06-13

AI Technical Summary

Technical Problem

The existing fruit and vegetable picking robots are costly and have high maintenance costs, and are inconvenient for picking a variety of fruits and vegetables. In particular, the identification of bar fruit and vegetable requires precise calculation and adjustment of the field of view, which leads to a lack of widespread application.

Method used

Multiple micro cameras are configured as machine vision terminals, combining sensors and shear clamping mechanisms, and automatic identification and picking of fruits and vegetables can be achieved through machine vision analysis and control systems, reducing hardware and maintenance costs.

Benefits of technology

It realizes accurate identification and picking of bar fruits and vegetables, reduces hardware costs by about 30%, simplifies maintenance difficulty, and is suitable for a variety of fruits and vegetables, especially in the identification process of bar fruits and vegetables, without the need to accurately calculate the identification distance and adjust the field of vision.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118617433B_ABST
    Figure CN118617433B_ABST
Patent Text Reader

Abstract

The present invention provides a fruit and vegetable picking robot arm based on visual analysis, including a robot arm, a machine vision terminal is arranged on the robot arm, the machine vision terminal includes a plurality of micro cameras arranged in an array, each micro camera is connected to a control system, and the height of the distribution area of ​​all micro cameras is not less than the length of the picked fruits and vegetables; wherein all micro cameras are horizontally installed on a vertically arranged substrate, and the lenses of all micro cameras are substantially flush with the outer wall of the substrate. The present invention provides a brand-new solution for fruit and vegetable picking robots, which only requires the use of multiple conventional micro cameras configured into a machine vision terminal to smoothly and accurately identify strips of fruits and vegetables that meet the picking requirements. Compared with existing picking robots with the same specific functions, the hardware cost is greatly reduced, and the use and maintenance costs and difficulty can also be significantly reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the technical field of fruit and vegetable picking robots, and in particular relates to a fruit and vegetable picking robot arm based on visual analysis. Background Art

[0002] Traditional fruit and vegetable picking (such as greenhouse-grown tomatoes, strawberries, cucumbers, loofahs, and peppers) is the most time-consuming and labor-intensive link in the agricultural production chain. It is costly, seasonal, and requires a large number of labor-intensive jobs. To this end, intelligent harvesting technology using robots to harvest fruits and vegetables has developed rapidly. Many fruit and vegetable industrial bases have widely used fruit and vegetable harvesting robots, such as the Sweeper robot, which can automatically move in the greenhouse, observe the bell peppers to determine whether they are ripe enough, and then saw the peppers off the plants and put them into the basket; Crux Agribotics cucumber inspection and harvesting robot, a cucumber picking robot based on visual technology and machine learning, scans and harvests, and according to detailed 3D information of location and size, it transports the cucumbers to the packaging area after harvesting without human intervention, realizing full automation from harvesting to packaging.

[0003] In the prior art, there are cucumber picking robot systems and picking methods in a greenhouse environment (document CN101356877A), fruit and vegetable picking manipulators, picking robots and picking methods (document CN111757380A), which are mainly suitable for picking fruits and vegetables with high fruit stalk strength and no obvious absorptivity, such as cucumbers, loofahs, and tomatoes.

[0004] As mentioned above, almost all existing fruit and vegetable picking robots adopt the technical route of "using machine vision system to collect images, based on a series of actions such as image recognition, segmentation, positioning, and shearing". This technical route usually requires the configuration of high-precision visual terminals (one of the core components) and relatively complex algorithm models, resulting in a high cost for the entire set of fruit and vegetable picking robots, which is not conducive to the promotion and application of fruit and vegetable planting in greenhouses, and the maintenance cost of this type of fruit and vegetable picking robots is high. More importantly, there are many types of fruits and vegetables, which can be divided into spherical fruits and vegetables (such as tomatoes), strip fruits and vegetables (such as loofah, cucumber, bitter melon), and flat fruits and vegetables (such as lentils). In actual application, strip fruits and vegetables also need to be within a specific larger field of view to effectively collect images. Summary of the invention

[0005] At least in order to solve the technical problems mentioned in the background technology, the present invention provides a fruit and vegetable picking robot arm based on visual analysis.

[0006] The present invention adopts the following technical solution.

[0007] A fruit and vegetable picking robot arm based on visual analysis comprises a robot arm on which a machine vision terminal is arranged, the machine vision terminal comprises a plurality of micro cameras arranged in an array, the spacing between any adjacent micro cameras is equal and not greater than 2 cm, each micro camera is connected to a control system, and the height of the distribution area of ​​all micro cameras is not less than the length of the picked fruits and vegetables; wherein all the micro cameras are horizontally mounted on a vertically arranged base plate, and the lenses of all the micro cameras are substantially flush with the outer wall of the base plate.

[0008] As a preferred solution, the height of the distribution area of ​​all micro cameras is 30~60cm and the width is 20~30cm, and the picked fruits and vegetables are loofahs, cucumbers or bitter gourds grown in greenhouses.

[0009] Furthermore, a sensor is provided on the outer wall of the substrate and on the lens side of the micro camera. The sensor is connected to the control system and is used to sense whether the picked fruits and vegetables are close to the outer wall of the substrate.

[0010] As a preferred solution, the substrate is a flat plate or an arc-shaped open plate, and the width of the substrate is not less than the maximum diameter of the picked fruits and vegetables. Such a solution is also conducive to more smooth and flexible identification of the picked fruits and vegetables.

[0011] As a preferred solution, the lens area of ​​a single micro camera is no larger than 5 mm.

[0012] Furthermore, a shearing and clamping mechanism is arranged on the robot arm and above the top of the base plate, and the shearing and clamping mechanism is used to shear the fruit stalks of the picked fruits and vegetables.

[0013] Furthermore, the control system includes a memory, a processor, and a program stored in the memory and executable on the processor, and when the processor executes the program, the following steps are implemented:

[0014] S1, controlling the robot arm to move, and when the sensor senses that the picked fruits and vegetables are close to the outer wall of the substrate, controlling the robot arm to pause;

[0015] S2, controlling the machine vision terminal to work, so that all micro cameras can capture image units, and transmit the captured image units to the control system;

[0016] S3, stitching all captured image units into a large image;

[0017] S4, selecting target image units from the obtained large image, and calculating the total height difference of the micro cameras corresponding to all target image units (actually refers to the height difference between the positions of the micro cameras corresponding to the first target image unit and the last target image unit in the same column);

[0018] S5, judging whether the picked fruits and vegetables meet the picking requirements according to the sum of the obtained total height difference + δ, where δh represents the height difference between the centers of adjacent micro cameras in the same row;

[0019] S6, when the sum of the obtained total height difference + δ is within the threshold interval, the shearing and clamping mechanism is controlled to work, the stalks of the picked fruits and vegetables are cut off and the picked fruits and vegetables are moved to the specified position; when the sum of the obtained total height difference + δ is less than the minimum value of the threshold interval, the robot arm is controlled to move away;

[0020] S7, repeat steps S1 to S6.

[0021] As a preferred solution, the target image unit refers to a blurred pattern photographed at a shooting distance of no more than 1 cm.

[0022] Beneficial effects: The present invention provides a brand-new solution for fruit and vegetable picking robots. It only needs to use multiple conventional miniature cameras (such as pinhole cameras) to configure a machine vision terminal to smoothly and accurately identify strips of fruits and vegetables that meet the picking requirements. Compared with existing picking robots with the same specific functions, the hardware cost can be reduced by at least about 30%, and the cost and difficulty of use and maintenance can also be significantly reduced. When identifying strips of fruits and vegetables, as long as the strips of fruits and vegetables can be kept close to the outer wall of the substrate, the image can be effectively collected. It is very convenient and practical, and there is no need to accurately calculate the recognition distance and adjust to a specific field of view as in the existing solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a three-dimensional schematic diagram of a fruit and vegetable picking robot arm (partial) based on visual analysis in Example 1;

[0024] Figure 2 This is a side schematic diagram of a fruit and vegetable picking robot arm (partial) based on visual analysis in Example 1;

[0025] Figure 3 This is a schematic diagram of a fruit and vegetable picking robot arm (partial) based on visual analysis in Example 1;

[0026] Figure 4 It is a three-dimensional schematic diagram of a fruit and vegetable picking robot arm (partial) based on visual analysis in Example 2;

[0027] Figure 5 A single-column image of a large image in Example 1;

[0028] Figure 6 , Figure 7 for Figure 5 A partial enlarged view of the

[0029] Figure 8 It is a single-column image of a large picture in Example 2. DETAILED DESCRIPTION

[0030] The technical solution of the present invention is described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Example 1

[0031] Combination Figures 1 to 3 As shown, a fruit and vegetable picking robot arm based on visual analysis is mainly used for picking loofah (variety: stick loofah) planted in a greenhouse. The optimal picking length of this loofah is 24~32cm, that is, when its length reaches 24~32cm, it meets the picking requirements. The robot arm includes a robot arm 1, on which a machine vision terminal is arranged, and the machine vision terminal includes a plurality of micro cameras 2 arranged in an array, and the spacing between any adjacent micro cameras 2 is equal and 2cm, and each micro camera 2 is connected to a control system. The height of the distribution area of ​​all micro cameras 2 is 45cm (a total of 12*6 micro cameras 2 are arranged), and the width is 20cm. The height difference δh between the centers of adjacent micro cameras 2 in the same row is 3cm, and the height of the distribution area of ​​all micro cameras 2 is not less than the length of the picked fruits and vegetables. The micro cameras 2 use pinhole cameras; wherein all micro cameras 2 are horizontally mounted on a vertically arranged substrate 3, and the lenses of all micro cameras 2 are substantially flush with the outer wall of the substrate 3.

[0032] In this embodiment, a sensor 4 is provided on the outer wall of the substrate 3 and located on the lens side of the micro camera 2. The sensor 4 is a laser sensor. The sensor 4 is connected to the control system and is used to sense whether the picked fruits and vegetables are close to the outer wall of the substrate 3. During use, when the loofah is close to the sensor 4, the sensor 4 can sense and feedback information (sensing the target object through the sensor belongs to conventional technology and is not described in detail in the present invention); the substrate 3 is a flat plate. A shearing clamping mechanism 5 is provided on the robot arm 1 and located above the top of the substrate 3. The shearing clamping mechanism 5 is used to shear the fruit stems of the picked fruits and vegetables. The shearing clamping mechanism 5 can adopt the same structure as that of the existing fruit and vegetable picking robot.

[0033] In this embodiment, the control system includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the following steps are implemented:

[0034] S1, control the robot arm 1 to move, and the base plate 3 will move synchronously during the movement of the robot arm 1. When the sensor senses that the picked fruits and vegetables are close to the outer wall of the base plate 3, the robot arm 1 is controlled to pause. At this time, the distance between the lens surface of the micro camera 2 located on the side of the loofah and the surface of the loofah will not exceed 1 cm (it can be considered that the micro camera 2 is close to the surface of the loofah). In this case, the image taken is a blurred pattern. This process is similar to taking a photo with a lens close to an object;

[0035] S2, controlling the machine vision terminal to work, so that all the micro cameras 2 capture image units, and transmit the captured image units to the control system;

[0036] S3, stitching all captured image units into a large image;

[0037] S4, selecting target image units from the obtained large image, and calculating the total height difference of the micro cameras corresponding to all target image units (actually refers to the height difference between the positions of the micro cameras corresponding to the first target image unit and the last target image unit in the same column); the target image unit refers to the blurred pattern captured when the shooting distance is not greater than 1 cm;

[0038] S5, judging whether the picked fruits and vegetables meet the picking requirements according to the sum of the obtained total height difference + δ, where δh represents the height difference between the centers of adjacent micro cameras 2 in the same row. When the sum of the total height difference + δh reaches 24-30 cm, the fruits and vegetables meet the picking requirements;

[0039] S6, when the sum of the total height difference + δ is within the threshold range (24-32 cm), the cutting and clamping mechanism 5 is controlled to work, the stalks of the picked fruits and vegetables are cut and the cut fruits and vegetables are moved to the designated position; when the sum of the total height difference + δ is less than the minimum value of the threshold range, the robot arm 1 is controlled to move away;

[0040] S7, repeat steps S1 to S6.

[0041] In one of the schemes, a row of micro cameras 2 that just happened to capture a loofah (the actual length of the loofah is 29.4 cm) are numbered from top to bottom as A0, A1 (calibrated coordinates are {16 cm, 57 cm}), A2, A3, A4, A5, A6, A7, A8, A9, A10 (calibrated coordinates are {16 cm, 30 cm}), and A11. The image units captured by this row of micro cameras 2 are stitched together to form a large picture as shown in the figure. Figure 5 As shown, all image units captured by the micro camera (A0) and the micro camera (A1) are as follows Figure 6 As shown, all image units captured by the micro camera (A10) and the micro camera (A11) are as follows Figure 7 As shown. Figure 5-7 It can be clearly seen that the image unit 102 captured by the micro camera (A11) is relatively clear, the image unit 101 captured by the micro camera (A10) is relatively clear, and the image unit 101 captured by the micro cameras (A1~A10) are all very blurry. Image unit 101 is determined to be the target image unit. This is because the distance between the lens surface of the micro cameras (A1~A10) and the surface of the loofah (shooting distance) is very small (less than 1 cm), which makes the captured image look blurry, while the distance between the lens surface of the micro cameras (A0 and A11) and the surface of the loofah (shooting distance) is relatively large and is not blocked by the loofah, which makes the captured image look clearer. Therefore, the micro cameras (A1~A10) correspond to the side positions of the loofah. Since the positions of each micro camera are fixed and the positions of the micro cameras are pre-calibrated, it is only necessary to calculate the height of the position of the micro camera (A1~A10) to characterize the length of the loofah, that is, the first target image unit ( Figure 5 The image unit captured by the micro camera (A1) and the last target image unit ( Figure 5 The height difference + δh of the position of the micro camera corresponding to the image unit captured by the micro camera (A10) in the middle represents the length of the loofah, so as to judge whether the loofah meets the picking requirements. In this embodiment, the height difference + δh between the position of the micro camera A1 (calibrated coordinates are {16cm, 57cm}) and the micro camera A10 (calibrated coordinates are {16cm, 30cm}) is 30cm, which falls within the threshold range (24~32cm), meeting the picking requirements. It can be seen that the result of judging the length of the loofah by the scheme in this embodiment is 30cm, which is basically the same as the actual length of the loofah 29.4cm, and the recognition accuracy is good. Example 2

[0042] A fruit and vegetable picking robot arm based on visual analysis, the structure of which is based on Example 1 and combined with Figure 4As shown, the height difference δh between the centers of adjacent micro cameras 2 in the same row in this example is 4cm. The number and serial number of micro cameras 2 and their calibrated coordinate positions are adaptively adjusted, mainly for picking a certain variety of cucumbers grown in greenhouses. The optimal picking length of this cucumber is 12~20cm, that is, when its length reaches 11~20cm, it meets the picking requirements. In one of the schemes, a row of micro cameras 2 that just took pictures of cucumbers (the actual length of cucumbers is 15.5cm) are numbered from top to bottom as B0, B1, B2, B3 (calibrated coordinates are {12cm, 36cm}), B4 (calibrated coordinates are {12cm, 32cm}), B5, B6, B7 (calibrated coordinates are {12cm, 20cm}), B8, B9, B10, B11. All the image units captured by this row of micro cameras 2 are spliced ​​into a large picture as shown in the figure below. Figure 8 As shown. Figure 8 It can be clearly seen that the image unit 200 captured by the micro camera (B3) is relatively clear, the image unit 202 captured by the micro camera (B8) is relatively clear, and the image unit 201 captured by the micro cameras (B4~B7) are all very blurry and are determined to be target image units. This is because the distance between the lens surface of the micro cameras (B4~B7) and the cucumber surface (shooting distance) is very small (less than 1 cm), causing the captured image to look blurry, while the distance between the lens surface of the micro cameras (B4 and B8) and the cucumber surface (shooting distance) is relatively large and is not blocked by the cucumber, causing the captured image to look clearer. Therefore, the micro camera (B4~B8) corresponds to the positive side position of the cucumber. Since the position of each micro camera is fixed and the position of the micro camera is pre-calibrated, it is only necessary to calculate the height of the position of the micro camera (B4~B8) to characterize the length of the cucumber, that is, the height difference + δh of the position of the micro camera corresponding to the first target image unit (the image taken by the micro camera B4) and the last target image unit (the image taken by the micro camera B8) in the same column is used to characterize the length of the cucumber, so as to judge whether the cucumber meets the picking requirements. In this embodiment, the height difference + δh between the position of the micro camera B4 (calibrated coordinates are {12cm, 32cm}) and the micro camera B8 (calibrated coordinates are {12cm, 20cm}) is 16cm, which falls within the threshold range (11~20cm) and meets the picking requirements. It can be seen that the result of judging the length of the cucumber by the scheme in this embodiment is 16cm, which is basically the same as the actual length of the cucumber 15.5cm, and the recognition accuracy is good.

[0043] The present invention provides a brand-new solution for the fruit and vegetable picking robot. It only needs to use a plurality of conventional micro cameras (such as pinhole cameras) to configure a machine vision terminal to smoothly and accurately identify the strip-shaped fruits and vegetables that meet the picking requirements. Compared with the existing picking robots with the same specific functions, the hardware cost can be reduced by at least about 30% (especially the existing picking robots need to use high-definition cameras, while the present invention only needs to use conventional micro cameras), and the use and maintenance costs and difficulty can also be significantly reduced. When identifying strip-shaped fruits and vegetables, as long as the strip-shaped fruits and vegetables can be ensured to be close to the outer wall of the substrate, the image can be effectively collected, which is very convenient and practical, and there is no need to accurately calculate the recognition distance and adjust to a specific field of view state as in the existing solutions.

Claims

1. A fruit and vegetable picking robot arm based on visual analysis, comprising a robot arm (1), on which a machine vision terminal is arranged, characterized in that: The machine vision terminal comprises a plurality of micro cameras (2) arranged in an array, wherein the spacing between any adjacent micro cameras (2) is equal and not greater than 2 cm, each micro camera (2) is connected to a control system, and the height of the distribution area of ​​all micro cameras (2) is not less than the length of the picked fruits and vegetables; wherein all micro cameras (2) are horizontally mounted on a vertically arranged substrate (3), and the lenses of all micro cameras (2) are substantially flush with the outer wall of the substrate (3); a sensor is arranged on the outer wall of the substrate (3) and located on the lens side of the micro camera (2), the sensor being connected to the control system and used to sense whether the picked fruits and vegetables are close to the outer wall of the substrate (3); the substrate (3) is a flat plate or an arc-shaped open plate, and the width of the substrate (3) is not less than the maximum diameter of the picked fruits and vegetables; The control system includes a memory, a processor, and a program stored in the memory and running on the processor. When the processor executes the program, the following steps are implemented: S1, controlling the robot arm (1) to move, and when the sensor senses that the picked fruits and vegetables are against the outer wall of the substrate (3), controlling the robot arm (1) to pause; S2, controlling the machine vision terminal to work, so that all the micro cameras (2) capture image units, and transmit the captured image units to the control system; S3, stitching all captured image units into a large image; S4, selecting target image units from the obtained large image, and calculating the total height differences of the micro cameras corresponding to all target image units; S5, judging whether the picked fruits and vegetables meet the picking requirements according to the sum of the obtained total height difference + δ, where δh represents the height difference between the centers of adjacent micro cameras (2) in the same row; S6, when the sum of the obtained total height difference + δ is within the threshold interval, the shearing and clamping mechanism is controlled to work, the fruit stalk of the picked fruit and vegetable is cut off and moved to a specified position; when the sum of the obtained total height difference + δ is less than the minimum value of the threshold interval, the robot arm (1) is controlled to move away; S7, repeat steps S1 to S6.

2. The fruit and vegetable picking robot arm according to claim 1, characterized in that: The height of the distribution area of ​​all the micro cameras (2) is 30~60cm and the width is 20~30cm. The picked fruits and vegetables are loofahs, cucumbers or bitter melons grown in greenhouses.

3. The fruit and vegetable picking robot arm according to claim 2, characterized in that: The lens area of ​​a single micro camera (2) is no larger than 5 mm.

4. According to the fruit and vegetable picking robot arm of claim 3, a shearing and clamping mechanism is arranged on the robot arm (1) and above the top of the base plate (3), and the shearing and clamping mechanism is used to shear and clamp the stems of the picked fruits and vegetables.

5. The fruit and vegetable picking robot arm according to claim 4, characterized in that: The target image unit refers to a blurred pattern captured when the shooting distance is not greater than 1 cm.

Citation Information

Cited By

  • Small berry picking system and method based on depth camera and navigation positioning

    CN121816954A