Agricultural work apparatus, agricultural work method, and program

Abstract

This agricultural work apparatus is provided with: an arm apparatus provided with a hand apparatus for performing agricultural work on a target group including a plurality of targets; an imaging apparatus for imaging the target group; a target position estimation unit for estimating the position of the target group on the basis of image information obtained via the imaging apparatus imaging the target group; an arm apparatus control unit for moving the arm apparatus relative to the target group on the basis of the estimated position; and a hand apparatus control unit for, on the basis of the estimated position, controlling the hand apparatus in order to execute the agricultural work on the target group at the position. As a result, it is possible to provide technology for automatically performing agricultural work on a target group including a plurality of targets.

Description

Agricultural work equipment, agricultural work method, and program

[0001] The present invention relates to an agricultural work device, an agricultural work method, and a program.

[0002] Traditionally, harvesting work on large-scale farms has been accomplished by manpower, resulting in high labor costs. Large-scale farms have very long lanes, measuring 50 to 100 meters each, and the transportation costs are particularly high when using manpower. Therefore, there is a demand for automation (mechanization) of harvesting work. For example, Patent Document 1 describes a device that can automatically harvest vegetables with outer leaves, such as broccoli.

[0003] Japanese Patent Application Laid-Open No. 2018-046752

[0004] However, no technology has been developed to automatically perform harvesting operations on a group of objects containing multiple objects (e.g., a bunch containing multiple tomatoes).

[0005] Therefore, the disclosed technology aims to provide a technology for automatically performing agricultural work on a group of objects including multiple objects.

[0006] An agricultural work apparatus, which is one aspect of the disclosed technology, comprises an arm apparatus equipped with a hand device that performs agricultural work on a group of objects including multiple objects, an imaging device that images the group of objects, an object position estimation unit that estimates the position of the group of objects based on image information obtained by the imaging device imaging the group of objects, an arm apparatus control unit that moves the arm apparatus relative to the group of objects based on the estimated position, and a hand apparatus control unit that controls the hand apparatus to perform agricultural work on the group of objects at that position based on the estimated position.

[0007] According to the present invention, it is possible to automatically perform farm work on a group of objects including a plurality of objects.

[0008] 5 is a diagram showing an example of a harvesting robot according to an embodiment. FIG. 6 is a diagram showing an example of a block configuration of a harvesting robot according to an embodiment. FIG. 7 is a diagram showing an example of the functional configuration of a control PC according to an embodiment. FIG. 8 is a diagram showing an example of a hand unit according to an embodiment. FIG. 9 is a diagram showing an example of a right belt unit according to an embodiment. FIG. 10 is a diagram showing an example of a cross-sectional view of the right belt unit shown in FIG. 5. FIG. 11 is a diagram showing another example of a right belt unit according to an embodiment. FIG. 12 is a diagram showing another example of a hand unit according to an embodiment. FIG. 13 is a diagram showing another example of a right belt unit according to an embodiment. A rear view of an example of a hand unit according to an embodiment. FIG. 14 is a diagram showing another example of a hand unit according to an embodiment. A top view of an example of a hand unit according to an embodiment. A side view of an example of a hand unit according to an embodiment. FIG. 15 is a diagram showing an example of a left belt unit and a transport unit according to an embodiment. FIG. 16 is a diagram showing another example of a left belt unit and a transport unit according to an embodiment. A flowchart showing an example of an information processing flow of a harvesting robot according to an embodiment. A flowchart showing another example of an information processing flow of a harvesting robot according to an embodiment. A flowchart showing another example of an information processing flow of a harvesting robot according to an embodiment.

[0009] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. The following embodiments are merely examples for explaining the present invention, and are not intended to limit the present invention to the embodiments. Furthermore, the present invention can be modified in various ways without departing from the spirit of the present invention. Furthermore, the same components in each drawing will be designated by the same reference numerals whenever possible, and redundant explanations will be omitted whenever possible.

[0010] 1 is a diagram illustrating an example of a harvesting robot (agricultural work device) according to an embodiment of the present invention. The harvesting robot 2 is a robot that automatically performs harvesting (agricultural work) on a group of objects, for example, including multiple objects grown on a farm. The harvesting robot 2 includes an arm unit 600 (arm device) equipped with a hand unit 1 (hand device) that performs harvesting (agricultural work) on a group of objects, for example, including multiple objects grown on a farm, a camera unit 4 equipped with a harvesting camera 13 (imaging device) that captures images of the group of objects, a slider 6 for adjusting the position of the arm unit 600, a harvest box 5 for storing the harvested object (or group of objects), and a vehicle body 3 that enables the harvesting robot 2 to move along a lane R arranged on the farm.

[0011] For example, the harvesting robot 2 is a harvesting robot that can harvest (farm work) a bunch (group of targets) formed by a group of tomatoes (multiple targets) with one stroke of the arm unit 600. For example, the harvesting robot 2 recognizes a bunch with many red fruits (targets) using the harvesting camera 13 (image capturing device) and causes the arm unit 600 to approach the recognized bunch. The harvesting robot 2 applies a twisting force to the bunch using the belt provided in the hand unit 1, separating the multiple fruits from the bunch.

[0012] Here, we will provide an overview of the harvesting robot 2 according to this embodiment. As mentioned above, harvesting work on large-scale farms has traditionally been accomplished by manual labor, resulting in high labor costs. Large-scale farms have extremely long distances, with each lane measuring 50 to 100 meters, making manual labor particularly costly. Despite demands for automation (mechanization), automated harvesting has not yet been put to practical use. Specifically, cost-effectiveness is an issue. The difficulty in improving cost-effectiveness lies in the trade-off between harvest rate and harvesting speed, and the fact that the unit price of tomatoes is lower than the unit price of an automated harvesting robot. Regarding the trade-off between harvest rate and harvesting speed, if an automated harvesting robot were to harvest all tomatoes in the same way as a human, the harvesting speed would tend to decrease. The lower unit price of tomatoes than that of an automated harvesting robot is due to the difficulty in lowering the unit price of an automated harvesting robot (machine cost / serviceable stroke count) compared to the unit price of tomatoes.

[0013] The trade-off between harvest rate and harvesting speed can be addressed as follows. For example, an automatic harvesting robot can harvest tomatoes at night, and then a human can use a cart to harvest the remaining tomatoes. This allows the automatic harvesting robot's weakness (harvesting everything) to be complemented by the weakness of a human (increasing the workload). Furthermore, the unit price of tomatoes is lower than the unit price of an automatic harvesting robot, and the following can be addressed. For example, by adopting an automatic group-picking hand that can simultaneously harvest four to six tomatoes in one stroke, harvesting efficiency can be expected to be several times higher than before.

[0014] Therefore, the harvesting robot 2 according to this embodiment includes an arm device with a hand device that performs agricultural work on a group of objects that includes multiple objects, an imaging device that images the group of objects, an object position estimation unit that estimates the position of the group of objects based on image information obtained by imaging the group of objects with the imaging device, an arm device control unit that moves the arm device relative to the group of objects based on the estimated position, and a hand device control unit that controls the hand device to perform agricultural work on the group of objects at the estimated position. The harvesting robot 2 according to this embodiment can automatically perform agricultural work on a group of objects that includes multiple objects.

[0015] "Agricultural work" refers to work for agricultural production. Agricultural work includes, for example, at least one of grasping, cutting, separating from bunches, and harvesting tomatoes (objects) grown on a farm. The "objects" in agricultural work are not limited to tomatoes, which are agricultural produce, but may also include other vegetables, fruits, grains, etc. that form a group of objects (e.g., bunches) containing multiple objects.

[0016] The harvesting robot 2 automatically performs agricultural work based on agricultural work details preset by a user who manages the harvesting robot 2. The user sets the agricultural work details to be performed by the harvesting robot 2 via a mobile terminal (not shown) operated by the user. In this case, the user sends the setting information via the mobile terminal to a control PC (Personal Computer) 10 (see FIG. 2 ) provided in the harvesting robot 2.

[0017] In the following, the harvesting robot 2 is described as an agricultural work device that automatically performs agricultural work, but the user can manually operate the harvesting robot 2 as appropriate while it is operating automatically.

[0018] The user sets, as the content of the agricultural work, for example, the scope of the harvesting work (e.g., at least a partial area within the farm) of the target of the agricultural work (e.g., tomatoes). The user sets, for example, the movement path of the harvest robot 2 (e.g., the path along which the harvest robot 2 moves within the farm, as well as the start point in lane R where the harvest robot 2 begins agricultural work and the end point in lane R where the harvest robot 2 ends agricultural work). The user may also set a direction change (switching) point in lane R for the harvest robot 2, for example, the end of lane R.

[0019] The harvesting robot 2 moves along a route on a predetermined lane R within the farm. For example, the harvesting robot 2 may stop after moving, for example, several tens of centimeters, confirm the position of the tomato bunch, and then perform the harvesting operation. Once the harvesting operation is completed, the harvesting robot 2 may move again along the route for several tens of centimeters and perform the harvesting operation on another tomato bunch, and repeat this process.

[0020] 2 is a diagram showing an example of the block configuration of a harvesting robot according to an embodiment. As shown in FIG. 2, the harvesting robot 2 includes, for example, a control PC 10, a sensor group including multiple sensors, and an actuator group including multiple actuators. The control PC 10 is, for example, a personal computer, and controls each component so that the harvesting robot 2 can perform agricultural work.

[0021] For example, the sensor group includes a lane end detector 11 that detects the lane end of lane R shown in Figure 1, an operation pad 12 that accepts user input, a harvesting camera 13 for capturing images of the harvest target, and a lane camera 14 for capturing images of lane R.

[0022] The lane edge detector 11 transmits, for example, information regarding whether a lane edge has been detected to the control PC 10. The lane edge detector 11 can detect, for example, the lane edge of lane R shown in FIG. 1 . The lane edge detector 11 may use any method for detecting lane edges. The lane edge detector 11 may detect that the current position is at the lane edge by detecting whether or not the vehicle has come into contact with a structure that indicates the lane edge and is located at the lane edge. The lane edge detector 11 may also detect that the current position is at the lane edge by recognizing code information, such as a QR code, located at the lane edge using a camera or the like. The lane edge detector 11 may be, for example, a proximity sensor. In this case, for example, a transmitter that transmits information indicating the lane edge may be located at the lane edge, and the lane edge detector 11 may detect the lane edge by receiving the information from the transmitter.

[0023] The operation pad 12 transmits, for example, information regarding manual movement instructions and start / stop instructions for the harvesting robot 2 input by the user to the control PC 10.

[0024] Harvesting camera 13 transmits, for example, at least one of an RGB image and a depth image acquired by capturing an image of the surroundings including the group of subjects to control PC 10. Harvesting camera 13 has, for example, the function of at least one of a camera capable of generating an RGB image and a camera capable of generating a depth image.

[0025] The lane camera 14 transmits at least one of an RGB image and a depth image acquired by capturing an image of the surroundings including at least a part of the lane R to the control PC 10. The lane camera 14 has at least one of the functions of a camera capable of generating an RGB image and a camera capable of generating a depth image, for example.

[0026] For example, the actuator group includes a drive wheel motor 15 , an arm unit actuator 16 , a hand unit motor 17 , and a camera motor 18 .

[0027] The drive wheel motor 15 is, for example, a motor for controlling the drive of the drive wheels provided in the vehicle body 3 shown in Fig. 1. The drive wheel motor 15, for example, transmits information regarding the rotation amount of the drive wheels to the control PC 10 and receives information regarding the target speed of the drive wheels from the control PC 10.

[0028] The arm unit actuator 16 controls, for example, the arm unit 600 shown in Fig. 1. The arm unit actuator 16, for example, transmits information relating to the current position of the arm unit 600 to the control PC 10 and receives information relating to the target position of the arm unit 600 from the control PC 10. The arm unit actuator 16 may, for example, transmit information relating to the current orientation / current tilt of the arm unit 600 to the control PC 10 and receive information relating to the target orientation / target tilt of the arm unit 600 from the control PC 10.

[0029] 1 . The hand unit motor 17, for example, transmits information relating to the current position of the hand unit 1 to the control PC 10 and receives information relating to the target position of the hand unit 1 from the control PC 10. The hand unit motor 17 may also transmit information relating to the current orientation / current tilt of the hand unit 1 to the control PC 10 and receive information relating to the target orientation / target tilt of the hand unit 1 from the control PC 10.

[0030] The camera motor 18, for example, transmits information regarding the current position of at least one of the harvest camera 13 and the lane camera 14 to the control PC 10, and receives information regarding the target position of at least one of the harvest camera 13 and the lane camera 14 from the control PC 10. The camera motor 18 may, for example, transmit information regarding the current orientation / current tilt of at least one of the harvest camera 13 and the lane camera 14 to the control PC 10, and receive information regarding the target orientation / target tilt of at least one of the harvest camera 13 and the lane camera 14 from the control PC 10.

[0031] 3 is a diagram showing an example of the functional configuration of a control PC according to an embodiment. The control PC 10 includes, for example, a central processing unit (CPU) that controls the various functions required for the harvesting robot 2 to perform agricultural work. The central processing unit is, for example, a CPU or MPU, and operates according to a program stored in the storage unit 30.

[0032] For example, the control PC 10 (central processing unit) functionally includes a target position estimation unit 21, an arm unit control unit 23 (arm device control unit), a hand unit control unit 25 (hand device control unit), and an estimation unit 27.

[0033] The target position estimation unit 21 estimates the position of the tomato bunches based on image information obtained by capturing images of the tomato bunches (target group) using the harvesting camera 13 (imaging device) shown in Figures 1 and 2, for example.

[0034] The arm unit control unit 23 moves the arm unit 600 shown in FIG. 1 relative to the tomato bunch, for example, based on the position of the tomato bunch estimated by the target position estimation unit 21 .

[0035] 1 to perform harvesting work (agricultural work) on the tomato bunch at the position based on, for example, the position of the tomato bunch estimated by the target position estimation unit 21. The hand unit control unit 25 adjusts the orientation or inclination of the hand unit 1 according to the orientation or inclination of the tomato bunch. The hand unit control unit 25 controls the hand unit 1 so that the hand unit 1 applies frictional force to the tomato bunch. The hand unit control unit 25 controls multiple belt units (e.g., right belt 101 and left belt 201) so that the tomato bunch is twisted while being sandwiched between them.

[0036] The estimation unit 27 estimates the possibility of successful harvesting of the tomato bunches, for example, based on image information obtained by the harvesting camera 13 shown in Figures 1 and 2 capturing an image of a specified range including the tomato bunches and the shape of the hand unit 1.

[0037] Here, the harvesting robot 2 is, for example, a harvesting device for harvesting fruit. For example, as shown in FIG. 1 , the harvesting robot 2 includes an arm unit 600 and a hand unit 1, and the hand unit 1 includes a belt. During harvesting, the harvesting robot 2 harvests fruit by contacting the belt with the fruit. Here, for example, the features of the hand unit 1 of the harvesting robot 2 are as follows.

[0038] (1) The belts provided in the hand unit 1 harvest fruit by rotating it. (2) Multiple belts (e.g., right and left belts) are arranged facing each other so as to sandwich the fruit. (3) The multiple belts arranged facing each other harvest fruit by moving in opposite directions during harvesting. (4) The width of the belts is configured to be wide enough to cover at least one fruit. (5) The opening width of the multiple belts of the hand unit 1 is adjustable. (5-1) The opening width can be adjusted using a slide rail (slide rail 510 shown in Figure 10). (5-2) Only the belt unit is exposed on the side of the slide rail that faces the harvested fruit. (5-3) The belt drive unit is located on the arm unit 600 side of the slide rail. (6) The belt is arranged at an angle. (6-1) The belt is arranged at an angle relative to the direction of the fruit stems. (6-2) A gutter (e.g., half pipe 252 shown in Figure 14, etc.) is located below the tilted belt unit. (6-3) One side of the gutter is convex, and the other is open. (6-4) A stopper is placed at the open end of the gutter. (6-5) Fruit harvested by the belt falls into the gutter and is stored in the harvesting device (harvesting box 5 shown in Figure 1). (7) The tip roller of the belt is the same diameter or smaller than the other rollers. (7-1) The belt width increases from the tip roller toward the base roller. (7-2) The tip of the gutter is positioned closer to the fruit than the tip roller. (8) The fruit-side belt of hand unit 1 protrudes more than the other belts. (8-1) The cover of hand unit 1 is positioned in a lowered position.

[0039] 4 to 16, the specific configuration of the hand unit 1 of the harvesting robot 2 having at least the above-mentioned features (1) to (8) will be described.

[0040] Fig. 4 is a diagram showing an example of a hand unit according to an embodiment. As shown in Fig. 4, the hand unit 1 is connected to an arm unit 600. The hand unit 1 includes, for example, a right belt unit 100 including a right motor unit 300, a left belt unit 200 including a left motor unit 400, and a hand opening / closing unit 500 that controls the opening and closing of the right belt unit 100 and the left belt unit 200. The left belt unit 200 includes, for example, a transport unit 250.

[0041] An example of the configuration of the right belt unit will be described with reference to Figs. 5 to 7. Fig. 5 is a diagram showing an example of the right belt unit according to the embodiment. Fig. 6 is a diagram showing an example of a cross-section of the right belt unit 100 shown in Fig. 5. Fig. 7 is a diagram of the right belt unit 100 shown in Fig. 5 with the upper right cover 102 and the lower right cover 103 removed.

[0042] As shown in Fig. 5, the right belt unit 100 includes, for example, a right belt 101, an upper right cover 102, and a lower right cover 103. As shown in Fig. 6, the right belt unit 100 includes, for example, a crosspiece guide 106 for preventing the right belt 101 from meandering, a tip rotation shaft 110 for rotating an idler 109, the idler 109 that rotates by receiving power from the right belt 101, a tensioner unit 108 for adjusting slack, etc., of the right belt 101, a drive pulley 105 for driving the right belt 101, a drive timing pulley 104 for the right belt 101, and a drive shaft 107. As shown in Figure 7, when the upper right cover 102 and the lower right cover 103 are removed, the right belt unit 100 includes, for example, a right belt 101, a tensioner unit 108, a drive timing pulley 104, a tensioner holder 111, a tensioner shaft 112, an upper right holder 113 (plate-shaped member), and a lower right holder 114 (plate-shaped member).

[0043] The left belt unit has the same configuration as the right belt unit except that it is connected to the conveying unit 250. For example, the left belt unit has the same configuration as the right belt unit described with reference to FIGS.

[0044] Fig. 8 is a perspective view of the hand unit 1 shown in Fig. 4 with the right belt unit 100 and left belt unit 200 removed (not shown). As shown in Fig. 8, the arm unit 600 holds the hand opening / closing unit 500 via components such as a yaw motor 601 and a yaw horn metal plate 602. For example, the hand opening / closing unit 500 holds the slide rail 510 via a right rail holder 505 shown in Fig. 10 and left rail holder A 506 and left rail holder B 507 shown in Fig. 11.

[0045] The slide rail 510 restricts movement in the up and down direction (the short side direction of the slide rail 510), and also allows the right rail block 511 and the left rail block 512 shown in Fig. 10 to slide in the left and right directions (slide rail 510). The right rail block 511 holds the right belt unit 100 and the right motor unit 300, and the left rail block 512 holds the left belt unit 200 and the left motor unit 400, so they can be operated in conjunction with each other.

[0046] Figure 9 is a perspective view of the right belt unit 100 in the case where a part of the upper right cover 102 of the right motor unit 300 and the right motor cover 301 in Figure 8 are not shown. As described above, the right motor unit 300 is held by the right motor holder 302 via the right rail block 511 and the right belt block 520 shown in Figure 10 and via bolts not shown. The right motor holder 302 holds the right motor 303, and a drive timing pulley 304 is attached to the tip end.

[0047] When the right motor 303 operates, the drive timing pulley 304 rotates in conjunction with the right motor 303. The teeth of the timing belt 305 mesh with the teeth of the drive timing pulley 304, and the timing belt 305 rotates in synchronization with the drive timing pulley 304. The other teeth of the timing belt 305 mesh with the drive timing pulley 104 of the right belt unit 100, and the timing belt 305 rotates in synchronization with the drive timing pulley 104.

[0048] The drive timing pulley 104 is fixed to a drive shaft 107, and the drive shaft 107 is fixed to a drive pulley 105. Therefore, when the right motor 303 operates, the drive pulley 105 rotates, and the frictional force of the drive pulley 105 rotates the right belt 101. The left motor unit 400 operates in a similar manner, which can rotate the left belt 201 of the left belt unit 200.

[0049] 10 is a perspective view of the hand unit 1 in a case where the right motor unit 300 and the left motor unit 400 are not shown in FIG. 4. The hand opening / closing motor 501 of the hand opening / closing unit 500 is held by a hand opening / closing link A 502 via bolts (not shown). The hand opening / closing link A 502 is rotatable in accordance with the rotation of the hand opening / closing motor 501. The hand opening / closing link A 502 is fastened together with the right rail block 511 and right belt block 520 and the left rail block 512 and left belt block 530 via hand opening / closing links B 503 and C 504, respectively. The tip end of the hand opening / closing link A 502, both ends of the hand opening / closing link B 503, and the hand opening / closing link C 504 are rotatably connected by bearing connections.

[0050] The opening and closing operations of the right belt unit 100 and the left belt unit 200 of the hand unit 1 are now described. First, when the hand opening and closing motor 501 is energized via a harness (not shown) and the rotating shaft rotates counterclockwise, the hand opening and closing link A 502 also rotates counterclockwise. At the same time, the hand opening and closing link B 503 and the hand opening and closing link B 504 on the right belt unit 100 side move toward the left belt unit 200 side (left side), and the hand opening and closing link B 503 and the hand opening and closing link C 504 on the left belt unit 200 side move toward the right belt unit 100 side (right side). In conjunction with this movement, the right rail block 511 moves leftward along the slide rail 510, and the left rail block 512 moves rightward along the slide rail 510, thereby moving the right belt unit 100 and the left belt unit 200 closer to each other. This causes the hand unit 1 to perform a closing operation.

[0051] On the other hand, when the hand opening / closing motor 501 is operated to rotate the hand opening / closing link A 502 clockwise, the right belt unit 100 and the left belt unit 200 move apart, and the hand unit 1 opens.

[0052] The opening and closing operation of the hand unit 1 will be described with reference to Figures 11 and 12. Figure 11 is a perspective view showing a state in which both belts of the hand unit 1 are open. As described with reference to Figure 10, the hand unit 1 opens when the hand opening / closing motor 501 is rotated clockwise. The right belt unit 100 and the right motor unit 300 are fixed to the right rail block 511, and the left belt unit 200 and the left motor unit 400 are fixed to the left rail block 512. Therefore, the right belt unit 100 and the right motor unit 300 move toward the right belt unit 100 (right side), and the left belt unit 200 and the left motor unit 400 move toward the left belt unit 200 (left side).

[0053] Figure 12 is a perspective view showing a state in which both belts of the hand unit 1 are closed. As described with reference to Figure 10, when the hand opening / closing motor 501 is rotated counterclockwise, the hand unit 1 performs a closing operation. In accordance with the movement of the right rail block 511 and the left rail block 512, the right belt unit 100 and the right motor unit 300 move toward the right belt unit 100 side (right side), and the left belt unit 200 and the left motor unit 400 move toward the left belt unit 200 side (left side).

[0054] FIG. 13 is a top view of the right belt unit 100 and the left belt unit 200 operating with the hand unit 1 closed. The harvesting robot 2 clamps the harvested crop by closing the hand unit 1, and then operates the right belt unit 100 and the left belt unit 200 to separate the harvested crop from the bunch, performing the harvesting operation. As shown in FIG. 9 , operating the right motor 303 of the right motor unit 300 clockwise rotates the right belt 101 clockwise. On the other hand, operating the left motor 404 of the left motor unit 400 counterclockwise rotates the left belt 201 counterclockwise. At this time, the rotation direction D1 of the right belt 101 and the rotation direction D2 of the left belt 201 are different (the right belt 101 and the left belt 201 operate in different directions), and due to the frictional force generated by the right belt 101 and the left belt 201, the harvested crop is twisted and separated from the bunch.

[0055] In this case, the right belt unit 100 and the left belt unit 200 are each formed so that the thickness of their tip portions gradually decreases from the thickness of their base portions, making them easier to enter narrow spaces during harvesting operations. The right belt unit 100 and the left belt unit 200 are also offset, for example, by α (α is an arbitrary length). Therefore, when the hand unit 1 is closed, a sufficient distance is ensured between the tip rotation shaft 110 of the right belt unit 100 (see FIG. 6 ) and the tip rotation shaft (not shown) of the left belt unit 200. This prevents soft harvested crops from being crushed.

[0056] Fig. 14 is a side view when the right belt unit 100 and the left belt unit 200 are operated with the hand unit 1 closed. As shown in Fig. 14, the height of the upper left cover 202 of the left belt unit 200 is formed to be higher than the height of the upper right cover 102 of the right belt unit 100 by, for example, β (β is an arbitrary length). Because the hand unit 1 is disposed at an angle with respect to the arm unit 600 as shown in Fig. 15, the above-described configuration makes it possible to prevent harvested products from flying out from the upper left cover 202 side of the left hand unit 200 due to the rotational force of the right belt 101 and the left belt 201.

[0057] FIG. 15 is a diagram illustrating an example of a left belt unit and a conveying unit according to an embodiment. As shown in FIG. 15 , the left belt unit 200 includes, for example, a conveying unit 250 (conveying member) for conveying at least some of the tomatoes (targets) to be separated from the tomato bunch (target group) and adjusting the drop position of the tomatoes. The conveying unit 250 is disposed on the left belt unit 200 with a direction or inclination corresponding to the drop position of the tomatoes. The conveying unit 250 includes, for example, a half pipe 252, a pipe holder 251 for holding the half pipe 252, a pipe end cover 253, and a conveying guide 254 (correcting member) such as an oblique brush for correcting the drop trajectory of the tomatoes. The drop trajectory of at least some of the tomatoes separated from the tomato bunch is corrected by the conveying guide 254 and stored in the harvest box 5 shown in FIG. 1 . As described above, the left belt unit 200 includes the conveying unit 250. However, the right belt unit 100 may include the conveying unit 250 instead of the left belt unit 200.

[0058] Fig. 16 is a diagram showing another example of the left belt unit and the conveying unit according to the embodiment. As shown in Fig. 16, the conveying unit 250 may include, for example, a tray 255 (storage unit) that is a dish-shaped part that stores at least some of the tomatoes separated from the tomato bunches, instead of the conveying guide 254 shown in Fig. 15. At least some of the tomatoes separated from the tomato bunches may be stored in the harvest box 5 shown in Fig. 1 via the tray 255, or may be stored in the harvest box 5 without passing through the tray 255. Note that the tray 255 may be connected to the conveying unit 250 together with the conveying guide 254.

[0059] Fig. 17 is a flowchart showing an example of a process flow for performing a harvesting operation by a harvesting robot according to an embodiment. As shown in Fig. 17 , the harvesting robot 2 (the control PC 10 shown in Figs. 2 and 3 ) estimates the position of a tomato bunch based on image information obtained by capturing an image of the tomato bunch with the harvesting camera 13 shown in Figs. 1 and 2 (step S1). The harvesting robot 2 moves the arm unit 600 shown in Fig. 1 relative to the tomato bunch based on the estimated position of the tomato bunch (step S3). The harvesting robot 2 controls the hand unit 1 shown in Fig. 1 to perform the harvesting operation on the tomato bunch at that position based on the estimated position of the tomato bunch (step S5).

[0060] 18 is a flowchart showing another example of the information processing flow of the harvesting robot according to the embodiment. As shown in FIG. 18, the harvesting robot 2 shown in FIG. 1 captures an image of the surroundings of the harvesting robot 2 with the harvesting camera 13 shown in FIGS. 1 and 2 (step S11). The harvesting robot 2 determines whether the target group is included in the captured image (step S13). If the target group is not included in the captured image (No), the process returns to step S11.

[0061] On the other hand, if the group of objects is included in the imaging range (Yes), the harvesting robot 2 determines whether or not a certain number of objects can be harvested (step S15). If the group of objects cannot be harvested (No), the process returns to step S11 to search for other groups of objects. Note that if multiple groups of objects are detected in step S13, it may be possible to determine whether or not a certain number of objects can be harvested for a group of objects different from the group for which the determination was made earlier as to whether or not a certain number of objects can be harvested. If the group of objects can be harvested (Yes in step S15), at least some of the objects are harvested from the group of objects (step S17).

[0062] The processing of steps S13 and S15 will be described in detail below. In order to pick as many fruits as possible in one approach, the harvesting robot 2 recognizes the position of the harvest target under the following conditions (1) to (3).

[0063] (1) A certain number or more of red fruits to be harvested are present in a cluster around the target position. (2) There are few unripe fruits in the grip range around the target position that the hand unit 1 approaches. (3) There are few obstacles around the target position, and it is possible to expect to harvest a certain number or more of fruits when approaching. Here, if at least one of conditions (1), (2), and (3) is met, it is estimated that there is a high possibility that the harvesting operation for the target group will be successful. For example, if all of conditions (1), (2), and (3) are met, it is estimated that there is a high possibility that the harvesting operation for the target group will be successful.

[0064] First, regarding condition (1), the harvesting robot 2 estimates the position of the fruit clusters by filtering the colored point cloud data acquired by the harvesting camera 13 shown in Figures 1 and 2 by color (e.g., red) and performing clustering processing. The harvesting robot 2 estimates the number of fruits based on the size of the processed clusters. The harvesting robot 2 may adjust the target red color range based on user specification.

[0065] Regarding condition (2), the harvesting robot 2 filters the colored point cloud data with a color different from red (e.g., green) and estimates the number of unripe fruits present within the gripping range from the position of the point cloud. Instead of or in addition to this estimation method, the harvesting robot 2 may employ an estimation method using a learning model, as described below. The gripping range is set according to the shape of the hand unit 1 (e.g., the shape of the right belt unit 100 and the left belt unit 200). For example, the width (height) of the target to be harvested can be varied depending on the design of the belt width (height). If a hand unit with a wide belt width (capable of picking many at once) were to wrap around and pick the entire bunch of tomatoes, there is a risk of picking unripe fruits (e.g., green fruits). Therefore, it is desirable to set the harvesting robot 2 to harvest fruits that are ripe from top to bottom. This is because tomatoes tend to turn red from the top of the bunch, while the bottom tends to be unripe. For example, the harvesting robot 2 preferably searches for an area where four or more red fruits are estimated to be present and performs the harvesting operation. The set number of clusters included in this area ("four or more" in the above) (cluster size) is variable and can be adjusted as appropriate.

[0066] Regarding condition (3), the harvesting robot 2 filters the colored point cloud data with a color different from red (e.g., green) and estimates the number of obstacles (e.g., stems and leaves) present within the gripping range from the positions of the point cloud. Instead of or in addition to this estimation method, the harvesting robot 2 may employ an estimation method using a learning model, which will be described later.

[0067] The harvesting robot 2 may further improve the efficiency of its harvest target search time by employing the following method. For example, in addition to the harvesting camera 13 shown in FIGS. 1 and 2 , the harvesting robot 2 may be equipped with an RGB camera that views an area a certain distance ahead of the harvesting camera 13. The harvesting robot 2 uses this RGB camera to recognize in advance areas where potential harvest targets may exist. The harvesting robot 2 may stop traveling at a point where the potential harvest targets can be harvested, and perform recognition and estimation processing using the harvesting camera 13.

[0068] The following describes in more detail a method for estimating unripe fruit and / or obstacles using a learning model. For example, (1) the harvesting robot 2 estimates the expected number of fruit to be harvested from RGB images and depth images of the area around the target harvesting location. (2) The harvesting robot 2 also estimates the degree of entrapment of unripe fruit in the same way as in (1) and penalizes the expected value.

[0069] (3) The harvesting robot 2 places a certain number of approach reference points around the central position of the recognized harvest target (group of targets). The harvesting robot 2 performs estimation (1) and / or (2) for each approach reference point. The harvesting robot 2 adopts the approach reference point that is best for each point and that has an evaluation that exceeds a certain standard as the approach position.

[0070] (4) Training data can be collected during automatic harvesting as follows: The harvesting robot 2 saves a pair of RGB images of the area around the target position, one before and one after the hand's approach. The harvesting robot 2 calculates the actual number of harvested crops from the difference between the pair of images before and after the approach. The harvesting robot 2 collects the actual number of harvested crops for the image before the approach as correct answer data. The harvesting robot 2 updates the learning model each time it performs a certain number of automatic harvests. Note that with regard to the hardware configuration of the harvesting robot 2, since a decision on whether or not to harvest must be made immediately to ensure a certain harvesting speed, the recognition and estimation processes may be performed within the harvesting robot 2 rather than on a server device or the like separate from the harvesting robot 2.

[0071] Fig. 19 is a flowchart showing an example of a movement process for a harvesting robot to move along a lane. As shown in Fig. 19, the harvesting robot 2 shown in Fig. 1 determines whether the lane edge of the outbound lane R has been detected by the lane edge detector 11 shown in Fig. 2 (step S21). If the lane edge of the outbound lane is not detected in step S21 (No), the harvesting robot 2 resumes movement along lane R. If the lane edge of the outbound lane is detected in step S21 (Yes), the harvesting robot 2 switches direction (changes direction) on lane R and resumes movement (step S23).

[0072] Next, the harvesting robot 2 determines whether the lane edge detector 11 has detected the lane edge of the return lane of lane R (step S25). If the lane edge of the return lane is not detected in step S25 (No), the harvesting robot 2 resumes movement on lane R. If the lane edge of the return lane is detected in step S25 (Yes), the harvesting robot 2 determines that it has reached the end of lane R and ends the process.

[0073] The above-described embodiments are merely examples of the present invention in all respects. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. In other words, specific configurations according to the embodiments may be appropriately adopted when implementing the present invention. Furthermore, the above-described embodiments are intended to facilitate understanding of the present invention and are not intended to limit the present invention. The elements included in each embodiment, as well as their arrangement, materials, conditions, shape, size, and the like, are not limited to those exemplified and may be appropriately modified.

[0074] [Additional Notes] Aspects of this embodiment include the following disclosures.

[0075] (Supplementary Note 1) An agricultural work apparatus comprising: an arm apparatus equipped with a hand device that performs agricultural work on a group of objects including a plurality of objects; an imaging device that images the group of objects; an object position estimation unit that estimates the position of the group of objects based on image information obtained by the imaging device imaging the group of objects; an arm apparatus control unit that moves the arm apparatus relative to the group of objects based on the estimated position; and a hand apparatus control unit that controls the hand apparatus to perform the agricultural work on the group of objects at the estimated position based on the estimated position. (Supplementary Note 2) The agricultural work apparatus according to Supplementary Note 1, wherein the hand apparatus control unit adjusts the orientation or inclination of the hand apparatus according to the orientation or inclination of the group of objects. (Supplementary Note 3) The agricultural work apparatus according to Supplementary Note 1 or 2, wherein the hand apparatus includes a plate-shaped member and is configured so that the thickness of a tip of the hand apparatus is smaller than the thickness of a base of the hand apparatus. (Supplementary Note 4) The agricultural work apparatus according to any of Supplements 1 to 3, wherein the hand apparatus control unit controls the hand apparatus so that the hand apparatus applies a frictional force to the group of objects. (Supplementary Note 5) The agricultural work apparatus according to any one of Supplements 1 to 4, wherein the hand apparatus further includes a plurality of belt units, and the hand apparatus control unit controls the plurality of belt units so as to twist the group of objects while sandwiching the group of objects between them. (Supplementary Note 6) The agricultural work apparatus according to any one of Supplements 1 to 5, wherein the hand apparatus further includes a plurality of belt units, and wherein the movement direction of one of the plurality of belt units is different from the movement direction of the other belt units. (Supplementary Note 7) The agricultural work apparatus according to any one of Supplements 1 to 6, wherein the arm apparatus further includes a transport member for transporting at least some of the objects separated from the group of objects and adjusting the drop position of the objects. (Supplementary Note 8) The agricultural work apparatus according to Supplementary Note 7, wherein the transport member is arranged on the arm apparatus with an orientation or inclination corresponding to the drop position. (Supplementary Note 9) The agricultural work apparatus according to Supplementary Note 7, wherein a correction member for correcting the drop trajectory of the objects is arranged at the tip of the transport member.(Supplementary Note 10) The agricultural work apparatus according to any one of Supplements 1 to 9, wherein the arm device further comprises a storage unit that stores at least some of the objects separated from the group of objects. (Supplementary Note 11) The agricultural work apparatus according to any one of Supplements 1 to 10, further comprising an estimation unit that estimates the likelihood of success of work on the group of objects based on image information obtained by the imaging device capturing an image of a predetermined range including the group of objects and the shape of the hand device. (Supplementary Note 12) The agricultural work apparatus according to any one of Supplements 1 to 11, wherein the group of objects includes a bunch formed by a group of multiple objects. (Supplementary Note 13) An agricultural work method performed by an agricultural work apparatus including an arm device with a hand device that performs agricultural work on a group of objects including a plurality of objects, and an imaging device that images the group of objects, the agricultural work method including the steps of: estimating the position of the group of objects based on image information obtained by the imaging device imaging the group of objects; moving the arm device relative to the group of objects based on the estimated position; and controlling the hand device to perform the agricultural work on the group of objects at that position based on the estimated position. (Supplementary Note 14) A program for causing an agricultural work apparatus, which includes an arm apparatus equipped with a hand device for performing agricultural work on a group of objects including a plurality of objects, and an imaging device for imaging the group of objects, to function as: an object position estimation unit that estimates the position of the group of objects based on image information obtained by the imaging device imaging the group of objects; an arm apparatus control unit that moves the arm apparatus relative to the group of objects based on the estimated position; and a hand apparatus control unit that controls the hand apparatus to perform the agricultural work on the group of objects at the position based on the estimated position.

[0076] 1...Hand unit (hand device), 2...Harvesting robot (agricultural work device), 13...Harvesting camera (imaging device), 21...Target position estimation unit, 23...Arm unit control unit (arm device control unit), 25...Hand unit control unit (hand device control unit), 27...Estimation unit, 101...Right belt (belt portion), 113...Upper right holder (plate-shaped member), 114...Lower right holder (plate-shaped member), 201...Left belt (belt portion), 250...Transport unit (transport member), 254...Transport guide (correction member), 255...Tray (storage unit), 600...Arm unit (arm device)

Claims

1. an arm device having a hand device that performs farm work on an object; an imaging device that images the object; an object position estimation unit that estimates a position of the object based on image information obtained by the imaging device capturing an image of the object; an arm device control unit that moves the arm device relative to the target based on the estimated position; a hand device control unit that controls the hand device to perform the agricultural work on the target at the estimated position based on the estimated position, agricultural equipment.

2. the hand device control unit adjusts the orientation or inclination of the hand device in accordance with the orientation or inclination of the object. The agricultural work implement according to claim 1.

3. The hand device includes a plate-like member, and is configured so that the thickness of a tip portion of the hand device is smaller than the thickness of a base portion of the hand device. The agricultural work implement according to claim 1.

4. the hand device control unit controls the hand device so that the hand device applies a friction force to the object. The agricultural work implement according to claim 1.

5. the hand device further includes a plurality of belt portions, the hand device control unit controls the plurality of belt units so that the plurality of belt units twist the object while sandwiching the object. The agricultural work implement according to claim 1.

6. the hand device further includes a plurality of belt portions, a moving direction of one of the plurality of belt portions is different from a moving direction of the other belt portions; The agricultural work implement according to claim 1.

7. the arm device further includes a transport member for transporting at least a portion of the object separated from the target and adjusting a drop position of the object; The agricultural work implement according to claim 1.

8. the conveying member is disposed on the arm device in a direction or inclination corresponding to the drop position; Agricultural work implement according to claim 7.

9. A correction member for correcting the falling trajectory of the target is disposed at the tip of the transport member. Agricultural work implement according to claim 7.

10. The arm device further includes a storage unit configured to store at least a portion of the object separated from the object. The agricultural work implement according to claim 1.

11. based on image information obtained by the imaging device capturing an image of a predetermined range including the target and the shape of the hand device, An estimation unit that estimates the success probability of the task on the target is further provided. The agricultural work implement according to claim 1.

12. The object includes a cluster formed by a group of multiple objects. The agricultural work implement according to claim 1.

13. An agricultural work method performed by an agricultural work apparatus including an arm device having a hand device that performs agricultural work on an object, and an imaging device that images the object, estimating a position of the object based on image information obtained by the imaging device capturing an image of the object; moving the arm device relative to the object based on the estimated position; and controlling the hand device to perform the agricultural task on the object at the estimated position based on the estimated position. Farming methods.

14. An agricultural work apparatus including an arm device having a hand device for performing agricultural work on an object, and an imaging device for imaging the object, an object position estimation unit that estimates a position of the object based on image information obtained by the imaging device capturing an image of the object; an arm device control unit that moves the arm device relative to the target based on the estimated position; a hand device control unit that controls the hand device to perform the agricultural work on the target at the estimated position based on the estimated position; A program to function as a