Harvesting device and control method of harvesting device
The harvesting device enhances efficiency by using a robot with precise part recognition and manipulation controls to optimize agricultural product separation, addressing inefficiencies in existing technologies.
Patent Information
- Application Number
- JP2023204241
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-18
- Filing Date
- 2023-12-01
- Publication Date
- 2024-11-28
AI Technical Summary
Existing harvesting devices and methods lack efficiency in accurately identifying and handling specific parts of agricultural products, leading to suboptimal harvesting operations.
A harvesting device equipped with a robot and an end effector, along with control units for precise part recognition and manipulation, allows for differentiated harvesting operations based on whether specific parts are recognized, enabling efficient separation of agricultural products.
The device improves work efficiency by stabilizing harvesting operations and reducing the risk of damage to surrounding crops, while ensuring reliable separation of products.
Smart Images

Figure 2024166044000001_ABST
Abstract
Description
[Technical field]
[0001] The present disclosure relates to a harvesting device and a method for controlling a harvesting device. [Background technology]
[0002] Patent Document 1 discloses a fruit and vegetable harvesting device. Patent Document 2 discloses a fruit and vegetable separation device. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] Japanese Patent Application Publication No. 8-103138 [Patent Document 2] Patent Publication No. 2021-23164 Summary of the Invention [Problem to be solved by the invention]
[0004] The present disclosure provides a harvesting device and a control method for the harvesting device that are useful for improving work efficiency. [Means for solving the problem]
[0005] A harvesting apparatus according to one aspect of the present disclosure includes a robot having an end effector, a first approach control unit that controls the robot to bring the end effector closer to a first portion of a harvest target crop, a second approach control unit that controls the robot to move the end effector from the first portion to a second portion of the harvest target crop and have the end effector hold the second portion, and a harvest control unit that controls the robot to harvest the harvest target crop while the end effector holds the second portion.
[0006] A harvesting device according to one aspect of the present disclosure includes a robot, a target detection unit that detects harvest objects in the crop from an image obtained by imaging at least a portion of the crop, a recognition unit that recognizes a specific part of the detected harvest object, and a harvest control unit that controls the robot to harvest the harvest object by different operations depending on whether or not the specific part is recognized.
[0007] A harvesting device according to one aspect of the present disclosure includes a robot, a target detection unit that detects harvest objects in the crop from an image obtained by imaging at least a portion of the crop, a recognition unit that recognizes a specific part of the detected harvest object, a harvest control unit that controls the robot to harvest the harvest object when the specific part is recognized, and an information notification unit that transmits individual information of the harvest object to another device when the specific part is not recognized.
[0008] A method for controlling a harvesting device according to one aspect of the present disclosure includes controlling the robot to bring an end effector held by the robot close to a first portion of a crop to be harvested, controlling the robot to move the end effector from the first portion to a second portion of the crop to be harvested so that the end effector holds the second portion, and controlling the robot to harvest the crop with the end effector holding the second portion.
[0009] A method for controlling a harvesting device according to one aspect of the present disclosure includes detecting a harvest object in the crop from an image obtained by imaging at least a portion of the crop, recognizing a specific portion of the detected harvest object, and controlling a robot to harvest the harvest object by different operations depending on whether or not the specific portion is recognized. Effect of the Invention
[0010] According to the present disclosure, a harvesting device and a method for controlling the harvesting device are provided that are useful for improving work efficiency. [Brief description of the drawings]
[0011] [Figure 1] FIG. 1 is a schematic diagram showing an example of a harvesting device. [Diagram 2] FIG. 2 is a top view illustrating an example of a harvesting operation performed by the harvesting device. [Diagram 3] FIG. 3 is a top view diagrammatically illustrating an example of an end effector. [Figure 4] FIG. 4 is a side view diagrammatically illustrating an example of an end effector. [Diagram 5] FIG. 5 is a block diagram illustrating an example of a functional configuration of the controller. [Figure 6] FIG. 6(a), FIG. 6(b), and FIG. 6(c) show examples of captured images. [Figure 7] FIG. 7 is a block diagram showing an example of a hardware configuration of the controller. [Figure 8] FIG. 8 is a flowchart showing an example of a series of processes executed by the controller. [Figure 9] FIG. 9 is a flowchart illustrating an example of a series of processes executed by the controller. [Figure 10] 10(a) and 10(b) are schematic diagrams illustrating an example of an operation when approaching one end of a harvest target and harvesting it. [Figure 11] 11(a) and 11(b) are schematic diagrams illustrating an example of an operation when approaching and harvesting one end of a harvest target. [Figure 12] 12(a) and 12(b) are schematic diagrams illustrating an example of the operation when approaching and harvesting the other end of the harvest target. [Figure 13] 13(a) and 13(b) are schematic diagrams illustrating an example of the operation when approaching and harvesting the other end of the harvest target. [Figure 14] FIG. 14 is a side view diagrammatically illustrating an example of an end effector. [Figure 15] FIG. 15 is a top view diagrammatically illustrating an example of an end effector. [Figure 16]FIG. 16 is a block diagram showing an example of a functional configuration of the controller. [Figure 17] FIG. 17 is a flowchart showing an example of a series of processes executed by the controller. [Figure 18] Fig. 18(a) is a schematic diagram illustrating the state of the end effector before approaching the harvesting target, and Fig. 18(b) shows an example of an image used to identify the approach site. [Figure 19] FIG. 19 is a flowchart showing an example of a series of processes executed in the first approach control. [Figure 20] 20(a) and 20(b) are schematic diagrams illustrating the operation in the first approach control. [Figure 21] FIG. 21 is a flowchart showing an example of a series of processes executed in the second approach control. [Figure 22] Fig. 22(a) is a schematic diagram illustrating an example of the operation in the second approach control, and Fig. 22(b) shows an example of an image used for pattern recognition. [Figure 23] Figures 23(a) and 23(b) are photographs illustrating the harvesting operations. [Figure 24] Figures 24(a) and 24(b) are photographs illustrating the harvesting operations. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0012] Hereinafter, an embodiment will be described with reference to the drawings. In the description, the same elements or elements having the same functions are denoted by the same reference numerals, and duplicated description will be omitted.
[0013] First Embodiment [Harvesting Equipment] An overview of the harvesting device according to the first embodiment and the agricultural crop to be harvested will first be described with reference to Figs. 1 and 2. The harvesting device 1 shown in Fig. 1 is a device (system) that automatically executes at least a part of the work for harvesting one or more harvest objects 210 from the agricultural crop 200. The agricultural crop 200 is a plant that bears fruit that can be harvested by the harvesting device 1, and the type is not limited. The harvesting device 1 harvests the fruit of the agricultural crop 200 as the harvest object. The agricultural crop 200 may be a fruit vegetable, a fruit-like vegetable, or a fruit tree. The agricultural crop 200 may be a type of vegetable of the Cucurbitaceae family, and in one example, it is a cucumber.
[0014] The fruit (harvested object) of the agricultural crop 200 may be edible. The harvested object 210 may be connected to a stem or branch of the agricultural crop 200 via a stalk 220 as shown in FIG. 1. The stalk 220 may be referred to as a stalk or a peduncle. The harvested object 210 may include an end connected to the stalk 220. The harvested object 210 may have a shape extending from the end connected to the stalk 220 in the opposite direction to the stalk 220. That is, the harvested object 210 may be formed in an elongated shape. The harvested object 210 having an elongated shape may extend in a straight line or in a curved shape. The harvested object 210 may be formed to extend in the opposite direction from the stalk 220. Hereinafter, the contents of the present disclosure will be described using an example in which the agricultural crop 200 is a cucumber.
[0015] The agricultural crop 200 includes, for example, a stem 202, one or more leaves 204, and one or more harvest objects 210. Each of the one or more harvest objects 210 is connected to the stem 202 via a stem 220. The harvest objects 210 hang down from the stem 202. The harvest objects 210 include an upper end 212a (first end) connected to the stem 220, and a lower end 212b (second end) opposite the upper end 212a. The upper end 212a is located higher than the lower end 212b.
[0016] 2, the crops 200 may be planted in a row along one horizontal direction. The crops 200 are planted, for example, in a greenhouse on a shelf 290 extending along one horizontal direction. The crops 200 may include a plurality of plants 201 (seedlings) lined up in a horizontal direction on the shelf 290. For each plant 201 included in the crops 200, there may be one or more harvest objects 210.
[0017] 1, the harvesting device 1 includes, for example, a robot 2, a cart 4, a storage unit 6, an imaging device 8, and a controller 100. Hereinafter, an example of each of the multiple devices constituting the harvesting device 1 will be described.
[0018] Robot 2 is a device that performs at least an operation of separating harvest target 210 from crop 200 based on an operation command from controller 100. Robot 2 is, for example, a six-axis vertical articulated robot, and includes articulated arm 10 and end effector 50. Articulated arm 10 changes the position and posture of end effector 50 by the operation of the articulated arm.
[0019] The articulated arm 10 has, for example, a base portion 11, a rotating portion 12, a first arm 13, a second arm 14, a wrist portion 17, and actuators 41, 42, 43, 44, 45, and .
[0020] The base 11 is placed on the cart 4. As a result, the entire robot 2 moves as the cart 4 travels. The rotating part 12 is provided on the base 11 so as to rotate about a vertical axis 21. The first arm 13 is connected to the rotating part 12 so as to rotate about an axis 22 that intersects (for example, perpendicular to) the axis 21, and extends in a direction away from the axis 22. In the present disclosure, the intersection also includes a case where there is a twisted relationship, such as a so-called three-dimensional intersection.
[0021] The second arm 14 is connected to the tip of the first arm 13 so as to rotate about an axis 23 that is substantially parallel to the axis 22, and extends in a direction away from the axis 23. The second arm 14 includes an arm base 15 and an arm end 16. The arm base 15 is connected to the tip of the first arm 13. The arm end 16 is connected to the tip of the arm base 15 so as to rotate about an axis 24 that intersects (e.g., is perpendicular to) the axis 23, and extends in a direction away from the arm base 15 along the axis 24.
[0022] Wrist 17 is connected to the tip of arm end 16 so as to rotate about axis 25 that intersects (e.g., perpendicular to) axis 24. An end effector 50 is provided at the tip of wrist 17 as a member to be operated, and end effector 50 is rotatable about axis 26 that intersects (e.g., perpendicular to) axis 25.
[0023] As described above, the multi-joint arm 10 has a joint 31 connecting the base 11 and the swivel 12, a joint 32 connecting the swivel 12 and the first arm 13, a joint 33 connecting the first arm 13 and the second arm 14, a joint 34 connecting the arm base 15 and the arm end 16 in the second arm 14, a joint 35 connecting the arm end 16 and the wrist 17, and a joint 36 connecting the wrist 17 and the end effector 50.
[0024] The actuators 41, 42, 43, 44, 45, and 46 include, for example, electric motors and reducers, and drive the joints 31, 32, 33, 34, 35, and 36, respectively. For example, the actuator 41 rotates the rotating part 12 about the axis 21, the actuator 42 rotates the first arm 13 about the axis 22, and the actuator 43 rotates the second arm 14 about the axis 23. In addition, the actuator 44 rotates the arm end part 16 about the axis 24, the actuator 45 rotates the wrist part 17 about the axis 25, and the actuator 46 rotates the end effector 50 about the axis 26.
[0025] The end effector 50 is a harvesting tool, and is provided at the tip of the articulated arm 10. The end effector 50 is connected to the tip of the articulated arm 10 in a state in which it can rotate around an axis 26 (first axis). The tip of the wrist 17 corresponds to the tip of the articulated arm 10. Details of the end effector 50 will be described later.
[0026] The configuration of the robot 2 (or the articulated arm 10) can be changed as appropriate. The robot 2 may be, for example, a seven-axis redundant robot in which one more joint is added to the six-axis configuration described above, or may be a so-called SCARA type articulated robot.
[0027] The cart 4 is a cart on which the robot 2 is mounted and which can run. The entire robot 2 moves as the cart 4 runs. As shown in FIG. 2, the cart 4 can run, for example, along the direction in which the multiple plants 201 are lined up. The cart 4 has a motor as a drive source for running, and may run on rails 5 extending in the direction in which the multiple plants 201 are lined up. As the cart 4 runs, the position of the base 11 of the robot 2 relative to the crops 200 changes. The cart 4 may operate to repeatedly run and stop based on an operation command from the controller 100.
[0028] The storage unit 6 is a member that stores one or more harvest objects 210 harvested by the robot 2. The storage unit 6 may be configured in any manner as long as it can store one or more harvest objects 210, and may be, for example, a container with an open top. The storage unit 6 is disposed within a range that the end effector 50 of the robot 2 can approach. The storage unit 6 may be mounted on the dolly 4. In one example, the storage unit 6 is mounted on the dolly 4 in a detachable state. When the storage unit 6 is mounted on the dolly 4, the storage unit 6 also moves as the dolly 4 travels. Instead of being mounted on the dolly 4, the storage unit 6 may be installed or disposed at a predetermined position that the end effector 50 of the robot 2 mounted on the dolly 4 can approach.
[0029] Imaging device 8 is a device capable of capturing an image of at least a portion of crop 200. Imaging device 8 captures an image of at least a portion of crop 200 to generate an image (hereinafter referred to as a "captured image PI") for detecting harvest target 210 of crop 200 that is to be harvested by robot 2. As shown in Fig. 2, in the direction in which cart 4 travels, a portion of the field of view (imaging range PR) that imaging device 8 can capture may overlap with cart 4, and the remaining portion may be located in front of cart 4.
[0030] The imaging device 8 has, for example, one or more cameras capable of capturing an image of the imaging range PR. The one or more cameras included in the imaging device 8 may capture an image of the imaging range PR using visible light. The imaging device 8 may have a distance sensor in addition to the one or more cameras. The distance sensor included in the imaging device 8 is, for example, a sensor capable of acquiring distance data (depth map) indicating the distance to each of multiple points on an object present in the imaging range PR. The imaging device 8 may be mounted on the dolly 4. The imaging device 8 is fixed to the dolly 4 via, for example, a fixing member 8a. When the imaging device 8 is mounted on the dolly 4, the imaging device 8 moves as the dolly 4 travels.
[0031] 3 and 4 show an example of the end effector 50. The end effector 50 is connected to the tip of the wrist 17 (the tip of the articulated arm 10) in a state in which it can rotate around the axis 26. In the example shown in FIG. 3 and FIG. 4, the end effector 50 is connected to the tip of the wrist 17 so that the direction in which the end effector 50 extends is along the axis 26. Hereinafter, for convenience of explanation, the direction in which the wrist 17 and the end effector 50 are aligned in the example shown in FIG. 3 and FIG. 4 is referred to as "direction D1". In addition, the direction (direction) from the wrist 17 toward the end effector 50 is referred to as "forward", and the direction (direction) from the end effector 50 toward the wrist 17 is referred to as "rear".
[0032] Direction D1 coincides with the direction in which axis 26 extends, and wrist 17 and end effector 50 face each other in direction D1. End effector 50 may be formed to extend in direction D1 as a whole. End effector 50 shown in Fig. 3 and Fig. 4 has a function of supporting (holding) harvesting object 210 and a function of cutting handle 220 connected to harvesting object 210.
[0033] The end effector 50 includes a main body 60 and a support 80. The main body 60 is connected to the tip of the wrist 17 in a state that is rotatable about an axis 26. When the main body 60 rotates about the axis 26, the support 80 also rotates about the axis 26.
[0034] The main body 60 has a base 62, a first movable part 64a, a second movable part 64b, a drive part 66, and a cutting part 70. The base 62 is rotatably connected to the tip of the wrist 17, and is a part that supports other members included in the main body 60. When the base 62 rotates about the axis 26, the entire main body 60 rotates about the axis 26.
[0035] The first movable part 64a and the second movable part 64b are parts that support the support part 80. The first movable part 64a and the second movable part 64b are arranged at an interval from each other in one direction (hereinafter referred to as "direction D2") perpendicular to the direction D1. In one example, the first movable part 64a and the second movable part 64b are each formed in a plate shape so as to extend in the direction D1. The first movable part 64a is provided on the base part 62 in a state capable of moving in the direction D2, and the second movable part 64b is provided on the base part 62 in a state capable of moving in the direction D2.
[0036] The drive unit 66 is a part that drives each of the first movable part 64a and the second movable part 64b in the direction D2. The drive unit 66 includes a drive source such as an electric motor. The drive unit 66 drives each of the first movable part 64a and the second movable part 64b based on an operation instruction from the controller 100. The distance between the first movable part 64a and the second movable part 64b changes due to the drive by the drive unit 66.
[0037] The cutting part 70 is a part capable of cutting the vicinity of the part of the harvest object 210 supported by the support part 80. The cutting part 70 is capable of cutting the handle 220 connected to the upper end part 212a (see FIG. 1 ) of the harvest object 210 in a state in which the support part 80 supports the upper end part 212a.
[0038] The cutting unit 70 includes, for example, a blade 72, a cover member 74, and a drive unit 76. As shown in Fig. 4, the blade 72 includes a sharp cutting edge 72a facing forward. The blade 72 is provided on the base unit 62 in a state in which it can move in the direction D1. The portion of the blade 72 including the cutting edge 72a may be formed in a plate shape so as to extend along the direction D1 and the direction D2.
[0039] The blade 72 is disposed at a position away from the first movable part 64a and the second movable part 64b in a direction (hereinafter referred to as "direction D3") perpendicular to the directions D1 and D2. When the axis 26 is disposed horizontally and the rotation position of the end effector 50 (main body part 60) about the axis 26 is in the initial position, for example, the blade 72 is located above the first movable part 64a and the second movable part 64b. Hereinafter, the end effector 50 will be described based on the case where the axis 26 is disposed horizontally and the rotation position of the end effector 50 (main body part 60) about the axis 26 is in the initial position.
[0040] The cover member 74 is a member that covers the blade 72. The cover member 74 covers the cutting edge 72a of the blade 72. The cover member 74 is fixed to the base portion 62. That is, as the blade 72 moves, the position of the blade 72 relative to the cover member 74 changes. The cover member 74 is disposed above the blade 72. The cover member 74 may be formed in a plate shape extending along the direction D1 and the direction D2.
[0041] The drive unit 76 is a part that drives the blade 72 in the direction D1. The drive unit 76 includes a drive source such as an electric motor. The drive unit 76 drives the blade 72 in the direction D1 based on an operation instruction from the controller 100. The drive by the drive unit 76 changes the relative position of the blade 72 with respect to the cover member 74. The drive unit 76 may drive the blade 72 so as to switch between a state in which the blade 72 is covered by the cover member 74 and a state in which the blade 72 (the blade tip 72a) is exposed from the cover member 74.
[0042] When the blade 72 is covered by the cover member 74, the cover member 74 overlaps the blade 72 when viewed vertically from above, and the blade 72 is not exposed. In this state, the cutting edge 72a of the blade 72 is located rearward compared to the tip of the cover member 74 which is located forward. When the blade 72 is exposed from the cover member 74, a part of the blade 72 including the cutting edge 72a is located forward of the tip of the cover member 74 when viewed from the direction D3. Figures 3 and 4 show an example of the blade 72 exposed from the cover member 74.
[0043] The support part 80 is a part capable of supporting the harvest object 210. The support part 80 is connected to the main body part 60 in a state in which it can rotate around an axis 88 (second axis) that intersects with the axis 26. The axis 88 may be perpendicular to the axis 26. The axis 88 may be set so as to pass through the first movable part 64a and the tip part located in the front of the first movable part 64a. For example, the support part 80 grasps (holds in a clamped state) the harvest object 210 so as to support the harvest object 210. Since the support part 80 supports the harvest object 210, the harvest object 210 is held, and therefore the support part 80 can also be referred to as a "holding part".
[0044] The support portion 80 includes, for example, a first gripping portion 82a, a second gripping portion 82b, a first claw portion 84a, and a second claw portion 84b. The first gripping portion 82a and the second gripping portion 82b are configured to be able to pinch and hold the harvest target 210 from both sides. The first gripping portion 82a and the second gripping portion 82b have similar shapes and are arranged at an interval in the direction D2.
[0045] Each of the first gripping portion 82a and the second gripping portion 82b may be formed in a plate shape. The first gripping portion 82a includes an inner wall surface 83a facing the second gripping portion 82b, and the second gripping portion 82b includes an inner wall surface 83b facing the first gripping portion 82a. When the first gripping portion 82a and the second gripping portion 82b grip the harvest target 210, at least a part of the inner wall surface 83a and at least a part of the inner wall surface 83b come into contact with the harvest target 210.
[0046] The first gripping portion 82a may be bent or curved such that a central portion of the inner wall surface 83a in the direction D1 is recessed in a direction away from the second gripping portion 82b. The second gripping portion 82b may be bent or curved such that a central portion of the inner wall surface 83b in the direction D1 is recessed in a direction away from the first gripping portion 82a. Unlike the example shown in Fig. 3, the inner wall surface 83a and the inner wall surface 83b may be flat surfaces along the directions D1 and D3, respectively.
[0047] The first gripping portion 82a is provided on the first movable portion 64a in a state in which it can rotate about an axis 88, and the second gripping portion 82b is provided on the second movable portion 64b in a state in which it can rotate about the axis 88. The first movable portion 64a and the second movable portion 64b are each driven by the drive portion 66 of the main body portion 60, thereby changing the distance between the first gripping portion 82a and the second gripping portion 82b. The first gripping portion 82a and the second gripping portion 82b come close to each other and apply a force inwardly on the side of the harvest object 210, whereby the harvest object 210 is gripped (supported) by the first gripping portion 82a and the second gripping portion 82b.
[0048] The first gripping portion 82a is rotatable about an axis 88, for example, between a state in which its tip is located in front of the first movable portion 64a and a state in which its tip is located below the first movable portion 64a. The first gripping portion 82a may be supported by the first movable portion 64a so that the state in which its tip is located in front of the first movable portion 64a is maintained when no external force is applied. A torsion spring, for example, is provided at the connection portion between the first gripping portion 82a and the first movable portion 64a.
[0049] The second gripping portion 82b is rotatable about the axis 88 between, for example, a state in which its tip is located in front of the second movable portion 64b and a state in which its tip is located below the second movable portion 64b. The second gripping portion 82b may be supported by the second movable portion 64b so that the tip is maintained in a state in which it is located in front of the second movable portion 64b when no external force is applied. A torsion spring, for example, is provided at the connection portion between the second gripping portion 82b and the second movable portion 64b. "θ" shown in FIG. 4 is an angle representing the movable range of the first gripping portion 82a and the second gripping portion 82b in the circumferential direction around the axis 88. The angle θ may be about 90°.
[0050] The first claw portion 84a and the second claw portion 84b allow the harvest object 210 to move from a position forward of the first gripping portion 82a and the second gripping portion 82b to the space between the first gripping portion 82a and the second gripping portion 82b. In addition, the first claw portion 84a and the second claw portion 84b restrict the movement of the harvest object 210 from the space between the first gripping portion 82a and the second gripping portion 82b to a position forward of the first gripping portion 82a and the second gripping portion 82b.
[0051] The first claw portion 84a is provided on the inside of the tip of the first grip portion 82a, and the second claw portion 84b is provided on the inside of the tip of the second grip portion 82b. When a rearward force is applied to each of the first claw portion 84a and the second claw portion 84b, they rotate about an axis extending in the direction D3, and when a forward force is applied to each of the first claw portion 84a and the second claw portion 84b, they do not rotate about the axis.
[0052] 4, the end effector 50 may include a detection unit 90. The detection unit 90 is capable of detecting the harvesting object 210 and the handle 220 connected thereto. The detection unit 90 may be provided on the base unit 62 of the main body unit 60. The detection unit 90 includes, for example, a camera capable of capturing images of the harvesting object 210 and the handle 220. The detection unit 90 may include a sensor capable of measuring the distance to the harvesting object 210 and the handle 220 in addition to or instead of the camera.
[0053] 1, the controller 100 is one or more computer devices configured to control each of the robot 2, the dolly 4, and the imaging device 8. The controller 100 may be configured by a plurality of computer devices that individually control the robot 2, the dolly 4, and the imaging device 8. In this case, the plurality of computer devices may be connected to each other so as to be able to communicate with each other.
[0054] 5, the controller 100 has, as functional components (hereinafter referred to as "functional modules"), for example, an imaging control unit 102, an object detection unit 104, a cart control unit 106, a recognition unit 108, a harvesting control unit 110, and a storage control unit 112. The processes executed by these functional modules correspond to the processes executed by the controller 100.
[0055] Imaging control unit 102 acquires image data from imaging device 8 by having imaging device 8 capture images of at least a portion of crop 200. When imaging device 8 is mounted on truck 4, imaging control unit 102 may cause imaging device 8 to capture images both during a period when truck 4 is traveling and during a period when truck 4 is stopped. When imaging device 8 acquires still images, imaging control unit 102 may cause imaging device 8 to repeatedly capture images at a predetermined imaging period.
[0056] The object detection unit 104 executes a process of detecting harvest objects 210 in the crop 200 from the captured image PI based on image data acquired from the imaging device 8. The harvest objects 210 detected by the object detection unit 104 are to be harvested by the robot 2. As shown in FIG. 6(a) and other figures, the object detection unit 104 detects, for example, rectangular bounding boxes that surround one or more harvest objects 210 included in the captured image PI. The object detection unit 104 may detect the harvest objects 210 from the captured image PI using a detection model constructed in advance by machine learning.
[0057] The target detection unit 104 may detect the harvest target 210 from information obtained by the imaging device 8 while the dolly 4 is traveling. The target detection unit 104 may detect the harvest target 210 from information obtained by the imaging device 8 while the dolly 4 is stopped. Depending on the imaging timing, the harvest target 210 may not be detected from the captured image PI, in which case the target detection unit 104 may determine that the harvest target 210 is not detected from the captured image PI.
[0058] The carriage control unit 106 controls the carriage 4 to repeatedly run and stop. For example, when a harvest object 210 is detected from an image PI captured by the imaging device 8 while the carriage 4 is running, the carriage control unit 106 stops the carriage 4 according to the position of the harvest object 210. When the harvest object 210 is not detected while the carriage 4 is running, the carriage control unit 106 causes the carriage 4 to continue running.
[0059] For example, if the harvest target 210 is not detected from the captured image PI captured by the imaging device 8 while the cart 4 is stopped (while it continues to be stopped), the cart control unit 106 causes the cart 4 to travel. If the harvest target 210 is detected while the cart 4 is stopped, the cart control unit 106 maintains the cart 4 in a stopped state.
[0060] The recognition unit 108 recognizes a specific part of the harvest object 210 detected by the target detection unit 104. In the captured image PI obtained by the imaging device 8, a part of the harvest object 210 may be hidden by other obstacles (e.g., the stem 202 or the leaf 204). The specific part recognized by the recognition unit 108 is, for example, a part for which the operation of the robot 2 needs to be changed depending on whether or not the part can be approached. The specific part may be the upper end 212a of the harvest object 210.
[0061] The recognition unit 108 may directly or indirectly recognize the upper end 212a in the captured image PI. For example, the recognition unit 108 recognizes the upper end 212a by using a recognition model previously constructed by machine learning to recognize (detect) the upper end 212a from the captured image PI. The recognition unit 108 may recognize the upper end 212a by recognizing the handle 220 by using a recognition model previously constructed by machine learning to recognize (detect) the handle 220 connected to the upper end 212a from the captured image PI. The recognition unit 108 may determine that the upper end 212a cannot be recognized when the upper end 212a or the handle 220 is hidden by another obstacle.
[0062] The recognition unit 108 may recognize the upper end 212a by determining the hiding state of the harvest object 210. The recognition unit 108, for example, determines whether the hiding state of the detected harvest object 210 is a state in which the whole is not hidden, a state in which the upper end 212a is not hidden and the lower end 212b is hidden, or a state in which the upper end 212a is hidden and the lower end 212b is hidden. A state in which a certain part is not hidden is a state in which that part is reflected in the captured image PI. A state in which a certain part is hidden is a state in which that part is not reflected in the captured image PI.
[0063] Fig. 6(a) shows a state in which the entire harvest target object 210 is not hidden in the captured image PI (hereinafter referred to as "not hidden state"). Fig. 6(b) shows a state in which the upper end 212a is not hidden and the lower end 212b is not hidden in the captured image PI (hereinafter referred to as "lower hidden state"), and Fig. 6(c) shows a state in which the upper end 212a is hidden and the lower end 212b is not hidden in the captured image PI (hereinafter referred to as "upper hidden state").
[0064] The recognition unit 108 may determine (classify) the hidden state of the detected harvest object 210 using a discrimination model constructed in advance by machine learning to determine which of the above three states the hidden state of the harvest object 210 is in. The recognition unit 108 determining a non-hidden state and a bottom hidden state correspond to the recognition unit 108 recognizing the top end 212a. The recognition unit 108 determining an top hidden state corresponds to the recognition unit 108 not recognizing the top end 212a.
[0065] The harvest control unit 110 controls the robot 2 to harvest the harvest object 210 by a different operation depending on whether or not the specific portion is recognized. That is, the harvest control unit 110 controls the robot 2 to harvest the harvest object 210 by a different operation when the specific portion is recognized by the recognition unit 108 and when the specific portion is not recognized by the recognition unit 108. The harvest control unit 110 may control the robot 2 to harvest the harvest object 210 by a different operation depending on whether or not the upper end portion 212a is recognized as the specific portion.
[0066] When the upper end 212a of the harvest target 210 is recognized, the harvest control unit 110 controls the robot 2, for example, so that the support unit 80 of the end effector 50 supports the upper end 212a. In this case, the harvest control unit 110 controls the robot 2 so that the end effector 50 directly applies force to the handle 220 with the upper end 212a supported by the support unit 80. In one example, when the upper end 212a is recognized, the harvest control unit 110 controls the robot 2 so that the cutting unit 70 cuts the handle 220 with the upper end 212a supported by the support unit 80. The handle 220 is cut by the cutting unit 70, and the harvest target 210 is separated from the crop 200.
[0067] When the upper end 212a of the harvesting target object 210 is not recognized, the harvesting control unit 110 controls the robot 2, for example, so that the support unit 80 supports the lower end 212b. In this case, the harvesting control unit 110 controls the robot 2 so that, with the lower end 212b supported by the support unit 80, a force is indirectly applied to the handle 220 by the end effector 50. In one example, when the upper end 212a is not recognized, the harvesting control unit 110 controls the robot 2 so that, with the lower end 212b supported by the support unit 80, the end effector 50 is rotated around the axis 26 to twist the handle 220.
[0068] Twisting the handle 220 means rotating the handle 220 in the circumferential direction around the extension direction of the handle 220. The extension direction of the handle 220 is the direction in which an imaginary line connecting one end and the other end of the handle 220 extends. The harvesting control unit 110 may control the robot 2 to rotate the handle 220 alternately in opposite rotational directions in the circumferential direction. Alternatively, the harvesting control unit 110 may control the robot 2 to rotate the handle 220 continuously in one rotational direction in the circumferential direction.
[0069] The harvest control unit 110 may control the robot 2 so that the support unit 80 approaches the harvest object 210 from a direction intersecting the longitudinal direction of the harvest object 210, whether the upper end 212a is recognized or not. The longitudinal direction of the harvest object 210 is the direction in which an imaginary line connecting one end (e.g., the upper end) and the other end (e.g., the lower end) of the harvest object 210 extends. The harvest control unit 110 may cause the robot 2 to perform a harvesting operation to harvest the harvest object 210 while the carriage 4 is stopped.
[0070] The storage control unit 112 causes the robot 2 to execute a storage operation of storing the harvested objects 210 in the storage unit 6 at least during the period when the cart 4 is traveling. In other words, when the storage operation is performed while the cart 4 is traveling, the storage control unit 112 controls the robot 2 to store the harvested objects 210 in the storage unit 6 while the cart 4 is traveling. The storage control unit 112 may also cause the robot 2 to execute a storage operation of storing the harvested objects 210 in the storage unit 6 during the period when the cart 4 is stopped.
[0071] 7, the controller 100 includes a circuit 150. The circuit 150 includes one or more processors 152, a memory 154, a storage 156, one or more drivers 158, and an input / output port 162. The storage 156 is a non-volatile storage medium (e.g., a flash memory) that is readable by a computer. The storage 156 stores programs and data for causing the controller 100 to control the robot 2 including the end effector 50, the dolly 4, and the imaging device 8. The memory 154 temporarily stores the programs loaded from the storage 156, the results of calculations by the processor 152, and the like.
[0072] The processor 152 executes the above programs in cooperation with the memory 154 to configure each of the functional modules described above. One or more drivers 158 output drive power to the actuators 41-46 of the robot 2 and to a drive motor included in the dolly 4 in response to a command from the processor 152. The input / output port 162 inputs and outputs electric signals to and from the imaging device 8 and the like. Note that the controller 100 is not necessarily limited to one that configures each function by a program. At least some of the functions of the controller 100 may be configured by, for example, a dedicated logic circuit or an ASIC (Application Specific Integrated Circuit) that integrates the dedicated logic circuit.
[0073] [Control method of harvesting device] Next, a control method for the harvesting device 1 will be described with reference to Figures 8 to 13. This control method includes at least detecting harvest object 210 from captured image PI obtained by capturing an image of at least a portion of crop 200, recognizing a specific part in the detected harvest object 210, and controlling robot 2 to harvest harvest object 210 by different operations depending on whether or not the specific part has been recognized.
[0074] 8 is a flowchart showing an example of a series of processes executed by controller 100. With dolly 4 placed at an initial position, which is one end of rail 5, controller 100 first executes step S01. In step S01, for example, dolly control unit 106 starts traveling of dolly 4. Dolly control unit 106 may start traveling of dolly 4 along the direction in which multiple stalks 201 included in crop 200 are lined up. Imaging control unit 102 may start imaging by imaging device 8 in response to the start of traveling of dolly 4. As a result, imaging by imaging device 8 is performed during the period in which dolly 4 is traveling.
[0075] Next, controller 100 executes step S02. In step S02, for example, imaging control unit 102 acquires captured image PI from imaging device 8, which is an image obtained by capturing an image of at least a portion of crop 200. Then, target detection unit 104 executes processing for detecting harvest target object 210 that is to be the target of harvesting operation by robot 2 in captured image PI.
[0076] Next, controller 100 executes step S03. In step S03, for example, controller 100 determines whether or not harvest object 210 was detected in step S02. If it is determined in step S03 that harvest object 210 has not been detected (step S03: NO), the process executed by controller 100 returns to step S02. In this case, since cart 4 continues traveling, the imaged location of crop 200 is changed, and object detection unit 104 executes a process for detecting harvest object 210 from captured image PI.
[0077] If it is determined in step S03 that the harvest target 210 has been detected (step S03: YES), the process executed by the controller 100 proceeds to step S04. In step S04, for example, the carriage control unit 106 stops the carriage 4. In one example, the carriage control unit 106 stops the carriage 4 so that the end effector 50 of the robot 2 can approach the detected harvest target 210 when the carriage 4 is stopped.
[0078] Next, the controller 100 executes step S05. In step S05, for example, the harvest control unit 110 controls the robot 2 to harvest the detected harvest object 210. The harvest control unit 110 may cause the robot 2 to harvest the harvest object 210 by different operations depending on the recognition result of the upper end 212a of the harvest object 210 on the captured image PI in which the harvest object 210 is detected. Since the cart 4 is stopped in step S04, the harvest control unit 110 causes the robot 2 to execute the harvest operation of step S05 while the cart 4 is stopped. Details of step S05 will be described later.
[0079] Next, the controller 100 executes step S06. In step S06, for example, the imaging control unit 102 acquires a captured image PI from the imaging device 8. That is, after the robot 2 harvests the harvest object 210 in step S05, the imaging control unit 102 acquires a captured image PI (second captured image) from the imaging device 8. Then, the target detection unit 104 executes a process for detecting the next harvest object 210 to be the target of the harvest operation by the robot 2 in the captured image PI.
[0080] Next, the controller 100 executes step S07. In step S07, for example, the controller 100 determines whether or not the next harvest target 210 was detected in step S06. If it is determined in step S07 that the next harvest target 210 was detected (step S07: YES), the process executed by the controller 100 proceeds to step S08.
[0081] In step S08, for example, the storage control unit 112 controls the robot 2 to store in the storage unit 6 the harvest object 210 supported (e.g., gripped) by the support unit 80 of the end effector 50 at the time when execution of step S05 is completed. The storage operation in step S08 is performed while the carriage 4 is stopped. After execution of step S08, the process performed by the controller 100 returns to step S05. Therefore, while the carriage 4 is stopped at the same position, the robot 2 performs the harvest operation of the next harvest object 210 detected in step S06.
[0082] On the other hand, if it is determined in step S07 that the next harvest object 210 has not been detected (step S07: NO), the process executed by the controller 100 proceeds to step S09. In step S09, for example, the carriage control unit 106 starts the travel of the carriage 4. As the carriage 4 starts traveling, the imaging range PR of the imaging device 8 changes. As described above, in step S06, if the target detection unit 104 does not detect the next harvest object 210 from the captured image PI, the carriage control unit 106 causes the carriage 4 to travel. In step S06, if the target detection unit 104 detects the next harvest object 210 from the captured image PI, the carriage control unit 106 maintains the carriage 4 stopped.
[0083] After executing step S09, the controller 100 executes step S10. In step S10, for example, the storage control unit 112 controls the robot 2 to store the harvest object 210 supported (e.g., gripped) by the support unit 80 of the end effector 50 at the time of completion of execution of step S05 in the storage unit 6. Unlike step S08, the storage operation in step S10 is performed while the carriage 4 is traveling. After executing step S10, the process executed by the controller 100 returns to step S02.
[0084] The controller 100 continues the series of processes shown in Fig. 8 until a predetermined end condition is satisfied, for example. In one example, the controller 100 ends the series of processes shown in Fig. 8 when the carriage 4 reaches an end position that is the other end opposite to the one end (initial position) of the rail 5.
[0085] 9 is a flowchart showing an example of a series of processes executed by the controller 100 in step S05. In step S05, the controller 100 first executes step S21. In step S21, for example, the recognition unit 108 determines the occlusion state of the detected harvest object 210 in the captured image PI obtained in step S02 or step S06. In one example, the recognition unit 108 determines whether the occlusion state of the detected harvest object 210 is the non-occlusion state, the lower occlusion state, or the upper occlusion state.
[0086] Next, the controller 100 executes step S22. In step S22, for example, the controller 100 determines whether or not the upper end 212a of the detected harvesting object 210 has been recognized. In one example, the controller 100 determines that the upper end 212a has been recognized when the obscuration state of the harvesting object 210 is determined to be a non-obscured state or a bottom obscured state. The controller 100 determines that the upper end 212a has not been recognized when the obscuration state of the harvesting object 210 is determined to be an top obscured state.
[0087] If it is determined in step S22 that the upper end 212a has been recognized (step S22: YES), the process executed by the controller 100 proceeds to steps S31 to S33. On the other hand, if it is determined in step S22 that the upper end 212a has not been recognized (step S22: NO), the process executed by the controller 100 proceeds to steps S41 to S44. That is, if the upper end 212a has been recognized, the controller 100 executes a series of processes from steps S31 to S33, and if the upper end 212a has not been recognized, the controller 100 executes a series of processes from steps S41 to S44.
[0088] 10(a) and 10(b), for example, the harvest control unit 110 controls the robot 2 so that the support unit 80 of the end effector 50 approaches the upper end 212a of the harvest object 210. The harvest control unit 110 may also control the robot 2 so that the support unit 80 approaches the upper end 212a from a direction intersecting the extension direction of the harvest object 210. The harvest control unit 110 may cause the support unit 80 to approach the upper end 212a using the detection result by the detection unit 90 provided in the end effector 50, in addition to the position information of the harvest object 210 obtained from the captured image PI.
[0089] While step S31 is being performed, the first gripping portion 82a and the second gripping portion 82b of the support portion 80 may be disposed at a position where the distance between them is greater than the width of the harvest object 210. After the harvest object 210 is disposed in the space between the first gripping portion 82a and the second gripping portion 82b, the harvest control portion 110 may adjust the height position of the end effector 50 by the articulated arm 10 to a position where the handle 220 can be cut by the cutting portion 70 after the harvest object 210 is disposed in the space between the first gripping portion 82a and the second gripping portion 82b. When adjusting the height position of the end effector 50, even if the harvest object 210 attempts to move forward from the space between the first gripping portion 82a and the second gripping portion 82b, the movement may be restricted by the first claw portion 84a and the second claw portion 84b.
[0090] In step S32, for example, the harvest control unit 110 controls the drive unit 66 to narrow the gap between the first movable unit 64a and the second movable unit 64b and to grip the upper end 212a by the first gripping unit 82a and the second gripping unit 82b, as shown in Fig. 11(a). The drive amount of the first movable unit 64a and the second movable unit 64b is set, for example, to an amount that holds the harvest object 210 and does not apply excessive force to the harvest object 210. While steps S31 and S32 are being executed, the harvest control unit 110 may maintain the blade 72 of the cutting unit 70 in a state where it is covered by the cover member 74 and not exposed.
[0091] In step S33, for example, the harvest control unit 110 controls the drive unit 76 to switch the blade 72 from a state in which the blade 72 is covered by the cover member 74 to a state in which the blade 72 is exposed from the cover member 74. As shown in Fig. 11(b), in step S33, the harvest control unit 110 causes the drive unit 76 to move the blade 72 so that the cutting edge (tip) of the blade 72 comes into contact with the handle 220 and the handle 220 is cut. The amount of drive of the handle 220 may be preset to an amount that allows the handle 220 to be cut when the support unit 80 supports the upper end portion 212a, even if there are some individual differences in the harvest object 210.
[0092] By executing a series of processes from steps S31 to S33, the harvesting operation when upper end 212a is recognized is completed, and harvest object 210 is separated from crop 200 while being supported by support unit 80. Immediately after harvest object 210 is harvested by executing step S33, harvest control unit 110 may control cutting unit 70 to retract blade 72 so that it is covered by cover member 74.
[0093] 12(a) and 12(b), for example, the harvest control unit 110 controls the robot 2 so that the support unit 80 of the end effector 50 approaches the lower end 212b of the harvest object 210. The harvest control unit 110 may also control the robot 2 so that the support unit 80 approaches the lower end 212b from a direction intersecting the extension direction of the harvest object 210. The harvest control unit 110 may cause the support unit 80 to approach the lower end 212b using the detection result by the detection unit 90 provided in the end effector 50, in addition to the position information of the harvest object 210 obtained from the captured image PI.
[0094] In step S42, for example, the harvest control unit 110 controls the drive unit 66 to narrow the gap between the first movable unit 64a and the second movable unit 64b so that the lower end 212b is gripped by the first gripping unit 82a and the second gripping unit 82b. The drive amount of the first movable unit 64a and the second movable unit 64b in step S42 may be set to the same as in step S32. In step S42, the harvest control unit 110 may cause the support unit 80 to support the harvest object 210 with the extension direction of the handle 220 intersecting the axis 26.
[0095] In step S43, for example, the harvest control unit 110 causes the robot 2 to execute an operation to change the posture of the harvest target 210 supported by the support unit 80 relative to the main body unit 60 of the end effector 50. As shown in Figures 13(a) and 13(b), the harvest control unit 110 causes the main body unit 60 to move by the articulated arm 10 so that the support unit 80 rotates about the axis 88 and the angle between the extension direction of the handle 220 and the axis 26 decreases.
[0096] When the extension direction of the handle 220 (a virtual line passing through both ends of the handle 220) and the axis 26 are in a twisted relationship, the angle between the extension direction of the handle 220 and the axis 26 is calculated by moving either one in parallel so that they intersect at an intersection. In step S43, for example, the harvest control unit 110 operates the articulated arm 10 so as to lift the main body 60 diagonally backward. By moving the main body 60 so as to be lifted diagonally backward, the first gripping portion 82a and the second gripping portion 82b supporting the harvest target 210 rotate about the axis 88, and the posture of the harvest target 210 changes.
[0097] In one example, the harvest control unit 110 moves the main body 60 diagonally backward by the articulated arm 10 until the tips of the first gripping portion 82a and the second gripping portion 82b face downward. The angle between the extension direction of the handle 220 and the axis 26 after execution of step S43 is smaller than the angle before execution of step S43.
[0098] In step S44, for example, the harvest control unit 110 rotates the end effector 50 about the axis 26 so that the handle 220 is twisted. In one example, the harvest control unit 110 controls the actuator 46 of the articulated arm 10 so that rotation continues in one direction in the circumferential direction about the axis 26. The harvest control unit 110 rotates the end effector 50 by the actuator 46 of the articulated arm 10 so that the handle 220 is twisted and cut off (so that the handle 220 is twisted off).
[0099] The harvest control unit 110 may control the actuator 46 of the articulated arm 10 so that the end effector 50 rotates a predetermined number of rotations. The number of rotations is set to a value that allows the handle 220 to be cut even if there are some individual differences in the harvest object 210. The harvest control unit 110 may detect the cutting of the handle 220 from the state (e.g., torque) of the actuator 46 that rotates the end effector 50 about the axis 26 or the detection result by the detection unit 90, and may stop the rotation of the end effector 50 about the axis 26.
[0100] By executing a series of processes from steps S41 to S44, the harvesting operation when upper end 212a is not recognized is terminated, and harvest object 210 is separated from crop 200 while being supported by support portion 80. After handle 220 is twisted off in step S44, first gripping portion 82a and second gripping portion 82b may return to the positions they had before step S43 was executed due to the restoring forces of torsion springs provided in the shaft portions of first gripping portion 82a and second gripping portion 82b.
[0101] [Variations] The series of processes shown in each of Fig. 8 and Fig. 9 are examples and can be modified as appropriate. In the series of processes described above, the controller 100 may execute one step and the next step in parallel, or may execute each step in an order different from the above-mentioned example. The controller 100 may omit any step, or may execute a process different from the above-mentioned example in any step.
[0102] In processing the captured image PI in step S02, the controller 100 may detect the harvest object 210 and recognize a specific portion of the detected harvest object 210 (processing content of step S21) at substantially the same timing. For example, the controller 100 may detect the harvest object 210 in response to the input of the captured image PI, and may detect the harvest object 210 and recognize a specific portion (e.g., the upper end 212a) using a learning model constructed by machine learning to determine the hidden state of the harvest object 210. In this manner, at substantially the same timing, the object detection unit 104 of the controller 100 may detect the harvest object 210 from the captured image PI, and the recognition unit 108 of the controller 100 may recognize the upper end 212a of the detected harvest object 210.
[0103] After starting execution of step S10, the controller 100 may start traveling of the cart 4 in step S09 while the storage operation in step S10 is continuing. Even in this case, the storage control unit 112 causes the robot 2 to perform the storage operation while the cart 4 is traveling. The harvesting control unit 110 may operate the end effector 50 by the articulated arm 10 so that the handle 220 is twisted and cut off in a state where the support unit 80 supports the lower end portion 212b, without executing step S43.
[0104] The support part 80 of the end effector 50 may support the harvesting object 210 by a method other than gripping. The support part 80 may support the harvesting object 210 by suction, for example. The cutting part 70 of the end effector 50 may cut the handle 220 by any method other than cutting with a blade, as long as the handle 220 can be cut by directly applying a force to the handle 220.
[0105] Even if the harvest object 210 does not have an elongated shape (for example, the crop 200 is a tomato), the recognition unit 108 may recognize the end connected to the stem. Then, the harvest control unit 110 may cause the robot 2 to perform a harvesting operation in a different manner depending on whether the end has been recognized.
[0106] In the above example, the harvesting operation differs depending on the recognition result of a specific part of the harvest object 210 (for example, the end connected to the handle), but instead, the controller 100 may change the processing content depending on whether or not the specific part is recognized. Specifically, the controller 100 controls the robot 2 to harvest the harvest object 210 when the specific part is recognized, and transmits individual information of the harvest object 210 to another device when the specific part is not recognized. For example, the harvest control unit 110 causes the robot 2 to perform the harvesting operation when the upper end 212a is recognized, and does not cause the robot 2 to perform the harvesting operation when the upper end 212a is not recognized.
[0107] The controller 100 may include, for example, as a functional module, an information notification unit 118 that transmits the individual information of the harvest object 210 to another device, as shown in FIG. 5. When a specific portion (for example, the upper end portion 212a) is not recognized, the information notification unit 118 transmits the individual information of the harvest object 210 to another device. The information notification unit 118 transmits the individual information to, for example, a higher-level controller as another device, or a user terminal held by an operator who harvests the harvest object 210 together with the harvesting device 1. The individual information of the harvest object 210 may include information indicating the position of the harvest object 210. In one example, the operator receives a notification from the controller 100, identifies the harvest object 210 based on the individual information, and then harvests the harvest object 210.
[0108] In one example among the various examples described above, at least a part of the matters described in the other examples may be combined.
[0109] [Summary of the first embodiment] This disclosure includes the following configurations [1] to
[13] and the configuration of Appendix 1.
[0110] [1] A harvesting device 1 comprising: a robot 2; an object detection unit 104 that detects a harvest target 210 in a crop 200 from an image PI obtained by capturing an image of at least a portion of the crop 200; a recognition unit 108 that recognizes a specific part of the detected harvest target 210; and a harvest control unit 110 that controls the robot 2 to harvest the harvest target 210 by performing different operations depending on whether or not the specific part has been recognized. When harvesting the harvest target in the agricultural crop 200, the operation suitable for harvesting differs depending on whether or not the tool of the robot can approach a specific part of the harvest target. In the harvesting device 1, the operation of the robot 2 when harvesting differs depending on whether or not the specific part is recognized, so it is possible to cause the robot 2 to perform an operation suitable for harvesting. This improves the efficiency of the harvesting work. Therefore, the harvesting device 1 is useful for improving the efficiency of work.
[0111] [2] The harvesting device 1 described in [1] above, wherein the harvesting object 210 includes a first end connected to the handle 220, and the recognition unit 108 recognizes the first end as a specific portion. The operation of the robot 2 suitable for harvesting differs depending on whether the harvesting device 1 can approach the end of the harvest object 210 near the handle 220. In the above configuration, the operation of the robot 2 differs depending on the recognition result of the first end, so the robot 2 can be operated to perform an operation suitable for harvesting. Therefore, this is useful for improving the efficiency of the work of harvesting the harvest object 210.
[0112] [3] The harvesting device 1 described in [2] above, wherein the harvesting object 210 has a shape extending from a first end (upper end 212a) in a direction opposite to the handle 220, the robot 2 has an end effector 50 including a support portion 80 capable of supporting the harvesting object 210, and the harvesting control unit 110 controls the robot 2 so that, when the first end (upper end 212a) is recognized, the support portion 80 supports the first end (upper end 212a), and when the first end (upper end 212a) is not recognized, the support portion 80 controls the robot 2 so that the support portion 80 supports a second end (lower end 212b) opposite the first end (upper end 212a). [4] The harvesting device 1 described in [2] above, wherein the harvesting object 210 has a shape extending from a first end (upper end 212a) in a direction opposite to the handle 220, the robot 2 has an end effector 50 including a support portion 80 capable of supporting the harvesting object 210, and when the first end (upper end 212a) is recognized, the harvesting control unit 110 controls the robot 2 so that the support portion 80 supports the first end (upper end 212a). By cutting the handle 220, the harvest object 210 is separated from the crop 200 and harvested. In the above configuration, when the first end (upper end 212a) connected to the handle 220 is recognized, a portion of the harvest object 210 close to the handle 220 is supported, making it possible to perform harvesting more stably. Even if the first end (upper end 212a) is not recognized, a portion far from the handle 220 is supported to indirectly apply force to the handle 220, making it possible to perform harvesting.
[0113] [4] The harvesting control unit 110 controls the robot 2 so that the support unit 80 approaches the harvest object 210 from a direction intersecting the longitudinal direction of the harvest object 210, whether the first end (upper end 212a) is recognized or not, in the harvesting device 1 described in [3] above. In this case, the movable range of the robot 2 is not expanded due to differences in the recognition results of the first end (upper end 212a), so the possibility of damaging parts of the crop 200 surrounding the harvest target 210 can be reduced.
[0114] [5] The robot 2 further has a multi-joint arm 10, the end effector 50 is connected to the tip of the multi-joint arm 10 in a state where it can rotate around the axis 26, and the harvesting control unit 110 controls the robot 2 so that when the first end (upper end 212a) is not recognized, the end effector 50 is rotated around the axis 26 with the second end (lower end 212b) supported by the support unit 80, thereby twisting the handle 220. The harvesting device 1 described in [3] or [4] above. In this case, even if end effector 50 cannot approach the first end portion close to handle 220, harvest object 210 can be separated from crop 200 more reliably.
[0115] [6] The end effector 50 further includes a main body portion 60 connected to the tip of the articulated arm 10 in a state where it can rotate around the axis 26, and the support portion 80 is connected to the main body portion 60 in a state where it can rotate around an axis 88 that intersects the axis 26, and when the first end portion (upper end portion 212a) is not recognized, the harvest control portion 110 sequentially performs the following operations: supporting the harvest object 210 by the support portion 80 in a state where the extension direction of the handle 220 intersects the axis 26; moving the main body portion 60 by the articulated arm 10 so that the support portion 80 rotates around the axis 88 to reduce the angle between the extension direction of the handle 220 and the axis 26; and rotating the end effector 50 around the axis 26 so that the handle 220 is twisted. The harvesting device 1 described in any one of [3] to [5] above. In this case, even if the support part 80 approaches and supports the harvest object 210 from a direction intersecting the extension direction of the handle 220, the posture of the harvest object 210 changes to a posture that makes it easy to twist off the handle 220. Therefore, it is possible to achieve both more reliable twisting off of the handle 220 and a reduced movable range of the articulated arm 10 of the robot 2.
[0116] [7] The end effector 50 further includes a cutting section 70 capable of cutting the vicinity of the portion of the harvest object 210 supported by the support section 80, and the harvest control section 110 controls the robot 2 to cause the cutting section 70 to cut the handle 220 with the first end (upper end 212a) supported by the support section 80 when the first end (upper end 212a) is recognized, a harvesting device 1 described in any one of [3] to [6] above. In this case, force acts directly on the handle 220 connected to the harvest object 210, causing the handle 220 to be severed, so that the harvest object 210 can be harvested in a short period of time.
[0117] [8] The cutting unit 70 includes a blade 72, a cover member 74 that covers the blade 72, and a drive unit 76 that moves the blade 72 relative to the cover member 74, and the harvesting control unit 110 controls the drive unit 76 to switch the blade 72 from a state in which the blade 72 is covered by the cover member 74 to a state in which the blade 72 is exposed from the cover member 74 when the first end (upper end 212a) is recognized, in the harvesting device 1 described in [7] above. In this case, since the blade 72 is covered by the cover member 74 when the cutting unit 70 is not in use, it is possible to reduce the possibility of damaging parts other than the harvest target 210. Since the cutting unit 70 is not used when the upper end 212a is not recognized, the above configuration is more useful when harvesting is performed with different operations depending on whether the cutting unit 70 is used or not.
[0118] [9] A harvesting device 1 described in any one of [1] to [8] above, further comprising a carriage 4 on which the robot 2 is mounted and which can run, and a storage control unit 112, wherein the harvesting control unit 110 causes the robot 2 to perform a harvesting operation to harvest the harvest objects 210 while the carriage 4 is stopped, and the storage control unit 112 causes the robot 2 to perform a storage operation to store the harvested harvest objects 210 in the storage unit 6, at least while the carriage 4 is running. The carriage 4 is stopped to harvest the harvest objects 210, thereby stabilizing the harvesting work by the robot 2. In addition, since the operation of storing the harvest objects 210 is performed while the carriage 4 is traveling, the efficiency of the entire work can be improved compared to when all storage operations are performed with the carriage 4 stopped.
[0119]
[10] The harvesting device 1 described in any one of [1] to [8] above, further comprising an imaging device 8 capable of imaging at least a portion of a crop 200, a carriage 4 on which a robot 2 and the imaging device 8 are mounted and which can move, an imaging control unit 102, and a carriage control unit 106 which controls the carriage 4 to repeatedly move and stop, wherein the harvesting control unit 110 causes the robot 2 to perform a harvesting operation to harvest the harvest target 210 while the carriage 4 is stopped, the imaging control unit 102 acquires a second captured image from the imaging device 8 after the robot 2 harvests the harvest target 210, and the carriage control unit 106 causes the carriage 4 to move when the target detection unit 104 does not detect the next harvest target 210 from the second captured image. After one harvest object 210 is harvested, the state of the crop 200 changes, and a new harvest object 210 may become available for harvest by the robot 2. In the above configuration, the harvesting operation continues at the location where the cart 4 has stopped until the harvest object 210 is no longer detected, thereby improving the efficiency of the operation.
[0120]
[11] The harvesting device 1 described in
[10] above, wherein the imaging control unit 102 causes the imaging device 8 to capture images while the cart 4 is traveling, and the target detection unit 104 detects the next harvest target 210 from information obtained by the imaging device 8 while the cart 4 is traveling. In this case, since the robot 2 searches for the next object to be harvested 210 while moving, the robot 2 can quickly transition to harvesting operation for the next object to be harvested 210 compared to when the cart 4 stops and starts moving after searching for the next object to be harvested 210. Therefore, the efficiency of the work can be improved.
[0121]
[12] A harvesting device 1 comprising: a robot 2; a target detection unit 104 that detects harvest targets 210 in a crop 200 from an image PI obtained by capturing an image of at least a portion of the crop 200; a recognition unit 108 that recognizes a specific part of the detected harvest target 210; a harvest control unit 110 that controls the robot 2 to harvest the harvest target 210 when the specific part is recognized; and an information notification unit 118 that transmits individual information of the harvest target 210 to another device when the specific part is not recognized. When harvesting the harvest target in the agricultural crop 200, the operation suitable for harvesting differs depending on whether or not the tool of the robot can approach a specific part of the harvest target, and there are cases where approaching and harvesting a specific part allows for a shorter harvest time. In the above-mentioned harvesting device 1, when a specific part is recognized, a harvesting operation is performed by the robot 2, and when the specific part is not recognized, individual information of the harvest target 210 is transmitted to another device. For example, when a specific part is not recognized, the harvesting work of the harvest target 210 can be performed by a worker or another harvesting device based on the individual information. This improves the efficiency of the harvesting work. Therefore, the above-mentioned harvesting device 1 is useful for improving the efficiency of work.
[0122]
[13] A method for controlling a harvesting device 1, comprising: detecting a harvest target 210 in a crop 200 from an image PI obtained by capturing an image of at least a portion of the crop 200; recognizing a specific portion of the detected harvest target 210; and controlling a robot 2 to harvest the harvest target 210 by different operations depending on whether or not the specific portion is recognized. This control method is useful for improving work efficiency, similar to the harvesting device 1 described in [1] above.
[0123] [Appendix 1] Robots and A carriage on which the robot is mounted and which can run; an object detection unit that detects an object to be harvested from an image obtained by capturing an image of at least a portion of the agricultural crop; A harvesting control unit that controls the robot so as to harvest the detected harvesting object; A storage control unit; The harvest control unit causes the robot to perform a harvesting operation to harvest the harvest target object during a period in which the carriage is stopped, The storage control unit causes the robot to perform a storage operation of storing the harvested harvest objects in a storage unit at least during a period when the cart is traveling.
[0124] <Second embodiment> Next, a harvesting device according to a second embodiment and a control method thereof will be described with reference to Figs. 14 to 22. The robot 2 may have an end effector 50A instead of the end effector 50. The end effector 50A shown in Figs. 14 and 15 is a harvesting tool, and like the end effector 50, is connected to the wrist 17 of the articulated arm 10 in a state capable of rotating around the axis 26. The end effector 50A may be connected to the wrist 17 so that the extension direction thereof intersects (for example, perpendicular to) the axis 26.
[0125] In Fig. 14, direction D1 corresponds to the direction in which end effector 50A extends. In the following description, "forward" refers to the direction (orientation) from the base end, which is the end connected to wrist 17 of end effector 50A, to the tip end, which is the end opposite to the base end of end effector 50A. "Rear" refers to the direction (orientation) from the tip end of end effector 50A to the base end. Directions D2 and D3 each have the same meaning as the directions used in the description of end effector 50.
[0126] The end effector 50A includes a main body 60A and a holding part 80A. The main body 60A includes a base 62, a first movable part 64a, a second movable part 64b, a driving part 66, a cutting part 70A, and a stopper 69. In the end effector 50A, the holding part 80A is supported by the first movable part 64a and the second movable part 64b.
[0127] The cutting unit 70A has the same function as the cutting unit 70, and can cut the vicinity of the part of the harvest object 210 held by the holding unit 80A. Unlike the cutting unit 70, the cutting unit 70A does not have a cover member 74, but has a blade 72 and a drive unit 76 (cutting drive unit). The stopper 69 restricts the movement of the harvest object 210 such that the blade 72 hits the harvest object 210 from the front. The stopper 69 is provided, for example, in the direction D1 between the holding unit 80A and a position where the blade 72 is placed on standby and not performing a cutting operation. The stopper 69 may be connected to the second movable unit 64b and formed to extend in the direction D2 from the connection part of the second movable unit 64b. In the direction D3, the stopper 69 is provided at a position that does not interfere with the drive (in and out) of the blade 72.
[0128] The holding unit 80A is a portion capable of holding the harvest object 210. Unlike the support unit 80, the holding unit 80A may not be rotatable about an axis along the direction D3 with respect to the main body unit 60 (the first movable unit 64a and the second movable unit 64b). The holding unit 80A may include a pair of gripping units that hold the harvest object 210 by sandwiching it from the sides. The sides of the harvest object 210 mean a direction intersecting with the extension direction of the harvest object 210. The holding unit 80A may have a first gripping unit 82a and a second gripping unit 82b as a pair of gripping units, similar to the support unit 80 of the end effector 50. The first gripping unit 82a may be fixed to the first movable unit 64a in a state in which it is not allowed to move relative to the first movable unit 64a. The second gripping portion 82b may be fixed to the second movable portion 64b in a state in which the second gripping portion 82b is not permitted to move relative to the second movable portion 64b.
[0129] The driving unit 66 may include a motor 66a. The motor 66a drives the first gripping unit 82a and the second gripping unit 82b so as to change the interval (hereinafter, referred to as "interval d") between the first gripping unit 82a and the second gripping unit 82b. The motor 66a may drive the first movable unit 64a and the second movable unit 64b so as to change the shortest distance between the first gripping unit 82a and the second gripping unit 82b in the direction D2. The motor 66a may include a detector that detects the rotation position. Note that the main body 60A of the end effector 50A may have a cutting unit 70 instead of the cutting unit 70A, and may not have a stopper 69.
[0130] The harvesting device 1 may include an imaging device 9. The imaging device 9 has at least one of a camera provided at the tip (wrist 17) of the articulated arm 10 and a camera provided at the end effector 50A. The one or more cameras provided in the imaging device 9 may be installed so that their field of view (image target) changes depending on the position of the end effector 50A. The imaging device 9 has, for example, a camera 92 and a camera 94.
[0131] Camera 92 may image within the field of view using visible light. Camera 92 is provided on wrist 17. Camera 92 may be installed so that its field of view faces forward. Camera 92 is capable of imaging at least a portion of crop 200. Camera 94 may image within the field of view using visible light. Camera 94 is provided on base 62. Camera 94 may be installed so that its field of view faces forward and images the surroundings of holding unit 80A. Camera 94 may be capable of imaging at least a portion of harvest object 210 when harvest object 210 is located between first gripping unit 82a and second gripping unit 82b. Note that wrist 17, camera 92, and camera 94 are omitted in FIG. 15.
[0132] 5, the controller 100 may have, as shown in Fig. 16, a target detection unit 122, a placement control unit 124, a harvest determination unit 126, a part identification unit 128, a first approach control unit 130, a holding determination unit 132, a second approach control unit 134, a pattern recognition unit 136, and a harvest control unit 138. The processes executed by these functional modules correspond to the processes executed by the controller 100.
[0133] Object detection unit 122 executes a process of detecting harvest object 210 in crop 200 based on one or more images obtained by capturing an image of at least a portion of crop 200. Object detection unit 122 detects harvest object 210 from captured image PI acquired from imaging device 8, for example, in a manner similar to object detection unit 104. Object detection unit 122 may calculate the position of harvest object 210 (for example, the relative position from a reference position of robot 2 or cart 4) from distance data acquired by imaging device 8.
[0134] The position control unit 124 controls the robot 2 to place the end effector 50A at a position where an operation for harvesting the harvest object 210 is to be started (hereinafter referred to as the "operation start position"). The position control unit 124 moves the end effector 50A by the articulated arm 10 to a position where the entire harvest object 210 is included within the field of view of the camera 92 provided on the wrist unit 17, for example.
[0135] The harvest assessment unit 126 executes a harvest assessment process. In the harvest assessment process, the harvest assessment unit 126 calculates the size of the harvest object 210 based on one or more images obtained by capturing an image of at least a portion of the crop 200, and determines whether or not to harvest the harvest object 210 based on the result of the size calculation. The harvest assessment unit 126 may calculate the size (dimension) of the harvest object 210 based on an image captured by the camera 92 with the end effector 50A placed in the operation start position.
[0136] Part identifying unit 128 identifies a part of harvest object 210 that is accessible to end effector 50A, based on one or more images obtained by capturing an image of at least a portion of crop 200. Hereinafter, the part identified by part identifying unit 128 will be referred to as a "first part." For example, part identifying unit 128 identifies a part of harvest object 210 that is not hidden by an obstacle or the like, in any of the one or more images, as the first part. Part identifying unit 128 may identify the first part based on an image obtained by capturing an image of camera 92 with end effector 50A placed at the operation start position. Part identifying unit 128 may calculate the position of the first part (for example, a relative position from a reference position of robot 2 or cart 4).
[0137] Part identifying unit 128 may identify the first part from a region of harvest object 210 located between an end (upper end 212a) connected to handle 220 and an end (lower end 212b) opposite handle 220. Part identifying unit 128 may perform the following steps: identify two or more candidate parts in harvest object 210 based on one or more images obtained by capturing an image of at least a portion of crop 200; and identify, as the first part, a part of the two or more candidate parts that is accessible to end effector 50A and is closest to handle 220.
[0138] The first approach control unit 130 controls the robot 2 to bring the end effector 50A closer to a first portion of the harvest target 210 of the agricultural crop 200. The first approach control unit 130 moves (brings) the end effector 50A closer to the first portion by the articulated arm 10 so that the first portion is located between the first gripping unit 82a and the second gripping unit 82b, for example. The first approach control unit 130 may control the robot 2 to bring the end effector 50A closer to the first portion and reduce the distance d between the first gripping unit 82a and the second gripping unit 82b so as to contact the first portion.
[0139] The first approach control unit 130 may perform the following operations: using the multi-joint arm 10 to move the first gripping portion 82a and the second gripping portion 82b closer to the first portion; driving the first gripping portion 82a and the second gripping portion 82b with the motor 66a so that the distance d between the first gripping portion 82a and the second gripping portion 82b becomes smaller; and stopping the driving by the motor 66a based on the reaction force received by the first gripping portion 82a and the second gripping portion 82b.
[0140] The holding determination unit 132 determines whether the first gripping unit 82a and the second gripping unit 82b hold the harvest target 210 based on the position of the motor 66a when the driving of the first gripping unit 82a and the second gripping unit 82b by the motor 66a is stopped (hereinafter referred to as the "stop position"). The holding determination unit 132 may determine that the holding unit 80A holds the harvest target 210 when it can be determined from the stop position of the motor 66a that the distance d between the first gripping unit 82a and the second gripping unit 82b is greater than a predetermined value. The holding determination unit 132 may determine that the holding unit 80A is not holding the harvest target 210 when it can be determined from the stop position of the motor 66a that the distance d is equal to or less than a predetermined value.
[0141] The second approach control unit 134 controls the robot 2 to move the end effector 50A from the first portion to a portion of the harvest object 210 different from the first portion (hereinafter referred to as the "second portion") and have the end effector 50A hold the second portion. The second approach control unit 134 controls the robot 2, for example, to move the end effector 50A from the first portion toward the handle 220 connected to the harvest object 210 and have the end effector 50A hold a portion located between the first portion and the handle 220 as the second portion.
[0142] The second approach control unit 134 may control the robot 2 to move the end effector 50A along the longitudinal direction of the harvest object 210 toward the handle 220 and cause the end effector 50A to hold, as the second portion, the end portion (upper end portion 212a) of the harvest object 210 that is connected to the handle 220. The second approach control unit 134 may control the robot 2 to move the end effector 50A to the second portion while maintaining the state in which the harvest object 210 is positioned between the first gripping unit 82a and the second gripping unit 82b.
[0143] The second approach control unit 134 may control the motor 66a so that the interval d between the movement of the end effector 50A from the first part to the second part is smaller than the interval d between the movement of the end effector 50A by the first approach control unit 130 to the first part. The second approach control unit 134 may execute control to move the end effector 50A from the first part to the second part when the holding determination unit 132 determines that the holding unit 80A holds the first part. The second approach control unit 134 may sequentially execute the following operations: stopping the end effector 50A based on the recognition result of the handle 220 described later, and holding the part of the harvest target 210 corresponding to the stopping position of the end effector 50A as the second part by the end effector 50A.
[0144] The pattern recognition unit 136 recognizes the pattern 220 based on the recognition image obtained from the camera 94 (or the camera 92) while the end effector 50A is moving from the first part toward the pattern 220. For example, while the end effector 50A is moving toward the pattern 220, the pattern recognition unit 136 continues capturing images by the camera 94, and executes a process of detecting the pattern 220 in each recognition image. The pattern recognition unit 136 executes a process of detecting the pattern 220 for each recognition image, and does not recognize the pattern 220 if the pattern 220 is not detected, and recognizes the pattern 220 if the pattern 220 is detected. The pattern recognition unit 136 may detect the pattern 220 using a detection model that is constructed in advance by machine learning to detect the pattern 220 according to the input of the image.
[0145] The harvest control unit 138 controls the robot 2 to harvest the harvest target 210 with the end effector 50A holding the second portion. The harvest control unit 138 controls the robot 2, for example, to cut the handle 220 with the cutting unit 70A with the holding unit 80A holding the second portion (upper end 212a). The harvest control unit 138 may also control the robot 2 to have the holding unit 80A hold the second portion, and have the cutting unit 70A cut the handle 220 with the end effector 50A displaced to apply tension to the handle 220.
[0146] After the second approach control unit 134 causes the holding unit 80A to hold the second portion, the harvest control unit 138 may control the robot 2 to move the harvest target 210 away from the stem 202 (e.g., the main stem) connected to the stem 220 in a direction intersecting the stem 220, and may drive the blade 72 using the drive unit 76 to cut the stem 220 with the blade 72. The drive amount of the blade 72 during cutting may be determined in advance, or the harvest control unit 138 may adjust the drive amount of the blade 72 based on information indicating the reaction force received by the blade 72 (e.g., the torque of the motor included in the drive unit 76).
[0147] Next, a control method for harvesting device 1 equipped with end effector 50A will be described with reference to Figures 17 to 24. This control method includes at least controlling robot 2 to bring end effector 50A possessed by robot 2 close to a first portion of object to be harvested 210 of agricultural crop 200, and controlling robot 2 to move end effector 50A from the first portion to a second portion of object to be harvested 210 and cause end effector 50A to hold the second portion. This control method also includes controlling robot 2 to harvest object to be harvested 210 with end effector 50A holding the second portion.
[0148] Fig. 17 is a flowchart showing an example of a series of processes executed by the controller 100 shown in Fig. 16. The series of processes shown in Fig. 17 may be executed in the harvesting operation of step S05 in the series of processes shown in Fig. 8. Steps S02 and S06 may be executed by the target detection unit 122.
[0149] As shown in Fig. 17, the controller 100 first executes step S51. In step S51, for example, the placement control unit 124 controls the robot 2 so as to place the end effector 50A at the operation start position. Fig. 18(a) illustrates a state after step S51 has been executed. By executing step S51, the camera 92 may be in a state in which it is possible to capture an image of an area including the harvest target 210.
[0150] Next, the controller 100 executes step S52. In step S52, for example, the harvest judgment unit 126 causes the camera 92 to capture an image, thereby acquiring image data from the camera 92. Fig. 18(b) shows a captured image PI1 obtained by capturing an image with the camera 92.
[0151] Next, the controller 100 executes step S53. In step S53, for example, the harvest judgment unit 126 detects the harvest object 210 from the captured image PI1 and then calculates the size of the harvest object 210. For example, the harvest judgment unit 126 detects a rectangular bounding box that surrounds the harvest object 210 in the captured image PI1. The harvest judgment unit 126 may calculate the size of the harvest object 210 based on the distance between the end effector 50A (camera 92) and the harvest object 210 and the size of the harvest object 210 on the image.
[0152] The harvest judgment unit 126 may calculate the size of the harvest object 210 on the image by using a bounding box corresponding to the harvest object 210. In one example, the harvest judgment unit 126 extracts the area of the harvest object 210 from the bounding box and then calculates the horizontal size of the harvest object 210 (for example, the size of a representative position, or the average value of the sizes of a plurality of representative positions). The harvest judgment unit 126 may then multiply the vertical size of the bounding box by the calculation result of the horizontal size of the harvest object 210 to calculate the size of the harvest object 210 on the image. Alternatively, the harvest judgment unit 126 may calculate the size of the harvest object 210 on the image by multiplying the vertical size and horizontal size of the bounding box, or may calculate the size of the harvest object 210 on the image from the number of pixels of the area extracted as the harvest object 210.
[0153] Next, the controller 100 executes step S54. In step S54, for example, the harvest determination unit 126 determines whether the harvest target 210 satisfies the harvest criterion based on the result of the size calculation in step S53. The harvest determination unit 126 may determine that the harvest criterion is met when the size of the harvest target 210 is larger than a predetermined threshold, and may determine that the harvest criterion is not met when the size of the harvest target 210 is equal to or smaller than the predetermined threshold.
[0154] If it is determined in step S54 that the harvest criterion is met (step S54: YES), the process executed by the controller 100 proceeds to step S55. In step S55, for example, the part identification unit 128 identifies a first part that is a part that the end effector 50A approaches first based on the captured image PI1. In one example, the part identification unit 128 calculates three candidate parts from within a bounding box detected in the captured image PI1.
[0155] In FIG. 18(b), three candidate parts are indicated by "Cp1", "Cp2", and "Cp3". Candidate part Cp1, candidate part Cp2, and candidate part Cp3 are arranged in this order from the top (top of the image). Candidate part Cp2 is located at the vertical center of the bounding box. Candidate part Cp1 is located midway between the top end of the bounding box and candidate part Cp2. Candidate part Cp3 is located midway between candidate part Cp2 and the bottom end of the bounding box. The horizontal positions of candidate part Cp1, candidate part Cp2, and candidate part Cp3 may each be the center of the bounding box.
[0156] The part identifying unit 128 may identify, as the first part, a part among the candidate parts Cp1, Cp2, and Cp3 that is accessible to the end effector 50A and is closest (highest up) to the handle 220. The part identifying unit 128 may determine whether the end effector 50A can access the candidate part depending on whether an obstacle is present around the candidate part and whether the candidate part is hidden in the captured image PI1.
[0157] In one example, if candidate part Cp1, candidate part Cp2, and candidate part Cp3 are all accessible, part identification unit 128 identifies candidate part Cp1 as the first part. In another example, if candidate part Cp1 is inaccessible due to the presence of an obstacle and the other two candidate parts are accessible, part identification unit 128 identifies candidate part Cp2 as the first part. As described above, candidate parts that are close to the vicinity of handle 220 to be cut are preferentially selected.
[0158] Next, the controller 100 executes step S56. In step S56, for example, the controller 100 executes the first approach control. In the first approach control, the first approach control unit 130 may control the robot 2 so as to bring the end effector 50A closer to the first portion identified in step S55. Details of the first approach control in step S56 will be described later.
[0159] Next, the controller 100 executes step S57. In step S57, for example, the controller 100 executes the second approach control. In the second approach control, the second approach control unit 134 may control the robot 2 to move the end effector 50A from the first part to the second part and to make the end effector 50A hold the second part. Details of the second approach control in step S57 will be described later.
[0160] Next, the controller 100 executes step S58. In step S58, for example, the controller 100 executes harvest control. In the harvest control, the harvest control unit 138 may control the robot 2 to harvest the harvest target 210 with the end effector 50A holding the second portion. Details of the harvest control in step S58 will be described later.
[0161] On the other hand, if it is determined in step S54 that the harvest criterion is not met (step S54: NO), the process executed by the controller 100 proceeds to step S59. In this case, the controller 100 does not execute steps S55 to S58. In step S59, for example, the controller 100 stops the harvesting operation of the harvest target 210. The controller 100 may control the robot 2 to return the end effector 50A to the position before execution of step S51 or to the standby position. After execution of step S59, the controller 100 may execute the processes from step S06 onwards shown in FIG. 8.
[0162] 19 is a flowchart showing a series of processes executed by the controller 100 in the first approach control of step S56. The controller 100 first executes step S61. In step S61, for example, the first approach control unit 130 controls the robot 2 to move the end effector 50A to the first portion specified in step S55. The first approach control unit 130 may control the motor 66a so that the above-mentioned distance d between the first gripping unit 82a and the second gripping unit 82b becomes larger than the width of the harvest target object 210 while the end effector 50A is being moved to the first portion.
[0163] In one example, the first approach control unit 130 controls the robot 2 to move the end effector 50A until a first portion of the harvest target 210 is within the area between the first gripping unit 82a and the second gripping unit 82b. While moving the end effector 50A to the first portion, the controller 100 may move the end effector 50A to the first portion by the articulated arm 10 without obtaining new image data during the movement. Fig. 20(a) illustrates a state after the end effector 50A has been moved to the first portion.
[0164] Next, the controller 100 executes step S62. In step S62, for example, the first approach control unit 130 starts driving the first gripping unit 82a and the second gripping unit 82b by the motor 66a so as to reduce the distance d. By executing step S62, the first gripping unit 82a and the second gripping unit 82b each start to approach so as to come into contact with the first portion of the harvesting target 210. FIG. 20(b) illustrates an example of the first gripping unit 82a and the second gripping unit 82b operating to pinch the harvesting target 210.
[0165] Next, the controller 100 executes step S63. In step S63, for example, the first approach control unit 130 determines whether or not the torque corresponding to the reaction force received by the first gripping unit 82a and the second gripping unit 82b is greater than a threshold value Tht. The first approach control unit 130 may obtain the torque from a control command to the motor 66a or a detection value of the motor 66a. The threshold value Tht is determined in advance, and is determined based on, for example, a torque (actual measurement value or command value) obtained when the first gripping unit 82a and the second gripping unit 82b actually grip a harvest target 210 that satisfies the harvesting standard.
[0166] Next, the controller 100 executes step S64. In step S64, for example, the first approach control unit 130 controls the motor 66a to stop driving the first gripper 82a and the second gripper 82b. When the driving of the first gripper 82a and the second gripper 82b is stopped, if no trouble occurs in the approach of the end effector 50A to the harvest object 210, the first gripper 82a and the second gripper 82b will be in contact with the harvest object 210.
[0167] Next, the controller 100 executes step S65. In step S65, for example, the holding determination unit 132 calculates the interval d from the stop position of the motor 66a at the time of execution of step S64. Then, the holding determination unit 132 determines whether the interval d is greater than a threshold value Thp. The threshold value Thp is determined in advance, and is determined, for example, based on the measurement value of the interval d when the first gripping unit 82a and the second gripping unit 82b actually hold the harvest object 210 that satisfies the harvesting standard. If the interval d is greater than the threshold value Thp, it can be determined that the first gripping unit 82a and the second gripping unit 82b are holding the harvest object 210. On the other hand, if the interval d is equal to or less than the threshold value Thp, it can be determined that the first gripping unit 82a and the second gripping unit 82b are not holding the harvest object 210, but are holding obstacles such as leaves or stems around the harvest object 210, or are not holding anything.
[0168] If it is determined in step S65 that the interval d is greater than the threshold value Thp (step S65: YES), the process executed by the controller 100 proceeds to step S66. In step S66, for example, the first approach control unit 130 drives the first gripping unit 82a and the second gripping unit 82b by the motor 66a so as to increase the interval d. The first approach control unit 130 may control the motor 66a so as to increase the interval d to such an extent that the harvest target 210 once held does not move out of the region between the first gripping unit 82a and the second gripping unit 82b. The amount by which the interval d is increased may be determined in advance. The interval d increased by the execution of step S65 may be smaller than the interval d during the execution of step S61 (during the approach to the first portion).
[0169] On the other hand, if it is determined in step S65 that the interval d is equal to or less than the threshold value Thp (step S65: NO), the process executed by the controller 100 proceeds to step S67. In step S67, for example, the controller 100 stops the harvesting operation of the harvest target 210, similar to step S59. After executing step S67, the controller 100 may execute the process of step S06 and subsequent steps shown in FIG.
[0170] 21 is a flowchart showing a series of processes executed by the controller 100 in the second approach control of step S57. The controller 100 first executes step S71. In step S71, for example, the second approach control unit 134 controls the robot 2 so that the first gripping portion 82a and the second gripping portion 82b of the end effector 50A start moving toward the handle 220.
[0171] In one example, the second approach control unit 134 calculates a path from the first part toward the handle 220 based on the captured image PI1, and controls the robot 2 to start moving the first gripping unit 82a and the second gripping unit 82b along the path. The second approach control unit 134 may control the motor 66a so that the distance d is maintained smaller than the distance d during execution of step S61 even after the movement starts. FIG. 22(a) illustrates the first gripping unit 82a and the second gripping unit 82b being moved toward the handle 220.
[0172] Next, the controller 100 executes step S72. In step S72, for example, the pattern recognition unit 136 starts capturing images by the camera 94. After step S72, the pattern recognition unit 136 may cause the camera 94 to capture images at a predetermined cycle. FIG. 22(b) illustrates a recognition image PI2 obtained by capturing images by the camera 94. The camera 94 may be installed so that the recognition image PI2 captures an upper area near the first gripping portion 82a and the second gripping portion 82b. The pattern recognition unit 136 executes image processing to detect the pattern 220 in the recognition image PI2 obtained each time an image is captured.
[0173] Next, the controller 100 executes step S73. In step S73, for example, the second approach control unit 134 waits until the pattern 220 is recognized (detected) from the recognition image PI2 by the pattern recognition unit 136. By repeating step S73, the first gripping unit 82a and the second gripping unit 82b continue to move toward the pattern 220 until the pattern 220 is detected from the recognition image PI2.
[0174] Next, the controller 100 executes step S74. In step S74, for example, the second approach control unit 134 controls the robot 2 to stop the end effector 50A. At the time of executing step S74, the harvest object 210 may be maintained in a state of being sandwiched between the first gripper 82a and the second gripper 82b. At the time of executing step S74, the orientation of the camera 94 may be adjusted so that the upper end 212a of the harvest object 210 is positioned between the first gripper 82a and the second gripper 82b.
[0175] Next, the controller 100 executes step S75. In step S75, for example, the second approach control unit 134 drives the first gripping unit 82a and the second gripping unit 82b by the motor 66a so that the interval d is reduced and the first gripping unit 82a and the second gripping unit 82b come into contact with the harvest object 210. By executing step S75, a portion of the harvest object 210 corresponding to the stopping position of the first gripping unit 82a and the second gripping unit 82b at the time of executing step S74 (for example, the upper end portion 212a) is held by the first gripping unit 82a and the second gripping unit 82b as the second portion. In step S75, the second approach control unit 134 may stop the driving of the first gripping unit 82a and the second gripping unit 82b based on a torque corresponding to the reaction force received by the first gripping unit 82a and the second gripping unit 82b, similar to steps S62 to S64. As in step S65, the second approach control unit 134 may determine whether the first gripping unit 82a and the second gripping unit 82b are holding the harvest target 210 based on the stopping position of the motor 66a when the drive of the first gripping unit 82a and the second gripping unit 82b is stopped.
[0176] After execution of step S75, the harvest control of step S58 described above is executed. In the harvest control of step S58, for example, the harvest control unit 138 controls the drive unit 76 to advance the blade 72 to cut the handle 220 while the first gripping unit 82a and the second gripping unit 82b hold the upper end 212a as the second part. Figures 23(a), 23(b), 24(a), and 24(b) are photographs showing the operation of the end effector 50A involved in the harvest control.
[0177] The photograph in Fig. 23(a) illustrates the state after step S75 is executed and before the harvest control in step S58 is started, with the holding part 80A holding the upper end part 212a. In Fig. 23(a), "L" is an imaginary line that passes through the blade 72 (more specifically, the blade tip) and follows the direction in which the blade 72 is inserted and removed (advanced and retreated). The imaginary line L can also be said to be a line formed from the movement trajectory of the blade 72 when the blade 72 is driven by the drive part 76 at the current position. The shortest distance on the imaginary line L between the handle 220 and the stalk 202 is indicated by "Di1".
[0178] 23(b) illustrates an example of how the posture of the harvesting target 210 is changed by the operation of the end effector 50A immediately before starting cutting with the blade 72. The harvesting control unit 138 may control the robot 2 to execute the following first and second operations. The execution period of the first operation and the execution period of the second operation may at least partially overlap.
[0179] The first operation is an operation of displacing the end effector 50A by the articulated arm 10 so as to apply tension to the handle 220. The harvest control unit 138, for example, moves the end effector 50A downward by the articulated arm 10 while the holding unit 80A holds the upper end 212a. The distance by which the end effector 50A is moved downward is set, for example, to such an extent that the handle 220 is not pulled and cut, and the stump 201 (seedling) including the harvest target 210 and the handle 220 is not pulled and cut. From the viewpoint of facilitating cutting by the blade 72, the distance by which the end effector 50A is moved downward may be 2 mm or more, 3 mm or more, 4 mm or more, or 5 mm or more. From the viewpoint of avoiding troubles caused by pulling stalk 202 in addition to handle 220 and not applying appropriate tension to handle 220 (for example, cutting of stump 201), the distance by which end effector 50A is moved downward may be 20 mm or less, 18 mm or less, 16 mm or less, or 15 mm or less. In one example, the distance by which end effector 50A is moved downward is 1 mm to 20 mm, or 5 mm to 15 mm. The above-mentioned numerical values are only examples, and the distance by which end effector 50A is moved downward may be appropriately set depending on the type or growth condition of crop 200, etc.
[0180] The second operation is an operation for changing the position and posture of the end effector 50A by the articulated arm 10 so as to increase the shortest distance between the handle 220 and the stalk 202 on the imaginary line L. For example, with the holding part 80A holding the upper end part 212a, the harvesting control part 138 controls the robot 2 to move the rear part of the base part 62 of the end effector 50A upward to tilt the end effector 50A and to lower the entire end effector 50A backward.
[0181] The photograph in FIG. 24(a) illustrates the state after the second operation. The shortest distance between the handle 220 and the stem 202 on the virtual line L is indicated by "Di2". The distance Di2 is greater than the distance Di1. The controller 100 may determine the approach direction (movement path) from the above-mentioned operation start position to the harvesting object 210 so that the handle 220 is farther away from the stem 202 than before the second operation is started by the second operation of moving the rear part of the base part 62 upward and lowering the end effector 50A backward. In one example, the handle 220 may be farther away from the stem 202 than before the second operation is started by the end effector 50A approaching the harvesting object 210 so as to avoid an obstacle, and the second operation may result in the handle 220 being farther away from the stem 202 than before the second operation is started.
[0182] The photograph of FIG. 24(b) illustrates the cutting of the handle 220 by the blade 72. In the portion surrounded by "A", the cutting of the handle 220 by the advance of the blade 72 is shown. On the imaginary line L, the shortest distance between the handle 220 and the stem 202 is widened, so that the possibility that the cutting edge of the advanced blade 72 will hit the stem 202 is reduced. From the viewpoint of both suppressing an increase in the working time and preventing damage to the stem 202, the angle at which the end effector 50A is tilted in the second action may be 2° to 20°, 3° to 18°, or 5° to 15°. From the viewpoint of both suppressing an increase in the working time and preventing damage to the stem 202, the distance by which the end effector 50A is moved backward in the second action may be 1 mm to 10 mm, 1.5 mm to 9 mm, or 2 mm to 8 mm. The above-mentioned numerical values are merely examples, and the amount of movement of end effector 50A in the second operation may be set as appropriate depending on the type or growth condition of crop 200. Harvest control unit 138 may control robot 2 to execute either the first operation or the second operation, instead of both the first operation and the second operation.
[0183] [Variations] In the series of processes shown in FIG. 21, the end effector 50A stops when the pattern recognition unit 136 recognizes the pattern 220, but the first approach control unit 130 may control the robot 2 to move the end effector 50A only a predetermined condition, the contents of which are arbitrarily determined, from the point at which the pattern recognition unit 136 recognizes the pattern 220, and then stop the end effector 50A.
[0184] In step S61 of the first approach control in step S56, unlike the above example, the controller 100 may continue to acquire image data while moving the end effector 50A to the first location. The controller 100 may move the end effector 50A to the first location by the articulated arm 10 based on image data acquired during the movement. In this case, even if the position of the harvest target 210 shifts while the end effector 50A is being moved, the target position can be corrected. Note that, if image data is not acquired during the movement of the end effector 50A to the first location, the amount of image data to be processed is small, and therefore the processing time of the controller 100 is shortened, and as a result, the operation time can be shortened.
[0185] The end effector 50A may include a hand unit having five fingers imitating a human hand as a holding unit, instead of the holding unit 80A. In this case, a pair of gripping units is formed by a part of the hand unit corresponding to the first finger and a part of the hand unit corresponding to the fingers other than the thumb (for example, the second finger). In one example among the various examples described in the second embodiment, at least a part of the matters described in the other examples may be combined.
[0186] At least a part of the matters described in the second embodiment and at least a part of the matters described in the first embodiment may be combined. In the example described in the first embodiment, the controller 100 (harvest control unit 110) controls the robot 2 to harvest the harvest object 210 by different operations depending on whether or not a specific part is recognized. The controller 100 (harvest control unit 110) may control the robot 2 to harvest the harvest object 210 by the operation described in the second embodiment as one of the different operations. In one example, the controller 100 may execute steps S55 to S58 of the series of processes shown in FIG. 17 instead of steps S31 to S33 in the series of processes shown in FIG. 9. Alternatively, the controller 100 may execute step S58 (at least one of the first operation and the second operation) instead of step S33 in the series of processes shown in FIG. 9.
[0187] [Summary of the second embodiment] The present disclosure includes the following configurations [1A] to [15A].
[0188] [1A] A harvesting device 1 comprising: a robot 2 having an end effector; a first approach control unit 130 that controls the robot 2 to bring the end effector 50A close to a first portion of a harvest target 210 of a crop 200; a second approach control unit 134 that controls the robot 2 to move the end effector 50A from the first portion to a second portion of the harvest target 210 and cause the end effector 50A to hold the second portion; and a harvesting control unit 138 that controls the robot 2 to harvest the harvest target 210 with the end effector 50A holding the second portion. In this harvesting device 1, the end effector 50A first approaches a first portion different from the second portion held during harvesting, and can approach the second portion in accordance with the surrounding conditions of the second portion (for example, while avoiding obstacles around the second portion or while moving the obstacles). Therefore, the robot 2 can be operated so that the end effector 50A more appropriately holds the second portion. This can prevent the occurrence of a problem in which the end effector 50A cannot hold the harvest object 210 in a state suitable for harvesting. Therefore, the harvesting device 1 is useful for improving work efficiency.
[0189] [2A] The harvesting device 1 described in [1A] above, wherein the second approach control unit 134 controls the robot 2 to move the end effector 50A from the first portion toward a handle 220 connected to the harvest object 210, and cause the end effector 50A to hold the second portion located between the first portion and the handle 220. Handle 220 is located close to stem 202 of crop 200, and there may be obstacles around handle 220. Even in such a case, by moving end effector 50A from the first portion to the second portion and holding the second portion, end effector 50A can hold the second portion while reducing the influence of obstacles around handle 220.
[0190] [3A] The harvesting device 1 described in [2A] above, wherein the harvest object 210 is formed to extend in the opposite direction from the handle 220, and the second approach control unit 134 controls the robot 2 to move the end effector 50A along the longitudinal direction of the harvest object 210 toward the handle 220, and cause the end (upper end 212a) of the harvest object 210 connected to the handle 220 to be held by the end effector 50A as the second part. For example, from the viewpoint of efficient harvesting, it is conceivable to harvest by directly applying force to the handle 220 while holding the end portion connected to the handle 220. However, there are many obstacles such as main stems or leaves near the handle 220, and there is a concern that when the end effector 50A is moved from a position away from the harvest target 210 to approach the vicinity of the handle 220 to hold the end portion, the influence of the obstacles may prevent the end effector 50A from performing an appropriate holding operation. In contrast, in the above configuration, the end effector 50A first approaches the first portion, which is located at a position different from the second portion (the end portion near the handle 220), and then moves the end effector 50A along the longitudinal direction of the harvesting target 210 to the second portion to hold the second portion. Therefore, even if there is an obstacle around the second portion, the obstacle can be moved when the end effector 50A is moved from the first portion to the second portion. This can reduce the occurrence of a problem in which the end effector 50A cannot properly hold the second portion due to an obstacle around the second portion.
[0191] [4A] The harvesting device 1 described in [2A] above, further comprising an area identification unit 128 that identifies an area of the harvest object 210 that is accessible by the end effector 50A as the first area based on one or more images obtained by capturing an image of at least a portion of the crop 200. In this case, the end effector 50A can more reliably approach a location other than the second portion.
[0192] [5A] The harvesting object 210 is formed to extend in the opposite direction from the handle 220, and the second approach control unit 134 controls the robot 2 to move the end effector 50A along the longitudinal direction of the harvesting object 210 toward the handle 220 and cause the end effector 50A to hold the end part (upper end 212a) of the harvesting object 210 connected to the handle 220 as the second part, and the part identification unit 128 identifies the first part from the area located between the end part (upper end 212a) of the harvesting object 210 connected to the handle 220 and the end part (lower end 212b) opposite the handle 220, the harvesting device 1 described in [4A] above. In this case, the distance that the end effector 50A travels from the first portion to the second portion is shorter than when the end portion (lower end portion 212b) opposite the handle 220 is set as the first portion and the end portion is approached first, which is useful for further improving the efficiency of the work.
[0193] [6A] The harvesting device 1 described in [5A] above, wherein the part identification unit 128 identifies two or more candidate parts (Cp1, Cp2, Cp3) in the harvest object 210 based on any of the one or more images, and identifies, among the two or more candidate parts (Cp1, Cp2, Cp3), a part that is accessible to the end effector 50A and is closest to the handle 220 as the first part. In this case, it is useful for reducing the occurrence of a problem in which the end effector 50A cannot properly hold the harvest object 210, and for improving the efficiency of the work.
[0194] [7A] The end effector 50A includes a pair of gripping portions (first gripping portion 82a and second gripping portion 82b) that hold the harvest object 210 by sandwiching it from the sides, and the first approach control portion 130 moves the end effector 50A to the first location by an arm (articulated arm 10) possessed by the robot 2 so that the first location is located between the pair of gripping portions (first gripping portion 82a and second gripping portion 82b), and the second approach control portion 134 controls the robot 2 to move the end effector 50A to the second location while maintaining the state in which the harvest object 210 is located between the pair of gripping portions (first gripping portion 82a and second gripping portion 82b). The harvesting device 1 described in any one of [1A] to [6A] above. In this case, when the end effector 50A moves from the first location to the second location, the possibility of the harvest object 210 being no longer able to be held by the pair of gripping portions (the first gripping portion 82a and the second gripping portion 82b) can be reduced.
[0195] [8A] The first approach control unit 130 controls the robot 2 to bring the end effector 50A closer to the first portion and reduce the distance d between the pair of gripping portions (the first gripping portion 82a and the second gripping portion 82b) to contact the first portion, as described in [7A] above. In this case, after confirming whether the end effector 50A has been able to access the harvest object 210, the harvesting operation of the harvest object 210 can be continued. For example, if the end effector 50A is unable to access the harvest object 210, the harvesting operation can be interrupted. This is useful for further improving the efficiency of the work, since the operation for the next harvest can be started sooner after the occurrence of the trouble. Note that, from the viewpoint of reducing the processing load of the controller 100, it is particularly beneficial to use the above configuration, since troubles of inaccessibility are more likely to occur when the harvest object 210 is recognized by the camera but control is not performed to access the harvest object 210.
[0196] [9A] The end effector 50A further includes a holding determination unit 132, and further includes a motor 66a that changes the distance d between the pair of gripping portions (first gripping portion 82a and second gripping portion 82b). The first approach control unit 130 controls the arm (articulated arm 10) to bring the pair of gripping portions (first gripping portion 82a and second gripping portion 82b) closer to the first region, and controls the motor 66a to move the pair of gripping portions (first gripping portion 82a and second gripping portion 82b) closer to the first region so that the distance d between the pair of gripping portions (first gripping portion 82a and second gripping portion 82b) becomes smaller. The harvesting device 1 described in [8A] above drives the pair of gripping parts (first gripping part 82a and second gripping part 82b) and stops the driving by the motor 66a based on the reaction force received by the pair of gripping parts (first gripping part 82a and second gripping part 82b), and the hold determination unit 132 determines whether the pair of gripping parts (first gripping part 82a and second gripping part 82b) are holding the harvest target 210 based on the position of the motor 66a when the drive of the pair of gripping parts (first gripping part 82a and second gripping part 82b) by the motor 66a is stopped. In this case, it is possible to easily check whether the first portion is being held or not without using other devices such as a camera.
[0197] [10A] The harvesting device 1 described in any one of [7A] to [9A] above, wherein the end effector 50A further includes a motor 66a that changes the distance between a pair of gripping portions (first gripping portion 82a and second gripping portion 82b), and the second approach control portion 134 controls the motor 66a so that the distance d between when the end effector 50A is moved from the first location to the second location is smaller than the distance d between when the first approach control portion 130 moves the end effector 50A to the first location. In this case, when the end effector 50A moves from the first location to the second location, the possibility of the state in which the harvest object 210 can be held by the pair of gripping portions (the first gripping portion 82a and the second gripping portion 82b) being lost (the state in which the harvest object 210 is positioned between the pair of gripping portions) can be further reduced.
[0198] [11A] The harvesting device 1 described in any one of [2A] to [6A] above, wherein the end effector 50A includes a holding portion 80A capable of holding the harvest object 210, and cutting portions 70, 70A capable of cutting the vicinity of the portion of the harvest object 210 held by the holding portion 80A, and the harvest control portion 138 controls the robot 2 to cut the handle 220 with the cutting portions 70, 70A while the holding portion 80A holds the second portion. In this case, an external force is applied directly to handle 220 to separate harvest object 210 from crop 200, thereby improving the efficiency of harvesting work.
[0199] [12A] The harvesting control unit 138 controls the robot 2 to hold the second portion using the holding unit 80A, and to displace the end effector 50A to apply tension to the handle 220 while causing the cutting units 70, 70A to cut the handle 220, as described in [11A] above. If a cutting force is applied directly to the handle 220, a problem may occur in which the handle 220 bends and becomes impossible to cut. In the above configuration, the handle 220 is cut while tension is applied to the handle 220, so the occurrence of such a problem can be suppressed and the success rate of the harvest can be improved.
[0200] [13A] The cutting unit 70, 70A includes a blade 72 and a drive unit 76 that drives the blade 72, and the harvesting control unit 138 controls the robot 2 to move the harvesting object 210 away from the stem 202 connected to the handle 220 in a direction intersecting the handle 220 after the holding unit 80A holds the second portion via the second approach control unit 134, and drives the blade 72 via the drive unit 76 so as to cut the handle 220 with the blade 72, in the harvesting device 1 described in [11A] or [12A] above. There is a risk that the driving of the blade 72 may damage the stem 202 (e.g., the main stem) around the handle 220. In contrast, in the above configuration, the handle 220 is moved away from the stem 202 and the blade 72 is driven, so that the possibility of damaging the stem 202 can be reduced.
[0201] [14A] The harvesting device 1 described in any one of [2A] to [6A] above, further comprising a camera (camera 94) provided at the tip of the arm (articulated arm 10) of the robot 2 or on the end effector 50A, and a handle recognition unit 136 that recognizes the handle 220 based on a recognition image PI2 obtained from the camera (camera 94) while the end effector 50A moves from the first portion toward the handle 220, wherein the second approach control unit 134 controls the robot 2 to sequentially stop the end effector 50A based on the recognition result of the handle 220 by the handle recognition unit 136, and cause the end effector 50A to hold the portion of the harvest object 210 corresponding to the stopping position of the end effector 50A as the second portion. In this case, the handle 220 is more likely to be recognized than a camera installed farther from the harvest object 210 than the tip of the arm or the end effector 50A (for example, a camera installed on the cart 4). As a result, the success rate of harvesting can be improved.
[0202] [15A] A harvesting device 1 described in any one of [1A] to [14A] above, further comprising a harvest assessment unit 126 that executes a harvest assessment process before the first approach control unit 130 moves the end effector 50A, the harvest assessment process including calculating the size of the harvest target 210 based on one or more images obtained by capturing an image of at least a portion of the crop 200, and determining whether or not to harvest the harvest target 210 based on the result of the size calculation. In this case, small-sized harvest objects 210 are not harvested, so that it is possible to achieve both automation of the work and harvesting work according to the growth status of the crops 200.
[0203] [16A] A method for controlling a harvesting device 1, comprising: controlling the robot 2 to bring an end effector 50 held by the robot 2 close to a first portion of a harvest target 210 of a crop 200; controlling the robot 2 to move the end effector 50A from the first portion to a second portion of the harvest target 210 so as to cause the end effector 50A to hold the second portion; and controlling the robot 2 to harvest the harvest target 210 with the end effector 50A holding the second portion. In this control method, similar to the harvesting device 1 described in [1A] above, it is possible to suppress the occurrence of a problem in which the end effector 50A is unable to hold the harvest target 210 in a state suitable for harvesting. Therefore, the above control method is useful for improving work efficiency. [Explanation of symbols]
[0204] 1...harvesting device, 2...robot, 4...cart, 6...storage section, 8...imaging device, 10...articulated arm, 26...axis, 50, 50A...end effector, 60, 60A...main body, 70, 70A...cutting section, 72...blade, 74...cover member, 76...driving section, 80...support section, 80A...holding section, 82a...first gripping section, 82b...second gripping section, d...spacing, 88...axis, 92, 94... Camera, 100...controller, 102...imaging control unit, 104, 122...target detection unit, 106...cart control unit, 108...recognition unit, 110, 138...harvesting control unit, 112...storage control unit, 118...information notification unit, 126...harvesting judgment unit, 128...part identification unit, 130...first approach control unit, 132...holding judgment unit, 134...second approach control unit, 136...pattern recognition unit.
Claims
1. a robot having an end effector; a first approach control unit that controls the robot so as to bring the end effector closer to a first portion of a harvest target of a crop; a second approach control unit that controls the robot to move the end effector from the first portion to a second portion of the harvesting object and to cause the end effector to hold the second portion; a harvesting control unit that controls the robot to harvest the harvest object while the end effector holds the second portion; A harvesting device comprising:
2. The harvesting device of claim 1, wherein the second approach control unit controls the robot to move the end effector from the first portion toward a handle connected to the harvesting object and cause the end effector to hold the second portion located between the first portion and the handle.
3. The harvesting object is formed to extend in an opposite direction from the stem, The harvesting device of claim 2, wherein the second approach control unit controls the robot to move the end effector along the longitudinal direction of the harvest object toward the handle and to cause the end effector to hold an end of the harvest object connected to the handle as the second part.
4. The harvesting device of claim 2, further comprising a part identification unit that identifies a part of the harvest object that is accessible to the end effector as the first part based on one or more images obtained by capturing at least a portion of the crop.
5. The harvesting object is formed to extend in an opposite direction from the stem, the second approach control unit controls the robot to move the end effector toward the handle along the longitudinal direction of the harvesting object and to cause the end effector to hold, as the second portion, an end of the harvesting object that is connected to the handle; The harvesting device according to claim 4 , wherein the part identification unit identifies the first part from an area located between an end of the harvest object connected to the handle and an end of the harvest object opposite the handle.
6. The part specifying unit is Identifying two or more candidate sites in the harvest object based on any of the one or more images; The harvesting device of claim 5 , further comprising: identifying, as the first portion, a portion of the two or more candidate portions that is accessible to the end effector and is closest to the handle.
7. The end effector includes a pair of gripping portions that hold the harvesting object by sandwiching it from the sides, the first approach control unit moves the end effector to the first portion by an arm of the robot so that the first portion is located between the pair of gripping portions; The harvesting device of claim 1 , wherein the second approach control unit controls the robot to move the end effector to the second location while maintaining the harvest object positioned between the pair of gripping units.
8. The harvesting device according to claim 7 , wherein the first approach control unit controls the robot to bring the end effector closer to the first portion and to reduce a gap between the pair of gripping portions so as to contact the first portion.
9. A retention determination unit is further provided, The end effector further includes a motor that changes the distance between the pair of grippers, The first approach control unit is moving the pair of gripping portions closer to the first portion by the arms; Driving the pair of gripping portions by the motor so that a distance between the pair of gripping portions becomes small; and stopping the driving by the motor based on a reaction force received by the pair of gripping portions. The harvesting device described in claim 8, wherein the holding determination unit determines whether the pair of grippers holds the harvest target based on the position of the motor when the motor stops driving the pair of grippers.
10. The end effector further includes a motor that changes the distance between the pair of grippers, 8. The harvesting device of claim 7, wherein the second approach control controls the motor such that the interval between moving the end effector from the first location to the second location is smaller than the interval between moving the end effector to the first location by the first approach control.
11. The end effector includes a holding portion capable of holding the harvesting object, and a cutting portion capable of cutting the harvesting object in the vicinity of a portion held by the holding portion, A harvesting device as described in any one of claims 2 to 6, wherein the harvesting control unit controls the robot to cut the stem with the cutting unit while the holding unit holds the second part.
12. The harvesting device of claim 11, wherein the harvesting control unit controls the robot to hold the second portion with the holding unit and to displace the end effector to apply tension to the handle and to cut the handle with the cutting unit.
13. The cutting unit includes a blade and a cutting drive unit that drives the blade. The harvest control unit is After the holding unit holds the second portion by the second approach control unit, the robot is controlled to move the harvesting object away from the stem connected to the handle in a direction intersecting the handle; and driving the blade with the cutting drive such that the blade cuts the stalk.
14. a camera provided at the tip of an arm of the robot or at the end effector; a pattern recognition unit that recognizes the pattern based on a recognition image obtained from the camera while the end effector is moving from the first portion toward the pattern, The second approach control unit is A harvesting device as described in any one of claims 2 to 6, wherein the robot is controlled to sequentially stop the end effector based on the recognition result of the handle by the handle recognition unit, and cause the end effector to hold the part of the harvest object corresponding to the stopping position of the end effector as the second part.
15. a harvest determination unit that executes a harvest determination process before the first approach control unit moves the end effector, A harvesting device as claimed in any one of claims 1 to 10, wherein the harvest assessment process includes calculating a size of the harvest object based on one or more images obtained by capturing at least a portion of the crop, and determining whether or not to harvest the harvest object based on a result of the size calculation.
16. Robots and an object detection unit that detects an object to be harvested from an image obtained by capturing an image of at least a portion of the agricultural crop; A recognition unit that recognizes a specific part of the detected harvest object; A harvesting device comprising: a harvesting control unit that controls the robot to harvest the harvest object by different operations depending on whether or not the specific portion is recognized.
17. Robots and an object detection unit that detects an object to be harvested from an image obtained by capturing an image of at least a portion of the agricultural crop; A recognition unit that recognizes a specific part of the detected harvest object; a harvesting control unit that controls the robot to harvest the harvest object when the specific portion is recognized; A harvesting device comprising: an information notification unit that transmits individual information of the harvest object to another device if the specific portion is not recognized.
18. Controlling the robot so as to bring an end effector of the robot closer to a first portion of a harvest target of the agricultural crop; controlling the robot to move the end effector from the first portion to a second portion of the harvesting object and cause the end effector to hold the second portion; Controlling the robot to harvest the harvesting object while holding the second portion with the end effector; A method for controlling a harvesting device, comprising:
19. Detecting a harvest target from an image obtained by capturing an image of at least a portion of the crop; Recognizing a specific part of the detected harvest object; and controlling a robot to harvest the harvest object by different operations depending on whether or not the specific portion is recognized.
Citation Information
Patent Citations
Fruit vegetable harvesting device
JP1996103138A
Fruit vegetable separation device and fruit vegetable separation method
JP2021023164A
Cited By
Multilegged walking robot and walking method
JP2026140217A
Multilegged walking robot and walking method
JP7831805B1
Hand device and hand system
WO2026164266A1
