Item Acquisition System
The article acquisition system addresses the challenge of uniform crop selection and positioning by using an imaging and control system to remotely manage crop acquisition, improving yield management and efficiency.
Patent Information
- Application Number
- JP2022529862
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2021-04-23
- Filing Date
- 2022-02-24
- Publication Date
- 2024-11-14
- Estimated Expiration
- 2042-02-24
AI Technical Summary
Existing article acquisition systems, particularly for agricultural crops, face challenges in uniformly selecting crops for harvesting or thinning and determining the precise positions for crop acquisition, leading to inefficiencies in yield management.
An article acquisition system equipped with an imaging section, arm, control section, and display unit that allows remote selection and positioning of crops for harvesting or thinning, using a gantry-type robot to move the arm to the designated location for acquisition.
Enables remote and uniform sorting of articles, improving yield management by allowing precise selection and positioning of crops for harvesting or thinning, enhancing productivity and efficiency.
Smart Images

Figure 0007583457000001 
Figure 0007583457000002 
Figure 0007583457000003
Abstract
Description
[Technical field]
[0001] The present disclosure relates to an item acquisition system. [Background technology]
[0002] Conventionally, an item acquisition system for acquiring items such as agricultural products is known (see, for example, Patent Document 1). In the agricultural product management system of Patent Document 1, a gantry-type robot is used to acquire the agricultural products. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2020-89345 A Summary of the Invention [Problem to be solved by the invention]
[0004] For example, it is necessary to judge the growth status of agricultural crops, select the crops as necessary, and harvest or thin them out, etc. Since the selection of whether to harvest or thin such crops affects the yield, it is difficult to uniformly select the crops using a robot, etc. In addition, it is difficult to uniformly determine which part of the crop to grasp or cut off in order to harvest or thin the crop.
[0005] An object of the present disclosure is to provide an item acquisition system that can acquire items remotely. [Means for solving the problem]
[0006] The object acquisition system according to the present disclosure includes an imaging unit that images an object, an arm that can acquire the object, a control unit that determines a shape of the object from image data captured by the imaging unit, and a display unit that allows a user to select whether or not to acquire the object. When the control unit determines to acquire the object imaged by the imaging unit, the control unit displays a position of the object on the display unit and displays a screen for inputting an acquisition position for acquiring the object, and when the acquisition position is input, the control unit moves the arm toward the position, and then moves the arm to align it with the acquisition position, thereby acquiring the object. Effect of the Invention
[0007] According to the present disclosure, it is possible to provide an item acquisition system that can remotely select items. [Brief description of the drawings]
[0008] [Figure 1] FIG. 1 is a system diagram illustrating a crop management system according to a first embodiment of the present disclosure. [Diagram 2] FIG. 1 is a side view showing the layout of a crop management system according to a first embodiment of the present disclosure. [Diagram 3] FIG. 1 is a top view showing the layout of a crop management system according to a first embodiment of the present disclosure. [Figure 4] 4 is a flowchart showing a process performed by a control device according to the first embodiment of the present disclosure. [Diagram 5] 4A to 4C are diagrams showing an example of strawberries in image processing performed by the control unit according to the first embodiment of the present disclosure. [Figure 6] 4A and 4B are diagrams illustrating an example of a cucumber in image processing performed by a control unit according to the first embodiment of the present disclosure. [Figure 7] FIG. 4 is a diagram showing an example of a screen displayed on a display unit according to the first embodiment of the present disclosure. [Figure 8] FIG. 4 is a diagram showing an example of a screen displayed on the display unit when thinning out branches according to the first embodiment of the present disclosure. [Figure 9] FIG. 4 is a diagram showing an example of image data acquired by a control unit according to the first embodiment of the present disclosure. [Figure 10] 1 shows an item sorting system according to a second embodiment of the present disclosure. [Figure 11] 10 is a flowchart showing a process performed by a control device according to a second embodiment of the present disclosure. [Figure 12] FIG. 11 is a schematic diagram showing an example of a selection screen according to a second embodiment of the present disclosure. [Figure 13] 5 is a flowchart for controlling an acquisition position according to the first and second embodiments of the present disclosure. [Figure 14] FIG. 11 is a diagram showing an example of displaying an acquisition position on a selection screen according to a second embodiment of the present disclosure. [Figure 15] FIG. 11 is a diagram showing another example of displaying an acquisition position on a selection screen according to the second embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0009] First Embodiment Hereinafter, a first embodiment of the present disclosure will be described with reference to the drawings. The item acquisition system according to the first embodiment is a crop management system. In this embodiment, the longitudinal direction of a vinyl greenhouse (one example of a greenhouse) 10 shown in Figures 2 and 3 is referred to as the Y direction, the lateral direction as the X direction, and the up-down direction as the Z direction in the specification and drawings.
[0010] As shown in FIG. 1, a crop management system 1 according to the present disclosure includes a robot 2, a camera (an example of an imaging unit) 4, a control device 6 (an example of a control unit), and a display unit 8.
[0011] The robot 2 has a boom 2a, a camera arm 2b, a harvesting arm 2c, and a harvesting chuck 2d provided at the tip of the harvesting arm 2c. The boom 2a, the camera arm 2b, the harvesting arm 2c, and the harvesting chuck 2d have a first actuator 2e, a second actuator 2f, a third actuator 2g, and a fourth actuator 2h for moving them, respectively.
[0012] 2 and 3, in this embodiment, the robot 2 is a gantry type robot that is installed in a vinyl greenhouse 10 in which the agricultural crop A is grown. Such a gantry type robot 2 is installed above and straddling the ridges in which the agricultural crop A is planted.
[0013] The robot 2 has multiple support posts 21 and at least two rails 22. The rails 22 extend in the longitudinal direction of the vinyl greenhouse 10 and are fixed to the upper parts of the multiple support posts 21. The boom 2a is suspended between the two rails 22. The boom 2a is driven by a first actuator 2e and can freely move on the rails 22 in the Y direction.
[0014] The camera arm 2b is attached to the boom 2a. The camera arm 2b is driven by a second actuator 2f and moves freely in the X and Z directions relative to the boom 2a toward the crops A. A camera 4 is attached to the tip of the camera arm 2b.
[0015] The harvesting arm 2c is attached to the boom 2a. The harvesting arm 2c is driven by the third actuator 2g shown in FIG. 1 and moves freely in the X and Z directions relative to the boom 2a, moving toward the crop A. A harvesting chuck 2d is attached to the tip of the harvesting arm 2c. The harvesting chuck 2d can cut and grab the crop A. In this embodiment, the harvesting chuck 2d is an end effector that rotates, for example, around the X axis relative to the harvesting arm 2c and has claws that open and close. A blade may be attached to the tip of the claw. The harvesting chuck 2d can access the crop A from both sides and above the crop A. The harvesting chuck 2d is driven by the fourth actuator 2h, and the claws are opened, closed, and rotated. The harvesting chuck 2d thus rotates and uses the claws and blades to cut the crop A. The camera arm 2b and the harvesting arm 2c may be a single arm that moves toward the crop A.
[0016] Camera 4 is a device for capturing images of crop A. Camera 4 is, for example, a high-resolution camera that can capture detailed images of the color, shape, etc. of crop A. Camera 4 is electrically connected to control device 6, captures images of crop A, and transmits image data to control device 6. Camera 4 is capable of capturing images of both sides and above crop A.
[0017] As shown in FIG. 1, the control device 6 is connected to the first actuator 2e to the fourth actuator 2h, the camera 4, and the like, and controls these devices. The control device 6 also performs control to manage the growth state of the crop A using image data of at least the crop A captured by the camera 4. The control device 6 is actually configured by a microcomputer including a calculation device (an example of a calculation unit) 6a, a memory (an example of a storage unit) 6b, a communication interface (an example of a communication unit) 6c, an input / output buffer, and the like. The control device 6 executes various controls using software stored in the memory 6b. The control device 6 may be electrically connected to sensors and systems such as a soil sensor 6f that detects the pH value, moisture, and temperature of the soil in which the crop A is planted, a sunshine sensor 6g that detects the sunshine intensity, and a temperature control system 6h in the vinyl house 10.
[0018] The control device 6 has an image processing unit 6d and an actuator control unit 6e. The image processing unit 6d and the actuator control unit 6e are functional configurations realized by software stored in the memory 6b. The image processing unit 6d acquires the size, color, and shape of the crop A from the image data acquired by the camera 4. The actuator control unit 6e controls each actuator to move the boom 2a and each arm to a predetermined position in the vinyl greenhouse 10.
[0019] The control device 6 records the harvest forecast data in the memory 6b. More specifically, the control device 6 records the harvestable number of crop A from data such as the number of seeds or seedlings planted for crop A, the number of sunshine days, the state of the soil, and the number of growing days. The control device 6 compares this information with the number of sunshine days and the changes in the state of the soil up to the present time, creates harvest forecast data that increases or decreases the estimated harvest number, and records it.
[0020] The display unit 8 displays a screen that allows the user to select whether to harvest or thin the crop A. When thinning is selected, the display unit 8 displays a screen that prompts the user to decide whether to thin the branches. When the user thins the branches, the display unit 8 displays a screen for determining the line at which the branches are to be cut off. That is, thinning the crop A includes not only the harvested product such as fruit, but also the selection and cutting of branches around the harvested product. The display unit 8 may also display a screen required by the management user who operates the crop management system 1. The display unit 8 is, for example, a touch screen type liquid crystal panel of a smartphone, a tablet, or the like, or a liquid crystal screen of a desktop personal computer. The display unit 8 is electrically connected to the control device 6 by wired or wireless communication. In this embodiment, the display unit 8 is wirelessly connected to the control device 6 via the Internet 9.
[0021] Next, a control procedure executed by the control device 6 will be described with reference to the flowchart of FIG.
[0022] As shown in Fig. 4, the control device 6 moves the camera arm 2b to capture an image of the crop A with the camera 4 (step S1). The control device 6 acquires image data of the captured image (step S2). The control device 6 continues to monitor the size, color, and shape of the crop A by periodically acquiring images of all the crops A in the vinyl greenhouse 10. Upon acquiring the image data, the control device 6 records the position where the image data was acquired in the memory 6b together with the image data (step S3).
[0023] The control device 6 processes the image data and judges the growth status (step S4). More specifically, the control device 6 processes the captured image data of the crop A, compares it with reference data that is data on the ideal size, color, and shape of the crop A according to the number of growing days, and judges the growth status in three stages, for example, "good", "needs observation", and "poor". The judgment is not limited to three stages, and may be any number of stages. After judging the growth status, the control device 6 judges whether thinning is necessary (step S5).
[0024] Specifically, as shown in FIG. 5, for example, if crop A is strawberries, the control device 6 compares the strawberries in the image data with reference data that defines the size, color, and shape of strawberries according to the number of growing days, which is recorded in the harvest forecast data. The control device 6 compares with the reference data and determines whether the strawberries in the image data match the reference data at a predetermined rate or more (90% or more in this embodiment). For example, the color of the strawberry on the far left of FIG. 5 is slightly blue, and the proportion of this blue color matches only about 70% of the red color in the reference data. In this case, the control device 6 determines that the growth condition of the strawberry on the far left of FIG. 5 is "poor."
[0025] On the other hand, suppose that the color of the strawberry at the very center of FIG. 5 roughly matches the reference data, but the size only matches about 80% of the reference data. In this case, the control device 6 judges the growth state of the strawberry at the center of FIG. 5 to "requires observation." Crop A shown in FIG. 6 is a cucumber. Suppose that the shape of the cucumber on the right side of FIG. 6 only matches about 80% of the reference data. In this case, the control device 6 judges the cucumber on the right side of FIG. 6 to "requires observation."
[0026] When the control device 6 determines that thinning is necessary (step S5: YES), the process proceeds to step S13. In this embodiment, when the control device 6 determines in step S5 that the growth condition is "needs observation" or "poor", it determines that thinning is necessary. On the other hand, when the control device 6 determines that thinning is not necessary (step S5: NO), the process proceeds to step S6. In this embodiment, the control device 6 determines that thinning is not necessary when the growth condition of the crop A for which the image was acquired is "good".
[0027] The control device 6 determines whether harvesting is necessary (step S6). When all items of size, color, and shape match 90% or more with the reference data and when the number of growing days has reached a predetermined number of days, the control device 6 determines that harvesting is necessary.
[0028] On the other hand, even if all the items of size, color, and shape match the reference data by 90% or more, there are cases where it is preferable to allow the crop to mature further. It is preferable that such a judgment be made by the management user of the crop management system 1. Therefore, in such a case, it is preferable that the control device 6 judges that the management user needs to judge whether harvesting is necessary. Therefore, the control device 6 judges whether harvesting approval confirmation is necessary (step S7). In this embodiment, for example, the control device 6 judges that harvesting approval confirmation is necessary for crop A, which matches the reference data by 90% or more in all the items of size, color, and shape, but at least one of size, color, and shape is between 90% and 95%.
[0029] When the control device 6 determines that confirmation of harvest approval is necessary (Yes in step S7), it displays the harvest approval screen 12 on the display unit 8 (step S8). More specifically, the control device 6 generates the harvest approval screen 12, transmits it to the display unit 8 via the communication interface 6c, and causes it to be displayed. FIG. 7 is an example of the harvest approval screen 12. The harvest approval screen 12 includes, for example, information 13 of image data of the crop A captured by the camera 4, a frame 14 displaying information of the reference data, a button 15 for approving the harvest, a button 16 for approving thinning, and information 17 displaying the position where the image data was acquired. The administrative user of the crop management system 1 can select whether to harvest or not by touching or clicking the button 15 or button 16 on the harvest approval screen 12 displayed on the display unit 8.
[0030] The control device 6 determines whether the harvest has been approved by the administrative user (step S9). If the control device 6 determines that the harvest has been approved by the administrative user (step S9 Yes), it moves the harvesting arm 2c to the position where the image of the crop A was acquired, and harvests the crop A (step S10). The control device 6 records the harvest position (step S11), reduces the harvest amount from the harvest forecast data, and removes the harvested crop A from the monitoring target (step S12), and returns the process to before step S1.
[0031] If the administrative user determines that harvesting is not necessary (step S6 NO) or that harvesting is not permitted (step S9 NO), the control device 6 returns the process to before step S1. If the control device 6 determines that harvesting permission confirmation is not necessary (step S7 NO), the control device 6 advances the process to step S10 and moves the harvesting arm 2c.
[0032] Next, a case where it is determined in step S5 that thinning is necessary will be described. When it is determined that thinning is necessary (YES in step S5), the control device 6 determines whether thinning acceptance confirmation is necessary or not (step S13).
[0033] If the control device 6 determines that the crop A is to be uniformly thinned, the yield will decrease and the productivity of the crop A will deteriorate. Therefore, when the control device 6 determines that the management user needs to make a decision regarding the crop A, it determines that thinning approval confirmation is necessary. In this embodiment, for example, the control device 6 determines that thinning approval confirmation is necessary for the crop A that has been determined to be "requiring observation" in the growth status determination in step S4.
[0034] When the control device 6 determines that thinning approval is necessary (Yes in step S13), it displays a thinning approval screen on the display unit 8 (step S14). More specifically, the control device 6 generates a thinning approval screen, transmits it to the display unit 8 via the communication interface 6c, and causes it to be displayed. In this embodiment, the thinning approval screen is the same as the harvest approval screen 12. However, the thinning approval screen may be displayed on a screen separate from the harvest approval screen 12. A user of the agricultural crop management system 1 can select whether or not to thin by touching or clicking the button 16.
[0035] When the thinning is approved by the management user, the control device 6 may display a screen for prompting the management user to decide whether or not to thin the branches. In this embodiment, the control device 6 prompts the management user to decide whether or not to thin the branches by displaying a button 8 for specifying the thinning position of the branches on the harvest feasibility screen 12. When the button 18 is touched or clicked by the management user, the control device 6 displays a screen 19 for specifying the thinning position of the branches. On the screen 19, the management user specifies the thinning position of the branches. For example, the management user specifies the position (obtaining position) of the branches to be thinned by touching and tracing (see the arrow from X to Y in FIG. 8). The control device 6 converts the line traced by the management user into coordinates by image processing. The control of the obtaining position will be described together with the second embodiment.
[0036] The control device 6 determines whether or not the thinning has been approved (step S15). If the thinning has been approved (step S15 Yes), the control device 6 moves the harvesting arm 2c to the position where the image of the crop A was acquired, and thins out the crop A (step S15). If branches are to be thinned, the control device 6 moves the harvesting arm 2c towards the branches and cuts them off. The control device 6 records the thinning position (step S16), reduces the harvest number from the harvest forecast data, and returns the process to before step S1.
[0037] If the thinning is not accepted (step S15 NO), the control device 6 returns the process to before step S1. If the control device 6 determines that thinning acceptance confirmation is not necessary (step S13 NO), the control device 6 advances the process to step S16 and moves the harvesting arm 2c.
[0038] In this embodiment, the control device 6 processes the images of the crop A captured by the camera 4 for each captured image to generate the harvest propriety screen 12, but the present disclosure is not limited to this. That is, as shown in Fig. 9, the control device 6 may capture image data f1 to f10 by the camera 4 while moving the camera arm 2b. Furthermore, the control device 6 may combine the image data f1 to f10 in advance and acquire them as one image data F. The control device 6 may acquire one image data F for one ridge where the crop A is planted, or may acquire one or multiple image data F for any section.
[0039] Furthermore, although the control device 6 immediately displays the generated harvest possibility screen 12 on the display unit 8, the present disclosure is not limited to this. That is, the control device 6 may image the agricultural crop A with the camera 4 at a predetermined time, generate the harvest possibility screen 12, and then display the harvest possibility screen 12 on the display unit 8 after any time has passed. For example, the control device 6 acquires image data F in advance during a time period when the imaging conditions for the camera 4 are favorable, and generates the harvest possibility screen 12. After that, after any time has passed, the administrative user may select whether to harvest while looking at the harvest possibility screen 12 displayed on the display unit 8. That is, the administrative user may select whether to harvest by displaying the previously generated harvest possibility screen 12 on the display unit 8 at any time.
[0040] This allows the management user to select whether or not to harvest based on the pre-created image data F 1, regardless of the interval or duration of image capture by the camera 4. This allows the management user to select whether or not to harvest the crop A at any time. As a result, the crop management system 1 can perform work remotely at any time and at any time selected by the management user.
[0041] <Second embodiment> As shown in FIG. 10, the item acquisition system according to the second embodiment of the present disclosure is an item sorting system 101. The item sorting system 101 includes a gripping device 102, a camera 104, a control device 106, a display 108, an input device 1010, and a sorting area 1012. In this embodiment, the item sorting system 101 is a waste sorting system that sorts a plurality of wastes X (an example of an item) that are mixed with various materials and shapes, such as wood, metal, rubber, glass, and resin, that are brought into a waste disposal site. When such wastes X are brought into the waste disposal site, they are temporarily piled up in a state where materials or materials Y (an example of an item type) are mixed regardless of the type of wastes X, and rough sorting (an example of a sorting process) is performed. In the rough sorting, the item sorting system 1 sorts the plurality of wastes X into predetermined materials or materials Y.
[0042] The gripping device 102 has an actuator (not shown) and an arm (one example of a gripping unit) 102a. In this embodiment, the gripping device 102 is a gantry-type robot. In such a gantry-type robot, the boom 102b moves forward and backward on a pair of rails 102c by an actuator, and the arm 2a attached to the boom 102b moves left and right and up and down. The rails 102c may be attached to a structural member of a building of a waste treatment plant to which multiple wastes X are transported. The transport device is electrically connected to a control device 106 (described later), and the actuator is controlled by the control device 106 to grip multiple wastes X with the arm 102a and move them to a predetermined position. The gripping device 102 is not limited to a gantry-type robot, and may be a device such as a robot arm. A plurality of gripping devices 102 may be provided.
[0043] The camera 104 captures an image of the waste X. In this embodiment, the camera 104 is attached to the boom 102b of the gripping device 102. The camera 104 may also be attached to a ceiling or beams of a factory where the sorting process is performed. However, the camera 104 may also be attached to a location other than the boom 102b of the gripping device 102. The camera 104 captures an image of the waste X contained in the imaging area A. The camera 104 may move in conjunction with the movement of the boom 102b and repeatedly capture still images at regular intervals. The camera 104 is electrically connected to the control device 106 described later, and transmits image data D, which is a captured video or still image (an example of a video), to the control device 106.
[0044] The control device 106 is provided to control the item sorting system 101. The control device 106 is electrically connected to the camera 104, a display 108 (described later), and an input device 1010, and is capable of communicating with each of them. The control device 106 has a processing unit 106a and a storage unit 106b. The control device 106 is actually configured by a microcomputer having the processing unit 106a including a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit), the storage unit 106b including a HDD (Hard Disk Drive) and a RAM (Random Access Memory), and an input / output buffer, etc. The processing unit 106a generates a selection screen S (described later) based on the image captured by the camera 104. The storage unit 106b stores combination data (one example of a combination) Z of waste X and the material or material Y of the waste X.
[0045] The display 108 displays the selection screen S. In this embodiment, the display 108 is a portable communication terminal configured integrally with the input device 1010 described below. The display 108 may also be a smartphone or a tablet configured integrally with the input device 1010. However, the display 108 may also be configured separately from the input device 1010. The display 108 may also be a tower-type personal computer used together with the input device 1010.
[0046] The input device 1010 receives input of combination data Z of waste X and a material or material Y corresponding to the waste X. The input device 1010 may be connected to the control device 106 in a wired or wireless manner via a line such as the Internet or a VPN (Virtual Private Network) so as to be able to communicate with the control device 106. In this embodiment, the input device 1010 is installed together with the display 108 and is an input terminal having a key corresponding to the material or material Y of the waste X. The input device 1010 may also be configured as a physical key integrally configured with the display 8. However, the input device 1010 may also be configured as an input icon displayed on the display 108 together with the selection screen S. The input device 1010 is disposed in a room or building different from the location where the camera 104, the gripping device 102, and the sorting area 1012 are disposed, and is operated by an operator H. For this reason, rough sorting by the item sorting system 101 can be performed via the input device 1010 without disposing the operator H near the gripping device 102 on which the waste X is piled up. As a result, the article sorting system 101 allows the operator H to carry out rough sorting without being restricted by the working environment, such as the smell of the waste X or dust generated by sorting.
[0047] The sorting area 1012 has a plurality of sorting areas 1012a to 1012f. In this embodiment, the sorting area 1012 is provided with six sorting areas: a first sorting area 1012a for sorting waste X whose material or material Y is wood, a second sorting area 1012b for sorting waste X whose material or material Y is metal, a third sorting area 1012c for sorting waste X whose material or material Y is resin, a fourth sorting area 1012d for sorting waste X whose material or material Y is rubber, a fifth sorting area 1012e for sorting waste X whose material or material Y is glass, and a sixth sorting area 1012f for sorting waste X whose material or material Y is other materials. However, it is sufficient that there are a plurality of sorting areas 1012, and for example, different sorting areas may be provided according to not only the material or material but also the size or shape of the waste. In this embodiment, the first sorting area 1012a to the third sorting area 1012c are provided to the right of the position where the waste X is carried in, and the fourth sorting area 1012d to the sixth sorting area 1012f are provided on the left side opposite the waste X. By arranging the sorting areas 1012 in this manner, the gripping device 102 can efficiently transport the waste X. However, the sorting areas 1012 only need to be provided within a range that the gripping device 102 can transport. A belt conveyor (not shown) is provided in each of the sorting areas 1012a to 1012f, and the waste X moved by the gripping device 102 is placed on the belt conveyor and sent to the subsequent detailed sorting process.
[0048] Next, a control procedure performed by the control device 106 will be described with reference to the flowchart of Fig. 11 and Fig. 12. The control device 106 starts the control procedure when a start button (not shown) is pressed.
[0049] The control device 106 acquires image data D captured by the camera 104 (step S201). The processing unit 106a of the control device 106 generates an item area x based on the acquired image data D (step S102). The item area x is an area obtained by dividing the image data D into areas for each waste X contained in the image data D. When the image data D contains one waste X, the image data D is divided into one item area x1 and an area xs excluding the item area. As shown in FIG. 12(a), when the image data D contains three wastes X1 to X3, the image data D is divided into three item areas x1 to x3 and an area xs excluding these item areas.
[0050] The processing unit 6a of the control device 106 designates the item category C of the item area x (step S203). The memory unit 106b of the control device 106 pre-stores combination data Z, which is a combination of waste X and the material or material Y of the waste X. The item category C includes a first item C1 and a second item C2. The first item C1 is the other item excluding the second item C2. Based on the existing combination data Z pre-stored in the memory unit 106b of the control device 106, the item category C of the waste X that cannot identify the material or material Y of the waste X is designated as the first item C1. Based on the existing combination data Z pre-stored in the memory unit 106b of the control device 106, the item category C of the waste X that can identify the material or material Y of the waste X is designated as the second item C2. The processing unit 106a designates either the first item C1 or the second item C2 as the item category C for each item area x. In this embodiment, of the wastes X1 to X3 included in the image data D, for the waste X3, the material or ingredient Y can be identified as metal based on the existing combination data Z stored in the memory unit 106b of the control device 106, so the second item C2 is specified as the item category C of the waste X3. Also, for the item categories C of the wastes X1 and X2, the material or ingredient Y cannot be identified based on the existing combination data Z, so the first item C1 is specified.
[0051] The processing unit 106a of the control device 106 judges whether the item class C of the item areas x1 to x3 corresponds to the first item C1 (step S204). If the item area x corresponds to the first item C1 (step S204 Yes), the processing unit 106a of the control device 106 generates an item selection section xc, which is a screen display corresponding to the first item C1 (step S205). The processing unit 106a of the control device 106 performs image processing on the item areas x1 and x2 corresponding to the waste X1 and X2 designated in step S203 as the first item C1, to generate the item selection section xc. In this embodiment, the processing unit 106a of the control device 106 generates an image display of the item areas x1 and x2 generated in step S202, which is filled with a diagonal line pattern as an image display different from the area xs other than the item area x3 and the item area. That is, in this embodiment, the item selection section xc is an image display generated as a fill with a diagonal line pattern in a shape corresponding to the item area x of the waste X whose item category C is the first item C1. The item selection section xc only needs to be able to distinguish the item area x of the waste X whose item category C is the first item C1 from other item areas x, and may be an image display such as a thick line indicating the boundary of the item area x or a semi-transparent filled area in addition to a fill with a diagonal line pattern.
[0052] On the other hand, if not (step S204 No), the processing unit 106a generates a selected region xu for the item region x (step S206). In this embodiment, the processing unit 106a of the control device 106 generates a selected region xu for the item region x3 generated in step S202 as an image display of a thick line indicating the boundary of the item region x3. The selected region xu may be an image display filled with a pattern different from that of the item selection region xc, as long as the operator H described below can recognize that it is not the item selection region xc. Furthermore, an image display corresponding to the selected region xu does not have to be generated for the selected region xu.
[0053] The processing unit 106a of the control device 106 generates a selection screen S (step S207). As shown in FIG. 12(b), the selection screen S is screen data obtained by overlapping and synthesizing the image display of the item selection section xc generated in step S205 on the image data D. That is, in the selection screen S, only the item area x corresponding to the waste X determined to be the first item C1 in step S204 among the item areas x corresponding to the waste X included in the image data D is synthesized as the item selection section xc on the image data D. As shown in FIG. 12(b), the processing unit 106a may further overlap the selected area xu generated in step S206 on the image data D. In this embodiment, image information obtained by synthesizing the item selection section xc generated for the item areas x1 and x2 corresponding to the waste X1 and X2 whose item classification C is the first item C1 and the image data D is generated as the selection screen S. That is, the selection screen S displays waste materials X1 and X2, whose item category C is the first item C1, clearly distinguished from other waste materials X3, whose item category C is the second item C2.
[0054] The control device 106 transmits the selection screen S to the display 8 (step S208). The control device 106 displays the selection screen S on the display 108 (step S209). Here, the operator H, while watching the selection screen S displayed on the display 108, infers the material or material Y of the waste X for which the item selection section xc is displayed, selects the item selection section xc and the material or material Y of the waste X corresponding to the item selection section xc, and inputs it to the input device 1010. The processing unit 106a of the control device 106 acquires the waste X corresponding to the item selection section xc input by the input device 1010 (step S210). The processing unit 106a of the control device 106 acquires the material or material Y input by the input device 1010 (step S211). Furthermore, the processing unit 106a of the control device 106 combines the waste X acquired in step S209 with the material or material Y acquired in step S210 to generate combination data Z (step S212). The storage unit 106b of the control device 106 stores the combination data Z (step S213).
[0055] The processing unit 106a of the control device 106 generates evaluation data r corresponding to the combination data Z generated in step S212 (step S214). In this embodiment, the processing unit 106a of the control device 106 selects one of the predetermined evaluation stages set in advance as the evaluation data r and generates it.
[0056] The processing unit 106a may generate the evaluation data r based on the time required from when the selection screen S is displayed on the display 108 in step S209 until the combination data Z is generated in step S212. If the time required until the combination data Z is generated is short, that is, if the operator H completes inputting the waste X and the material or material Y in a short time, the processing unit 106a may select a high evaluation stage to generate the evaluation data r. In this way, the processing unit 106a may select an evaluation stage according to the time required until the combination data Z is generated to generate the evaluation data r.
[0057] The processing unit 106a of the control device 106 may generate the evaluation data r based on the difference between the combination data Z previously stored in the storage unit 106b of the control device 106 and the combination data Z generated in step S212. Specifically, when the processing unit 106a of the control device 106 judges that the combination data Z of the material or material Y of the waste X input by the operator H to the input device 1010 is similar within a predetermined range by comparing it with the combination data Z previously stored in the storage unit 106b of the control device 106, that is, when it judges that the material or material Y of the waste X has been appropriately selected and input, the processing unit 106a may select a high evaluation stage and generate the evaluation data r. In this way, the processing unit 106ba may select an evaluation stage according to the accuracy of the combination data Z generated in step S212 and generate the evaluation data r.
[0058] Furthermore, the evaluation data r may be changed according to various conditions, such as the time period when the waste X and the material or material Y of the waste X are input via the input device 1010, and the identification information of the operator H previously stored in the storage unit 106b of the control device 106. Specifically, the processing unit 106a may select a high evaluation stage to generate the evaluation data r when an input is made to the input device 1010 during a preset time period or working time. Furthermore, the processing unit 106a may select an evaluation stage according to the proficiency of the operator H to generate the evaluation data r.
[0059] The memory unit 106b of the control device 106 stores the evaluation data r generated in step S213 in association with the combination data Z generated in step S212 (step S215). This allows the evaluation data r associated with the combination data Z to be accumulated in the memory unit 106b. As a result, the item selection system 101 can improve the speed and accuracy of rough selection based on the combination data Z and the evaluation data r stored in the memory unit 106b.
[0060] The processing unit 106a of the control device 106 changes the item selection area xc corresponding to the waste X acquired in step S210 to the selected area xu (step S216). This makes it possible to prevent the repeated input of the material or ingredient Y for the same waste X via the input device 1010. As a result, the item sorting system 101 can improve the speed at which rough sorting is performed.
[0061] The processing unit 106a of the control device 106 determines whether the selection screen S includes an item selection section xc (step S217). When the processing unit 106a of the control device 106 determines that the selection screen S does not include an item selection section xc (step S217 Yes), it determines the reward R based on the evaluation r stored in the storage unit 106b (step S218). In this embodiment, the processing unit 106a of the control device 106 calculates and determines the reward R as a monetary reward to be given to the operator H who inputs the waste X and the material or material Y of the waste X via the input device 1010. The reward R is determined by the evaluation data r generated in step S214 and the number of inputs of the waste X and the material or material Y made by the operator H to the input device 1010. Specifically, the higher the evaluation stage of the evaluation data r and the more the number of inputs, the larger the reward R given to the operator H. Therefore, an appropriate reward R can be given to the operator H according to the accuracy and the number of inputs of the combination data Z of the waste X and the material or material Y input to the input device 1010. This can increase the motivation of the operator H to perform input work on the input device 1010. As a result, it is possible to widely secure human resources to engage in rough sorting of the waste X.
[0062] On the other hand, if not (No in step S217), the processing unit 106a of the control device 106 returns the process to before step S208.
[0063] The processing unit 106a of the control device 106 controls the gripping device 102 to move the waste X to the sorting area 1012, and returns the process to before step S201 (step S219). In this embodiment, the processing unit 106a of the control device 106 generates a movement path to the sorting area 1012 corresponding to the material or material Y of the waste X based on the image data D and the combination data Z generated in step S211, and controls the gripping device 102 to move the waste X to the specified sorting area 1012.
[0064] Next, the acquisition position in the first embodiment and the display method of the acquisition position in the second embodiment will be described with reference to Fig. 13 to Fig. 15. In the following description, a case where the gripping device 102 grips and moves the waste X in the second embodiment will be described.
[0065] The control device 106 acquires the position where the operator H touches the display 108 (step S301). Specifically, the operator H looks at the selection screen S displayed on the display 108 and selects the waste X1 to be gripped by the gripping device 102 by a touch operation. The control device 106 acquires the position on the display 108 where the touch operation was performed.
[0066] The control device 106 determines the touch position acquired in step S301 as an acquired position (step S302). Furthermore, the control device 106 acquires the coordinates on the selection screen S based on the acquired position determined in step S302 (step S303).
[0067] The control device 106 converts the coordinates on the screen acquired in step S303 into an actual arrangement position (step S304). That is, the control device 106 converts the acquired position on the selection screen S into coordinates in the environment in which the gripping device 102 actually works, and sets the coordinates as the arrangement position.
[0068] Finally, the control device 106 transmits the placement position, which is the coordinate converted and acquired in step S304, to the arm 102a of the gripping device 102 (step S305).
[0069] Without being limited thereto, this embodiment can also be applied to the control device 6 in the first embodiment. That is, the control device 6 may acquire, as an acquired position, the position at which the harvesting arm 2c thins out the branches by the manager user performing a touch operation while viewing the harvest propriety screen 12 displayed on the display unit 8, and control the harvesting arm 2c (for example, see FIG. 8).
[0070] FIG. 14(a) is a schematic diagram showing a state in which a plurality of selectable objects X1 to X3 are displayed on a selection screen S of a display 108 operated by an operator H. FIG. 14(b) is a schematic diagram showing a state in which an object X1 is selected from the plurality of objects X1 to X3. The operator H selects one object X1 from the plurality of objects X1 to X3 displayed on the selection screen S by tracing it. When the operator H performs a touch operation while tracing the display 108, as shown in FIG. 14(b), the control device 106 displays a straight line L1 superimposed on the image of the object X1 selected by the touch operation by approximating the position traced with the finger to a straight line. The control device 6 determines an acquisition position for the gripping device 102 to acquire the object X1 based on the position and direction of this straight line L1.
[0071] 14, the control device 6 may display a rotating cursor XR superimposed on the selected item X1. The operator H can change the angle of the straight line L1 by rotating the cursor XR to an arbitrary angle on the display 108. This allows the operator H to change the direction of the straight line L1 superimposed on the selected item X1 by tracing the display 108 to an arbitrary angle without canceling the selected state. As a result, the control device 106 can acquire position information for the gripping device 102 to grip the selected item X1 with a simple operation by the operator H.
[0072] <Other embodiments> Although the embodiments of the present disclosure have been described above, the present disclosure is not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the invention.
[0073] (a) In the above first embodiment, a gantry-type robot 2 has been described as an example, but the present invention is not limited to this. The robot 2 may be a self-propelled gantry-type robot without rails, or a robot with a self-propelled robot arm. In any case, it is sufficient that the robot is capable of capturing an image of the crop A and harvesting it.
[0074] (b) In the above second embodiment, the item sorting system 1 that sorts waste X has been described as an example, but the present disclosure is not limited to this. The item sorting system 1 may be any system for sorting items to be transported, and may be a system for sorting and transporting a plurality of stacked items of different shapes or colors (e.g., agricultural products, etc.).
[0075] (c) In the second embodiment, the processing unit 6a of the control device 6 selects one of the predetermined evaluation stages as the evaluation data r and generates it, but the present disclosure is not limited to this. The evaluation data r can be selected from various forms for determining the reward R, such as a numerical value or a variable.
[0076] (d) In the above second embodiment, the remuneration R is calculated and determined as a monetary remuneration to be given to the operator H, but the present disclosure is not limited to this. That is, the remuneration R may be points that can be exchanged for cash or other services, or various other rewards. [Explanation of symbols]
[0077] 1: Crop management system 2a: Boom (an example of a moving part) 2b: Camera arm (one example of an arm) 2c: Harvesting arm (one example of an arm) 2d: Harvesting chuck 4 (104): Camera (an example of an imaging unit) 6 (106): Control device (an example of a control unit) 8 (108): Display (display unit) 21: Support 22: Rail A: Crops 101: Item sorting system 102: Grasping device 106a: Processing unit 106b: memory unit 1010: input device S: selection screen
Claims
1. An imaging unit that images an article; An arm capable of acquiring the item; A control unit that determines a shape of the article from image data captured by the imaging unit; A display unit that allows the user to select whether or not to acquire the item; Equipped with The display unit is a touch panel type, The control unit is When it is determined to acquire the item imaged by the imaging unit, displaying the placement position of the item on the display unit and displaying a screen for inputting the acquisition position of the item; When the acquisition position is input, the user touches the display unit to trace the acquisition position on the screen, the control unit approximates the position where the user traces the display unit to a straight line, determines the acquired position based on the position and orientation of the straight line, acquires coordinates on the screen based on the acquired position, converts the coordinates on the screen into coordinates in an environment in which the arm works, determines the coordinates in the environment in which the arm works as the placement position, moves the arm towards the coordinates of the placement position, and then aligns the arm with the acquired position and moves the arm to acquire the item. Item Acquisition System.
2. The control unit is A plurality of pieces of data captured while moving the imaging unit are synthesized in advance and acquired as one piece of image data. The item acquisition system according to claim 1 .
3. The control unit captures an image of the item at a predetermined time using the imaging unit to obtain the image data, and displays the screen on the display unit after an arbitrary time interval. The article acquisition system according to claim 1 or 2.
4. the commodity is an agricultural crop, the control unit, when the crop imaged by the imaging unit is selected as the harvest target or the thinning target, moves the arm to harvest the crop. The article acquisition system according to any one of claims 1 to 3.
5. The control unit records the positions of the crops to be harvested or thinned, displaying, on the display unit, a position of at least one of the harvest target and the thinning target, and at least one of the harvest target and the crop to be thinned; The article acquisition system according to claim 4 .
6. The control unit determines whether the crop is a target for harvesting or a target for thinning by using one or both of a shape and a color of the crop imaged by the imaging unit, When a crop management user needs to determine whether a crop is a harvest target or a thinning target among the crops determined to be thinning targets, at least one of the crops to be harvested and the crops to be thinned is displayed on the display unit. The article acquisition system according to claim 4 or 5.
7. The control unit records harvest forecast data of the agricultural crops, updating the harvest forecast data when the crop is thinned; The article acquisition system according to any one of claims 4 to 6.
8. A plurality of supports erected at intervals inside the greenhouse capable of accommodating the agricultural crops; At least one movable member is hung on the plurality of posts and is movable over the top of the crops in the greenhouse; The arm is movable from the moving member toward the crop. The article acquisition system according to any one of claims 4 to 7.
Citation Information
Patent Citations
Fruit harvesting apparatus
JP1984034816A
Fruit harvesting apparatus
JP1984034817A
Information providing system, information providing device, information providing method, and program
JP2013005726A
Agricultural robot device
JP2020089345A
Fruit vegetable harvesting device and fruit vegetable harvesting method
JP2020195335A
