Work systems, learning model generation systems, and work tools

Abstract

The objective is to provide a work system that can acquire teaching information even when performing tasks such as inserting teaching tools into agricultural products. [Solution] The work system is a work system for teaching movements to a robot hand comprising an arm and a hand portion attached to the tip of the arm, and comprises a work tool and a sensing device. The work tool includes a handle portion corresponding to the arm, a head portion corresponding to the hand portion and located on the tip side of the handle portion, and a detectable marker portion located on the tip side of the handle portion. The sensing device detects the behavior of the marker portion located on the tip side of the handle portion and acquires teaching information including the behavior of the marker portion.

Description

Technical Field

[0001] The present disclosure relates to a work system, a learning model generation system, and a work tool.

Background Art

[0002] Patent Document 1 describes a teaching tool in which a marker for position and orientation detection is fixed to a main body. The teaching tool is operated by a human and can grasp an object.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When attempting to obtain teaching information in agricultural work, such as pruning work, using the teaching tool described in Patent Document 1, the teaching tool may be inserted into the agricultural crop to perform pruning work. In this case, the leaves of the agricultural crop may hide the marker, and there is a risk that the teaching information cannot be obtained.

[0005] In view of the above problems, an object of the present disclosure is to provide a work system capable of obtaining teaching information even in work where a teaching tool is inserted into an agricultural crop.

Means for Solving the Problems

[0006] A work system according to one aspect of the present disclosure is a work system for teaching movements to a robot hand comprising an arm and a hand portion attached to the tip of the arm, the work system comprising a work tool and a sensing device, the work tool including a handle portion corresponding to the arm, a head portion corresponding to the hand portion and located on the tip side of the handle portion, and a detectable marker portion located on the tip side of the handle portion, the sensing device detecting the behavior of the marker portion located on the tip side of the handle portion and acquiring teaching information including the behavior of the marker portion.

[0007] A work tool according to one aspect of this embodiment is a work tool for teaching movements to a robot hand comprising an arm and a hand portion attached to the tip of the arm, and comprises a handle portion corresponding to the arm, a head portion corresponding to the hand portion and located on the tip side of the handle portion, and a detectable marker portion located on the tip side of the handle portion. [Effects of the Invention]

[0008] According to this disclosure, teaching information can be obtained even in tasks such as inserting a teaching tool into crops. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 is a schematic diagram showing an example of the work system according to Embodiment 1. [Figure 2] Figure 2 is an explanatory diagram illustrating the configuration of the work tools. [Figure 3] Figure 3 is a perspective view showing an example of a work tool. [Figure 4] Figure 4 is a schematic diagram showing an overview of the learning model generation system. [Modes for carrying out the invention]

[0010] <Summary of the embodiments of this disclosure> The embodiments of this disclosure are outlined below.

[0011] (1) A work system according to one aspect of this embodiment is a work system for teaching movements to a robot hand comprising an arm and a hand portion attached to the tip of the arm, comprising a work tool and a sensing device, wherein the work tool includes a handle portion corresponding to the arm, a head portion corresponding to the hand portion and located on the tip side of the handle portion, and a detectable marker portion located on the tip side of the handle portion, and the sensing device detects the behavior of the marker portion located on the tip side of the handle portion and acquires teaching information including the behavior of the marker portion.

[0012] According to the work system of this embodiment, since the marker portion is located at the tip of the handle portion, i.e., at the base end of the head portion, even when the user inserts the head portion into a crop to perform agricultural work, the leaves of the crop do not obscure the marker portion. This allows the work system to acquire teaching information.

[0013] (2) In (1) above, the sensing device may include an imaging unit that acquires an image of a user performing work using the work tool, and an acquisition unit that acquires the teaching information based on the image of the marker portion included in the image.

[0014] (3) In (1) or (2) above, the marker portion is a polyhedron, and the polyhedron may have a pattern on a surface other than the surface facing the handle portion.

[0015] Because the marker portion is polyhedron-shaped, the sensing device can recognize the pattern on the marker portion regardless of the orientation of the work tool.

[0016] (4) In (3) above, the pattern may be an AR (Augmented Reality) marker.

[0017] Based on AR markers, the work system can provide teaching information indicating the position and orientation of the work tool.

[0018] (5) In any one of (1) to (4) above, the input device may further include an input device that outputs a signal for instructing the operation of the head unit, and the sensing device may further obtain information indicating the instruction as the teaching information from the signal output by the input device.

[0019] Thereby, even when the leaves of the agricultural crop hide the head unit, the work system can obtain, as teaching information, the event that the head unit is operating.

[0020] (6) In (5) above, the input device may be provided on the handle portion.

[0021] Since the input device is provided at the user's hand, the user can easily input a work instruction.

[0022] (7) In any one of (1) to (6) above, the marker portion may be further provided at the tip of the head portion.

[0023] When the sensing device can recognize the marker portion provided at the tip of the head portion, the work system can specify the position of the tip of the work tool portion by the marker portion.

[0024] (8) The learning model generation system according to one aspect of the present embodiment includes a generation device that generates a learning model based on the teaching information obtained by the sensing device of the work system according to any one of (1) to (7) above.

[0025] According to the learning model generation system of the present embodiment, a learning model can be generated based on the teaching information obtained by the work system according to any one of (1) to (7) of the present disclosure.

[0026] (9) A work tool according to one aspect of this embodiment is a work tool for teaching an action to a robot hand comprising an arm and a hand portion attached to the tip of the arm, comprising a handle portion corresponding to the arm, a head portion corresponding to the hand portion and located on the tip side of the handle portion, and a detectable marker portion located on the tip side of the handle portion.

[0027] According to the work tool of this embodiment, the marker portion is located at the tip of the handle portion, that is, at the base end of the head portion. Therefore, even when the user inserts the head portion into a crop to perform agricultural work, the leaves of the crop do not cover the marker portion.

[0028] (10) In (9) above, the marker portion is a polyhedron, and the polyhedron may have patterns on the surfaces other than the surface facing the handle portion.

[0029] Because the marker area is polyhedron-shaped, the pattern on the marker area can be recognized regardless of the direction the work tool is facing.

[0030] (11) In (10) above, the pattern may be an AR marker.

[0031] AR markers allow the work system to determine its location.

[0032] (12) Any one of the above (9) to (11) may further include an input device that outputs a signal to command the operation of the head unit.

[0033] This allows the work tool to output a signal indicating that the head is in operation, even if the leaves of crops obscure the head.

[0034] (13) In the above (12), the input device may be provided on the handle portion.

[0035] Since an input device is provided at hand, users can easily input work commands.

[0036] (14) In any one of (9) to (13) above, the marker portion may be further provided at the tip of the head portion.

[0037] If the marker located at the tip of the head can be recognized, the work system can use the marker to pinpoint the position of the tip of the work tool.

[0038] [Details of the embodiments of this disclosure] The embodiments of this disclosure will be described in detail below with reference to the drawings. At least some of the embodiments described below may be combined in any way.

[0039] [Embodiment 1] [1-1 Overall Configuration of the Work System] Figure 1 is a schematic diagram showing an example of the work system according to Embodiment 1. The work system 1 of this disclosure will be described based on Figure 1. The work system 1 is a system for teaching operations to a robot hand, which comprises an arm and a hand portion attached to the tip of the arm. Here, teaching means that the user pre-programs the robot hand to perform specific tasks or actions accurately. Specifically, the work system 1 is a system for acquiring control information for controlling the robot hand 21. The work system 1 comprises a work tool 2 and a sensing device 3. The work tool 2 includes a handle portion 4 corresponding to the arm, a head portion 5 corresponding to the hand portion and located on the tip side of the handle portion 4, and a detectable marker portion 6 located on the tip side of the handle portion 4. The sensing device 3 detects the behavior of the marker portion 6 provided on the tip side of the handle portion 4 and acquires teaching information TI including the behavior of the marker portion 6. Here, equivalent means configured to perform the same function. However, the materials and structures of each may be the same or different. Before explaining the details of each component, let's first explain the prerequisite: the robot hand.

[0040] <Robot Hand> A robotic hand is a device that mimics a human arm and hand. It is a part of a robot capable of grasping and manipulating objects like a human hand. A robotic hand includes, for example, a hand section corresponding to a human hand and an arm section corresponding to a human arm. Similar to the relationship between a human hand and arm, the hand section is connected to the arm section. The hand section is driven, for example, by actuators to open and close, and grasps objects. The arm section, connected to the hand section, moves the hand section to a working position on the object and changes the orientation of the hand section to match the direction of the object. To perform such actions, the arm section includes, for example, multiple link sections, joint sections connecting the link sections, and actuator sections driving the link sections. A control device controls the actuators, enabling the robotic hand to perform fine movements similar to a human arm.

[0041] The control device controls the actuators based on control information, causing the robot hand to perform actions. To make the robot hand perform actions similar to those of a human arm, a large amount of control information is required. If this control information is entered via a keyboard, the input time can be substantial. In the work system 1 of this disclosure, the sensing device 3 captures images of the user performing the work, including the work tool 2, and generates control information to control the robot hand. As a result, the work system 1 can reduce the time required to generate control information compared to the time required to input it via a keyboard.

[0042] [1-2 Details of each component] Next, we will explain the details of each component. [1-2-1 Work Tools] Figure 2 is an explanatory diagram illustrating the configuration of the work tool 2. The work tool 2 includes a handle 4, a head 5, and a marker 6. The work tool 2 may be made of any material that can withstand the agricultural work to be performed. For example, the material of the work tool 2 may be metal, synthetic resin, or wood.

[0043] <Handle section> The handle portion 4 corresponds to the arm 22 of the robot hand 21. The user U grasps the handle portion 4 and performs agricultural work. For this reason, the handle portion 4 has a shape that the user U can grasp, for example, a cylindrical shape. The handle portion 4 may also have a hollow structure. This allows the handle portion 4 to be lighter or to incorporate actuators that drive the head portion 5.

[0044] <Head section> The head unit 5 corresponds to the hand unit 23 attached to the tip of the arm 22 of the robot hand 21. The head unit 5 is the part that performs work on agricultural crops. The head unit 5 is located at the tip of the handle unit 4. The head unit 5 has a shape and structure according to the agricultural work. For example, when the work is gripping, the head unit 5 has a shape and structure as shown in Figure 2. When the work is pruning, the head unit 5 has the shape and structure of pruning shears. When the work is spraying pesticides, the head unit 5 has the shape and structure of a sprayer. The head unit may also be configured to be operated by an actuator built into the handle unit. The hand unit 23 performs the same function as the head unit 5 and has a similar shape and configuration. For example, if the head unit 5 is pruning shears, then the hand unit 23 is also pruning shears.

[0045] <Marker Department> The marker portion 6 is a detectable marker located on the tip side of the handle portion 4. In other words, the sensing device 3 can detect the marker portion 6 by acquiring an image of the user U performing work with the work tool 2.

[0046] Figure 3 is a perspective view of an example of the work tool 2. The marker portion 6 is located at the tip of the handle portion 4. The head portion 5 is connected to the handle portion 4 in a manner that is located on the tip side of the handle portion 4; in other words, the marker portion 6 is located on the base side of the head portion 5. As a result, even if the leaves obscure the head portion 5 when the head portion 5 is inserted into the crop AP to cut the branch, the sensing device 3 can still recognize the marker portion 6.

[0047] The marker portion 6 may also be a polyhedron, for example, a cube. The polyhedron has a pattern 7 on all faces except the one facing the handle portion 4. This allows the sensing device to recognize the pattern 7 on the marker portion 6 even if the work tool 2 is tilted. Furthermore, the imaging unit 9, which will be described later, can simultaneously photograph multiple faces of the marker portion 6, allowing for more accurate determination of its position and orientation than with a single face. The marker portion 6 may also have a second polyhedron on the opposite side of the handle portion 4 from the side where the first polyhedron is located. This allows the sensing device 3 to recognize the second polyhedron even if the handle portion 4 or the head portion 5 obscures the first polyhedron from the perspective of the sensing device 3.

[0048] The pattern 7 on the polyhedron may be any pattern that the sensing device 3 can detect the behavior of the marker portion, for example, an AR marker. An AR marker is an image recognition technology designed to recognize specific visual information (such as an image) and overlay creative works such as 3D models or videos onto it. In this disclosure, the AR marker is used as a mark for detecting the position and orientation of the work tool 2. The sensing device 3 detects the AR marker, which is designed to be detectable, and acquires teaching information TI indicating the position and orientation of the work tool 2 based on the detected AR marker. Note that the pattern 7 is not limited to an AR marker and may also be a QR code (registered trademark).

[0049] The marker portion 6 may be further provided at the tip of the head portion 5. If the sensing device 3 can recognize the marker portion 6, the sensing device 3 can determine the position of the tip of the head portion 5 by the marker portion 6. As a result, the work system 1 can acquire teaching information TI as the behavior of the marker portion 6 based on the behavior of the head portion 5.

[0050] <Input device> Returning to Figure 2, let's explain the input device 8. The input device 8 is a device that outputs a signal to command the operation of the head unit 5. Hereinafter, this signal will be referred to as the command signal CS. For example, if the head unit 5 is pruning shears, the signal that commands the operation of the head unit 5 is a signal indicating a command to close or open the pruning shears. The input device 8 is, for example, a push-button switch. When user U operates the push-button switch, the push-button switch connects or disconnects the two signal lines connected to the push-button switch. As a result, the push-button switch outputs the command signal CS. The command signal CS is input to the control circuit (not shown) that controls the head unit 5 and to the sensing device 3.

[0051] The control circuit that controls the head unit 5 is located, for example, inside the handle unit 4. The command signal CS is input to the control circuit. Based on the input command signal CS, the control circuit drives, for example, an actuator to open and close the head unit 5. If the head unit is pruning shears, the actuator opens and closes the blades of the pruning shears, and the blades cut the branches of the crop AP. The input device is not limited to a push-button switch, but may be a sensor that detects the actions of the user U. The sensor is, for example, a gesture sensor. The gesture sensor detects the hand movements and body movements of the user U and outputs a signal indicating the detection result.

[0052] The input device 8 may be provided on the handle portion 4. Since the input device 8 is located within easy reach, the user U can easily input commands.

[0053] [1-2-2 Sensing Devices] Returning to Figure 1, the sensing device 3 will be described. The sensing device 3 detects the behavior of the marker portion 6 provided on the tip side of the handle portion 4 and acquires teaching information TI including the behavior of the marker portion 6. The sensing device 3 includes an imaging unit 9 and a processing unit 10.

[0054] <Photography Department> The imaging unit 9 acquires images of user U working with the work tool 2. The imaging unit 9 includes an image sensor, such as a CMOS (Complementary Metal-Oxide-Semiconductor) imager. The image sensor converts the image captured by the imaging unit 9 into an image signal VS. The imaging unit 9 converts the image signal VS into, for example, digital information and outputs it. The output digital information is input to the processing unit 10. The imaging unit 9 is positioned to capture all or part of the crop AP, user U, and work tool 2. The imaging unit 9 is mounted on a support member, such as a tripod. User U may wear the imaging unit 9.

[0055] <Processing device> The processing unit 10 performs processing such as acquiring teaching information TI. The processing unit 10 includes a processor 11, memory 12, and an input / output interface (hereinafter referred to as "input / output I / F") 13.

[0056] Processor 11 is, for example, a CPU (Central Processing Unit). However, processor 11 is not limited to a CPU. Processor 11 may also be a GPU (Graphics Processing Unit). Processor 11 may be, for example, a multi-core processor. Processor 11 may also be a single-core processor. Processor 11 may be, for example, an ASIC (Application Specific Integrated Circuit), or a programmable logic device such as a gate array or FPGA (Field Programmable Gate Array).

[0057] Memory 12 includes volatile memory and non-volatile memory. Volatile memory is semiconductor memory such as SRAM (Static Random Access Memory) and DRAM (Dynamic Random Access Memory). Non-volatile memory is flash memory, hard disk, ROM (Read Only Memory), etc. Non-volatile memory stores a control program for controlling the working system 1, which is a computer program, and data used to execute the control program. Each function of the processor 11 is performed when the control program is executed by the processor 11. The control program can be stored in a recording medium such as flash memory, ROM, or CD-ROM.

[0058] The input / output interface 13 is connected to the imaging unit 9. The input / output interface 13 transmits commands to the imaging unit 9 to control it, and receives signals containing captured images output by the imaging unit 9. The input / output interface 13 passes the captured image information contained in the input signal to the processor 11. The input / output interface 13 is also connected to the input device 8. The input device 8 inputs a command signal CS, which is a signal that commands the operation of the head unit 5, to the input / output interface 13. The input / output interface 13 passes the information contained in the command signal CS (hereinafter referred to as "command information CI") to the processor 11.

[0059] [1-3 Operation of the work system] [1-3-1 Work Tools] Work tool 2 is a tool operated by user U when acquiring teaching information TI. When work tool 2 performs work on crop AP, such as pruning, work system 1 can acquire the position and orientation of work tool 2 as teaching information. Further details will be explained in the sensing device section below.

[0060] The work tool 2 further includes an input device 8. When the input device 8 outputs a command signal CS, the head unit 5 operates based on the command signal CS. Meanwhile, the command signal CS is input to the sensing device 3. As a result, even if the leaves of the crop AP obscure the head unit 5, the sensing device 3 can recognize from the command signal CS that the head unit 5 is operating.

[0061] [1-3-2 Sensing device] The sensing device 3 detects the behavior of the marker portion 6 provided on the tip side of the handle portion 4 and acquires teaching information TI including the behavior of the marker portion 6. The sensing device 3 includes an imaging unit 9 and an acquisition unit 14 as a functional block. The processor 11 executes a control program stored in the memory 12, thereby causing the acquisition unit 14 to perform its function. The pattern 7 on the marker portion 6 is described as an AR marker.

[0062] <Photography Department> The imaging unit 9 acquires an image of the user performing work using a work tool. The captured image includes at least an image of the marker portion 6 of the work tool 2. The imaging unit 9 outputs a signal containing information indicating the captured image (hereinafter referred to as "image information VI"). This signal is input to the input / output I / F 13 of the processing unit 10. The input / output I / F 13 stores the image information VI contained in the input signal in the memory 12.

[0063] <Acquisition part> The acquisition unit 14 acquires teaching information TI based on the image of the marker unit 6 included in the captured image. Specifically, (1) The acquisition unit 14 reads out the image information VI stored in the memory 12. (2) The acquisition unit 14 detects AR markers from the read image information VI. For detection, for example, a pattern matching technique is used, which compares the image information VI with information indicating the pattern 7 of the AR marker that is stored in memory 12 in advance, and detects the AR markers. (3) The acquisition unit 14 extracts feature points such as corners and edges of the AR marker from the detected AR marker. (4) The acquisition unit 14 calculates the position and orientation of the AR marker based on the corners and edges of the extracted AR marker. (5) The acquisition unit 14 stores the calculated position and orientation information of the AR marker (hereinafter referred to as "position and orientation information POI") in, for example, the memory 12. Such position and orientation information POI can be used as teaching information TI. (6) The acquisition unit 14 repeats the steps (1) to (5) above. This allows the acquisition unit 14 to acquire continuous position and orientation information POI, that is, information indicating the behavior of the marker unit 6. As described above, the acquisition unit 14 acquires teaching information TI based on the image of the marker portion 6 included in the captured image. In other words, the sensing device 3 can detect the behavior of the marker portion 6 provided on the tip side of the handle portion 4 and acquire teaching information TI including the behavior of the marker portion 6.

[0064] Furthermore, the sensing device 3 may acquire information indicating a command (command information CI) as teaching information TI from the signal output by the input device 8. Specifically, the acquisition unit 14 associates the command information CI with the position and attitude information POI. The position and attitude information POI to be associated is the position and attitude information POI calculated based on the captured image taken at the same time that the command signal CS was output. As a result, the sensing device 3 can acquire information indicating a command (command information CI) from the signal output by the input device 8 as teaching information TI.

[0065] [1-4 Effects] As explained above, since the marker portion 6 is located at the tip end of the handle portion 4, that is, at the base end of the head portion 5, even if the user U inserts the head portion 5 into the crop AP to perform agricultural work, the leaves of the crop AP do not cover the marker portion 6. As a result, even if the leaves of the crop AP cover the head portion 5, the work system 1 can acquire teaching information TI. Teaching information TI can be used to teach movements to a robot hand, which is equipped with an arm and a hand portion attached to the tip end of the arm.

[0066] [Embodiment 2] [2-1 Learning Model Generation System] Figure 4 is a schematic diagram showing an overview of the learning model generation system. The learning model generation system 40 includes a generation device 41 that generates a learning model based on teaching information TI acquired by the sensing device 3. The learning model generation system 40 differs from the work system 1 in that it further includes a generation device 41. Other aspects are the same. The same reference numerals are used for components identical to those in the work system 1, and descriptions of identical components, functions, and operations are omitted.

[0067] <Structure> The learning model generation system 40 includes a processor 42, memory 43, and input / output interface 44. The processor 42, memory 43, and input / output interface 44 have the same configuration as the processor 11, memory 12, and input / output interface 13 included in the processing unit 10, so their description is omitted.

[0068] <Operation> The generation device 41 generates a learning model through the following process. (1) The processor 42 controls the input / output interface 44 to acquire position and orientation information POI stored in the memory 12. The position and orientation information POI may also be associated with command information CI. The processor 42 generates a training dataset from the acquired multiple position and orientation information POIs and stores it in the memory 43. (2) The processor 42 generates a learning model based on the dataset stored in memory 43.

[0069] <Effects and Effects> The learning model generation system 40 generates a learning model based on the teaching information TI acquired by the work system, i.e., the position and orientation information POI of the work tool 2. This allows the learning model generation system 40 to generate a learning model based on the work of user U. For example, if user U's work is pruning, the learning model generation system 40 can generate a learning model that suggests to user U where on the branch of the crop AP it is acceptable to cut.

[0070] The embodiments disclosed herein are illustrative and not restrictive in all respects. The scope of the present invention is not limited to the embodiments described above, and includes all modifications within the scope equivalent to the configurations described in the claims. [Explanation of symbols]

[0071] 1. Work System 2. Work Tools 3. Sensing device 4. Handle section 5. Head section 6 Marker section 7 patterns 8 Input devices 9. Imaging Unit 10 Processing Unit 11 processors 12 memory 13. Input / Output Interface (I / F) 14 Acquisition Department 21 Robot Hand 22 Arms 23 Handball Section 40. Learning Model Generation System 41 Generator 42 processors 43 memory 44 Input / Output Interfaces AP Agricultural Products CI directive information CS command signal VI Image Information VS Image signal POI position and orientation information TI Teaching Information U User

Claims

1. A work system for teaching movements to a robot hand comprising an arm and a hand portion attached to the tip of the arm, comprising a work tool and a sensing device, The aforementioned work tool is, The handle portion corresponding to the aforementioned arm, A head portion corresponding to the hand portion and located on the tip side of the handle portion, The handle portion includes a detectable marker portion located at the tip end, The sensing device is The behavior of the marker located at the tip of the handle is detected, and teaching information including the behavior of the marker is acquired. Work system.

2. The sensing device is An imaging unit that acquires images of a user performing work using the aforementioned work tool, Includes an acquisition unit that acquires teaching information based on the image of the marker portion included in the captured image, The work system according to claim 1.

3. The aforementioned marker portion is a polyhedron, The polyhedron has a pattern on a surface other than the surface facing the handle portion. The work system according to claim 1.

4. The aforementioned pattern is an AR (Augmented Reality) marker. The work system according to claim 3.

5. The system further includes an input device that outputs a signal to command the operation of the head unit, The sensing device further acquires information indicating the command from the signal output by the input device as teaching information. The work system according to claim 1.

6. The work system according to claim 5, wherein the input device is provided on the handle portion.

7. The marker portion is further provided at the tip of the head portion, The work system according to claim 1.

8. The present invention provides a generation device that generates a learning model based on the teaching information acquired by the sensing device described in any one of claims 1 to 7, A learning model generation system.

9. A work tool for teaching movements to a robot hand comprising an arm and a hand portion attached to the tip of the arm, The handle portion corresponding to the aforementioned arm, A head portion corresponding to the hand portion and located on the tip side of the handle portion, A detectable marker portion located at the tip of the handle portion, Prepare Work tools.

10. The aforementioned marker portion is a polyhedron, The polyhedron has a pattern on a surface other than the surface facing the handle portion. The work tool according to claim 9.

11. The work tool according to claim 10, wherein the pattern is an AR marker.

12. The system further includes an input device that outputs a signal to command the operation of the head unit. The work tool according to claim 9.

13. The input device is provided on the handle portion of the work tool according to claim 12.

14. The marker portion is further provided at the tip of the head portion, The work tool according to claim 9.

Images