Cooperation operation system, method, and program

Abstract

To facilitate introduction of a robot to non-stationary operation.SOLUTION: A cooperation operation system for enabling a robot to perform operation in cooperation with an operator includes: a reception part for receiving an instruction of operation; an analysis part for decomposing the received instruction of the operation into an instruction of operation of the robot; and a target creation part for creating an operation target including a target position and a target attitude of the robot, on the basis of the decomposed instruction of the operation.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a collaborative work system in which a robot works in collaboration with a worker. [Background technology]

[0002] In recent years, in the field of FA (Factory Automation), development of a collaborative work system in which humans and robots work in cooperation has been progressing. For example, JP 2023-87705 A (Patent Document 1) describes a collaborative work system in which humans and robots work in cooperation. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2023-87705 Summary of the Invention [Problem to be solved by the invention]

[0004] In a factory, a series of tasks such as transporting workpieces, processing the workpieces, and assembling the processed workpieces are so-called routine tasks, in which the work procedures (workpieces, type of work, time required for work, and order of work) are determined in advance and do not change. In contrast, in non-routine tasks, such work procedures are not fixed, but are changed by people as appropriate.

[0005] Due to the recent labor shortage, there is a shortage of workers not only for routine tasks but also for non-routine tasks. When robots are introduced to routine tasks, they only need to repeat a set motion, and introduction is relatively easy, making it possible to make up for the labor shortage. In contrast, when introducing robots to non-routine tasks, the robot must carry out the work procedure determined by a human, that is, it must cooperate with a human, making introduction impractical and difficult. Patent Document 1 does not propose introducing robots to non-routine tasks.

[0006] The present disclosure has been made in consideration of the above-mentioned problems, and has an object to provide a collaborative work system that makes it easy to introduce robots into non-routine tasks. [Means for solving the problem]

[0007] The collaborative work system according to this disclosure is a collaborative work system in which a robot works in collaboration with a worker, and includes a reception unit that receives work instructions, an analysis unit that breaks down the received work instructions into work commands for the robot, and a goal generation unit that generates operation targets including a target position and target posture for the robot based on the broken down work commands.

[0008] According to this disclosure, a worker can give a task instruction to a collaborative task system to cause the system to generate a target motion, making it easier to introduce robots into non-routine tasks.

[0009] In the above disclosure, work instructions include instructions based on voice spoken by a worker and instructions based on information indicating the surrounding environment of the robot.

[0010] According to this disclosure, task instructions can be given to a collaborative task system by speech and information indicating the surrounding environment.

[0011] In the above disclosure, the robot operates in the same work area as the worker, and the information indicative of the surrounding environment includes a three-dimensional image representing a space corresponding to the work area.

[0012] According to this disclosure, a three-dimensional image of a work area can be used as information indicating the surrounding environment for work instructions to a collaborative work system.

[0013] In the above disclosure, the three-dimensional image includes an image representing a gesture of a worker in a work area. According to this disclosure, the image representing the gesture of the worker can be used as information indicating the surrounding environment of the work instruction.

[0014] In the above disclosure, the gesture of the worker includes pointing at an object related to the work in the work area. According to this disclosure, an image of the worker pointing at an object related to the work can be used as an instruction for the work.

[0015] In the above disclosure, the collaborative task system further includes a storage unit that stores the movement target, and an adjustment unit that adjusts the movement target in the storage unit based on an instruction received by the reception unit. According to this disclosure, the generated movement target can be adjusted based on the instruction.

[0016] In the above disclosure, the task includes a task in which the robot handles a workpiece in cooperation with a worker, and the task instruction includes an instruction based on a force applied by the worker to the workpiece. According to this disclosure, the force applied by the worker to the workpiece can be used as the task instruction.

[0017] In the above disclosure, the robot has a movable part including an arm and a drive part that drives the movable part, and the collaborative work system generates a control command that controls the drive part based on the operation target so that the movable part operates in accordance with the operation target.

[0018] According to this disclosure, a control command for driving a robot can be generated from the generated operation target.

[0019] The method of this disclosure is a computer-implemented method that includes receiving instructions for a task to be performed by a robot in collaboration with a worker, breaking down the received work instructions into task commands for the robot, and generating operation targets including a target position and a target posture for the robot based on the broken down work commands.

[0020] The program of this disclosure is a program for causing a computer to execute a method, the method comprising: receiving instructions for a task to be performed by a robot in collaboration with a worker; breaking down the received work instructions into task commands for the robot; and generating an operational target including a target position and a target posture for the robot based on the broken down task commands. Effect of the Invention

[0021] According to the present disclosure, it is possible to easily introduce robots into non-routine tasks. [Brief description of the drawings]

[0022] [Figure 1] FIG. 1 illustrates an example of a collaborative work system according to an embodiment. [Diagram 2] FIG. 2 is a schematic diagram illustrating an example of a hardware configuration of a control device according to the present embodiment. [Diagram 3] FIG. 1 is a schematic diagram showing a configuration example of a robot according to an embodiment of the present invention. [Figure 4] FIG. 2 is a schematic diagram illustrating an example of a hardware configuration of an information processing device according to an embodiment. [Diagram 5] FIG. 2 is a diagram illustrating an example of a configuration of modules of an information processing device according to the present embodiment. [Figure 6] 1A to 1C are diagrams illustrating an example of a process for generating and adjusting a motion target according to the present embodiment. [Figure 7] FIG. 2 is a diagram showing a schematic diagram of a relationship between a work instruction, a work command, and an operation target according to the present embodiment. [Figure 8] FIG. 2 is a diagram illustrating an example of a motion target according to the present embodiment. [Figure 9] 5A to 5C are diagrams illustrating a process of calculating a control command from a target position of an axis according to the present embodiment. [Figure 10] FIG. 2 is a diagram illustrating an example of a display screen according to the present embodiment. [Figure 11] 3 is a flowchart of a process according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] The embodiments of the present invention will be described in detail with reference to the drawings. In the drawings, the same or corresponding parts are denoted by the same reference characters and the description thereof will not be repeated. The modifications described below may be combined as appropriate.

[0024] <A. Application Example> The outline of the collaborative work system according to this embodiment will be described. FIG. 1 is a diagram showing an example of the collaborative work system according to the embodiment. As shown in FIG. 1, the collaborative work system 1 includes a control device 100, a robot 200, a plurality of sensing devices 300, an information processing device 500, a camera 600, a display device 700, an input device 800, a microphone 900, and gloves worn by an operator 400. The gloves have a force sensor 9 that measures the force applied to the gloves.

[0025] The collaborative work system 1 is provided at a production site such as a factory. As shown in FIG. 1, the production site includes a process in which the operator 400 and the robot 200 cooperate to perform an unsteady operation (hereinafter referred to as the "collaborative work process"). The collaborative work system 1 includes one or more collaborative work processes. In each collaborative work process, in the same work area, the robot 200 performs an unsteady operation in cooperation with the operator 400. The collaborative work process in FIG. 1 is, for example, an equipment maintenance process. More specifically, an unsteady operation is shown in which the robot 200 grips the member 5 on the table 8, moves while gripping it, and cooperates with the operator 400 at the movement destination to fit the member 5 into the member 6. The members 5 and 6 include parts that make up the equipment to be maintained or materials that make up the parts. Note that the unsteady operation is not limited to such maintenance work.

[0026] When the operator 400 grips objects such as the members 5 and 6, the force applied from the operator 400 to the objects is measured by the force sensor 9. The force sensor 9 outputs the force and moment separately in three directions of the X-axis, Y-axis, and Z-axis. The output data from the force sensor 9 is transferred to the information processing device 500.

[0027] The robot 200 includes a manipulator 202 and a manipulator controller 204.

[0028] The manipulator 202 has a plurality of arms 211, 212, and 213, a plurality of joints 221, 222, and 223, a rotating unit 230, and a base unit 240. Specifically, in the manipulator 202, from the tip side, the arm 211, the joint 221, the arm 212, the joint 222, the arm 213, the joint 223, the rotating unit 230, and the base unit 240 are provided in this order. The number of arms and joints provided in the manipulator 202 is not limited as long as they are plural.

[0029] An end effector is attached to the tip of arm 211. A rear end of arm 211 is connected to joint 221. A tip of arm 212 is connected to joint 221. A rear end of arm 212 is connected to joint 222. A tip of arm 213 is connected to joint 222. A rear end of arm 213 is connected to joint 223.

[0030] The joint 221 moves the arm 211 relative to the arm 212. The joint 222 moves the arm 212 relative to the arm 213. The joint 223 moves the arm 213 relative to the rotation portion 230.

[0031] The rotating part 230 supports the joint 223 and is configured to be rotatable about an axis. The base part 240 supports the rotating part 230 so that the rotating part 230 is rotatable.

[0032] The manipulator controller 204 controls the operation of the manipulator 202 in accordance with a control command from the control device 100. Specifically, the manipulator controller 204 acquires a control command from the control device 100 instructing a target position and a moving speed of the tip of the manipulator 202, that is, the tip of the arm 211 (hereinafter referred to as "tool center point (TCP)"). The manipulator controller 204 controls the operations of the joints 221 to 223 and the rotating part 230 so that the tool center point moves in accordance with the acquired control command.

[0033] The sensing device 300 is a device that is placed in a work area and detects the position of an object in the work area. The objects include a workpiece, a manipulator 202, and a worker 400. The sensing device 300 is, for example, an RGB-D camera or a laser range finder, and generates point cloud data of a field of view in which each pixel (each point) in a captured three-dimensional image has a three-dimensional coordinate value. It is preferable that a plurality of sensing devices 300 are placed at different positions (so as to be in the image capturing direction) so that there are no blind spots in the three-dimensional space including the work area.

[0034] The control device 100 is communicatively connected to a plurality of sensing devices 300. The control device 100 communicates with the plurality of sensing devices 300 using, for example, GigE Vision (registered trademark) or a Universal Serial Bus (USB).

[0035] The control device 100 is communicatively connected to a manipulator controller 204. The control device 100 communicates with the manipulator controller 204 using, for example, Ethernet / IP (registered trademark). The control device 100 recognizes the current position of the manipulator 202 through communication with the manipulator controller 204.

[0036] 1, the control device 100 includes an input / output interface 10 (shown as input / output I / F 10 in the figure) and a robot interface 12 (shown as robot I / F 12 in the figure). When the control device 100 controls robots 200 installed in each of a plurality of collaborative work processes, the control device 100 includes a robot interface 12 for each robot 200 to be controlled.

[0037] The input / output interface 10 controls the exchange of information (data, signals) with an external device. More specifically, the input / output interface 10 controls the exchange between the control device 100 and the information processing device 500 and the exchange between the control device 100 and the sensing device 300. The robot interface 12 controls the exchange between the control device 100 and the manipulator controller 204.

[0038] The information processing device 500 provides information for collaborative work. The information processing device 500 is communicably connected to the control device 100. A display device 700 and an input device 800 are also connected to the information processing device 500. The display device 700 is, for example, a liquid crystal display. The input device 800 includes, for example, a keyboard, a mouse, a touch panel, etc.

[0039] The camera 600 captures images of the robot 200 and its surroundings. When the collaborative work system 1 includes a plurality of collaborative work processes, a camera 600 is provided for each of the plurality of collaborative work processes. The camera 600 is connected to the information processing device 500, and outputs a moving image obtained by capturing the image to the information processing device 500. The information processing device 500 may play back the moving image acquired from the camera 600 on the display device 700.

[0040] The information processing device 500 generates an "action target" for the robot 200 for a collaborative task. More specifically, the information processing device 500 receives instructions for a non-routine task (hereinafter, referred to as a "task instruction"). Such a task instruction for a non-routine task includes an instruction based on the voice of the worker 400 from the microphone 900 and an instruction based on information indicating the surrounding environment of the robot 200.

[0041] 1, the information indicating the surrounding environment indicates three-dimensional image data (point cloud data) from the sensing device 300. The information processing device 500 breaks down the received work instruction into work commands for the robot 200 for the work, and generates operation targets including a target position and a target posture of the robot 200 based on the work commands obtained by the break down. Such operation targets represent a trajectory for the manipulator 202 to move from the start position to the end position.

[0042] The control device 100 generates (calculates) a control command based on the operation target received from the information processing device 500. More specifically, the control device 100 determines (calculates) a trajectory of the tool center point from the current position to the end position of the goal based on the operation target. The control device 100 specifies (calculates) a target position and a moving speed of the tool center point for each control cycle based on the determined trajectory. The control device 100 generates a control command related to the specified target position and moving speed for each control cycle, and outputs the generated control command to the manipulator controller 204.

[0043] The manipulator controller 204 controls the robot 200 (more specifically, the manipulator 202) to move from the start position to the end position according to the control command. By such control, in the non-stationary task, the manipulator 202 moves from the start position to the end position indicated by the operation target according to the above-mentioned trajectory.

[0044] The collaborative work system 1 may include a control device 100 for each of one or more collaborative work processes. In this case, the control device 100 controls the robot 200 installed in the corresponding collaborative work process. Alternatively, the collaborative work system 1 may include a control device 100 that is common to a plurality of collaborative work processes. In this case, the control device 100 controls the robot 200 installed in each of the plurality of collaborative work processes.

[0045] Note that the sensing device 300 communicates with the control device 100 in FIG. 1, but may also communicate with the information processing device 500. Further, although the control device 100 and the information processing device 500 are independent, the control device 100 may incorporate a module of the information processing device 500.

[0046] Note that the information indicating the surrounding environment of the robot 200 is not limited to an image (point cloud data) and is changed according to the non-steady operation.

[0047] According to the cooperative work system 1 in FIG. 1, a work instruction for non-steady work is given to the information processing device 500. The information processing device 500 generates an operation target of the robot 200 from the work instruction, and the control device 100 controls the robot 200 to operate along the trajectory indicated by the operation target. In this way, if the operator 400 gives a work instruction to the cooperative work system 1, the target operation can be automatically generated. As a result, it becomes easy to introduce a robot into non-steady work.

[0048] Hereinafter, a specific example of the cooperative work system 1 according to the present embodiment will be described. <B. Hardware Configuration of Control Device> FIG. 2 is a schematic diagram showing a hardware configuration example of the control device according to the present embodiment. The control device 100 typically has a configuration according to a computer architecture.

[0049] As shown in FIG. 2, the control device 100 includes a control processing circuit 110 and a field network controller 120.

[0050] The control processing circuit 110 executes arithmetic processing necessary to drive the robot 200. As an example, the control processing circuit 110 includes a processor 112, a main memory 114, a storage 116, and interface circuits 118 and 119.

[0051] The processor 112 executes control operations for driving the robot 200. The main memory 114 is composed of a volatile storage device such as a DRAM (Dynamic Random Access Memory) or an SRAM (Static Random Access Memory), for example. The storage 116 is composed of a non-volatile storage device such as an SSD (Solid State Drive) or an HDD (Hard Disk Drive), for example.

[0052] The storage 116 stores a system program 130 including an OS (Operating System) and a user program 131 for realizing the control of the robot 200. The user program 131 includes instructions for executing control operations related to the operation of the robot 200 and instructions related to the interface with the robot 200. The input / output interface 10 and the robot interface 12 in FIG. 1 include a circuit including the interface circuits 119 and 118 in FIG. 2, or a program for controlling the interface circuits 119 and 118. The processor 112 reads a program from the storage 116, expands the read program in the main memory, and executes it.

[0053] The trajectory data 132 is data related to the trajectory of the tool center point of the manipulator 202 determined in advance.

[0054] The interface circuit 118 exchanges data with the robot 200. The interface circuit 119 exchanges data with the information processing device 500.

[0055] The field network controller 120 mainly exchanges data with the sensing device 300 via the field network.

[0056] <C. Configuration of Robot 200> 3 is a schematic diagram showing a configuration example of a robot according to the present embodiment. Referring to FIG. 3, the robot 200 is shown as, for example, an articulated robot.

[0057] The robot 200 includes a rotating unit 230, which is an example of a movable unit, and a plurality of movable units 13, 14, 15, 16, and 17. These movable units correspond to joints of the robot 200. Each of the movable units constitutes a link constituting the robot 200, which rotates in the direction of the arrow in the figure along a rotation axis (not shown). Each movable unit rotates by an amount of rotation (angle and direction of rotation) according to a control command from the manipulator controller 204. An end effector 18 is attached to the tip of the arm of the robot 200. A load sensor 19 that detects a load generated on the end effector 18 is provided at a portion of the arm of the robot 200 where the end effector 18 is attached. The load sensor 19 outputs a detection result indicating the magnitude of the generated load and the direction in which the load is generated. The detection result of the load sensor 19 may be output in the form of a kind of vector.

[0058] The end effector 18 is provided with grippers 20 for gripping a workpiece. The end effector 18 grips any workpiece placed between the grippers 20 by changing the spacing between the grippers 20 in accordance with a control command.

[0059] Any mechanism may be used as long as the robot 200 can hold a workpiece. That is, it is not limited to the gripper 20, but may be a mechanism that adsorbs a workpiece using negative pressure, or a mechanism that holds a workpiece using electromagnetic force.

[0060] In the following description, the operation of the robot 200 will be mainly described based on a coordinate system XYZ (hereinafter also referred to as the "TCP coordinate system") based on the end effector 18. In the TCP coordinate system, the vertical direction of the end effector 18 corresponds to the Z axis. In other words, the Z axis corresponds to the direction in which the workpiece gripped by the gripper 20 is inserted.

[0061] The load sensor 19 outputs, as detection results, the load in the X-axis direction (X), the load in the Y-axis direction (Y), and the load in the Z-axis direction (Z) of the TCP coordinate system, respectively, and also outputs the load (moment) in the rotation direction (RX) centered on the X-axis, the load (moment) in the rotation direction (RY) centered on the Y-axis, and the load (moment) in the rotation direction (RZ) centered on the Z-axis.

[0062] In the present embodiment, since the length of the arm of the robot 200 is fixed, the position of the robot 200 is indicated by the coordinate values (x, y, z) of the TCP coordinate system of the end effector 18, and the posture of the robot 200 is indicated by the posture of the end effector 18. The posture of the end effector 18 is indicated by the inclination angles (Rx, Ry, Rz) of the X-axis, Y-axis, and Z-axis, respectively. In this way, the position and posture of the robot 200 are indicated by the coordinate values (x, y, z) and the inclination angles (Rx, Ry, Rz).

[0063] <D. Hardware Configuration of Information Processing Device> FIG. 4 is a schematic diagram showing a hardware configuration example of an information processing device according to an embodiment. The information processing device 500 includes a processor 501 such as a CPU (Central Processing Unit) or an MPU (Micro-Processing Unit), a memory 502, a storage 503, a display controller 504, an input interface 505, a communication interface 506, a camera interface 507, and a network interface 516 for connecting to a communication network. Further, the information processing device 500 includes a force sense interface 508 that communicates with the force sense sensor 9, a microphone interface 509 that communicates audio data with the microphone 900, and a device driver 510 that reads and writes data with respect to the storage medium 901. These respective units are connected via a bus so as to be capable of data communication with each other.

[0064] The processor 501 realizes various processes according to the present embodiment by expanding and executing the programs stored in the storage 503 in the memory 502.

[0065] The memory 502 is typically a volatile storage device such as a DRAM, and stores an application program 513 read from the storage 503 .

[0066] The storage 503 is typically a non-volatile magnetic storage device such as a hard disk drive. The storage 503 stores data including a system program 512 such as an OS, an application program 513, and a machine learning learning model 514. The application program 513 or the learning model 514 to be installed in the storage 503 is downloaded from the storage medium 901 to the storage 503, or downloaded to the storage 503 via a network interface 516 and a communication network.

[0067] The storage medium 901 includes a volatile storage medium or a non-volatile storage medium, for example, a general-purpose semiconductor storage device such as a memory card such as CF (registered trademark, Compact Flash) or SD (registered trademark, Secure Digital), a magnetic storage medium such as a flexible disk, or an optical storage medium such as a CD-ROM (Compact Disk Read Only Memory).

[0068] The display controller 504 is connected to the display device 700 and outputs signals for displaying various types of information to the display device 700 in accordance with internal commands from the processor 501 .

[0069] The input interface 505 mediates data transmission between the processor 501 and an input device 800 such as a keyboard, a mouse, a touch panel, a dedicated console, etc. In other words, the input interface 505 accepts operation commands given by a user operating the input device 800.

[0070] The communication interface 506 mediates data transmission between the processor 501 and external devices (such as the control device 100 (see FIG. 1), a work database not shown, a production management device not shown, etc.). The communication interface 506 typically includes Ethernet (registered trademark), USB (Universal Serial Bus), and the like. Note that the application program 513 may be downloaded from a distribution server or the like via the communication interface 506.

[0071] When using a computer having a structure according to the computer architecture as described above, the application program 513 for providing the functions according to the present embodiment is executed in cooperation with the OS.

[0072] In the embodiment, a configuration example in which the necessary processing is realized by the processor 501 executing the application program 513 has been shown. However, some or all of the provided processing may be implemented using a dedicated hardware circuit (for example, a dedicated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array)).

[0073] <E. Module Configuration of Information Processing Apparatus 500> FIG. 5 is a diagram showing a schematic configuration example of a module of an information processing device according to the present embodiment. In FIG. 5, the information processing device 500 has a memory unit 50, a reception unit 51, an analysis unit 52, a goal generation unit 53, and an adjustment unit 54. The memory unit 50 is realized by the memory 502 and the storage 503 shown in FIG. 4. The reception unit 51 is realized by the communication interface 506, the microphone interface 509, the haptic interface 508, and the processor 501 that executes the application program 513 shown in FIG. 4. The analysis unit 52, the goal generation unit 53, and the adjustment unit 54 are realized by the processor 501 that executes the application program 513. The memory unit 50 stores the learning model 514, the movement target 320, the voice data 57 transferred from the microphone 900, and the data of the video 58 captured by the camera 600. The voice data 57 includes the voice and the time when the voice was collected.

[0074] The reception unit 51, the analysis unit 52, the goal generation unit 53, and the adjustment unit 54 will be described in detail later with reference to FIGS.

[0075] The learning model 514 receives a work instruction as input and outputs a work command. In this manner, the learning model 514 can decompose the work instruction into one or more work commands determined from the work instruction. In this specification, "decomposing a work command" refers to identifying the type of the work command.

[0076] Such a learning model 514 includes, for example, a neural network. In a neural network, a weight is assigned to each node for an appropriate output relative to an input. The weight is determined by machine learning using training data.

[0077] More specifically, in a neural network, when multiple inputs are given to the input layer, each input is multiplied by a weight in the input layer, and the multiplication result is sent to the next layer, where the multiplication result is multiplied by a weight, and the multiplication result is sent to the next layer, and finally, an output is obtained from the output layer.

[0078] Machine learning of the learning model 514 is performed as follows, for example. First, a large number of work instructions with known types of work commands are obtained. Next, a set of a work instruction that is an input to the learning model 514 and the type of work command that should be the output (correct answer) for the input is used as learning data to train the learning model 514. As a result, the learning model 514 becomes a model that can derive the output that should be for the input.

[0079] Note that the "decompose work command" process described above can also be realized by a module that does not use the learning model 514. For example, table search can be used. More specifically, the storage unit 50 stores a table in which a plurality of operation patterns having a sentence (text data) indicating a work instruction for each of a plurality of non-steady operations, the type of work, the position of the worker, and an operation command are registered. The information processing device 500 obtains a sentence semantically associated from the speech 290 of the utterance by voice recognition processing and natural language processing, and performs image processing on the image 280 to obtain the type of work and the position information of the worker 400. The information processing device 500 searches the table based on the set of these obtained information. By the search, a pattern having information that matches the set of the sentence, the type of work, and the position information is extracted from the table, and an operation command is obtained from the extracted pattern. In this way, by searching the table, it is possible to "decompose the work command" from the work instruction.

[0080] <F. Process of generating and adjusting operation target> FIG. 6 is a diagram schematically showing an example of a process of generating and adjusting an operation target according to the present embodiment. FIG. 7 is a diagram schematically showing the relationship among a work instruction, a work command, and an operation target according to the present embodiment. With reference to FIGS. 6 and 7, the process of generating and adjusting the operation target will be described.

[0081] The reception unit 51 receives a work instruction. The work instruction includes an instruction based on an image (point cloud data) from the sensing device 300, an instruction based on a voice uttered from the microphone 900, a text-based instruction from the input device 800, and the like. For example, when the worker 400 utters "attach the member 5 to the member 6," the voice 290 (FIG. 7) of the utterance is shown as an instruction based on a voice. When the worker 400 indicates the work with a pointing gesture, the image 280 (FIG. 7) of the pointing finger of the worker 400 included in the image from the sensing device 300 is shown as an instruction based on an image. Note that the gesture of the worker 400 representing the work instruction is not limited to pointing. In FIG. 7, a three-dimensional space 305 configured by point cloud data of the field of view from the sensing device 300 is shown. The three-dimensional space 305 corresponds to a coordinate system in which the three axes of XYZ are mutually orthogonal.

[0082] The analysis unit 52 acquires a work instruction by using the learning model 514 (process T51 in FIG. 6). More specifically, the learning model 514 receives the work instruction (e.g., data of the voice 290 and the image 280) received by the reception unit 51 as an input, and outputs, for example, the work instruction 310 (type of work instruction) in FIG. 7. The image 280 is composed of a first image 280 pointing at the component 5 and a second image 280 pointing at the component 6. The learning model 514 receives the positions of the first and second images 280 in the three-dimensional space 305 as an input, and outputs the start position PS and end position PE of the work, which are three-dimensional coordinate values, respectively. The work instruction 310 includes different types of work instructions 311 and 312, as shown in FIG. 7. More specifically, the work command 311 indicates that the end effector 18 picks up a workpiece in a posture PSS at the work start position PS, and the work command 312 indicates that the end effector 18 places the workpiece in a posture PSE at the work end position PE. In this way, the work instructions of the worker 400 are decomposed into multiple types of work commands 311 and 312 for the robot 200.

[0083] The posture PSS and the posture PSE are determined, for example, as follows: The information processing device 500 receives, as a work instruction, the moments of the X-axis, the Y-axis, and the Z-axis detected by the force sensor 9 when the worker 400 picks up the member 5 on the table 8. The information processing device 500, as the target generation unit 53, sets these moments of the X-axis, the Y-axis, and the Z-axis in a predetermined function, and executes a calculation based on the function to calculate the posture PSS.

[0084] Similarly, for example, the information processing device 500 receives, as a work instruction, moments about the X-axis, Y-axis, and Z-axis detected by the force sensor 9 when the worker 400 places the member 5. The information processing device 500, as the target generation unit 53, sets these moments about the X-axis, Y-axis, and Z-axis in a predetermined function, and executes a calculation based on the function to calculate the posture PSE. Note that the method of determining the posture PSS and the posture PSE is not limited to this.

[0085] The goal generator 53 generates an operation target including a target position and a target posture of the robot 200 based on the task command 310 (i.e., the decomposed task commands 311, 312) from the analyzer 52 (process T52 in FIG. 6). More specifically, the analyzer 52 generates an operation target which is a trajectory (path) from the start position PS to the end position PE of the end effector 18. The goal generator 53 calculates such a trajectory according to, for example, an algorithm of RRT (Rapidly-exploring Random Trees). Note that the generation of the trajectory is not limited to the method using RRT.

[0086] In RRT, a trajectory is searched for by growing a tree that connects the point clouds between the start position PS and the end position PE in a three-dimensional space indicated by an image of the point cloud from the sensing device 300. Since the search is performed in a three-dimensional space in which joints are movable, each point constituting the tree is assigned a vector corresponding to the posture of the robot 200 (more specifically, the end effector 18). For example, if the robot 200 has six joints, the vector is six-dimensional. In such a search, an obstacle is identified from the image of the point cloud, and a trajectory is searched for to avoid the identified obstacle. In the working area, the worker 400 becomes an obstacle to the robot 200's movement. Therefore, the target generator 53 searches for a trajectory that does not interfere with the partial area of ​​the obstacle between the robot 200 and the worker 400. In the search, a trajectory is searched for such that the movement time does not exceed a predetermined time.

[0087] 7, a motion target 320 is generated based on the task command 310 through such a search and stored in the storage unit 50. More specifically, the motion target 320 includes, for example, trajectories 321 shown by solid lines corresponding to each of the axes No. 1 to No. 6 of the six joints.

[0088] The adjustment unit 54 adjusts the movement target 320 stored in the storage unit 50 (process T53 in FIG. 6). This adjustment is performed based on a work instruction received by the reception unit 51, for example, and is repeatedly performed each time a work instruction is received.

[0089] More specifically, for example, when the worker 400 utters "move a little further away," the adjustment unit 54 expands the area of ​​the work command 310 where the obstacle (worker 400) is located, i.e., the obstacle area, and searches for a trajectory to avoid that area. The adjustment unit 54 stores such an adjusted movement target 320 in the storage unit 50. Through such adjustment, the original solid line trajectory 321 of the movement target 320 is changed to a dashed line trajectory 322 as shown in FIG. 7.

[0090] In this embodiment, a part of the module shown in FIG. 5 can be mounted on a computer on the cloud (not shown). For example, one or more of the analysis unit 52, the target generation unit 53, and the adjustment unit 54 that utilize the learning model 514 may be mounted on a computer on the cloud. In this case, the information processing apparatus 500 exchanges data with the computer on the cloud via the network interface 516.

[0091] <G. Control of Robot> As shown in FIG. 3, the robot 200 has a movable part that can be moved by a plurality of axes. The manipulator controller 204 corresponds to a drive unit that drives the movable part. The axes of the movable part are driven by corresponding servo motors (not shown) to rotate. The manipulator controller 204 drives the servo motor according to a control command. By controlling the movement of each arm according to the control command in this way, the movement of the robot 200 is realized. The movement amount (movement direction, movement distance) of the robot 200 is determined by the rotation amount (rotation direction, angle) of the servo motor indicated by the control command.

[0092] FIG. 8 is a diagram schematically showing an example of an operation target according to this embodiment. FIG. 8 shows an operation target 320 for one axis. The operation target 320 is data in which the target position changes over time for the axis. The speed and trajectory of the movement of the arm corresponding to the axis change so as to be the speed and trajectory according to the target operation. For each axis of the robot 200, an operation target similar to FIG. 8 is generated.

[0093] FIG. 9 is a diagram showing a schematic diagram of a process of calculating a control command from a target position of an axis according to the present embodiment. As shown in FIG. 9, a servo motor rotation amount (αA, αB, αC) corresponding to the axis can be calculated by applying a predetermined function (for example, a kinematic function) to the three-dimensional coordinate P (x, y, z) which is the target position (FIG. 8) of the axis in the three-dimensional space. The servo motor rotation amounts (αA, αB, αC) corresponding to the target positions of the other axes can be calculated in the same manner. FIG. 9 shows a method of determining the servo motor rotation amount for the target position, but for the tilt angle (Rx, Ry, Rz) corresponding to the target posture, a predetermined function can be applied to the tilt angle (Rx, Ry, Rz) to calculate the rotation amount of the servo motor that controls the tilt. In this way, a control command for the robot 200 can be generated from the operation target 320.

[0094] <H.モニタ> 10 is a diagram showing an example of a display screen according to the present embodiment. For example, display device 700 displays screen 60B of FIG.

[0095] Screen 60B includes area 63 and area 65. Area 63 displays an image of graph 70a showing the time series change in the degree of approach of robot 200 (more specifically, end effector 18) to worker 400 while robot 200 moves from a start position to an end position in a collaborative task. The shorter the distance between robot 200 and worker 400, the greater the degree of approach. Area 65 displays video from camera 600.

[0096] The information processing device 500 detects a change in the position (three-dimensional coordinates M(x, y, z)) of the worker 400 in three-dimensional space from a point cloud image acquired by the sensing device 300 during a collaborative task. The information processing device 500 compares the three-dimensional coordinates P(x, y, z) of the axis indicated by the motion target 320 in each control cycle during the task with the three-dimensional coordinates M(x, y, z) of the worker in each control cycle, and calculates a proximity indicating the degree to which the robot 200 (more specifically, the arm and the end effector 18) approaches the worker 400 from the comparison result and a predetermined arithmetic expression. A graph 70a shows a time-series change in the proximity calculated for each control cycle. The graph 70a in FIG. 10 shows that the robot 200 approaches the worker 400 closer at the end position PE of the task than at the start position PS.

[0097] <I.フローチャート> 11 is a flowchart of processing according to this embodiment. This flowchart is a method implemented by a computer for collaborative work according to this embodiment. The method is mainly realized by the processor 501 of the information processing device 500 executing the application program 513. Note that some of the processing in the flowchart can be realized by the processor 112 of the control device 100 executing the user program 131.

[0098] The information processing device 500 receives a work instruction as the reception unit 51 (step S1). More specifically, the information processing device 500 collects data of the voice 290 of the speech from the microphone 900 (step S1a), and receives data of the image 280 of the point cloud from the sensing device 300 (step S1c). The information processing device 500 performs pre-processing on each of the data of the voice 290 and the image 280 (steps S1b and S1d). Such pre-processing includes, but is not limited to, removing noise from the data of the voice 290 and the image 280 and extracting features for processing using the learning model 514 described later. The extracted features can be treated as input to the learning model 514.

[0099] The information processing device 500, as the analysis unit 52, breaks down the work instruction received by the reception unit 51 into work commands for the robot 200 (step S2). Such processing by the analysis unit 52 includes a process of “breaking down the work command” using the learning model 514.

[0100] The information processing device 500, as the goal generating unit 53, generates an operation target including a target position and a target posture of the robot 200 based on the decomposed task command (step S3), and transfers the operation target to the control device 100.

[0101] The control device 100 generates (calculates) a control command based on the motion target from the information processing device 500 (step S4), and outputs the generated control command to the manipulator controller 204.

[0102] The manipulator controller 204 starts to control the robot 200 so that the robot 200 moves from the start position to the end position according to the control command (step S5). After the start of such movement, the control device 100 judges whether there is a disturbance input (step S6). Such a disturbance input refers to interference (force) that the robot 200 receives from the outside while the robot 200 is operating (moving). The disturbance input includes, for example, a force applied to the arm of the robot 200, a force applied to a workpiece that the end effector 18 has grasped, and the like. The subject that applies such a force to the robot 200 includes a person (worker 400) or an object (equipment, etc.). A gravity sensor provided in association with the axis of the joint of the robot 200 detects such a force (load). The force detected for each axis is transferred from the robot 200 to the control device 100.

[0103] The control device 100 determines whether there is an external disturbance input based on whether the force (load) transferred from the gravity sensor exceeds a first threshold value (step S6). When it is determined that there is no external disturbance input (NO in step S6), the control device 100 controls the robot 200 to move via the manipulator controller 204 until it reaches the end position of the goal (step S7). When the robot 200 reaches the end position, the manipulator controller 204 enters a state of waiting for a control command for the next work instruction.

[0104] When it is determined that there is an external disturbance input (YES in step S6), the control device 100 determines whether the force detected by the gravity sensor of each axis satisfies a predetermined condition (step S8). This predetermined condition indicates, for example, that the load transferred from the gravity sensor of each axis exceeds a second threshold value (where the first threshold value < the second threshold value).

[0105] When the control device 100 determines that the force detected by the gravity sensor of each axis does not satisfy the predetermined condition (NO in step S8), it proceeds to step S7. When it is determined that the force detected by the gravity sensor of each axis satisfies the predetermined condition (YES in step S8), the control device 100 outputs a stop signal to the manipulator controller 204 (step S9). The manipulator controller 204 stops the robot 200 according to the stop signal.

[0106] <J. Appendix> This embodiment includes the following disclosure.

[0107] (Configuration 1) A cooperative work system (1) in which a robot (200) works in cooperation with a worker (400), a reception unit (51) that receives a work instruction, an analysis unit (52) that decomposes the received work instruction into work commands (311, 312) for the robot, a target generating unit (53) that generates an operation target (320) including a target position and a target posture of the robot based on the decomposed task command.

[0108] (Configuration 2) The instructions for the work are: 2. The collaborative work system according to configuration 1, including instructions based on a voice (290) spoken by the worker and instructions based on information (280) indicating the surrounding environment of the robot.

[0109] (Configuration 3) The robot operates in the same work area as the worker, 3. The collaborative work system of claim 2, wherein the information indicative of the surrounding environment includes a three-dimensional image representing a space corresponding to the work area.

[0110] (Configuration 4) 4. The collaborative work system of claim 3, wherein the three-dimensional image includes an image representing a gesture of the worker in the work area.

[0111] (Configuration 5) 5. The collaborative work system of claim 4, wherein the gesture of the worker includes pointing at an object related to the work in the work area.

[0112] (Configuration 6) A storage unit (50) for storing the operation target; 3. The collaborative work system according to configuration 1 or 2, further comprising an adjustment unit (54) that adjusts the operation target of the storage unit based on an instruction received by the reception unit.

[0113] (Configuration 7) The work includes an operation in which the robot handles a workpiece in cooperation with the worker, 3. The collaborative work system according to claim 1, wherein the work instructions include instructions based on the force applied by the worker to the workpiece.

[0114] (Configuration 8) The robot includes a movable part including an arm and a drive part (204) that drives the movable part; The collaborative work system includes: 3. The collaborative work system according to configuration 1 or 2, further comprising: a control command for controlling the driving unit based on the operation target so that the movable unit operates in accordance with the operation target.

[0115] (Configuration 9) A computer-implemented method (500), comprising: The robot (200) receives instructions for a task to be performed in cooperation with a worker (400); Breaking down the received work instructions into work commands (311, 312) for the robot; and generating an action goal (320) including a target position and a target posture of the robot based on an instruction for the decomposed task.

[0116] (Configuration 10) A program (513) for causing a computer (500) to carry out a method, The method comprises: The robot (200) receives instructions for a task to be performed in cooperation with a worker (400); Breaking down the received work instructions into work commands (311, 312) for the robot; and generating an operation target (320) including a target position and a target posture of the robot based on an instruction for the decomposed task.

[0117] The embodiments disclosed herein should be considered to be illustrative and not restrictive in all respects. The scope of the present invention is defined by the claims, not the above description, and is intended to include all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0118] 1 Collaborative work system, 5,6 Member, 8 Table, 9 Force sensor, 10 Input / output interface, 12 Robot interface, 13,14,15,16,17 Moving part, 18 End effector, 19 Load sensor, 20 Gripper, 50 Memory unit, 51 Reception unit, 52 Analysis unit, 53 Target generation unit, 54 Adjustment unit, 57 Voice data, 58 Video, 60B Screen, 63,65 Area, 70a Graph, 100 Control device, 110 Control processing circuit, 112,501 Processor, 114 Main memory, 116,503 Storage, 118,119 Interface circuit, 120 Field network controller, 130,512 System program, 131 User program, 132 Trajectory data, 200,202 Manipulator, 204 Manipulator controller, 211, 212, 213 Arm, 221, 222, 223 Joint, 230 Rotating part, 240 Base part, 280 Image, 290 Voice, 300 Sensing device, 305 Three-dimensional space, 310, 311, 312 Work instruction, 320 Movement target, 321, 322 Trajectory, 400 Worker, 500 Information processing device, 502 Memory, 504 Display controller, 505 Input interface, 506 Communication interface, 507 Camera interface, 508 Force sense interface, 509 Microphone interface, 510 Device driver, 513 Application program, 514 Learning model, 516 Network interface, 600 Camera, 700 Display device, 800 Input device, 900 Microphone, 901 Storage medium, PE End position, PS Start position, PSE, PSS Posture.

Claims

1. A collaborative work system in which a robot works in collaboration with a worker, A reception unit that receives work instructions; an analysis unit that breaks down the received work instruction into work commands for the robot; a target generating unit that generates an operation target including a target position and a target posture of the robot based on a command for the decomposed task.

2. The instructions for the work are: The collaborative work system according to claim 1 , including instructions based on voice uttered by the worker and instructions based on information indicating a surrounding environment of the robot.

3. The robot operates in the same work area as the worker, The collaborative work system according to claim 2 , wherein the information indicating the surrounding environment includes a three-dimensional image representing a space corresponding to the work area.

4. The collaborative work system according to claim 3 , wherein the three-dimensional image includes an image representing a gesture of the worker in the work area.

5. The collaborative work system according to claim 4 , wherein the gesture of the worker includes pointing at an object related to the work in the work area.

6. A storage unit for storing the operation target; The collaborative work system according to claim 1 , further comprising an adjustment unit that adjusts the operation target of the storage unit based on an instruction received by the reception unit.

7. The work includes an operation in which the robot handles a workpiece in cooperation with the worker, The collaborative work system according to claim 1 , wherein the task instruction includes an instruction based on a force applied by the worker to the workpiece.

8. The robot includes a movable unit including an arm and a drive unit that drives the movable unit, The collaborative work system includes: The collaborative work system according to claim 1 , further comprising: a control command for controlling the driving unit based on the operation target so that the movable unit operates in accordance with the operation target.

9. 1. A computer-implemented method comprising: The robot receives instructions for a task to be performed in cooperation with a worker; Breaking down the received work instructions into work commands for the robot; and generating an operation goal including a target position and a target posture of the robot based on a decomposed task command.

10. A program for causing a computer to carry out a method, The method comprises: The robot receives instructions for a task to be performed in cooperation with a worker; Breaking down the received work instructions into work commands for the robot; and generating an operation target including a target position and a target posture of the robot based on an instruction for the decomposed task.

Images