Robot control system, robot control method, and robot control program

Abstract

To teach a task to a robot easily.SOLUTION: A robot control system includes: an inquiry part which makes an inquiry to a user when a robot performs a task; a specifying part which specifies user's reaction to the inquiry by using a sensor; a complementary part which complements at least a part of the task based on the specified action; and a robot control unit which controls the robot so that the robot carries out the complemented task.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] One aspect of the present disclosure relates to a robot control system, a robot control method, and a robot control program.

Background Art

[0002] Patent Document 1 describes a robot teaching system for suppressing an increase in the operation burden on a user. This system includes a teaching tool including an operation unit operated by a user to specify a teaching position, a measurement unit that measures the position and orientation of the teaching position specified by the teaching tool, and a control unit that determines the teaching position of the robot based on the position and orientation.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] A mechanism for easily teaching a task to a robot is desired.

Means for Solving the Problems

[0005] A robot control system according to one aspect of the present disclosure includes an inquiry unit that executes an inquiry to a user while the robot is executing a task, an identification unit that identifies a user's action with respect to the inquiry using a sensor, a complement unit that complements at least a part of the task based on the identified action, and a robot control unit that controls the robot so that the robot executes the complemented task.

[0006] A robot control method relating to one aspect of the present disclosure is a robot control method performed by a robot control system having at least one processor, and includes the steps of: making a query to a user while the robot is performing a task; identifying the user's action in response to the query using sensors; completing at least a portion of the task based on the identified action; and controlling the robot so that it performs the completed task.

[0007] A robot control program relating to one aspect of this disclosure causes a computer to perform the following steps while the robot is performing a task: to make a query to a user; to identify the user's action in response to the query using sensors; to complete at least a part of the task based on the identified action; and to control the robot so that it performs the completed task. [Effects of the Invention]

[0008] According to one aspect of this disclosure, robots can be easily taught tasks. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an example of the application of a robot control system. [Figure 2] This figure shows an example of a computer hardware configuration used for a robot control system. [Figure 3] This flowchart shows an example of processing in a robot control system. [Figure 4] This flowchart shows an example of the process for defining a task. [Figure 5] This flowchart shows an example of a process that complements the work. [Figure 6] This flowchart shows an example of a process for interpolating the target position. [Figure 7] This figure shows an example of a document. [Figure 8]This diagram shows scenes related to document identification. [Figure 9] This diagram shows scenes related to work completion. [Figure 10] This figure shows a scenario related to the completion of the target position. [Figure 11] This figure shows a scenario related to the completion of the target position. [Figure 12] This figure shows a scenario related to the completion of the target position. [Modes for carrying out the invention]

[0010] The embodiments described herein will be described in detail below with reference to the attached drawings. In the description of the drawings, the same or equivalent elements are denoted by the same reference numerals, and redundant descriptions are omitted.

[0011] [System Configuration] As an example, the robot control system relating to this disclosure is shown as a component of robot system 1. Robot system 1 is a mechanism that automates a given task by having a robot perform actions to achieve a given objective.

[0012] Figure 1 shows an example of the configuration of a robot system 1, as well as an example of the application of a robot control system. In this example, the robot system 1 comprises a robot control system 10, one or more robots 2, and one or more robot controllers 3 corresponding to one or more robots 2. Figure 1 shows one robot 2 and one robot controller 3, illustrating a configuration in which one robot 2 is connected to one robot controller 3. In another example, the robot control system 10 may be connected to multiple pairs of robots 2 and robot controllers 3. Alternatively, one robot controller 3 may be connected to multiple robots 2. The communication network connecting the devices may be a wired network or a wireless network. The communication network may consist of at least one of the internet and an intranet. Alternatively, the communication network may simply be implemented by a single communication cable.

[0013] The robot control system 10 is a computer system for autonomously operating the robot 2 in at least some situations. The robot control system 10 executes a given operation to generate a command signal for controlling the robot 2. In one example, the command signal includes data for controlling the robot 2, for example, a path indicating the trajectory of the robot 2. The trajectory of the robot 2 refers to the path of the movement of the robot 2 or its components. For example, the trajectory of the robot 2 can be the trajectory of the tip. The robot control system 10 transmits the generated command signal towards the robot controller 3.

[0014] The robot controller 3 is a device that operates the robot 2 according to a command signal from the robot control system 10. In one example, the robot controller 3 calculates a joint angle target value (the angle target value of each joint of the robot 2) for matching the position and orientation of the tip with the target value indicated by the command signal, and controls the robot 2 according to the angle target value.

[0015] The robot 2 is a device or machine that works in place of a person. In one example, the robot 2 is a multi-axis serial link type vertical articulated robot. The robot 2 includes a manipulator 2a and an end effector 2b which is a tool attached to the tip of the manipulator 2a. The robot 2 can execute various processes using its end effector 2b. The robot 2 can freely change the position and orientation of the end effector 2b within a given range. The robot 2 may be a 6-axis vertical articulated robot or a 7-axis vertical articulated robot with one redundant axis added to the 6 axes.

[0016] Robot 2 operates based on the control by the robot control system 10 to execute a given task. In the present disclosure, a task refers to a series of processes to be executed by the robot 2 to achieve a certain purpose. By the robot 2 executing the task, the result desired by the user of the robot system 1 can be obtained. For example, a task is set to process some workpiece. Examples of tasks include "grip the workpiece and place it on the conveyor", "grab the workpiece and attach the workpiece to another structure", and "spray-paint the workpiece". A workpiece refers to a physical object processed by the robot 2.

[0017] The robot control system 10 assists the user in teaching the robot 2. Teaching refers to the operation of directly or indirectly transmitting the work to be executed to the robot. Conventionally, the user executes teaching for the robot through a user interface such as a programming pendant, a keyboard, a mouse, or a dedicated controller. However, in that conventional method, the user has to memorize complex commands that are difficult to intuitively understand or acquire complex operation methods of the user interface for teaching. Therefore, a lot of time is wasted until the user can perform teaching. In addition, costs for introducing a dedicated user interface are incurred.

[0018] The robot control system 10 performs teaching to robot 2 based on interaction with the user. The robot control system 10 queries the user while robot 2 is performing a task. When robot 2 is performing a task, it means that robot 2 has started operating based on a predetermined program. Therefore, the robot control system 10 queries the user when robot 2 is online. When robot 2 is performing a task, this includes when robot 2 is in a standby position. A query to the user refers to the process of requesting instruction input from the user. The robot control system 10 identifies the user's action in response to the query using sensors. Based on that action, the robot control system 10 completes part of the task and causes robot 2 to perform the completed task. An action refers to an expression resulting from the user's actions. For example, an action is an expression that can be perceived through a person's sight or hearing. Examples of actions include gestures, sound production (e.g., clapping, vocalization, etc.), and presentation of objects. An action indicates an instruction from the user. The robot control system 10 generates tasks through interaction with the user while robot 2 is performing the task. For example, this mechanism makes it possible to have the robot 2 process workpieces while flexibly adapting to changes in the environment of the real-world workspace where the robot 2 is located.

[0019] In the robot control system 10, the user only needs to perform a predetermined action toward a sensor in their workspace for teaching purposes. The user does not need to operate a user interface for teaching. For example, the action is similar to the user's normal movements and does not require the user to have any special skills. Therefore, the user can easily take the predetermined action and give instructions to the robot 2 more intuitively. Thus, even users unfamiliar with robot operation can easily teach the robot 2 a task. In contrast to conventional teaching, which is highly specific and machine-centered, teaching in the robot control system 10 can be said to be abstract and human-centered.

[0020] Examples of sensors for identifying user actions include visual sensors such as cameras and auditory sensors such as microphones. In one example, the robot control system 10 uses a camera 20 and a speakerphone 30 as sensors. The camera 20 is an imaging device that captures images of the area around the end effector 2b. The camera 20 may be placed on the manipulator 2a, or it may be attached, for example, near the tip of the manipulator 2a. The camera 20 moves in response to the movement of the robot 2. This movement may include a change in at least one of the position and orientation of the camera 20. The camera 20 may be located in a different place from the robot 2, as long as it moves in response to the movement of the robot 2. The speakerphone 30 is a device that integrates a microphone and a speaker. The speakerphone 30 detects sounds around the robot 2 and outputs sound to the user. The speakerphone 30 may be placed on the floor or on the manipulator 2a.

[0021] Figure 1 also shows an example of the functional configuration of the robot control system 10. In this example, the robot control system 10 comprises an inquiry unit 11, a specification unit 12, a task management unit 13, and a robot control unit 14 as functional components. The inquiry unit 11 is a functional module that makes inquiries to the user while the robot is executing a task. The specification unit 12 is a functional module that uses sensors to identify the user's action in response to that inquiry. The task management unit 13 is a functional module that generates tasks based on that action. In this example, the task management unit 13 comprises a definition unit 15, a completion unit 16, and a storage unit 17. The definition unit 15 is a functional module that defines tasks. The completion unit 16 is a functional module that completes the defined tasks and generates them. The storage unit 17 is a functional module that stores at least a portion of the completed tasks as task parts. The robot control unit 14 is a functional module that controls the robot 2 so that it executes the generated tasks.

[0022] Task definition refers to the process of setting one or more actions of the robot that make up the task. For example, task definition is the process of setting only a part of the task, not the whole thing. For example, definition unit 15 defines a task without setting at least one of the work to be processed by the task or the target position of the task. Task completion refers to the process of setting the unset parts of a task. The task is generated when the task is completed by completion unit 16. For example, definition unit 15 defines a task consisting of two instructions: "Place <object A> at <location P>" and "Terminate if the object is not found." In these instructions, "Place ~ at ..." and "Terminate if the object is not found" are syntax related to the task. Definition unit 15 defines the task based on this syntax. On the other hand, "object A" and "location P" in those instructions correspond to variables whose definition has been reserved as unset parts. Completion unit 16 sets these unset parts. For example, completion unit 16 sets "red object" for "object A" and a specific location for "location P". If we conveniently represent that location as position (Xa, Ya), the completion unit 16 completes the command "Place <object A> at <location P>" to "Place the red object at position (Xa, Ya)" and generates the task.

[0023] The robot control system 10 can be implemented using any type of computer. This computer may be a general-purpose computer such as a personal computer or a business server, or it may be incorporated into a dedicated device that performs specific processing.

[0024] Figure 2 shows an example of the hardware configuration of a computer 100 used for a robot control system 10. In this example, the computer 100 comprises a main unit 110, a monitor 120, and an input device 130.

[0025] The main unit 110 is a device having a circuit 160. The circuit 160 has at least one processor 161, memory 162, storage 163, input / output port 164, and communication port 165. The storage 163 stores programs for configuring each functional module of the main unit 110. The storage 163 is a computer-readable recording medium such as a hard disk, non-volatile semiconductor memory, magnetic disk, or optical disk. The memory 162 temporarily stores programs loaded from the storage 163, calculation results of the processor 161, etc. The processor 161 configures each functional module by executing programs in cooperation with the memory 162. The input / output port 164 performs input and output of electrical signals to and from the monitor 120 or input device 130 in response to commands from the processor 161. The input / output port 164 may also perform input and output of electrical signals to and from other devices such as the robot controller 3, camera 20, and speakerphone 30. The communication port 165 performs data communication with other devices via the communication network N in response to commands from the processor 161.

[0026] The monitor 120 is a device for displaying information output from the main unit 110. For example, the monitor 120 is a device capable of displaying graphics, such as an LCD panel.

[0027] The input device 130 is a device for inputting information into the main unit 110. Examples of input devices 130 include operation interfaces such as keypads, mice, and operation controllers.

[0028] The monitor 120 and the input device 130 may be integrated as a touch panel. For example, the main unit 110, the monitor 120, and the input device 130 may be integrated as a single unit, similar to a tablet computer.

[0029] Each functional module of the robot control system 10 is implemented by loading a robot control program onto the processor 161 or memory 162 and having the processor 161 execute that program. The robot control program includes code for implementing each functional module of the robot control system 10. The processor 161 operates the input / output ports 164 and communication ports 165 according to the robot control program and performs data reading and writing to the memory 162 or storage 163.

[0030] The robot control program may be provided on a non-temporary recording medium such as a CD-ROM, DVD-ROM, or semiconductor memory. Alternatively, the robot control program may be provided via a communication network as a data signal superimposed on a carrier wave.

[0031] [Robot control method] As an example of the robot control method relating to this disclosure, the process performed by the robot control system 10 will be described with reference to Figures 3 to 6. Figure 3 is a flowchart showing an example of this process as process flow S1. That is, the robot control system 10 executes process flow S1. Figures 4 to 6 are flowcharts showing details of a part of process flow S1. Figure 4 shows an example of the process of defining a task, Figure 5 shows an example of the process of complementing the workpiece, and Figure 6 shows an example of the process of complementing the target position.

[0032] In step S11, the robot control system 10 defines the task based on the document. The document is a record of information for defining the task. Therefore, the document indicates the task. The document can also be said to be an instruction manual for the robot 2. The definition of the task will be explained in detail with reference to Figure 4.

[0033] In step S111, the inquiry unit 11 inquires with the user about the document. Document inquiry refers to the process of requesting the user to input the document. In one example, the inquiry unit 11 operates robot 2 so that it assumes a predetermined inquiry posture, which is a posture that responds to inquiries. When robot 2 assumes the inquiry posture, the inquiry unit 11 may output a guidance voice such as "Please show me the instructions" from the speakerphone 30. In one example, the inquiry unit 11 executes the inquiry after robot 2 assumes the inquiry posture.

[0034] In step S112, the identification unit 12 identifies a document from the image captured by the camera 20. The user presents the document to the camera 20. The document may be written on a medium such as a board or sheet, or it may be displayed electronically on a display device. Presenting the document is one example of a user action in response to an inquiry. In one example, the identification unit 12 pre-stores recognition targets set to identify a document from an image. Recognition targets refer to information pre-set regarding the user's action. The identification unit 12 extracts these recognition targets from the image and identifies the document based on these recognition targets.

[0035] In step S113, the definition unit 15 defines the task based on the document. The definition unit 15 sets one or more actions of the robot 2 that constitute the task according to the syntax shown in the document. For example, the definition unit 15 sets various parameters or conditions related to the actions of the robot 2, such as the robot 2's operating speed and the task completion conditions. At this stage, the task includes undefined parts. The definition unit 15 sets the actions of the robot 2 at the start of the task, i.e., the initial actions of the robot 2 in the task. The definition unit 15 may output an audio message from the speakerphone 30 indicating that the task has been defined.

[0036] In step S114, the robot control unit 14 begins executing the defined task. At this stage, the entire task is not yet complete, but the actions required to start the task are set. Therefore, the robot control unit 14 can begin executing the task. However, for the robot 2 to execute the entire task, values ​​must be assigned to the variables indicated by the documentation, and the task must be completed. This completion is performed from step S12 onward.

[0037] Returning to Figure 3, in step S12, the robot control system 10 complements the workpieces to be processed by the defined task. In one example, the query unit 11 queries the user regarding the variables corresponding to the workpieces. Subsequently, the identification unit 12 identifies the user's action in response to that query. Then, the complementation unit 16 complements the task by substituting the workpieces to be processed by the robot 2 into the variables based on that action. The complementation of the task by workpieces will be explained in detail with reference to Figure 5.

[0038] In step S121, the inquiry unit 11 inquires with the user about a reference object. A reference object is an object that indicates information necessary to identify a workpiece. The reference object may be the same object as the workpiece to be processed, or it may be a different object from the workpiece. Inquiring about a reference object is the process of requesting the user to input a reference object. In one example, the inquiry unit 11 operates the robot 2 so that it assumes a predetermined inquiry posture. When the robot 2 assumes the inquiry posture, the inquiry unit 11 may output a voice prompt such as "Please show me the reference object" from the speakerphone 30. In one example, the inquiry unit 11 executes the inquiry after the robot 2 assumes the inquiry posture.

[0039] In step S122, the identification unit 12 identifies the subject from the image captured by the camera 20. In one example, the user presents a reference object to the camera 20. The reference object may be a physical object, written on a medium such as a board or sheet, or displayed electronically on a display device. Alternatively, the user presents a pre-configured shortcut gesture to the camera to complement the work as part of the task. The shortcut gesture is represented by a hand gesture, i.e., a hand shape. Both presenting a reference object and a shortcut gesture are examples of user actions in response to an inquiry. In one example, the identification unit 12 pre-stores recognition targets configured to identify the subject from the image. The identification unit 12 extracts either the reference object or the user's hand from the image as its recognition target and identifies the subject based on that recognition target. If a reference object is extracted, the identification unit 12 identifies the feature quantities of that reference object as user actions. The feature quantities of a reference object are physical quantities that indicate the characteristics or properties of the reference object. For example, physical quantities may indicate color, shape, texture, or type of object. When the user's hand is extracted and its hand shape corresponds to a shortcut gesture, the identification unit 12 identifies that shortcut gesture as the user's action.

[0040] As shown in step S123, the identification unit 12 identifies either the feature quantities of the reference object or the shortcut gesture as the subject. If the feature quantities of the reference object are identified, the process proceeds to step S124. If the shortcut gesture is identified, the process proceeds to step S125.

[0041] In step S124, the completion unit 16 sets up a work based on the feature quantities of the reference object and completes the task with that work. That is, the completion unit 16 completes the task with the work as at least part of it. For example, the completion unit 16 sets up a work such as "red object" and "T-shaped object," and completes the task by assigning that work to a variable in the document. The completion unit 16 may also output audio from the speakerphone 30 indicating that the task has been completed with the work.

[0042] In step S125, the completion unit 16 retrieves the work stored in the storage unit 17 as a task part corresponding to the shortcut gesture, and completes the task with that work. For example, the completion unit 16 completes the task by assigning the work to a variable in the document. In this case, the task includes a task part. Similar to step S124, the completion unit 16 may output an audio message from the speakerphone 30 indicating that the task has been completed with the work.

[0043] Returning to Figure 3, in step S13, the robot control system 10 completes the task's target position. The task's target position is the position where processing of the workpiece is completed. For example, if the task is pick and place, the target position is the position where the workpiece grasped by the robot 2 is placed. In one example, the query unit 11 queries the user regarding the variable corresponding to the target position. Subsequently, the identification unit 12 identifies the user's action in response to that query. Then, the completion unit 16 substitutes the target position into the variable based on that action, completing the task. The completion of the task by target position will be explained in detail with reference to Figure 6.

[0044] In step S131, the inquiry unit 11 asks the user for the target position. An inquiry for the target position refers to the process of requesting the user to input the target position. In one example, the inquiry unit 11 operates the robot 2 so that it assumes a predetermined inquiry posture. When the robot 2 assumes the inquiry posture, the inquiry unit 11 may output a guidance voice such as "Please tell me the target position" from the speakerphone 30. In one example, the inquiry unit 11 executes the inquiry after the robot 2 assumes the inquiry posture.

[0045] In step S132, the identification unit 12 changes the orientation of the camera 20 based on the guidance sound. In one example, the user emits a guidance sound by clapping their hands or making a sound as a prerequisite action before performing an action indicating the target position. The identification unit 12 acquires the guidance sound via the speakerphone 30 and estimates the direction from which the guidance sound was emitted. The identification unit 12 then controls the robot 2 to orient the camera 20 mounted on the robot 2 in that direction. In one example, the identification unit 12 generates a path for the robot 2 to change the orientation of the camera 20 through planning and outputs a command signal indicating that path to the robot controller 3. The robot controller 3 controls the robot 2 according to the command signal. The robot 2 moves along that path, and as a result, the camera 20 moves so that it faces the direction from which the guidance sound was emitted.

[0046] In step S133, the identification unit 12 identifies the subject from the image captured by the moved camera 20. In one example, the user points to the target location with their hand towards the camera 20. Alternatively, the user presents a pre-configured shortcut gesture to the camera to complement the target location as part of the task. Both the hand shape indicating the target location and the shortcut gesture are examples of user actions in response to the query. In one example, the identification unit 12 pre-stores recognition targets configured to identify the subject from the image. The identification unit 12 extracts the user's hand from the image as its recognition target and identifies the subject based on that recognition target. If the hand shape corresponds to a shortcut gesture, the identification unit 12 identifies that shortcut gesture as the user action. If the hand shape does not correspond to a shortcut gesture, the hand shape is identified as the user action.

[0047] As shown in step S134, the identification unit 12 identifies either a shortcut gesture or a hand shape other than a shortcut gesture as the subject.

[0048] If a hand shape other than a shortcut gesture is identified in step S134, the process proceeds to step S135. In step S135, the identification unit 12 estimates the target position based on the hand shape. In one example, the identification unit 12 sets a reference axis for the hand shape and estimates the target position based on this reference axis. For example, the identification unit 12 sets a reference axis along the longitudinal direction of the hand shape and estimates the intersection point of this reference axis with the area shown in the image as the target position. The area that intersects with the reference axis corresponds, for example, to the location where the workpiece is to be placed.

[0049] In step S136, the identification unit 12 determines whether the camera 20 has approached the target position. In one example, the identification unit 12 calculates the distance between the camera 20 and the target position and makes the determination based on this distance. The identification unit 12 may calculate the distance based on a depth image provided by the camera 20. Alternatively, the identification unit 12 may calculate the distance based on an image from the camera 20 and depth information provided separately from the image. If the calculated distance is less than or equal to a given threshold, the identification unit 12 determines that the camera 20 has approached the target position. On the other hand, if the calculated distance is greater than the threshold, the identification unit 12 determines that the camera 20 has not approached the target position.

[0050] If it is determined that the camera 20 is not approaching the target position (NO in step S136), the process proceeds to step S137. In step S137, the identification unit 12 controls the robot 2 to move the camera 20 toward the target position and identifies the subject from the image captured by the moved camera 20. In one example, the identification unit 12 generates a path for the robot 2 to bring the camera 20 closer to the user's hand, which is the object to be recognized, through planning, and outputs a command signal indicating that path to the robot controller 3. The robot controller 3 controls the robot 2 according to that command signal. The robot 2 moves along that path, and as a result, the camera 20 approaches the user's hand and the target position. The identification unit 12 may also operate the robot 2 so that the object to be recognized (the user's hand) is positioned in the center of the camera 20's field of view. The identification unit 12 extracts the object to be recognized from the image captured by the camera 20 as it approaches the object, and identifies the hand shape indicating the target position as the user's action.

[0051] After step S137, the process returns to step S135, and the identification unit 12 estimates the target position for the newly identified hand shape. The process from steps S135 to S137 is repeated until the camera 20 approaches the target position.

[0052] If it is determined that camera 20 is approaching the target position (YES in step S136), the process proceeds to step S138. In step S138, the completion unit 16 completes the task using the target position estimated based on the hand shape. That is, the completion unit 16 completes the task with the target position as at least part of it. For example, the completion unit 16 completes the task by assigning the target position to a variable in the document. The completion unit 16 may also output audio from speakerphone 30 indicating that the task has been completed with the target position.

[0053] If a shortcut gesture is identified in step S134, the process proceeds to step S139. In step S139, the completion unit 16 retrieves the target position stored in the storage unit 17 as a task part corresponding to the shortcut gesture, and completes the task using that target position. For example, the completion unit 16 completes the task by assigning the target position to a variable in the document. In this case, the task includes a task part. Similar to step S138, the completion unit 16 may output audio from the speakerphone 30 indicating that the task has been completed using the target position.

[0054] Returning to Figure 3, in step S14, the robot control unit 14 controls the robot 2 to perform the complementary task. In one example, the robot control unit 14 generates a path for searching for a workpiece in the workspace through path planning and outputs a command signal indicating that path to the robot controller 3. The robot controller 3 controls the robot 2 according to that command signal. The robot 2 moves along the path and searches for the workpiece. If a workpiece is found, the robot control unit 14 generates a path for performing the complementary task on that workpiece through planning and outputs a command signal indicating that new path to the robot controller 3. The robot controller 3 controls the robot 2 according to that command signal. The robot 2 moves along the new path, and as a result, the robot 2 processes the workpiece. For example, the robot 2 performs a pick-and-place operation, moving the workpiece to a target position. If there are multiple workpieces in the workspace, the robot control unit 14 may repeatedly search for workpieces and perform tasks on the found workpieces.

[0055] In one example, the completion unit 16 stores at least a portion of the completed task as a task part in the storage unit 17. For example, the task part may be the completed work or target location, or it may be the entire task.

[0056] As shown in step S15, the robot control system 10 may perform the next complement after processing one or more workpieces by performing the complemented task. For example, the robot control system 10 repeatedly performs the necessary processing according to the new target to be complemented. For example, if a new workpiece is to be complemented, the process returns to step S12. In the repeated step S12, the robot control system 10 queries the user regarding the variable corresponding to the workpiece, identifies the user's action in response to the query, and complements the task by assigning the workpiece to the variable based on that action. The workpiece assigned to the variable may be the same as or different from the workpiece used in the previous complement. In the repeated step S13, the robot control system 10 queries the user regarding the variable corresponding to the target position, identifies the user's action in response to the query, and complements the task by assigning the target position to the variable based on that action. The target position assigned to the variable may be the same as or different from the target position used in the previous complement. In the repeated step S14, the robot control unit 14 controls the robot 2 to perform the newly complemented task. As another example, if a new target position is to be added, the process returns to step S13, and the robot control system 10 executes steps S13 and S14.

[0057] The entire processing flow S1 may be executed repeatedly. That is, the robot control system 10 may define the next task based on another document (step S11), complement the next task (steps S12, S13), and control the robot 2 to execute the complemented next task (step S14).

[0058] In processing flow S1, the identification unit 12 may execute an additional query to the user asking whether or not to confirm the identified action, and may confirm the action based on the user's response to the additional query. The additional query can be said to be a process that gives the user an opportunity to reconsider the action to input to the robot control system 10. In one example, the identification unit 12 extracts a first hand shape from the first image of the camera 20 and identifies that first shape as an action. For example, the first shape is a hand shape pointing to a target position. The identification unit 12 executes an additional query to the user asking whether or not to confirm that action. For example, the identification unit 12 outputs an audio for the additional query from the speakerphone 30. The identification unit 12 extracts a second hand shape for the additional query from a second image taken by the camera 20 after the first image. Then, the identification unit 12 confirms the identified action in response to the second shape. The completion unit 16 completes the task based on the confirmed action. For example, the completion unit 16 completes the task by adding the target position as at least part of the task based on that action.

[0059] In relation to step S132, the identification unit 12 may change the orientation of the camera 20 based on a preceding action detected by the camera 20 rather than the speakerphone 30. For example, the user points towards the camera 20 with their hand or finger in the direction of a target position. Based on this gesture, the identification unit 12 controls the robot 2 to orient the camera 20 in the indicated direction.

[0060] Step S132 may be executed again at any point in step S13. For example, in response to the user emitting a guidance sound to re-specify the target position in step S133 or later, the process may return to step S132 in step S13. In this case, the identification unit 12 changes the orientation of the camera 20 again based on the guidance sound. Then, the process from step S133 onwards is executed again.

[0061] An example related to the processing flow S1 will be explained with reference to Figures 7 to 12. All of Figures 7 to 12 show scenes in the workspace where user 300 teaches robot 2. Figure 7 shows an example of a document. Figure 8 shows a scene related to document identification. Figure 9 shows a scene related to workpiece completion. All of Figures 10 to 12 show scenes related to target position completion.

[0062] In the example in Figure 7, document 200 is represented by two-dimensional codes placed on the board. For example, the user creates document 200 by arranging magnetic sheets with two-dimensional codes written on them on the board. The four two-dimensional codes located at the four corners of the board are markers for positioning the camera 20 to capture the entire board. In this example, document 200 consists of two-dimensional codes 210, 220, 230, and 240 that indicate syntax, and two-dimensional codes 211 and 212 that indicate variables. Two-dimensional code 210 indicates the syntax "Place <object A> at <location P>". Related to this two-dimensional code 210 are two-dimensional codes 211 that indicate the variable "object A" and two-dimensional code 212 that indicates the variable "location P". Two-dimensional code 220 indicates the syntax "the object is in front". Two-dimensional code 230 indicates the syntax "end if the object is not found". Two-dimensional code 240 indicates the syntax "work quickly".

[0063] In the scenario shown in Figure 8, the inquiry unit 11 queries for a document (step S111). The user 300 presents the document 200 on the board to the camera 20 in response to the query. The identification unit 12 identifies the document 200 from the image from the camera 20 (step S112). The definition unit 15 defines a task based on the document 200. The robot control unit 14 starts executing the task (step S114). For example, as the start of the task, the robot control unit 14 causes the robot 2 to assume a query posture to query the user for a reference object.

[0064] In the scenario shown in Figure 9, the query unit 11 queries for a reference object (step S121). The user 300 presents the reference object 400 to the camera 20 in response to the query. The identification unit 12 identifies the feature quantities of the reference object 400 from the image from the camera 20 (step S122). The completion unit 16 sets the work based on those feature quantities and completes the work as at least part of the task (step S124).

[0065] In the scene shown in Figure 10, user 300 claps their hands to change the orientation of camera 20. Inquiry unit 11 inquires about the target position (step S131). In response to the inquiry, user 300 claps their hands near the inclined surface 500 where the workpiece should be placed. Identification unit 12 estimates the direction of the sound obtained from speakerphone 30 and controls robot 2 to orient camera 20 in that direction (step S132). As shown in Figure 11, after the orientation of camera 20 has changed, user 300 points with their hand to the inclined surface 500 as the target position. Identification unit 12 identifies the hand shape from the image of camera 20 (step S133) and estimates the target position based on the hand shape (step S135). As shown in Figure 12, identification unit 12 determines that camera 20 is not approaching the target position and moves camera 20 toward that target position (steps S136, S137). In the example shown in Figure 12, the identification unit 12 controls the robot 2 to move the camera 20 toward the target position. Then, when it is determined that the camera 20 is approaching the target position, the completion unit 16 completes the task by adding the target position as at least part of the task (step S138).

[0066] Subsequently, the robot control unit 14 controls the robot 2 to perform the complementary task (step S14). The robot 2 searches for the workpiece in the workspace according to the command signals from the robot control unit 14, grasps the workpiece it finds, and places it on the inclined surface 500, which is the target position. The workpiece is then placed in a box located below the inclined surface 500.

[0067] [Differentiation] The above has been a detailed description based on embodiments of the present disclosure. However, the technical features of the present disclosure are not limited to the embodiments described above. The technical features of the present disclosure can be modified in various ways without departing from the spirit thereof.

[0068] The task definition may be a process that sets up the entire task. In this case, task completion by the completion unit is unnecessary. The robot control unit controls the robot so that it executes the defined task. The definition unit may also retrieve the entire task from the storage unit as a task part corresponding to the hand shape when the user presents a predetermined hand shape to the camera instead of a document, and redefine the task as is. In this case, the robot control unit controls the robot so that it executes the redefined task.

[0069] The system's hardware configuration is not limited to a configuration in which each functional module is realized by program execution. For example, at least a portion of the above-mentioned group of functional modules may be composed of logic circuits specialized for that function, or they may be composed of an ASIC (Application Specific Integrated Circuit) that integrates such logic circuits.

[0070] The processing steps for a method executed by at least one processor are not limited to the examples above. For example, some of the steps or processes described above may be omitted, or each step may be performed in a different order. Also, any two or more of the steps described above may be combined, or some of the steps may be modified or deleted. Alternatively, other steps may be performed in addition to each of the steps described above.

[0071] When comparing the relative magnitudes of two numbers in a computer system or within a computer, either the two criteria "greater than or equal to" and "greater than" may be used, or either the two criteria "less than or equal to" and "less than" may be used.

[0072] [Note] As can be seen from the various examples above, this disclosure includes the following aspects: (Note 1) A query unit that executes user inquiries while the robot is performing a task, An identification unit that identifies the user's action in response to the aforementioned inquiry using a sensor, A complementation unit that complements at least a portion of the task based on the identified action, A robot control unit that controls the robot to perform the complementary tasks, A robot control system equipped with the following features. (Note 2) The aforementioned sensor is a camera, The specified part is, The recognition target, which has been pre-configured for the aforementioned action, is extracted from the image captured by the camera. Based on the extracted recognition target, the Action is identified. The robot control system described in Appendix 1. (Note 3) The specified part is, The camera moves in accordance with a preceding action taken by the user before the aforementioned action. The object to be recognized is extracted from the image captured by the moving camera. The robot control system described in Appendix 2. (Note 4) The aforementioned camera is mounted on the robot, The preceding action includes a sound emitted by the user, The specified unit directs the camera in the direction from which the sound was emitted. The robot control system described in Appendix 3. (Note 5) The aforementioned camera is mounted on the robot, The specified part is, The robot is operated to bring the camera closer to the object to be recognized. The recognition target is extracted from an image captured by a camera that approaches the recognition target. A robot control system as described in Appendix 3 or 4. (Note 6) The specified unit operates the robot so that the object to be recognized is positioned in the center of the camera's field of view. The robot control system described in Appendix 5. (Note 7) The object of recognition is the user's hand. The identifying unit identifies the shape of the hand extracted from the image as the action. A robot control system described in any one of the appendices 2 to 6. (Note 8) The complementation unit complements the target position of the task performed by the robot as at least part of the task, based on the shape of the hand. The robot control system described in Appendix 7. (Note 9) The specified part is, The first shape of the hand is extracted from the first image, and the first shape is identified as the action. An additional query is made to the user regarding whether or not to confirm the identified action. The second shape of the hand in response to the additional query is extracted from the second image to determine the identified action. The complementation unit complements at least a portion of the task based on the determined action. A robot control system as described in Appendix 7 or 8. (Note 10) The system further comprises a storage unit that stores at least a portion of the aforementioned completed tasks as task parts, The complementation unit retrieves the task part from the storage unit when the identified action corresponds to a pre-set hand shape. The robot control unit controls the robot so that it performs a task including the task parts. A robot control system described in any one of the appendices 7 to 9. (Note 11) The specified part is, The reference object presented by the user is extracted from the image. The feature quantities of the aforementioned reference object are identified as the actions, The interpolation unit, based on the feature quantities, interpolates the workpiece to be processed by the robot as at least part of the task. A robot control system described in any one of the appendices 2 to 10. (Note 12) The system further comprises a definition unit for defining the aforementioned task, The inquiry unit inquires the user for a document indicating the task, The definition unit defines the task based on the document presented by the user, The complementation unit complements at least a portion of the defined task based on the identified action. A robot control system described in any one of the appendices 1 to 11. (Note 13) The aforementioned document includes the syntax and variables relating to the task, The definition unit defines the task based on the syntax, The query unit executes the query to the user regarding the variable, The identifying unit identifies the user's action in response to the inquiry, The completion unit completes the values ​​to be substituted into the variables based on the identified action. The robot control system described in Appendix 12. (Note 14) The aforementioned inquiry unit is, The robot is operated so that it assumes an inquiry posture, which is the posture that responds to the aforementioned inquiry. After the robot assumes the query posture, it executes the query. A robot control system described in any one of the appendices 1 to 13. (Note 15) A query unit that asks the user for documentation indicating the tasks to be performed by the robot, An identification unit that identifies the document presented by the user from an image captured by a camera, A definition unit that defines the task based on the identified document, A robot control unit that controls the robot to perform the defined task, A robot control system equipped with the following features. (Note 16) A robot control method performed by a robot control system comprising at least one processor, The robot performs a task, and the steps include: making a query to the user while the robot is executing the task; The steps include: identifying the user's action in response to the aforementioned inquiry using a sensor; A step of supplementing at least part of the task based on the identified action, The steps include controlling the robot so that it performs the complementary task, A robot control method including the following. (Note 17) The robot performs a task, and the steps include: making a query to the user while the robot is executing the task; The steps include: identifying the user's action in response to the aforementioned inquiry using a sensor; A step of supplementing at least part of the task based on the identified action, The steps include controlling the robot so that it performs the complementary task, A robot control program that instructs a computer to execute.

[0073] According to appendices 1, 16, and 17, at least part of the tasks performed by the robot is supplemented based on user actions identified using sensors, thus reducing the user's teaching burden. Since user actions are identified using sensors, users can quickly become familiar with teaching the robot without having to learn how to operate input / output devices. Furthermore, since input / output devices are not required, the system configuration can be simplified, and it is expected that system development costs will be reduced.

[0074] According to Appendix 2, since user actions are identified using the camera, users can perform teaching through the natural action of performing an action towards the camera. Furthermore, since the recognition targets necessary to identify actions are pre-set, these recognition targets can be extracted more reliably from the image. As a result, actions can be identified more reliably.

[0075] According to Appendix 3, the camera used to photograph the object to be recognized moves automatically in response to the user's actions, further reducing the user's teaching burden.

[0076] According to Appendix 4, the camera turns in the direction of the sound emitted by the user, allowing the user to easily point the camera to the desired location, thereby further reducing the user's teaching burden.

[0077] According to Appendix 5, since the camera moves closer to the object to be recognized, the object to be recognized necessary to identify the action can be extracted more reliably. In addition, by operating the robot to bring the camera closer to the object to be recognized, the user can be shown that the robot is trying to identify the user's action. As a result, the user can be visually informed that interactive teaching is taking place.

[0078] According to Appendix 6, because the camera reliably captures the object being recognized, user actions can be identified more accurately.

[0079] According to Appendix 7, since the user's hands are processed as the object of recognition, the user can easily teach using their own hands.

[0080] According to Appendix 8, the target location of the task can be easily instructed to the robot.

[0081] According to Appendix 9, a query is executed to confirm the action of the identified user before adopting that action. Therefore, it is possible to give the user an opportunity to reconsider the action and reduce the probability of incorrect teaching.

[0082] According to Appendix 10, at least some of the tasks previously used are retrieved from the memory unit and used. Therefore, the user can easily relay past instructions to the robot again.

[0083] According to Appendix 11, the task is defined by complementing at least a part of the task based on the features of the reference object. Therefore, it is easy to teach the work to be processed by the robot. Since the features of the reference object are used, the robot can be made to process various workpieces as long as the workpiece has those features. In other words, the user only needs to give the robot abstract instructions about the workpiece, thus reducing the teaching burden. When the feature is color, the way the color appears in the workspace is identified as the feature. Therefore, it is possible to absorb differences in how colors appear due to light properties such as light intensity and reliably identify the workpiece in each workspace.

[0084] According to Appendix 12, since tasks are defined simply by the user presenting a document, the teaching burden on the user can be reduced.

[0085] According to Appendix 13, tasks are first defined based on syntax using documentation that shows tasks through syntax and variables. Subsequently, the tasks are completed by setting values ​​in variables based on user actions. By setting tasks in this step-by-step manner, it becomes possible to easily teach even complex tasks to the robot.

[0086] According to Appendix 14, the robot assumes a specific posture before making an inquiry to the user, thus clearly communicating to the user the timing for taking action.

[0087] According to Appendix 15, since tasks are defined simply by the user presenting a document, the user's teaching burden can be reduced. Because the document is identified using a camera, the user can easily teach the robot without having to learn how to operate input / output devices. In addition, since input / output devices are not required, the system configuration can be simplified, and it is expected that system development costs will be reduced. [Explanation of Symbols]

[0088] 1...Robot system, 2...Robot, 3...Robot controller, 10...Robot control system, 20...Camera, 30...Speakerphone, 11...Inquiry unit, 12...Identification unit, 13...Task management unit, 14...Robot control unit, 15...Definition unit, 16...Complementary unit, 17...Storage unit, 200...Document, 400...Reference object.

Claims

1. A query unit that executes user inquiries while the robot is performing a task, A unit that extracts the user's hand, which is a pre-set recognition target in relation to the user's action in response to the aforementioned inquiry, from an image captured by a camera, and identifies the shape of the extracted hand as the action, A complementation unit that complements at least a portion of the task based on the identified action, A robot control unit that controls the robot to perform the complementary tasks, Equipped with, The specified part is, The first shape of the hand is extracted from the first image, and the first shape is identified as the action. An additional query is made to the user regarding whether or not to confirm the identified action. The second shape of the hand in response to the additional query is extracted from the second image to determine the identified action. The complementation unit complements at least a portion of the task based on the determined action. Robot control system.

2. A query unit that makes inquiries to the user while the robot is performing a task, A unit that extracts the user's hand, which is a pre-set recognition target in relation to the user's action in response to the aforementioned inquiry, from an image captured by a camera, and identifies the shape of the extracted hand as the action, A complementation unit that complements at least a portion of the task based on the identified action, A robot control unit that controls the robot to perform the complementary tasks, A storage unit that stores at least a portion of the aforementioned completed tasks as task parts, Equipped with, The complementation unit retrieves the task part from the storage unit when the identified action corresponds to a pre-set hand shape. The robot control unit controls the robot so that it performs a task including the task parts. Robot control system.

3. The specified part is, The camera moves in accordance with a preceding action taken by the user before the aforementioned action. The object to be recognized is extracted from the image captured by the moving camera. The robot control system according to claim 1 or 2.

4. The aforementioned camera is mounted on the robot, The preceding action includes a sound emitted by the user, The specified unit directs the camera in the direction from which the sound was emitted. The robot control system according to claim 3.

5. The aforementioned camera is mounted on the robot, The specified part is, The robot is operated to bring the camera closer to the object to be recognized. The recognition target is extracted from an image captured by a camera that approaches the recognition target. The robot control system according to claim 3.

6. The specified unit operates the robot so that the object to be recognized is positioned in the center of the camera's field of view. The robot control system according to claim 5.

7. The complementation unit complements the target position of the task performed by the robot as at least part of the task, based on the shape of the hand. The robot control system according to claim 1 or 2.

8. The specified part is, The reference object presented by the user is extracted from the image. The feature quantities of the aforementioned reference object are identified as the actions, The interpolation unit, based on the feature quantities, interpolates the workpiece to be processed by the robot as at least part of the task. The robot control system according to claim 1 or 2.

9. The aforementioned inquiry unit is, The robot is operated so that it assumes an inquiry posture, which is the posture that responds to the aforementioned inquiry. After the robot assumes the query posture, it executes the query. The robot control system according to claim 1 or 2.

10. A definition unit for defining a task, While the robot is performing the aforementioned task, a query unit executes a query to the user, An identification unit that identifies the user's action in response to the aforementioned inquiry using a sensor, A complementation unit that complements at least a portion of the task based on the identified action, A robot control unit that controls the robot to perform the complementary tasks, Equipped with, The query unit queries the user for documentation including syntax and variables related to the task. The definition unit defines the task based on the syntax of the document presented by the user, The query unit executes the query to the user regarding the variable, The identifying unit identifies the user's action in response to the inquiry, The completion unit completes the values ​​to be substituted into the variables based on the identified action. Robot control system.

11. A robot control method performed by a robot control system comprising at least one processor, The robot performs a task, and the steps include: making a query to the user while the robot is executing the task; The steps include extracting the user's hand, which is a pre-configured recognition target in relation to the user's action in response to the aforementioned inquiry, from an image captured by a camera, and identifying the shape of the extracted hand as the action, A step of supplementing at least part of the task based on the identified action, The steps include controlling the robot so that it performs the complementary task, Includes, In the aforementioned step of identification, The first shape of the hand is extracted from the first image, and the first shape is identified as the action. An additional query is made to the user regarding whether or not to confirm the identified action. The second shape of the hand in response to the additional query is extracted from the second image to determine the identified action. In the aforementioned supplementing step, at least a portion of the task is supplemented based on the determined action. Robot control methods.

12. A robot control method performed by a robot control system comprising at least one processor, The robot performs a task, and the steps include: making a query to the user while the robot is executing the task; The steps include extracting the user's hand, which is a pre-configured recognition target in relation to the user's action in response to the aforementioned inquiry, from an image captured by a camera, and identifying the shape of the extracted hand as the action, A step of supplementing at least part of the task based on the identified action, The steps include controlling the robot so that it performs the complementary task, The steps include storing at least a portion of the completed task as a task part in the memory unit, Includes, In the aforementioned complementary step, if the identified action corresponds to a pre-set hand shape, the task part is retrieved from the storage unit. In the control step, the robot is controlled so that it performs the task including the task part. Robot control methods.

13. A robot control method performed by a robot control system comprising at least one processor, The steps to define the task, The robot performs the aforementioned task, and the steps include: The steps include: identifying the user's action in response to the aforementioned inquiry using a sensor; A step of supplementing at least part of the task based on the identified action, The steps include controlling the robot so that it performs the complementary task, Includes, In the step of executing the above query, the user is asked for documentation including syntax and variables related to the task. In the step defined above, the task is defined based on the syntax of the document presented by the user, In the step of executing the aforementioned query, the aforementioned query to the user is executed with respect to the aforementioned variable, In the aforementioned identifying step, the user's action in response to the query is identified, In the aforementioned complementing step, the values ​​to be substituted for the variables are complemented based on the identified action. Robot control methods.

14. The robot performs a task, and the steps include: making a query to the user while the robot is executing the task; The steps include extracting the user's hand, which is a pre-configured recognition target in relation to the user's action in response to the aforementioned inquiry, from an image captured by a camera, and identifying the shape of the extracted hand as the action, A step of supplementing at least part of the task based on the identified action, The steps include controlling the robot so that it performs the complementary task, Have the computer run it, In the aforementioned step of identification, The first shape of the hand is extracted from the first image, and the first shape is identified as the action. An additional query is made to the user regarding whether or not to confirm the identified action. The second shape of the hand in response to the additional query is extracted from the second image to determine the identified action. In the aforementioned supplementing step, at least a portion of the task is supplemented based on the determined action. Robot control program.

15. The step of making a query to the user while the robot is performing a task, The steps include extracting the user's hand, which is a pre-configured recognition target in relation to the user's action in response to the aforementioned inquiry, from an image captured by a camera, and identifying the shape of the extracted hand as the action, A step of supplementing at least part of the task based on the identified action, The steps include controlling the robot so that it performs the complementary task, The steps include storing at least a portion of the completed task as a task part in the memory unit, Have the computer run it, In the aforementioned complementary step, if the identified action corresponds to a pre-set hand shape, the task part is retrieved from the storage unit. In the control step, the robot is controlled so that it performs the task including the task part. Robot control program.

16. A step of defining a task, The robot performs the aforementioned task, and the steps include: The steps include: identifying the user's action in response to the aforementioned inquiry using a sensor; A step of supplementing at least part of the task based on the identified action, The steps include controlling the robot so that it performs the complementary task, Have the computer run it, In the step of executing the above query, the user is asked for documentation including syntax and variables related to the task. In the step defined above, the task is defined based on the syntax of the document presented by the user, In the step of executing the aforementioned query, the aforementioned query to the user is executed with respect to the aforementioned variable, In the aforementioned identifying step, the user's action in response to the query is identified, In the aforementioned complementing step, the values ​​to be substituted for the variables are complemented based on the identified action. Robot control program.

Images