Control device, control method, and program

Abstract

This control device 1X mainly has a recognition result acquisition means 14X, a display control means 15X, and a correction acceptance means 16X. The recognition result acquisition means 14X acquires a result of recognizing an object related to a task executed by a robot. The display control means 15X displays information representing the recognition result so as to be visually confirmed while superposed on real scenery or on a scenery image. The correction acceptance means 16X accepts a correction of the recognition result, the correction being based on external input.

Description

[Technical field]

[0001] The present disclosure relates to the technical fields of a control device, a control method, and a storage medium for a robot that executes a task. [Background technology]

[0002] A robot system has been proposed that recognizes the robot's environment using a sensor and makes the robot execute a task based on the recognized environment. For example, Patent Document 1 discloses a robot system that issues an operation command to the robot based on the detection result of a surrounding environment detection sensor and a determined behavior plan for the robot. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-046779 A Summary of the Invention [Problem to be solved by the invention]

[0004] If there is an error in the robot's recognition of the environment, the robot's determined motion (action) plan may not be executed correctly. On the other hand, it may be difficult in some cases to recognize the robot's environment with perfect accuracy based on the output signals of the sensors.

[0005] In view of the above-mentioned problems, one of the objects of the present disclosure is to provide a control device, a control method, and a storage medium that are capable of obtaining accurate recognition results of objects related to a task. [Means for solving the problem]

[0006] One aspect of the control device is A recognition result acquisition means for acquiring an object recognition result related to a task to be executed by the robot; a display control means for displaying information representing the recognition result so as to be visually recognized by superimposing the information on a real landscape or an image of the landscape; A correction receiving means for receiving a correction of the recognition result based on an external input. death, the recognition result includes a confidence level indicating a degree of confidence regarding the correctness of the recognition of the object; The correction receiving means determines whether or not the correction is necessary based on the degree of confidence. It is a control device. Another aspect of the control device is A recognition result acquisition means for acquiring an object recognition result related to a task to be executed by the robot; a display control means for displaying information representing the recognition result so as to be visually recognized by superimposing the information on a real landscape or an image of the landscape; and a correction receiving means for receiving a correction of the recognition result based on an external input, the recognition result includes a confidence level indicating a degree of confidence regarding the correctness of the recognition of the object; The display control means is a control device that highlights information representing the recognition result for which the degree of confidence is less than a threshold value.

[0007] One aspect of the control method includes: The computer Obtaining object recognition results related to a task to be performed by the robot; displaying information representing the recognition result so as to be visually recognized overlapping with a real landscape or an image of the landscape; Accepts corrections to the recognition results based on external input 、 the recognition result includes a confidence level indicating a degree of confidence regarding the correctness of the recognition of the object; determining whether the correction is necessary based on the degree of confidence; It is a control method.

[0008] One aspect of the program is Obtaining object recognition results related to a task to be performed by the robot; displaying information representing the recognition result so as to be visually recognized overlapping with a real landscape or an image of the landscape; Accepts corrections to the recognition results based on external input 、 the recognition result includes a confidence level indicating a degree of confidence regarding the correctness of the recognition of the object; Based on the degree of confidence, it is determined whether the correction is necessary. It is a program that causes a computer to execute processing. Effect of the Invention

[0009] This makes it possible to preferably obtain accurate recognition results of objects related to the task. [Brief description of the drawings]

[0010] [Figure 1] 1 shows a configuration of a robot control system in a first embodiment. [Diagram 2] (A) shows the hardware configuration of the robot controller, and (B) shows the hardware configuration of the instruction device. [Diagram 3] 13 shows an example of a data structure of application information. [Figure 4] 2 is an example of a functional block of a robot controller. [Diagram 5] (A) shows a first embodiment of the modification; (B) shows a second embodiment of the modification; (C) shows a third embodiment of the modification; (D) shows a fourth embodiment of the modification. [Figure 6] 1 shows the state of the workspace before correction as visually recognized by a worker when a pick-and-place is the target task. [Figure 7] 13 shows the state of the workspace as viewed by the worker after modifications have been made to the virtual object. [Figure 8] 4 is an example of a functional block showing a functional configuration of a motion planning unit. [Figure 9] 1 shows an overhead view of a workspace when a target task is pick-and-place. [Figure 10] 4 is an example of a flowchart showing an overview of a robot control process executed by a robot controller in the first embodiment. [Figure 11] 13 is an example of a functional block of a robot controller in the second embodiment. [Figure 12] FIG. 4 is a diagram showing trajectory information in a first specific example. [Figure 13] 1A is a diagram showing a schematic diagram of a corrected trajectory (corrected trajectory) of a robot hand and an object corrected based on an input for correcting the trajectory of the robot hand and the object, etc. in the first concrete example. FIG. [Figure 14]13A is a diagram showing pre-correction orbit information in the second specific example from a first viewpoint, and FIG. 13B is a diagram showing pre-correction orbit information in the second specific example from a second viewpoint. [Figure 15] 13A is a diagram showing an overview of an operation related to the correction of orbit information in the second specific example from a first perspective, and FIG. 13B is a diagram showing an overview of an operation related to the correction of orbit information in the second specific example from a second perspective. [Figure 16] 13A is a diagram showing the orbit information after correction in the second specific example from a first viewpoint, and FIG. 13B is a diagram showing the orbit information after correction in the second specific example from a second viewpoint. [Figure 17] 13 is an example of a flowchart illustrating an overview of a robot control process executed by a robot controller in a second embodiment. [Figure 18] FIG. 13 shows a schematic configuration diagram of a control device according to a third embodiment. [Figure 19] 13 is an example of a flowchart executed by the control device in the third embodiment. [Figure 20] FIG. 13 shows a schematic configuration diagram of a control device according to a fourth embodiment. [Figure 21] 13 is an example of a flowchart executed by the control device in the fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, embodiments of a control device, a control method, and a storage medium will be described with reference to the drawings.

[0012] First Embodiment (1) System Configuration 1 shows the configuration of a robot control system 100 according to the first embodiment. The robot control system 100 mainly includes a robot controller 1, an instruction device 2, a storage device 4, a robot 5, and a sensor (detection device) 7.

[0013] When a task (also called a "target task") to be executed by the robot 5 is specified, the robot controller 1 converts the target task into a sequence of time steps (time increments) of simple tasks that the robot 5 can accept, and controls the robot 5 based on the generated sequence.

[0014] The robot controller 1 also performs data communication with the instruction device 2, the storage device 4, the robot 5, and the sensor 7 via a communication network or by direct wireless or wired communication. For example, the robot controller 1 receives an input signal "S1" related to a motion plan for the robot 5 from the instruction device 2. The robot controller 1 also transmits a display control signal "S2" to the instruction device 2, causing the instruction device 2 to execute a predetermined display or sound output. The robot controller 1 also transmits a control signal "S3" related to the control of the robot 5 to the robot 5. The robot controller 1 also receives a sensor signal "S4" from the sensor 7.

[0015] The instruction device 2 is a device that receives instructions from a worker regarding a motion plan for the robot 5. The instruction device 2 performs a predetermined display or sound output based on a display control signal S2 supplied from the robot controller 1, and supplies an input signal S1 generated based on an input by the worker to the robot controller 1. The instruction device 2 may be a tablet terminal equipped with an input unit and a display unit, a stationary personal computer, or any terminal used for augmented reality.

[0016] The storage device 4 has an application information storage unit 41. The application information storage unit 41 stores application information required to generate an operation sequence, which is a sequence to be executed by the robot 5, from a target task. Details of the application information will be described later with reference to FIG. 3. The storage device 4 may be an external storage device such as a hard disk connected to or built into the robot controller 1, or may be a storage medium such as a flash memory. The storage device 4 may also be a server device that performs data communication with the robot controller 1 via a communication network. In this case, the storage device 4 may be composed of multiple server devices.

[0017] The robot 5 performs work related to the target task based on a control signal S3 supplied from the robot controller 1. The robot 5 is a robot that operates in, for example, various factories such as an assembly factory or a food factory, or at a logistics site. The robot 5 may be a vertical articulated robot, a horizontal articulated robot, or any other type of robot. The robot 5 may supply a status signal indicating the status of the robot 5 to the robot controller 1. This status signal may be an output signal of a sensor that detects the status (position, angle, etc.) of the entire robot 5 or a specific part such as a joint, or may be a signal generated by the control unit of the robot 5 that indicates the progress status of the operation sequence of the robot 5.

[0018] The sensor 7 is one or more sensors, such as a camera, a range sensor, a sonar, or a combination of these, that detect the state in the workspace in which the target task is executed. For example, the sensor 7 includes at least one camera that captures an image of the workspace of the robot 5. The sensor 7 supplies the generated sensor signal S4 to the robot controller 1. The sensor 7 may be a self-propelled or flying sensor (including a drone) that moves in the workspace. The sensor 7 may also include a sensor provided on the robot 5 and a sensor provided on another object in the workspace. The sensor 7 may also include a sensor that detects sound in the workspace. In this way, the sensor 7 may include various sensors that detect the state in the workspace and are provided at any location.

[0019] The configuration of the robot control system 100 shown in FIG. 1 is an example, and various modifications may be made to the configuration. For example, there may be a plurality of robots 5, and there may be a plurality of control objects, such as robot arms, each of which operates independently. Even in these cases, the robot controller 1 generates an operation sequence to be executed for each robot 5 or each control object based on the target task, and transmits a control signal S3 based on the operation sequence to the target robot 5. The robot 5 may also perform a collaborative work with another robot, a worker, or a machine tool operating in the working space. The sensor 7 may also be a part of the robot 5. The instruction device 2 may also be configured as the same device as the robot controller 1. The robot controller 1 may also be configured from a plurality of devices. In this case, the plurality of devices constituting the robot controller 1 transmit and receive information required to execute a process assigned in advance between these plurality of devices. The robot controller 1 and the robot 5 may also be configured as one unit.

[0020] (2) Hardware Configuration 2(A) shows the hardware configuration of the robot controller 1. The robot controller 1 includes, as hardware, a processor 11, a memory 12, and an interface 13. The processor 11, the memory 12, and the interface 13 are connected via a data bus 10.

[0021] The processor 11 executes a program stored in the memory 12 to function as a controller (arithmetic device) that performs overall control of the robot controller 1. The processor 11 is, for example, a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a TPU (Tensor Processing Unit). The processor 11 may be composed of multiple processors. The processor 11 is an example of a computer.

[0022] The memory 12 is composed of various types of volatile and non-volatile memories such as a RAM (Random Access Memory), a ROM (Read Only Memory), and a flash memory. The memory 12 also stores programs for executing the processes executed by the robot controller 1. Note that part of the information stored in the memory 12 may be stored in one or more external storage devices (e.g., the storage device 4) capable of communicating with the robot controller 1, or may be stored in a storage medium that is detachable from the robot controller 1.

[0023] The interface 13 is an interface for electrically connecting the robot controller 1 to other devices. These interfaces may be wireless interfaces such as a network adapter for wirelessly transmitting and receiving data to and from other devices, or may be hardware interfaces for connecting to other devices via cables or the like.

[0024] The hardware configuration of the robot controller 1 is not limited to the configuration shown in Fig. 2(A). For example, the robot controller 1 may be connected to or built-in at least one of a display device, an input device, and a sound output device. The robot controller 1 may also be configured to include at least one of an instruction device 2 and a storage device 4.

[0025] 2(B) shows a hardware configuration of the instruction device 2. The instruction device 2 includes, as hardware, a processor 21, a memory 22, an interface 23, an input unit 24a, a display unit 24b, and a sound output unit 24c. The processor 21, the memory 22, and the interface 23 are connected via a data bus 20. The interface 23 is also connected to the input unit 24a, the display unit 24b, and the sound output unit 24c.

[0026] The processor 21 executes a predetermined process by executing a program stored in the memory 22. The processor 21 is a processor such as a CPU or a GPU. The processor 21 generates an input signal S1 by receiving a signal generated by the input unit 24a via the interface 23, and transmits the input signal S1 to the robot controller 1 via the interface 23. The processor 21 also controls at least one of the display unit 24b and the sound output unit 24c via the interface 23 based on a display control signal S2 received from the robot controller 1 via the interface 23.

[0027] The memory 22 is configured with various types of volatile and non-volatile memories such as RAM, ROM, flash memory, etc. The memory 22 also stores programs for executing the processes executed by the instruction device 2.

[0028] The interface 23 is an interface for electrically connecting the instruction device 2 to other devices. These interfaces may be wireless interfaces such as a network adapter for wirelessly transmitting and receiving data to and from other devices, or may be hardware interfaces for connecting to other devices via cables or the like. The interface 23 also performs interface operations for the input unit 24a, the display unit 24b, and the sound output unit 24c.

[0029] The input unit 24a is an interface that accepts user input, and may be, for example, a touch panel, a button, a keyboard, or a voice input device. The input unit 24a may also include various input devices (such as an operation controller) used in virtual reality. In this case, the input unit 24a may be, for example, various sensors (including, for example, a camera, a wearable sensor, etc.) used in motion capture, or, if the display unit 24b is a glasses-type terminal that realizes augmented reality, may be an operation controller that is paired with the terminal.

[0030] The display unit 24b performs augmented reality display based on the control of the processor 21. In a first example, the display unit 24b is a glasses-type terminal that displays information about the state of objects in a scene (here, the work space) viewed by the worker, superimposed on the scene. In a second example, the display unit 24b is a display, projector, or the like that displays information about objects superimposed on an image (also called a "real-life image") of a scene (here, the work space). The real-life image is supplied by the sensor 7. The sound output unit 24c is, for example, a speaker, and performs sound output based on the control of the processor 21.

[0031] The hardware configuration of the instruction device 2 is not limited to the configuration shown in Fig. 2(B). For example, at least one of the input unit 24a, the display unit 24b, and the sound output unit 24c may be configured as a separate device electrically connected to the instruction device 2. The instruction device 2 may be connected to various devices such as a camera, or may have these built-in.

[0032] (3) Application Information Next, the data structure of the application information stored in the application information storage unit 41 will be described.

[0033] 3 shows an example of the data structure of the application information. As shown in FIG. 3, the application information includes abstract state designation information I1, constraint condition information I2, operation limit information I3, subtask information I4, abstract model information I5, and object model information I6.

[0034] The abstract state designation information I1 is information that designates an abstract state that needs to be defined when generating an action sequence. This abstract state is an abstract state of an object in the working space, and is defined as a proposition to be used in a goal logical formula, which will be described later. For example, the abstract state designation information I1 designates an abstract state that needs to be defined for each type of target task.

[0035] The constraint information I2 is information indicating constraint conditions when executing a target task. For example, when the target task is pick-and-place, the constraint information I2 indicates a constraint condition that the robot 5 (robot arm) must not come into contact with an obstacle, a constraint condition that the robots 5 (robot arms) must not come into contact with each other, etc. Note that the constraint information I2 may be information recording constraint conditions appropriate for each type of target task.

[0036] The motion limit information I3 indicates information related to the motion limit of the robot 5 controlled by the robot controller 1. The motion limit information I3 is, for example, information that specifies the upper limit of the speed, acceleration, or angular velocity of the robot 5. Note that the motion limit information I3 may be information that specifies the motion limit for each movable part or joint of the robot 5.

[0037] The subtask information I4 indicates information on subtasks that are components of an operation sequence. A "subtask" is a task obtained by breaking down a target task into units that the robot 5 can accept, and indicates the subdivided operation of the robot 5. For example, when the target task is pick-and-place, the subtask information I4 specifies the reaching, which is the movement of the robot arm of the robot 5, and the grasping, which is the gripping by the robot arm, as subtasks. The subtask information I4 may indicate information on subtasks that can be used for each type of target task. Note that the subtask information I4 may include information on subtasks that require an operation command from an external input. In this case, the subtask information I4 on the external input type subtask includes, for example, information that identifies it as an external input type subtask (for example, flag information) and information that indicates the operation content of the robot 5 in the external input type subtask.

[0038] The abstract model information I5 is information on an abstract model that abstracts dynamics in the working space. For example, the abstract model is represented by a model that abstracts real-world dynamics using a hybrid system, as described below. The abstract model information I5 includes information indicating the conditions for switching dynamics in the above-mentioned hybrid system. For example, in the case of pick-and-place, in which the robot 5 grasps an object (also called an "object") to be worked on and moves it to a specified position, the switching condition corresponds to a condition that the object cannot be moved unless it is grasped by the robot 5. The abstract model information I5 has information on an abstract model suitable for each type of target task.

[0039] The object model information I6 is information on the object model of each object in the working space to be recognized from the sensor signal S4 generated by the sensor 7. The above-mentioned objects include, for example, the robot 5, obstacles, tools and other objects handled by the robot 5, and working bodies other than the robot 5. The object model information I6 includes, for example, information required for the robot controller 1 to recognize the type, position, posture, and currently executed operation of each of the above-mentioned objects, and three-dimensional shape information such as CAD (Computer Aided Design) data for recognizing the three-dimensional shape of each object. The former information includes parameters of an inference device obtained by learning a learning model in machine learning such as a neural network. For example, when an image is input, this inference device is trained in advance to output the type, position, posture, and the like of an object that is a subject in the image.

[0040] In addition to the above-mentioned information, the application information storage unit 41 may store various information related to the generation process of the operation sequence and the generation process of the display control signal S2.

[0041] (4) Processing Overview Next, an overview of the processing of the robot controller 1 in the first embodiment will be described. In summary, the robot controller 1 displays the recognition result of an object in the working space recognized based on the sensor signal S4 on the instruction device 2 by augmented reality, and accepts an input for correction of the recognition result. As a result, even if an erroneous recognition of an object in the working space occurs, the robot controller 1 appropriately corrects the part where the erroneous recognition occurred based on the user input, and realizes accurate formulation of an operation plan for the robot 5 and execution of the target task.

[0042] Fig. 4 is an example of a functional block showing an overview of the processing of the robot controller 1. The processor 11 of the robot controller 1 functionally has a recognition result acquisition unit 14, a display control unit 15, a correction acceptance unit 16, an action planning unit 17, and a robot control unit 18. Note that Fig. 4 shows an example of data exchanged between each block, but is not limited to this. The same applies to other functional block diagrams described later.

[0043] The recognition result acquisition unit 14 recognizes the state and attributes of objects in the work space based on the sensor signal S4 and the like, and supplies information representing the recognition result (also referred to as the "first recognition result Im1") to the display control unit 15. In this case, for example, the recognition result acquisition unit 14 refers to the abstract state designation information I1 and recognizes the state and attributes of objects in the work space that need to be considered when executing a target task. The objects in the work space are, for example, the robot 5, objects such as tools or parts handled by the robot 5, obstacles, and other work bodies (people or other objects performing work other than the robot 5). For example, the recognition result acquisition unit 14 refers to the object model information I6 and generates the first recognition result Im1 by analyzing the sensor signal S4 using any technology that recognizes the environment of the work space. Examples of the technology that recognizes the environment include image processing technology, image recognition technology (including object recognition using an AR marker), voice recognition technology, and technology using RFID (Radio Frequency Identifier).

[0044] In this embodiment, the recognition result acquisition unit 14 recognizes at least the position, posture, and attributes of an object. The attributes are, for example, the type of object, and the type of object recognized by the recognition result acquisition unit 14 is classified according to the granularity according to the type of target task to be executed. For example, when the target task is pick-and-place, the object is classified into "obstacle", "object to be grasped", and the like. The recognition result acquisition unit 14 supplies the generated first recognition result Im1 to the display control unit 15. Note that the first recognition result Im1 is not limited to information representing the position, posture, and type of the object, and may include information regarding various states or attributes (for example, the size, shape, etc. of the object) recognized by the recognition result acquisition unit 14.

[0045] Furthermore, when the recognition result acquisition unit 14 receives recognition correction information "Ia" indicating the correction content of the first recognition result Im1 from the correction acceptance unit 16, the recognition result acquisition unit 14 generates information reflecting the recognition correction information Ia on the first recognition result Im1 (also referred to as "second recognition result Im2"). Here, the recognition correction information Ia is, for example, information indicating whether or not correction is necessary, and if correction is necessary, an object to be corrected, an index to be corrected, and an amount of correction. Note that the "object to be corrected" is an object that requires correction of the recognition result, and the "index to be corrected" corresponds to an index related to a position (for example, a coordinate value for each coordinate axis), an index related to a posture (for example, Euler angles), an index representing an attribute, and the like. Then, the recognition result acquisition unit 14 supplies the second recognition result Im2 reflecting the recognition correction information Ia to the operation planning unit 17. Note that, when the recognition correction information Ia indicates that no correction is required, the second recognition result Im2 is the same as the first recognition result Im1 that the recognition result acquisition unit 14 initially generated based on the sensor signal S4.

[0046] The display control unit 15 generates a display control signal S2 for displaying or outputting a sound on the instruction device 2 used by the worker, and transmits the display control signal S2 to the instruction device 2 via the interface 13. In this embodiment, the display control unit 15 generates an object (also called a "virtual object") that virtually represents each object based on the recognition result of the object in the working space indicated by the first recognition result Im1. Then, the display control unit 15 generates a display control signal S2 that controls the display of the instruction device 2 so that each virtual object is visually recognized by the worker by being superimposed on a corresponding object in a real landscape or a real-life image. The display control unit 15 generates this virtual object based on, for example, the type of object indicated by the first recognition result Im1 and the three-dimensional shape information for each type of object included in the object model information I6. In another example, the display control unit 15 generates a virtual object by combining a primitive shape (a polygon registered in advance) according to the shape of the object indicated by the first recognition result Im1.

[0047] The correction receiving unit 16 receives a correction to the first recognition result Im1 by an operation of a worker using the instruction device 2. Then, when the operation related to the correction is completed, the correction receiving unit 16 generates recognition correction information Ia indicating the correction content of the first recognition result Im1. In this case, the correction receiving unit 16 receives an input signal S1 generated by the instruction device 2 via the interface 13 during display control of the recognition result of the object by the augmented reality, and supplies the recognition correction information Ia generated based on the input signal S1 to the recognition result acquisition unit 14. Furthermore, before the correction is confirmed, the correction receiving unit 16 supplies an instruction signal such as a correction of the display position of the virtual object based on the input signal S1 supplied from the instruction device 2 to the display control unit 15, and the display control unit 15 supplies a display control signal S2 reflecting the correction based on the instruction signal to the instruction device 2. As a result, the instruction device 2 displays a virtual object that immediately reflects the operation by the worker.

[0048] The motion planning unit 17 determines a motion plan for the robot 5 based on the second recognition result Im2 supplied from the recognition result acquisition unit 14 and the application information stored in the storage device 4. In this case, the motion planning unit 17 generates a motion sequence "Sr" which is a sequence of subtasks (subtask sequence) to be executed by the robot 5 to achieve the target task. The motion sequence Sr specifies a series of motions of the robot 5, and includes information indicating the execution order and execution timing of each subtask. The motion planning unit 17 supplies the generated motion sequence Sr to the robot control unit 18.

[0049] The robot control unit 18 controls the operation of the robot 5 by supplying a control signal S3 to the robot 5 via the interface 13. The robot control unit 18 controls the robot 5 to execute each subtask constituting the operation sequence Sr at a respective determined execution timing (time step) based on the operation sequence Sr supplied from the operation planning unit 17. Specifically, the robot control unit 18 transmits the control signal S3 to the robot 5 to execute position control or torque control of the joints of the robot 5 to realize the operation sequence Sr.

[0050] The function corresponding to the robot control unit 18 may be included in the robot 5 instead of the robot controller 1. In this case, the robot 5 operates based on the operation sequence Sr generated by the operation plan unit 17.

[0051] Here, each of the components of the recognition result acquisition unit 14, the display control unit 15, the correction acceptance unit 16, the operation planning unit 17, and the robot control unit 18 can be realized, for example, by the processor 11 executing a program. Also, each component may be realized by recording a necessary program in an arbitrary non-volatile storage medium and installing it as necessary. Note that at least a part of each of these components may be realized by any combination of hardware, firmware, and software, without being limited to being realized by software by a program. Also, at least a part of each of these components may be realized by using an integrated circuit programmable by a user, such as an FPGA (Field-Programmable Gate Array) or a microcontroller. In this case, a program composed of each of the above components may be realized using this integrated circuit. Also, at least a part of each of the components may be configured by an ASSP (Application Specific Standard Produce), an ASIC (Application Specific Integrated Circuit), or a quantum computer control chip. In this way, each of the components may be realized by various hardware. The above is the same in other embodiments described later. Furthermore, each of these components may be realized by the cooperation of multiple computers, for example, using cloud computing technology.

[0052] (5) Generate correction information Next, a specific description will be given of a method for generating the recognition correction information Ia based on the control of the display control unit 15 and the correction receiving unit 16. The display control unit 15 causes the instruction device 2 to display a virtual object of the recognized object so as to be superimposed on an actual object (real object) in a landscape or live-action image visually recognized by the worker according to the position and posture of the object. Then, the correction receiving unit 16 receives an operation to correct the position and posture of the virtual object so that the real object and the virtual object match when there is a difference between the real object and the virtual object in the landscape or live-action image.

[0053] First, the modification modes of the virtual objects of each object will be described. Fig. 5(A) to Fig. 5(D) respectively show the modes (first mode to fourth mode) of modifications received by the modification receiving unit 16. In Fig. 5(A) to Fig. 5(D), the left side of the arrows shows how the real object and virtual object look before modification, and the right side of the arrows shows how the real object and virtual object look after modification. Here, the cylindrical real object is shown by a solid line, and the virtual object is shown by a dashed line.

[0054] In the first mode shown in FIG. 5(A), in the state before correction, the position and orientation of the real object and the virtual object are visually shifted. Therefore, in this case, the operator performs an operation to correct the position and orientation (roll, pitch, yaw) on the input unit 24a of the instruction device 2 so that the virtual object overlaps with the real object. In the state after correction, the position and orientation of the virtual object are appropriately changed based on the input signal S1 generated by the above-mentioned operation to correct the position and orientation. Then, the correction receiving unit 16 generates recognition correction information Ia that instructs correction of the position and orientation of the object corresponding to the target virtual object, and supplies the recognition correction information Ia to the recognition result acquisition unit 14. As a result, the correction of the position and orientation of the virtual object is reflected in the second recognition result Im2 as a correction of the recognition result of the position and orientation of the corresponding object.

[0055] In the second mode shown in FIG. 5(B), in the state before correction, the recognition result acquisition unit 14 was unable to recognize the presence of the target object based on the sensor signal S4, and therefore a virtual object for the target object is not displayed. Therefore, in this case, the operator performs an operation on the input unit 24a of the instruction device 2 to instruct the generation of a virtual object for the target object. In this case, the operator may perform an operation to directly specify attributes such as the position, orientation, and type of the object for which a virtual object is to be generated, or may perform an operation to instruct re-execution of the object recognition process centered on a portion of the real object that has not been recognized by specifying the portion. Then, in the state after correction, a virtual object for the target object is appropriately generated with a position and orientation that match the real object based on the input signal S1 generated by the above-mentioned operation. Then, the correction acceptance unit 16 generates recognition correction information Ia indicating the addition of the recognition result of the object corresponding to the generated virtual object, and supplies the recognition correction information Ia to the recognition result acquisition unit 14. As a result, the addition of the virtual object is reflected in the second recognition result Im2 as the addition of the recognition result of the corresponding object.

[0056] In the third mode shown in FIG. 5(C), a virtual object for an object that does not actually exist is generated in the pre-correction state due to erroneous object recognition in the recognition result acquisition unit 14. In this case, the operator performs an operation on the input unit 24Aa of the instruction device 2 to instruct the deletion of the target virtual object. Then, in the post-correction state, the virtual object generated due to the erroneous object recognition is appropriately deleted based on the input signal S1 generated by the above-mentioned operation. Then, the correction receiving unit 16 generates recognition correction information Ia instructing the deletion of the recognition result of the object corresponding to the target virtual object, and supplies the recognition correction information Ia to the recognition result acquisition unit 14. As a result, the deletion of the virtual object is reflected in the second recognition result Im2 as the deletion of the recognition result of the corresponding object.

[0057] In the fourth mode shown in FIG. 5(D), due to an error in the object attribute (here, type) recognition process in the recognition result acquisition unit 14, a virtual object having an attribute "object to be grasped" different from the original attribute "obstacle" of the target object is generated in the pre-correction state. In this case, the worker performs an operation on the input unit 24Aa of the instruction device 2 to instruct correction of the attribute of the target virtual object. Then, in the post-correction state, the attribute of the virtual object is appropriately corrected based on the input signal S1 generated by the above-mentioned operation. Then, the correction receiving unit 16 generates recognition correction information Ia indicating a change in the attribute of the object corresponding to the target virtual object, and supplies the recognition correction information Ia to the recognition result acquisition unit 14. As a result, the change in the attribute of the virtual object is reflected in the second recognition result Im2 as a correction of the recognition result regarding the attribute of the corresponding object.

[0058] Next, a specific example of correction based on the above-mentioned first to fourth modes will be described. FIG. 6 shows a state of the work space before correction visually recognized by the worker when pick-and-place is set as a target task. In FIG. 6, a first object 81, a second object 82, and a third object 83 are present on a work table 79. The display control unit 15 displays virtual objects 81V, 82V and their attribute information 81T, 82T so as to be superimposed on the scenery (real world) or the actual image visually recognized by the worker, together with text information 78 that prompts the correction of the position, posture, and attributes of the objects. Note that, here, it is assumed that any calibration process performed in augmented reality or the like is performed, and that coordinate conversion or the like is appropriately performed between various coordinate systems such as the coordinate system of the sensor 7 and the display coordinate system that displays the virtual objects.

[0059] In this case, the virtual object 81V is misaligned with the position of the real object. Also, the attribute of the virtual object 82V indicated by the attribute information 82T (here, "object to be grasped") is different from the attribute of the original second object 82 (here, "obstacle"). Furthermore, the third object 83 is not recognized by the robot controller 1, and a corresponding virtual object is not generated. Then, the correction receiving unit 16 receives corrections for these differences based on the input signal S1 supplied from the instruction device 2, and the display control unit 15 instantly displays the latest virtual object reflecting the corrections.

[0060] FIG. 7 shows the state of the workspace as viewed by the worker after the correction of the virtual object is performed. In FIG. 7, the virtual object 81V is appropriately arranged at a position where it overlaps with the real object based on an operation instructing the movement of the virtual object 81V. Furthermore, based on an operation instructing the attribute change of the virtual object 82V, the attribute "obstacle" indicated by the attribute information 82T matches the attribute of the second object 82 to be recognized. Furthermore, based on an operation instructing the generation of a virtual object for the third object 83, the virtual object 83V for the third object 83 is generated with an appropriate position and posture. Furthermore, the attribute of the virtual object 83V indicated by the attribute information 83T matches the attribute of the third object 83 to be recognized.

[0061] 7, after various correction operations are performed, the correction receiving unit 16 receives an input signal S1 corresponding to an operation to confirm the correction, and supplies recognition correction information Ia indicating the received correction content to the recognition result acquiring unit 14. After that, the recognition result acquiring unit 14 supplies a second recognition result Im2 reflecting the recognition correction information Ia to the motion planning unit 17, and the motion planning unit 17 starts calculating a motion plan for the robot 5 based on the second recognition result Im2. In this case, the display control unit 15 displays text information 78A notifying the operator that the correction has been accepted and that a motion plan has been established and that robot control has begun.

[0062] In this way, the robot controller 1 can accurately correct recognition errors of objects in the working space, and can accurately formulate an operation plan and control the robot based on accurate recognition results.

[0063] Next, a supplementary explanation will be given about the determination of whether or not correction is necessary by the correction receiving unit 16. In a first example, the correction receiving unit 16 receives an input specifying whether or not correction is necessary, and determines whether or not correction is necessary based on an input signal S1 corresponding to the input.

[0064] In a second example, the correction receiving unit 16 may determine whether correction is necessary based on a degree of confidence indicating the degree of confidence in the accuracy of the recognition (estimation) of the position, orientation, and attributes of each object. In this case, the correction receiving unit 16 associates a degree of confidence with each estimation result of the position, orientation, and attributes of each object in the first recognition result Im1, and when all of these degrees of confidence are equal to or greater than a predetermined threshold, the correction receiving unit 16 determines that correction of the first recognition result Im1 is unnecessary and supplies recognition correction information Ia to the recognition result acquisition unit 14 to the effect that correction is unnecessary. The above-mentioned threshold is stored in, for example, the memory 12 or the storage device 4.

[0065] On the other hand, when there is a confidence level lower than the above-mentioned threshold, the correction receiving unit 16 determines that a correction is necessary, and issues a display control instruction to receive the correction to the display control unit 15. After that, the display control unit 15 performs display control to realize a display as shown in FIG.

[0066] Preferably, in the second example, the display control unit 15 determines the display mode of various information represented by the first recognition result Im1 based on the confidence level. For example, in the example of Fig. 6, when the confidence level of either the position or the orientation of the first object 81 is less than a threshold, the display control unit 15 highlights the virtual object 81V representing the position and the orientation of the first object 81. In addition, when the confidence level of the attribute of the first object 81 is less than a threshold, the display control unit 15 highlights the attribute information 81T representing the attribute of the first object 81.

[0067] In this way, the display control unit 15 highlights information about the recognition result that is particularly in need of correction (i.e., the confidence level is less than the threshold value). This makes it possible to effectively prevent correction omissions and smoothly support the worker in making corrections.

[0068] Here, the confidence level included in the first recognition result Im1 will be explained in more detail. When estimating the position, orientation, and attributes of an object detected based on the sensor signal S4, the recognition result acquisition unit 14 calculates the confidence level for each estimated element, and generates the first recognition result Im1 in which the calculated confidence levels are associated with each of the estimated position, orientation, and attributes of the object. In this case, for example, when an estimation model based on a neural network is used to estimate the position, orientation, and attributes of an object, the correction acceptance unit 16 uses the certainty level (reliability level) output by the estimation model together with the estimation result as the above-mentioned confidence level. For example, the estimation model for estimating the position and orientation of an object is a regression model, and the estimation model for estimating the attributes of the object is a classification model.

[0069] (6) Details of the operation sequence generation part Next, the detailed processing of the motion planning unit 17 will be described.

[0070] (5-1) Functional Blocks 8 is an example of a functional block diagram showing the functional configuration of the motion planning unit 17. The motion planning unit 17 functionally includes an abstract state setting unit 31, a target logical formula generating unit 32, a time step logical formula generating unit 33, an abstract model generating unit 34, a control input generating unit 35, and a subtask sequence generating unit 36.

[0071] The abstract state setting unit 31 sets an abstract state in the work space based on the second recognition result Im2 supplied from the recognition result acquisition unit 14. In this case, the abstract state setting unit 31 defines a proposition for expressing, in a logical formula, each abstract state that needs to be considered when executing a target task based on the second recognition result Im2. The abstract state setting unit 31 supplies information indicating the set abstract state (also referred to as "abstract state setting information IS") to the target logical formula generation unit 32.

[0072] The target logical formula generation unit 32 converts the target task into a logical formula of temporal logic (also called "target logical formula Ltag") representing a final achieved state based on the abstract state setting information IS. In other words, the target logical formula generation unit 32 generates the target logical formula Ltag based on the initial state of the workspace before the operation of the robot 5 specified based on the abstract state setting information IS and the final achieved state of the workspace. In addition, the target logical formula generation unit 32 adds a constraint condition to be satisfied in the execution of the target task to the target logical formula Ltag by referring to the constraint condition information I2 from the application information storage unit 41. Then, the target logical formula generation unit 32 supplies the generated target logical formula Ltag to the time-step logical formula generation unit 33.

[0073] The goal logical formula generating unit 32 may recognize the final achieved state of the workspace based on information stored in advance in the storage device 4, or based on the input signal S1 supplied from the instruction device 2.

[0074] The time-step logical formula generation unit 33 converts the target logical formula Ltag supplied from the target logical formula generation unit 32 into a logical formula (also called a "time-step logical formula Lts") that represents a state at each time step. Then, the time-step logical formula generation unit 33 supplies the generated time-step logical formula Lts to the control input generation unit 35.

[0075] The abstract model generation unit 34 generates an abstract model "Σ" that abstracts the real dynamics in the working space, based on the abstract model information I5 stored in the application information storage unit 41 and the second recognition result Im2 supplied from the abstract state setting unit 31. In this case, the abstract model generation unit 34 regards the target dynamics as a hybrid system in which continuous dynamics and discrete dynamics are mixed, and generates an abstract model Σ based on the hybrid system. A method for generating the abstract model Σ will be described later. The abstract model generation unit 34 supplies the generated abstract model Σ to the control input generation unit 35.

[0076] The control input generation unit 35 determines a control input to the robot 5 for each time step that satisfies the time step logical formula Lts supplied from the time step logical formula generation unit 33 and the abstract model Σ supplied from the abstract model generation unit 34 and optimizes an evaluation function (for example, a function representing the amount of energy consumed by the robot). Then, the control input generation unit 35 supplies information indicating the control input to the robot 5 for each time step (also referred to as "control input information Icn") to the subtask sequence generation unit 36.

[0077] The subtask sequence generation unit 36 ​​generates an operation sequence Sr, which is a sequence of subtasks, based on the control input information Icn supplied from the control input generation unit 35 and the subtask information I4 stored in the application information storage unit 41, and supplies the operation sequence Sr to the robot control unit 18.

[0078] (6-2) Abstract State Setting Part The abstract state setting unit 31 sets an abstract state in the workspace based on the second recognition result Im2 and the abstract state designation information I1 acquired from the application information storage unit 41. In this case, the abstract state setting unit 31 first refers to the abstract state designation information I1 and recognizes the abstract state to be set in the workspace. Note that the abstract state to be set in the workspace differs depending on the type of the target task.

[0079] Fig. 9 shows an overhead view of a workspace when a target task is pick-and-place. In the workspace shown in Fig. 9, there are two robot arms 52a and 52b, four objects 61 (61a to 61d), an obstacle 62, and an area G which is the destination of the object 61.

[0080] In this case, first, the abstract state setting unit 31 recognizes the state of the object 61, the range in which the obstacle 62 exists, the state of the robot 5, the range in which the area G exists, and the like.

[0081] Here, the abstract state setting unit 31 recognizes the position vectors "x1" to "x4" of the centers of the objects 61a to 61d as the positions of the objects 61a to 61d. In addition, the abstract state setting unit 31 recognizes the position vector "x r1 " and the position vector of the robot hand 53b "x r2 " are recognized as the positions of the robot arm 52a and the robot arm 52b.

[0082] Similarly, the abstract state setting unit 31 recognizes the postures of the objects 61a to 61d (not necessary in the example of FIG. 9 because the objects are spherical), the range of existence of the obstacle 62, the range of existence of the area G, etc. Note that, for example, when the obstacle 62 is regarded as a rectangular parallelepiped and the area G is regarded as a rectangle, the abstract state setting unit 31 recognizes the position vectors of each vertex of the obstacle 62 and the area G.

[0083] Furthermore, the abstract state setting unit 31 determines an abstract state to be defined in the target task by referring to the abstract state designation information I1. In this case, the abstract state setting unit 31 determines a proposition indicating an abstract state based on the second recognition result Im2 (e.g., the number of each type of object) regarding the objects existing in the workspace and the abstract state designation information I1.

[0084] In the example of FIG. 9, the abstract state setting unit 31 assigns identification labels "1" to "4" to the objects 61a to 61d identified by the second recognition result Im2, respectively. In addition, the abstract state setting unit 31 assigns the proposition "g i In addition, the abstract state setting unit 31 assigns an identification label "O" to the obstacle 62 and defines a proposition "o i Furthermore, the abstract state setting unit 31 defines a proposition "h" that the robot arms 52 interfere with each other. The abstract state setting unit 31 also defines a proposition "v" that the object "i" exists in the working table (the table in which the object and the obstacle exist in the initial state). i ” , the proposition “wi " etc. The non-work area is, for example, an area (floor surface, etc.) in which the object would be present if the object were to fall from the work table.

[0085] In this way, the abstract state setting unit 31 recognizes the abstract state to be defined by referring to the abstract state designation information I1, and determines the proposition expressing the abstract state (in the above example, g i ,o i , h, etc.) are defined according to the number of objects 61, the number of robot arms 52, the number of obstacles 62, the number of robots 5, etc. Then, the abstract state setting unit 31 supplies information indicating a proposition expressing an abstract state to the target logical formula generation unit 32 as abstract state setting information IS.

[0086] (6-3) Target formula generator First, the target logical expression generating unit 32 converts the target task into a logical expression using temporal logic.

[0087] For example, in the example of FIG. 9, suppose that a goal task of "Eventually, the object (i=2) exists in the area G" is given. In this case, the goal logical formula generating unit 32 generates the goal task by dividing the goal task into an operator "◇" corresponding to "eventually" in the linear logical formula (LTL: Linear Temporal Logic) and a proposition "g i " to generate the logical formula "◇g2". Furthermore, the target logical formula generation unit 32 may express a logical formula using any temporal logic operator other than the operator "◇" (logical product "∧", logical sum "∨", negation "￢", logical inclusion "⇒", always "□", next "○", until "U", etc.). Furthermore, the logical formula may be expressed using any temporal logic, such as MTL (Metric Temporal Logic) or STL (Signal Temporal Logic), not limited to linear temporal logic.

[0088] The target task may be specified in a natural language. There are various techniques for converting a task expressed in a natural language into a logical formula.

[0089] Next, the target logical expression generating unit 32 generates a target logical expression Ltag by adding the constraint condition indicated by the constraint condition information I2 to the logical expression indicating the objective task.

[0090] For example, when the constraint condition information I2 includes two constraint conditions corresponding to the pick-and-place shown in FIG. 9, namely, "the robot arms 52 do not interfere with each other" and "the target object i does not interfere with the obstacle O", the target logical formula generating unit 32 converts these constraint conditions into logical formulas. Specifically, the target logical formula generating unit 32 converts the proposition "o i Using " and proposition "h", the two constraints above are converted into the following logical formulas, respectively. □￢h ∧ i □￢o i

[0091] Therefore, in this case, the target logical formula generation unit 32 generates the following target logical formula Ltag by adding the logical formulas of these constraint conditions to the logical formula "◇g2" corresponding to the objective task "the object (i=2) ultimately exists in area G." (◇g2)∧(□￢h)∧(∧ i □￢o i )

[0092] In reality, the constraint conditions corresponding to pick-and-place are not limited to the above two, and there are other constraint conditions such as "the robot arm 52 does not interfere with the obstacle O," "the multiple robot arms 52 do not grab the same object," and "the objects do not come into contact with each other." These constraint conditions are also stored in the constraint condition information I2 and reflected in the target logical formula Ltag.

[0093] (6-4) Time step formula generator The time-step logical formula generation unit 33 determines the number of time steps (also called the "target number of time steps") to complete the target task, and determines a combination of propositions that express the state at each time step that satisfies the target logical formula Ltag in the target number of time steps. Since there are usually multiple combinations, the time-step logical formula generation unit 33 generates a logical formula that combines these combinations with a logical sum as the time-step logical formula Lts. The above combinations become candidates for a logical formula that expresses a sequence of operations to be commanded to the robot 5, and are hereinafter also called "candidates φ".

[0094] Here, a specific example of the process of the time-step logical expression generating unit 33 when a goal task of "finally the object (i=2) exists in the area G" is set as exemplified in the description of FIG. 9 will be described.

[0095] In this case, the following target logical formula Ltag is supplied from the target logical formula generator 32 to the time step logical formula generator 33. (◇g2)∧(□￢h)∧(∧ i □￢o i ) In this case, the time step formula generator 33 generates the proposition "g i " is extended to include the concept of time steps, i,k " is used. Here, the proposition "g i,k " is a proposition that "object i exists in region G at time step k." If the target time step number is set to "3," the target logical formula Ltag can be rewritten as follows. (◇g 2,3 )∧(∧ k=1,2,3 □￢h k )∧(∧ i,k=1,2,3 □￢o i,k )

[0096] Also, ◇g 2,3 can be rewritten as shown in the following equation:

[0097]

number

[0098] At this time, the above-mentioned target logical formula Ltag is expressed by the logical sum (φ1∨φ2∨φ3∨φ4) of the four candidates “φ1” to “φ4” shown below.

[0099]

number

[0100] Therefore, the time-step logical expression generating unit 33 determines the logical sum of the four candidates φ1 to φ4 as the time-step logical expression Lts. In this case, the time-step logical expression Lts is true when at least one of the four candidates φ1 to φ4 is true.

[0101] Next, a method for setting the target number of time steps will be explained in more detail.

[0102] The time step logical formula generating unit 33 determines the target number of time steps based on, for example, the expected time of the task specified by the input signal S1 supplied from the instruction device 2. In this case, the time step logical formula generating unit 33 calculates the target number of time steps from the expected time based on information on the time width per time step stored in the memory 12 or the storage device 4. In another example, the time step logical formula generating unit 33 stores in advance in the memory 12 or the storage device 4 information that associates a target number of time steps appropriate for each type of target task, and determines the target number of time steps according to the type of target task to be executed by referring to the information.

[0103] Preferably, the time-step logical formula generating unit 33 sets the target number of time steps to a predetermined initial value. Then, the time-step logical formula generating unit 33 gradually increases the target number of time steps until a time-step logical formula Lts that allows the control input generating unit 35 to determine the control input is generated. In this case, if the control input generating unit 35 performs optimization processing using the set target number of time steps and is unable to derive an optimal solution, the time-step logical formula generating unit 33 adds a predetermined number (an integer equal to or greater than 1) to the target number of time steps.

[0104] At this time, the time-step logical formula generating unit 33 may set the initial value of the target number of time steps to a value smaller than the number of time steps corresponding to the working time of the target task expected by the user. This allows the time-step logical formula generating unit 33 to preferably prevent setting an unnecessarily large target number of time steps.

[0105] (6-5) Abstract Model Generation The abstract model generation unit 34 generates the abstract model Σ based on the abstract model information I5 and the second recognition result Im2. Here, the abstract model information I5 records information required for generating the abstract model Σ for each type of target task. For example, when the target task is pick-and-place, a general-purpose abstract model that does not specify the position and number of objects, the position of the area where the objects are placed, the number of robots 5 (or the number of robot arms 52), etc. is recorded in the abstract model information I5. Then, the abstract model generation unit 34 generates the abstract model Σ by reflecting the second recognition result Im2 in the general-purpose abstract model including the dynamics of the robot 5 recorded in the abstract model information I5. As a result, the abstract model Σ becomes a model in which the state of the object in the working space and the dynamics of the robot 5 are abstractly represented. In the case of pick-and-place, the state of the object in the working space indicates the position and number of objects, the position of the area where the objects are placed, the number of robots 5, etc.

[0106] When other work objects exist, information on the abstracted dynamics of the other work objects may be included in the abstract model information I5. In this case, the abstract model Σ is a model in which the state of the objects in the work space, the dynamics of the robot 5, and the dynamics of the other work objects are abstractly represented.

[0107] Here, when the robot 5 is performing a target task, the dynamics in the workspace frequently changes. For example, in pick-and-place, when the robot arm 52 is holding an object i, the object i can be moved, but when the robot arm 52 is not holding the object i, the object i cannot be moved.

[0108] Considering the above, in the present embodiment, in the case of pick-and-place, the action of grabbing the target object i is expressed as a logical variable “δ i In this case, for example, the abstract model generation unit 34 can determine the abstract model Σ to be set for the work space shown in FIG.

[0109]

number

[0110] Here, "u j " indicates the control input for controlling the robot hand j ("j=1" is the robot hand 53a, and "j=2" is the robot hand 53b), "I" indicates the unit matrix, and "0" indicates the zero matrix. Note that the control input is assumed to be velocity here as an example, but it may be acceleration. Also, "δ j,i " is a logical variable that is "1" when the robot hand j is holding the object i, and is "0" otherwise. Also, "x r1 ", "x r2" indicates the position vector of robot hand j (j = 1, 2), and "x1" to "x4" indicate the position vector of object i (i = 1 to 4). If object i has a shape other than a sphere and its posture needs to be taken into consideration, vectors "x1" to "x4" contain elements that represent its posture, such as Euler angles. "h(x)" is a variable that satisfies "h(x) ≧ 0" when a robot hand is close enough to an object to grasp it, and satisfies the following relationship with logical variable δ. δ=1 ⇔ h(x)≧0 In this equation, if the robot hand is close enough to an object to grasp it, it is assumed that the robot hand is grasping the object, and the logical variable δ is set to 1.

[0111] Here, equation (1) is a differential equation that shows the relationship between the state of the object at time step k and the state of the object at time step k + 1. In the above equation (1), the grip state is represented by a logical variable that is a discrete value, and the movement of the object is represented by a continuous value, so equation (1) shows a hybrid system.

[0112] Equation (1) takes into consideration only the dynamics of the robot hand, which is the end effector of the robot 5 that actually grasps an object, rather than the detailed dynamics of the entire robot 5. This makes it possible to suitably reduce the amount of calculation in the optimization process performed by the control input generation unit 35.

[0113] Furthermore, the abstract model information I5 records logical variables corresponding to the action of switching dynamics (the action of grasping the object i in the case of pick-and-place) and information for deriving the differential equation of formula (1) from the second recognition result Im2. Therefore, even if the positions and number of objects, the area where the objects are placed (area G in FIG. 9), the number of robots 5, etc. vary, the abstract model generation unit 34 can determine an abstract model Σ that is suited to the environment of the target workspace based on the abstract model information I5 and the second recognition result Im2.

[0114] Note that the abstract model generating unit 34 may generate a model of a mixed logical dynamical (MLD) system or a hybrid system that combines a Petri net, an automaton, and the like, instead of the model shown in equation (1).

[0115] (6-6) Control Input Generation Unit The control input generation unit 35 determines an optimal control input for the robot 5 for each time step based on the time step logical formula Lts supplied from the time step logical formula generation unit 33 and the abstract model Σ supplied from the abstract model generation unit 34. In this case, the control input generation unit 35 defines an evaluation function for the target task, and solves an optimization problem of minimizing the evaluation function with the abstract model Σ and the time step logical formula Lts as constraints. The evaluation function is, for example, predetermined for each type of target task, and stored in the memory 12 or the storage device 4.

[0116] For example, when a pick-and-place task is set as a target task, the control input generating unit 35 calculates the distance "d k " and the control input "u k The evaluation function is determined so that the above-mentioned distance d k In the case of a target task in which "the object (i=2) is ultimately present in the region G," corresponds to the distance between the object (i=2) and the region G at time step k.

[0117] In this case, the control input generator 35 calculates the distance d k The square of the norm of and the control input u k The control input generation unit 35 determines the sum of the square of the norm of the abstract model Σ and the time step logical formula Lts (i.e., the candidate φ i The constrained mixed integer optimization problem shown in the following equation (2) is solved, with the constraint being the logical sum of

[0118]

number

[0119] Here, "T" is the number of time steps to be optimized, and may be the target number of time steps, or may be a predetermined number smaller than the target number of time steps, as described below. In this case, preferably, the control input generation unit 35 approximates the logical variables to continuous values ​​(a continuous relaxation problem). This allows the control input generation unit 35 to preferably reduce the amount of calculation. Note that when STL is adopted instead of linear logic (LTL), it is possible to describe it as a nonlinear optimization problem.

[0120] Furthermore, when the target number of time steps is long (for example, larger than a predetermined threshold), the control input generation unit 35 may set the number of time steps used for optimization to a value smaller than the target number of time steps (for example, the above-mentioned threshold). In this case, the control input generation unit 35 sequentially calculates the control input u by solving the above-mentioned optimization problem, for example, every time a predetermined number of time steps elapses. k Determine.

[0121] Preferably, the control input generation unit 35 solves the above optimization problem for each predetermined event corresponding to an intermediate state with respect to the achievement state of the target task, and determines the control input u to be used. k may be determined. In this case, the control input generating unit 35 sets the number of time steps until the next event occurs to the number of time steps used for optimization. The above-mentioned event is, for example, an event in which dynamics in the workspace is switched. For example, when pick-and-place is set as the target task, the robot 5 grasps an object, the robot 5 finishes carrying one of a plurality of objects to be carried to a destination point, and the like are defined as events. The events are, for example, defined in advance for each type of target task, and information for identifying the event for each type of target task is stored in the storage device 4.

[0122] (6-7) Subtask sequence generation part The subtask sequence generation unit 36 ​​generates an operation sequence Sr based on the control input information Icn supplied from the control input generation unit 35 and the subtask information I4 stored in the application information storage unit 41. In this case, the subtask sequence generation unit 36 ​​recognizes subtasks that the robot 5 can accept by referring to the subtask information I4, and converts the control input for each time step indicated by the control input information Icn into a subtask.

[0123] For example, in the subtask information I4, functions indicating two subtasks, namely, moving (reaching) of the robot hand and grasping (grasping) of the robot hand, are defined as subtasks that the robot 5 can accept when the target task is pick-and-place. In this case, the function "Move" representing reaching is a function that takes as arguments, for example, the initial state of the robot 5 before the execution of the function, the final state of the robot 5 after the execution of the function, and the time required to execute the function. Also, the function "Grasp" representing grasping is a function that takes as arguments, for example, the state of the robot 5 before the execution of the function, the state of the object to be grasped before the execution of the function, and the logical variable δ. Here, the function "Grasp" represents a grasping action when the logical variable δ is "1", and represents a releasing action when the logical variable δ is "0". In this case, the subtask sequence generation unit 36 ​​determines the function "Move" based on the trajectory of the robot hand determined by the control input for each time step indicated by the control input information Icn, and determines the function "Grasp" based on the transition of the logical variable δ for each time step indicated by the control input information Icn.

[0124] Then, the subtask sequence generation unit 36 ​​generates an operation sequence Sr consisting of a function "Move" and a function "Grasp", and supplies the operation sequence Sr to the robot control unit 18. For example, when the target task is "finally, the object (i=2) is present in the area G", the subtask sequence generation unit 36 ​​generates an operation sequence Sr of the function "Move", the function "Grasp", the function "Move", and the function "Grasp" for the robot hand closest to the object (i=2). In this case, the robot hand closest to the object (i=2) moves to the position of the object (i=2) by the first function "Move", grasps the object (i=2) by the first function "Grasp", moves to the area G by the second function "Move", and places the object (i=2) in the area G by the second function "Grasp".

[0125] (7) Processing flow FIG. 10 is an example of a flowchart showing an outline of the robot control process executed by the robot controller 1 in the first embodiment.

[0126] First, the robot controller 1 acquires a sensor signal S4 from the sensor 7 (step S11). Then, the recognition result acquisition unit 14 of the robot controller 1 recognizes the state (including the position and posture) and attributes of the object in the working space based on the acquired sensor signal S4 (step S12). As a result, the recognition result acquisition unit 14 generates a first recognition result Im1 regarding the object in the working space.

[0127] Next, the display control unit 15 causes the pointing device 2 to display virtual objects superimposed on real objects in the landscape or real-life image based on the first recognition result Im1 (step S13). In this case, the display control unit 15 generates a display control signal S2 for displaying virtual objects corresponding to each object identified by the first recognition result Im1, and supplies the display control signal S2 to the pointing device 2.

[0128] Then, the correction receiving unit 16 determines whether or not the first recognition result Im1 needs to be corrected (step S14). In this case, the correction receiving unit 16 may determine whether or not the first recognition result Im1 needs to be corrected based on the confidence level included in the first recognition result Im1, or may receive an input specifying whether or not the first recognition result Im1 needs to be corrected and determine whether or not the first recognition result Im1 needs to be corrected based on the received input.

[0129] Then, when the correction acceptance unit 16 determines that the first recognition result Im1 needs to be corrected (step S14; Yes), it accepts the correction of the first recognition result Im1 (step S15). In this case, the correction acceptance unit 16 accepts the correction based on any operation method using the input unit 24a, which is any user interface provided in the instruction device 2 (more specifically, the specification of the target to be corrected and the specification of the content to be corrected, etc.). Then, the recognition result acquisition unit 14 generates the second recognition result Im2 reflecting the recognition correction information Ia generated by the correction acceptance unit 16 (step S16). On the other hand, when the correction acceptance unit 16 determines that the first recognition result Im1 does not need to be corrected (step S14; No), the process proceeds to step S17. In this case, the correction acceptance unit 16 supplies the recognition correction information Ia indicating that the correction is not necessary to the recognition result acquisition unit 14, and the recognition result acquisition unit 14 supplies the first recognition result Im1 to the operation planning unit 17 as the second recognition result Im2.

[0130] Then, the motion planning unit 17 determines a motion plan for the robot 5 based on the second recognition result Im2 (step S17). As a result, the motion planning unit 17 generates a motion sequence Sr, which is a motion sequence of the robot 5. Then, the robot control unit 18 performs robot control based on the determined motion plan (step S18). In this case, the robot control unit 18 sequentially supplies the control signals S3 generated based on the motion sequence Sr to the robot 5, and controls the robot 5 to operate in accordance with the generated motion sequence Sr. (8) Variations The block configuration of the motion planning unit 17 shown in FIG. 8 is an example, and various modifications may be made.

[0131] For example, information on candidates φ for a sequence of operations to be commanded to the robot 5 is stored in advance in the storage device 4, and the motion planning unit 17 executes an optimization process of the control input generation unit 35 based on the information. As a result, the motion planning unit 17 selects the optimal candidate φ and determines the control input of the robot 5. In this case, the motion planning unit 17 does not need to have functions corresponding to the abstract state setting unit 31, the target logical formula generation unit 32, and the time step logical formula generation unit 33 in generating the motion sequence Sr. In this way, information on the execution results of some of the functional blocks of the motion planning unit 17 shown in FIG. 8 may be stored in advance in the application information storage unit 41.

[0132] In another example, the application information may include design information such as a flowchart for designing an operation sequence Sr corresponding to a target task, and the operation planner 17 may generate the operation sequence Sr by referring to the design information. A specific example of executing a task based on a task sequence designed in advance is disclosed in, for example, JP 2017-39170 A.

[0133] <Second embodiment> The robot controller 1 according to the second embodiment performs a process of displaying information (also called "trajectory information") on the trajectory of an object (target object, robot 5) based on the formulated motion plan after formulating the motion plan, instead of or in addition to the process of accepting corrections to the first recognition result Im1, and accepting corrections to the trajectory. As a result, the robot controller 1 according to the second embodiment appropriately corrects the motion plan so that the target task is executed according to the flow intended by the operator.

[0134] Hereinafter, in the robot control system 100, the same components as those in the first embodiment are appropriately denoted by the same reference numerals, and the description thereof will be omitted. Note that the configuration of the robot control system 100 in the second embodiment is the same as the configuration shown in FIG.

[0135] Fig. 11 is an example of a functional block of a robot controller 1A in the second embodiment. The robot controller 1A has a hardware configuration shown in Fig. 2(A), and the processor 11 of the robot controller 1A functionally has a recognition result acquisition unit 14A, a display control unit 15A, a correction acceptance unit 16A, an operation planning unit 17A, and a robot control unit 18A.

[0136] The recognition result acquisition unit 14A generates a first recognition result Im1 and generates a second recognition result Im2 based on the recognition correction information Ia, similarly to the first embodiment.

[0137] In addition to the processing executed by the display control unit 15 in the first embodiment, the display control unit 15A acquires trajectory information specified from the operation plan determined by the operation planning unit 17A from the operation planning unit 17A, and controls the display of the instruction device 2 regarding the trajectory information. In this case, the display control unit 15A generates a display control signal S2 for the instruction device 2 to display trajectory information regarding the trajectory of the object, etc. for each time step indicated by the operation sequence Sr, and supplies the display control signal S2 to the instruction device 2. In this case, the display control unit 15A may display trajectory information representing the trajectory of the robot 5 in addition to the trajectory of the object. Here, the display control unit 15A may display, as the trajectory information, information representing the state transition of the object, etc. for each time step number, or information representing the state transition of the object, etc. for each predetermined number of time steps.

[0138] In addition to the generation process of the recognition correction information Ia executed by the correction receiving unit 16 in the first embodiment, the correction receiving unit 16A also receives corrections to the trajectory information by the operation of the operator using the instruction device 2. Then, when the operation related to the correction is completed, the correction receiving unit 16A generates trajectory correction information "Ib" indicating the correction content related to the trajectory of the object, etc., and supplies the trajectory correction information Ib to the motion planning unit 17A. Note that, when there is no input related to the correction, the correction receiving unit 16A supplies the trajectory correction information Ib indicating that there is no correction to the motion planning unit 17A.

[0139] In addition to the generation process of the operation sequence Sr executed by the operation planning unit 17 in the first embodiment, the operation planning unit 17A generates an operation sequence Sr (also called "second operation sequence Srb") that reflects the trajectory correction information Ib supplied from the correction receiving unit 16A. As a result, the operation planning unit 17A formulates a new operation plan in which the original operation plan is corrected so that the state of the target object specified by the correction is realized. Then, the operation planning unit 17A supplies the generated second operation sequence Srb to the robot control unit 18. Hereinafter, for convenience, the operation sequence Sr before reflecting the trajectory correction information Ib is also called "first operation sequence Sr". In addition, the operation plan based on the first operation sequence Sr is called "first operation plan", and the operation plan based on the second operation sequence Srb is called "second operation plan". When trajectory correction information Ib is generated to the effect that correction of the first operation plan is unnecessary, the second operation sequence Srb becomes the same as the first operation sequence Sr.

[0140] The motion planning unit 17A may determine whether the second motion sequence Srb reflecting the trajectory correction information Ib satisfies the constraint conditions, and may supply the generated second motion sequence Srb to the robot control unit 18 only when the constraint conditions indicated by the constraint condition information I2 are satisfied. This allows the robot 5 to suitably execute only motion plans that satisfy the constraint conditions. When the motion planning unit 17A determines that the second motion sequence Srb does not satisfy the constraint conditions, it instructs the display control unit 15A and the motion planning unit 17A to execute a process of accepting re-correction.

[0141] The robot control unit 18A controls the operation of the robot 5 by supplying a control signal S3 to the robot 5 via the interface 13 based on the second operation sequence Srb supplied from the operation planning unit 17A.

[0142] Here, a supplementary explanation will be given on the generation of trajectory information. After generating the first operation sequence Sr, the operation planning unit 17A supplies the display control unit 15A with trajectory information of the object (and the robot 5) required for display control by the display control unit 15A. In this case, the position (posture) vectors (see formula (1)) of the robot 5 (specifically, the robot hand) and the object for each time step are obtained by the optimization process based on formula (2) executed in formulating the first operation plan. Therefore, the operation planning unit 17A supplies these position (posture) vectors to the display control unit 15A as trajectory information. Here, the trajectory information supplied by the operation planning unit 17A to the display control unit 15A includes information on the timing at which the robot hand grasps (and releases) the object (i.e., δ in formula (1)). j,i The information may include the information on the gripping (and releasing) direction, and the posture of the robot hand when gripping (and releasing). The gripping (and releasing) direction of the target object may be specified based on, for example, the trajectory of the position vector of the robot hand and the gripping (and releasing) timing.

[0143] In addition, the functional blocks shown in FIG. 11 are based on the premise that the processing of the first embodiment is performed, but this is not limited thereto, and the robot controller 1A does not have to execute processing related to correction of the first recognition result Im1 (generation of recognition correction information Ia by the correction receiving unit 16A, generation of the second recognition result Im2 by the recognition result acquisition unit 14A, display control of the first recognition result Im1 by the display control unit 15, etc.).

[0144] Next, specific examples (first specific example and second specific example) of the process relating to display and correction of trajectory information of an object or the like in the second embodiment will be described.

[0145] 12 is a diagram showing trajectory information in the first specific example. The first specific example is a specific example related to a target task of placing an object 85 in a box 86, and the robot controller 1A roughly displays the trajectories of the robot hand 53 of the robot 5 and the object 85 specified by the first motion plan determined by the motion planner 17A.

[0146] In FIG. 12, positions "P1" to "P9" indicate the positions of the robot hand 53 for each predetermined number of time steps based on the first operation plan, and a trajectory line 87 indicates the trajectory (path) of the robot hand 53 based on the first operation plan. Moreover, virtual objects "85Va" to "85Ve" indicate virtual objects representing the position and posture of the target object 85 for each predetermined number of time steps based on the first operation plan. Moreover, an arrow "Aw1" indicates the direction in which the robot hand 53 grasps the target object 85 when the robot hand 53 switches to a grasping state based on the first operation plan, and an arrow "Aw2" indicates the direction in which the robot hand 53 moves away from the target object 85 when the robot hand 53 switches to a non-grasping state based on the first operation plan. Moreover, virtual robot hands "53Va" to "53Vh" are virtual objects representing the posture of the robot hand 53 immediately before and after the robot hand 53 switches between a grasping state and a non-grasping state based on the first operation plan. In addition, in the first operation plan formulated by the operation planning unit 17A, the trajectory of the robot hand 53, which is the end effector of the robot 5, is generated as the trajectory of the robot 5, and therefore the trajectory of the robot hand 53 is shown here as the trajectory of the robot 5.

[0147] Based on the trajectory information received from the motion planning unit 17A, the display control unit 15A causes the instruction device 2 to display positions P1 to P9, a trajectory line 87, and arrows Aw1 and Aw2 representing the trajectory of the robot hand 53, virtual robot hands 53Va to 53Vh representing the posture of the robot hand 53 immediately before and after switching between the grasping state and the non-grasping state, and virtual objects 85Va to 85Ve representing the trajectory of the target object 85. This allows the display control unit 15A to allow the worker to appropriately grasp the outline of the first motion plan that has been formulated. Note that, when the trajectory of each joint of the robot 5 is required in the first motion plan, the display control unit 15A may display the trajectory of each joint of the robot 5 in addition to the trajectory of the robot hand 53.

[0148] Then, the correction receiving unit 16A receives a correction of each element based on the first operation plan shown in Fig. 12. The object to be corrected in this case may be the state of the robot hand 53 or the object 85 at any time step (including the posture of the robot hand 53 specified by the virtual robot hands 53Va to 53Vh), or each timing of grasping / non-grasping, etc.

[0149] Here, in the first operation plan shown in FIG. 12, the robot hand 53 is trying to place the object 85 in the box 86 while holding the handle portion of the object 85, and there is a possibility that the robot hand 53 may come into contact with the box 86 and not be able to properly place the object 85 in the box 86, resulting in a failure of the task. In consideration of the above, the worker carries the object 85 to the vicinity of the box 86 with the robot hand 53 holding the handle of the object 85, and then performs an operation to generate a correction by using the instruction device 2 to add an operation to change the grip of the object 85 so as to grasp the upper part of the object 85. Then, the correction receiving unit 16A generates trajectory correction information Ib based on the input signal S1 generated by the above-mentioned operation, and supplies the generated information to the operation planning unit 17A. Then, the operation planning unit 17A generates a second operation sequence Srb reflecting the trajectory correction information Ib, and the display control unit 15A causes the instruction device 2 to display the trajectory information specified by the second operation sequence Srb again.

[0150] FIG. 13 is a diagram showing roughly the corrected trajectory information of the robot hand 53 and the object 85 corrected based on the input for correcting the trajectories and the like of the robot hand 53 and the object 85 in the first concrete example.

[0151] 13, positions "P11" to "P20" indicate the positions of the robot hand 53 for each predetermined number of time steps based on the revised second operation plan, and a trajectory line 88 indicates the trajectory (path) of the robot hand 53 based on the revised second operation plan. Also, virtual objects "85Vf" to "85Vj" indicate virtual objects representing the position and posture of the target object 85 for each predetermined number of time steps based on the second operation plan. Also, arrows "Aw11" and "Aw13" indicate the direction in which the robot hand 53 grasps the target object 85 when the robot hand 53 switches from a non-grasping state to a grasping state based on the second operation plan, and arrows "Aw12" and "Aw14" indicate the direction in which the robot hand 53 moves away from the target object 85 when the robot hand 53 switches from a grasping state to a non-grasping state based on the second operation plan. Moreover, the virtual robot hands "53Vj" to "53Vm" are virtual objects representing the posture of the robot hand 53 immediately before the robot hand 53 switches between the grasping state and the non-grasping state based on the second operation plan. Note that, instead of the example of Fig. 13, the correction receiving unit 16A may also display the virtual object representing the posture of the robot hand 53 immediately after the robot hand 53 switches between the grasping state and the non-grasping state based on the second operation plan in the same manner as in Fig. 12.

[0152] In the example of FIG. 13, the correction receiving unit 16A generates trajectory correction information Ib indicating the addition of an action of placing the object 85 on a horizontal plane and an action of gripping the upper part of the object 85 (i.e., the addition of an action of changing the grip of the object 85) in a time step corresponding to the position P6 or the position P7 of FIG. 12 based on the operation of the operator. The action of placing the object 85 on a horizontal plane is an action related to the position P16, the virtual object 85Vg, the virtual robot hand 53Vk, and the arrow Aw12, and the action of gripping the upper part of the object 85 is an action related to the position P17, the virtual robot hand 53Vl, and the arrow Aw13. The trajectory correction information Ib also includes information regarding the correction of the action of placing the object 85 in the box 86, which is corrected in association with the change of these actions. For example, regarding the action of placing the object 85 in the box 86, a virtual robot hand 53Vm showing a posture different from the posture of the robot hand 53 specified by the virtual robot hands 53Vg and 53Vh shown in FIG. 12 is displayed in FIG. 13. Then, the motion planning unit 17A determines a second motion plan shown in FIG. 13 based on this trajectory correction information Ib.

[0153] Here, a method for determining the second operation plan will be explained in more detail. In the first example, the operation planner 17A recognizes the correction contents indicated in the trajectory correction information Ib as additional constraint conditions. Then, the operation planner 17A calculates the state of the robot hand 53 and the object 85 for each time step by executing the optimization process indicated in the formula (2) again based on the additional constraint conditions and the existing constraint conditions indicated in the constraint condition information I2. Then, the display controller 15 causes the instruction device 2 to display again the trajectory information based on the above-mentioned calculation results. Furthermore, when the operation planner 17A receives an input signal S1 or the like indicating that the trajectories of the redisplayed robot hand 53 and the object 85 are approved, the operation planner 17A supplies the second operation sequence Srb based on the above-mentioned calculation results to the robot controller 18A.

[0154] In a second example of generating a second operation plan, when the trajectories of the modified robot hand 53 and the target object 85 are specified based on the operation of the instruction device 2, the correction receiving unit 16A supplies the operation planning unit 17A with trajectory correction information Ib including trajectory information of the modified robot hand 53 and the target object 85. In this case, the operation planning unit 17A determines whether the trajectory information of the modified robot hand 53 and the target object 85 specified by the trajectory correction information Ib satisfies the existing constraint condition (i.e., the constraint condition indicated by the constraint condition information I2), and if it does, generates a second operation sequence Srb based on the trajectory information and supplies the second operation sequence Srb to the robot control unit 18A.

[0155] According to these first and second examples, the operation planning unit 17A can preferably formulate a second operation plan that modifies the first operation plan so that the state of the object specified by the modification is realized.

[0156] 13 is merely an example, and the robot controller 1A may accept various corrections regarding the trajectories of the robot hand 53 and the object 85. For example, the robot controller 1A may accept, as a correction of the posture of the object 85 in the gripping state, a correction of tilting the object 85 from a vertical state to an oblique 45 degree angle midway, a correction of changing the orientation of the object 85 when it reaches the vicinity of the box 86 so that the object 85 can be easily put into the box 86, and the like.

[0157] In this way, the robot controller 1A according to the second embodiment can favorably accept modifications to the state of the target object, such as its position and posture, and modifications to the point at which the target object is grasped, and determine a second operation plan that reflects these modifications.

[0158] FIG. 14(A) is a diagram showing trajectory information before correction in the second specific example from a first viewpoint, and FIG. 14(B) is a diagram showing trajectory information before correction in the second specific example from a second viewpoint. Here, the second specific example is a specific example regarding a goal task of moving the object 93 to a position on the work table 79 behind the first obstacle 91 and the second obstacle 92, and the robot controller 1A displays the trajectory of the object 93 specified by the first motion plan determined by the motion planner 17A. The virtual objects 93Va to 93Vd are virtual objects showing the position and posture of the object 85 for each predetermined number of time steps based on the first motion plan. The virtual object 93Vd shows the position and posture of the object 93 (i.e., the object 93 at the goal position) when the goal task is accomplished.

[0159] 14(A) and 14(B), in the first operation plan, the trajectory of the object 93 is set so that the object 93 passes through the space between the first obstacle 91 and the second obstacle 92. On the other hand, in the case of such a trajectory, there is a possibility that the robot hand of the robot 5 (not shown) may come into contact with the first obstacle 91 or the second obstacle 92, and the operator judges that it is necessary to correct the trajectory of the object 93.

[0160] FIG. 15(A) is a diagram showing an outline of an operation related to the correction of trajectory information in the second concrete example from a first viewpoint, and FIG. 15(B) is a diagram showing an outline of an operation related to the correction of trajectory information in the second concrete example from a second viewpoint. In this case, the worker performs an operation on the instruction device 2 to correct the trajectory of the object 93 so that the object 93 does not pass through the space between the first obstacle 91 and the second obstacle 92 but passes beside the second obstacle 92. Specifically, the worker performs an operation by drag-and-drop operation or the like to place the virtual object 93Vb existing between the first obstacle 91 and the second obstacle 92 at a position beside the second obstacle 92. Then, the display control unit 15A newly generates and displays a virtual object 93Vy located beside the second obstacle 92 based on the input signal S1 generated by the above-mentioned operation. In this case, the worker adjusts the position of the object 93 as well as the attitude of the virtual object 93Vy so that the object 93 has a desired attitude.

[0161] Then, the correction receiving unit 16A supplies the trajectory correction information Ib including information on the position and posture of the virtual object 93Vy to the motion planning unit 17A. In this case, the motion planning unit 17A recognizes, for example, that the object 93 transitions to the state of the virtual object 93Vy based on the trajectory correction information Ib as an additional constraint condition. Then, the motion planning unit 17A performs the optimization process shown in formula (2) based on the additional constraint condition to determine the trajectory of the object 93 after the correction (and the trajectory of the robot 5), etc. In addition, when information on the scheduled operation time (i.e., the time step) corresponding to the virtual object 93Vb before the change is included in the trajectory correction information Ib, the motion planning unit 17A may set, as the above-mentioned additional constraint condition, that the object 93 transitions to the state of the virtual object 93Vy at the above-mentioned scheduled operation time.

[0162] Fig. 16(A) is a diagram showing trajectory information based on the second operation plan in the second specific example from a first viewpoint, and Fig. 16(B) is a diagram showing trajectory information based on the second operation plan in the second specific example from a second viewpoint. In Fig. 16(A) and Fig. 16(B), virtual objects 93Vx to 93Vz represent the positions and postures of target object 93 for each predetermined number of time steps based on the second operation plan.

[0163] 16(A) and 16(B), in this case, the display control unit 15A uses trajectory information based on the second operation plan regenerated by the operation planning unit 17A to display the transition of the object 93 reflecting the correction by virtual objects 93Vx to 93Vz and 93Vd. Note that, here, the trajectory of the object 93 is appropriately corrected so that the object 93 passes beside the second obstacle 92, since the state of the virtual object 93Vy is taken into consideration as a constraint condition (subgoal) in the second operation plan. Then, the robot controller 1A controls the robot 5 based on the second operation plan in this way, thereby allowing the robot 5 to appropriately complete the target task.

[0164] FIG. 17 is an example of a flowchart showing an outline of a robot control process executed by the robot controller 1 in the second embodiment.

[0165] First, the robot controller 1 acquires a sensor signal S4 from the sensor 7 (step S21). Then, the recognition result acquisition unit 14A of the robot controller 1A recognizes the state (including position and posture) and attributes of the object in the working space based on the acquired sensor signal S4 (step S22). Furthermore, the recognition result acquisition unit 14A generates a second recognition result Im2 by correcting the first recognition result Im1 based on the processing of the first embodiment. Note that in the second embodiment, the correction processing of the first recognition result Im1 based on the processing of the first embodiment is not essential processing.

[0166] Then, the motion planning unit 17A determines a first motion plan (step S23). Then, the display control unit 15A acquires trajectory information based on the first motion plan determined by the motion planning unit 17A, and causes the instruction device 2 to display the trajectory information (step S24). In this case, the display control unit 15A causes the instruction device 2 to display at least the trajectory information related to the target object.

[0167] Then, the correction receiving unit 16A determines whether or not the orbit information needs to be corrected (step S25). In this case, the correction receiving unit 16A receives, for example, an input specifying whether or not the orbit information needs to be corrected, and determines whether or not the correction is needed based on the received input.

[0168] Then, when the correction receiving unit 16A determines that the trajectory information needs to be corrected (step S25; Yes), it accepts the correction regarding the trajectory information (step S26). In this case, the correction receiving unit 16A accepts the correction based on any operation method using the input unit 24a, which is any user interface provided in the instruction device 2. Then, the motion planning unit 17A determines a second motion plan that reflects the accepted correction based on the trajectory correction information Ib generated by the correction receiving unit 16A (step S27). Then, the motion planning unit 17A determines whether the determined second motion plan satisfies the constraint condition (step S28). Then, if the second motion plan satisfies the constraint condition (step S28; Yes), the motion planning unit 17A proceeds to the process of step S29. On the other hand, if the second motion plan does not satisfy the constraint condition (step S28; No), the correction receiving unit 16A considers the previous correction to be invalid, and accepts the correction regarding the trajectory information again in step S26. If the second operation plan is determined by the operation planning unit 17A so as to satisfy the additional constraint condition in step S27, the second operation plan is deemed to satisfy the constraint condition in step S28.

[0169] Then, if the trajectory information does not need to be corrected (step S25; No), or if it is determined in step S28 that the constraint condition is satisfied (step S28; Yes), the robot control unit 18A controls the robot based on the second operation sequence Srb based on the second operation plan determined by the operation planning unit 17A (step S29). In this case, the robot control unit 18A sequentially supplies the control signal S3 generated based on the second operation sequence Srb to the robot 5, and controls the robot 5 to operate according to the generated second operation sequence Srb. Note that, if the trajectory information does not need to be corrected, the robot controller 1A executes the process of step S18 by regarding the first operation plan as the second operation plan.

[0170] In the second embodiment, instead of displaying the trajectory information by augmented reality, the robot controller 1A may display the trajectory information on the instruction device 2 by superimposing it on a CG (computer graphics) image or the like that typically represents the working space, and accept various corrections to the trajectory of the object or the robot 5. Even in this mode, the robot controller 1A can suitably accept corrections to the trajectory information made by the operator.

[0171] <Third embodiment> Fig. 18 shows a schematic configuration diagram of a control device 1X in the third embodiment. The control device 1X mainly includes a recognition result acquisition means 14X, a display control means 15X, and a correction receiving means 16X. The control device 1X may be composed of a plurality of devices. The control device 1X may be, for example, the robot controller 1 in the first embodiment or the robot controller 1A in the second embodiment.

[0172] The recognition result acquisition means 14X acquires the recognition result of an object related to a task executed by the robot. The "object related to the task" refers to any object related to the task executed by the robot, and includes a target object (workpiece) to be grasped or processed by the robot, other workpieces, and the robot. The recognition result acquisition means 14X may acquire the recognition result of the object by generating the recognition result based on information generated by a sensor that senses the environment in which the task is executed, or may acquire the recognition result of the object by receiving the recognition result from an external device that generates the recognition result. The former recognition result acquisition means 14X may be, for example, the recognition result acquisition unit 14 in the first embodiment or the recognition result acquisition unit 14A in the second embodiment.

[0173] The display control means 15X displays the information representing the recognition result so that it is visually recognized overlapping with a real landscape or an image of the landscape. The "landscape" here corresponds to the work space in which the task is executed. The display control means 15X may be a display device that performs display by itself, or may perform display by transmitting a display signal to an external display device. The display control means 15X may be, for example, the display control unit 15 in the first embodiment or the display control unit 15A in the second embodiment.

[0174] The correction receiving means 16X receives a correction of the recognition result based on an external input. The correction receiving means 16X can be, for example, the correction receiving unit 16 in the first embodiment or the correction receiving unit 16A in the second embodiment.

[0175] 19 is an example of a flowchart in the third embodiment. The recognition result acquisition means 14X acquires the recognition result of an object related to a task to be executed by the robot (step S31). The display control means 15X displays information representing the recognition result so as to be visually recognized overlapping with a real landscape or an image of the landscape (step S32). The correction receiving means 16X receives a correction of the recognition result based on an external input (step S33).

[0176] According to the third embodiment, the control device 1X can suitably accept a correction of the object recognition result related to the task executed by the robot, and can obtain an accurate recognition result based on the correction.

[0177] <Fourth embodiment> Fig. 20 shows a schematic configuration diagram of a control device 1Y in the fourth embodiment. The control device 1Y mainly includes an operation planning means 17Y, a display control means 15Y, and a correction receiving means 16Y. The control device 1Y may be composed of a plurality of devices. The control device 1X may be, for example, the robot controller 1 in the first embodiment.

[0178] The motion planning means 17Y determines a first motion plan of the robot that executes a task using an object. The motion planning means 17Y also determines a second motion plan of the robot based on a correction received by a correction receiving means 16Y, which will be described later. The motion planning means 17Y can be, for example, the motion planning unit 17A in the second embodiment.

[0179] The display control means 15Y displays trajectory information regarding the trajectory of the object based on the first operation plan. The display control means 15Y may be a display device that performs display by itself, or may perform display by transmitting a display signal to an external display device. The display control means 15Y may be, for example, the display control unit 15A in the second embodiment.

[0180] The correction receiving means 16Y receives a correction related to the trajectory information based on an external input. The correction receiving means 16Y can be, for example, the correction receiving unit 16A in the second embodiment.

[0181] 21 is an example of a flowchart in the fourth embodiment. The motion planning means 17Y determines a motion plan for the robot that executes a task using an object (step S41). The display control means 15Y displays trajectory information on the trajectory of the object based on the motion plan (step S42). The correction receiving means 16Y receives a correction on the trajectory information based on an external input (step S43). Then, the motion planning means 17Y determines a second motion plan for the robot based on the correction received by the correction receiving means 16Y (step S44).

[0182] According to the fourth embodiment, the control device 1X can display trajectory information regarding the trajectory of an object based on a determined motion plan for the robot, and can suitably accept corrections thereto and reflect them in the motion plan.

[0183] In each of the above-described embodiments, the program can be stored using various types of non-transitory computer readable media and can be supplied to a computer, such as a processor. The non-transitory computer readable medium includes various types of tangible storage media. Examples of the non-transitory computer readable medium include magnetic storage media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROM (Read Only Memory), CD-R, CD-R / W, and semiconductor memory (e.g., mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, and RAM (Random Access Memory)). The program may also be supplied to a computer by various types of transitory computer readable media. Examples of the transitory computer readable medium include electric signals, optical signals, and electromagnetic waves. The transitory computer readable medium can supply the program to a computer via a wired communication path such as an electric wire and an optical fiber, or via a wireless communication path.

[0184] In addition, some or all of the above embodiments may be described as, but are not limited to, the following supplementary notes.

[0185] [Appendix 1] A recognition result acquisition means for acquiring an object recognition result related to a task to be executed by the robot; a display control means for displaying information representing the recognition result so as to be visually recognized by superimposing the information on a real landscape or an image of the landscape; a correction receiving means for receiving a correction of the recognition result based on an external input; A control device having the above configuration. [Appendix 2] The control device according to claim 1, wherein the display control means displays an object that imitates the object based on the recognition result so that the object is visually superimposed on the object in the scenery or the image. [Appendix 3] The control device according to claim 2, wherein the correction receiving means receives the correction regarding at least one of a position or an orientation of the object based on the external input that changes at least one of a position or an orientation of the object. [Appendix 4] The control device according to claim 2 or 2, wherein the correction accepting means accepts the correction indicating deletion of a recognition result of an object corresponding to the object, based on the external input instructing deletion of the object. [Appendix 5] The control device according to any one of appendices 2 to 4, wherein the correction accepting means accepts the correction indicating the addition of a recognition result of an object corresponding to the object based on the external input instructing the addition of the object. [Appendix 6] the display control means displays information about an attribute of the object included in the recognition result in association with the object in the scenery or the image; 3. The control device according to claim 1, wherein the modification receiving means receives the modification regarding the attribute based on the external input instructing a change to the attribute. [Appendix 7] the recognition result includes a confidence level indicating a degree of confidence regarding the correctness of the recognition of the object; The control device according to any one of claims 1 to 6, wherein the correction receiving means determines whether or not the correction is necessary based on the degree of confidence. [Appendix 8] the recognition result includes a confidence level indicating a degree of confidence regarding the correctness of the recognition of the object; The control device according to any one of claims 1 to 7, wherein the display control means determines a display mode of information represented by the recognition result based on the degree of confidence. [Appendix 9] The control device according to claim 8, wherein the display control means highlights information represented by the recognition result for which the degree of confidence is less than a threshold value. [Appendix 10] The control device according to any one of claims 1 to 9, further comprising an action planning means for determining an action plan for the robot based on a second recognition result in which the recognition result is modified. [Appendix 11] The motion planning means a logical expression conversion means for converting a task to be executed by the robot into a logical expression based on temporal logic based on the second recognition result; a time step logical expression generating means for generating a time step logical expression, which is a logical expression representing a state for each time step for executing the task, from the logical expression; a subtask sequence generation means for generating a sequence of subtasks to be executed by the robot based on the time step logical expression; 11. The control device of claim 10, comprising: [Appendix 12] The computer Obtaining object recognition results related to a task to be performed by the robot; displaying information representing the recognition result so as to be visually recognized overlapping with a real landscape or an image of the landscape; Accepting a correction of the recognition result based on an external input; Control methods. [Appendix 13] Obtaining object recognition results related to a task to be performed by the robot; displaying information representing the recognition result so as to be visually recognized overlapping with a real landscape or an image of the landscape; A storage medium storing a program for causing a computer to execute a process for accepting corrections to the recognition result based on external input.

[0186] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-mentioned embodiments. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. In other words, the present invention naturally includes various modifications and corrections that a person skilled in the art could make in accordance with the entire disclosure, including the claims, and the technical ideas. In addition, the disclosures of the above cited patent documents and the like are incorporated herein by reference. [Explanation of symbols]

[0187] 1. 1A Robot Controller 1X, 1Y control device 2 Indicating device 4 Storage device 5. Robot 7 Sensors 41 Application information storage unit 100 Robot Control System

Claims

1. A recognition result acquisition means for acquiring an object recognition result related to a task to be executed by the robot; a display control means for displaying information representing the recognition result so as to be visually recognized by superimposing the information on a real landscape or an image of the landscape; and a correction receiving means for receiving a correction of the recognition result based on an external input, the recognition result includes a confidence level indicating a degree of confidence regarding the correctness of the recognition of the object; The correction receiving means determines whether or not the correction is necessary based on the degree of confidence.

2. The control device according to claim 1 , wherein the display control means displays an object that imitates the object based on the recognition result so as to be visually recognized superimposed on the object in the scenery or the image.

3. The control device according to claim 2 , wherein the modification receiving means receives the modification regarding at least one of a position or a posture of the object based on the external input that changes at least one of a position or a posture of the object.

4. The control device according to claim 2 , wherein the correction accepting means accepts the correction indicating deletion of a recognition result of an object corresponding to the object, based on the external input instructing deletion of the object.

5. The control device according to any one of claims 2 to 4, wherein the correction accepting means accepts the correction indicating the addition of a recognition result of an object corresponding to the object based on the external input instructing the addition of the object.

6. the display control means displays information about an attribute of the object included in the recognition result in association with the object in the scenery or the image; The control device according to claim 1 , wherein the modification accepting means accepts the modification regarding the attribute based on the external input instructing a change to the attribute.

7. The control device according to claim 1 , wherein the display control means determines a display mode of the information represented by the recognition result based on the degree of confidence.

8. A recognition result acquisition means for acquiring an object recognition result related to a task to be executed by the robot; a display control means for displaying information representing the recognition result so as to be visually recognized by superimposing the information on a real landscape or an image of the landscape; and a correction receiving means for receiving a correction of the recognition result based on an external input, the recognition result includes a confidence level indicating a degree of confidence regarding the correctness of the recognition of the object; The display control means highlights information representing the recognition result for which the degree of confidence is less than a threshold.

9. The computer Obtaining object recognition results related to a task to be performed by the robot; displaying information representing the recognition result so as to be visually recognized overlapping with a real landscape or an image of the landscape; accepting a correction of the recognition result based on an external input; the recognition result includes a confidence level indicating a degree of confidence regarding the correctness of the recognition of the object; determining whether the correction is necessary based on the degree of confidence; Control methods.

10. Obtaining object recognition results related to a task to be performed by the robot; displaying information representing the recognition result so as to be visually recognized overlapping with a real landscape or an image of the landscape; accepting a correction of the recognition result based on an external input; the recognition result includes a confidence level indicating a degree of confidence regarding the correctness of the recognition of the object; A program that causes a computer to execute a process of determining whether or not the correction is necessary based on the degree of confidence.

Images