Control device, control method, and program

Abstract

This control device 1Y primarily has an operation planning means 17Y, a display controlling means 15Y, and a correction receiving means 16Y. The operation planning means 17Y determines an operation plan of a robot that executes a task using an object. The display controlling means 15Y displays track information relating to the track of the object based on the operation plan. The correction receiving means 16Y receives corrections relating to the track information based on external input. The operation planning means 17Y determines a second operation plan of the robot on the basis of the corrections received by the correction receiving means 16Y.

Description

[Technical Field]

[0001] The present disclosure relates to the technical fields of a control device, a control method, and a storage medium for a robot that performs a task. [Background technology]

[0002] A robot system has been proposed that uses sensors to recognize the robot's environment and causes the robot to perform a task based on the recognized environment. For example, Patent Document 1 discloses a robot system that issues operation commands to the robot based on the detection results of a surrounding environment detection sensor and a determined behavior plan for the robot. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2020-046779 Summary of the Invention [Problem to be solved by the invention]

[0004] When a robot's motion (action) plan is automatically generated from a given task, the generated motion plan does not necessarily execute the task as intended by the user. Therefore, it is convenient for the user to be able to check the generated motion plan as appropriate and make corrections to the motion plan.

[0005] In view of the above-mentioned problems, one of the objects of the present disclosure is to provide a control device, a control method, and a storage medium that are capable of suitably correcting an operation plan. [Means for solving the problem]

[0006] One aspect of the control device is a motion planning means for determining a first motion plan for a robot that executes a task using an object; a three-dimensional position of the object, the movement of which is caused by the robot based on the first motion plan, at each predetermined time interval during the movement; and a display control means for displaying trajectory information showing a plurality of objects of the object, the trajectory information representing a state of the object including at least an attitude; The corrections to the orbital information are Select one of the Operation via external input and an operation by external input that modifies at least one of the three-dimensional position or orientation of the object selected by said operation. and a correction accepting means for accepting the correction by The motion planning means is a control device that determines a second motion plan for the robot based on the correction.

[0007] One aspect of the control method includes: The computer determining a first motion plan for the robot to perform a task using the object; a three-dimensional position of the object, the movement of which is caused by the robot based on the first motion plan, at each predetermined time interval during the movement; and displaying trajectory information showing a plurality of objects of the object representing a state of the object including at least a pose; The corrections to the orbital information are Select one of the Operation via external input and an operation by external input that modifies at least one of the three-dimensional position or orientation of the object selected by said operation. Accepted by determining a second motion plan for the robot based on the modification; It is a control method.

[0008] One aspect of the program is determining a first motion plan for the robot to perform a task using the object; a three-dimensional position of the object, the movement of which is caused by the robot based on the first motion plan, at each predetermined time interval during the movement; and displaying trajectory information showing a plurality of objects of the object, which represents a state of the object including at least a pose; The corrections to the orbit information are Select one of the Operation via external input and an operation by external input that modifies at least one of the three-dimensional position or orientation of the object selected by said operation. Accepted by The program causes a computer to execute a process of determining a second operation plan for the robot based on the correction. [Effects of the Invention]

[0009] The motion plan can be suitably modified. [Brief explanation of the drawings]

[0010] [Figure 1] 1 shows the configuration of a robot control system in a first embodiment. [Figure 2] (A) shows the hardware configuration of the robot controller. (B) shows the hardware configuration of the instruction device. [Figure 3] 10 shows an example of a data structure of application information. [Figure 4] 2 is an example of a functional block of a robot controller. [Figure 5] (A) shows a first embodiment of the modification. (B) shows a second embodiment of the modification. (C) shows a third embodiment of the modification. (D) shows a fourth embodiment of the modification. [Figure 6] When pick-and-place is the target task, the state of the workspace before correction as visually recognized by the worker is shown. [Figure 7] 10 shows the state of the workspace as viewed by the worker after modifications have been made to the virtual object. [Figure 8] 3 is an example of a functional block showing the functional configuration of an action planning unit. [Figure 9] 1 shows an overhead view of the workspace when the target task is pick-and-place. [Figure 10] 1 is an example of a flowchart illustrating an outline of a robot control process executed by a robot controller in the first embodiment. [Figure 11] 10 is an example of a functional block of a robot controller according to a second embodiment. [Figure 12] FIG. 2 is a diagram showing trajectory information in a first specific example. [Figure 13]10A and 10B are diagrams showing, in the first specific example, the corrected trajectories (corrected trajectories) of the robot hand and the object corrected based on inputs for correcting the trajectories of the robot hand and the object, etc. FIG. [Figure 14] 10A is a diagram showing the orbit information before correction in the second specific example from a first viewpoint, and FIG. 10B is a diagram showing the orbit information before correction in the second specific example from a second viewpoint. [Figure 15] 10A is a diagram showing an outline of an operation related to the correction of orbit information in the second specific example from a first perspective, and FIG. 10B is a diagram showing an outline of an operation related to the correction of orbit information in the second specific example from a second perspective. [Figure 16] 10A is a diagram showing the orbit information after correction in the second specific example from a first viewpoint, and FIG. 10B is a diagram showing the orbit information after correction in the second specific example from a second viewpoint. [Figure 17] 10 is an example of a flowchart showing an outline of a robot control process executed by a robot controller in the second embodiment. [Figure 18] FIG. 10 shows a schematic configuration diagram of a control device according to a third embodiment. [Figure 19] 10 is an example of a flowchart executed by the control device in the third embodiment. [Figure 20] FIG. 10 shows a schematic configuration diagram of a control device according to a fourth embodiment. [Figure 21] 13 is an example of a flowchart executed by the control device in the fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of a control device, a control method, and a storage medium will be described with reference to the drawings.

[0012] First Embodiment (1) System Configuration 1 shows the configuration of a robot control system 100 according to the first embodiment. The robot control system 100 mainly includes a robot controller 1, an instruction device 2, a storage device 4, a robot 5, and a sensor (detection device) 7.

[0013] When a task (also called a "target task") to be executed by the robot 5 is specified, the robot controller 1 converts the target task into a sequence of time steps (time increments) of simple tasks that the robot 5 can accept, and controls the robot 5 based on the generated sequence.

[0014] The robot controller 1 also communicates data with the instruction device 2, storage device 4, robot 5, and sensor 7 via a communication network or by direct wireless or wired communication. For example, the robot controller 1 receives an input signal "S1" related to a motion plan for the robot 5 from the instruction device 2. The robot controller 1 also transmits a display control signal "S2" to the instruction device 2, causing the instruction device 2 to execute a predetermined display or sound output. The robot controller 1 also transmits a control signal "S3" related to the control of the robot 5 to the robot 5. The robot controller 1 also receives a sensor signal "S4" from the sensor 7.

[0015] The instruction device 2 is a device that receives instructions from a worker regarding a motion plan for the robot 5. The instruction device 2 performs a predetermined display or sound output based on a display control signal S2 supplied from the robot controller 1, and supplies an input signal S1 generated based on an input from the worker to the robot controller 1. The instruction device 2 may be a tablet terminal equipped with an input unit and a display unit, a stationary personal computer, or any terminal used for augmented reality.

[0016] The storage device 4 has an application information storage unit 41. The application information storage unit 41 stores application information required to generate an operation sequence, which is a sequence to be executed by the robot 5, from a target task. Details of the application information will be described later with reference to FIG. 3. The storage device 4 may be an external storage device such as a hard disk connected to or built into the robot controller 1, or may be a storage medium such as a flash memory. The storage device 4 may also be a server device that performs data communication with the robot controller 1 via a communication network. In this case, the storage device 4 may be composed of multiple server devices.

[0017] The robot 5 performs work related to the target task based on a control signal S3 supplied from the robot controller 1. The robot 5 is a robot that operates, for example, in various factories such as assembly factories and food factories, or in logistics sites. The robot 5 may be a vertical articulated robot, a horizontal articulated robot, or any other type of robot. The robot 5 may supply a status signal indicating the status of the robot 5 to the robot controller 1. This status signal may be an output signal from a sensor that detects the status (position, angle, etc.) of the entire robot 5 or a specific part such as a joint, or may be a signal generated by the control unit of the robot 5 that indicates the progress of the operation sequence of the robot 5.

[0018] The sensor 7 is one or more sensors such as a camera, a range sensor, a sonar, or a combination thereof that detects conditions within the workspace in which the target task is performed. For example, the sensor 7 includes at least one camera that captures an image of the workspace of the robot 5. The sensor 7 supplies the generated sensor signal S4 to the robot controller 1. The sensor 7 may be a self-propelled or flying sensor (including a drone) that moves within the workspace. The sensor 7 may also include a sensor provided on the robot 5 and a sensor provided on another object in the workspace. The sensor 7 may also include a sensor that detects sound within the workspace. In this way, the sensor 7 may include various sensors that detect conditions within the workspace and may include sensors provided at any location.

[0019] The configuration of the robot control system 100 shown in FIG. 1 is merely an example, and various modifications may be made to the configuration. For example, there may be multiple robots 5, or multiple control objects, such as robot arms, each of which operates independently. Even in these cases, the robot controller 1 generates an operation sequence to be executed for each robot 5 or each control object based on a target task, and transmits a control signal S3 based on the operation sequence to the corresponding robot 5. The robot 5 may also perform a collaborative operation with other robots, workers, or machine tools operating in the workspace. The sensor 7 may also be part of the robot 5. The instruction device 2 may also be configured as the same device as the robot controller 1. The robot controller 1 may also be configured as multiple devices. In this case, the multiple devices that make up the robot controller 1 exchange information necessary to execute pre-assigned processes between these multiple devices. The robot controller 1 and the robot 5 may also be configured as an integrated unit.

[0020] (2) Hardware Configuration 2(A) shows the hardware configuration of the robot controller 1. The robot controller 1 includes, as hardware, a processor 11, a memory 12, and an interface 13. The processor 11, the memory 12, and the interface 13 are connected via a data bus 10.

[0021] The processor 11 functions as a controller (arithmetic device) that performs overall control of the robot controller 1 by executing a program stored in the memory 12. The processor 11 is, for example, a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a TPU (Tensor Processing Unit). The processor 11 may be composed of multiple processors. The processor 11 is an example of a computer.

[0022] The memory 12 is composed of various types of volatile and non-volatile memory, such as RAM (Random Access Memory), ROM (Read Only Memory), and flash memory. The memory 12 also stores programs for executing processes performed by the robot controller 1. Some of the information stored in the memory 12 may be stored in one or more external storage devices (e.g., storage device 4) that can communicate with the robot controller 1, or may be stored in a storage medium that is detachable from the robot controller 1.

[0023] The interface 13 is an interface for electrically connecting the robot controller 1 to other devices. These interfaces may be wireless interfaces such as network adapters for wirelessly transmitting and receiving data to and from other devices, or may be hardware interfaces for connecting to other devices via cables or the like.

[0024] The hardware configuration of the robot controller 1 is not limited to the configuration shown in Fig. 2(A). For example, the robot controller 1 may be connected to or have built-in at least one of a display device, an input device, and a sound output device. The robot controller 1 may also be configured to include at least one of an instruction device 2 and a storage device 4.

[0025] 2(B) shows the hardware configuration of the instruction device 2. The instruction device 2 includes, as hardware, a processor 21, a memory 22, an interface 23, an input unit 24a, a display unit 24b, and a sound output unit 24c. The processor 21, the memory 22, and the interface 23 are connected via a data bus 20. The interface 23 is also connected to the input unit 24a, the display unit 24b, and the sound output unit 24c.

[0026] The processor 21 executes a predetermined process by executing a program stored in the memory 22. The processor 21 is a processor such as a CPU or a GPU. The processor 21 receives a signal generated by the input unit 24a via the interface 23, thereby generating an input signal S1, and transmits the input signal S1 to the robot controller 1 via the interface 23. The processor 21 also controls at least one of the display unit 24b and the sound output unit 24c via the interface 23, based on a display control signal S2 received from the robot controller 1 via the interface 23.

[0027] The memory 22 is configured by various types of volatile and non-volatile memories such as RAM, ROM, flash memory, etc. The memory 22 also stores programs for executing the processes executed by the instruction device 2.

[0028] The interface 23 is an interface for electrically connecting the instruction device 2 with other devices. These interfaces may be wireless interfaces such as network adapters for wirelessly transmitting and receiving data to and from other devices, or may be hardware interfaces for connecting to other devices via cables or the like. The interface 23 also performs interface operations for the input unit 24a, display unit 24b, and sound output unit 24c.

[0029] The input unit 24a is an interface that accepts user input, and may be, for example, a touch panel, buttons, a keyboard, or a voice input device. The input unit 24a may also include various input devices (such as an operation controller) used in virtual reality. In this case, the input unit 24a may be, for example, various sensors (including, for example, a camera, a wearable sensor, etc.) used in motion capture, or, if the display unit 24b is an eyeglass-type terminal that realizes augmented reality, an operation controller that is paired with the terminal.

[0030] The display unit 24b performs augmented reality display under the control of the processor 21. In a first example, the display unit 24b is a glasses-type terminal that displays information about the state of objects in a scene (here, the work space) superimposed on the scene viewed by the worker. In a second example, the display unit 24b is a display, projector, or the like that displays information about objects superimposed on an image (also called a "real-life image") of a scene (here, the work space). The real-life image is supplied by the sensor 7. The sound output unit 24c is, for example, a speaker, and outputs sound under the control of the processor 21.

[0031] 2(B) 。 For example, at least one of the input unit 24a, the display unit 24b, and the sound output unit 24c may be configured as a separate device electrically connected to the instruction device 2. The instruction device 2 may be connected to various devices such as a camera, or may have these built-in.

[0032] (3) Application Information Next, the data structure of the application information stored in the application information storage unit 41 will be described.

[0033] 3 shows an example of the data structure of application information. As shown in FIG. 3, the application information includes abstract state designation information I1, constraint condition information I2, operational limit information I3, subtask information I4, abstract model information I5, and object model information I6.

[0034] The abstract state designation information I1 is information that designates the abstract state that needs to be defined when generating an action sequence. This abstract state is the abstract state of an object in the workspace, and is defined as a proposition used in the goal logical formula, which will be described later. For example, the abstract state designation information I1 designates the abstract state that needs to be defined for each type of target task.

[0035] The constraint information I2 is information indicating constraints for executing a target task. For example, if the target task is pick-and-place, the constraint information I2 indicates constraints such as that the robot 5 (robot arm) must not come into contact with obstacles, and that the robots 5 (robot arms) must not come into contact with each other. Note that the constraint information I2 may be information recording constraints appropriate for each type of target task.

[0036] The motion limit information I3 indicates information relating to the motion limits of the robot 5 controlled by the robot controller 1. The motion limit information I3 is, for example, information that defines the upper limit of the speed, acceleration, or angular velocity of the robot 5. Note that the motion limit information I3 may also be information that defines the motion limits for each movable part or joint of the robot 5.

[0037] The subtask information I4 indicates information about subtasks that are components of an operation sequence. A "subtask" is a task obtained by breaking down a target task into units that the robot 5 can accept, and refers to the subdivided operations of the robot 5. For example, if the target task is pick-and-place, the subtask information I4 defines reaching, which is the movement of the robot arm of the robot 5, and grasping, which is the gripping by the robot arm, as subtasks. The subtask information I4 may indicate information about subtasks that can be used for each type of target task. Note that the subtask information I4 may also include information about subtasks that require operation commands from external input. In this case, the subtask information I4 related to an external input-type subtask includes, for example, information identifying it as an external input-type subtask (e.g., flag information) and information indicating the operation of the robot 5 in that external input-type subtask.

[0038] The abstract model information I5 is information about an abstract model that abstracts the dynamics in the workspace. For example, as will be described later, the abstract model is represented by a model that abstracts real-world dynamics using a hybrid system. The abstract model information I5 includes information indicating the conditions for switching dynamics in the hybrid system described above. For example, in the case of pick-and-place, in which the robot 5 grasps an object (also called an "object") that is the work target and moves it to a predetermined position, the switching condition corresponds to a condition that the object cannot be moved unless it is grasped by the robot 5. The abstract model information I5 includes information about an abstract model appropriate for each type of target task.

[0039] The object model information I6 is information about the object model of each object in the workspace to be recognized from the sensor signal S4 generated by the sensor 7. The above-mentioned objects include, for example, the robot 5, obstacles, tools and other objects handled by the robot 5, and workpieces other than the robot 5. The object model information I6 includes, for example, information necessary for the robot controller 1 to recognize the type, position, posture, and currently executing operation of each of the above-mentioned objects, as well as three-dimensional shape information such as CAD (Computer Aided Design) data for recognizing the three-dimensional shape of each object. The former information includes parameters of an inference device obtained by training a learning model in machine learning such as a neural network. For example, when an image is input, this inference device is trained in advance to output the type, position, posture, and other information of the object that is the subject of the image.

[0040] In addition to the above-mentioned information, the application information storage unit 41 may store various information related to the generation process of the operation sequence and the generation process of the display control signal S2.

[0041] (4) Processing Overview Next, an overview of the processing of the robot controller 1 in the first embodiment will be described. In summary, the robot controller 1 uses augmented reality to display the recognition results of objects in the workspace recognized based on the sensor signal S4 on the instruction device 2, and accepts input for correction of the recognition results. As a result, even if an erroneous recognition of an object in the workspace occurs, the robot controller 1 appropriately corrects the part where the erroneous recognition occurred based on user input, thereby enabling the robot 5 to accurately formulate an operation plan and execute a target task.

[0042] Fig. 4 is an example of functional blocks showing an overview of the processing of the robot controller 1. Functionally, the processor 11 of the robot controller 1 has a recognition result acquisition unit 14, a display control unit 15, a correction acceptance unit 16, an action planning unit 17, and a robot control unit 18. Note that Fig. 4 shows an example of data exchanged between each block, but is not limited to this. The same applies to other functional block diagrams described later.

[0043] The recognition result acquisition unit 14 recognizes the state, attributes, etc. of objects in the workspace based on the sensor signal S4, etc., and supplies information representing the recognition result (also referred to as the "first recognition result Im1") to the display control unit 15. In this case, for example, the recognition result acquisition unit 14 refers to the abstract state designation information I1 to recognize the state, attributes, etc. of objects in the workspace that need to be taken into consideration when executing the target task. The objects in the workspace include, for example, the robot 5, objects such as tools or parts handled by the robot 5, obstacles, and other workers (people performing work other than the robot 5 or other objects). For example, the recognition result acquisition unit 14 refers to the object model information I6 and analyzes the sensor signal S4 using any technology that recognizes the environment of the workspace, thereby generating the first recognition result Im1. Examples of the technology for recognizing the environment include image processing technology, image recognition technology (including object recognition using an AR marker), voice recognition technology, and technology using RFID (Radio Frequency Identifier).

[0044] In this embodiment, the recognition result acquisition unit 14 recognizes at least the position, orientation, and attributes of an object. The attributes are, for example, the type of object, and the types of objects recognized by the recognition result acquisition unit 14 are classified at a granularity according to the type of target task to be executed. For example, if the target task is pick-and-place, the objects are classified into "obstacles," "objects to be grasped," and the like. The recognition result acquisition unit 14 supplies the generated first recognition result Im1 to the display control unit 15. Note that the first recognition result Im1 is not limited to information representing the position, orientation, and type of the object, and may also include information regarding various states or attributes (e.g., the size and shape of the object) recognized by the recognition result acquisition unit 14.

[0045] Furthermore, when the recognition result acquisition unit 14 receives recognition correction information “Ia” indicating correction details of the first recognition result Im1 from the correction receiving unit 16, the recognition result acquisition unit 14 generates information (also referred to as “second recognition result Im2”) in which the recognition correction information Ia is reflected in the first recognition result Im1. Here, the recognition correction information Ia is, for example, information indicating whether correction is necessary, and if correction is necessary, the object to be corrected, the index to be corrected, and the amount of correction. Note that the “object to be corrected” is an object whose recognition result needs to be corrected, and the “index to be corrected” corresponds to an index related to position (e.g., coordinate values ​​for each coordinate axis), an index related to posture (e.g., Euler angles), an index representing an attribute, etc. Then, the recognition result acquisition unit 14 supplies the second recognition result Im2 reflecting the recognition correction information Ia to the action planning unit 17. Note that when the recognition correction information Ia indicates that no correction is necessary, the second recognition result Im2 is identical to the first recognition result Im1 that the recognition result acquisition unit 14 initially generated based on the sensor signal S4.

[0046] The display control unit 15 generates a display control signal S2 for displaying predetermined information or outputting sound on the instruction device 2 used by the worker, and transmits the display control signal S2 to the instruction device 2 via the interface 13. In this embodiment, the display control unit 15 generates objects (also referred to as "virtual objects") virtually representing each object based on the recognition result of the object in the workspace indicated by the first recognition result Im1. The display control unit 15 then generates the display control signal S2 for controlling the display of the instruction device 2 so that each virtual object is visually recognized by the worker as being superimposed on a corresponding object in a real landscape or a live-action image. The display control unit 15 generates this virtual object based on, for example, the type of object indicated by the first recognition result Im1 and the three-dimensional shape information for each type of object included in the object model information I6. In another example, the display control unit 15 generates a virtual object by combining primitive shapes (pre-registered polygons) according to the shape of the object indicated by the first recognition result Im1.

[0047] The correction accepting unit 16 accepts corrections to the first recognition result Im1 made by an operator using the instruction device 2. When the correction operation is completed, the correction accepting unit 16 generates recognition correction information Ia indicating the corrections to the first recognition result Im1. In this case, the correction accepting unit 16 receives an input signal S1 generated by the instruction device 2 via the interface 13 during display control of the object recognition result using augmented reality, and supplies the recognition correction information Ia generated based on the input signal S1 to the recognition result acquiring unit 14. Before the correction is finalized, the correction accepting unit 16 supplies an instruction signal, such as a correction to the display position of the virtual object based on the input signal S1 supplied from the instruction device 2, to the display control unit 15, and the display control unit 15 supplies a display control signal S2 reflecting the correction based on the instruction signal to the instruction device 2. As a result, the instruction device 2 displays the virtual object in a manner that immediately reflects the operator's operation.

[0048] The motion planning unit 17 determines a motion plan for the robot 5 based on the second recognition result Im2 supplied from the recognition result acquisition unit 14 and the application information stored in the storage device 4. In this case, the motion planning unit 17 generates a motion sequence "Sr", which is a sequence of subtasks (subtask sequence) that the robot 5 should execute to achieve the target task. The motion sequence Sr defines a series of motions of the robot 5 and includes information indicating the execution order and execution timing of each subtask. The motion planning unit 17 supplies the generated motion sequence Sr to the robot control unit 18.

[0049] The robot control unit 18 controls the operation of the robot 5 by supplying a control signal S3 to the robot 5 via the interface 13. The robot control unit 18 controls the robot 5 to execute each subtask constituting the operation sequence Sr at a predetermined execution timing (time step) based on the operation sequence Sr supplied from the operation planning unit 17. Specifically, the robot control unit 18 transmits the control signal S3 to the robot 5 to execute position control or torque control of the joints of the robot 5 to realize the operation sequence Sr.

[0050] The robot 5 may have a function equivalent to the robot control unit 18 instead of the robot controller 1. In this case, the robot 5 operates based on the operation sequence Sr generated by the operation planning unit 17.

[0051] Here, each of the components of the recognition result acquisition unit 14, the display control unit 15, the correction acceptance unit 16, the action planning unit 17, and the robot control unit 18 can be realized, for example, by the processor 11 executing a program. Alternatively, each component may be realized by recording the necessary program in an arbitrary non-volatile storage medium and installing it as needed. Note that at least a portion of these components may not necessarily be realized by software programs, but may be realized by any combination of hardware, firmware, and software. Also, at least a portion of these components may be realized using a user-programmable integrated circuit, such as an FPGA (Field-Programmable Gate Array) or a microcontroller. In this case, the integrated circuit may be used to realize a program consisting of the above components. Also, at least a portion of each component may be configured by an ASSP (Application Specific Standard Produce), an ASIC (Application Specific Integrated Circuit), or a quantum computer control chip. In this way, each component may be realized by various hardware. The same applies to other embodiments described below. Furthermore, each of these components may be realized by collaboration between multiple computers, for example, using cloud computing technology.

[0052] (5) Generate correction information Next, a specific description will be given of a method for generating recognition correction information Ia based on the control of display control unit 15 and correction receiving unit 16. Display control unit 15 causes instruction device 2 to display a virtual object of the recognized object so as to be superimposed on an actual object (real object) in a landscape or live-action image visually recognized by the worker, according to the position and posture of the object. Then, correction receiving unit 16 receives an operation to correct the position and posture of the virtual object so that the real object and the virtual object match when there is a difference between the real object and the virtual object in the landscape or live-action image.

[0053] First, the manner in which each virtual object is modified will be described. Figures 5(A) to 5(D) show the manners (first to fourth manners) of modifications received by the modification receiving unit 16. In Figures 5(A) to 5(D), the left side of the arrows shows how the real object and virtual object appear before modification, and the right side of the arrows shows how the real object and virtual object appear after modification. Here, cylindrical real objects are indicated by solid lines, and virtual objects are indicated by dashed lines.

[0054] In the first mode shown in FIG. 5A, before correction, the position and orientation of the real object and the virtual object are visually perceived as being misaligned. Therefore, in this case, the operator performs an operation on the input unit 24a of the instruction device 2 to correct the position and orientation (roll, pitch, yaw) so that the virtual object overlaps the real object. After correction, the position and orientation of the virtual object are appropriately changed based on the input signal S1 generated by the above-mentioned operation to correct the position and orientation. Then, the correction receiving unit 16 generates recognition correction information Ia that instructs correction of the position and orientation of the object corresponding to the target virtual object, and supplies the recognition correction information Ia to the recognition result acquisition unit 14. As a result, the correction of the position and orientation of the virtual object is reflected in the second recognition result Im2 as a correction of the recognition result of the position and orientation of the corresponding object.

[0055] In the second mode shown in FIG. 5B , before correction, the recognition result acquisition unit 14 was unable to recognize the presence of the target object based on the sensor signal S4, and therefore no virtual object for the target object was displayed. Therefore, in this case, the operator performs an operation on the input unit 24a of the instruction device 2 to instruct the generation of a virtual object for the target object. In this case, the operator may directly specify attributes, such as the position, orientation, and type, of the object for which a virtual object is to be generated, or may perform an operation to specify a location on the real object that was not recognized and instruct the re-execution of object recognition processing centered on that location. After correction, based on the input signal S1 generated by the above operation, a virtual object for the target object is appropriately generated with a position and orientation consistent with the real object. The correction receiving unit 16 then generates recognition correction information Ia indicating the addition of a recognition result for an object corresponding to the generated virtual object and supplies the recognition correction information Ia to the recognition result acquisition unit 14. As a result, the addition of the virtual object is reflected in the second recognition result Im2 as the addition of the recognition result for the corresponding object.

[0056] In a third mode shown in FIG. 5(C), a virtual object for an object that does not actually exist is generated in the pre-correction state due to, for example, erroneous object recognition by the recognition result acquisition unit 14. In this case, the operator performs an operation on the input unit 24Aa of the instruction device 2 to instruct deletion of the target virtual object. Then, in the post-correction state, the virtual object generated due to the erroneous object recognition is appropriately deleted based on the input signal S1 generated by the above-described operation. Then, the correction receiving unit 16 generates recognition correction information Ia instructing deletion of the recognition result of the object corresponding to the target virtual object, and supplies the recognition correction information Ia to the recognition result acquisition unit 14. As a result, the deletion of the virtual object is reflected in the second recognition result Im2 as deletion of the recognition result of the corresponding object.

[0057] In the fourth mode shown in FIG. 5(D), due to an error in the object attribute (here, type) recognition process by the recognition result acquisition unit 14, a virtual object with an attribute "object to be grasped," which differs from the target object's original attribute "obstacle," is generated in the pre-correction state. In this case, the operator performs an operation on the input unit 24Aa of the instruction device 2 to instruct correction of the attribute of the target virtual object. Then, in the post-correction state, the attribute of the virtual object is appropriately corrected based on the input signal S1 generated by the above-mentioned operation. Then, the correction receiving unit 16 generates recognition correction information Ia indicating a change in the attribute of the object corresponding to the target virtual object and supplies the recognition correction information Ia to the recognition result acquisition unit 14. As a result, the change in the attribute of the virtual object is reflected in the second recognition result Im2 as a correction of the recognition result regarding the attribute of the corresponding object.

[0058] Next, specific examples of corrections based on the first to fourth aspects will be described. FIG. 6 shows the state of the workspace before correction as viewed by the worker when pick-and-place is the target task. In FIG. 6, a first object 81, a second object 82, and a third object 83 are present on a work table 79. The display control unit 15 displays virtual objects 81V and 82V and their attribute information 81T and 82T superimposed on the scenery (real world) or a real-life image viewed by the worker, along with text information 78 prompting the worker to correct the positions, orientations, and attributes of the objects. It is assumed here that any calibration process performed in augmented reality or the like has been performed, and that coordinate conversions between various coordinate systems, such as the coordinate system of the sensor 7 and the display coordinate system displaying the virtual objects, have been appropriately performed.

[0059] In this case, the position of virtual object 81V is misaligned with the position of the real object. Also, the attribute of virtual object 82V indicated by attribute information 82T (here, "object to be grasped") is different from the attribute of the original second object 82 (here, "obstacle"). Furthermore, third object 83 is not recognized by the robot controller 1, and a corresponding virtual object has not been generated. Then, the correction receiving unit 16 receives corrections for these differences based on the input signal S1 supplied from the instruction device 2, and the display control unit 15 immediately displays the latest virtual object reflecting the corrections.

[0060] 7 shows the state of the workspace as viewed by the worker after corrections have been made to the virtual objects. In FIG. 7, based on an operation instructing the movement of virtual object 81V, virtual object 81V is appropriately positioned at a position where it overlaps with the real object. Furthermore, based on an operation instructing the attribute change of virtual object 82V, the attribute "obstacle" indicated by attribute information 82T matches the attribute of second object 82 that should be recognized. Furthermore, based on an operation instructing the generation of a virtual object for third object 83, virtual object 83V for third object 83 is generated with an appropriate position and orientation. Furthermore, the attribute of virtual object 83V indicated by attribute information 83T matches the attribute of third object 83 that should be recognized.

[0061] 7, after various correction operations are performed, the correction receiving unit 16 receives an input signal S1 corresponding to an operation to confirm the corrections, and supplies recognition correction information Ia indicating the received corrections to the recognition result acquiring unit 14. Thereafter, the recognition result acquiring unit 14 supplies a second recognition result Im2 reflecting the recognition correction information Ia to the motion planning unit 17, and the motion planning unit 17 starts calculating a motion plan for the robot 5 based on the second recognition result Im2. In this case, the display control unit 15 also displays text information 78A notifying the operator that the corrections have been received and that a motion plan has been created and robot control has begun.

[0062] In this way, the robot controller 1 can accurately correct any recognition error of an object in the workspace, and can accurately formulate an operation plan and control the robot based on the accurate recognition result.

[0063] Next, a supplementary explanation will be given of the determination of whether or not a correction is necessary by the correction receiving unit 16. In a first example, the correction receiving unit 16 receives an input specifying whether or not a correction is necessary, and determines whether or not a correction is necessary based on an input signal S1 corresponding to the input.

[0064] In a second example, the correction receiving unit 16 may determine whether correction is necessary based on a degree of confidence indicating the degree of confidence in the accuracy of the recognition (estimation) of the position, orientation, and attributes of each object. In this case, the correction receiving unit 16 associates a degree of confidence with each estimation result of the position, orientation, and attributes of each object in the first recognition result Im1, and if all of these degrees of confidence are equal to or greater than a predetermined threshold, the correction receiving unit 16 determines that correction of the first recognition result Im1 is unnecessary and supplies recognition correction information Ia indicating that correction is unnecessary to the recognition result acquisition unit 14. The above-mentioned threshold is stored, for example, in the memory 12 or the storage device 4.

[0065] On the other hand, if there is a confidence level lower than the threshold value, the correction receiving unit 16 determines that a correction is necessary and issues a display control instruction to receive the correction to the display control unit 15. Thereafter, the display control unit 15 performs display control to realize the display shown in FIG.

[0066] Preferably, in the second example, the display control unit 15 determines the display manner of the various information represented by the first recognition result Im1 based on the confidence level. For example, in the example of Fig. 6, when the confidence level for either the position or the orientation of the first object 81 is less than a threshold, the display control unit 15 highlights the virtual object 81V representing the position and orientation of the first object 81. Furthermore, when the confidence level for the attribute of the first object 81 is less than a threshold, the display control unit 15 highlights the attribute information 81T representing the attribute of the first object 81.

[0067] In this way, the display control unit 15 highlights information about recognition results that are particularly in need of correction (i.e., the confidence level is less than the threshold value). This makes it possible to effectively prevent oversight of corrections and smoothly support corrections by the worker.

[0068] Here, a supplementary explanation will be given of the confidence level included in the first recognition result Im1. When estimating the position, orientation, and attributes of an object detected based on the sensor signal S4, the recognition result acquisition unit 14 calculates a confidence level for each estimated element and generates the first recognition result Im1 by associating the calculated confidence levels with each of the estimated position, orientation, and attributes of the object. In this case, for example, when an estimation model based on a neural network is used to estimate the position, orientation, and attributes of an object, the correction receiving unit 16 uses the certainty level (reliability) output by the estimation model together with the estimation result as the confidence level. For example, the estimation model for estimating the position and orientation of an object is a regression model, and the estimation model for estimating the attributes of the object is a classification model.

[0069] (6) Details of the operation sequence generation section Next, the detailed processing of the motion planning unit 17 will be described.

[0070] (5-1) Functional Blocks 8 is an example of a functional block diagram showing the functional configuration of the motion planning unit 17. Functionally, the motion planning unit 17 includes an abstract state setting unit 31, a target logical formula generation unit 32, a time step logical formula generation unit 33, an abstract model generation unit 34, a control input generation unit 35, and a subtask sequence generation unit 36.

[0071] The abstract state setting unit 31 sets an abstract state in the workspace based on the second recognition result Im2 supplied from the recognition result acquisition unit 14. In this case, the abstract state setting unit 31 defines a proposition to express, as a logical formula, each abstract state that needs to be considered when executing the target task, based on the second recognition result Im2. The abstract state setting unit 31 supplies information indicating the set abstract state (also referred to as "abstract state setting information IS") to the target logical formula generation unit 32.

[0072] The target logical formula generation unit 32 converts the target task into a logical formula of temporal logic (also referred to as a "target logical formula Ltag") that represents a final achieved state, based on the abstract state setting information IS. In other words, the target logical formula generation unit 32 generates the target logical formula Ltag based on the initial state of the workspace before the operation of the robot 5, which is specified based on the abstract state setting information IS, and the final achieved state of the workspace. Furthermore, the target logical formula generation unit 32 adds constraints that must be satisfied in the execution of the target task to the target logical formula Ltag by referring to constraint information I2 from the application information storage unit 41. Then, the target logical formula generation unit 32 supplies the generated target logical formula Ltag to the time-step logical formula generation unit 33.

[0073] The goal logical formula generating unit 32 may recognize the final achieved state of the workspace based on information stored in advance in the storage device 4, or may recognize it based on the input signal S1 supplied from the instruction device 2.

[0074] The time-step logical formula generation unit 33 converts the target logical formula Ltag supplied from the target logical formula generation unit 32 into a logical formula (also called a "time-step logical formula Lts") that represents the state at each time step. Then, the time-step logical formula generation unit 33 supplies the generated time-step logical formula Lts to the control input generation unit 35.

[0075] The abstract model generation unit 34 generates an abstract model "Σ" that abstracts the real dynamics in the workspace, based on the abstract model information I5 stored in the application information storage unit 41 and the second recognition result Im2 supplied from the abstract state setting unit 31. In this case, the abstract model generation unit 34 regards the target dynamics as a hybrid system in which continuous dynamics and discrete dynamics are mixed, and generates an abstract model Σ based on the hybrid system. A method for generating the abstract model Σ will be described later. The abstract model generation unit 34 supplies the generated abstract model Σ to the control input generation unit 35.

[0076] The control input generation unit 35 determines a control input to the robot 5 for each time step that satisfies the time step logical formula Lts supplied from the time step logical formula generation unit 33 and the abstract model Σ supplied from the abstract model generation unit 34 and optimizes an evaluation function (for example, a function representing the amount of energy consumed by the robot).The control input generation unit 35 then supplies information indicating the control input to the robot 5 for each time step (also referred to as "control input information Icn") to the subtask sequence generation unit 36.

[0077] The subtask sequence generation unit 36 ​​generates an operation sequence Sr, which is a sequence of subtasks, based on the control input information Icn supplied from the control input generation unit 35 and the subtask information I4 stored in the application information storage unit 41, and supplies the operation sequence Sr to the robot control unit 18.

[0078] (6-2) Abstract State Setting Unit The abstract state setting unit 31 sets an abstract state in the workspace based on the second recognition result Im2 and the abstract state designation information I1 acquired from the application information storage unit 41. In this case, the abstract state setting unit 31 first refers to the abstract state designation information I1 and recognizes the abstract state to be set in the workspace. Note that the abstract state to be set in the workspace differs depending on the type of target task.

[0079] Fig. 9 shows an overhead view of the workspace when the target task is pick-and-place. The workspace shown in Fig. 9 includes two robot arms 52a and 52b, four objects 61 (61a to 61d), an obstacle 62, and an area G that is the destination of the object 61.

[0080] In this case, first, the abstract state setting unit 31 recognizes the state of the object 61, the range of existence of the obstacle 62, the state of the robot 5, the range of existence of the area G, and the like.

[0081] Here, the abstract state setting unit 31 recognizes the position vectors "x1" to "x4" of the centers of the objects 61a to 61d as the positions of the objects 61a to 61d. In addition, the abstract state setting unit 31 recognizes the position vector "x r1 " and the position vector of the robot hand 53b "x r2 " are recognized as the positions of the robot arms 52a and 52b.

[0082] Similarly, the abstract state setting unit 31 recognizes the postures of the objects 61a to 61d (not necessary in the example of FIG. 9 because the objects are spherical), the range of existence of the obstacle 62, the range of existence of the area G, etc. Note that, for example, when the obstacle 62 is regarded as a rectangular parallelepiped and the area G is regarded as a rectangle, the abstract state setting unit 31 recognizes the position vectors of the vertices of the obstacle 62 and the area G.

[0083] Furthermore, the abstract state setting unit 31 determines an abstract state to be defined in the target task by referring to the abstract state designation information I1. In this case, the abstract state setting unit 31 determines a proposition indicating an abstract state based on the second recognition result Im2 regarding objects present in the workspace (e.g., the number of each type of object) and the abstract state designation information I1.

[0084] In the example of FIG. 9, the abstract state setting unit 31 assigns identification labels "1" to "4" to the objects 61a to 61d identified by the second recognition result Im2, respectively. In addition, the abstract state setting unit 31 assigns the proposition "g i Furthermore, the abstract state setting unit 31 assigns an identification label "O" to the obstacle 62 and defines the proposition "o i Furthermore, the abstract state setting unit 31 defines a proposition "h" that the robot arms 52 interfere with each other. The abstract state setting unit 31 also defines a proposition "v" that the object "i" exists in the working table (a table in which the object and obstacles exist in the initial state). i ”, the proposition that there is an object in the non-work area other than the work table and area G, “wi " etc. The non-work area is, for example, an area (floor surface, etc.) where the object will be present if it falls from the work table.

[0085] In this way, the abstract state setting unit 31 recognizes the abstract state to be defined by referring to the abstract state designation information I1, and determines the proposition that represents the abstract state (in the above example, g i , o i , h, etc.) are defined according to the number of objects 61, the number of robot arms 52, the number of obstacles 62, the number of robots 5, etc. Then, the abstract state setting unit 31 supplies information indicating a proposition that represents the abstract state to the target logical formula generation unit 32 as abstract state setting information IS.

[0086] (6-3) Target formula generation part First, the target logical formula generator 32 converts the target task into a logical formula using temporal logic.

[0087] For example, in the example of FIG. 9, suppose that a goal task of "Eventually, the object (i=2) exists in the area G" is given. In this case, the goal logical formula generation unit 32 generates the goal task by combining the operator "◇" corresponding to "eventually" in the linear logical formula (LTL: Linear Temporal Logic) and the proposition "g i " to generate the logical formula "◇g2". Furthermore, the target logical formula generation unit 32 may express a logical formula using any temporal logic operator other than the operator "◇" (logical product "∧", logical sum "∨", negation "￢", logical inclusion "⇒", always "□", next "○", until "U", etc.). Furthermore, the logical formula may be expressed using any temporal logic, such as MTL (Metric Temporal Logic) or STL (Signal Temporal Logic), rather than being limited to linear temporal logic.

[0088] The target task may be specified in a natural language. There are various techniques for converting a task expressed in a natural language into a logical formula.

[0089] Next, the target logical formula generating unit 32 generates a target logical formula Ltag by adding the constraints indicated by the constraint information I2 to the logical formula indicating the target task.

[0090] For example, if the constraint information I2 includes two constraints corresponding to the pick-and-place shown in FIG. 9, namely, "the robot arms 52 do not interfere with each other" and "the target object i does not interfere with the obstacle O", the target logical formula generation unit 32 converts these constraints into a logical formula. Specifically, the target logical formula generation unit 32 converts the proposition "o i " and proposition "h", the above two constraints are converted into the following logical formulas, respectively. □￢h ∧ i □￢o i

[0091] Therefore, in this case, the target logical formula generation unit 32 generates the following target logical formula Ltag by adding the logical formulas of these constraint conditions to the logical formula "◇g2" corresponding to the target task "the target object (i=2) will ultimately exist in area G." (◇g2)∧(□￢h)∧(∧ i □￢o i )

[0092] In reality, the constraints for pick-and-place are not limited to the two mentioned above, and there are other constraints such as "the robot arm 52 does not interfere with the obstacle O," "multiple robot arms 52 do not grab the same object," and "objects do not come into contact with each other." These constraints are also stored in the constraint information I2 and reflected in the target logical formula Ltag.

[0093] (6-4) Time step formula generation part The time-step logical formula generation unit 33 determines the number of time steps required to complete the target task (also referred to as the "target number of time steps"), and determines a combination of propositions that represent the state at each time step that satisfies the target logical formula Ltag in the target number of time steps. Since there are usually multiple combinations, the time-step logical formula generation unit 33 generates a logical formula that combines these combinations using a logical sum as the time-step logical formula Lts. The above combinations become candidates for a logical formula that represents a sequence of actions to be instructed to the robot 5, and will hereinafter be referred to as "candidates φ."

[0094] Here, a specific example of the processing of the time-step logical expression generating unit 33 when a goal task of "finally, the object (i=2) exists in the area G" is set as exemplified in the description of FIG. 9 will be described.

[0095] In this case, the following target logical formula Ltag is supplied from the target logical formula generator 32 to the time step logical formula generator 33. (◇g2)∧(□￢h)∧(∧ i □￢o i ) In this case, the time step logical formula generator 33 generates the proposition "g i " is extended to include the concept of time steps, i,k " is used. Here, the proposition "g i,k " is a proposition that "object i exists in region G at time step k." If the target number of time steps is set to "3," the target logical formula Ltag can be rewritten as follows: (◇g 2,3 )∧(∧ k=1,2,3 □￢h k )∧(∧ i,k=1,2,3 □￢o i,k )

[0096] Also, ◇g 2,3 can be rewritten as shown in the following equation:

[0097]

number

[0098] In this case, the above-mentioned target logical formula Ltag is expressed by the logical sum (φ1∨φ2∨φ3∨φ4) of the four candidates "φ1" to "φ4" shown below.

[0099]

number

[0100] Therefore, the time-step logical expression generation unit 33 determines the logical sum of the four candidates φ1 to φ4 as the time-step logical expression Lts. In this case, the time-step logical expression Lts is true when at least one of the four candidates φ1 to φ4 is true.

[0101] Next, a method for setting the target number of time steps will be explained in more detail.

[0102] The time step logical formula generation unit 33 determines the target number of time steps based on, for example, the expected time of the task specified by the input signal S1 supplied from the instruction device 2. In this case, the time step logical formula generation unit 33 calculates the target number of time steps from the expected time based on information on the time width per time step stored in the memory 12 or the storage device 4. In another example, the time step logical formula generation unit 33 stores in advance in the memory 12 or the storage device 4 information that associates the target number of time steps appropriate for each type of target task, and determines the target number of time steps according to the type of target task to be executed by referring to the information.

[0103] Preferably, the time-step logical formula generator 33 sets the target number of time steps to a predetermined initial value. Then, the time-step logical formula generator 33 gradually increases the target number of time steps until a time-step logical formula Lts that allows the control input generator 35 to determine the control input is generated. In this case, if the control input generator 35 performs optimization processing using the set target number of time steps and is unable to derive an optimal solution, the time-step logical formula generator 33 adds a predetermined number (an integer equal to or greater than 1) to the target number of time steps.

[0104] In this case, the time-step logical expression generator 33 may set the initial value of the target number of time steps to a value smaller than the number of time steps corresponding to the expected working time of the target task by the user, thereby preventing the time-step logical expression generator 33 from setting an unnecessarily large target number of time steps.

[0105] (6-5) Abstract Model Generation Unit The abstract model generation unit 34 generates an abstract model Σ based on the abstract model information I5 and the second recognition result Im2. Here, the abstract model information I5 records information necessary for generating the abstract model Σ for each type of target task. For example, if the target task is pick-and-place, the abstract model information I5 records a generic abstract model that does not specify the positions and number of objects, the position of the area where the objects are placed, the number of robots 5 (or the number of robot arms 52), etc. The abstract model generation unit 34 then generates the abstract model Σ by reflecting the second recognition result Im2 in the generic abstract model recorded in the abstract model information I5, which includes the dynamics of the robot 5. As a result, the abstract model Σ becomes a model that abstractly represents the states of objects in the workspace and the dynamics of the robot 5. In the case of pick-and-place, the states of objects in the workspace indicate the positions and number of objects, the position of the area where the objects are placed, the number of robots 5, etc.

[0106] If other work objects exist, information on the abstracted dynamics of the other work objects may be included in the abstract model information I5. In this case, the abstract model Σ is a model that abstractly represents the state of objects in the workspace, the dynamics of the robot 5, and the dynamics of the other work objects.

[0107] Here, when the robot 5 is performing a target task, the dynamics in the workspace frequently change. For example, in pick-and-place, if the robot arm 52 is holding an object i, the object i can be moved, but if the robot arm 52 is not holding the object i, the object i cannot be moved.

[0108] Taking the above into consideration, in this embodiment, in the case of pick and place, the operation of grasping the object i is represented by the logical variable "δ i In this case, for example, the abstract model generation unit 34 can determine the abstract model Σ to be set for the work space shown in FIG. 9 by the following equation (1).

[0109]

number

[0110] Here, "u j " indicates the control input for controlling the robot hand j ("j=1" is the robot hand 53a, and "j=2" is the robot hand 53b), "I" indicates the identity matrix, and "0" indicates the zero matrix. Note that the control input is assumed to be velocity here as an example, but it may also be acceleration. Also, "δ j,i " is a logical variable that is "1" when robot hand j is grasping object i, and is "0" otherwise. r1 ", "x r2" indicates the position vector of robot hand j (j = 1, 2), and "x1" to "x4" indicate the position vector of object i (i = 1 to 4). If object i has a shape other than a sphere and its posture needs to be taken into consideration, vectors "x1" to "x4" contain elements that represent its posture, such as Euler angles. Furthermore, "h(x)" is a variable that satisfies "h(x) ≥ 0" when the robot hand is close enough to the object to grasp it, and satisfies the following relationship with the logical variable δ. δ=1 ⇔ h(x)≧0 In this equation, if the robot hand is close enough to the object to be able to grasp it, it is assumed that the robot hand is grasping the object, and the logical variable δ is set to 1.

[0111] Here, equation (1) is a difference equation that shows the relationship between the state of the object at time step k and the state of the object at time step k + 1. In the above equation (1), the grip state is represented by a logical variable, which is a discrete value, and the movement of the object is represented by a continuous value, so equation (1) shows a hybrid system.

[0112] Equation (1) takes into consideration only the dynamics of the robot hand, which is the end effector of the robot 5 that actually grasps an object, rather than the detailed dynamics of the entire robot 5. This makes it possible to suitably reduce the amount of calculation required for optimization processing by the control input generation unit 35.

[0113] Furthermore, the abstract model information I5 stores logical variables corresponding to the action of switching dynamics (the action of grasping the object i in the case of pick-and-place), and information for deriving the difference equation of formula (1) from the second recognition result Im2. Therefore, even if the positions and number of objects, the area where the objects are placed (area G in FIG. 9), the number of robots 5, etc. change, the abstract model generation unit 34 can determine an abstract model Σ that is suited to the environment of the target workspace based on the abstract model information I5 and the second recognition result Im2.

[0114] The abstract model generating unit 34 may generate a model of a mixed logical dynamical (MLD) system or a hybrid system that combines Petri nets, automata, etc., instead of the model shown in equation (1).

[0115] (6-6) Control Input Generation Unit The control input generation unit 35 determines an optimal control input for the robot 5 for each time step based on the time step logical formula Lts supplied from the time step logical formula generation unit 33 and the abstract model Σ supplied from the abstract model generation unit 34. In this case, the control input generation unit 35 defines an evaluation function for the target task and solves an optimization problem to minimize the evaluation function using the abstract model Σ and the time step logical formula Lts as constraints. The evaluation function is, for example, predetermined for each type of target task and stored in the memory 12 or the storage device 4.

[0116] For example, when a pick-and-place task is set as the target task, the control input generating unit 35 calculates the distance "d k ” and control input “u k The evaluation function is determined so that the above-mentioned distance d k In the case of a target task in which "the object (i=2) is finally present in the region G", corresponds to the distance between the object (i=2) and the region G at time step k.

[0117] In this case, the control input generator 35 calculates the distance d k The square of the norm of and the control input u k The control input generation unit 35 determines the sum of the square of the norm of the abstract model Σ and the time step logical formula Lts (i.e., the candidate φ i The constrained mixed integer optimization problem shown in the following equation (2) is solved, with the constraint being the logical sum of

[0118]

number

[0119] Here, "T" is the number of time steps to be optimized, and may be the target number of time steps, or, as will be described later, may be a predetermined number smaller than the target number of time steps. In this case, the control input generation unit 35 preferably approximates the logical variables to continuous values ​​(treating the problem as a continuous relaxation problem). This allows the control input generation unit 35 to preferably reduce the amount of calculation. Note that if STL is used instead of linear logic (LTL), the problem can be written as a nonlinear optimization problem.

[0120] Furthermore, when the target number of time steps is long (for example, when it is larger than a predetermined threshold), the control input generation unit 35 may set the number of time steps used for optimization to a value smaller than the target number of time steps (for example, the above-mentioned threshold). In this case, the control input generation unit 35 sequentially calculates the control input u by solving the above-mentioned optimization problem, for example, every time the predetermined number of time steps elapses. k Determine.

[0121] Preferably, the control input generator 35 solves the above optimization problem for each predetermined event corresponding to an intermediate state relative to the achievement state of the target task, and determines the control input u to be used. k may be determined. In this case, the control input generation unit 35 sets the number of time steps until the next event occurs as the number of time steps used for optimization. The above-mentioned event is, for example, an event in which the dynamics in the workspace switches. For example, if pick-and-place is set as the target task, the events defined include the robot 5 grasping an object, or the robot 5 completing the transportation of one of multiple objects to be transported to a destination point. Events are defined in advance for each type of target task, for example, and information identifying an event for each type of target task is stored in the storage device 4.

[0122] (6-7) Subtask sequence generation part The subtask sequence generation unit 36 ​​generates the operation sequence Sr based on the control input information Icn supplied from the control input generation unit 35 and the subtask information I4 stored in the application information storage unit 41. In this case, the subtask sequence generation unit 36 ​​recognizes subtasks that the robot 5 can accept by referring to the subtask information I4, and converts the control input for each time step indicated by the control input information Icn into a subtask.

[0123] For example, the subtask information I4 defines functions indicating two subtasks, namely, moving the robot hand (reaching) and grasping (grasping) of the robot hand, as subtasks that the robot 5 can accept when the target task is pick-and-place. In this case, the function "Move" representing reaching is a function that takes as arguments, for example, the initial state of the robot 5 before the function is executed, the final state of the robot 5 after the function is executed, and the time required to execute the function. Furthermore, the function "Grasp" representing grasping is a function that takes as arguments, for example, the state of the robot 5 before the function is executed, the state of the object to be grasped before the function is executed, and a logical variable δ. Here, the function "Grasp" represents a grasping action when the logical variable δ is "1," and represents a releasing action when the logical variable δ is "0." In this case, the subtask sequence generation unit 36 ​​determines the function "Move" based on the trajectory of the robot hand determined by the control input for each time step indicated by the control input information Icn, and determines the function "Grasp" based on the transition of the logical variable δ for each time step indicated by the control input information Icn.

[0124] The subtask sequence generator 36 then generates an operation sequence Sr consisting of the functions "Move" and "Grasp" and supplies the operation sequence Sr to the robot control unit 18. For example, if the target task is "finally, the object (i=2) is present in area G," the subtask sequence generator 36 generates an operation sequence Sr of the function "Move," the function "Grasp," the function "Move," and the function "Grasp" for the robot hand closest to the object (i=2). In this case, the robot hand closest to the object (i=2) moves to the position of the object (i=2) by the first function "Move," grasps the object (i=2) by the first function "Grasp," moves to area G by the second function "Move," and places the object (i=2) in area G by the second function "Grasp."

[0125] (7) Processing flow FIG. 10 is an example of a flowchart showing an outline of the robot control process executed by the robot controller 1 in the first embodiment.

[0126] First, the robot controller 1 acquires a sensor signal S4 from the sensor 7 (step S11). Then, the recognition result acquisition unit 14 of the robot controller 1 recognizes the state (including position and posture) and attributes of the object in the workspace based on the acquired sensor signal S4 (step S12). As a result, the recognition result acquisition unit 14 generates a first recognition result Im1 regarding the object in the workspace.

[0127] Next, the display control unit 15 causes the pointing device 2 to display virtual objects superimposed on real objects in the landscape or real-life image based on the first recognition result Im1 (step S13). In this case, the display control unit 15 generates a display control signal S2 for displaying virtual objects corresponding to each object identified by the first recognition result Im1, and supplies the display control signal S2 to the pointing device 2.

[0128] Then, the correction receiving unit 16 determines whether the first recognition result Im1 needs to be corrected (step S14). In this case, the correction receiving unit 16 may determine whether the first recognition result Im1 needs to be corrected based on the confidence level included in the first recognition result Im1, or may receive an input specifying whether the first recognition result Im1 needs to be corrected and determine whether the first recognition result Im1 needs to be corrected based on the received input.

[0129] If the correction accepting unit 16 determines that the first recognition result Im1 needs to be corrected (step S14; Yes), it accepts the correction of the first recognition result Im1 (step S15). In this case, the correction accepting unit 16 accepts the correction (more specifically, the specification of the target to be corrected and the content to be corrected, etc.) based on an arbitrary operation method using the input unit 24a, which serves as an arbitrary user interface included in the instruction device 2. Then, the recognition result acquiring unit 14 generates a second recognition result Im2 that reflects the recognition correction information Ia generated by the correction accepting unit 16 (step S16). On the other hand, if the correction accepting unit 16 determines that the first recognition result Im1 does not need to be corrected (step S14; No), the process proceeds to step S17. In this case, the correction accepting unit 16 supplies the recognition correction information Ia indicating that the correction is not necessary to the recognition result acquiring unit 14, and the recognition result acquiring unit 14 supplies the first recognition result Im1 as the second recognition result Im2 to the action planning unit 17.

[0130] The motion planning unit 17 then determines a motion plan for the robot 5 based on the second recognition result Im2 (step S17). As a result, the motion planning unit 17 generates a motion sequence Sr, which is a motion sequence of the robot 5. The robot control unit 18 then controls the robot based on the determined motion plan (step S18). In this case, the robot control unit 18 sequentially supplies the control signals S3 generated based on the motion sequence Sr to the robot 5, and controls the robot 5 to operate in accordance with the generated motion sequence Sr. (8) Variations The block configuration of the operation planning unit 17 shown in FIG. 8 is an example, and various modifications may be made.

[0131] For example, information on candidate φ for a sequence of operations to be commanded to the robot 5 is stored in advance in the storage device 4, and the operation planning unit 17 executes the optimization process of the control input generation unit 35 based on the information. As a result, the operation planning unit 17 selects the optimal candidate φ and determines the control input for the robot 5. In this case, the operation planning unit 17 does not need to have functions corresponding to the abstract state setting unit 31, the target logical formula generation unit 32, and the time step logical formula generation unit 33 in generating the operation sequence Sr. In this way, information on the execution results of some of the functional blocks of the operation planning unit 17 shown in FIG. 8 may be stored in advance in the application information storage unit 41.

[0132] In another example, the application information may include in advance design information such as a flowchart for designing an operation sequence Sr corresponding to a target task, and the operation planning unit 17 may generate the operation sequence Sr by referring to the design information. Note that a specific example of executing a task based on a pre-designed task sequence is disclosed in, for example, Japanese Patent Application Laid-Open No. 2017-39170.

[0133] Second Embodiment Instead of or in addition to the process of accepting corrections to the first recognition result Im1, the robot controller 1 according to the second embodiment displays information (also referred to as "trajectory information") about the trajectory of the object (target object, robot 5) based on the formulated motion plan after formulating the motion plan, and performs a process of accepting corrections to the trajectory. As a result, the robot controller 1 according to the second embodiment appropriately corrects the motion plan so that the target task is executed in the flow intended by the operator.

[0134] Hereinafter, in the robot control system 100, the same components as those in the first embodiment will be appropriately denoted by the same reference numerals, and the description thereof will be omitted. Note that the configuration of the robot control system 100 in the second embodiment is the same as the configuration shown in FIG.

[0135] Fig. 11 shows an example of functional blocks of a robot controller 1A in the second embodiment. The robot controller 1A has the hardware configuration shown in Fig. 2(A), and the processor 11 of the robot controller 1A functionally has a recognition result acquisition unit 14A, a display control unit 15A, a correction acceptance unit 16A, an operation planning unit 17A, and a robot control unit 18A.

[0136] As in the first embodiment, the recognition result acquisition unit 14A generates a first recognition result Im1 and a second recognition result Im2 based on the recognition correction information Ia.

[0137] In addition to the processing performed by the display control unit 15 in the first embodiment, the display control unit 15A acquires trajectory information identified from the operation plan determined by the operation planning unit 17A from the operation planning unit 17A and controls the display of the instruction device 2 regarding the trajectory information. In this case, the display control unit 15A generates a display control signal S2 for the instruction device 2 to display trajectory information regarding the trajectory of the object, etc. for each time step indicated by the operation sequence Sr, and supplies the display control signal S2 to the instruction device 2. In this case, the display control unit 15A may display trajectory information indicating the trajectory of the robot 5 in addition to the trajectory of the object. Here, the display control unit 15A may display, as the trajectory information, information indicating the state transition of the object, etc. for each time step, or information indicating the state transition of the object, etc. for each predetermined number of time steps.

[0138] In addition to the process of generating recognition correction information Ia executed by the correction receiving unit 16 in the first embodiment, the correction receiving unit 16A also receives corrections to trajectory information through operations by the operator using the instruction device 2. When the operation related to the correction is completed, the correction receiving unit 16A generates trajectory correction information "Ib" indicating the details of the correction related to the trajectory of the object, etc., and supplies the trajectory correction information Ib to the motion planning unit 17A. Note that when there is no input related to the correction, the correction receiving unit 16A supplies trajectory correction information Ib indicating that no correction has been made to the motion planning unit 17A.

[0139] In addition to the process of generating the movement sequence Sr executed by the movement planning unit 17 in the first embodiment, the movement planning unit 17A generates a movement sequence Sr (also referred to as a "second movement sequence Srb") that reflects the trajectory correction information Ib supplied from the correction receiving unit 16A. As a result, the movement planning unit 17A formulates a new movement plan by correcting the original movement plan so that the state of the object specified by the correction is realized. Then, the movement planning unit 17A supplies the generated second movement sequence Srb to the robot control unit 18. Hereinafter, for convenience, the movement sequence Sr before reflecting the trajectory correction information Ib will also be referred to as a "first movement sequence Sr." Furthermore, a movement plan based on the first movement sequence Sr will be referred to as a "first movement plan," and a movement plan based on the second movement sequence Srb will be referred to as a "second movement plan." If trajectory correction information Ib indicating that correction of the first movement plan is unnecessary is generated, the second movement sequence Srb will be the same as the first movement sequence Sr.

[0140] The action planning unit 17A may determine whether the second action sequence Srb reflecting the trajectory correction information Ib satisfies the constraint conditions, and supply the generated second action sequence Srb to the robot control unit 18 only if the constraint conditions indicated by the constraint condition information I2 are satisfied. This allows the robot 5 to preferably execute only action plans that satisfy the constraint conditions. If the action planning unit 17A determines that the second action sequence Srb does not satisfy the constraint conditions, it instructs the display control unit 15A and the action planning unit 17A to execute processing to accept further corrections.

[0141] The robot control unit 18A controls the operation of the robot 5 by supplying a control signal S3 to the robot 5 via the interface 13 based on the second operation sequence Srb supplied from the operation planning unit 17A.

[0142] Here, a supplementary explanation will be given on the generation of trajectory information. After generating the first operation sequence Sr, the operation planning unit 17A supplies the display control unit 15A with trajectory information of the object (and the robot 5) required for display control by the display control unit 15A. In this case, the position (posture) vectors (see formula (1)) of the robot 5 (specifically, the robot hand) and the object for each time step are obtained by the optimization process based on formula (2) executed in formulating the first operation plan. Therefore, the operation planning unit 17A supplies these position (posture) vectors to the display control unit 15A as trajectory information. Here, the trajectory information supplied by the operation planning unit 17A to the display control unit 15A includes information on the timing at which the robot hand grasps (and releases) the object (i.e., δ in formula (1)). j,i The information may include the position of the robot hand (information specified by the robot hand position vector), the gripping (and release) direction, and the posture of the robot hand when gripping (and release). The gripping (and release) direction of the object may be specified based on, for example, the trajectory of the position vector of the robot hand and the gripping (and release) timing.

[0143] Note that the functional blocks shown in FIG. 11 are based on the premise that the processing of the first embodiment is performed, but this is not limiting, and the robot controller 1A does not have to perform processing related to the correction of the first recognition result Im1 (generation of recognition correction information Ia by the correction receiving unit 16A, generation of the second recognition result Im2 by the recognition result acquisition unit 14A, display control of the first recognition result Im1 by the display control unit 15, etc.).

[0144] Next, specific examples (first and second specific examples) of processing related to displaying and correcting trajectory information of an object or the like in the second embodiment will be described.

[0145] 12 is a diagram showing trajectory information in the first specific example. The first specific example is a specific example related to a target task of placing an object 85 in a box 86, and the robot controller 1A displays a schematic representation of the trajectories of the robot hand 53 of the robot 5 and the object 85 specified by the first motion plan determined by the motion planning unit 17A.

[0146] 12, positions "P1" to "P9" indicate the positions of the robot hand 53 for each predetermined number of time steps based on the first operation plan, and a trajectory line 87 indicates the trajectory (path) of the robot hand 53 based on the first operation plan. Virtual objects "85Va" to "85Ve" indicate virtual objects representing the position and posture of the object 85 for each predetermined number of time steps based on the first operation plan. Arrow "Aw1" indicates the direction in which the robot hand 53 grasps the object 85 when switching to a grasping state based on the first operation plan, and arrow "Aw2" indicates the direction in which the robot hand 53 moves away from the object 85 when switching to a non-grasping state based on the first operation plan. Virtual robot hands "53Va" to "53Vh" are virtual objects representing the postures of the robot hand 53 immediately before and after the robot hand 53 switches between a grasping state and a non-grasping state based on the first operation plan. In addition, in the first operation plan formulated by the operation planning unit 17A, the trajectory of the robot hand 53, which is the end effector of the robot 5, is generated as the trajectory of the robot 5, so here the trajectory of the robot hand 53 is shown as the trajectory of the robot 5.

[0147] Based on the trajectory information received from the motion planning unit 17A, the display control unit 15A causes the instruction device 2 to display positions P1 to P9, a trajectory line 87, and arrows Aw1 and Aw2 indicating the trajectory of the robot hand 53, virtual robot hands 53Va to 53Vh indicating the posture of the robot hand 53 immediately before and after switching between the grasping state and the non-grasping state, and virtual objects 85Va to 85Ve indicating the trajectory of the target object 85. In this way, the display control unit 15A allows the worker to easily grasp the outline of the formulated first motion plan. Note that, if the trajectories of each joint of the robot 5 are determined in the first motion plan, the display control unit 15A may display the trajectories of each joint of the robot 5 in addition to the trajectory of the robot hand 53.

[0148] Then, the correction receiving unit 16A receives a correction of each element based on the first operation plan shown in Fig. 12. The object to be corrected in this case may be the state of the robot hand 53 or the object 85 at any time step (including the posture of the robot hand 53 specified by the virtual robot hands 53Va to 53Vh), or may be the timing of grasping or non-grasping, etc.

[0149] 12, the robot hand 53 attempts to place the object 85 in the box 86 while holding the handle portion of the object 85. Therefore, the robot hand 53 may come into contact with the box 86, resulting in failure of the task. Taking the above into consideration, the worker carries the object 85 to the vicinity of the box 86 with the robot hand 53 holding the handle, and then performs an operation using the instruction device 2 to generate a correction to add a motion to re-grasp the object 85 so as to grasp the upper portion of the object 85. The correction receiving unit 16A then generates trajectory correction information Ib based on the input signal S1 generated by the above operation, and supplies the generated information to the motion planning unit 17A. The motion planning unit 17A then generates a second motion sequence Srb that reflects the trajectory correction information Ib. The display control unit 15A then causes the instruction device 2 to display the trajectory information identified by the second motion sequence Srb again.

[0150] FIG. 13 is a diagram showing, in the first specific example, corrected trajectory information of the robot hand 53 and the object 85 based on inputs for correcting the trajectories of the robot hand 53 and the object 85, etc.

[0151] 13, positions "P11" to "P20" indicate the positions of the robot hand 53 for each predetermined number of time steps based on the modified second operation plan, and a trajectory line 88 indicates the trajectory (path) of the robot hand 53 based on the modified second operation plan. Virtual objects "85Vf" to "85Vj" indicate virtual objects that represent the position and posture of the object 85 for each predetermined number of time steps based on the second operation plan. Arrows "Aw11" and "Aw13" indicate the direction in which the robot hand 53 grasps the object 85 when the robot hand 53 switches from a non-grasping state to a grasping state based on the second operation plan, and arrows "Aw12" and "Aw14" indicate the direction in which the robot hand 53 moves away from the object 85 when the robot hand 53 switches from a grasping state to a non-grasping state based on the second operation plan. Furthermore, the virtual robot hands "53Vj" to "53Vm" are virtual objects representing the posture of the robot hand 53 immediately before the robot hand 53 switches between the grasping state and the non-grasping state based on the second operation plan. Note that, instead of the example of Fig. 13, the correction receiving unit 16A may also display the virtual objects representing the posture of the robot hand 53 immediately after the robot hand 53 switches between the grasping state and the non-grasping state based on the second operation plan in the same manner as in Fig. 12.

[0152] In the example of FIG. 13 , the correction receiving unit 16A generates trajectory correction information Ib, based on the operator's operation, indicating the addition of an action to place the object 85 on a horizontal plane and an action to grasp the upper part of the object 85 (i.e., the addition of an action to change the grip of the object 85) at a time step corresponding to the position P6 or the position P7 in FIG. 12 . The action to place the object 85 on a horizontal plane is an action related to the position P16, the virtual object 85Vg, the virtual robot hand 53Vk, and the arrow Aw12, and the action to grasp the upper part of the object 85 is an action related to the position P17, the virtual robot hand 53Vl, and the arrow Aw13. The trajectory correction information Ib also includes information regarding the correction of the action to place the object 85 in a box 86, which is corrected in conjunction with the change in these actions. For example, with respect to the action of placing the object 85 in the box 86, a virtual robot hand 53Vm exhibiting a posture different from the posture of the robot hand 53 specified by the virtual robot hands 53Vg and 53Vh shown in FIG. 12 is displayed in FIG. Then, the motion planning unit 17A determines the second motion plan shown in FIG. 13 based on this trajectory correction information Ib.

[0153] Here, a method for determining the second operation plan will be explained in more detail. In a first example, the operation planning unit 17A recognizes the correction contents indicated in the trajectory correction information Ib as additional constraints. Then, the operation planning unit 17A calculates the states of the robot hand 53 and the object 85 for each time step by again executing the optimization process indicated in equation (2) based on the additional constraints and the existing constraints indicated by the constraint information I2. Then, the display control unit 15 causes the instruction device 2 to again display trajectory information based on the above-mentioned calculation results. Furthermore, when the operation planning unit 17A receives an input signal S1 or the like indicating approval of the redisplayed trajectories of the robot hand 53 and the object 85, it supplies a second operation sequence Srb based on the above-mentioned calculation results to the robot control unit 18A.

[0154] In a second example of generating a second operation plan, when the trajectories of the robot hand 53 and the object 85 after correction are specified based on an operation on the instruction device 2, the correction receiving unit 16A supplies trajectory correction information Ib including trajectory information of the robot hand 53 and the object 85 after correction to the operation planning unit 17A. Then, in this case, the operation planning unit 17A determines whether the trajectory information of the robot hand 53 and the object 85 after correction specified by the trajectory correction information Ib satisfies the existing constraint conditions (i.e., the constraint conditions indicated by the constraint condition information I2), and if the trajectory information satisfies the existing constraint conditions, generates a second operation sequence Srb based on the trajectory information and supplies the second operation sequence Srb to the robot control unit 18A.

[0155] According to these first and second examples, the operation planning unit 17A can preferably formulate a second operation plan that modifies the first operation plan so that the specified state of the object is realized by the modification.

[0156] 13 is just an example, and the robot controller 1A may accept various corrections regarding the trajectories of the robot hand 53 and the object 85. For example, the robot controller 1A may accept corrections to the posture of the object 85 in the gripping state, such as a correction to tilt the object 85 from a vertical position to a 45-degree angle midway, or a correction to change the orientation of the object 85 when it reaches the vicinity of the box 86 so that the object 85 can be easily placed in the box 86.

[0157] In this way, the robot controller 1A according to the second embodiment can suitably accept modifications to the state of the object, such as its position and posture, and modifications to the point at which the object is grasped, and determine a second operation plan that reflects these modifications.

[0158] 14(A) is a diagram showing trajectory information before correction in the second specific example from a first viewpoint, and FIG. 14(B) is a diagram showing trajectory information before correction in the second specific example from a second viewpoint. Here, the second specific example is a specific example related to a goal task of moving an object 93 to a position on the work table 79 behind a first obstacle 91 and a second obstacle 92, and the robot controller 1A displays the trajectory of the object 93 specified by a first motion plan determined by the motion planning unit 17A. Virtual objects 93Va to 93Vd are virtual objects representing the position and posture of the object 93 for each predetermined number of time steps based on the first motion plan. Note that the virtual object 93Vd represents the position and posture of the object 93 (i.e., the object 93 located at the goal position) when the goal task is accomplished.

[0159] 14(A) and 14(B), in the first operation plan, the trajectory of the object 93 is set so that the object 93 passes through the space between the first obstacle 91 and the second obstacle 92. On the other hand, in the case of such a trajectory, there is a possibility that the robot hand of the robot 5 (not shown) may come into contact with the first obstacle 91 or the second obstacle 92, and the operator determines that it is necessary to correct the trajectory of the object 93.

[0160] FIG. 15(A) is a diagram illustrating an outline of an operation related to the correction of trajectory information in the second specific example from a first viewpoint, and FIG. 15(B) is a diagram illustrating an outline of an operation related to the correction of trajectory information in the second specific example from a second viewpoint. In this example, the worker performs an operation on the instruction device 2 to correct the trajectory of the object 93 so that the object 93 passes beside the second obstacle 92 without passing through the space between the first obstacle 91 and the second obstacle 92. Specifically, the worker performs an operation such as a drag-and-drop operation to place a virtual object 93Vb existing between the first obstacle 91 and the second obstacle 92 at a position beside the second obstacle 92. Then, based on the input signal S1 generated by the above operation, the display control unit 15A newly generates and displays a virtual object 93Vy located beside the second obstacle 92. In this example, the worker adjusts not only the position of the object 93 but also the orientation of the virtual object 93Vy so that the object 93 has a desired orientation.

[0161] Then, the correction receiving unit 16A supplies trajectory correction information Ib including information about the position and posture of the virtual object 93Vy to the action planning unit 17A. In this case, the action planning unit 17A recognizes, for example, that the object 93 transitions to the state of the virtual object 93Vy based on the trajectory correction information Ib as an additional constraint. Then, the action planning unit 17A performs the optimization process shown in equation (2) based on the additional constraint, and determines the trajectory of the object 93 (and the trajectory of the robot 5) after the correction, etc. Note that if the trajectory correction information Ib includes information about the scheduled operation time (i.e., of the time step) corresponding to the virtual object 93Vb before the change, the action planning unit 17A may set, as the above-mentioned additional constraint, that the object 93 transitions to the state of the virtual object 93Vy at the above-mentioned scheduled operation time.

[0162] Fig. 16(A) is a diagram showing trajectory information based on the second operation plan in the second specific example from a first viewpoint, and Fig. 16(B) is a diagram showing trajectory information based on the second operation plan in the second specific example from a second viewpoint. In Fig. 16(A) and Fig. 16(B), virtual objects 93Vx to 93Vz represent the position and posture of the target object 93 for each predetermined number of time steps based on the second operation plan.

[0163] 16(A) and 16(B), in this case, the display control unit 15A uses trajectory information based on the second operation plan regenerated by the operation planning unit 17A to display the transition of the object 93 reflecting the correction by virtual objects 93Vx to 93Vz and 93Vd. Note that, here, the second operation plan takes into consideration the state of virtual object 93Vy as a constraint condition (subgoal), and therefore the trajectory of the object 93 is appropriately corrected so that the object 93 passes beside the second obstacle 92. Then, by controlling the robot 5 based on the second operation plan in this way, the robot controller 1A can cause the robot 5 to appropriately complete the target task.

[0164] FIG. 17 is an example of a flowchart showing an outline of the robot control process executed by the robot controller 1 in the second embodiment.

[0165] First, the robot controller 1 acquires a sensor signal S4 from the sensor 7 (step S21). Then, the recognition result acquisition unit 14A of the robot controller 1A recognizes the state (including position and posture) and attributes of the object in the workspace based on the acquired sensor signal S4 (step S22). Furthermore, the recognition result acquisition unit 14A generates a second recognition result Im2 by correcting the first recognition result Im1 based on the processing of the first embodiment. Note that in the second embodiment, the correction processing of the first recognition result Im1 based on the processing of the first embodiment is not essential processing.

[0166] Then, the action planning unit 17A determines a first action plan (step S23). Then, the display control unit 15A acquires trajectory information based on the first action plan determined by the action planning unit 17A, and causes the instruction device 2 to display the trajectory information (step S24). In this case, the display control unit 15A causes the instruction device 2 to display at least the trajectory information related to the object.

[0167] Then, the correction receiving unit 16A determines whether or not the orbit information needs to be corrected (step S25). In this case, the correction receiving unit 16A receives, for example, an input specifying whether or not the orbit information needs to be corrected, and determines whether or not the correction is needed based on the received input.

[0168] If the correction receiving unit 16A determines that the trajectory information needs to be corrected (step S25; Yes), it accepts the correction related to the trajectory information (step S26). In this case, the correction receiving unit 16A accepts the correction based on any operation method using the input unit 24a, which is any user interface included in the instruction device 2. Then, the motion planning unit 17A determines a second motion plan that reflects the accepted correction based on the trajectory correction information Ib generated by the correction receiving unit 16A (step S27). Then, the motion planning unit 17A determines whether the decided second motion plan satisfies the constraint conditions (step S28). If the second motion plan satisfies the constraint conditions (step S28; Yes), the motion planning unit 17A proceeds to the process of step S29. On the other hand, if the second motion plan does not satisfy the constraint conditions (step S28; No), the correction receiving unit 16A considers the previous correction to be invalid and accepts a correction related to the trajectory information again in step S26. If the second operation plan is determined by the operation planning unit 17A so as to satisfy the additional constraint condition in step S27, the second operation plan is deemed to satisfy the constraint condition in step S28.

[0169] Then, if correction of the trajectory information is not required (step S25; No), or if it is determined in step S28 that the constraints are satisfied (step S28; Yes), the robot control unit 18A controls the robot based on the second movement sequence Srb based on the second movement plan determined by the movement planning unit 17A (step S29). In this case, the robot control unit 18A sequentially supplies control signals S3 generated based on the second movement sequence Srb to the robot 5, and controls the robot 5 to move in accordance with the generated second movement sequence Srb. Note that if correction of the trajectory information is not required, the robot controller 1A regards the first movement plan as the second movement plan and executes the process of step S18.

[0170] In the second embodiment, instead of displaying the trajectory information using augmented reality, the robot controller 1A may superimpose the trajectory information on a computer graphics (CG) image or the like that schematically represents the working space and display it on the instruction device 2, and accept various corrections to the trajectory of the object or the robot 5. In this mode as well, the robot controller 1A can suitably accept corrections to the trajectory information made by the worker.

[0171] Third Embodiment FIG. 18 shows a schematic configuration diagram of a control device 1X in the third embodiment. The control device 1X mainly includes a recognition result acquisition means 14X, a display control means 15X, and a correction acceptance means 16X. Note that the control device 1X may be composed of multiple devices. The control device 1X may be, for example, the robot controller 1 in the first embodiment or the robot controller 1A in the second embodiment.

[0172] The recognition result acquisition means 14X acquires the recognition result of an object related to a task executed by the robot. The "object related to the task" refers to any object related to the task executed by the robot, such as a target object (workpiece) to be grasped or processed by the robot, another workpiece, or the robot. The recognition result acquisition means 14X may acquire the recognition result of the object by generating it based on information generated by a sensor that senses the environment in which the task is executed, or may acquire the recognition result of the object by receiving it from an external device that generates the recognition result. The former recognition result acquisition means 14X can be, for example, the recognition result acquisition unit 14 in the first embodiment or the recognition result acquisition unit 14A in the second embodiment.

[0173] The display control means 15X displays information representing the recognition result so that it is visually superimposed on a real landscape or an image of a landscape. Here, the "landscape" corresponds to the workspace in which the task is executed. The display control means 15X may be a display device that performs display itself, or may perform display by transmitting a display signal to an external display device. The display control means 15X may be, for example, the display control unit 15 in the first embodiment or the display control unit 15A in the second embodiment.

[0174] The correction receiving means 16X receives a correction of the recognition result based on an external input. The correction receiving means 16X can be, for example, the correction receiving unit 16 in the first embodiment or the correction receiving unit 16A in the second embodiment.

[0175] 19 is an example of a flowchart in the third embodiment. The recognition result acquisition means 14X acquires the recognition result of an object related to a task to be performed by the robot (step S31). The display control means 15X displays information representing the recognition result so that it is visually recognized as being superimposed on the actual scenery or an image of the scenery (step S32). The correction acceptance means 16X accepts a correction of the recognition result based on an external input (step S33).

[0176] According to the third embodiment, the control device 1X can suitably accept a correction of the recognition result of an object related to a task to be executed by a robot, and can acquire an accurate recognition result based on the correction.

[0177] <Fourth embodiment> 20 shows a schematic configuration diagram of a control device 1Y in the fourth embodiment. The control device 1Y mainly includes an operation planning unit 17Y, a display control unit 15Y, and a correction receiving unit 16Y. The control device 1Y may be composed of multiple devices. The control device 1X may be, for example, the robot controller 1 in the first embodiment.

[0178] The motion planning means 17Y determines a first motion plan for the robot that executes a task using an object. The motion planning means 17Y also determines a second motion plan for the robot based on a correction received by a correction receiving means 16Y, which will be described later. The motion planning means 17Y can be, for example, the motion planning unit 17A in the second embodiment.

[0179] The display control means 15Y displays trajectory information regarding the trajectory of the object based on the first operation plan. The display control means 15Y may be a display device that performs display by itself, or may perform display by transmitting a display signal to an external display device. The display control means 15Y can be, for example, the display control unit 15A in the second embodiment.

[0180] The correction receiving means 16Y receives a correction related to the trajectory information based on an external input. The correction receiving means 16Y can be, for example, the correction receiving unit 16A in the second embodiment.

[0181] 21 is an example of a flowchart in the fourth embodiment. The motion planning means 17Y determines a motion plan for the robot that executes a task using an object (step S41). The display control means 15Y displays trajectory information regarding the trajectory of the object based on the motion plan (step S42). The correction receiving means 16Y receives a correction regarding the trajectory information based on an external input (step S43). Then, the motion planning means 17Y determines a second motion plan for the robot based on the correction received by the correction receiving means 16Y (step S44).

[0182] According to the fourth embodiment, the control device 1X can display trajectory information relating to the trajectory of an object based on the determined motion plan of the robot, and can suitably accept and reflect corrections thereto in the motion plan.

[0183] In each of the above-described embodiments, the program can be stored using various types of non-transitory computer-readable media and supplied to a computer processor or the like. Non-transitory computer-readable media include various types of tangible storage media (tangible storage media). Examples of non-transitory computer-readable media include magnetic storage media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). The program may also be supplied to a computer by various types of transient computer-readable media (transitory computer-readable media). Examples of transient computer-readable media include electrical signals, optical signals, and electromagnetic waves. The transient computer-readable media can supply the program to a computer via a wired communication path such as an electric wire or optical fiber, or via a wireless communication path.

[0184] In addition, part or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes.

[0185] [Appendix 1] a motion planning means for determining a first motion plan for a robot that executes a task using an object; a display control means for displaying trajectory information regarding a trajectory of the object based on the first operation plan; a correction receiving means for receiving a correction regarding the orbit information based on an external input, The motion planning means determines a second motion plan for the robot based on the correction. [Appendix 2] The control device according to claim 1, wherein the motion planning means determines the second motion plan by modifying the first motion plan so that the state of the object specified by the modification is realized. [Appendix 3] 3. The control device according to claim 1, wherein the correction receiving means receives the correction regarding at least one of a position and an attitude of the object on the trajectory. [Appendix 4] the display control means displays an object of the object representing a state of the object at each predetermined time interval; 4. The control device according to claim 3, wherein the modification receiving means receives the modification regarding the state of the object on the trajectory based on the external input that changes the state of the object. [Appendix 5] the display control means displays, as the trajectory information, information regarding a position at which the robot grasps the object, a grasping direction, or an attitude of an end effector of the robot; 5. The control device according to any one of claims 1 to 4, wherein the correction receiving means receives the correction regarding a position at which the robot grasps the object, a direction in which the robot grasps the object, or an attitude of the end effector. [Appendix 6] the correction accepting means accepts the correction specifying addition of an action of changing the object by the robot; 6. The control device according to any one of appendices 1 to 5, wherein the motion planning means determines the second motion plan including a motion of the robot changing over the object. [Appendix 7] 7. The control device according to any one of appendices 1 to 6, wherein the display control means displays a trajectory related to the robot together with a trajectory of the object as the trajectory information. [Appendix 8] A control device described in any one of appendices 1 to 7, further comprising a robot control means for controlling the robot based on the second operation plan when the second operation plan satisfies the constraint conditions set in the first operation plan. [Appendix 9] The motion planning means a logical expression conversion means for converting a task to be executed by the robot into a logical expression based on temporal logic; a time step logical expression generating means for generating, from the logical expressions, a time step logical expression which is a logical expression representing a state for each time step in order to execute the task; a subtask sequence generation means for generating a sequence of subtasks to be executed by the robot based on the time step logical formula; 9. The control device according to any one of claims 1 to 8, comprising: [Appendix 10] The computer determining a first motion plan for the robot to perform a task using the object; displaying trajectory information regarding a trajectory of the object based on the first motion plan; accepting a correction to the orbit information based on an external input; determining a second motion plan for the robot based on the modification; Control method. [Appendix 11] determining a first motion plan for the robot to perform a task using the object; displaying trajectory information regarding a trajectory of the object based on the first motion plan; accepting a correction to the orbit information based on an external input; A storage medium storing a program that causes a computer to execute a process for determining a second operation plan for the robot based on the correction.

[0186] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above embodiments. Various modifications within the scope of the present invention that would be understood by those skilled in the art can be made to the configuration and details of the present invention. In other words, the present invention naturally includes various modifications and alterations that would be possible for those skilled in the art based on the entire disclosure, including the claims, and the technical ideas. Furthermore, the disclosures of the above-cited patent documents and other documents are incorporated herein by reference. [Explanation of symbols]

[0187] 1. 1A robot controller 1X, 1Y control device 2 Indicating device 4 Storage device 5. Robot 7 Sensors 41 Application information storage unit 100 Robot Control System

Claims

1. a motion planning means for determining a first motion plan for a robot that executes a task using an object; a display control means for displaying trajectory information showing a plurality of objects of the object, which represents a state of the object including at least a three-dimensional position and orientation of the object at each predetermined time interval during the movement of the object, the movement of which is caused by the robot based on the first operation plan; a correction receiving means for receiving a correction to the trajectory information by an external input operation for selecting one of the plurality of objects and an external input operation for correcting at least one of a three-dimensional position or an attitude of the object selected by the operation, The motion planning means determines a second motion plan for the robot based on the correction.

2. The control device according to claim 1 , wherein the motion planning means determines the second motion plan so as to satisfy the modification as an additional constraint.

3. The control device according to claim 1 or 2, wherein the motion planning means determines the second motion plan by modifying the first motion plan so that the state of the object specified by the modification is realized.

4. the display control means displays, as the trajectory information, information regarding a position at which the robot grasps the object, a direction in which the robot grasps the object, or an attitude of an end effector of the robot; 4. The control device according to claim 1, wherein the correction receiving means receives the correction regarding a position where the robot grasps the object, a direction in which the robot grasps the object, or an attitude of the end effector.

5. the correction accepting means accepts the correction specifying addition of an action of changing the object by the robot; 5. The control device according to claim 1, wherein the motion planning means determines the second motion plan including a motion of the robot changing over the object.

6. 6. The control device according to claim 1, wherein the display control means displays a trajectory related to the robot together with the trajectory of the object as the trajectory information.

7. The control device according to any one of claims 1 to 6, further comprising a robot control means for controlling the robot based on the second operation plan when the second operation plan satisfies the constraint conditions set in the first operation plan.

8. The computer determining a first motion plan for a robot to perform a task using an object; displaying trajectory information indicating a plurality of objects of the object, which represents a state of the object including at least a three-dimensional position and orientation of the object at each predetermined time interval during the movement of the object caused to move by the robot based on the first operation plan; accepts a correction related to the trajectory information by an external input operation of selecting one of the plurality of objects and an external input operation of correcting at least one of a three-dimensional position or an attitude of the object selected by the operation; determining a second motion plan for the robot based on the modification; Control method.

9. determining a first motion plan for a robot to perform a task using an object; displaying trajectory information indicating a plurality of objects of the object, which represents a state of the object including at least a three-dimensional position and orientation of the object at each predetermined time interval during the movement of the object caused to move by the robot based on the first operation plan; accepts a correction related to the trajectory information by an external input operation of selecting one of the plurality of objects and an external input operation of correcting at least one of a three-dimensional position or an attitude of the object selected by the operation; A program that causes a computer to execute a process of determining a second operation plan for the robot based on the correction.

Images