Motion Planning Device, Motion Planning Method, and Program

Abstract

This motion planning device 1X mainly comprises a status setting means 31X and a motion planning means 16X. The status setting means 31X sets a status in a work space where a mobile robot equipped with a manipulator for handling an object is to perform work. The motion planning means 16X decides a motion plan for the movement of the robot and the motion of the manipulator on the basis of the status set by the status setting means 31X, restricting conditions regarding the robot movement and the manipulator motion, and an evaluation function based on the dynamics of the robot.

Description

Technical Field

[0001] The present disclosure relates to the technical field of an operation planning device, an operation planning method, and a storage medium for performing operation planning of a robot.

Background Art

[0002] Mobile working robots have been proposed. For example, Patent Document 1 discloses a control unit that controls each operation axis of a robot arm provided on a cart and each operation axis of the cart to operate in cooperation.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a mobile robot having a manipulator, the movement of the robot body and the operation of the manipulator affect each other, and it is difficult to plan these simultaneously. On the other hand, when each of the movement of the robot body and the operation of the manipulator is planned independently, an inefficient plan may be calculated.

[0005] One object of the present disclosure is to provide an operation planning device, an operation planning method, and a storage medium capable of suitably determining the operation plan of a mobile robot having a manipulator in view of the above-described problems.

Means for Solving the Problems

[0006] One aspect of the operation planning device is state setting means for setting a state in a work space where a robot having a manipulator for handling an object and a self-propelled body works, The above state, the constraint conditions regarding the movement of the main body and the operation of the manipulator, and the evaluation function based on the dynamics of the robot, Based on this, operation planning means for determining an operation plan regarding the movement of the main body and the operation of the manipulator, having and The operation planning means determines a first operation plan for moving the main body to a reference point within a designated area designated as the area where the robot performs work, and determines a second operation plan, which is the operation plan based on the state, the constraint conditions, and the evaluation function after the robot reaches the reference point, after the robot reaches the reference point. It is an operation planning device.

[0007] Another aspect of the operation planning device is area designating means for receiving a designation of an area where a robot having a manipulator for handling an object and a self-propelled main body works, object designating means for receiving a designation regarding an object within the area, Based on the designation of the area and the designation regarding the object, operation planning means for determining an operation plan regarding the movement of the main body and the operation of the manipulator, having and The operation planning means determines a first operation plan for moving the main body to a reference point within a designated area designated as the area where the robot performs work, and determines a second operation plan, which is the operation plan after the robot reaches the reference point, after the robot reaches the reference point. It is an operation planning device.

[0008] One aspect of the operation planning method is a computer sets the state in the working space where a robot having a manipulator for handling an object and a self-propelled main body works, Based on the above state, the constraint conditions regarding the movement of the main body and the operation of the manipulator, and the evaluation function based on the dynamics of the robot, determines an operation plan regarding the movement of the main body and the operation of the manipulator and In the determination of the operation plan, a first operation plan for moving the main body to a reference point within a designated area designated as the area where the robot performs work is determined, and a second operation plan, which is the operation plan based on the state, the constraint conditions, and the evaluation function after the robot reaches the reference point, is determined by the robot after the robot reaches the reference point. operation planning method.

[0009] Another aspect of the operation planning method is a computer Receives a designation of an area where a robot having a manipulator for handling an object and a self-propelled main body works, Receives a designation regarding an object within the area 、 Based on the designation of the area and the designation regarding the object, determines an operation plan regarding the movement of the main body and the operation of the manipulator and , In the determination of the operation plan, a first operation plan for moving the main body to a reference point within a designated area designated as the area where the robot performs work is determined, and a second operation plan, which is the operation plan after the robot reaches the reference point, is determined by the robot after the robot reaches the reference point. It is an operation planning method.

[0010] One aspect of the program is, Sets a state in a work space where a robot having a manipulator for handling an object and a self-propelled main body works, Based on the state, constraint conditions regarding the movement of the main body and the operation of the manipulator, and an evaluation function based on the dynamics of the robot, Causes a computer to execute a process of determining an operation plan regarding the movement of the main body and the operation of the manipulator, In the determination of the operation plan, a first operation plan for moving the main body to a reference point within a designated area designated as the area where the robot performs work is determined, and a second operation plan, which is the operation plan based on the state, the constraint conditions, and the evaluation function after the robot reaches the reference point, is determined by the robot after the robot reaches the reference point. It is a program.

[0011] Another aspect of the program is, Receives a designation of an area where a robot having a manipulator for handling an object and a self-propelled main body works, Receives a designation regarding an object within the area 、 Based on the designation of the area and the designation regarding the object, causes a computer to execute a process of determining an operation plan regarding the movement of the main body and the operation of the manipulator 、 In the determination of the operation plan, a first operation plan for moving the main body to a reference point within a designated area designated as the area where the robot performs work is determined, and a second operation plan, which is the operation plan after the robot reaches the reference point, is determined by the robot after the robot reaches the reference point. It is a program.

Advantages of the Invention

[0012] It is possible to suitably determine the operation plan of a mobile robot having a manipulator.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Mode for Carrying Out the Invention

[0014] Hereinafter, embodiments of the motion planning device, the motion planning method, and the storage medium will be described with reference to the drawings.

[0015] <First Embodiment> (1) System Configuration FIG. 1 shows the configuration of a robot control system 100 according to the first embodiment. The robot control system 100 mainly includes a robot controller 1, an instruction device 2, a storage device 4, a robot 5, and a sensor 7.

[0016] When a task to be executed by the robot 5 (also referred to as a "target task") is specified, the robot controller 1 converts the target task into a sequence for each time step (time increment) of a simple task that the robot 5 can accept, and controls the robot 5 based on the generated sequence.

[0017] Also, the robot controller 1 performs data communication with the instruction device 2, the storage device 4, the robot 5, and the sensor 7 via a communication network or by direct communication, either wirelessly or by wire. For example, the robot controller 1 receives an input signal "S1" regarding the specification of the target task from the instruction device 2. Also, the robot controller 1 transmits an output control signal "S2" to the instruction device 2 to cause the instruction device 2 to perform a predetermined display or sound output. Further, the robot controller 1 transmits a control signal "S3" regarding the control of the robot 5 to the robot 5. Also, the robot controller 1 receives a sensor signal "S4" from the sensor 7.

[0018] The instruction device 2 is a device that receives instructions from an operator for the robot 5. The instruction device 2 performs a predetermined display or sound output based on the output control signal S2 supplied from the robot controller 1, or supplies an input signal S1 generated based on the operator's input to the robot controller 1. The instruction device 2 may be a tablet terminal including an input unit and a display unit, or may be a stationary personal computer.

[0019] The memory device 4 has an application information storage unit 41. The application information storage unit 41 stores application information necessary for generating an operation sequence, which is a sequence that the robot 5 should execute, from the target task. Details of the application information will be described later with reference to FIG. 3. The memory device 4 may be an external storage device such as a hard disk connected to or built into the robot controller 1, or may be a storage medium such as a flash memory. Further, the memory device 4 may be a server device that performs data communication with the robot controller 1 via a communication network. In this case, the memory device 4 may be composed of a plurality of server devices.

[0020] The robot 5 is a mobile (self-propelled) robot that performs work related to the target task based on the control signal S3 supplied from the robot controller 1. FIG. 1 shows, as an example, the state in which the robot 5 performs pick and place of the object (workpiece) 6. The robot 5 is, for example, a robot that operates in various factories such as an assembly factory or a food factory, or at a logistics site. The robot 5 has a robot body 50 that moves and a manipulator (here, a robot arm) 52 that has a robot hand (end effector) 53. The manipulator 52 may be a vertically articulated robot, a horizontally articulated robot, or any other type of robot. Hereinafter, the manipulator 52 shall be synonymous with the robot arm. The robot 5 may supply a state signal indicating the state of the robot 5 to the robot controller 1. This state signal may be an output signal of a sensor that detects the state (position, angle, etc.) of the entire robot 5 or a specific part such as a joint, or may be a signal indicating the progress state of the operation sequence of the robot 5 generated by the control unit of the robot 5.

[0021] Note that the robot body 50 and the manipulator 52 of the robot 5 do not necessarily have to be integrated, and the robot body 50 and the manipulator 52 may be separate devices. For example, a configuration in which the manipulator 52 is mounted on a mobile robot corresponding to the robot body 50 may be used.

[0022] The sensor 7 is one or more sensors that are a camera, a ranging sensor, a sonar, or a combination thereof that detect the state within the work space where the target task is executed. In the present embodiment, the sensor 7 is assumed to include at least one camera that images the work space of the robot 5. The sensor 7 supplies the generated sensor signal S4 to the robot controller 1. The sensor 7 may be a self-propelled or flying sensor (including a drone) that moves within the work space. Further, the sensor 7 may include sensors provided on the robot 5 and sensors provided on other objects within the work space. Further, the sensor 7 may include a sensor that detects sound within the work space. Thus, the sensor 7 is various sensors that detect the state within the work space and may include sensors provided at arbitrary locations.

[0023] Note that the configuration of the robot control system 100 shown in FIG. 1 is an example, and various changes may be made to the configuration. For example, there may be a plurality of robots 5, and each may have a plurality of manipulators 52 that operate independently. Even in these cases, the robot controller 1 generates an operation sequence to be executed for each robot 5 (and for each of the robot body 50 and the manipulator 52) based on the target task, and transmits a control signal S3 based on the operation sequence to the target robot 5. Further, the robot 5 may perform collaborative work with other robots, workers, or machine tools operating within the work space. Further, the sensor 7 may be a part of the robot 5. Further, the instruction device 2 may be configured as the same device as the robot controller 1. Further, the robot controller 1 may be composed of a plurality of devices. In this case, the plurality of devices constituting the robot controller 1 exchange information necessary for executing the pre-assigned processing among these plurality of devices. Further, the robot controller 1 and the robot 5 may be integrally configured.

[0024] (2) Hardware Configuration Figure 2(A) shows the hardware configuration of the robot controller 1. The robot controller 1 includes, as hardware, a processor 11, a memory 12, and an interface 13. The processor 11, the memory 12, and the interface 13 are connected via a data bus 10.

[0025] The processor 11 functions as a controller (arithmetic unit) that controls the entire robot controller 1 by executing a program stored in the memory 12. The processor 11 is, for example, a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a TPU (Tensor Processing Unit). The processor 11 may be composed of a plurality of processors. The processor 11 is an example of a computer.

[0026] The memory 12 is composed of various volatile memories and non-volatile memories such as RAM (Random Access Memory), ROM (Read Only Memory), and flash memory. Further, a program for executing the processing executed by the robot controller 1 is stored in the memory 12. Note that a part of the information stored in the memory 12 may be stored by one or more external storage devices (for example, the storage device 4) that can communicate with the robot controller 1, or may be stored by a storage medium detachable from the robot controller 1.

[0027] The interface 13 is an interface for electrically connecting the robot controller 1 and other devices. These interfaces may be wireless interfaces such as network adapters for wirelessly transmitting and receiving data with other devices, or may be hardware interfaces for connecting to other devices via cables or the like.

[0028] Note that the hardware configuration of the robot controller 1 is not limited to the configuration shown in FIG. 2(A). For example, the robot controller 1 may be connected to or incorporated with at least any one of a display device, an input device, or a sound output device. Further, the robot controller 1 may be configured to include at least one of the instruction device 2 or the storage device 4.

[0029] FIG. 2(B) shows the hardware configuration of the instruction device 2. As hardware, the instruction device 2 includes a processor 21, a memory 22, an interface 23, an input unit 24a, a display unit 24b, and a sound output unit 24c. The processor 21, the memory 22, and the interface 23 are connected via a data bus 20. Further, the input unit 24a, the display unit 24b, and the sound output unit 24c are connected to the interface 23.

[0030] The processor 21 executes a predetermined process by executing a program stored in the memory 22. The processor 21 is a processor such as a CPU or a GPU. The processor 21 generates an input signal S1 by receiving a signal generated by the input unit 24a via the interface 23, and transmits the input signal S1 to the robot controller 1 via the interface 23. Further, the processor 21 controls at least one of the display unit 24b or the sound output unit 24c via the interface 23 based on an output control signal S2 received from the robot controller 1 via the interface 23.

[0031] The memory 22 is composed of various volatile memories and non-volatile memories such as RAM, ROM, and flash memory. Further, a program for executing the process executed by the instruction device 2 is stored in the memory 22.

[0032] The interface 23 is an interface for electrically connecting the pointing device 2 and other devices. These interfaces may be wireless interfaces such as network adapters for wirelessly transmitting and receiving data with other devices, or may be hardware interfaces for connecting to other devices via cables or the like. Further, the interface 23 performs interface operations for the input unit 24a, the display unit 24b, and the sound output unit 24c. The input unit 24a is an interface for receiving user input, and examples thereof include a touch panel, buttons, a keyboard, a voice input device, and the like. The display unit 24b is, for example, a display, a projector, or the like, and performs display based on the control of the processor 21. The sound output unit 24c is, for example, a speaker, and performs sound output based on the control of the processor 21.

[0033] Note that the hardware configuration of the pointing device 2 is not limited to the configuration shown in FIG. 2(B). For example, at least one of the input unit 24a, the display unit 24b, or the sound output unit 24c may be configured as a separate device electrically connected to the pointing device 2. Further, the pointing device 2 may be connected to various devices such as a camera, or may incorporate these devices.

[0034] (3) Application Information Next, the data structure of the application information stored in the application information storage unit 41 will be described.

[0035] FIG. 3 shows an example of the data structure of the application information. As shown in FIG. 3, the application information includes abstract state designation information I1, constraint condition information I2, operation limit information I3, subtask information I4, abstract model information I5, object model information I6, and map information I7.

[0036] The abstract state specification information I1 is information that specifies the abstract states that need to be defined when generating an operation sequence. This abstract state is an abstract state of an object in the work space and is defined as a proposition used in the target logical formula described later. For example, the abstract state specification information I1 specifies the abstract states that need to be defined for each type of target task.

[0037] The constraint condition information I2 is information indicating the constraint conditions when executing the target task. The constraint condition information I2 indicates, for example, when the target task is pick-and-place, constraint conditions such as the robot 5 must not touch an obstacle, and the robots 5 (manipulators) must not touch each other. Note that the constraint condition information I2 may be information that records the constraint conditions suitable for each type of target task.

[0038] The motion limit information I3 indicates information regarding the motion limits of the robot 5 controlled by the robot controller 1. The motion limit information I3 is, for example, information that defines the upper limits of the speed, acceleration, or angular velocity of the robot 5. Further, the motion limit information I3 includes information that defines the motion limits for each movable part (including the robot body 50) or joint of the robot 5. In the present embodiment, the motion limit information I3 includes information regarding the reach range of the manipulator 52. The reach range of the manipulator 52 is the range in which the manipulator 52 can perform work on the object 6 and may be indicated by the maximum distance from the reference position of the robot 5, or may be indicated by a region in the coordinate system with the robot 5 as the reference.

[0039] Subtask information I4 indicates information on subtasks that are components of the operation sequence. A "subtask" is a task obtained by decomposing the target task into units that the robot 5 can accept, and refers to the subdivided operations of the robot 5. For example, when the target task is pick and place, the subtask information I4 defines reaching, which is the movement of the manipulator 52, and grasping, which is the gripping by the manipulator 52, as subtasks. Also, at least the subtasks corresponding to the movement of the robot body 50 are defined in the subtask information I4. The subtask information I4 may indicate information on subtasks that can be used for each type of target task. Note that the subtask information I4 may include information on subtasks that require an operation command by external input.

[0040] The abstract model information I5 is information on an abstract model that abstracts the dynamics in the work space. For example, as will be described later, the abstract model is represented by a model that abstracts the actual dynamics by a hybrid system. The abstract model information I5 includes information indicating the conditions for switching the dynamics in the above hybrid system. The switching conditions include, for example, in the case of pick and place where the robot 5 grasps the object 6 to be the work target and moves it to a predetermined position, the condition that the object 6 cannot move unless it is gripped by the robot 5. The abstract model information I5 has information on an abstract model suitable for each type of target task.

[0041] The object model information I6 is information regarding the object models of each object in the work space to be recognized from the sensor signal S4 generated by the sensor 7. Each of the above-mentioned objects corresponds to, for example, the robot 5, an obstacle, a tool handled by the robot 5 or other objects, a working body other than the robot 5, and the like. The object model information I6 includes, for example, information necessary for the robot controller 1 to recognize the type, position, posture, currently executing operation, etc. of each of the above-mentioned objects, and 3D shape information such as CAD (Computer Aided Design) data for recognizing the 3D shape of each object. The former information includes the parameters of an inference device obtained by training a learning model in machine learning such as a neural network. This inference device is pre-trained, for example, to output the type, position, posture, etc. of the object that is the subject in the image when an image is input.

[0042] The map information I7 is information indicating the map (layout) of the work space in which the robot 5 exists. The map information I7 includes, for example, information indicating the movable range (passage) in the work space, the positions of fixed objects such as walls, installations, stairs (including the size and range), and information indicating obstacles.

[0043] In addition, the application information storage unit 41 may store various information related to the generation process of the operation sequence and the generation process of the output control signal S2 in addition to the above-mentioned information.

[0044] (4) Processing Overview Next, the outline of the processing of the robot controller 1 will be described. Generally speaking, after moving the robot main body 50 to the area specified based on the input signal S1 (also referred to as the "specified area"), the robot controller 1 simultaneously executes the motion plan regarding the movement of the robot main body 50 and the motion plan of the manipulator 52. Thereby, the robot controller 1 efficiently and surely causes the robot 5 to perform the target task.

[0045] FIG. 4 is an example of a functional block showing an outline of the processing of the robot controller 1. Functionally, the processor 11 of the robot controller 1 includes an output control unit 15, an operation planning unit 16, and a robot control unit 17. In FIG. 4, an example of data exchanged between the blocks is shown, but it is not limited thereto. The same applies to the diagrams of other functional blocks described later.

[0046] Based on the map information I7 and the like, the output control unit 15 generates an output control signal S2 for displaying or audibly outputting predetermined information to the instruction device 2 used by the operator, and transmits the output control signal S2 to the instruction device 2 via the interface 13.

[0047] For example, the output control unit 15 generates an output control signal S2 for causing the instruction device 2 to display an input screen (also referred to as a "task specification screen") regarding the specification of the target task. Thereafter, the output control unit 15 receives the input signal S1 generated by the instruction device 2 through the interface 13 from the instruction device 2 in response to an input operation on the task specification screen. In this case, the input signal S1 includes information (also referred to as "task specification information Ia") for roughly specifying the target task. The task specification information Ia is, for example, information corresponding to a rough command to the robot 5 and does not include information for defining the specific operation of the robot 5 (for example, information on control inputs and subtasks described later). In the present embodiment, the task specification information Ia includes at least information representing a specified area specified by the user as the space in which the robot 5 performs work and information regarding the object 6 (for example, an object to be grasped) specified by the user. The information regarding the object 6 may include the position of the object serving as the object 6 and information indicating the transport destination (goal point) for transporting each object serving as the object 6.

[0048] Also, preferably, the task specification information Ia may further include information specifying priorities in the execution of the target task. Here, the priorities indicate matters to be emphasized in the execution of the target task, such as safety priority, long working time priority, power consumption priority, etc. The priorities are selected by the user, for example, on the task specification screen. Specific examples of the specified area, the object 6, and the specification method for the priorities will be described later with reference to FIGS. 11 and 12.

[0049] Then, the output control unit 15 supplies the task specification information Ia based on the input signal S1 supplied from the instruction device 2 to the operation planning unit 16.

[0050] The operation planning unit 16 generates an operation sequence to be executed by the robot 5 based on the task specification information Ia supplied from the output control unit 15, the sensor signal S4, and the application information stored in the storage device 4. The operation sequence corresponds to a sequence of subtasks (subtask sequence) that the robot 5 should execute to achieve the target task and defines a series of operations of the robot 5. In the present embodiment, the operation planning unit 16 formulates a first operation plan to move the robot 5 to the specified area indicated by the task specification information Ia and a second operation plan to complete the target task for the robot 5 after the robot 5 arrives at the specified area, respectively. Then, the operation planning unit 16 generates a first operation sequence "Sr1" based on the first operation plan and a second operation sequence "Sr2" based on the second operation plan. Then, the operation planning unit 16 sequentially supplies the generated first operation sequence Sr1 and the second operation sequence Sr2 to the robot control unit 17. Here, each operation sequence includes information indicating the execution order and execution timing of each subtask.

[0051] The robot control unit 17 controls the operation of the robot 5 by supplying a control signal S3 to the robot 5 via the interface 13. When the robot control unit 17 receives an operation sequence from the operation planning unit 16, it performs control for the robot 5 to execute each subtask constituting the operation sequence at a respectively determined execution timing (time step). Specifically, the robot control unit 17 executes position control or torque control of the joints of the robot 5 for realizing the operation sequence by transmitting the control signal S3 to the robot 5.

[0052] Note that the robot 5 may have a function corresponding to the robot control unit 17 instead of the robot controller 1. In this case, the robot 5 executes an operation based on the operation sequence generated by the operation planning unit 16. Also, the robot control unit 17 may be provided separately by the robot body 50 and the manipulator 52. In this case, for example, the robot control unit 17 may be configured such that one of them exists in the robot controller 1 and the other exists in the robot 5.

[0053] Here, each component of the output control unit 15, the operation planning unit 16, and the robot control unit 17 can be realized, for example, by the processor 11 executing a program. Also, by recording the necessary program in an arbitrary non-volatile storage medium and installing it as needed, each component may be realized. Note that at least a part of each of these components is not limited to being realized by software based on a program, and may be realized by any combination of hardware, firmware, and software. Further, at least a part of each of these components may be realized using a user-programmable integrated circuit such as an FPGA (Field-Programmable Gate Array) or a microcontroller. In this case, a program composed of the above-described components may be realized using this integrated circuit. Also, at least a part of each component may be composed of an ASSP (Application Specific Standard Produce), an ASIC (Application Specific Integrated Circuit), or a quantum computer control chip. Thus, each component may be realized by various hardware. The above also applies to other embodiments described later. Furthermore, each of these components may be realized by the cooperation of a plurality of computers using, for example, cloud computing technology or the like.

[0054] (5) Details of the Operation Planning Unit Next, the detailed processing of the operation planning unit 16 will be described.

[0055] (5-1) Functional Blocks FIG. 5 is an example of a functional block diagram showing the functional configuration of the operation planning unit 16. Functionally, the operation planning unit 16 includes a path setting unit 30, an abstract state setting unit 31, a target logical formula generation unit 32, a time step logical formula generation unit 33, an abstract model generation unit 34, a control input generation unit 35, and a subtask sequence generation unit 36.

[0056] In the first operation plan, the route setting unit 30 sets the route of the robot 5 to the point (also referred to as the "reference point") that the robot 5 should reach within the designated area represented by the task designation information Ia. The reference point is set to be an appropriate position as the operation start position of the robot 5 in the second operation plan that generates the second operation sequence Sr2. In other words, the route setting unit 30 sets the reference point so that it becomes the operation start position of the robot 5 suitable for formulating the second operation plan. The route setting unit 30 sets the reference point based on, for example, the designated area indicated by the task designation information Ia and the position of the object 6, the self-position estimated based on the sensor signal S4 output by the sensor 7 provided on the robot 5, and the reach range of the robot 5 (specifically, the manipulator 52) indicated by the operation limit information I3. Further, the route setting unit 30 determines the route from the self-position to the reference point (also referred to as the "robot route") based on the self-position obtained by self-position estimation, the reference point, and the map information I7. Note that the self-position estimation may be based on a GPS receiver and an internal sensor, or may be based on a camera or other external sensors (including SLAM (Simultaneous Localization and Mapping), etc.). Specific examples of setting the reference point and the route will be described later. The route setting unit 30 supplies information indicating the set robot route (also referred to as the "route information Irt") to the subtask sequence generation unit 36.

[0057] When formulating the second operation plan, the abstract state setting unit 31 sets the abstract state of an object or the like related to the target task based on the sensor signal S4 supplied from the sensor 7, the task designation information Ia supplied from the output control unit 15, the abstract state designation information I1, and the object model information I6. In this case, the abstract state setting unit 31 first determines a space for setting the abstract state (also referred to as the "abstract state setting space"), recognizes an object that needs to be considered when executing the target task within the abstract state setting space, and generates a recognition result "Im" regarding the object. Note that the abstract state setting space may be the entire designated area, or may be set to a rectangular area or the like that at least includes the position of the object 6 designated in the task designation information Ia, the position of the destination, and the robot 5 within the designated area. Then, based on the recognition result Im, the abstract state setting unit 31 defines a proposition for expressing in a logical formula for each abstract state that needs to be considered when executing the target task. The abstract state setting unit 31 supplies information indicating the set abstract state (also referred to as the "abstract state setting information IS") to the target logical formula generation unit 32.

[0058] When formulating the second operation plan, based on the abstract state setting information IS, the target logical formula generation unit 32 converts the target task within the designated area indicated by the task designation information Ia into a logical formula of temporal logic representing the final achievement state (also referred to as the "target logical formula Ltag"). In this case, the target logical formula generation unit 32 refers to the constraint condition information I2 from the application information storage unit 41 and adds the constraint conditions that should be satisfied in the execution of the target task within the designated area to the target logical formula Ltag. Then, the target logical formula generation unit 32 supplies the generated target logical formula Ltag to the time step logical formula generation unit 33.

[0059] When formulating the second operation plan, the time step logical formula generation unit 33 converts the target logical formula Ltag supplied from the target logical formula generation unit 32 into a logical formula representing the state at each time step (also referred to as the "time step logical formula Lts"). Then, the time step logical formula generation unit 33 supplies the generated time step logical formula Lts to the control input generation unit 35.

[0060] When formulating the second operation plan, the abstract model generation unit 34 generates an abstract model "Σ" that abstracts the actual dynamics in the abstract state setting space based on the abstract model information I5 stored in the application information storage unit 41 and the recognition result Im supplied from the abstract state setting unit 31. In this case, the abstract model generation unit 34 regards the target dynamics as a hybrid system in which continuous dynamics and discrete dynamics coexist, and generates an abstract model Σ based on the hybrid system. The method for generating the abstract model Σ will be described later. The abstract model generation unit 34 supplies the generated abstract model Σ to the control input generation unit 35.

[0061] When formulating the second operation plan, the control input generation unit 35 generates a control input for each time step to the robot 5 (that is, the trajectory information of the robot 5). The control input generation unit 35 satisfies the time step logical formula Lts supplied from the time step logical formula generation unit 33 and the abstract model Σ supplied from the abstract model generation unit 34, and determines the control input to the robot 5 for each time step that optimizes an evaluation function (for example, a function representing the amount of energy consumed by the robot). In this case, the evaluation function is a function that indirectly or directly considers the dynamics of the robot 5 represented by the abstract model Σ. Then, the control input generation unit 35 supplies information indicating the control input to the robot 5 for each time step (also referred to as "control input information Icn") to the subtask sequence generation unit 36.

[0062] The subtask sequence generation unit 36 generates a first motion sequence Sr1 when formulating a first motion plan, and generates a second motion sequence Sr2 when formulating a second motion plan. When the path information Irt is supplied from the path setting unit 30 at the time of formulating the first motion plan, the subtask sequence generation unit 36 generates a first motion sequence Sr1, which is a sequence of subtasks that instructs the movement of the robot 5 along the robot path indicated by the path information Irt based on the subtask information I4. On the other hand, when formulating the second motion plan, the subtask sequence generation unit 36 generates a second motion sequence Sr2 related to the movement of the robot main body 50 and the operation of the manipulator 52 based on the control input information Icn supplied from the control input generation unit 35 and the subtask information I4. The subtask sequence generation unit 36 sequentially supplies the generated motion sequence to the robot control unit 17.

[0063] (5-2) Route Setting Unit The process executed by the path setting unit 30 will be specifically described. FIG. 6 is an example of a schematic diagram showing the work space of the robot 5 from an overhead view.

[0064] In the example shown in FIG. 6, in the work space, there are a robot 5, a workbench 54, and other workbenches 54A to 54E other than the workbench 54. On the workbench 54, there are an object 61a (also referred to as "object A") and an object 61b (also referred to as "object B"). Further, beside the workbench 54, there is a second workbench 63 (also referred to as "destination G") which is the conveyance destination of the object A and the object B. Furthermore, for convenience of explanation, in FIG. 6, a speech balloon 56 representing a schematic command specified by the user for the robot 5 is clearly shown. The schematic command here designates an "area I" with a broken line frame 55 as the outer edge, and instructs to convey one object A (i.e., the object 61a) and one object B (i.e., the object 61b) within the area I to the destination G (i.e., the table 63).

[0065] Note that the destination G is not limited to being within the same area I. When the destination G is designated as another area outside the area I (also referred to as "other area"), for example, the robot controller 1 places the object existing in the area I on the robot 5 once and then moves the robot 5 to the other area, and then formulates an operation plan to place the object on the designated destination G in the other area on the robot 5. Such an operation of the robot 5 is realized by generating a sequence of operations of moving to the area again by the first operation plan and placing on the destination by the second operation plan after the operation based on the second operation plan of placing the object on the robot 5 once is completed (that is, by alternately repeating the first operation plan and the second operation plan).

[0066] In this case, the route setting unit 30 first sets a reference point where the robot 5 should move within the area I which is the designated area. In this case, for example, the route setting unit 30 sets the reference point at the position closest to the robot 5 (or the position where the robot 5 can reach the fastest) among the positions at a predetermined distance from the object (here, object B) close to the robot 5 within the area I. Here, the above-mentioned predetermined distance is set to a value obtained by adding the reach range (or a predetermined multiple thereof) of the robot 5 (specifically, the manipulator 52) and the movement error of the robot 5. The above-mentioned movement error of the robot 5 is, specifically, the sum of the error of the self-position estimation accuracy obtained in the process of performing self-position estimation and the error related to the movement control of the robot 5, and this error information may be stored in advance in the storage device 4, for example. Note that, as the error information of the self-position estimation, the error information obtained at the time of self-position estimation may be used.

[0067] In this way, the route setting unit 30 determines the reference point based on the position of the object, the movement error of the robot 5, and the reach range of the robot 5. Thereby, the route setting unit 30 can determine the reference point at an appropriate position as the operation start point of the robot 5 in the second operation plan.

[0068] Here, a supplementary explanation will be given regarding the necessity of setting an appropriate reference point. Generally, if the reference point is too close to the object 6, it is impossible to calculate the optimal second operation sequence Sr2 that defines the movement of the robot body 50 and the operation of the manipulator 52 respectively. On the other hand, if the reference point is too far from the object 6, the amount of calculation for calculating the second operation sequence Sr2 will increase. Considering the above, the path setting unit 30 determines a reference point that minimizes the movement of the robot body 50 in the second operation plan based on the position of the object, the movement error of the robot 5, and the reach range of the robot 5. Thereby, while preventing the above-mentioned increase in the amount of calculation, it is possible to preferably formulate the movement of the robot 5 and the operation plan of the manipulator 52.

[0069] In addition, the path setting unit 30 determines the robot path as the path that can reach the reference point in the shortest time length without the robot 5 colliding with the obstacles (here, other workbenches 54A to 54E, etc.) recorded in the map information I7. In this case, the path setting unit 30 may determine the robot path based on an arbitrary path search method.

[0070] FIG. 7 is a diagram showing the reference point and the robot path set by the path setting unit 30. In FIG. 7, the point 57 indicates the reference point, and the path line 58 indicates the robot path. The path setting unit 30 determines the reference point and the robot path shown in FIG. 7, and supplies the path information Irt indicating the determined robot path to the subtask sequence generation unit 36. Thereafter, the subtask sequence generation unit 36 generates the first operation sequence Sr1 based on the path information Irt. Then, the robot control unit 17 supplies the control signal S3 based on the first operation sequence Sr1 to the robot 5 to control the robot 5 to move along the robot path.

[0071] (5-3) Abstract State Setting Unit Next, the processing of the abstract state setting unit 31 will be described. When the abstract state setting unit 31 determines that the operation of the robot 5 based on the first operation sequence Sr1 has been completed (i.e., the robot 5 has reached the reference point), it performs the setting of the abstract state setting space, the generation of the recognition result Im, and the setting of the abstract state. In this case, for example, when the abstract state setting unit 31 receives a completion notification of the first operation sequence Sr1 from the robot 5 or the robot control unit 17, it determines that the robot 5 has reached the reference point. In other examples, the abstract state setting unit 31 may determine whether the robot 5 has reached the reference point based on a sensor signal S4 such as a camera image.

[0072] In the generation of the recognition result Im, the abstract state setting unit 31 refers to the object model information I6 and analyzes the sensor signal S4 by a technique (such as an image processing technique, an image recognition technique, a voice recognition technique, a technique using RFID (Radio Frequency Identifier), etc.) for recognizing the environment of the abstract state setting space. Thereby, the abstract state setting unit 31 generates information such as the type, position, and posture of the objects in the abstract state setting space as the recognition result Im. Also, the objects in the abstract state setting space are, for example, the robot 5, objects such as tools or parts handled by the robot 5, obstacles, and other working bodies (persons or other objects that perform work other than the robot 5).

[0073] Next, the abstract state setting unit 31 sets the abstract state in the abstract state setting space based on the recognition result Im and the abstract state designation information I1 obtained from the application information storage unit 41. In this case, first, the abstract state setting unit 31 refers to the abstract state designation information I1 and recognizes the abstract state to be set in the abstract state setting space. Note that the abstract state to be set in the abstract state setting space differs depending on the type of the target task. Therefore, when the abstract state to be set for each type of the target task is defined in the abstract state designation information I1, the abstract state setting unit 31 refers to the abstract state designation information I1 corresponding to the target task to be currently executed and recognizes the abstract state to be set.

[0074] FIG. 8 shows a bird's-eye view of the abstract state setting space 59 set by the abstract state setting unit 31. In the abstract state setting space 59 shown in FIG. 8, there are a robot 5 with two manipulators 52a and 52b attached to the robot body 50, a workbench 54 on which there are two objects 61 (object A and object B) and an obstacle 62, and a second workbench 63 (destination G) which is the conveyance destination of the object 61.

[0075] In this case, first, the abstract state setting unit 31 recognizes the existence range of the workbench 54, the states of the object A and the object B, the existence range of the obstacle 62, the state of the robot 5, the existence range of the destination G, and the like. Then, the abstract state setting unit 31 expresses the positions of the recognized elements using, for example, a coordinate system based on the set abstract state setting space 59. For example, the reference of the coordinate system in this abstract state setting space 59 can also be the aforementioned reference point, that is, the start point of the second operation plan.

[0076] Here, the abstract state setting unit 31 recognizes the position vectors "x 1 ", "x 2 " of the centers of the object A and the object B as the positions of the object A and the object B. Further, the abstract state setting unit 31 recognizes the position vector "x r1 " of the robot hand 53a that grips the object and the position vector "x r2 " of the robot hand 53b as the positions of the manipulator 52a and the manipulator 52b. Furthermore, the abstract state setting unit 31 recognizes the position vector "x r " of the robot body 50 as the position of the robot body 50. In this case, the position vector x r1 and the position vector x r2 may be relative vectors based on the position vector x r of the robot body 50. In this case, the position of the manipulator 52a in the coordinate system of the abstract state setting space 59 is represented by "x r + x r1 ", and the position of the manipulator 52b in the coordinate system of the abstract state setting space 59 is represented by the position vector "x r + x r2 ".

[0077] Similarly, the abstract state setting unit 31 recognizes the postures of the objects A and B, the existence range of the obstacle 62, the existence range of the destination G, and the like. For example, when the abstract state setting unit 31 regards the obstacle 62 as a rectangular parallelepiped and the destination G as a rectangle, the abstract state setting unit 31 recognizes the position vectors of the respective vertices of the obstacle 62 and the destination G.

[0078] The abstract state set by the abstract state setting unit 31 is represented by, for example, the following abstract state vector "z". z T =(x r ,x r1 ,x r2 ,x 1 ,x 2 ,…) In addition to the abstract states of the manipulators 52a and 52b, the abstract state vector also includes the abstract state of the robot body 50. Therefore, the abstract state vector z is also called an extended state vector. Then, by the optimization process of the control input generation unit 35 using such an abstract state vector z, the trajectory information of the manipulators 52a and 52b and the robot body 50 for each time step is preferably generated.

[0079] In addition, the abstract state setting unit 31 determines the abstract state to be defined in the target task by referring to the abstract state designation information I1. In this case, the abstract state setting unit 31 defines a proposition indicating the abstract state based on the recognition result Im (for example, the number for each type of object) regarding the objects existing in the abstract state setting space and the abstract state designation information I1.

[0080] In the example of FIG. 8, the abstract state setting unit 31 assigns identification labels "1" and "2" to the objects A and B specified by the recognition result Im, respectively. Further, the abstract state setting unit 31 defines a proposition "g i " that the object "i" (i = 1 to 2) exists in the destination G, which is the target point where the object should be finally placed. Further, the abstract state setting unit 31 assigns an identification label "O" to the obstacle 62, and a proposition "o iDefine ". Furthermore, the abstract state setting unit 31 defines the proposition "h" that the manipulators 52 interfere with each other. The proposition "h" may also include the condition that the manipulator 52 and the robot body 50 do not interfere. Note that the abstract state setting unit 31 defines the proposition "v" that the object "i" exists within the workbench 54 (a table where the object and obstacles exist in the initial state). i ", and the proposition "w" that the object exists in the non - working area other than the workbench 54 and the transfer destination G. i ". etc. may be further defined. The non - working area is, for example, the area (such as the floor surface) where the object exists when the object falls from the workbench 54.

[0081] In this way, by referring to the abstract state designation information I1, the abstract state setting unit 31 recognizes the abstract state to be defined, and defines the propositions (in the above example, g i , o i , h, etc.) representing the abstract state according to the number of objects 61, the number of manipulators 52, the number of obstacles 62, the number of robots 5 (manipulators 52), etc. Then, the abstract state setting unit 31 supplies the information indicating the proposition representing the abstract state to the target logical formula generation unit 32 as the abstract state setting information IS.

[0082] (5 - 4) Goal Logic Formula Generation Unit First, the target logical formula generation unit 32 converts the target task indicated by the task designation information Ia into a logical formula using temporal logic.

[0083] For example, in the example of FIG. 8, assume that the target task of "finally, the object (i = 2) exists at the transfer destination G" is given. In this case, the target logical formula generation unit 32 uses the operator "◇" corresponding to "eventually" in linear temporal logic (LTL: Linear Temporal Logic) for the target task, and the proposition "g" defined by the abstract state setting unit 31 i ", and uses them to form the logical formula "◇g 2It generates "」. Further, the target logical formula generation unit 32 may represent a logical formula using any temporal logic operator other than the operator "◇" (logical product "∧", logical sum "∨", negation "￢", logical inclusion "⇒", always "□", next "○", until "U", etc.). Also, not limited to linear temporal logic, a logical formula may be represented using any temporal logic such as MTL (Metric Temporal Logic) or STL (Signal Temporal Logic).

[0084] Note that the task specification information Ia may be information that specifies the target task in natural language. There are various techniques for converting a task represented in natural language into a logical formula.

[0085] Next, the target logical formula generation unit 32 generates a target logical formula Ltag by adding the constraint conditions indicated by the constraint condition information I2 to the logical formula indicating the target task.

[0086] For example, as constraint conditions corresponding to the pick-and-place shown in FIG. 8, if "the manipulators 52 never interfere with each other" and "the object i never interferes with the obstacle O" are included in the constraint condition information I2, the target logical formula generation unit 32 converts these constraint conditions into logical formulas. Specifically, the target logical formula generation unit 32 uses the proposition "o i " and the proposition "h" defined by the abstract state setting unit 31 in the description of FIG. 8 to convert the above two constraint conditions into the following logical formulas respectively. □￢h ∧ i □￢o i

[0087] Therefore, in this case, the target logical formula generation unit 32 adds the logical formulas of these constraint conditions to the logical formula "◇g 2 " corresponding to the target task of "finally, the object (i = 2) exists at the destination G" to generate the following target logical formula Ltag. (◇g 2 )∧(□￢h)∧(∧ i □￢o i )

[0088] In practice, the constraint conditions corresponding to pick-and-place are not limited to the two described above, and there are also constraint conditions such as "the manipulator 52 does not interfere with the obstacle O", "multiple manipulators 52 do not grasp the same object", and "the objects do not contact each other". Similarly, such constraint conditions are stored in the constraint condition information I2 and reflected in the target logical formula Ltag.

[0089] (5-5) Time Step Logic Formula Generation Unit The time step logical formula generation unit 33 determines the number of time steps (also referred to as "target time step number") to complete the target task, and determines a combination of propositions representing the state at each time step that satisfies the target logical formula Ltag with the target time step number. Since there are usually multiple such combinations, the time step logical formula generation unit 33 generates a logical formula obtained by combining these combinations by logical sum as the time step logical formula Lts. The above combination is a candidate for the logical formula representing the sequence of operations commanded to the robot 5, and will also be referred to as "candidate φ" hereinafter.

[0090] Here, a specific example of the processing of the time step logical formula generation unit 33 when the target task of "finally the object (i = 2) exists at the destination G" exemplified in the description of FIG. 8 is set will be described.

[0091] In this case, the following target logical formula Ltag is supplied from the target logical formula generation unit 32 to the time step logical formula generation unit 33. (◇g 2 )∧(□￢h)∧(∧ i □￢o i ) In this case, the time step logical formula generation unit 33 uses the proposition "g i " extended to include the concept of time step, that is, the proposition "g i,k ". Here, the proposition "g i,k " is the proposition that "the object i exists at the destination G at time step k". Here, when the target time step number is "3", the target logical formula Ltag is rewritten as follows. (◇g 2,3 )∧(∧ k=1,2,3 □￢h k )∧(∧ i,k=1,2,3 □￢o i,k )

[0092] Also, ◇g 2,3 can be rewritten as shown in the following formula.

[0093]

Number

[0094] At this time, the above-mentioned target logical formula Ltag is represented by the logical sum (φ 1 ” ~ “φ 4 ) of the following four candidates “φ 1 ∨φ 2 ∨φ 3 ∨φ 4 ).

[0095]

Number

[0096] Therefore, the time step logical formula generation unit 33 determines the logical sum of the four candidates φ 1 ~φ 4 as the time step logical formula Lts. In this case, the time step logical formula Lts becomes true when at least any one of the four candidates φ 1 ~φ 4 becomes true.

[0097] Next, a supplementary explanation will be given about the method for setting the target time step number.

[0098] The time step logic formula generation unit 33 determines the target number of time steps based on, for example, the expected time of the operation specified by the input signal S1 supplied from the pointing device 2. In this case, the time step logic formula generation unit 33 calculates the target number of time steps from the above-described expected time based on the information on the time width per time step stored in the memory 12 or the storage device 4. In another example, the time step logic formula generation unit 33 stores in advance in the memory 12 or the storage device 4 information associating the target number of time steps suitable for each type of target task, and refers to the information to determine the target number of time steps corresponding to the type of target task to be executed.

[0099] Preferably, the time step logic formula generation unit 33 sets the target number of time steps to a predetermined initial value. Then, the time step logic formula generation unit 33 gradually increases the target number of time steps until the time step logic formula Lts for which the control input generation unit 35 can determine the control input is generated. In this case, when the time step logic formula generation unit 33 cannot derive an optimal solution as a result of the optimization process performed by the control input generation unit 35 with the set target number of time steps, the time step logic formula generation unit 33 adds a predetermined number (an integer of 1 or more) to the target number of time steps.

[0100] At this time, the time step logic formula generation unit 33 may set the initial value of the target number of time steps to a value smaller than the number of time steps corresponding to the operation time of the target task expected by the user. Thereby, the time step logic formula generation unit 33 preferably suppresses setting an unnecessarily large target number of time steps.

[0101] (5-6) Abstract Model Generation Unit The abstract model generation unit 34 generates an abstract model Σ based on the abstract model information I5 and the recognition result Im. Here, in the abstract model information I5, for each type of target task, information necessary for generating the abstract model Σ is recorded. For example, when the target task is pick and place, a general - purpose abstract model that does not specify the position and number of objects, the position of the area where the objects are to be placed, the number of robots 5 (or the number of manipulators 52), etc. is recorded in the abstract model information I5. Also, in the abstract model information I5, an abstract model related to the movement of the robot main body 50 is recorded. Then, the abstract model generation unit 34 reflects the recognition result Im on the general - purpose abstract model recorded in the abstract model information I5, which includes the dynamics of the robot 5 (specifically, the dynamics related to the movement of the robot main body 50 and the dynamics of the manipulator 52), to generate the abstract model Σ. As a result, the abstract model Σ becomes a model in which the state of the object in the abstract state - setting space and the dynamics of the robot 5 are abstractly represented. The state of the object in the abstract state - setting space indicates, in the case of pick and place, the position and number of the object, the position of the area where the object is to be placed, the number of robots 5 (manipulators 52), etc.

[0102] In addition, when there are other working bodies, information regarding the abstracted dynamics of the other working bodies may be included in the abstract model information I5. In this case, the abstract model Σ becomes a model in which the state of the object in the abstract state - setting space, the dynamics of the robot 5, and the dynamics of the other working bodies are abstractly represented.

[0103] Here, during the operation of the target task by the robot 5, the dynamics in the abstract state - setting space frequently change. For example, in pick and place, when the manipulator 52 is grasping the object i, the object i can be moved, but when the manipulator 52 is not grasping the object i, the object i cannot be moved.

[0104] Taking the above into consideration, in this embodiment, the operation of grasping the object i is represented by the logical variable "δ" iIt is abstractly represented by "". In this case, for example, the abstract model generation unit 34 can define the abstract model Σ to be set for the abstract state setting space shown in FIG. 8 by the following formula (1).

[0105] [Number]

[0106] Here, "u 0 " represents the control input for controlling the robot body 50, and "u 1 " represents the control input for controlling the manipulator 52a, and "u 2 " represents the control input for controlling the manipulator 52b. Also, "I" represents the identity matrix, and "0" represents the zero row example. Note that the control input is assumed to be speed here as an example, but it may also be acceleration. Also, "δ j,i " is a logical variable that is "1" when the manipulator j ("j = 1" represents the manipulator 52a, "j = 2" represents the manipulator 52b) is grasping the object i, and "0" in other cases. Also, "δ 12,i " is a logical variable that becomes "1" when the object i moves due to the movement of the robot body 50 when the manipulator 1 or the manipulator 2 is grasping the object i, and "0" in other cases. Also, "x r1 ", "x r2 " are the position vectors of the manipulator j (j = 1, 2), and "x 1 ", "x 2 " represent the position vectors of the object i (i = 1, 2). Note that here, the coordinates x r1 , x r2 of the manipulator 52 and the coordinates x1, x2 of the object i are represented in the reference coordinate system within the abstract state setting space, and when the robot body 50 moves (i.e., by "u 0 "), the positions of the manipulator 52 and the object i grasped by the manipulator 52 also move.

[0107] Also, "h(x)" is a variable that becomes "h(x) ≥ 0" when a robot hand (i.e., the position vector x of the manipulator 52 r1 or x r2 ) exists in the vicinity of the object to the extent that the object can be grasped, and satisfies the following relationship with the logical variable δ. δ = 1 ⇔ h(x) ≥ 0 In this equation, when a robot hand exists in the vicinity of the object to the extent that the object can be grasped, it is considered that the robot hand is grasping the object, and the logical variable δ is set to "1".

[0108] Here, Equation (1) is a difference equation showing the relationship between the state of the object at time step k and the state of the object at time step k + 1. In the above Equation (1), the grasping state is represented by a logical variable with discrete values, and the movement of the object is represented by continuous values. Therefore, Equation (1) represents a hybrid system.

[0109] Equation (1) considers only the dynamics of the robot hand, which is the tip of the robot 5 that actually grasps the object, and the dynamics of the robot body 50, rather than the detailed dynamics of the entire robot 5. This can preferably reduce the computational amount of the optimization process by the control input generation unit 35.

[0110] Also, the abstract model information I5 records a logical variable corresponding to an operation in which the dynamics switches (in the case of pick and place, the operation of grasping the object i), and information for deriving the difference equation of Equation (1) from the recognition result Im. Therefore, even when the position and number of objects, the area where the objects are placed (the destination G in FIG. 8), the number of robots 5, etc. vary, the abstract model generation unit 34 can determine the abstract model Σ according to the environment of the target abstract state setting space based on the abstract model information I5 and the recognition result Im.

[0111] Note that, instead of the model shown in Equation (1), the abstract model generation unit 34 may generate a model of a hybrid system that combines a Mixed Logical Dynamical (MLD) system, a Petri net, an automaton, or the like.

[0112] (5-7) Control Input Generation Unit Based on the time step logical formula Lts supplied from the time step logical formula generation unit 33 and the abstract model Σ supplied from the abstract model generation unit 34, the control input generation unit 35 determines the optimal control input for the robot 5 for each time step. In this case, the control input generation unit 35 defines an evaluation function for the target task and solves an optimization problem that minimizes the evaluation function with the abstract model Σ and the time step logical formula Lts as constraint conditions. The evaluation function is predefined for each type of target task, for example, and is stored in the memory 12 or the storage device 4.

[0113] For example, the control input generation unit 35 sets the distance "d k " between the object to be carried and the target point to carry the object and the control input "u k " to be minimized (i.e., to minimize the energy consumed by the robot 5). The above-mentioned distance d k corresponds to the distance between the object (i = 2) and the destination G at time step k in the case of the target task of "finally, the object (i = 2) exists at the destination G".

[0114] In this case, the control input generation unit 35 defines the sum of the square of the norm of the distance d k at all time steps and the square of the norm of the control input u k as the evaluation function. Then, the control input generation unit 35 solves the following constrained mixed-integer optimization problem shown in Equation (2) with the abstract model Σ and the time step logical formula Lts (i.e., the logical sum of the candidate φ i ) as constraint conditions.

[0115]

Equation

[0116] Here, "T" is the number of time steps to be optimized, which may be the target number of time steps or, as will be described later, a predetermined number smaller than the target number of time steps. In this case, preferably, the control input generation unit 35 approximates the logical variable to a continuous value (regards it as a continuous relaxation problem). Thereby, the control input generation unit 35 can preferably reduce the computational amount. When STL is adopted instead of the linear logic formula (LTL), it can be described as a non-linear optimization problem.

[0117] Also, when the target number of time steps is long (for example, when it is larger than a predetermined threshold value), the control input generation unit 35 may set the number of time steps used for optimization to a value smaller than the target number of time steps (for example, the above-mentioned threshold value). In this case, the control input generation unit 35, for example, solves the above-mentioned optimization problem every time a predetermined number of time steps elapses, and sequentially determines the control input u k to be determined.

[0118] Preferably, the control input generation unit 35 solves the above-mentioned optimization problem for each predetermined event corresponding to an intermediate state with respect to the achievement state of the target task, and determines the control input u k to be used. In this case, the control input generation unit 35 sets the number of time steps until the next event occurs to the number of time steps used for optimization. The above-mentioned event is, for example, an event in which the dynamics in the abstract state setting space switches. For example, when the pick-and-place is the target task, events such as the robot 5 grasping the object and the robot 5 finishing transporting one of the plurality of objects to be transported to the target point are defined. Events are, for example, predetermined in advance for each type of target task, and information for specifying events for each type of target task is stored in the storage device 4.

[0119] FIG. 9 is a diagram schematically showing the trajectories of the robot body 50 and the manipulators 52a and 52b based on the control input information Icn generated by the control input generation unit 35. In FIG. 9, the trajectories of the manipulators 52a and 52b are shown up to the point in time when the manipulator 52a grips the object A and the manipulator 52b grips the object B. The points 56a to 56c represent the predicted positions of the robot body 50 at each time step (i.e., the transition of x r ), the points 58a to 58e represent the predicted positions of the manipulator 52a at each time step (i.e., the transition of x r1 ), and the points 59a to 59d represent the predicted positions of the manipulator 52b at each time step (i.e., the transition of x r2 ). Note that the positions of the manipulators 52a and 52b are abstractly represented by the positions of the robot hands 53a and 53b.

[0120] Here, the time steps corresponding to the points 56a to 56c, the time steps corresponding to the points 58a to 58e, and the time steps corresponding to the points 59a to 59d may or may not be time steps in overlapping periods with each other. Whether there is such an overlap is determined according to the setting of priorities (such as safety priority, tact time priority, etc.) in the execution of the target task set by the user. The setting of priorities will be described in detail in the section of "(6) Determination of Constraint Conditions and Evaluation Function According to the Setting of Priorities ".

[0121] Based on the above-described optimization process, the control input generation unit 35 can determine the abstract state vector z and the control input for each time step, thereby specifying the respective trajectories of the robot body 50 and the manipulators 52a and 52b as shown in FIG. 9.

[0122] (5-8) Sub-task sequence generation unit The sub-task sequence generation unit 36 generates an operation sequence Sr based on the control input information Icn supplied from the control input generation unit 35 and the sub-task information I4 stored in the application information storage unit 41. In this case, by referring to the sub-task information I4, the sub-task sequence generation unit 36 recognizes the sub-tasks that the robot 5 can accept, and converts the control input for each time step indicated by the control input information Icn into sub-tasks.

[0123] For example, in the case where the pick-and-place is the target task in the subtask information I4, functions indicating three subtasks of the movement (moving) of the robot body 50, the movement (reaching) of the robot hand, and the grasping (grasping) of the robot hand are defined as subtasks that the robot 5 can accept. In this case, the function "Move" representing moving is, for example, a function that takes as arguments the initial state of the robot 5 before the execution of the function, the path indicated by the path information Irt (or the final state of the robot 5 after the execution of the function), and the time required for the execution of the function (or the moving speed of the robot body 50). The function "Reach" representing reaching is, for example, a function that takes as arguments the initial state of the robot 5 before the execution of the function, the final state of the robot 5 after the execution of the function, and the time required for the execution of the function. Further, the function "Grasp" representing grasping is, for example, a function that takes as arguments the state of the robot 5 before the execution of the function, the state of the object to be grasped before the execution of the function, and the logical variable δ. Here, the function "Grasp" represents performing a grasping operation when the logical variable δ is "1", and represents performing a releasing operation when the logical variable δ is "0". In this case, the subtask sequence generation unit 36 determines the function "Reach" based on the trajectory of the manipulator 52 determined by the control input for each time step indicated by the control input information Icn, and determines the function "Grasp" based on the transition of the logical variable δ for each time step indicated by the control input information Icn. Further, the subtask sequence generation unit 36 determines the function "Move" based on the trajectory of the robot body 50 determined by the control input for each time step indicated by the path information Irt or the control input information Icn. Here, the arguments and characteristics of the function "Move" for moving the robot body 50 (moving), and parameters for adjusting them, etc., may be different when moving the robot body 50 to the reference point based on the path information Irt, and when moving the robot body 50 (moving) together with the movement (reaching) and grasping (grasping) by the manipulator 52 based on the control input information Icn after reaching the reference point.That is, the format (arguments, parameters, etc.) of the function "Move" used when generating the first operation sequence Sr1 may be different from the format of the function "Move" used when generating the second operation sequence Sr2.

[0124] When the path information Irt is supplied from the path setting unit 30 to the sub-task sequence generation unit 36, the sub-task sequence generation unit 36 generates a first operation sequence Sr1 composed of the function "Move" and supplies the first operation sequence Sr1 to the robot control unit 17. When the control input information Icn is supplied from the control input generation unit 35 to the sub-task sequence generation unit 36, the sub-task sequence generation unit 36 generates a second operation sequence Sr2 composed of the functions "Move", "Reach", and "Grasp" and supplies the second operation sequence Sr2 to the robot control unit 17.

[0125] For example, when the target task is "finally, the object (i = 2) exists at the destination G", in the second operation plan, the sub-task sequence generation unit 36 generates a second operation sequence Sr2 including a sequence of the functions "Reach", "Grasp", "Reach", and "Grasp" for the manipulator 52 closest to the object (i = 2). In this case, the manipulator 52 closest to the object (i = 2) moves to the position of the object (i = 2) by the first function "Reach", grasps the object (i = 2) by the first function "Grasp", moves to the destination G by the second function "Reach", and places the object (i = 2) on the destination G by the second function "Grasp".

[0126] (6) Determination of constraint conditions and evaluation functions according to the setting of priorities Based on the input signal S1 supplied from the instruction device 2, the robot controller 1 recognizes the setting of priorities in the execution of the target task specified by the user, and determines the constraint conditions and evaluation function according to the setting of priorities. Here, the priorities to be set include, for example, "safety first", "operation time length first (tact time first)", "power consumption first", and the like. In this case, information (also referred to as "priority correspondence information") associating each priority that can be set with at least one of the constraint conditions or evaluation functions to be used is stored in the application information storage unit 41 or the like, and the robot controller 1 determines the constraint conditions or / and evaluation function to be used based on the priority correspondence information and the priority indicated by the task specification information Ia.

[0127] FIG. 10(A) is a diagram showing the operation period of the robot body 50 when safety first is set, and the operation period of the manipulator 52, explicitly shown on the time steps corresponding to the execution period of the second operation sequence Sr2.

[0128] When safety first is set, the robot controller 1 determines that it is necessary to operate the robot body 50 and the manipulator 52 exclusively on the time axis, and operates the manipulator 52 after the movement of the robot body 50 is completed. In this case, the target logical formula generation unit 32 of the robot controller 1 generates, for example, a formula of a constraint condition representing "not operating the robot body 50 and the manipulator 52 simultaneously", and adds the generated formula of the constraint condition to the target logical formula Ltag. In another example, the control input generation unit 35 separately adds a constraint condition of "the control input to the robot body 50 and the control input to the manipulator 52 do not occur at the same time step" and performs an optimization process. Thereby, the robot controller 1 can generate the second operation sequence Sr2 in consideration of the safety first specified by the user.

[0129] FIG. 10(B) is a diagram showing a band 68C indicating the operation period of the robot body 50 when the operation time length priority is set, and bands 68D and 68E indicating the operation periods of the manipulator 52, explicitly shown on the time steps defined in the second operation sequence Sr2.

[0130] When the operation time length priority is set, the robot controller 1 generates a second operation sequence Sr2 that actively overlaps the operation of the robot body 50 and the manipulator 52 on the time axis. In this case, the control input generation unit 35 may set, for example, an evaluation function (e.g., min{T}) that includes at least a term contributing to the minimization of the number of time steps T, instead of the evaluation function shown in Equation (2). In other words, the control input generation unit 35 sets an evaluation function having a positive correlation with the number of time steps T (a negative correlation when maximizing the evaluation function instead of Equation (2)), that is, additionally sets a term that becomes a penalty for the time step. Such a penalty term can be, for example, the number of time steps required to reach the first object. Thereby, the control input generation unit 35 can execute the operation of the robot body 50 and the operation of the manipulator 52 in parallel and generate a second operation sequence Sr2 that minimizes the operation time length as much as possible.

[0131] Also, when the power consumption priority is set, the control input generation unit 35 uses, for example, the evaluation function shown in Equation (2) as it is, or in the evaluation function, sets the weight of the term related to u k to be larger than that of other terms (here, the terms related to d k ). Alternatively, the control input generation unit 35 may set an evaluation function representing the relationship between the value of the control input u and the power consumption. In that case, the above relationship varies depending on the robot, and information representing the evaluation function to be set for each robot type may be stored in the storage device 4 as application information. As a result, the robot controller 1 can formulate an operation plan that prioritizes power consumption. Similarly for other priorities, the robot controller 1 can recognize the constraint conditions and / or evaluation functions to be set based on the priority correspondence information, and by reflecting them in the operation plan, formulate an operation plan that takes into account the specified priorities.

[0132] In this way, in addition to spatial constraints, the control input generation unit 35 can determine the trajectory of the robot 5 for each time step in consideration of temporal constraints based on the priorities specified by the user.

[0133] (7) Receiving input related to the target task Next, the process of receiving input regarding the target task by the task specification screen will be described. Hereinafter, each display example of the task specification screen displayed by the indicating device 2 based on the control of the output control unit 15 will be described with reference to FIGS. 11 and 12.

[0134] FIG. 11 shows a first display example of a task specification screen for specifying a target task. The output control unit 15 generates an output control signal S2 and controls the indicating device 2 to display the task specification screen shown in FIG. 11 by transmitting the output control signal S2 to the indicating device 2. The task specification screen shown in FIG. 11 mainly has an area specification column 80, an object specification column 81, a priority specification column 82, an execution button 83, and a stop button 84.

[0135] The output control unit 15 receives an input for specifying the designated area where the robot 5 performs work in the area specification column 80. Here, symbols (such as "A", "B", "C", etc.) are assigned to the areas that are candidates for the designated area, and the output control unit 15 receives an input for selecting the symbol of the area to be the designated area (here, "B") through the area specification column 80. Note that the output control unit 15 may pop up and display a map of the work space representing the correspondence between each symbol and the area in a separate window.

[0136] The output control unit 15 receives an input for designating the destination (goal point) of the object (workpiece) in the object designation field 81. Here, the objects are classified by shape, and the output control unit 15 receives an input for designating the destination for each classified object in the object designation field 81. In the example of FIG. 11, the output control unit 15 receives an input in the object designation field 81 to move two cubic objects to "destination 2", three rectangular parallelepiped objects to "destination 1", one cylindrical object to "destination 1", and one cylindrical object to "destination 2".

[0137] Also, the output control unit 15 receives an input for selecting a priority item in the priority item designation field 82. Here, a plurality of candidates for priority items are listed in the priority item designation field 82, and the output control unit 15 receives the selection of one priority item (here, "safety") from among them.

[0138] Then, when the output control unit 15 detects that the execution button 83 has been selected, it receives the input signal S1 indicating the content specified in the area designation field 80, the object designation field 81, and the priority item designation field 82 from the indicating device 2, and generates task designation information Ia based on the received input signal S1. Also, when the output control unit 15 detects that the stop button 84 has been selected, it cancels the formulation of the operation plan.

[0139] FIG. 12 shows a second display example of the task designation screen for designating the target task. The output control unit 15 generates an output control signal S2 and controls the indicating device 2 to display the task designation screen shown in FIG. 12 by transmitting the output control signal S2 to the indicating device 2. The task designation screen shown in FIG. 12 mainly includes an area designation field 80A, an object designation field 81A, a destination mark display field 81B, a priority item designation field 82A, an execution button 83A, and a stop button 84A.

[0140] The output control unit 15 displays the map (floor plan) of the work space by referring to the map information I7 in the area designation field 80A. Here, the output control unit 15 divides the displayed work space into a predetermined number of areas (here, six areas), and displays each divided area as a designated area so that it can be selected. Note that instead of displaying a map of the work space by CG (Computer Graphics) based on the map information I7, the output control unit 15 may display a real image of the work space taken as a map.

[0141] Also, the output control unit 15 displays a real image or a CG image of the object (workpiece) existing in the area designated in the area designation field 80A in the object designation field 81A. In this case, the output control unit 15 acquires, as a sensor signal S4, an image generated by a camera that has photographed the area designated in the area designation field 80A, and displays the state of the latest object in the designated area based on the acquired sensor signal S4 in the object designation field 81A. Then, the output control unit 15 accepts an input for attaching a mark designating a conveyance destination to each object displayed in the object designation field 81A. Note that as shown in the conveyance destination mark display field 81B, a solid line mark corresponds to "conveyance destination 1" and a dotted line mark corresponds to "conveyance destination 2" as the marks for designating the conveyance destination.

[0142] Also, the output control unit 15 accepts an input for selecting a priority item in the priority item designation field 82A. Here, a plurality of candidates for priority items are listed in the priority item designation field 82, and the output control unit 15 accepts the selection of one priority item (here, "safety") from among them.

[0143] Then, when the output control unit 15 detects that the execution button 83 has been selected, it receives an input signal S1 indicating the content specified in the area designation field 80A, the object designation field 81A, and the priority item designation field 82A from the instruction device 2, and generates task designation information Ia based on the received input signal S1. Thereby, the output control unit 15 can suitably generate the task designation information Ia. Further, when the output control unit 15 detects that the cancel button 84 has been selected, it cancels the formulation of the operation plan.

[0144] As described above, according to the task designation screen shown in the first display example or the second display example, the output control unit 15 can receive the user input necessary for generating the task designation information Ia and suitably acquire the task designation information Ia.

[0145] (8) Processing flow FIG. 13 is an example of a flowchart showing an outline of the processing executed by the robot controller 1 in the first embodiment.

[0146] First, the robot controller 1 acquires the task designation information Ia (step S11). In this case, for example, the output control unit 15 causes the instruction device 2 to display a task designation screen as shown in FIG. 11 or FIG. 12, and acquires the task designation information Ia by receiving the input signal S1 regarding the designation of the target task. In another example, when the task designation information Ia is stored in advance in the storage device 4 or the like, the robot controller 1 may acquire the task designation information Ia from the storage device 4 or the like.

[0147] Next, the robot controller 1 determines a reference point for the designated area indicated by the task designation information Ia acquired in step S11, and determines a robot path that is the path to the determined reference point (step S12). In this case, the path setting unit 30 sets a reference point based on the position of the object 6, its own position, the reach range of the manipulator 52, etc., and generates path information Irt indicating the robot path for reaching the reference point.

[0148] Then, the robot controller 1 performs robot control according to the first operation sequence Sr1 based on the robot path (step S13). In this case, the robot control unit 17 sequentially supplies the control signal S3 based on the first operation sequence Sr1 for the robot 5 to move according to the robot path indicated by the path information Irt to the robot 5, and controls the robot 5 to operate according to the generated first operation sequence Sr1.

[0149] Then, the robot controller 1 determines whether the robot 5 has reached the reference point (step S14). And when the robot controller 1 determines that the robot 5 has not reached the reference point (step S14; No), it continues the robot control based on the first operation sequence Sr1 in step S13.

[0150] On the other hand, when the robot controller 1 determines that the robot 5 has reached the reference point (step S14; Yes), it generates a second operation sequence Sr2 regarding the movement of the robot main body 50 and the operation of the manipulator 52 (step S15). In this case, the generation of the abstract state setting information IS by the abstract state setting unit 31, the generation of the target logical formula Ltag by the target logical formula generation unit 32, the generation of the time step logical formula Lts by the time step logical formula generation unit 33, the generation of the abstract model Σ by the abstract model generation unit 34, the generation of the control input information Icn by the control input generation unit 35, and the generation of the second operation sequence Sr2 by the subtask sequence generation unit 36 are sequentially executed.

[0151] Next, the robot controller 1 performs robot control according to the second operation sequence Sr2 generated in step S15 (step S16). In this case, the robot control unit 17 sequentially supplies the control signal S3 based on the second operation sequence Sr2 to the robot 5, and controls the robot 5 to operate according to the generated second operation sequence Sr2.

[0152] Then, the robot controller 1 determines whether the target task has been completed (step S17). In this case, for example, when the output of the control signal S3 to the robot 5 based on the second operation sequence Sr2 by the robot control unit 17 of the robot controller 1 is completed (the output has disappeared), it is determined that the target task has been completed. In another example, when the robot control unit 17 recognizes based on the sensor signal S4 that the state of the object has become a completed state, it is determined that the target task has been completed. And when the robot controller 1 determines that the target task has been completed (step S17; Yes), the processing of the flowchart ends. On the other hand, when it is determined that the target task has not been completed (step S17; No), the robot controller 1 continues the robot control based on the second operation sequence Sr2 in step S16.

[0153] (9) Variant Next, a modification example of the first embodiment will be described. The following modification examples may be applied in any combination.

[0154] (First Modification Example) The robot controller 1 may generate the first operation sequence Sr1 and control the robot 5 based on the first operation sequence Sr1 in a situation where the state within the designated area (including the state of the object 6) can be observed.

[0155] In this case, when the robot controller 1 receives the task designation information Ia indicating the designated area, it determines a reference point based on an arbitrary point in the designated area indicated by the task designation information Ia, and generates the first operation sequence Sr1 indicating the robot path to the reference point, thereby moving the robot 5 to the reference point. In this case, the robot controller 1 sets, for example, the point in the designated area where the required time until the robot 5 reaches is the shortest, or the point closest to the starting position (current position) of the robot 5 as the reference point.

[0156] Then, after the robot 5 reaches the reference point, the abstract state setting unit 31 of the robot controller 1 generates a recognition result Im regarding an object within the designated area based on the sensor signal S4 generated by the sensor 7 provided on the robot 5. Thereafter, the robot controller 1 executes processing based on the above-described embodiment, generates the second operation sequence Sr2, and controls the robot 5 based on the second operation sequence Sr2.

[0157] In this way, even when the robot controller 1 recognizes the state of an object or the like after reaching the reference point within the designated area, it can suitably cause the robot 5 to execute the target task.

[0158] (Second Modification Example) The block configuration of the operation planning unit 16 shown in FIG. 5 is an example, and various modifications may be made.

[0159] For example, information on candidates φ of the sequence of operations to be commanded to the robot 5 is stored in advance in the storage device 4, and the operation planning unit 16 executes an optimization process of the control input generation unit 35 based on the information. Thereby, the operation planning unit 16 selects an optimal candidate φ and determines the control input for the robot 5. In this case, the operation planning unit 16 does not necessarily have functions corresponding to the abstract state setting unit 31, the target logical formula generation unit 32, and the time step logical formula generation unit 33 in generating the operation sequence Sr. In this way, information regarding the execution results of some functional blocks of the operation planning unit 16 shown in FIG. 5 may be stored in advance in the application information storage unit 41.

[0160] In another example, the application information may include in advance design information such as a flowchart for designing the operation sequence Sr corresponding to the target task, and the operation planning unit 16 may generate the operation sequence by referring to the design information. Note that a specific example of executing a task based on a pre-designed task sequence is disclosed in, for example, Japanese Patent Application Laid-Open No. 2017-39170.

[0161] (Third Modification Example) Instead of being supplied to the subtask sequence generation unit 36, the path information Irt generated by the path setting unit 30 may be supplied to other processing blocks. For example, the control input generation unit 35 may receive the path information Irt, generate control inputs for each time step along the robot path indicated by the path information Irt, and supply control input information Icn indicating the control inputs to the subtask sequence generation unit 36. In this case, the subtask sequence generation unit 36 generates the first operation sequence Sr1 based on the control input information Icn.

[0162] <Second Embodiment> FIG. 14 shows a schematic configuration diagram of the motion planning device 1X in the second embodiment. The motion planning device 1X mainly includes a state setting means 31X and a motion planning means 16X. Note that the motion planning device 1X may be composed of a plurality of devices. The motion planning device 1X can be, for example, the robot controller 1 in the first embodiment.

[0163] The state setting means 31X sets the state in the work space where a mobile robot having a manipulator for handling an object works. The state setting means 31X can be, for example, the abstract state setting unit 31 in the first embodiment (including modifications, the same applies hereinafter).

[0164] The motion planning means 16X determines a motion plan regarding the movement of the robot and the operation of the manipulator based on the state set by the state setting means 31X, the constraint conditions regarding the movement of the robot and the operation of the manipulator, and the evaluation function based on the dynamics of the robot. The motion planning means 16X can be, for example, the motion planning unit 16 (excluding the abstract state setting unit 31) in the first embodiment.

[0165] FIG. 15 is an example of a flowchart in the second embodiment. The state setting means 31X sets the state in the work space where a mobile robot having a manipulator for handling an object works (step S21). The motion planning means 16X determines a motion plan regarding the movement of the robot and the operation of the manipulator based on the state set by the state setting means 31X, the constraint conditions regarding the movement of the robot and the operation of the manipulator, and the evaluation function based on the dynamics of the robot (step S22).

[0166] According to the second embodiment, the motion planning device 1X can formulate an optimal motion plan taking into account both the movement of the mobile robot and the operation of the manipulator of the robot.

[0167] In each of the above-described embodiments, the program can be stored using various types of non-transitory computer-readable media and supplied to a processor or the like that is a computer. The non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic storage media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROM (Read Only Memory), CD-R, CD-R / W, semiconductor memories (e.g., mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (Random Access Memory)). Also, the program may be supplied to the computer by various types of transitory computer-readable media. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer-readable media can supply the program to the computer via a wired communication path such as electric wires and optical fibers or a wireless communication path.

[0168] In addition, some or all of each of the above embodiments may be described as follows in the appended notes, but are not limited thereto.

[0169] [Appendix 1] State setting means for setting a state in a work space where a mobile robot having a manipulator for handling an object works, Based on the state, constraint conditions related to the movement of the robot and the operation of the manipulator, and an evaluation function based on the dynamics of the robot, Motion planning means for determining a motion plan related to the movement of the robot and the operation of the manipulator, A motion planning device having the same. [Appendix 2] The motion planning means determines a first motion plan for moving the robot to a reference point within a designated area designated as an area where the robot performs work, and a second motion plan which is the motion plan based on the state, the constraint conditions, and the evaluation function after the robot reaches the reference point. The motion planning device according to Appendix 1. [Appendix 3] The motion planning means sets the reference point based on the position of the object. The motion planning device according to Appendix 2. [Appendix 4] The motion planning means determines the reference point based on the position of the object, the movement error of the robot, and the reach range of the manipulator. The motion planning device according to Appendix 3. [Appendix 15] When the robot reaches the reference point based on the first motion plan, the state setting means determines a state setting space in which at least the robot and the object exist, and sets the state within the state setting space. The motion planning device according to any one of Appendices 2 to 4. [Appendix 6] When a priority item to be prioritized in the motion plan is specified, the motion planning means determines at least one of the constraint conditions or the evaluation function based on the priority item. The motion planning device according to any one of Appendices 1 to 5. [Appendix 7] The operation planning means is the operation planning device according to Supplementary Note 6, which sets the evaluation function having a positive or negative correlation with the working time length when the priority is the working time length. [Supplementary Note 8] The operation planning means is the operation planning device according to Supplementary Note 6, which sets the constraint condition that the movement of the robot and the operation of the manipulator are executed exclusively when the priority is safety. [Supplementary Note 9] The operation planning means logic formula conversion means for converting the task to be executed by the robot into a logic formula based on temporal logic based on the state; time step logic formula generation means for generating a time step logic formula, which is a logic formula representing the state for each time step to execute the task, from the logic formula; sub-task sequence generation means for generating a sequence of sub-tasks to be executed by the robot as the operation plan based on the time step logic formula; The operation planning device according to any one of Supplementary Notes 1 to 8, comprising [Supplementary Note 10] area designating means for receiving a designation of an area where a mobile robot having a manipulator for handling an object works; object designating means for receiving a designation regarding the object within the area; operation planning means for determining an operation plan regarding the movement of the robot and the operation of the manipulator based on the designation of the area and the designation regarding the object; An operation planning device comprising [Supplementary Note 11] The operation planning means respectively determines a first operation plan for moving the robot to a predetermined reference point within the area and a second operation plan, which is the operation plan regarding the movement of the robot and the operation of the manipulator after the execution by the robot of the first operation plan. The operation planning device according to Supplementary Note 10. [Supplementary Note 12] A computer Set the state in the working space where a mobile robot having a manipulator for handling an object works, Based on the state, the constraint conditions related to the movement of the robot and the operation of the manipulator, and the evaluation function based on the dynamics of the robot, Determine an operation plan related to the movement of the robot and the operation of the manipulator, Operation planning method. [Appendix 13] Set the state in the working space where a mobile robot having a manipulator for handling an object works, Based on the state, the constraint conditions related to the movement of the robot and the operation of the manipulator, and the evaluation function based on the dynamics of the robot, A storage medium storing a program for causing a computer to execute a process of determining an operation plan related to the movement of the robot and the operation of the manipulator. [Appendix 14] The computer, Receives a designation of an area where a mobile robot having a manipulator for handling an object works, Receives a designation related to an object in the area, Based on the designation of the area and the designation related to the object, determine an operation plan related to the movement of the robot and the operation of the manipulator, Operation planning method. [Appendix 15] Receives a designation of an area where a mobile robot having a manipulator for handling an object works, Receives a designation related to an object in the area, A storage medium storing a program for causing a computer to execute a process of determining an operation plan related to the movement of the robot and the operation of the manipulator based on the designation of the area and the designation related to the object.

[0170] Although the invention of the present application has been described with reference to the embodiments, the invention of the present application is not limited to the above embodiments. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the invention of the present application within the scope of the invention of the present application. That is, the invention of the present application naturally includes various modifications and corrections that those skilled in the art could make in accordance with the entire disclosure including the claims and the technical idea. In addition, each disclosure of the above-cited patent documents and the like shall be incorporated herein by reference.

Explanation of Reference Numerals

[0171] 1 Robot Controller 1X Motion Planning Device 2 Instruction Device 4 Storage Device 5 Robot 7 Sensor 41 Application Information Storage Unit 100 Robot Control System

Claims

1. State setting means for setting a state in a work space where a robot having a manipulator for handling an object and a self-propelled body works; Based on the state, constraint conditions regarding the movement of the main body and the operation of the manipulator, and an evaluation function based on the dynamics of the robot; Operation planning means for determining an operation plan regarding the movement of the main body and the operation of the manipulator; It has, The operation planning means determines a first operation plan for moving the main body to a reference point within a designated area designated as an area where the robot performs work, and the robot determines a second operation plan which is the operation plan based on the state, the constraint conditions, and the evaluation function after reaching the reference point after reaching the reference point. Operation planning device.

2. The operation planning means sets the reference point based on the position of the object. The operation planning device according to claim 1.

3. The operation planning means determines the reference point based on the position of the object, the movement error of the main body, and the reach range of the manipulator. The operation planning device according to claim 2.

4. When the main body reaches the reference point based on the first operation plan, the state setting means determines a state setting space in which at least the robot and the object exist, and sets the state within the state setting space. The operation planning device according to any one of claims 1 to 3.

5. Area designating means for receiving a designation of an area where a robot having a manipulator for handling an object and a self-propelled body works; Object designating means for receiving a designation regarding an object within the area; Operation planning means for determining an operation plan regarding the movement of the main body and the operation of the manipulator based on the designation of the area and the designation regarding the object; It has, The operation planning means determines a first operation plan for moving the main body to a reference point within a designated area designated as an area where the robot performs work, and the robot determines a second operation plan which is the operation plan after reaching the reference point after reaching the reference point. Operation planning device.

6. A computer, Sets the state in the work space where a robot having a manipulator for handling an object and a self-propelled body works, Based on the aforesaid state, the constraint conditions regarding the movement of the main body and the operation of the manipulator, and the evaluation function based on the dynamics of the robot, determine an operation plan regarding the movement of the main body and the operation of the manipulator, In the determination of the operation plan, determine a first operation plan for moving the main body to a reference point within a designated area designated as an area where the robot performs work, and determine a second operation plan, which is the operation plan based on the state, the constraint conditions, and the evaluation function after the robot reaches the reference point, after the robot reaches the reference point. An operation planning method.

7. Set the state in the working space where a robot having a manipulator for handling an object and a self-propelled main body performs work, Based on the aforesaid state, the constraint conditions regarding the movement of the main body and the operation of the manipulator, and the evaluation function based on the dynamics of the robot, cause a computer to execute a process of determining an operation plan regarding the movement of the main body and the operation of the manipulator, In the determination of the operation plan, determine a first operation plan for moving the main body to a reference point within a designated area designated as an area where the robot performs work, and determine a second operation plan, which is the operation plan based on the state, the constraint conditions, and the evaluation function after the robot reaches the reference point, after the robot reaches the reference point. A program.

8. A computer receives a designation of an area where a robot having a manipulator for handling an object and a self-propelled main body performs work, receives a designation regarding an object within the area, determines an operation plan regarding the movement of the main body and the operation of the manipulator based on the designation of the area and the designation regarding the object, In the determination of the operation plan, determine a first operation plan for moving the main body to a reference point within a designated area designated as an area where the robot performs work, and determine a second operation plan, which is the operation plan after the robot reaches the reference point, after the robot reaches the reference point. An operation planning method.

9. Receive a designation of an area where a robot having a manipulator for handling an object and a self-propelled main body performs work, receive a designation regarding an object within the area, cause a computer to execute a process of determining an operation plan regarding the movement of the main body and the operation of the manipulator based on the designation of the area and the designation regarding the object, In the determination of the operation plan, a first operation plan for moving the main body to a reference point within a designated area designated as an area where the robot performs work is determined, and a second operation plan, which is the operation plan after the robot reaches the reference point, is determined by the robot after reaching the reference point. Program.

Images