Control system, control method, and program

Abstract

A control system according to the present invention comprises: a first processing means that determines whether or not to change a relation in position and orientation between an observation device for realizing a desired task which is input through an input device and a workpiece to be subject to the desired task, on the basis of at least one of information pertaining to the desired task, observation device information pertaining to the observation device, object model information pertaining to the workpiece, controlled device information pertaining to a controlled device for changing the relation in position and orientation between the observation device and the workpiece, and constraint condition information to be satisfied for realizing the desired task; a second processing means that determines whether or not the observation device can observe the workpiece; a third processing means that outputs, on the basis of a result of determination by the second processing means, plan information for executing the desired task; and a fourth processing means that controls the controlled device on the basis of the plan information.

Description

Control system, control method, and recording medium

[0001] The present disclosure relates to a control system, a control method, and a recording medium.

[0002] An example of a controlled device controlled by a control device is disclosed in, for example, Patent Document 1. The robot device disclosed in Patent Document 1 generates an operation with a short operation time while taking into consideration both the order in which the hand of the robot device used for outer ring inspection, i.e., the imaging device, is moved to the working point and the posture of the hand at that time.

[0003] International Publication No. 2021 / 070096

[0004] However, when the relationship between the hand of a robotic device and the workpiece (workpiece) is not ideal, i.e., when the hand is out of reach, outside the field of view of the hand's imaging device, or the workpiece is obscured by another object (i.e., an object other than the workpiece (workpiece) that can be manipulated by a control device (i.e., an object whose position and / or orientation can be changed)), it is difficult to operate the hand by planning only the order and orientation of the hand. Therefore, the device disclosed in Patent Document 1 is not necessarily able to control the hand of a robotic device to be controlled when the relationship between the hand of the robotic device and the workpiece is not ideal. Therefore, one of the objects of the present disclosure is to provide a motion plan that can continue control and perform a task even when the relationship between the hand of a robotic device and the workpiece is not ideal.

[0005] In one aspect of the present disclosure, a control system comprises a first processing means that determines whether to change the relationship between the position and orientation of the observation device and the work based on at least one of information regarding a target task input by an input device, observation device information regarding an observation device that realizes the target task, object model information regarding a work that is the target of the target task, controlled device information regarding a controlled device that changes the relationship between the position and orientation of the observation device and the work, and constraint condition information that must be satisfied to realize the target task; a second processing means that determines whether the observation device can observe the work; a third processing means that outputs planning information for executing the target task based on the determination result by the second processing means; and a fourth processing means that controls the controlled device based on the planning information.

[0006] In another aspect of the present disclosure, a control method determines whether to change the relationship between the position and posture of the observation device and the work based on at least one of information regarding the target task input by an input device, observation device information regarding the observation device that realizes the target task, object model information regarding the work that is the target of the target task, controlled device information regarding the controlled device that changes the relationship between the position and posture of the observation device and the work, and constraint condition information that must be satisfied to realize the target task, determines whether the observation device can observe the work, outputs planning information for executing the target task based on the determination result, and controls the controlled device based on the planning information.

[0007] In another aspect of the present disclosure, the recording medium stores a program that causes a computer to perform the following operations based on at least one of information regarding the target task input by an input device, observation device information regarding the observation device that realizes the target task, object model information regarding the work that is the target of the target task, controlled device information regarding the controlled device that changes the relationship between the position and posture of the observation device and the work, determine whether or not the observation device can observe the work, output planning information for executing the target task based on the determination result, and control the controlled device based on the planning information.

[0008] According to the device etc. according to the present disclosure, precise control of the controlled device can be realized.

[0009] 1 is a diagram illustrating an example of a configuration of a control system according to a first embodiment of the present disclosure. FIG. 2 is a diagram illustrating an example of a data structure of storage information stored in a storage device according to the first embodiment of the present disclosure. FIG. 3 is a flowchart illustrating an example of a processing procedure performed by the control system according to the first embodiment of the present disclosure. FIG. 4 is a diagram illustrating an example of a display of a task input screen according to the first embodiment of the present disclosure. FIG. 5 is a diagram illustrating an example of a specific configuration of a control system according to the first embodiment of the present disclosure. FIG. 6 is a diagram illustrating a first example of an abstract state according to the first embodiment of the present disclosure. FIG. 7 is a diagram illustrating a second example of an abstract state according to the first embodiment of the present disclosure. FIG. 8 is a diagram illustrating an example of a change in logical variables assuming a result of solving an optimization problem according to the embodiment of the present disclosure, and an operation corresponding to the example change. FIG. 9 is a diagram illustrating an example of another abstract state when the target task is an imaging task according to the first embodiment of the present disclosure. FIG. 10 is a diagram illustrating an example of a change in each logical variable when an optimization problem is solved by adding a constraint condition of Equation (10) according to the embodiment of the present disclosure. FIG. 11 is a diagram illustrating an example of a configuration of a control system according to a second embodiment of the present disclosure. FIG. 12 is a diagram illustrating an example of a configuration of a control system according to a third embodiment of the present disclosure. FIG. 13 is a diagram illustrating a first application example of the control system according to the first embodiment of the present disclosure. FIG. 14 is a diagram illustrating a second application example of the control system according to the first embodiment of the present disclosure. Fig. 1 is a diagram illustrating a third application example of the control system according to the first embodiment of the present disclosure. Fig. 2 is a diagram illustrating a control system with a minimum configuration according to an embodiment of the present disclosure. Fig. 3 is a diagram illustrating an example of a processing flow of the control system with a minimum configuration of the present disclosure. Fig. 4 is a schematic block diagram illustrating the configuration of a computer according to at least one embodiment.

[0010] The following describes embodiments of the present disclosure, but the following embodiments do not limit the scope of the claimed invention. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solutions of the present disclosure. In the following description of the embodiments and drawings, the same reference numerals denote similar objects unless otherwise specified. In the following description of the embodiments, repeated description of similar configurations or operations may be omitted.

[0011] First Embodiment (Configuration) Fig. 1 is a diagram illustrating an example of the configuration of a control system 100 according to a first embodiment of the present disclosure. As shown in Fig. 1, the control system 100 includes an input device 1, an observation device 2, a storage device 3, a controlled device 4, a control device 6 (an example of a fourth processing means), and a planning device 10. The control system 100 is a control system in which the planning device 10 controls the controlled device 4 by outputting information for changing the positional relationship between the observation device 2 and an object described below, based on information for the control system 100 to execute a task and information stored in the storage device 3.

[0012] The input device 1 accepts input of information necessary for the control system 100 to execute a task. Hereinafter, this task will be referred to as a "target task." The input device 1 may function as an interface with a user and accept data input by the user. For example, the input device 1 may be equipped with a GUI (Graphical User Interface) and include at least one of a touch panel, buttons, a keyboard, and a voice input device.

[0013] The observation device 2 observes the target object (workpiece) of the target task according to the target task received by the input device 1. Here, workpiece observation is a general term for acquiring information about the workpiece using the observation device 2. For example, if the target task is to obtain image information about the workpiece, the observation device 2 is equipped with a camera. The camera (2D camera or 3D camera) acquires still images or continuous images from a specific position and orientation. The acquired image information includes at least one of RGB images, 3D depth data, and point cloud data, and may be appropriately set according to the target task. Setting refers to the process of assigning acquired information expressed as numerical values ​​to variables. Any camera capable of acquiring desired image information may be used, and is not limited by the present disclosure. Such a target task for acquiring image information can be applied, for example, to workpiece inspection and management, or data collection for machine learning. Machine learning is, for example, learning for object recognition (estimating the position and orientation of an object from an image) or object identification (identifying a specific object from an image). Another example of a target task is using the observation device 2 as a dedicated sensor to obtain information about a workpiece. For example, the observation device 2 may be a barcode reader that reads a barcode attached to a workpiece, or a microscope camera that captures the surface pattern (fingerprint of an object) of a workpiece. The dedicated sensor and the information acquired by the observation device 2 are not limited to these, and are not limited by the present disclosure. The installation location, number of installations, and operation (control) method of the observation device 2 may be determined appropriately depending on the target task. Details of the control method for the observation device 2 will be described later.

[0014] The storage device 3 stores at least information about the target task to be executed by the control system 100. Specifically, for example, the storage device 3 stores information about the observation device 2, information about the workpiece that is the target of the target task, and information about the controlled device 4. Examples of information about the target task include conditions for completing the target task received by the input device 1 and constraints that must be satisfied. Specifically, in the case of a target task to obtain image information about the workpiece described above, the information about the target task includes the relationship between the position and orientation of the workpiece and the observation device 2, conditions such as the distance, orientation, and brightness from the workpiece at the time of image capture (image capture conditions), and environmental conditions in the work space such as "the workpiece and the observation device 2 are not obstructed by other objects" and "there are no other objects within the movable range of the controlled device 4." The information about the target task may be stored as numerical data, mathematical expressions (inequalities or equations), or propositions (a form that can determine the truth or falsity of a sentence or expression). Examples of information about the observation device 2 include information about the specifications, performance (specs), and limitations of the observation device 2. Specifically, in the case of a target task of obtaining image information of a workpiece, the information about the observation device 2 includes the range that can be imaged by the observation device 2, the time required for image capture, the size of the device, etc. Examples of information about the workpiece include information specifying the shape of the workpiece and the location to be imaged. Information about the workpiece may be data such as a CAD model or numerical values ​​indicating size, etc. Examples of information about the controlled device 4 include the range and speed of movement of the controlled device 4, and information required for control. Note that the storage device 3 may be an external storage device such as a hard disk connected to or built into any other device, or a storage medium such as a flash memory. Furthermore, the storage device 3 may be stored in a distributed manner across multiple storage devices or multiple media.

[0015] The controlled device 4 changes the relative positional relationship between the observation device 2 and the workpiece based on the motion plan output by the planning device 10. The controlled device 4 will be described using the example of the target task of obtaining image information about the workpiece described above. For example, if the controlled device 4 is a robotic device (robot arm, articulated robot) with a movable arm, the positional relationship between the observation device 2 and the workpiece can be changed by mounting the observation device 2 on the arm and moving the arm using control signals generated by the control device 6 based on the motion plan. Alternatively, the observation device 2 can be fixed in position, and the robotic device's arm can be equipped with a gripper with two or more claws that can perform manipulation by physical contact with the workpiece, such as grasping or pushing the workpiece, or an attraction end effector that can attract the workpiece using vacuum or magnetic force. This makes it possible to change the position and orientation of the workpiece, or to grip the workpiece and move it closer to the observation device 2. This allows the controlled device 4 to change the positional relationship between the observation device 2 and the workpiece. Alternatively, mounting the observation device 2 and the end effector on the arm simultaneously makes it possible to change both the position and orientation of the observation device 2 and the position and orientation of the workpiece. The above-described controlled device 4 is merely an example, and the type and configuration of the arm, the mounting method and number of the observation device 2, the type of end effector, and the like may be determined appropriately depending on the target task and the type of workpiece. Another example of the controlled device 4 may be one integrated into the observation device 2. For example, the observation device 2 may have a movable part that changes the imaging range by changing the position and orientation of the observation device 2, and this movable part may be the controlled device 4. A movable part is a movable device (including an actuator) that changes the rotation or translation of a part other than the arm in the robot device. The method of movement and configuration of the observation device 2 may be determined appropriately depending on the observation device 2, the target task, and the type of workpiece.

[0016] The control device 6 generates a control signal for controlling the controlled device 4 based on the operation plan. Then, the control device 6 controls the controlled device 4 by outputting the generated control signal to the controlled device 4. The control device 6 may be a device independent from the controlled device 4. Alternatively, the control device 6 may be a device provided in the controlled device 4.

[0017] The planning device 10 includes an operation determination unit 11 (an example of a first processing means), an observation determination unit 12 (an example of a second processing means), and a plan generation unit 13 (an example of a plan generation means). The planning device 10 outputs an operation plan (an example of plan information) for controlling the controlled device 4 based on information input from each of the input device 1, the observation device 2, and the storage device 3 (specifically, based on processing in an optimization problem described below). The planning device 10 may be a device independent of the input device 1, the observation device 2, the storage device 3, the controlled device 4, and the control device 6. The planning device 10 may also be coupled to any of the input device 1, the observation device 2, the storage device 3, the controlled device 4, and the control device 6. The connections between the planning device 10 and each of the input device 1, the observation device 2, the storage device 3, the controlled device 4, and the control device 6 may be wired or wireless.

[0018] The operation determination unit 11 receives current environmental status information and information stored in the storage device 3 and outputs a determination result on whether to operate an object. The current environmental status information is, for example, information representing the position and orientation of a workpiece. This position and orientation are expressed in a coordinate system based on the observation device 2 or the controlled device 4, or in a coordinate system based on an arbitrary point. It is desirable for the position and orientation to be expressed in six dimensions: three dimensions (X, Y, Z) for position and three dimensions (roll, pitch, yaw) for orientation. However, the positional relationships between the observation device 2, the controlled device 4, and the arbitrary point (i.e., the coordinates of the observation device 2, the controlled device 4, and the arbitrary point in the applicable coordinate system) are assumed to be known. Furthermore, the way in which the position and orientation of the workpiece are expressed is not limited to the above, and may also be the position and size of the center or center of gravity of the workpiece. This is, for example, the case in which "the widest surface is the imaging point" shown in Figure 7 (described later). In the state of the workpiece shown in Figure 7, it can be seen from the state of the workpiece that the widest surface of the workpiece is not facing the direction observable by the observation device 2. Furthermore, if other objects are present in the environment where the target task is being performed, the state information of the current environment includes information representing the position and posture of those objects. In other words, if other objects are present in the environment where the target task is being performed, the state information of the current environment is recognition information about the objects present in the environment where the target task is being performed. The recognition information includes identification of whether the object is a workpiece or another object. This recognition information may be output based on information acquired by the observation device 2, or may be acquired from another recognition means. Examples of other recognition means include a device separate from the observation device 2 (e.g., a device external to the observation device 2). The other recognition means recognizes workpieces and other objects using, for example, an inference device trained in advance by machine learning (deep learning) using a neural network.

[0019] As described above, the storage device 3 stores at least information about the target task to be executed by the control system 100. The storage device 3 may store information about the observation device 2, information about the workpiece that is the target of the target task, and information about the controlled device 4. Below, an example will be described in which the target task is an imaging task for obtaining image information about the workpiece. However, the above information is merely an example, and the information input as information stored in the storage device 3 is not limited to the above. For example, the information stored in the storage device 3 may be input as a proposition for completing the target task. The operation determination unit 11 determines whether to manipulate an object based on the current environmental state information and the information stored in the storage device 3. The operation determination unit 11 outputs the determination result to the plan generation unit 13. Here, manipulating an object refers to manipulating the workpiece included in the environment, and, if other objects are recognized, also manipulating the objects, i.e., changing the position or posture of the object. The determination result may be a numerical value or a binary value representing true or false. The determination result is not limited to a single value. For example, as described above, when other objects are included, the operation determination unit 11 may output a determination result that the workpiece is being manipulated and a determination result that the other object is being manipulated separately. Note that the determination result output by the operation determination unit 11 may be, for example, true or false for the proposition "manipulate the workpiece."

[0020] The observation determination unit 12 receives information about the abstract state and information about the observation device 2. Based on information about the workpiece, information about the observation device, and the proposition, the observation determination unit 12 determines whether the observation device 2 is within an area where the workpiece 20 can be observed by the observation device 2. For example, if the observation device 2 is configured to be movable or to control the position and orientation of the workpiece 20, the observation determination unit 12 determines that the workpiece 20 can be observed when the observation device 2 enters the observable area of ​​the workpiece 20. Also, for example, if the installation position of the observation device 2 is fixed, the observation determination unit 12 determines that the workpiece 20 can be observed when the workpiece enters the observable area of ​​the observation device 2. The determination result may be a binary value representing true or false, or may be another value. Examples of other values ​​include the overlap rate between the observable area of ​​the observation device 2 and the volume or area of ​​the workpiece 20. For example, the observation determination unit 12 may output true or false for the proposition "observable." The observation determination unit 12 then outputs the determination result. The workpiece status information is information that represents the position and posture of the workpiece, similar to the information input to the operation determination unit 11. The information stored in the storage device 3 is information that includes at least the specifications, performance (specs), or limitations of the observation device 2. However, the above information is merely an example, and the information input as information to be stored in the storage device 3 is not limited to the above. For example, a proposition for completing a target task may be input to the storage device 3.

[0021] The plan generation unit 13 receives input of the current environmental state information, the information stored in the storage device 3, the determination results by the operation determination unit 11, and the determination results by the observation determination unit 12, and outputs an operation plan for controlling the controlled device 4 to the control device 6. This operation plan is obtained, for example, based on processing in an optimization problem described below. The current environmental state information is similar to the information input to the operation determination unit 11 and is information representing the positions and postures of the workpiece and other objects. Note that the current environmental state information includes information about objects other than the workpiece 20. Furthermore, the information input to the operation determination unit 11 does not include information about objects other than the workpiece 20. The information stored in the storage device 3 includes at least information about a target task to be executed by the control system 100. An imaging task for obtaining image information about a workpiece, which is an example of a target task, will be described below. Conditions or propositions for completing the target task are input as information about the imaging task stored in the storage device 3. For example, the information about the imaging task stored in the storage device 3 is propositions such as "the observation device 2 is in the observable area," "there are no obstructing objects between the workpiece and the observation device 2," and "the current state of the workpiece satisfies the imaging location specification." Each of these set propositions corresponds to an output from either the operation determination unit 11 or the imaging determination unit 12, and the truth or falsity of the proposition is determined. Based on the determination result of the proposition, the plan generation unit 13 outputs an operation plan for controlling the controlled device 4 to the control device 6. The operation plan is preferably an operation plan that changes the position and orientation relationship between the observation device 2, the workpiece, and the object involved in a time series, i.e., for each time step. Specifically, the plan generation unit 13 generates information for each time step, such as moving an object to a specific position, moving the workpiece to a specific position, or moving the observation device 2 to a specific position. As described below, the plan generation unit 13 uses this information to determine whether or not to move for each time step and generates the specific position to be moved as a state vector value in an abstract model (e.g., Equations (6) and (7) described below). Then, the plan generating unit 13 outputs the information generated for each time step to the controlled device information 14. That is, the operation plan includes information about the order (sequence) of each operation.Although the controlled device 4 is controlled based on this time-series information, the motion plan does not have to be a control signal that directly controls the moving part (actuator) of the controlled device 4. For example, the motion plan may include information on target values ​​for the position and angle of the moving part at a certain time step, and control up to the target value may be realized by the control device 6 of this configuration or a control function included in the controlled device 4. In general, current state information (position and angle) of the controlled device 4 can be acquired from the controlled device 4. Therefore, by providing a target value through the motion plan, it is possible to realize control from the current value to the target value, for example, control that feeds back the angle of the moving part (actuator) so that it follows spatially continuous position information (trajectory).

[0022] (Stored Information) As described above, the storage device 3 stores at least information about a target task to be executed by the control system 100, information about the observation device 2, information about the workpiece that is the target of the target task, and information about the controlled device 4. A specific example will be given below. Fig. 2 is a flowchart showing an example of a processing procedure performed by the control system 100 according to the first embodiment of the present disclosure. As shown in Fig. 2, the storage device 3 may store abstract state information I1, constraint condition information I2, observation device information I3, controlled device information I4, subtask information I5, abstract model information I6, and object model information I7.

[0023] The abstract state information I1 is information about an abstract state that needs to be defined in order to control the controlled device 4. The abstract state is a state that represents an abstracted representation of a real object in the workspace in which the control system 100 operates. For example, the abstract state is information that numerically represents the position, posture, size, and other characteristics of an object. However, the abstract state is not limited to these. For example, the abstract state may be information represented by a function (e.g., a Gaussian distribution) that represents a position distribution or a surface shape.

[0024] The type and content of the target task input from the input device 1 may be associated with an abstract state that needs to be defined. For example, if the target task is an imaging task for obtaining image information about a workpiece, the position of the workpiece, the orientation of the workpiece, the size of the workpiece, the positions of other objects, the orientation of other objects, the size of other objects, the positions of obstacles that should not be contacted, the orientation of obstacles that should not be contacted, the size of obstacles that should not be contacted, the area of ​​obstacles that should not be contacted, the position of the observation device 2, the orientation of the observation device 2, the size of the observation device 2, etc. are stored as abstract state information I1. Note that the area of ​​obstacles that should not be contacted may be an area with a margin larger than the actual size of the obstacles that should not be contacted. Furthermore, the abstract state information I1 may be stored in advance before the target task is executed, or may be updated when new information is added. Any means may be used to add information.

[0025] The constraint information I2 is information indicating constraints for executing the target task. For example, if the target task is the aforementioned imaging task, the constraint information I2 indicates that the observation device 2 must not come into contact with the workpiece, that the observation device 2 must not come into contact with other objects or obstacles, and that the object controlled by the controlled device 4 must not enter a certain range (area). The conditions indicated by this information may be specified as numerical data (absolute values / relative values) or mathematical expressions (inequalities or equations) based on the respective abstract states. Furthermore, the conditions indicated by this information may be stored as propositions (in a format that can determine the truth or falsity of sentences or expressions) and may include conditions regarding the order between propositions. Furthermore, the type and content of the target task input from the input device 1 may be associated with the constraint information I2.

[0026] The observation device information I3 is information indicating the specifications and performance of the observation device 2. The observation device information I3 may include information associated with the target task and the type of observation device 2. For example, if the target task is an imaging task and the observation device 2 is a camera, the information included in the observation device information I3 associated with the target task and the type of observation device 2 is information such as the camera's field of view, focal length, focal depth, and required light intensity.

[0027] The controlled device information I4 is information indicating the specifications and performance of the controlled device 4. The controlled device information I4 may include information associating a target task with the configuration of the control system 100 and the type of the controlled device 4. For example, if the controlled device 4 is a robot arm, the information includes parameter information such as its movable range, limit values ​​for movable speed, and gain required for control. These pieces of information may be values ​​determined by the hardware of the controlled device 4 at the time of shipment, or may be values ​​set by the user according to the target task and the configuration of the control system 100.

[0028] The subtask information I5 is associated with the target task and the configuration of the control system 100 including the observation device 2 and the controlled device 4, and indicates information for the plan generation unit 13 to output an operation plan. The target task is executed by combining tasks defined in units in which the controlled device 4 can operate. Hereinafter, these defined tasks will be referred to as subtasks. The combination of subtasks is determined based on the plan information output by the plan generation unit 13. In other words, the subtask information I5 includes information defining the subtask and information indicating the correspondence between the information and the plan information, and is referenced in the process of outputting the plan information by the plan generation unit 13. For example, if the objective task is an imaging task that is given the task of "finally capturing an image of a specified location on the workpiece," the subtasks may be, for example, a "subtask (ST1) of approaching the position of the workpiece or object" when other objects are present, a "subtask (ST2) of changing the position and orientation of the object," a "subtask (ST3) of changing the position and orientation of the workpiece" when the current position and orientation of the workpiece do not satisfy the conditions for imaging, or a "subtask (ST4) of grasping the workpiece and bringing it closer to the observation device 2" when the installation position of the observation device 2 is fixed. For example, subtask ST1 is a task of moving the specified position of the arm of the controlled device 4 to a target value, and receives the target value. Therefore, the information defining the subtask stored in subtask information I5 includes information for controlling the arm from the current value to the target value. Subtask ST2 is a task of changing the current position and orientation of the object to a target position and orientation using the end effector of the controlled device 4, and receives the target position and orientation. The information defining the subtask stored in subtask information I5 includes information for controlling the arm from the current position and orientation to the target position and orientation. The plan generation unit 13 selects appropriate subtasks and combines the selected subtasks based on the plan information output according to differences between the target task and the environment and the correspondence relationships defined in the subtask information I5. In the above case, the plan generation unit 13 combines, for example, the subtasks of approaching an object (ST1), changing the position and orientation of the object (ST2), and bringing the workpiece 20 closer to the observation device 2 (ST4).

[0029] The subtask information I5 may also include adjustment parameters such as the time required to complete the execution of a subtask, the speed at which the subtask is executed, and constraints on the order of the subtasks. The subtask information I5 does not need to include information for generating a control signal for directly controlling the controlled device 4. The signal for controlling the controlled device 4 is only required to be associated with the motion plan output by the plan generation unit 13, and the subtask to be executed may be determined from the motion plan and generated based on the subtask. A method for determining subtasks from plan information is a method that uses the relationship between changes in logical variables and subtasks, as described below. In this case, the controlled device 4 may have a function for controlling the controlled device 4 from its current state to the target value when, for example, a state-change object (such as the workpiece 20, an obstacle 21, or the observation device 2, described below) whose position or posture is to be changed by the controlled device 4 and a target value are specified for each subtask. In other words, the controlled device 4 may be controlled from its current state to the target value by a general control device (controller) not shown in FIG. 1 .

[0030] The subtask information I5 preferably includes information about a function for controlling the controlled device 4 according to input values ​​corresponding to each subtask. Specifically, in the example of subtask ST1 described above, the subtask information I5 is information about a function that uses a current value and a target value as arguments to generate a trajectory (a point through which the specified position of the arm passes in space) from the current value to the target value. Note that the subtask information I5 is not limited to the above function and may also include information about a table (database) that outputs a trajectory based on the current value and the target value. In the case of the preferred subtask information I5 described above, each moving part (actuator) of the controlled device 4 is controlled to satisfy the trajectory information. This control is realized by the control device 6 shown in FIG. 1 . The difference between the preferred subtask information I5 and the subtask information I5 containing information about a table (database) that outputs a trajectory based on the current value and the target value is whether the information provided by the planning device 10 to the controlled device 4 is a target value or trajectory information. A target value is spatially a single piece of information. In contrast, a trajectory is continuous information. Therefore, by providing the controlled device 4 with subtask information I5 including information on a table (database) that outputs a trajectory, it is possible to improve the spatial control accuracy of the controlled device 4. This contributes to the achievement of appropriate subtasks, i.e., to an improvement in the degree of achievement of the target task.

[0031] The abstract model information I6 is information about a model (also referred to as an "abstract model") that abstracts the dynamics in the workspace of the control system 100. The abstract model is not limited to a model that abstracts continuous dynamics, as handled in mechanical systems, but may also include a model that abstracts discrete dynamics including logic. Generally, a system represented by the target system of the control system 100 (i.e., an overall model including an abstract model representing the state and dynamics of the target object or environment) is called a hybrid system. Therefore, the abstract model information I6 may include information about a dynamics switch, i.e., a logical branch, in the hybrid system. A "switch" refers to a change in the abstract model due to a logical branch. For example, if the target task is the imaging task described above, conditions for a switch include capturing an image of the workpiece 20 when the observation device 2 enters an observable area, or gripping the workpiece or other object when the end effector of the controlled device 4 approaches the workpiece or other object to a specified position and changing the position and orientation of the workpiece. The abstract model information I6 is preferably expressed as a state space model that represents the dynamics of a hybrid system including continuous variables and discrete (logical) variables. Dynamics refers to "dynamic behavior (change)" as opposed to "static behavior (change)." The state space model is a model that represents spatial and temporal changes (dynamics, i.e., dynamic changes) of a state (position and posture). The abstract model information I6 may also be stored in association with the type and content of the target task and the configuration of the control system 100. The type of target task represents differences in hardware, such as observation devices and controlled devices, depending on the differences in the target task itself, such as imaging, inspection, and identification. The content of the target task represents differences in operations within the same target task, such as the number of imaging attempts and the number of workpieces.

[0032] The object model information I7 includes information specifying the shape and imaging location of the workpiece 20 that is the target of the target task. The imaging location refers to the portion of the workpiece 20 that is to be imaged (e.g., the top surface of the workpiece 20 viewed from above). The imaging location is information that can be specified by area, coordinate values, features (vertices, etc.). The object model information I7 may also include information about other objects and obstacles. The information about other objects and obstacles is information for operations such as operating (controlling) the controlled device 4 so as not to collide with other objects or obstacles, or "moving" other objects, obstacles, or the workpiece 20. Specifically, the information about other objects and obstacles is information for estimating the state (position, posture) and size of the other objects and obstacles. For example, if the other objects and obstacles are known objects, the information about the other objects and obstacles may be CAD data, similar to the workpiece 20. Furthermore, if the other objects and obstacles are unknown objects, the information about the other objects and obstacles may be machine-learned information, similar to the workpiece 20. The information about other objects and obstacles need not include the "image capture location," and the rest of the information may be the same as that about the workpiece 20. The object model information I7 is used when the operation determination unit 11 makes a determination and the plan generation unit 13 outputs an operation plan. The object model information I7 includes, for example, information representing the type, shape, and posture of each object, CAD data representing a two-dimensional or three-dimensional shape, and other information. The information representing the type, shape, and posture of each object, CAD data representing a two-dimensional or three-dimensional shape, and other information may be associated with the type and content of the target task, the type of target workpiece, and other information and recorded as the object model information I7. Furthermore, the operation determination unit 11 and the plan generation unit 13 may use information representing the type, shape, and posture of each object, CAD data representing a two-dimensional or three-dimensional shape, and other information to obtain status information about the current environment, i.e., status information about the workpiece or other objects. If the operation determination unit 11 and the plan generation unit 13 recognize a workpiece or other objects using an inference device previously trained through machine learning (deep learning) using a neural network, the object model information I7 may include parameters of the inference device.The inference unit receives image information (2D or 3D) including an object and outputs state information (position and orientation) of the object. Typically, the inference unit learns in advance the relationship between the image information and the correct state information through deep learning (learning using a neural network) (i.e., the neural network weights are determined as parameters and the determined parameters are saved), and then uses the parameters to infer the state information from the image information. The recognition process may be performed by the control system 100 of this embodiment or by other means. The present invention does not limit the storage and use of the object model information I7. For example, if the recognition process is performed by other means, the object model information I7 may not be used. However, information about the workpiece in the object model information I7 is used when determining the "appropriate area Gi" for the determination process by the operation determination unit 11, which will be described later.

[0033] While the above examples of data stored in the storage device 3 have been presented, the storage (input) and use (output) of data may be performed by a device other than the storage device 3 (e.g., a device external to the control system 100). In this case, the timing and means of the storage (input) and use (output) of data by a device other than the storage device 3 are not limited to specific timing and means. Furthermore, while information I1 to I7 are presented, this information is not all-inclusive; it may be added or omitted as appropriate depending on the target task and the configuration and environment of the control system 100. For example, the information required for a configuration and environment consisting of only a target task and work is abstract state information I1 for that configuration and environment, abstract model information I6, subtask information I5 based on the target task, object model information I7 (work), observation device information I3, and controlled device information I4. In other words, constraint information I2 may be omitted.

[0034] (Operation) Next, a description will be given of the processing performed by the control system 100. Fig. 3 is a flowchart showing an example of the procedure of the processing performed by the control system 100. In the processing shown in Fig. 3, the control system 100 receives a target task from the input device 1 (step S101).

[0035] FIG. 4 is a diagram illustrating an example of a display of a task input screen according to the first embodiment of the present disclosure. FIG. 4 is a diagram illustrating an example of receiving a target task from the input device 1 when the target task is an imaging task. FIG. 4 illustrates an example of a display of a UI (user interface) screen that receives input operations by a user. The input device 1 may be equipped with a UI for display and input, or the UI may be configured as a device separate from the input device 1. In the example illustrated in FIG. 4 , task setting G1 selects the imaging task method and mode and inputs related setting values. Here, the imaging task mode includes options for capturing images of specified locations or capturing images randomly, with the imaging location and number of images set as setting values ​​for each. These options may be displayed and input using a pull-down menu. Work information G2 is information about the size and shape of the workpiece among the object model information I7 stored in the storage device 3. The example illustrated in FIG. 4 is an example of reading from information such as CAD data. The object model information, which is information about the workpiece read here, is displayed in imaging location designation G3 for specifying the imaging location shown in FIG. 4 and is stored in the storage device 3. The designation G3 is a GUI (Graphical User Interface) that reads and displays workpiece information (object model information) and specifies the imaging location. The imaging location can be specified using a mouse, touch panel, or the like. Alternatively, the workpiece information may be data previously stored in the storage device 3 (i.e., workpiece information in the object model information I7). In this case, the order in which the workpiece information is stored and displayed on the GUI may be any order. The designation G3 of the imaging location indicates the object model information I7 for the loaded workpiece. In the example shown in FIG. 4 , the designation G3 of the imaging location displays the three-dimensional (3D) shape of the workpiece loaded in the workpiece information G2 and the imaging location (circle). The designation of the imaging location may be performed by rotating the workpiece three-dimensionally on the screen of the designation G3 of the imaging location and specifying it with a mouse, or information about the imaging location may be included in pre-loaded CAD data, etc. The designation of the imaging location is finally completed by the user touching the confirm button G4.The execute button G5 shown in FIG. 4 is a button for instructing the start of execution of the target task. The stop button G6 shown in FIG. 4 is a button for canceling the execution of the target task. The data preview / output G7 shown in FIG. 4 previews the captured data and outputs it to a file. In the example shown in FIG. 4, the image specified in the data preview / output G7 is output to a specified file by the button G8. Note that the above-described operations on the UI are merely examples and are not limited by the present invention. For example, while FIG. 4 shows an example of a single workpiece and a single image capture location, multiple of each may be used.

[0036] 2 is a diagram illustrating an example of a data structure of stored information stored in the storage device 3 according to the first embodiment of the present disclosure. Next, the planning device 10 acquires the accumulated information illustrated in FIG. 2 from the storage device 3 (step S102). The accumulated information is information about at least a target task to be executed by the control system 100, which is stored in the above-described storage device 3. It is desirable that the planning device 10 acquires associated accumulated information based on the target task accepted in step S101 and the configuration of the control system 100, specifically, the observation device 2 and the controlled device 4.

[0037] Next, the planning device 10 sets a goal logical formula and an abstract model for the control system 100 to execute the goal task based on the goal task and the accumulated information (step S103). The goal logical formula is a logical formula that represents a final achievement state that is the goal of the goal task. The goal logical formula may be expressed in an abstract state. That is, the goal logical formula may be expressed as variables, and numerical values ​​may be substituted when information about the real environment is input. Note that, when actually performing calculations, numerical values ​​are substituted into the variables of the goal logical formula. Furthermore, the goal logical formula may express, in a single logical formula, the conditions for completing the goal task and the constraints that must be satisfied related to the environment and the control system 100.

[0038] Here, a specific example of setting the target logical formula by the planning device 10 shown in FIG. 3 will be described. The target logical formula is a logical formula that represents the final, targeted state of the target task acquired by the input device 1 in step S101. FIG. 5 is a diagram illustrating an example of a specific configuration of the control system 100 according to the first embodiment of the present disclosure. FIG. 5 illustrates an example of the configuration of the control system 100 in the first embodiment when an imaging task is set as the target task. FIG. 5 illustrates the configuration of the control system 100 in the case where the observation device 2 is a camera that acquires image information of the workpiece 20, and the controlled device 4 is an armed robot (robot arm) that changes the relative positional relationship between the workpiece 20 and the observation device 2. The observation device 2 is fixedly installed on the robot arm, and the position and orientation of the observation device 2 are changed by controlling the arm of the controlled device 4. Furthermore, the robot arm of the controlled device 4 is equipped with an end effector that can grasp the workpiece 20 and change its position and orientation. In other words, the position and orientation of the workpiece 20 can be changed by controlling the arm of the controlled device 4. The above configuration is an example and is not limited to this configuration. The process of step S103 will be described in more detail later.

[0039] Furthermore, the planning device 10 acquires current state information for the workpiece 20 that is the target of the target task and for objects other than the workpiece 20. The planning device 10 reflects the acquired current state information in the abstract model by setting it as an abstract state (step S104). The current state information for the workpiece 20 and other objects is preferably a quantity representing the position, orientation, and shape (e.g., the length of the long side). Furthermore, any means may be used to acquire the current state information. Note that more specific processing of the above-mentioned step S104 will be described later.

[0040] Next, the operation determination unit 11 outputs a determination result as to whether or not to operate the object based on the current state information and the information stored in the storage device 3 (step S105). The information stored in the storage device 3 is information about the target task included in information I7 and information about the workpiece 20 that is the target of the target task. Specifically, the information stored in the storage device 3 is preferably information about the conditions under which the workpiece 20 can be imaged and the observation location of the workpiece 20. Note that more specific processing of the above-mentioned step S105 will be described later.

[0041] Next, the observation determination unit 12 outputs a determination result as to whether the observation device 2 is within the area where the workpiece 20 can be observed by the observation device 2, based on the status information (information representing the position and orientation) of the workpiece 20 and the observation device 2 and the information stored in the storage device 3 (step S106). The information stored in the storage device 3 preferably includes specifications and performance information for the observation device 2, including at least the field of view and focal length. Note that the above-mentioned essential information may not be acquired, or the workpiece may not actually be observed even if it is determined by calculation based on the specifications (information) (e.g., due to shadows or reflections caused by ambient light). If the essential information cannot be acquired, the observation determination unit 12 may replace the missing information with a specified value (a value previously stored) and make a determination. The observation determination unit 12 then performs actual observation based on the planned information, and if the observation is not possible (the task cannot be accomplished), it replans the observation, or acquires information on other specifications of the observation device 2 (e.g., exposure time or aperture) and adjusts them. More specific processing of the above-mentioned step S106 will be described later.

[0042] Next, the plan generation unit 13 generates an operation plan that satisfies the target logical formula and the abstract model based on the outputs of the operation determination unit 11 and the observation determination unit 12. Then, the plan generation unit 13 outputs the generated operation plan to the control device 6 (step S107). Note that more specific processing of the above-mentioned step S107 will be described later.

[0043] Then, the control device 6 controls the controlled device 4 based on the operation plan (step S108). Note that the process of step S108 will be described in more detail later.

[0044] FIG. 6 is a diagram illustrating a first example of an abstract state according to the first embodiment of the present disclosure. In diagrams and equations, including FIG. 6, numerical values ​​are expressed using literal expressions. Part (a) of FIG. 6 illustrates an abstract state when an imaging task is the target task. In the abstract state illustrated in part (a) of FIG. 6, a certain point W is set as the reference point of the coordinate system, and the state vector Xc of the observation device 2, the state vector Xe of the end effector of the controlled device 4, and the state vector Xw of the workpiece 20 are represented. The reference point W can be defined arbitrarily, and can be, for example, the edge or center of the workspace, or a pedestal on which the robot is placed. However, in the present disclosure, the reference point W is not limited to the edge or center of the workspace, or the pedestal on which the robot is placed. Furthermore, it is desirable for the state vector to be represented in three dimensions (X, Y, Z) indicating the position and three dimensions (roll, pitch, yaw) indicating the orientation. The state vector indicates the reference position and orientation of each of the observation device 2, the controlled device 4, and the workpiece 20. Therefore, in the following description, the state vector indicating this reference will be used to represent each position and posture. Also, in FIG. 6 , the i-th imaging location of the workpiece 20 is represented as Pi, and the observation range when the observation device 2 is at position Xc is represented as Rxc. The observation range Rxc is determined by the camera's viewing angle and focal length, which are stored in the storage device 3 as observation device information I3. Part (b) of FIG. 6 is a schematic diagram in which the position Xc of the observation device 2 is changed within the range in which the i-th imaging location Pi of the workpiece 20 is included in the observation range Rxc. If the imaging location Pi and the observation range Rxc are known, the region of the position Xc of the observation device 2 within the observation range Rxc (i.e., the imageable region) can be determined. Here, the imageable region is represented as Hi. The imageable region Hi is the range in which the imaging location Pi can be observed when the position Xc of the observation device 2 is changed. Therefore, if the position Xc of the observation device 2 is included in the imageable region Hi, imaging is possible, i.e., the target task can be achieved. Here, in order to express the achievement state of this target task in a logical formula, the planning device 10 defines a proposition based on the abstract state information I1.The imaging task, which is the target task for the i-th imaging location Pi, defines the proposition "ai" as the target task, which is that "the position Xc of the observation device 2 is ultimately present within the imaging area Hi." "Eventually" corresponds to any step up to the preset final time step defined by the operator "?", which corresponds to "eventually," which will be described later. Here, i is an integer greater than or equal to 1, and represents an identification number that identifies the imaging location of the workpiece. This proposition is used to generate a target logical formula.

[0045] Here, a supplementary explanation will be given regarding a method for expressing a logical expression. A logical expression may be expressed by converting a target task described in natural language as described above into a logical expression. Various known methods can be used to convert a target task into a logical expression. As an example of a target task, consider a case where an imaging task is set in which "the observation device 2 and the imaging location Pi of the workpiece are ultimately located within the imaging area A." In this case, the planning device 10 may generate a target logical expression "?a1" using an operator "?" corresponding to "eventually" in linear temporal logic (LTL) and a proposition "ai" defined as an achievement state. Specifically, the planning device 10 generates a target logical expression as a constraint that equation (1) is satisfied at a certain time step. The operator "eventually" in linear temporal logic is also called "finally" or "future," and means "someday, eventually, or at some point in the future." That is, this operator does not specify a specific time, but can indicate the passage of time up to the final point in time (for example, an assumed finite target time step Tk, which will be described later). The target logical formula may be expressed using any linear temporal logic operator other than the operator "?". Note that the linear temporal logic operators may include general logical operators. For example, in addition to or instead of eventually "?", the logical product "#", the logical sum "V", the negation "!", the logical inclusion "¥", always "@", next "&", or until "U", or a combination of these, may be used to generate the target logical formula. Note that the target logical formula may be written using a temporal logic such as MTL (Metric Temporal Logic) or STL (Signal Temporal Logic) in addition to linear temporal logic.

[0046] Constraints that must be satisfied in the execution of the objective task may be added to the target logical formula. For example, the planning device 10 may generate propositions indicating the constraints based on the constraint information I2, and then use the generated propositions to generate a target logical formula in the form of a single logical formula including the constraints. Alternatively, the planning device 10 may generate a logical formula indicating the constraints as a logical formula separate from the target logical formula. In this case, it is determined that the objective task is achieved when the target logical formula and the constraints are all satisfied. Taking the aforementioned imaging task as an example, the constraint stored as the constraint information I2, "The controlled device 4 controlled by the control device 6 does not enter an area set as an obstacle," can be expressed as "@!h" if the proposition "The controlled part, which is the movable part of the controlled device 4, is present in the area set as an obstacle" is expressed as "h." Therefore, the target logical formula for the imaging location Pi including the constraint can be generated as "(?ai)#(@!h)."

[0047] As described above, in the environment of the workspace shown by the configuration of the control system 100 shown in FIG. 5 and the abstract state corresponding to that configuration shown in FIG. 6, by satisfying the target logical formula "(?ai)#(@!h)" for the imaging location Pi, the target task can be achieved while satisfying the constraint of not entering an area set as an obstacle.

[0048] FIG. 7 is a diagram illustrating a second example of an abstract state in the first embodiment of the present disclosure. Part (a) of FIG. 7 illustrates an abstract state in which the position and orientation of the workpiece 20 are general compared to the environments in the workspaces illustrated in FIGS. 5 and 6 . Here, the general position and orientation of the workpiece 20 refers to a case in which the position of the workpiece 20 is not included in the observation range Rxc illustrated in FIG. 6 , i.e., the difference between the position of the workpiece 20 and the position of the observation device 2 is greater than or equal to a certain threshold, or a case in which the angle between the normal to the plane on which the imaging location Pi exists and the normal to the observation range Rxc is greater than or equal to a certain threshold, i.e., the deviation between the orientation of the workpiece 20 and the orientation of the observation range Rxc is large. In other words, the general position and orientation of the workpiece 20 means that the position and orientation are not limited to a certain range. Here, each threshold value is determined appropriately depending on the type of workpiece 20, the performance, configuration, and arrangement of the observation device 2 and the controlled device 4, etc. Specifically, for example, the threshold value is determined based on the specifications (field of view and focal length) of the observation device 2. Alternatively, for example, the threshold value may be determined to provide a specified margin for the specification value of the observation device 2. Alternatively, for example, a provisional value may be determined as the threshold value without being based on known information such as the specifications of the observation device 2. By using the threshold value determined in this manner, the observation determination unit 12 can determine whether the observation device 2 is within the area where the workpiece 20 can be observed by the observation device 2 and output the determination result. In a workspace environment such as that shown in part (a) of Figure 7, if the position Xc of the observation device 2 is never included in the imageable area Hi, regardless of the value controlled, the aforementioned target logical formula "(?ai)#(@!h)" cannot be satisfied. In other words, the target task cannot be achieved.

[0049] Therefore, part (b) of Figure 7 shows an imageable area Hi similar to part (b) of Figure 6, and two workpieces 20 with different positions and orientations. One of the two workpieces 20 is the same workpiece 20 as the workpiece 20 shown in part (a) of Figure 7. The other is a workpiece 20 that exists in an appropriate area Gi such that the imageable area Hi exists within the range of motion of the controlled device 4. In other words, if the workpiece 20 exists in the appropriate area Gi, the controlled device 4 can move to the imageable area Hi, thereby achieving the target task. The appropriate area Gi is an area that includes the position and orientation of the workpiece 20, as shown in part (b) of Figure 7.

[0050] The appropriate area Gi can be defined, for example, as the angle between the normal vector of the imaging location Pi and this appropriate area Gi being equal to or less than a certain threshold. By defining the appropriate area Gi in this manner, a target logical formula can be determined for cases where the position and orientation of the workpiece 20 are general. For example, if the proposition "bi" is "the workpiece 20 is within the appropriate area Gi," the target task can be achieved by satisfying the proposition "bi" and then satisfying "(?ai)#(@!h)." In other words, when the workpiece 20 is present in the appropriate area Gi, the target task can be achieved by the observation device 2 being present in the imageable area Hi. Note that there is a constraint on the order of the propositions "ai" and "bi." That is, when the observation device 2 is in the imageable area Hi while the workpiece 20 is present in the appropriate area Gi, the observation device 2 can photograph the workpiece 20, but if the workpiece 20 is present in the appropriate area Gi after the observation device 2 has been in the imageable area Hi, the observation device 2 may not be able to photograph the workpiece 20. Therefore, if the constraints are such that the proposition "bi" comes first and the proposition "ai" comes later, and both are satisfied, the observation device 2 will be able to reliably photograph the workpiece 20. Such constraints regarding the order of propositions may be included in the constraint condition information I2 of the accumulated data shown in FIG. 2, or may be included in the subtask information I5 described later.

[0051] Next, a more specific process of setting the abstract model by the planning device 10 in step S103 will be described. As described above, the abstract model is a model that abstracts the dynamics in the workspace of the control system 100. The abstract model may be stored as abstract model information I6. To handle dynamics, i.e., time changes, it is necessary to add the concept of time to the above-described goal logical formula. When executing a target task, the control system 100 counts time in time steps. Furthermore, the control system 100 sets the number of time steps required to execute the target task, i.e., the number of time steps from the start to the completion of the target task. The number of time steps required to execute the target task is also referred to as the target time step number. Note that the method for setting the target time step number is not limited to a specific method. For example, the target time step number may be stored in the storage device 3 or may be specified by the user via the input device 1. Furthermore, the duration of the time step when the control system 100 executes the target task is not limited to a specific duration.

[0052] The above-mentioned proposition "?ai" is expanded to include time steps. That is, if the proposition "ai" is satisfied at time step k (k is an integer greater than or equal to 1), it is expressed as "ai, k." In this case, the proposition "?ai" expressed by the operator "?" (eventually, eventually) can be specified by setting the time steps "k = ..., Tk-2, Tk-1, Tk," where Tk is the target time step at which at least this proposition is satisfied, and the condition that the proposition "ai" always holds after a certain time step before Tk. In other words, it is not possible to set an infinite number of steps with the operator "?" (eventually, eventually). Therefore, here, "time step Tk" is the last step in the processing, and no processing is performed after that, but the goal is achieved. Here, the state vectors Xc, Xe, and Xw of the observation device 2, the end effector of the controlled device 4, and the workpiece 20, respectively, are also expanded to include time steps. That is, the respective state vectors at time step k are represented as Xc,k, Xe,k, and Xw,k. Furthermore, to represent whether or not a proposition is satisfied with a value of "0" or "1," a logical variable θi,k at imaging location Pi and time step k is introduced, which takes the value of "0" or "1." If the value of proposition "ai,k" in state vector Xc,k of observation device 2 is represented as "Xc,k[ai,k]," the fact that proposition "ai,k" is true at time step k is equivalent to the state vector Xc,k of observation device 2 being included in the imageable region Hi. Therefore, the logical variable θi,k can be expressed, for example, as in the following equation (1):

[0053]

[0054] Here, "E" is a symbol representing an element. For example, "a is an element of set A" is expressed as "a E A".

[0055] In addition, H i,k in formula (1) represents the imageable area H i as the area at time step k. From formula (1), when the value of the logical variable θ i,k is 1, the proposition “a i,k ” holds.

[0056] Similarly, the proposition "bi" shown in part (b) of Figure 7, "The workpiece 20 is within the range of the appropriate region Gi," is expanded to a proposition in a format that includes time step k. Then, a logical variable ηi,k at time step k is introduced. The fact that the proposition "bi,k" holds true at time step k is equivalent to the state vector Xw,k of the workpiece 20 being included in the appropriate range Gi. Therefore, the logical variable ηi,k can be expressed, for example, as in the following equation (2).

[0057]

[0058] In addition, Gi,k in formula (2) represents the appropriate region Gi as the region at time step k. From formula (2), when the value of the logical variable ηi,k is 1, the proposition “bi,k” holds.

[0059] Next, we introduce the concept of time steps and logical variables to express how the position and orientation of the observation device 2 and the end effector are changed by controlling the controlled device 4, i.e., the robot arm. However, in the configurations illustrated in FIGS. 5 to 7 in the embodiments of the present disclosure, the observation device 2 and the end effector are changed in position and orientation by the same arm (controlled device 4). In other words, changes to the position and orientation of the observation device 2 and the end effector are performed by a single controlled device 4. Therefore, the position and orientation of the observation device 2 and the position and orientation of the end effector cannot be independently brought closer to their target values. In the following description, we assume that one of the state vectors (i.e., either the position and orientation of the observation device 2 or the position and orientation of the end effector) is prioritized to bring closer to the target value. Prioritizing the movement of the state vector Xc,k of the observation device 2 at time step k is represented by a logical variable δc,k, which takes a value of 0 or 1. For example, when the value of the logical variable δc,k is 0, the state vector Xe,k of the end effector is controlled to approach the control target, and when the value of the logical variable δc,k is 1, the state vector Xc,k of the observation device 2 is controlled to approach the control target.

[0060] Next, changing the position and orientation of the workpiece 20 by moving the end effector will be described. For example, consider the case where the end effector grips the workpiece 20 and the position and orientation of the workpiece 20 are changed to target values ​​when the distance between the end effector and the workpiece 20 is equal to or less than a specified distance. This can be expressed using a logical variable δw,k that takes a value of 0 or 1 indicating whether the controlled device 4 is capable of controlling the position and orientation of the workpiece 20. For example, when the value of the logical variable δw,k is 0, the workpiece 20 is not gripped by the end effector and its position and orientation are not changed. Furthermore, when the value of the logical variable δw,k is 1, the workpiece 20 is gripped by the end effector and its position and orientation are changed.

[0061] Using the relationship between equations (1) and (2), the order of each proposition can be expressed as a constraint. First, the constraint for proposition "ai" to be true after proposition "bi" is true, that is, the constraint for satisfying "when the state vector Xw,k of the workpiece 20 enters the appropriate region Gi (the proposition "bi,k" is true), the state vector Xc,k of the observation device 2 can be moved" can be expressed, for example, by the following equations (3) to (5) using the logical operators negation "!", logical inclusion "¥", logical product "#", and temporal logic operators next "&" and always "@".

[0062]

[0063]

[0064]

[0065] Equation (3) indicates that the value of the logical variable δw,k, which indicates a change in the position and posture of the workpiece 20, is 1 at a certain time step k and 0 at the next time step, i.e., when the change in the position and posture of the workpiece 20 is completed, the logical variable ηi,k at the next step becomes 1, and the proposition "bi" is true.

[0066] Equation (4) expresses that the proposition “ai” holds when the value of the logical variable δc,k, which indicates a change in the position and attitude of the observation device 2, is 1 at a certain time step k and 0 at the next time step, i.e., when the change in the position and attitude of the observation device 2 is completed, the logical variable θi,k at the next step becomes 1.

[0067] Furthermore, equation (5) indicates that if the proposition "ai,k" is not true at a certain time step k and the proposition "bi,k" is true, the value of the logical variable δc,k that changes the position and attitude of the state vector Xc,k of the observation device 2 at the next step is 1 (true). Furthermore, equation (5) indicates that if the proposition "ai,k" is true at a certain time step k or the proposition "bi,k" is not true, the value of the logical variable δc,k that changes the position and attitude of the state vector Xc,k of the observation device 2 at the next step is 0 (false). Note that the constraint conditions shown in equations (3) to (5) are merely examples, and the constraint conditions are not limited to equations (3) to (5). Based on the above explanation, an example of a target logical formula that includes a constraint condition is one in which (?ai)#(@!h) and equations (3) to (5) are simultaneously true. Hereinafter, this constraint condition will be referred to as Φ.

[0068] The abstract model representing the dynamics (also called time change or time evolution) of the abstract state illustrated in FIG. 6 or FIG. 7 can be expressed, for example, as in the following equation (6) by using the state vector and logical variables that take the above-mentioned time steps into consideration.

[0069]

[0070] In equation (6), k represents a time step (an integer k≧1), and k−1 represents the step immediately preceding time step k. Therefore, equation (6) represents the relationship between the state vector Xc,k of the observation device 2 and the state vector Xw,k of the workpiece 20 at time step k, and the state vector Xc,k−1 of the observation device 2 and the state vector Xw,k−1 of the workpiece 20 at time step k−1, i.e., the dynamics. Note that in equation (6), uk and vk are vectors related to the control inputs when controlling the observation device 2 and the workpiece 20, respectively. It is desirable that uk and vk are vectors indicating the amount of change per time step. For example, if the control input is a position, uk and vk are vectors indicating a velocity. Furthermore, for example, if the control input is a position angle, uk and vk are vectors indicating an angular velocity. Furthermore, "I" represents a unit matrix. "0" represents a zero matrix. In equation (6), δc,k and δw,k are logical variables indicating whether or not the observation device 2 and the workpiece 20 are controlled, respectively, and take the value of 0 or 1. In other words, equation (6) represents dynamics that includes discrete (logical) variables in addition to continuous variables. For this reason, the system represented by equation (6) is generally called a hybrid system. Note that in the embodiment of the present disclosure, in the configurations exemplified in FIGS. 5 to 7, changes in the positions and attitudes of the observation device 2 and the end effector are executed by a single controlled device 4, so the control inputs uk and vk may be the same variable. In other words, equation (6) is

[0071]

[0072] It should be noted that the control of the observation device 2 and the workpiece 20 may be performed by multiple control devices 6 rather than a single control device 6. For this reason, formula (6) corresponding to multiple control devices 6 is more general. In the following explanation, formula (7) corresponding to one control device 6 will be used. However, the formulation of the abstract model is not limited to formula (6) or formula (7). For example, if there are multiple workpieces, the number of dimensions of the independent state vectors Xw,k in formulas (6) and (7) will increase accordingly.

[0073] Next, a more specific process in step S104 described above in which the planning device 10 acquires and sets the abstract state and reflects it in the abstract model will be described. The planning device 10 acquires, as the abstract state, at least the current values ​​of the state vectors Xc,k and Xw,k exemplified in Equation (6). The abstract state preferably includes both the position and orientation values ​​of the observation device 2 and the workpiece 20. In the configuration in which the observation device 2 illustrated here is mounted on the controlled device 4, the position and orientation of the observation device 2 can be calculated based on values ​​managed by the control device 6 that controls the controlled device 4. Generally, the control device 6 monitors the state (preferably angle information) of the movable part (actuator) of the controlled device 4. Therefore, the control device 6 can acquire this value. The relationship between the angle information indicating the state of the movable part and the state vector of the observation device 2 is determined by the configuration, preferably a geometric relationship. The geometric relationship is the translational and rotational relationship between the reference point of the state of the controlled device 4 and the reference point of the state of the observation device 2. Specific examples of the translational and rotational relationship include a vector representing parallel translation and a rotation matrix representing rotation. In other words, the relationship between translation and rotation indicates where the observation device 2 is installed on the controlled device 4. If the relationship between translation and rotation is given based on the configuration, the control device 6 can calculate the state vector of the observation device 2 from the angle information indicating the state of the movable part. This calculation means may be the general means described above and is not limited to a specific means. As described above, the position and orientation of the workpiece 20 may be acquired by the control system 100 according to an embodiment of the present disclosure, or by means other than the control system 100. Generally, the plan generation unit 13 can use an object recognition method to acquire the position and orientation of the workpiece 20. However, for example, the user may specify the position and orientation of the workpiece 20 via the input device 1. Note that the object recognition here is not applied to time step k, i.e., the entire path. Step S104 is a processing step for "acquiring and reflecting (substituting into an equation) the current state." Therefore, object recognition is not reflected at the present time, i.e., after the initial value of time step k.In other words, object recognition is performed before the processing and operation described below, and the state of the object is not continuously acquired thereafter.

[0074] As described above, once the value of the state vector at a certain time step is known, the values ​​of the state vector at subsequent time steps can be sequentially calculated using the abstract model exemplified by Equation (6). The abstract model exemplified by Equation (6) is preferably given the value of the state vector at the start of a target task and can calculate the change in the state vector until the target task is completed. The value of the state vector at the start of a target task is given by the object recognition or user input described above. Furthermore, the states at subsequent time steps are all given by calculations (e.g., simulations) based on the abstract model (dynamics).

[0075] Next, a more specific process in step S105 described above in which the operation determination unit 11 determines whether or not the workpiece 20 or another object is to be operated will be described.

[0076] In the example environment shown in FIG. 7 , whether or not to manipulate the workpiece 20 in the processing of step S105 corresponds to the truth or falsity of the aforementioned proposition "bi: the position and orientation of the workpiece 20 are within the appropriate region Gi." In other words, according to equation (2), when the value of the logical variable ηi,k is 0, the proposition "bi" is not satisfied, so the workpiece 20 is manipulated; when the value of the logical variable ηi,k is 1, the proposition "bi" is satisfied, so the workpiece 20 is not manipulated. Therefore, the processing of the operation determination unit 11 (step S105) outputs the value of the logical variable ηi,k. Note that the specific example of the environment shown in FIG. 7 is an example in which there are no objects other than the workpiece 20 in the environment, and the reverse of equation (2) holds. In other words, if the condition "Xw,k falls within the appropriate region Gi" on the right side of equation (2) is satisfied, then the proposition "bi,k" on the left side of equation (2) is satisfied. In other words, (by definition) the logical variable is 1. Therefore, the value of the logical variable η at time step k can be determined by evaluating the right-hand side of equation (2). That is, the value of the logical variable η can be determined based on the relationship between the position vector X of the workpiece 20 at time step k and the appropriate area Gi. For example, if the operation determination unit 11 determines that the angle between the normal vector of the imaged location Pi and the appropriate area Gi is equal to or less than a certain threshold, the operation determination unit 11 can set the value of the logical variable η to 1. Furthermore, if the operation determination unit 11 determines that the angle between the normal vector of the imaged location Pi and the appropriate area Gi exceeds a certain threshold, the operation determination unit 11 can set the value of the logical variable η to 0. Once the state of the workpiece 20 is calculated by object recognition, the normal vector of the imaged location Pi can be calculated based on the information stored about the imaged location Pi (the imaged location can also be determined if the current position and orientation of the workpiece 20 are known). As a result, the value of the logical variable can be determined by comparing the imaged location Pi with the appropriate area Gi (for example, comparing the angle between the normal vectors). This determination may be made using a general object recognition method in the first step of starting the target task, and is not limited to using a specific means. Once the state of the workpiece 20 is calculated by object recognition, the normal vector of the imaging point Pi can be calculated based on stored information about the imaging point P (if the current position and orientation of the workpiece are known, the imaging point can also be known).As a result, as described above, the value of the logical variable can be determined by comparing it with the appropriate region (for example, comparing the angle between the normal vectors). In subsequent time steps, the values ​​calculated sequentially by the abstract model illustrated in equation (6) can be referenced.

[0077] Next, a more specific process will be described in which the observation determination unit 12 determines whether the observation device 2 is within the region where the workpiece 20 can be observed by the observation device 2 in step S106. In the example environment shown in FIG. 7, whether the observation device 2 is within the region where the workpiece 20 can be observed by the observation device 2 in this process corresponds to the truth or falsity of the above-mentioned proposition "ai: the state vector Xc,k of the observation device 2 is included in the imageable region H." In other words, according to equation (1), when the value of the logical variable θi,k is 0, the proposition "ai" is not satisfied and the workpiece 20 is not observable, whereas when the value of the logical variable ηi,k is 1, the proposition "ai" is satisfied and the workpiece 20 is observable. Therefore, the process (step S106) of the observation determination unit 12 outputs the value of the logical variable θi,k. Note that the specific example of the environment shown in FIG. 7 is an example in which there are no objects other than the workpiece 20 in the environment, i.e., Xc falls within the imageable area Hi, and the converse of equation (1) holds (i.e., the logical variable is 1). Therefore, the value of the logical variable θi,k at time step k can be determined by evaluating the right-hand side of equation (1). That is, the value of the logical variable θi,k can be determined based on the relationship between the position vector Xc,k of the observation device 2 at time step k and the imageable area Hi. As described above, the observation determination unit 12 can calculate the value of the position vector Xc,k of the observation device 2 at time step k based on the state information of the controlled device 4. The observation determination unit 12 can determine the imageable area Hi based on the task information input by the input device 1, the accumulated data of FIG. 2 (preferably observation device information I3 and object model information I7) stored in the storage device 3, and the state vector Xw,k of the workpiece 20 at time step k. The value of the state vector Xw,k of the workpiece 20 can be obtained by any means, such as object recognition, in the first step of starting the target task, as described above. In subsequent time steps, values ​​calculated sequentially by the abstract model exemplified by Equation (6) can be obtained.

[0078] Next, a more specific process in step S107 in which the plan generation unit 13 generates and outputs an operation plan that satisfies the target logical formula and the abstract model will be described. As described above, in an example in which the target task is an imaging task, the target logical formula is a compilation of the proposition (?ai)#(@!h) and the constraint conditions represented by formulas (3), (4), and (5). In the following description, this compilation of the target logical formula will be represented as Φ. The abstract model will be represented by formula (6) (represented as "Σ"). An operation plan that satisfies the target logical formula Φ and the abstract model (formula (6)) can be obtained by determining the values ​​of the state vector Xc,k, state vector Xw,k, logical variable δc,k, and logical variable δw,k at each time step so as to satisfy the constraint conditions represented by formulas (3), (4), and (5). Note that the values ​​of the logical variables ηi,k and θi,k in equations (3), (4), and (5) representing the constraint conditions may be the values ​​of the logical variables ηi,k and θi,k output by the operation determination unit 11 and the observation determination unit 12. The state vector and the values ​​of the logical variables at each time step (i.e., the time-series motion plan for each step) can be obtained, for example, by minimizing the sum of squares of the norms of the control input uk in the following equation (8).

[0079]

[0080] In equation (8), Φk is an equation combining equations (3), (4), and (5) that indicate constraint conditions. This equation (8) represents an optimization problem in which the target logical formula is the constraint condition and the sum of squares of the norms of the control input uk is the evaluation function. In particular, since it contains logical variables, it is called a mixed integer optimization problem or a mixed integer programming problem. Solutions to mixed integer programming problems are collectively called mixed integer programming (MIP).

[0081] FIG. 8 is a diagram schematically illustrating an example of changes in logic variables and the corresponding actions, assuming the results of solving an optimization problem in an embodiment of the present disclosure. FIG. 8 illustrates an example of an action plan. In part (a) of FIG. 8, the horizontal direction represents the change in time step k (k = 1, 2, ..., 8), and for each logic variable listed at the top, the values ​​at each time step are shown: θi,k representing an observation judgment, δc,k representing control of the observation device, ηi,k representing an operation judgment, and δw,k representing control of the workpiece. Note that the time step interval, total number, and value changes are examples. Furthermore, part (b) of FIG. 8 separately illustrates the actions (controls) corresponding to the changes in the value of each logic variable. For example, when the value of the logic variable δw,k (referred to as δw in FIG. 8 ) representing workpiece control is 1, the position and orientation of the workpiece 20 are being changed, and at time step k = 3, the value of ηi,k (referred to as ηi in FIG. 8 ), indicating an operation judgment, changes from 0 to 1, i.e., the workpiece 20 is included in the appropriate region Gi. Furthermore, at time step k=4, the value of the workpiece control δw,k becomes 0, indicating that the workpiece control δw,k is complete. Therefore, FIG. 8 shows that the position and orientation of the workpiece 20 are changed so that it falls within the appropriate region Gi during time steps k=1, 2, and 3, as indicated by "control of workpiece" in FIG. 8. Next, at time step k=4, the value of the logical variable δc,k (denoted as δc in FIG. 8) representing the control of the observation device 2 changes from 0 to 1. This change indicates that control of the observation device 2 begins after control of the workpiece 20 is completed. Thereafter, the value of the logical variable δc,k remains 1 until time step k=6, so control of the workpiece 20 continues. At time step k=7, the value becomes 0, indicating that control of the observation device 2 is complete. At time step k=6, the value of θi,k representing the observation judgment changes from 0 to 1, indicating that the observation device 2 is included in the imageable region Hi. Therefore, during time steps k=4, 5, and 6, the position and attitude of the observation device 2 are changed so that it enters the imageable area Hi, and FIG. 8 indicates that "observation device is controlled."Here, at time step k=7, the value of θi,k (denoted as θi in FIG. 8 ), which represents the observation judgment, is 1, which indicates that the proposition “ai” is satisfied, i.e., the imaging task can be achieved, and is denoted as “imaging possible” in FIG. 8 .

[0082] Here, the relationship between changes in logical variables and subtasks will be described. A subtask is a task defined as a unit for operating the controlled device 4, which is combined to complete a target task. It is desirable for subtasks to be controlled on a subtask-by-subtask basis. Subtasks may be associated with logical variables. For example, FIG. 8 illustrates an example in which control is switched based on the values ​​of logical variables δw,k and δc,k. Therefore, the unit at which control is switched may be defined as a subtask. That is, in the example of FIG. 8, the period when logical variables δw,k are 1 can be defined as a "subtask controlling the workpiece," the period when logical variables δc,k are 1 can be defined as a "subtask controlling the observation device," and the period when logical variables θi,k are 1 can be defined as a "subtask capturing an image of the workpiece." Such relationships between changes in logical variables and subtasks and control may be stored as subtask information I5 in the storage device 3. Note that the above subtask division is merely an example and is not limited to the above.

[0083] Finally, a specific process in which the control device 6 controls the controlled device 4 in step S108 will be described. The control device 6 outputs a control signal generated based on the operation plan to the controlled device 4. It is desirable that the control device 6 output a control signal to the controlled device 4 in units of subtasks. For example, the operation plan may include, in addition to information indicating the subtask, a time-series target value associated with a time step. The control device 6 can generate a control signal in units of subtasks based on the operation plan. The control method by the control device 6 may use an existing means. However, the means is not limited thereto. For example, the control method by the control device 6 may be a method in which position, speed, etc. are fed back and controlled to follow a time-series target value.

[0084] Here, the order relationship between subtasks will be described. As described above, the change in the logical variable for each time step calculated as a result of solving the optimization problem of Equation (8) already reflects the target logical formula and constraints. In other words, the change in the logical variable satisfies the order constraint. Therefore, the subtasks defined based on the change in the logical variable also satisfy the order constraint. In this way, a feature of the present disclosure is that the order constraint of the subtasks, i.e., the control order constraint, is automatically satisfied by specifying the constraint in the form of a logical formula (proposition) without having to specify the order constraint for each subtask in advance. However, the method, means, and procedure for reflecting the constraint in the logical formula are not limited to specific means and procedures. For example, the constraint may be specified as constraint information I2 stored in the storage device 3, may be specified as subtask information I5, or may be additionally specified by the user via the input device 1.

[0085] The operation of the control system 100 according to the first embodiment has been described above using an example in which the target task is an imaging task, but the above formulation and calculation are merely examples and are not limited to these.

[0086] (Another Operation Example of the First Embodiment) Next, another operation example of the first embodiment will be described. The target task is similarly an imaging task, and an example is shown in which an environment different from the environments exemplified in Figures 5 to 7 is used. Note that the configuration and operation are similar.

[0087] FIG. 9 is a diagram illustrating an example of another abstract state when the target task is an imaging task in the first embodiment of the present disclosure. FIG. 9 illustrates an abstract state for an imaging task corresponding to FIG. 7 . However, unlike FIG. 7 , FIG. 9 illustrates a state in which an obstacle 21 overlaps the workpiece 20. FIG. 9 illustrates an example in which an object other than the workpiece 20 is present in the environment. Note that the number and arrangement of the obstacles are not limited to those illustrated in FIG. 9 . This example illustrates a case in which the planning device 10 acquires and sets the abstract state and reflects it in the abstract model (step S104), acquiring the state vector Xw of the workpiece 20 and the state vector Xo of the obstacle 21. Note that acquiring state information of the workpiece, obstacle, etc. in the environment in this manner depends on the means for acquiring the environment and state information, such as an object recognition means, but this means may be any means. Also, consider a case in which the acquired state information can be identified and classified as either the workpiece 20 or the obstacle 21.

[0088] In this example, the target logical formula and constraints (equations (3), (4), and (5)) remain unchanged. Here, we will explain this operation using an example of a motion plan (specifically, the motion plan shown in FIG. 10 , described later) without adding any additional formulas. While the following describes a case where the condition of which object to control first does not apply, such a condition may be included as a constraint and set appropriately depending on the environment, task, and object. However, the abstract model represented by equation (6) or (7) is modified. Similar to the workpiece 20, the obstacle 21 is assumed to be grasped by the controlled device 4, i.e., the end effector, and its position and orientation can be changed to target values ​​when the distance between the controlled device 4 and the obstacle 21 is within a certain specified range. Furthermore, it is assumed that the obstacle 21 is not grasped by the end effector, and its position and orientation are not changed when the distance between the end effector and the obstacle 21 exceeds a certain specified range. This can be expressed in the same way as the workpiece 20 by adding a new logical variable δo,k, which takes a value of 0 or 1 and indicates whether the controlled device 4 can control the position and orientation of the obstacle 21. That is, when the value of the logical variable δo,k is 0, the obstacle 21 is not grasped by the end effector, and its position and orientation are not changed. On the other hand, when the value of the logical variable δo,k is 1, the obstacle 21 is grasped by the end effector, and its position and orientation are changed. When the state vector and logical variables are newly added in this way, the abstract model can be expressed, for example, as in the following equation (9).

[0089]

[0090] Note that equation (9) is an expanded equation obtained by introducing logical variables δo,k into equation (7) representing the abstract model, but it is also possible to expand equation (6) by introducing logical variables δo,k, but the formulation of equation (9) is not limited to these. Equation (9) is the same as equation (7) except that the state vector Xo and logical variable δo for the obstacle 21 are added.

[0091] From the above, the plan generation unit 13 can generate an optimal motion plan for this environment and output the generated motion plan simply by replacing the abstract model "Σ" in the optimization problem expressed by equation (8) with equation (9). However, here, it is necessary to add constraints on the logical variables δw,k and δo,k that determine the control of the workpiece 20 and obstacle 21. This is because, in the configuration shown in FIG. 9, it is not possible to simultaneously control both the workpiece 20 and the obstacle 21 in the same time step. In other words, the values ​​of the respective logical variables cannot be set to true (1) at the same time. For this reason, it is necessary to add, for example, the following equation (10) as a constraint.

[0092]

[0093] In equation (10), j (an integer equal to or greater than 1) represents the object controlled by the controlled device 4, which in the configuration of FIG. 9 is the workpiece 20, the obstacle 21, or the observation device 2. That is, j in equation (10) includes everything represented by the logical variable δ. For example, j=1 represents the workpiece 20, and j=2 represents the obstacle 21. That is, equation (10) implies a constraint that n objects in the environment cannot be controlled simultaneously. Note that this constraint may not be necessary depending on the device configuration or environment. For example, if there are multiple controlled devices 4, i.e., multiple robot arms, and multiple end effectors are mounted, equation (10) is unnecessary.

[0094] FIG. 10 is a diagram schematically illustrating an example of changes in each logical variable and the corresponding operation when solving the optimization problem by adding the constraint of Equation (10) in an embodiment of the present disclosure. FIG. 10 illustrates an example of an operation plan. The logical variables illustrated in FIG. 10 are obtained by adding a logical variable δo (denoted as δo in FIG. 10) for the obstacle 21 to the logical variables illustrated in FIG. 8 . In FIG. 10, when the value of this logical variable δo is 1 for time steps k = 1 to 3, the obstacle 21 is first controlled, i.e., the position and orientation of the obstacle 21 are changed. The subtask corresponding to this control is labeled "Control Obstacle" in FIG. 10. Note that although a target value for changing the position and orientation of the obstacle 21 is not specified, it can be determined as appropriate. For example, by specifying a position a certain distance away based on the status information of other objects in the workspace, the obstacle 21 can be moved to an area away from the workpiece 20, as shown in FIG. 10. Such a target value may be stored, for example, as constraint condition information I5 in the storage device 3.

[0095] Next, it will be explained that the value of the operation judgment ηi,k, which is a logical variable, is 1 at time step k=3 in Fig. 10. The judgment of the operation judgment ηi,k (described as operation judgment ηi in Fig. 10) is expressed by equation (2). However, in the environments illustrated in Figs. 9 and 10, the converse of equation (2) does not hold. In other words, even if the workpiece 20 is included in the appropriate region Gi, the operation judgment ηi,k does not necessarily become 1 (true). 9 and 10 , the reason is that even if the position and posture of the workpiece 20 are appropriate (i.e., even if the status information of the workpiece 20 can be acquired and the workpiece 20 is included in the appropriate area Gi), there is an influence of other objects such as the obstacle 21 (for example, even if the target task is an imaging task and the relationship of the position and posture of the imaging location Pi and the imaging device 2 (an example of an observation device) is such that the imaging location Pi is included in the imaging range of the imaging device 2, if the obstacle 21 covers the imaging location Pi of the workpiece 20, the imaging device 2 will not actually be able to capture the imaging location Pi, which is an influence of the obstacle 21). Therefore, the determination process of the operation determination ηi,k performed by the operation determination unit 11 needs to be able to handle such an environment. The operation determination unit 11 can handle the determination of the operation determination ηi,k in such an environment by using image processing or object recognition means. For example, the operation determination unit 11 may identify the obstacle 21 and the workpiece 20 by using image processing or object recognition means. If the obstacle 21 is outside the appropriate region G, the operation determination ηi,k may be determined to be 1 (true). If the obstacle 21 is within the appropriate region G, the operation determination ηi,k may be determined to be false. The method for identifying the obstacle 21 and the workpiece 20 is not limited to the method using image processing or object recognition means. For example, the method for identifying the obstacle 21 and the workpiece 20 may be a method in which the user provides identification information via the input device 1. Note that the process for identifying the obstacle 21 and the workpiece 20 is performed at a timing prior to the operation plan (e.g., at the beginning of the process). Therefore, the object is not recognized or determined by the user at time step k=3, but is identified in the initial state (time step k=1).

[0096] In the example shown in FIG. 10 , the value of the operation decision ηi,k is 1 at time step k=3. In this case, no operation to control the workpiece 20 is necessary. For the sake of explanation, three columns for the "Control workpiece" item are shown at the top of FIG. 10 . However, since the value of the operation decision ηi,k is 1 and the value of the workpiece control δw,k, which is a logical variable representing the control of the workpiece 20, is 0 at every time step, the results of the optimization calculation show that it is not necessary to control the workpiece 20. Then, since the conditions for controlling the observation device 2 are satisfied, the value of the control δc,k for the observation device 2 becomes 1 at time step k=4, and control of the observation device 2 is initiated. The operation from time step k=4 onwards is the same as that shown in FIG. 8 , and therefore will not be described here.

[0097] While the above describes operations in different environments, the control system 100 of the embodiment of the present disclosure has the characteristic of being able to accomplish a target task without additional configuration or processing, even in different environments. For convenience of explanation, the cases where only the workpiece 20 is present and where other objects are present are described. Note that in the first embodiment, status information of the workspace is input, and processing proceeds based on constraints and information for executing the task; therefore, conditions for determining the "environment" are not input. In contrast, other operation examples of the first embodiment indicate that "adaptation to the environment is required." This indicates that identification of other objects (obstacles) and their status information are required, and that the presence of a means for identifying other objects (obstacles) indicates that adaptation to the environment is possible. In other words, even if the relationship between the target object (workpiece 20) and the observation device 2 is not ideal, the control system 100 can continue control and provide an operation plan that enables the task to be completed.

[0098] (Advantages) In this way, the control system 100 can achieve precise control of the controlled device.

[0099] Second Embodiment (Device Configuration) FIG. 11 is a diagram illustrating an example of the configuration of a control system 100 according to a second embodiment of the present disclosure. The control system 100 illustrated in FIG. 11 differs from the control system 100 according to the first embodiment in that the controlled device 4 includes multiple control devices, from the first controlled device 4a to the mth controlled device 4m. The number of controlled devices is at least two and is not limited. The other configurations are the same as those of the first embodiment, and therefore will not be described below. Note that FIG. 11 illustrates a configuration in which multiple controlled devices are provided with a control device 6 similar to that of the first embodiment, but the number of control devices 6 and their relationship to the controlled devices 4 are not limited to this configuration.

[0100] (Operation) As shown in FIG. 11 , the second embodiment has multiple controlled devices 4a to 4m, and thus differs from the first embodiment in terms of the abstract model and constraints. For example, in the first embodiment, the abstract model was rewritten from equation (6) to equation (7). However, in the second embodiment of the present disclosure, the control inputs u, v, ... corresponding to each of the controlled devices 4a to 4m can be set individually and independently, as in equation (7). Specifically, if there are two controlled devices, 4a and 4b, the control inputs u and v in equation (7) can be made to correspond to the respective controlled devices. Therefore, even in the same time step, the controlled device 4a and the controlled device 4b can be controlled individually and independently. This also affects the constraints. In the first embodiment, equation (10) had a constraint that only one object can be controlled in the same time step, i.e., the sum of the logical variables related to control is equal to or less than the number of controlled devices. However, this condition is relaxed. Specifically, in the second embodiment of the present disclosure having m controlled devices, the constraint corresponding to equation (10) in the first embodiment is expressed by the following equation (11).

[0101]

[0102] Therefore, in equation (11), the changes in the logical variables related to control are not exclusive, unlike equation (10) of the first embodiment. In other words, in equation (11), when the value of one logical variable is 1, the value of the other logical variables may not be 0, but may be 1. This means that, for the same number of time steps, the control system 100 according to the second embodiment can control more controlled devices 4 than the control system 100 according to the first embodiment, and is expected to improve work efficiency, such as by reducing work time.

[0103] However, Equation (11) holds only when the state change objects j are all different for each controlled device 4. For example, Equation (11) holds when the state of each of the state change objects j is changed by different controlled devices, such as the observation device 2 being the controlled device 4a, the workpiece 20 being the controlled device 4b, and the obstacle 21 being the controlled device 4c. This is because the controlled devices 4a to 4m controlled by multiple control devices 6a to 6m cannot simultaneously target the same state change object j. However, as in the above example, the number of state change objects j and the number m of controlled devices 4 do not need to match, and the correspondence relationship for changing the state can be determined arbitrarily. In other words, the number of state change objects may be fewer than the controlled devices, or conversely, the number of controlled devices may be fewer than the state change objects, and the correspondence relationship for changing the state is not limited. In addition, interference (contact) between the controlled devices 4a to 4m must be avoided. This can be added as a constraint in relation to the proposition "never enter an area defined as an obstacle (@!h)" described as the goal logical formula in the first embodiment. For example, other controlled devices can be included in the area defined as an obstacle, or a constraint on coordinates such as "X4a<X4b" can be included for the x-coordinates (X4a, X4b) of the end effectors of controlled device 4a and controlled device 4b. Note that these are merely examples, and constraints may be added as appropriate depending on the environment, the number of control devices and controlled devices, and the configurations of the control devices and controlled devices.

[0104] (Advantages) In this way, the control system 100 can achieve precise control of the controlled device.

[0105] <Third Embodiment> (Device Configuration) Fig. 12 is a diagram showing an example of the configuration of a control system 100 according to a third embodiment of the present disclosure. The control system 100 shown in Fig. 12 has a configuration in which an evaluation device 5 is further added to the configuration of the control system 100 according to the second embodiment. However, with regard to the number of controlled devices 4, the control system 100 according to the third embodiment of the present disclosure may include multiple controlled devices 4a to 4m as in the second embodiment, or may include a single controlled device 4 as in the first embodiment.

[0106] (Operation) The evaluation device 5 evaluates the results of observation by the observation device 2 executed as the target task. For example, if the target task is an imaging task, the evaluation device 5 receives image information captured by the observation device 2 and outputs an evaluation result. The evaluation result may, for example, indicate whether the range specified as the target task has been captured, whether the image is blurred, and whether the brightness (exposure) is appropriate. The observation determination unit 12 can also accept the evaluation result output by the evaluation device 5 as input. For example, the observation determination unit 12 typically determines whether the observation device 2 is within an observable area based on calculated values ​​for calculating plan information, i.e., before operation. The determination by the observation determination unit 12 is made after the observation determination unit 12 actually operates. Alternatively, a method of switching between before and after the observation determination unit 12 operates can be considered. Note that general image processing techniques can be used for the evaluation performed by the evaluation device 5. For the evaluation performed by the evaluation device 5, the evaluation results to be accepted as input can be appropriately set using image processing techniques appropriate for the target task and the environment.

[0107] A new effect of the evaluation device 5 in the third embodiment of the present disclosure will be described. In the first and second embodiments, the observation determination unit 12 performs the determination process in step S106, for example, based on whether the observation device 2 enters an area where the workpiece 20 can be observed by the observation device 2. That is, the observation determination unit 12 performs the determination in step S106 based on the state vectors, position, orientation, shape, etc., of the workpiece 20 and the observation device 2. This means that the observation determination unit 12 makes a determination based on an abstract state without using real-world information acquired by the observation device 2. However, for the purpose of achieving an imaging task, a determination based on real-world information actually acquired by the observation device 2 is important. Therefore, in the third embodiment of the present disclosure, the observation determination unit 12 performs a determination operation based on the output of the evaluation device 5, thereby enabling the achievement of the target task even when a determination based on an abstract state is inappropriate. That is, the observation determination unit 12 in the third embodiment has the function of performing a process to determine the truth or falsity of the proposition that imaging is possible using real-world information acquired by the observation device 2. For example, the observation / determination unit 12 typically determines the truth or falsity of the proposition that imaging is possible based on calculated values ​​for the calculation of the motion plan, i.e., before the operation. However, this determination by the observation / determination unit 12 is made after the observation / determination unit 12 actually operates. Alternatively, a method of switching between before and after the observation / determination unit 12 operates can be considered. For example, if the evaluation device 5 outputs an evaluation result indicating that the image is blurred (low contrast in the edge portion), the observation / determination unit 12 can determine that the value of the logical variable θi,k is 0, i.e., the proposition "bi" indicating that imaging is possible is false, even if the state vector of the observation device 2 is included in the observable region H. Based on this determination result, the control device 6 may, for example, use a function to change the distance between the observation device 2 and the workpiece 20 by controlling the controlled device 4 in combination with an autofocus function, a visual feedback function, or the like. The control method used by the control device 6 is not limited to a control method based on the position and orientation of the observation device 2.

[0108] Furthermore, the effects of the control system 100 according to the third embodiment of the present disclosure, which includes the evaluation device 5 and multiple controlled devices 4a-4m, will be described. When the output of the evaluation device 5 results in the value of the logical variable θi,k being 0 (i.e., when the evaluation result is inappropriate), the control system 100 according to the third embodiment is not limited to responding by control using only a single controlled device 4. That is, the control system 100 according to the third embodiment may include multiple controlled devices. For example, when the evaluation device 5 outputs an evaluation result indicating "low image brightness (dark)" and the observation determination unit 12 outputs a value of 0 as the logical variable θi,k, in addition to the controlled device 4a that changes the position and attitude of the observation device 2, the controlled device 4b may be a lighting device, and the controlled device 4b may be controlled to illuminate the workpiece 20. This lighting operation can be realized, for example, by adding a new logical variable that determines the control of the lighting device based on the output of the evaluation device 5. As described above, the third embodiment of the present disclosure is characterized by the ability to change the values ​​of multiple logical variables that determine the control of the multiple controlled devices 4a-4m based on the output of the evaluation device 5. In the above, an imaging task is used as an example, and the output of the evaluation device 5 is an evaluation result based on an image, but this is not limited to this. For example, in a task of reading a barcode attached to a workpiece, the reading device is a specific example of the evaluation device 5, and the reading result is an example of the output of the evaluation device 5.

[0109] (Advantages) In this way, the control system 100 can achieve precise control of the controlled device.

[0110] (Application Examples) Application examples based on the first to third embodiments will be described below.

[0111] (First Application Example) The first application example is an example in which the control system 100 in the first embodiment is applied to target tasks such as workpiece inspection, registration, and matching, which are performed at manufacturing sites, logistics sites, etc., with the observation device 2 being a dedicated sensor installed in the workspace and the controlled device 4 being an articulated robot arm. FIG. 13 is a diagram showing a first application example of the control system 100 according to the first embodiment of the present disclosure. FIG. 13 shows an example configuration of the control system 100 as this application example. The other configurations are the same as those in the first embodiment, so description thereof will be omitted.

[0112] In the first application example, examples of the dedicated sensor of the observation device 2 include image acquisition means such as a camera, a barcode reader, an RFID (Radio Frequency Identification) scanner, and a microscope camera (microscope). These dedicated sensors may be used as appropriate depending on the target task. For example, a microscope camera may be used when registering and comparing the surface pattern (fingerprint of a workpiece) for application to product management and traceability.

[0113] The first application example shows an example in which the observation device 2 is not provided in the controlled device 4, and the observation device 2 is fixedly installed at a predetermined position in a predetermined orientation. That is, in the first application example, the position and posture of the observation device 2 remain unchanged. On the other hand, the controlled device 4 is provided with a means capable of manipulating the workpiece 20, specifically, an end effector such as a robot hand. That is, the relative position and posture relationship between the observation device 2 and the workpiece 20 can be changed by changing the position and posture of the workpiece 20 using the controlled device 4. The operation of the control system 100 can be considered to be similar to that of the control system 100 according to the first embodiment.

[0114] The effect of fixing the installation position and orientation of the observation device 2, as in this application example, is described below. By not mounting the observation device 2 on the controlled device 4, the control system 100 can increase the payload capacity of the robot alone, especially when the controlled device 4 is a robot arm. Generally, the payload capacity of a robot arm is determined based on the payload capacity of the robot alone and the weight of the end effector. An example of an end effector is a robot hand. Therefore, if the observation device 2 is mounted near the end effector of the robot arm, the weight of the observation device 2 is added, reducing the payload capacity, i.e., there is a risk that the control of grasping a heavy workpiece or changing its position and orientation will be lost. Therefore, this risk can be reduced by fixing the observation device 2 at a predetermined position and orientation other than the controlled device 4. Furthermore, if the observation device 2 has a complex or large shape, and moves in conjunction with the controlled device 4, the observation device 2 may come into contact with surrounding obstacles 21. Of course, as shown in the first embodiment, for example, the plan generation unit 13 can obtain information about the shape of the observation device 2 from the observation device information I3 and set it as an obstacle area. By doing so, the planner 10 can output a contact-free motion plan based on the constraint "@!h" where "h" is the proposition "exists within the obstacle area." However, because the observation device 2 moves, there is a risk that the motion range of the observation device 2 will be more restricted and the calculation load on the planning device 10 will increase. On the other hand, by configuring the observation device 2 to be installed in the workspace, the observation device 2 can be treated as a static obstacle, and this configuration has the effect of reducing this risk.

[0115] The above describes a first application example in which the observation device 2 is a dedicated sensor with a fixed installation position, and the controlled device 4 is an articulated robot arm, and a target task such as inspection, registration, and verification of the workpiece 20 is assumed. In FIG. 13 , the workpiece 20 is shown as a single workpiece, as in the first embodiment. However, the shape and number of the workpieces 20 targeted for the target task are not limited to the workpiece 20 shown in FIG. 13 . Furthermore, as in the second and third embodiments, the control system 100 may include multiple controlled devices 4, or an evaluation device 5 may be added. However, the control system 100 is not limited to the environment or configuration shown in FIG. 13 .

[0116] (Advantages) In this way, the control system 100 can achieve precise control of the controlled device.

[0117] (Second Application Example) In the second application example, the controlled devices 4a to 4m of the control system 100 in the second or third embodiment are articulated robot arms, a specific area (area) Ak is added, and the control system is applied to a target task that involves not only observation of a workpiece but also manipulation. FIG. 14 is a diagram showing a second application example of the control system 100 according to the first embodiment of the present disclosure. FIG. 14 shows an example configuration of the control system 100 as this application example. The configuration other than that described above is the same as that of the second or third embodiment, and therefore description thereof will be omitted.

[0118] In the example configuration of the environment and control system 100 shown in FIG. 14 , which illustrates the second application example, the robot arm of the controlled device 4a is equipped with an observation device 2. That is, the position and posture of the observation device 2 can be changed by the controlled device 4a. Note that in the figures prior to the second application example, the environment is shown only as a block. However, in FIG. 14 , the environment indicates that a robot arm is located and that a destination (area A) exists. The controlled device 4b is equipped with an end effector such as a robot hand that can grasp the workpiece 20 and change its position and posture. In the second application example, the position and posture of the workpiece 20 are changed by the controlled device 4b. Note that the control system 100 does not necessarily have to include an end effector. As in this application example, each of the controlled devices 4a to 4m may perform a different role. Specifically, there is a degree of freedom in the correspondence between each controlled device and the logical variables that determine control. For example, as in the above example, the controlled device 4a may be controlled based on the logical variable δc that determines the control of the observation device 2, and the controlled device 4b may be controlled based on the logical variable δw that determines the control of the workpiece 20. In this application example, an area A indicating a specific region (area) is added. This can be used as a target value for the destination of the workpiece 20, for example. Specifically, a task such as "image the workpiece 20 and transport it to area A" can be given as a goal task. Such a goal task can be expressed, for example, by a proposition "c" that "the position Xw of the workpiece 20 is ultimately located within area A." As a result, the goal tasks for the entire process, including imaging the workpiece 20 and avoiding contact with obstacles, can be expressed, for example, as "(?ai)#(?c)#(@!h)." However, there is a constraint that the goal task "c" for transporting the workpiece to area A must be executed after the goal task "ai" for imaging. This constraint condition can be set, for example, as a condition that indicates the order between the logical variable θi that indicates observability and the logical variable that determines the control for transporting to area A. Note that the controlled device 4b may be used for control to change the position and posture of the workpiece 20 before transporting to area A.As explained in the above embodiment, this operation satisfies the constraints of the order relationship because the condition that the change in the position and orientation of the workpiece 20 is completed before imaging becomes possible has already been set. Note that the environment shown in FIG. 14 and the above operation are merely examples and are not limited to these. For example, there may be multiple area A and workpieces 20, and the area may be different for each workpiece. Furthermore, the destination area may be changed based on the evaluation results of the evaluation device 5.

[0119] The control system 100 of the second application example has been described above. Generally, when planning tasks involving such complex control, the order between tasks and between controlled devices is important. Therefore, generating a plan is typically time-consuming, and there is a risk that an inappropriate plan may result in malfunctions in the operation of the controlled device. The control system 100 of the second application example has the characteristic of being able to execute a complex task (a complete task), such as imaging and transportation, by associating different target tasks, i.e., propositions, with multiple controlled devices 4a-4m. Therefore, the control system 100 of the second application example generates an optimal operation plan without the user's awareness, even for complex and complex tasks involving different target tasks, simply by setting constraints between logical variables. Therefore, the control system 100 of the second application example has the effect of reducing the above-mentioned risks.

[0120] In the control system 100 of the second application example, the controlled devices 4a to 4m are multi-joint robot arms, a specific area is added, and complex target tasks such as imaging and transportation are assumed. However, the number of controlled devices 4a to 4m, the number of workpieces 20, and the number of areas in the control system 100 of the second application example are not limited to the example of the control system 100 shown in Figure 14. Furthermore, although imaging and transportation to the area are set as target tasks in the above example, this application example is not limited to these.

[0121] (Advantages) In this way, the control system 100 can achieve precise control of the controlled device.

[0122] (Third Application Example) The third application example is an application example in which the observation device 2 of the control system 100 in the second or third embodiment is replaced with a plurality of observation devices (for example, observation devices 2a and 2b), and the controlled devices 4a to 4m are replaced with devices such as an articulated robot arm and a belt conveyor that transports the workpiece 20. FIG. 15 is a diagram showing a third application example of the control system 100 according to the first embodiment of the present disclosure. FIG. 15 shows an example configuration of the control system 100 in this application example. The configuration other than that described above is the same as that of the second or third embodiment, and therefore description thereof will be omitted.

[0123] The example of the control system 100 shown in FIG. 15 is characterized by the inclusion of multiple observation devices 2a and 2b. For example, as in the first embodiment, the observation device 2a performs an imaging task for the workpiece 20 and is mounted on the controlled device 4a, thereby enabling its position and orientation to be changed. As in the first application example, the observation device 2b is an observation device with a fixed installation position and capable of acquiring state information of objects in the environment, including the workpiece 20. Specifically, in the above-described embodiment, any means may be used to acquire position and orientation information of workpieces and obstacles. However, in this application example, it is assumed that the observation device 2b is used as this means. That is, the observation device 2b acquires observation information for estimating the positions and orientations of objects in the environment, including the workpiece 20, preferably a three-dimensional state vector for the position and a three-dimensional state vector for the orientation. The method of estimating the state vector based on this observation information is the same as the method described in the first embodiment. Another feature is the difference in the types of the controlled devices 4a and 4b. In the embodiment of the present disclosure, there is no particular limitation on the type of controlled device 4, and the only assumption is that the movable part (actuator) of the controlled device 4 can be controlled based on the operation plan output by the planning device 10. Therefore, it is also possible to handle a controlled device 4b such as a belt conveyor shown in FIG. 15 . Specifically, a workpiece 20 is placed on the movable part of the controlled device 4b, and a target location is input, and the controlled device 4b is controlled to transport the workpiece 20 to the target location. In other words, the position of the workpiece 20 is changed by the controlled device 4b. Therefore, the control system 100 can be associated with a logical variable δw that determines the control of the position of the workpiece 20. For example, the control system 100 can control the workpiece 20 to move when the value of the logical variable δw is 1 and stop when the value of the logical variable δw is 0.

[0124] Next, the features of this application example will be described. First, because multiple observation devices exist, they can be used in cooperation or in conjunction with each other. A specific example is described below. In the above example, it was assumed that the observation device 2b could acquire observation information about objects in the environment, including the workpiece 20. However, one method is to set a proposition "d" such that "the workpiece 20 can be observed" and, before controlling the workpiece 20, set a constraint that this proposition "d" be 1 (true). Here, the observation device 2a is responsible for the imaging task of the workpiece 20 as the target task, so the observation device 2a may output observation information about the workpiece 20. However, as mentioned above, depending on the configuration of the environment, such as the position of the workpiece 20 and obstacles, it may be determined that either the observation device 2a or the observation device 2b can observe the workpiece 20, or that neither the observation device 2a nor the observation device 2b can observe the workpiece 20. Therefore, it is difficult to determine in advance which observation device to use for observation, and there is a risk that it may also be difficult to set the conditions and thresholds for making the determination. In such a case, using this application example, for example, a logical variable that determines whether the workpiece is observed by the observation device 2b can be added independently of the logical variable θi that determines whether the workpiece is observed by the observation device 2a, and proposition "d" can be further added. By solving the optimization problem under this condition, the value of the logical variable is output as the appropriate value for observation by the observation device 2a or 2b. Therefore, the control system 100 has the effect of reducing the risk of the above-mentioned presetting. Furthermore, one of the features of this application example is that the controlled device to be controlled is not limited to a robot arm, as exemplified by a belt conveyor as the controlled device 4b.

[0125] The above describes the control system 100 of the third application example, which has a plurality of observation devices 2a, 2b, and the controlled devices 4a to 4m are devices such as an articulated robot arm and a belt conveyor that transports the workpieces 20. However, the number, type, and configuration of the observation devices 2, the number, type, and configuration of the controlled devices 4a to 4m, and the number, type, and shape of the workpieces 20 are not limited to those of the control system 100 illustrated in FIG.

[0126] (Advantages) In this way, the control system 100 can achieve precise control of the controlled device.

[0127] The present invention has been described above using the above-mentioned embodiments and application examples as examples. However, the present invention is not limited to the above-mentioned contents. The present invention can be applied to various forms within the scope of the gist of the present invention.

[0128] A control system 100 with a minimum configuration according to an embodiment of the present disclosure will be described. FIG. 16 is a diagram illustrating the control system 100 with a minimum configuration according to an embodiment of the present disclosure. As shown in FIG. 16, the control system 100 with a minimum configuration according to an embodiment of the present disclosure includes a first processing unit 101 (an example of a first processing unit), a second processing unit 102 (an example of a second processing unit), a third processing unit 103 (an example of a third processing unit), and a fourth processing unit 104 (an example of a fourth processing unit). The first processing unit 101 determines whether to change the relationship between the position and orientation of the observation device and the workpiece based on at least one of information about the target task input via an input device, observation device information about the observation device that realizes the target task, object model information about the workpiece that is the target of the target task, controlled device information about the controlled device that changes the relationship between the position and orientation of the observation device and the workpiece, and constraint information that must be satisfied to realize the target task. The first processing unit 101 can be realized, for example, using the functions of the operation determination unit 11 illustrated in FIG. 1. The second processing unit 102 determines whether the observation device can observe the workpiece. The second processing unit 102 can be realized, for example, by using the functions of the observation and determination unit 12 illustrated in Fig. 1. The third processing unit 103 outputs an operation plan for executing the target task based on the determination result by the second processing unit 102. The third processing unit 103 can be realized, for example, by using the functions of the plan generation unit 13 illustrated in Fig. 1. The fourth processing unit 104 controls the controlled device based on the operation plan. The fourth processing unit 104 can be realized, for example, by using the functions of the control device 6 illustrated in Fig. 1.

[0129] Next, a description will be given of the processing of the control system 100 with the minimum configuration according to the present disclosure. Fig. 17 is a diagram showing an example of a processing flow of the control system with the minimum configuration according to the present disclosure. Here, the processing of the control system 100 with the minimum configuration will be described with reference to Fig. 17.

[0130] The first processing unit 101 determines whether to change the relationship between the position and orientation of the observation device and the workpiece based on at least one of information about the target task input by the input device, observation device information about the observation device that realizes the target task, object model information about the workpiece that is the target of the target task, controlled device information about the controlled device that changes the relationship between the position and orientation of the observation device and the workpiece, and constraint information that must be satisfied to realize the target task (step S1). The second processing unit 102 determines whether the observation device can observe the workpiece (step S2). The third processing unit 103 outputs an operation plan for executing the target task based on the determination result by the second processing unit 102 (step S3). The fourth processing unit 104 controls the controlled device based on the operation plan (step S4).

[0131] (Advantages) In this way, the control system 100 can achieve precise control of the controlled device.

[0132] The order of the processes in the embodiments of the present disclosure may be changed as long as the processes are performed appropriately.

[0133] Although the embodiments of the present disclosure have been described, the control system 100, the control device 6, and other control devices may have a computer device inside. The above-described processing steps are stored in the form of a program on a computer-readable recording medium, and the computer reads and executes this program to perform the above processing. Specific examples of computers are shown below.

[0134] FIG. 18 is a schematic block diagram showing the configuration of a computer according to at least one embodiment. As shown in FIG. 18, the computer 50 includes a CPU 60, a main memory 70, a storage 80, and an interface 90. For example, the control system 100, the control device 6, and other control devices described above are each implemented in the computer 50. The operations of the above-described processing units are stored in the storage 80 in the form of a program. The CPU 60 reads the program from the storage 80, loads it into the main memory 70, and executes the above-described processing in accordance with the program. The CPU 60 also allocates storage areas in the main memory 70 corresponding to the above-described storage units in accordance with the program.

[0135] Examples of storage 80 include a hard disk drive (HDD), a solid state drive (SSD), a magnetic disk, a magneto-optical disk, a compact disc read-only memory (CD-ROM), a digital versatile disc read-only memory (DVD-ROM), and a semiconductor memory. Storage 80 may be internal media directly connected to the bus of computer 50, or may be external media connected to computer 50 via interface 90 or a communication line. Furthermore, if this program is distributed to computer 50 via a communication line, computer 50 that receives the program may load the program into main memory 70 and execute the above-described processing. In at least one embodiment, storage 80 is a non-transitory tangible storage medium.

[0136] The program may also implement some of the functions described above. Furthermore, the program may be a file that can implement the functions described above in combination with a program already stored in the computer device, a so-called differential file (differential program).

[0137] Although several embodiments of the present disclosure have been described, these embodiments are merely examples and do not limit the scope of the disclosure. Various additions, omissions, substitutions, and modifications may be made to these embodiments without departing from the spirit of the disclosure.

[0138] Note that part or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes.

[0139] (Supplementary Note 1) A control system comprising: a first processing means that determines whether or not to change the relationship between the position and orientation of the observation device and the workpiece based on at least one of information about a target task input by an input device, observation device information about an observation device that realizes the target task, object model information about a workpiece that is the target of the target task, controlled device information about a controlled device that changes the relationship between the position and orientation of the observation device and the workpiece, and constraint condition information that must be satisfied to realize the target task; a second processing means that determines whether or not the observation device can observe the workpiece; a third processing means that outputs plan information for executing the target task based on the determination result by the second processing means; and a fourth processing means that controls the controlled device based on the plan information.

[0140] (Supplementary Note 2) The control system according to Supplementary Note 1, wherein the first processing means and the second processing means output a determination result based on abstract state information relating to an abstract state, which is an abstract state in a workspace in which the target task is executed.

[0141] (Supplementary Note 3) The control system according to Supplementary Note 1 or Supplementary Note 2, wherein the third processing means outputs the plan information based on abstract state information relating to an abstract state, which is an abstract state in a workspace in which the target task is executed, and abstract model information relating to a temporal or spatial change of the abstract state.

[0142] (Supplementary Note 4) The control system described in any one of Supplementary Note 1 to Supplementary Note 3, wherein the third processing means outputs the plan information in units of subtasks by correlating subtasks with changes in abstract states, which are abstract states in a workspace in which the target task is executed, based on subtask information regarding subtasks into which operations required to complete the target task are broken down; and the fourth processing means controls the controlled device in units of subtasks.

[0143] (Supplementary Note 5) The control system according to any one of Supplementary Note 1 to Supplementary Note 4, wherein the abstract state included in the abstract model information relating to the abstract state, which is an abstract state in a workspace in which the target task is executed, includes continuous variables that allow continuous change and logical variables that represent logical values, the judgments made by the first processing means and the second processing means are related to the logical variables, and the third processing means outputs the changes over time of the continuous variables and the logical variables as the plan information.

[0144] (Supplementary Note 6) The control system according to any one of Supplementary Note 1 to Supplementary Note 5, wherein there are a plurality of controlled devices, and the control of each of the controlled devices is associated with a continuous variable that allows continuous change and a logical variable that represents a logical value.

[0145] (Supplementary Note 7) The control system described in any one of Supplementary Notes 1 to 6, wherein the third processing means outputs planning information including information regarding the execution order of the subtasks over time, based on subtask information regarding subtasks into which operations required to complete the target task are broken down, and on time changes of continuous variables that allow continuous change and logical variables that represent logical values, and the fourth processing means, whether the fourth processing means is one or two or more, executes control of each subtask and each controlled device in the execution order based on the planning information.

[0146] (Supplementary Note 8) The control system according to any one of Supplementary Note 1 to Supplementary Note 7, further comprising an evaluation device that outputs an evaluation result of the observation information acquired by the observation device, and the second processing means makes a judgment based on the evaluation result.

[0147] (Supplementary Note 9) A control method comprising: determining whether to change the relationship between the position and posture of the observation device and the workpiece based on at least one of information regarding a target task input by an input device, observation device information about an observation device that realizes the target task, object model information about a workpiece that is the target of the target task, controlled device information about a controlled device that changes the relationship between the position and posture of the observation device and the workpiece, and constraint condition information that must be satisfied to realize the target task; determining whether the observation device can observe the workpiece; outputting plan information for executing the target task based on the determination result; and controlling the controlled device based on the plan information.

[0148] (Supplementary Note 10) A recording medium storing a program that causes a computer to execute the following: determining whether to change the relationship between the position and posture of the observation device and the work based on at least one of information about the target task input by an input device, observation device information about the observation device that realizes the target task, object model information about the work that is the target of the target task, controlled device information about the controlled device that changes the relationship between the position and posture of the observation device and the work, and constraint condition information that must be satisfied to realize the target task; determining whether the observation device can observe the work; outputting plan information for executing the target task based on the determination result; and controlling the controlled device based on the plan information.

[0149] According to the control system according to the present disclosure, precise control of the controlled device can be realized.

[0150] REFERENCE SIGNS LIST 1... Input device 2, 2a, 2b... Observation device 3... Storage device 4, 4a, ..., 4m... Controlled device 5... Evaluation device 6... Control device 11... Operation determination unit 12... Observation determination unit 13... Plan generation unit 20... Work 21... Obstacle 100... Control system

Claims

1. a first processing means for determining whether or not to change the relationship between the position and orientation of the observation device and the workpiece based on at least one of information about the target task input by an input device, observation device information about the observation device that realizes the target task, object model information about the workpiece that is the target of the target task, controlled device information about the controlled device that changes the relationship between the position and orientation of the observation device and the workpiece, and constraint condition information that must be satisfied to realize the target task; a second processing means for determining whether the observation device can observe the workpiece; a third processing means for outputting plan information for executing the target task based on the determination result by the second processing means; a fourth processing means for controlling the controlled device based on the plan information; A control system comprising:

2. the first processing means and the second processing means output a determination result based on abstract state information relating to an abstract state which is an abstract state in a workspace in which the target task is executed. The control system of claim 1 .

3. the third processing means outputs the plan information based on abstract state information relating to an abstract state, which is an abstract state in a workspace where the target task is executed, and abstract model information relating to a temporal or spatial change of the abstract state.

3. The control system according to claim 1 or 2.

4. the third processing means outputs the plan information in units of subtasks by associating a change in an abstract state, which is an abstract state within a workspace in which the target task is executed, with a subtask based on subtask information relating to subtasks into which actions required to complete the target task are broken down; the fourth processing means controls the controlled device in units of the subtasks, 3. A control system according to claim 1 or 2.

5. the abstract state included in the abstract model information relating to the abstract state, which is an abstract state in a workspace for executing the target task, includes continuous variables which allow continuous changes and logical variables which represent logical values, and the determinations made by the first processing means and the second processing means are related to the logical variables; the third processing means outputs the time changes of the continuous variables and the logical variables as the planning information; 3. A control system according to claim 1 or 2.

6. There are a plurality of the controlled devices, and the control of each of the controlled devices is associated with a continuous variable that allows continuous change and a logical variable that represents a logical value.

3. A control system according to claim 1 or 2.

7. the third processing means outputs the plan information including information regarding the execution order of the subtasks for each time period, based on subtask information regarding subtasks into which operations necessary to complete the target task are broken down, and on time changes of continuous variables that allow continuous changes and logical variables that represent logical values; The fourth processing means executes control of each subtask and each controlled device in an execution order based on the plan information, regardless of whether the fourth processing means is one or two or more.

3. A control system according to claim 1 or 2.

8. an evaluation device that outputs an evaluation result of the observation information acquired by the observation device, and the second processing means makes a judgment based on the evaluation result; 3. A control system according to claim 1 or 2.

9. determining whether or not to change the relationship between the position and orientation of the observation device and the workpiece based on at least one of information about the target task input by an input device, observation device information about an observation device that realizes the target task, object model information about a workpiece that is the target of the target task, controlled device information about a controlled device that changes the relationship between the position and orientation of the observation device and the workpiece, and constraint condition information that must be satisfied to realize the target task; determining whether the observation device can observe the workpiece; Based on the result of the judgment, outputting plan information for executing the target task; controlling the controlled device based on the planning information; Control method.

10. determining whether to change the relationship between the position and orientation of the observation device and the workpiece based on at least one of information about the target task input by an input device, observation device information about an observation device that realizes the target task, object model information about a workpiece that is the target of the target task, controlled device information about a controlled device that changes the relationship between the position and orientation of the observation device and the workpiece, and constraint condition information that must be satisfied to realize the target task; Determining whether the observation device can observe the workpiece; outputting plan information for executing the target task based on the determination result; controlling the controlled device based on the planning information; A program that causes a computer to execute the following.