Control Device, Control Method, and Program

Abstract

The present invention provides a control device comprising: a constraint condition selection means that selects environment-compliant constraint condition data which is of environment-compliant constraint condition data in which environment characteristic information, which is information pertaining to an execution environment of a process and a constraint condition for execution of the process in that environment are linked to each other and corresponds to an actual environment, which is an execution environment of a process to be executed; and a control execution means that controls a control object to execute the process to be executed on the basis of the constraint condition indicated by the selected environment-compliant constraint condition data.

Description

Control device, constraint condition selection device, data generation device, control method, constraint condition selection method, data generation method, and storage medium

[0001] The present invention relates to a control device, a constraint condition selection device, a data generation device, a control method, a constraint condition selection method, a data generation method, and a storage medium.

[0002] A technology has been proposed for acquiring control commands for a robot in accordance with the robot's operating environment and controlling the robot. For example, a trained model generation device described in Patent Literature 1 generates a training dataset in which feature values ​​extracted by an autoencoder from sensor data related to the robot's operating environment are associated with actions to be performed by the robot. The trained model generation device then references the obtained training dataset to generate a trained model that defines the relationship between the operating environment and the actions to be performed by the robot. Furthermore, a robot control device generates commands corresponding to the operating environment by inputting feature values ​​extracted from sensor data related to the operating environment of the controlled robot into the trained model. The robot control device operates the controlled robot by transmitting the generated commands to the controlled robot.

[0003] Japanese Patent Application Laid-Open No. 2020-113262

[0004] In cases where the processes to be performed by a controlled object such as a robot are different, it is preferable to be able to make the controlled object perform each of the processes.

[0005] An example of an object of this disclosure is to provide a control device, a constraint condition selection device, a data generation device, a control method, a constraint condition selection method, a data generation method, and a storage medium that can solve the above-mentioned problems.

[0006] According to a first aspect of the present invention, the control device comprises a constraint condition selection means for selecting environmentally-compatible constraint condition data corresponding to the actual environment, which is the execution environment of the process to be executed, from environmentally-compatible constraint condition data in which environmental characteristic information, which is information about the execution environment of the process, is linked to constraint conditions for executing the process in that environment, and a control execution means for controlling the controlled object to execute the process to be executed based on the constraint conditions indicated by the selected environmentally-compatible constraint condition data.

[0007] According to a second aspect of the present invention, the constraint selection device includes a constraint selection means for selecting environmentally-compatible constraint data corresponding to the actual environment, which is the execution environment of the process to be executed, from environmentally-compatible constraint data in which environmental characteristic information, which is information about the execution environment of the process, is linked to constraints for executing the process in that environment.

[0008] According to a third aspect of the present invention, a data generation device includes a constraint condition learning means for learning constraint conditions based on the results of a simulation of a process, and an environment-responsive constraint condition data generation means for linking information about the environment in the simulation of the process with the constraint conditions obtained by learning, thereby generating environment-responsive constraint condition data in which environmental feature information, which is information about the environment, is linked with the constraint conditions.

[0009] According to a fourth aspect of the present invention, the control method includes a computer selecting environmentally-adapted constraint condition data corresponding to the actual environment, which is the execution environment of the process to be executed, from environmentally-adapted constraint condition data that links environmental characteristic information, which is information about the execution environment of the process, with constraint conditions for executing the process in that environment, and controlling the controlled object to execute the process to be executed based on the constraint conditions indicated by the selected environmentally-adapted constraint condition data.

[0010] According to a fifth aspect of the present invention, the constraint selection method includes a computer selecting environmentally-adapted constraint data that corresponds to the actual environment, which is the execution environment of the process to be executed, from environmentally-adapted constraint data that links environmental characteristic information, which is information about the execution environment of the process, with constraints for executing the process in that environment.

[0011] According to a sixth aspect of the present invention, the data generation method includes a computer learning constraints based on the results of a simulation of a process, and linking information about the environment in the simulation of the process with the constraints obtained by learning, thereby generating environment-responsive constraint data in which environmental feature information, which is information about the environment, is linked with the constraints.

[0012] According to a seventh aspect of the present invention, the recording medium is a recording medium having recorded thereon a program for causing a computer to select environmentally-adapted constraint condition data corresponding to the actual environment, which is the execution environment of the process to be executed, from environmentally-adapted constraint condition data in which environmental characteristic information, which is information about the execution environment of the process, is linked to constraint conditions for executing the process in that environment, and to control the controlled object to execute the process to be executed based on the constraint conditions indicated by the selected environmentally-adapted constraint condition data.

[0013] According to an eighth aspect of the present invention, the recording medium is a recording medium having recorded thereon a program for causing a computer to select environmentally-adapted constraint condition data that corresponds to the actual environment, which is the execution environment of the processing to be executed, from environmentally-adapted constraint condition data that links environmental characteristic information, which is information about the execution environment of the processing, with constraint conditions for executing the processing in that environment.

[0014] According to a ninth aspect of the present invention, the recording medium is a recording medium having recorded thereon a program for causing a computer to learn constraint conditions based on the results of a simulation of a process, and to generate environmentally responsive constraint condition data in which environmental feature information, which is information about the environment, is linked to the constraint conditions by linking information about the environment in the simulation of the process with the constraint conditions obtained by learning.

[0015] According to the present invention, it is expected that it will be possible to accommodate cases where the processes to be performed by a controlled object such as a robot are different, and to have the controlled object perform each of the processes.

[0016] 1 is a diagram illustrating an example of the configuration of a control system according to the first embodiment. FIG. 1 is a diagram illustrating an example of the configuration of a control device according to the first embodiment. FIG. 2 is a diagram illustrating an example of a captured image of a real environment according to the first embodiment. FIG. 3 is a diagram illustrating a first example of an environment graph according to the first embodiment. FIG. 4 is a diagram illustrating a first example of an environment graph according to the first embodiment. FIG. 5 is a diagram illustrating a second example of an environment graph according to the first embodiment. FIG. 6 is a diagram illustrating a third example of an environment graph according to the first embodiment. FIG. 7 is a diagram illustrating a fourth example of an environment graph according to the first embodiment. FIG. 8 is a diagram illustrating an example of data flow when a control device according to the first embodiment controls a control target. FIG. 9 is a diagram illustrating an example of a processing procedure by which a control device according to the first embodiment controls a control target. FIG. 10 is a diagram illustrating an example of the configuration of a control device according to the second embodiment. FIG. 11 is a diagram illustrating an example of data flow when a control device according to the second embodiment generates new environmentally adaptive constraint data. FIG. 12 is a diagram illustrating an example of a processing procedure by which a control device according to the second embodiment controls a control target. FIG. 13 is a diagram illustrating an example of the configuration of a control device according to the third embodiment. FIG. 14 is a diagram illustrating an example of data flow when a control device according to the third embodiment generates new environmentally adaptive constraint data. FIG. 15 is a diagram illustrating an example of a processing procedure by which a control device according to the third embodiment controls a control target. FIG. 16 is a diagram illustrating an example of the configuration of a control device according to the fourth embodiment. FIG. 17 is a diagram illustrating an example of data flow when a control device according to the fourth embodiment generates new environmentally adaptive constraint data. FIG. 18 is a diagram illustrating an example of a processing procedure by which a control device according to the fourth embodiment controls a control target. FIG. 19 is a diagram illustrating an example of processing performed by a control target in accordance with control by the control device according FIG. 10 is a diagram showing an example of the configuration of a control device according to a fifth embodiment. FIG. 11 is a diagram showing an example of the configuration of a constraint condition selection device according to a sixth embodiment. FIG. 12 is a diagram showing an example of the configuration of a data generation device according to a seventh embodiment. FIG. 13 is a diagram showing an example of the processing procedure in a control method according to an eighth embodiment. FIG. 14 is a diagram showing an example of the processing procedure in a constraint condition selection method according to a ninth embodiment. FIG. 15 is a diagram showing an example of the processing procedure in a data generation method according to a tenth embodiment. FIG. 16 is a schematic block diagram showing the configuration of a computer according to at least one embodiment.

[0017] The following describes embodiments of the present invention, but the following embodiments do not limit the scope of the invention as claimed. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0018] <First embodiment> Fig. 1 is a diagram showing an example of the configuration of a control system according to the first embodiment. In the configuration shown in Fig. 1, the control system 1 includes a control device 100 and a control target 900. The control system 1 is a system in which the control target 900 operates under the control of the control device 100.

[0019] The control device 100 controls the control target 900 so that the control target 900 performs the processing to be executed. In particular, the control device 100 determines constraint conditions for the control target 900 to perform the processing, depending on the execution environment of the processing. Then, the control device 100 controls the control target 900 based on the determined constraint conditions.

[0020] Specifically, the control device 100 stores data linking information about the process execution environment with constraints on the execution of the process in that environment. The information about the process execution environment is also referred to as environmental characteristic information. The data linking information about the process execution environment with constraints on the execution of the process in that environment is also referred to as environment-related constraint data.

[0021] The control device 100 then compares the environmental characteristic information included in the environmentally adaptive constraint data with the environmental characteristic information in the execution environment of the process to be executed, and selects environmentally adaptive constraint data that is appropriate for the execution environment of the process to be executed. The control device 100 reads the constraint conditions from the selected environmentally adaptive constraint data, thereby obtaining the constraint conditions that are appropriate for the execution environment of the process to be executed. The control device 100 is an example of a constraint selection device. The process to be executed is also referred to as the target process. The execution environment of the target process is also referred to as the real environment. The environmental characteristic information in the real environment is also referred to as real environment characteristic information.

[0022] By acquiring constraint conditions according to the real environment, the control device 100 is expected to be able to handle not only differences in the execution environment of the process but also differences in the process to be executed. For example, a constraint condition that the control target 900 must avoid certain obstacles when moving can be applied to various processes that involve the movement of the control target 900.

[0023] Furthermore, the control device 100 may acquire constraints according to not only the actual environment but also the process to be executed. For example, as will be described later, information about the target state may be used as environmental characteristic information in addition to information about the initial state of the execution environment of the process.

[0024] The control device 100 may be configured using a computer such as a personal computer (PC) or a workstation (WS).

[0025] The control target 900 operates under the control of the control device 100. The control target 900 is not limited to a specific one, and can be various controllable objects. The processing performed by the control target 900 can be various processing depending on the control target 900.

[0026] For example, the control target 900 may be an articulated robot (manipulator). In this case, various processes, such as grasping and moving an object or processing a product, can be set as the process of the control target, depending on the type of end effector.

[0027] Furthermore, the control target 900 may be a mobile object such as an automated guided vehicle or a drone. In this case, various processes involving the movement of the control target 900, such as transporting an object, can be the processes to be executed. Furthermore, the control target 900 may be a single device, or a system including multiple devices, such as a power plant or a chemical plant. In this case, various processes can be the processes to be executed depending on the type of device or system.

[0028] The process execution environment is also referred to as the operating environment of the control target 900. However, there is no need for a one-to-one correspondence between the process execution environment and the control target 900. In a specific environment, one of the multiple control targets 900 may be selectively operated, or the multiple control targets 900 may be made to operate cooperatively. For example, the above-mentioned constraint that the control target 900 must avoid a predetermined obstacle when moving can be used when the control target 900 is various types of mobile equipment.

[0029] The control target 900 itself may be included in the operating environment of the control target 900. For example, the state of the operating environment of the control target 900 may include the state of the control target 900.

[0030] Fig. 2 is a diagram showing an example of the configuration of the control device 100. In the configuration shown in Fig. 2, the control device 100 includes a communication unit 110, a display unit 120, an operation input unit 130, a storage unit 180, and a processing unit 190. The storage unit 180 includes an environment-associated constraint condition data storage unit 181. The processing unit 190 includes a real-environment characteristic information generation unit 191, a similarity calculation unit 192, a constraint condition selection unit 193, and a control execution unit 194.

[0031] The communication unit 110 communicates with other devices. For example, the communication unit 110 may receive observation data of the real environment from a sensor that observes the real environment. The communication unit 110 may also transmit a control command to the control target 900.

[0032] The display unit 120 has a display screen such as a liquid crystal panel or an LED (Light Emitting Diode) panel, and displays various images. For example, the display unit 120 may display various information related to the execution of processing by the control target 900, such as constraint conditions selected by the control device 100 or real-time images of the control target 900.

[0033] The operation input unit 130 includes input devices such as a keyboard and a mouse, and receives user operations. For example, the operation input unit 130 may receive a user operation that specifies a process to be executed by the control system 1 (a process to be executed).

[0034] The storage unit 180 stores various types of data. The storage unit 180 is configured using a storage device provided in the control device 100. The environment-related constraint condition data storage unit 181 stores environment-related constraint condition data.

[0035] The processing unit 190 performs various processes by controlling each unit of the control device 100. The functions of the processing unit 190 are performed, for example, by a CPU (Central Processing Unit) included in the control device 100 reading and executing a program from the storage unit 180.

[0036] The real environment characteristic information generation unit 191 generates real environment characteristic information based on observation data of the real environment obtained by a sensor that observes the real environment. As described above, the real environment characteristic information is environmental characteristic information in the real environment. The environmental characteristic information is information related to the execution environment of the processing. The real environment characteristic information generation unit 191 corresponds to an example of a real environment characteristic information generation means.

[0037] The environmental characteristic information is not limited to specific information, and can be various information for which similarity can be calculated. For example, various sensors can be used to generate the real environment characteristic information depending on the process of the control target 900 or the execution target. A camera may be used as the sensor, and the real environment characteristic information generation unit 191 may acquire a captured image of the real environment as observation data of the real environment. Alternatively, a sensor that outputs some value, such as the air temperature around the control target 900 or the flow rate of raw materials handled by the control target 900, may be used as the sensor, and the real environment characteristic information generation unit 191 may acquire sensor measurement values. The real environment characteristic information generation unit 191 may acquire data from multiple sensors.

[0038] Furthermore, as the real environment feature information, observation data of the real environment may be used as is, or data obtained by processing the observation data may be used. For example, when a captured image of the real environment is obtained as the observation data, the real environment feature information generation unit 191 may perform image recognition processing on the captured image of the real environment to extract objects appearing in the image. Then, the real environment feature information generation unit 191 may generate, as the real environment feature information, information indicating the attributes or states of each extracted object, such as the position, size, color, material, or some of these, of each extracted object. Alternatively, the real environment feature information generation unit 191 may use the captured image of the real environment as is as the real environment feature information. Objects appearing in the captured image of the real environment are examples of objects located in the real environment.

[0039] Furthermore, the real environment characteristic information is not limited to a specific format. For example, when information indicating the attributes or status of each object located in the real environment is used as the real environment characteristic information, the real environment characteristic information may be in the form of a graph (directed graph or undirected graph). A graph indicating real environment characteristic information is also called an environment graph. Alternatively, the real environment characteristic information may be in the form of tabular data indicating the attributes or status of each object. Alternatively, the real environment information may be in the form of vector data that lists the attributes or status of each object in order.

[0040] Fig. 3 is a diagram showing an example of a captured image of a real environment. In the example of Fig. 3, the captured image of the real environment shows a tray, a rectangular box placed in the tray, and a cylindrical can placed on the box. The tray, the box, and the can are examples of objects located in the real environment.

[0041] Fig. 4 is a diagram showing a first example of an environment graph. Fig. 4 shows an example of an environment graph generated by the real environment characteristic information generation unit 191 from the captured image of the real environment shown in Fig. 3. In the example of Fig. 4, the environment graph is configured to include three nodes, "Tray 1," "Rectangular Parallelepiped 1," and "Cylinder 1," and edges that indicate the relationships between these nodes.

[0042] In the example of Figure 4, the nodes of the environment graph represent each object that appears in the captured image of the real environment. The node "Tray 1" represents a tray. The node "Rectangular Parallelepiped 1" represents a rectangular box. The node "Cylinder 1" represents a cylindrical can. The shapes of these objects that appear in the captured image of the real environment are used as identification information for the nodes of the environment graph. In the example of Figure 4, the shape of the tray is referred to as "Tray." Furthermore, the "1" in "Tray 1" and so on is a serial number for each object shape that is used to identify each node when there are multiple objects of the same shape.

[0043] In addition, in the example of Figure 4, each node is linked with information indicating the attributes or state of the object represented by the node, including the type of each object ("tray," "box," "can"), the position of each object ("x = . . .," "y = . . .," "z = . . ."), and the color of each object ("white," "blue," "red")

[0044] In the example of Fig. 4, each edge is associated with a relationship indicated by that edge. The relationship "in" indicates a relationship in which the object indicated by the node on the starting side of the directed edge is located inside the object indicated by the node on the terminal side. The relationship "on" indicates a relationship in which the object indicated by the node on the starting side of the directed edge is located on top of the object indicated by the node on the terminal side. As in the example of Fig. 4, the environment feature information may be represented by an environment graph including nodes indicating objects in the environment and edges indicating relationships between the nodes.

[0045] Fig. 5 is a diagram showing a second example of an environment graph. In the example of Fig. 5, in addition to each object appearing in a captured image of the real environment, the attributes or states of each object are also shown as nodes. A node indicating an object and a node indicating the type of that object are connected by an edge "type". A node indicating an object and a node indicating the position of that object are connected by an edge "position". A node indicating an object and a node indicating the color of that object are connected by an edge "color".

[0046] 5, for ease of viewing, nodes and edges are shown to indicate the attributes or states of the tray indicated by the node "Tray 1" and the can indicated by the node "Cylinder 1", and detailed information about the nodes and edges is omitted. As in the example of FIG. 5, the environmental feature information may be represented by an environmental graph that includes nodes other than the nodes that indicate the objects in the environment.

[0047] 6 is a diagram showing a third example of an environment graph. The example in FIG. 6 shows environment graphs for the initial state and the target state of the process execution environment. The environment graph showing the initial state shows a state in which there are two trays, "Tray 1" and "Tray 2," and two objects, "Rectangular Parallelepiped 1" and "Cylinder 1," are located in "Tray 1." The environment graph showing the target state shows a state in which two objects, "Rectangular Parallelepiped 1" and "Cylinder 1," are located in "Tray 2" of "Tray 1" and "Tray 2," and "Cylinder 1" is located above "Rectangular Parallelepiped 1."

[0048] 6 shows a process of moving the "rectangular parallelepiped 1" and the "cylinder 1" so as to change from the initial state to the goal state by combining the environment graph of the initial state and the environment graph of the goal state. As in the example of FIG. 6, the environment characteristic information may include the environment graph of the initial state and the environment graph of the goal state.

[0049] In this way, the combination of the initial state environment graph and the target state environment graph indicates information about the process to be executed, in addition to information about the execution environment of the process. By selecting environment-related constraint data using environmental characteristic information including the initial state environment graph and the target state environment graph, the control device 100 is expected to be able to acquire constraints that are appropriate not only for the actual environment but also for the process to be executed.

[0050] 7 is a diagram showing a fourth example of an environment graph. FIG. 7 shows an example of an environment graph in which the environment graph for the initial state and the environment graph for the target state in the example of FIG. 6 are combined into a single graph. The nodes "Tray 1 Initial," "Tray 2 Initial," "Rectangular Parallelepiped 1 Initial," and "Cylinder 1 Initial" represent objects in the initial state. The nodes "Tray 1 Target," "Tray 2 Target," "Rectangular Parallelepiped 1 Target," and "Cylinder 1 Target" represent objects in the target state. The "time" edge indicates the passage of time from the initial state to the target state.

[0051] As in the example of Fig. 7, the environmental characteristic information may include an environmental graph that combines an environmental graph of the initial state of the processing environment and an environmental graph of the target state into a single graph. By having the control device 100 select environment-compatible constraint condition data using environmental characteristic information that includes an environmental graph that combines an environmental graph of the initial state of the processing environment and an environmental graph of the target state into a single graph, it is expected that constraint conditions that are appropriate not only for the actual environment but also for the processing to be executed can be obtained, as in the example of Fig. 6.

[0052] The similarity calculation unit 192 calculates the similarity of the environmental characteristic information. As the similarity calculated by the similarity calculation unit 192, various indices according to the environmental characteristic information can be used.

[0053] For example, when the environmental feature information is represented as a graph (directed graph or undirected graph), the similarity calculation unit 192 may use the similarity of the nodes or subgraphs representing the objects, and the similarity of the edges between the nodes representing the objects, as the similarity of the environmental feature information.

[0054] When calculating the similarity between two nodes or subgraphs representing objects, for example, the attributes or states of the two objects can be represented by vectors, and an index representing the similarity of the vectors, such as cosine similarity or norm, can be used. When cosine similarity is used as an index representing the similarity of the vectors, a larger index value indicates that the two vectors are more similar. On the other hand, when norm is used as an index representing the similarity of the vectors, a smaller index value indicates that the two vectors are more similar.

[0055] When representing the attributes or states of two objects as vectors, the similarity calculation unit 192 arranges the items in the same order in the two vectors, for example, arranging the coordinate values ​​indicating the positions in the order of x-coordinate value, y-coordinate value, and z-coordinate value. Furthermore, the similarity calculation unit 192 may exclude items for which data is available for only one of the two objects from the elements of the vector, thereby excluding the items from the targets for similarity determination.

[0056] Furthermore, for items that are expressed other than numerically, such as the shape of an object, the similarity calculation unit 192 may digitize the similarity separately from the vector similarity by setting the similarity to a predetermined value if the values ​​of that item are the same for two objects, or to 0 if they are different, and add the digitized similarity to the vector similarity. Note that in cases where a norm is used as an index representing vector similarity, the smaller the index value, the more similar the vectors are, and so on, the similarity may be set to a sufficiently large predetermined value when the values ​​of the same item are different for two objects, or may be set to a value that is larger than when the values ​​of the same item are the same for two objects.

[0057] In calculating the similarity between nodes representing objects in two graphs, the similarity calculation unit 192 may, for example, associate nodes representing objects between the two graphs so that the average value of the similarities between the associated nodes across the entire graph is maximized.The similarity calculation unit 192 may then use the maximum average value of the similarities between the associated nodes across the entire graph as the similarity between the nodes in the two graphs.

[0058] In calculating the edge similarity, for example, if both nodes at both ends of an edge selected from each of the two graphs are associated by the above-mentioned node association, the selected edges may be defined as the same edge.The similarity calculation unit 192 may then count the number of identical edges in the entire graph.In order to avoid the edge similarity becoming higher as the scale of the graph increases, the similarity calculation unit 192 may use the value obtained by dividing the number of identical edges in the entire graph by the number of nodes representing objects as the edge similarity in the two graphs.

[0059] Edges may represent not only the connection relationships between nodes but also various values ​​related to the nodes connected by the edges, such as the distance between nodes. Edges may be represented by real numbers or vectors. When edges are represented by vectors, an index representing the similarity of vectors, such as cosine similarity or norm, can be used as an index representing the similarity of edges.

[0060] For edges, with respect to items that are expressed other than numerically, the similarity calculation unit 192 may quantify the similarity separately from the vector similarity by setting the similarity to a predetermined value if the values ​​of that item are the same for two edges, or to 0 if they are different, and add the quantified similarity to the vector similarity. Note that in a case where a norm is used as an index representing vector similarity, the smaller the index value, the more similar the vectors are, and so on, the similarity may be set to a sufficiently large predetermined value when the values ​​of the same item are different for two objects, or may be set to a value larger than when the values ​​of the same item are the same for two objects.

[0061] The similarity calculation unit 192 may then use the value obtained by weighting the similarity of the nodes in the two graphs and the similarity of the edges in the two graphs with a predetermined weighting coefficient as the similarity of the two graphs.

[0062] Alternatively, the similarity calculation unit 192 may be configured to include a neural network such as a graph neural network (GNN) or a graph convolutional network (GCN), and may be configured to learn in advance how to calculate the feature quantities of the graphs. The similarity calculation unit 192 may then calculate the feature quantities of the entire graphs using the neural network, and compare (cluster) the similarities between two graphs based on the calculated feature quantities. Alternatively, the similarity calculation unit 192 may calculate the similarity between two graphs using a kernel function by a kernel method such as a graph kernel.

[0063] When the environmental feature information is expressed as a real vector (a vector whose elements are real values), a known index of similarity between real vectors, such as cosine similarity or norm, may be used as the similarity of the environmental feature information.When the environmental feature information is expressed as an image, a known method for calculating the similarity of images may be used as the similarity of the environmental feature information.

[0064] The constraint condition selection unit 193 selects environmentally-adapted constraint condition data corresponding to the real environment from the environmentally-adapted constraint condition data stored in the environmentally-adapted constraint condition data storage unit 181. Specifically, the constraint condition selection unit 193 selects environmentally-adapted constraint condition data corresponding to the real environment based on the similarity between the real environment characteristic information and the environmental characteristic information included in the environmentally-adapted constraint condition data calculated by the similarity calculation unit 192. The constraint condition selection unit 193 corresponds to an example of a constraint condition selection means.

[0065] The similarity calculation unit 192 may calculate the similarity between the environmental characteristic information included in each piece of environmentally-adapted constraint condition data stored in the environmentally-adapted constraint condition data storage unit 181 and the actual environment characteristic information. Then, the constraint condition selection unit 193 may select, from the environmentally-adapted constraint condition data stored in the environmentally-adapted constraint condition data storage unit 181, the environmentally-adapted constraint condition data having the highest similarity calculated by the similarity calculation unit 192.

[0066] The control execution unit 194 controls the control target 900. Specifically, the control execution unit 194 controls the control target 900 to execute the target process based on the constraint conditions indicated by the environment-associated constraint condition data selected by the constraint condition selection unit 193. The control execution unit 194 generates a control command for the control target 900 and transmits the generated control command to the control target 900 via the communication unit 110, thereby controlling the control target 900. The control execution unit 194 corresponds to an example of a control execution means.

[0067] For example, if the process to be executed is for the control target 900 to reach a predetermined target point, an objective function indicating whether the process to be executed is achieved may be set in advance, such as a function indicating the distance between the control target 900 and the target point.The control execution unit 194 may then search for a time series of control commands for the control target 900 that satisfies the constraint conditions and results in a value for the objective function indicating that the process to be executed has been achieved.The control execution unit 194 may then send control commands to the control target 900 according to the obtained time series of control commands, thereby causing the control target 900 to execute the process to be executed.

[0068] Alternatively, the constraint condition selection unit 193 may select constraint conditions including constraint conditions that indicate a target state of the process to be executed. For example, the constraint conditions may be expressed in temporal logic, and the constraint conditions selected by the constraint condition selection unit 193 may include a constraint condition that the control object 900 is located at a target point when a predetermined time has elapsed. Then, the control execution unit 194 may search for a time series of control commands for the control object 900 that satisfies the constraint conditions. Then, the control execution unit 194 may transmit control commands to the control object 900 according to the obtained time series of control commands, thereby causing the control object 900 to perform the process to be executed.

[0069] Fig. 8 is a diagram showing an example of the flow of data when the control device 100 controls the control target 900. In the example of Fig. 8, the real environment characteristic information generation unit 191 generates real environment characteristic information based on observation data of the real environment obtained by a sensor that observes the real environment.

[0070] The similarity calculation unit 192 calculates the similarity between the environmental characteristic information included in each piece of environmentally-adapted constraint condition data stored in the environmentally-adapted constraint condition data storage unit 181 and the actual environment characteristic information. The constraint condition selection unit 193 selects, from the environmentally-adapted constraint condition data stored in the environmentally-adapted constraint condition data storage unit 181, the environmentally-adapted constraint condition data with the highest similarity calculated by the similarity calculation unit 192.

[0071] The control execution unit 194 generates a control command for the control target 900 based on the constraint conditions indicated by the environment-associated constraint condition data selected by the constraint condition selection unit 193. The control execution unit 194 controls the control target 900 by transmitting the generated control command to the control target 900 via the communication unit 110.

[0072] 9 is a diagram showing an example of a procedure of a process in which the control device 100 controls the control target 900. In the process of FIG. 9, the real environment characteristic information generating unit 191 acquires observation data of the real environment (step S101).

[0073] The real-world characteristic information generating unit 191 generates real-world characteristic information based on the acquired observation data (step S102). Next, the similarity calculating unit 192 calculates the similarity between the real-world characteristic information and the environmental characteristic information included in each piece of environmental-related constraint condition data stored in the environmental-related constraint condition data storage unit 181 (step S103).

[0074] Next, the constraint condition selection unit 193 selects the environmentally-adapted constraint condition data having the highest similarity calculated by the similarity calculation unit 192 from among the environmentally-adapted constraint condition data stored in the environmentally-adapted constraint condition data storage unit 181 (step S104). Then, the control execution unit 194 controls the control target 900 based on the constraint conditions indicated by the environmentally-adapted constraint condition data selected by the constraint condition selection unit 193 (step S105). Specifically, the control execution unit 194 generates a control command for the control target 900 based on the constraint conditions and transmits the generated control command to the control target 900 via the communication unit 110. After step S105, the control device 100 ends the processing of FIG. 9.

[0075] As described above, the constraint condition selection unit 193 selects environmentally-compatible constraint condition data corresponding to the actual environment, which is the execution environment of the process to be executed, from among the environmentally-compatible constraint condition data in which environmental characteristic information, which is information about the execution environment of the process, and constraint conditions for executing the process in that environment are linked. The control execution unit 194 controls the control target 900 to execute the process to be executed, based on the constraint conditions indicated by the selected environmentally-compatible constraint condition data.

[0076] By acquiring constraint conditions according to the real environment, the control device 100 is expected to be able to deal with not only differences in the execution environment of the process but also differences in the process to be executed. For example, a constraint condition that the control target 900 must avoid certain obstacles when moving can be applied to various processes that involve movement of the control target 900.

[0077] The real environment characteristic information generating unit 191 generates real environment characteristic information, which is environmental characteristic information for the real environment, based on observation data of the real environment obtained by a sensor that observes the real environment. The similarity calculating unit 192 calculates the similarity of the environmental characteristic information. The constraint condition selecting unit 193 selects environmentally-responsive constraint condition data corresponding to the real environment based on the similarity between the real environment characteristic information and environmental characteristic information included in the environmentally-responsive constraint condition data.

[0078] The control device 100 is expected to be able to acquire constraint conditions suited to the actual environment by selecting environmentally-adapted constraint condition data based on the similarity of the environmental feature information. In this respect, the control device 100 is expected to be able to control the controlled object 900 with high accuracy.

[0079] The environment feature information is expressed as a graph including nodes that represent objects in the processing execution environment and edges that represent relationships between the nodes. Since the environment feature information indicates relationships between objects, the control device 100 is expected to be able to determine the similarity of the processing execution environments with high accuracy and to acquire constraints that are suitable for the real environment.

[0080] Second Embodiment In the second to fourth embodiments, an example will be described in which the control device generates environmentally-adapted constraint condition data when it determines that there is no environmentally-adapted constraint condition data suitable for the actual environment.

[0081] In the second embodiment, an example is described in which the control device generates environmentally adaptive constraint condition data in response to a user operation to input constraint conditions according to the actual environment. In the third embodiment, an example is described in which the control device generates environmentally adaptive constraint condition data in response to a user operation to operate the control target 900. In the fourth embodiment, an example is described in which the control device generates environmentally adaptive constraint condition data based on a simulation of the operation of the control target 900.

[0082] Fig. 10 is a diagram showing an example of the configuration of a control device according to the second embodiment. In the configuration shown in Fig. 10, a control device 200 includes a communication unit 110, a display unit 120, an operation input unit 130, a storage unit 180, and a processing unit 290. The processing unit 290 includes a real environment characteristic information generation unit 191, a similarity calculation unit 192, a constraint condition selection unit 193, a control execution unit 194, a support request unit 291, and an environment-associated constraint condition data generation unit 292. In the second embodiment, the control device 200 is used instead of the control device 100 of Fig. 1.

[0083] 10, parts having the same functions as those in FIG. 1 are designated by the same reference numerals (110, 120, 130, 180, 181, 191, 192, 193, 194), and detailed description thereof will be omitted here. The control device 200 differs from the control device 100 in that the processing unit 290 further includes a support request unit 291 and an environment-associated constraint condition data generation unit 292 in addition to the units included in the processing unit 190. In other respects, the control device 200 is similar to the control device 100.

[0084] The support request unit 291 requests a user operation when there is no environmentally-adapted constraint condition data including environmental feature information whose similarity to the real environment feature information is equal to or greater than a predetermined threshold value among the environmentally-adapted constraint condition data stored in the environmentally-adapted constraint condition data storage unit 181. The threshold value here may be set in advance by the user and stored in the storage unit 180, for example. The support request unit 291 corresponds to an example of a support request means.

[0085] For example, the support requesting unit 291 may control the display unit 120 to display a message that a control plan for the control target 900 cannot be formulated and that support from the user is requested. Alternatively, in the second embodiment, the support requesting unit 291 may control the display unit 120 to display a message that requests input of constraint conditions for the operation of the control target 900.

[0086] If the environmentally-adapted constraint condition data stored in the environmentally-adapted constraint condition data storage unit 181 does not contain environmentally-adapted constraint condition data including environmental feature information whose similarity to the real environment feature information is equal to or greater than a predetermined threshold, the environmentally-adapted constraint condition data generation unit 292 generates environmentally-adapted constraint condition data in which the real environment feature information is linked to constraint conditions in the real environment. The environmentally-adapted constraint condition data generation unit 292 corresponds to an example of an environmentally-adapted constraint condition data generation means.

[0087] In the second embodiment, the environment-responsive constraint data generation unit 292 generates environment-responsive constraint data by linking the real environment characteristic information generated by the real environment characteristic information generation unit 191 with constraint conditions input by the user. The user may input constraint conditions including constraint conditions indicating a target state of the process to be executed. In this case, the environment-responsive constraint data generation unit 292 generates environment-responsive constraint data in which the real environment characteristic information is linked with constraint conditions including constraint conditions indicating a target state of the process to be executed.

[0088] 11 is a diagram showing an example of the data flow when the control device 200 newly generates environmental constraint condition data. In the example of Fig. 11, the data flow in the real environment characteristic information generation unit 191 and the similarity calculation unit 192 is the same as in Fig. 8.

[0089] Specifically, the real-environment characteristic information generating unit 191 generates real-environment characteristic information based on observation data of the real environment obtained by a sensor that observes the real environment. The similarity calculating unit 192 calculates the similarity between the real-environment characteristic information and the environment characteristic information included in each piece of environment-related constraint condition data stored in the environment-related constraint condition data storage unit 181.

[0090] The constraint condition selection unit 193 determines whether there is environment-related constraint condition data that indicates that the similarity calculated by the similarity calculation unit 192 is similar to or greater than a predetermined threshold. When the constraint condition selection unit 193 determines that there is corresponding environment-related constraint condition data, the data flow in the constraint condition selection unit 193 and the control execution unit 194 is the same as that described with reference to Figure 8, and therefore will not be illustrated or described here.

[0091] On the other hand, if it is determined that there is no environmentally-compatible constraint condition data that indicates that the similarity calculated by the similarity calculation unit 192 is similar to or greater than a predetermined threshold, the constraint condition selection unit 193 outputs information to the support request unit 291 indicating that there is no corresponding environmentally-compatible constraint condition data.

[0092] When the support request unit 291 receives input of information indicating that there is no relevant environmental constraint data, it requests support from the user. For example, as described above, the support request unit 291 may control the display unit 120 to display a message indicating that a control plan for the control target 900 cannot be formulated and that support from the user is requested.

[0093] In the second embodiment, a user who has received a request for assistance inputs constraint conditions to the control device 200. The control execution unit 194 generates a control command for the control target 900 so as to satisfy the constraint conditions input by the user, and transmits the generated control command to the control target 900 via the communication unit 110.

[0094] Furthermore, the environmentally-adapted constraint data generation unit 292 generates environmentally-adapted constraint data by linking the real environment characteristic information generated by the real environment characteristic information generation unit 191 with the constraints input by the user. The environmentally-adapted constraint data generation unit 292 stores the generated environmentally-adapted constraint data in the environmentally-adapted constraint data storage unit 181. As a result, the environmentally-adapted constraint data generated by the environmentally-adapted constraint data generation unit 292 is added to the environmentally-adapted constraint data stored in the environmentally-adapted constraint data storage unit 181.

[0095] 12 is a diagram showing an example of a procedure of a process in which the control device 200 controls the control target 900. Steps S201 to S203 in FIG. 12 are the same as steps S101 to S103 in FIG.

[0096] After step S203, the constraint condition selection unit 193 determines whether there is environment-related constraint condition data indicating that the similarity calculated by the similarity calculation unit 192 is similar to or greater than a predetermined threshold (step S204). If the constraint condition selection unit 193 determines that there is corresponding environment-related constraint condition data (step S204: YES), the process proceeds to step S211. Steps S211 to S212 are the same as steps S104 to S105 in FIG. 9. After step S212, the control device 200 ends the process of FIG. 12.

[0097] On the other hand, if the constraint condition selection unit 193 determines in step S204 that there is no corresponding environment-associated constraint condition data (step S204: NO), the support request unit 291 requests support from the user (step S221). Then, the control device 100 acquires the constraint conditions input by the user (step S222). For example, the operation input unit 130 accepts a user operation to input the constraint conditions.

[0098] Next, the control execution unit 194 controls the control target 900 so as to satisfy the constraint conditions input by the user (step S223). Furthermore, the environment-related constraint condition data generation unit 292 associates the real environment characteristic information generated by the real environment characteristic information generation unit 191 with the constraint conditions input by the user to generate environment-related constraint condition data, and stores the generated environment-related constraint condition data in the environment-related constraint condition data storage unit 181 (step S224). After step S224, the control device 200 ends the processing of FIG. 12.

[0099] As described above, when there is no environment-adapted constraint condition data including environment feature information whose similarity to the real environment feature information is equal to or greater than a predetermined threshold, the environment-adapted constraint condition data generator 292 generates environment-adapted constraint condition data linking the real environment feature information with the constraint conditions in the real environment. It is expected that the control device 200 will be able to add environment-adapted constraint condition data suitable for the real environment when there is no environment-adapted constraint condition data suitable for the real environment.

[0100] Furthermore, if there is no environment-adapted constraint condition data including environment characteristic information whose similarity to the real environment characteristic information is equal to or greater than a predetermined threshold, the support request unit 291 requests a user operation. The environment-adapted constraint condition data generation unit 292 generates environment-adapted constraint condition data in which the real environment characteristic information is linked to constraint conditions in the real environment, based on the user operation. The control device 200 is expected to be able to add environment-adapted constraint condition data suited to the real environment, in that it generates environment-adapted constraint condition data based on the user operation.

[0101] Furthermore, the environment-adapted constraint data generator 292 generates environment-adapted constraint data that links the real environment characteristic information with the constraints input by the user. The control device 200 is expected to be able to add environment-adapted constraint data that is suited to the real environment, since it generates environment-adapted constraint data using the constraints input by the user.

[0102] Third Embodiment Fig. 13 is a diagram showing an example of the configuration of a control device according to a third embodiment. In the configuration shown in Fig. 13, a control device 300 includes a communication unit 110, a display unit 120, an operation input unit 130, a storage unit 180, and a processing unit 390. The processing unit 390 includes a real-environment characteristic information generation unit 191, a similarity calculation unit 192, a constraint condition selection unit 193, a control execution unit 194, a support request unit 291, an environment-associated constraint condition data generation unit 292, and a constraint condition learning unit 391. In the third embodiment, the control device 300 is used instead of the control device 100 of Fig. 1.

[0103] 13, parts having the same functions as those in FIG. 10 are given the same reference numerals (110, 120, 130, 180, 181, 191, 192, 193, 194, 291, 292), and detailed description thereof will be omitted here. The control device 300 differs from the control device 200 in that the processing unit 390 further includes a constraint condition learning unit 391 in addition to the units included in the processing unit 290. In other respects, the control device 300 is similar to the control device 200.

[0104] The constraint condition learning unit 391 learns constraint conditions based on a time series of operations of the control object 900. In the third embodiment, the constraint condition learning unit 391 learns constraint conditions based on a time series of operations of the control object 900 in response to user operations. In the third embodiment, it can be said that the constraint condition learning unit 391 learns control conditions based on user operations that operate the control object 900.

[0105] For example, the storage unit 180 stores constraint conditions including parameters in advance. The constraint conditions including parameters are also referred to as a constraint condition template. The constraint condition learning unit 391 then determines the parameter values ​​of the constraint condition template based on the time series of the operation of the control target 900.

[0106] Furthermore, for example, the storage unit 180 stores a plurality of constraint condition templates. Furthermore, the control device 300 acquires a plurality of time series of operations of the control target 900. The constraint condition learning unit 391 then adopts a constraint condition template that has parameter values ​​that will satisfy the constraint conditions for all time series when the target process is successful. On the other hand, the constraint condition learning unit 391 rejects a constraint condition template that does not have parameter values ​​that will satisfy the constraint conditions for all time series when the target process is successful. Here, a success in the target process refers to a case where the target process is achieved. A case where the target process cannot be achieved is also referred to as a failure in the target process.

[0107] Alternatively, the constraint condition learning unit 391 may determine whether to adopt a constraint condition template based on whether a parameter value that satisfies the constraint condition exists at a rate higher than a predetermined rate for the time series of operations of the control target 900 when the target process is successful. For example, the constraint condition learning unit 391 may adopt a constraint condition template in which a parameter value that satisfies the constraint condition exists for a predetermined rate or more of the time series when the target process is successful. On the other hand, the constraint condition learning unit 391 may reject a constraint condition template that does not satisfy the condition that a parameter value that satisfies the constraint condition exists for a predetermined rate or more of the time series when the target process is successful.

[0108] Here, consider a case where the control device 300 can obtain both a time series of the operation of the control target 900 when the process to be executed is successful and a time series of the operation of the control target 900 when the process to be executed fails. In the third embodiment, a case where the user performs both an operation that will result in the process to be executed succeeding and an operation that will result in the process to be executed failing corresponds to a case where the control device 300 can obtain both a time series of the operation of the control target 900 when the process to be executed is successful and a time series of the operation of the control target 900 when the process to be executed is failed.

[0109] In this case, the constraint condition learning unit 391 searches for and sets combinations of parameter values ​​for all parameter combinations of the adopted constraint condition template such that the constraint conditions are met if the target process is successful, and the constraint conditions are not met if the target process fails.

[0110] Here, "the constraint conditions are satisfied when the target process is successful" means that when the target process is successful, the constraint conditions are satisfied for all constraint condition templates for the time series of operations of the control target 900. Also, "the constraint conditions are not satisfied when the target process fails" means that when the target process fails, the constraint conditions are not satisfied for at least one constraint condition template for the time series of operations of the control target 900.

[0111] If a combination of parameter values ​​cannot be obtained such that the constraint conditions are met if the target process is successful and the constraint conditions are not met if the target process fails, the constraint condition learning unit 391 may change one or more of the adopted constraint condition templates to unadopted.

[0112] Here, too, the condition that "the constraint is satisfied if the target process is successful" may be relaxed to "if the target process is successful, the constraint is satisfied at a rate higher than a predetermined rate." For example, the constraint learning unit 391 may search for and set a combination of parameter values ​​for all parameter combinations of the adopted constraint template such that the constraint is satisfied for at least a predetermined rate of time series when the target process is successful and the constraint is not satisfied for all time series when the target process is unsuccessful. Furthermore, if such a combination of parameter values ​​cannot be obtained, the constraint learning unit 391 may change one or more of the adopted constraint templates to unadopted.

[0113] Next, consider a case where the control device 300 can obtain a time series of the operation of the control target 900 when the target process is successful, but cannot obtain a time series of the operation of the control target 900 when the target process is unsuccessful. In this case, the constraint condition learning unit 391 searches for and sets combinations of parameter values ​​for all parameter combinations of the adopted constraint condition templates such that the constraint conditions are satisfied for all constraint condition templates with respect to the time series of the operation of the control target 900 when the target process is successful.

[0114] The constraint condition learning unit 391 may set a combination of parameter values ​​so that the constraint conditions are as strict as possible. For example, with respect to a parameter indicating the radius of an area that the control target 900 must not enter, the constraint condition learning unit 391 may set, as the parameter value, the largest radius among the radii that satisfy the constraint conditions for the time series of the operation of the control target 900 when the process to be executed is successful.

[0115] Furthermore, if a combination of parameter values ​​cannot be obtained that satisfies the constraint conditions for all constraint condition templates for the time series of the operation of the control object 900 when the target process is successful, the constraint condition learning unit 391 may change one or more of the adopted constraint condition templates to unadopted.

[0116] In this case, too, the condition of all time series when the target process is successful may be relaxed to the condition of a predetermined percentage or more of the time series when the target process is successful. For example, the constraint condition learning unit 391 may search for and set combinations of parameter values ​​such that the constraint conditions for all constraint condition templates are satisfied for a predetermined percentage or more of the time series of the operation of the control target 900 when the target process is successful, for all parameter combinations of the adopted constraint condition templates.

[0117] In this case, setting a combination of parameter values ​​so as to make the constraint conditions as strict as possible can be defined as setting a combination of parameter values ​​so that the constraint conditions are satisfied for time series selected as time series that account for a predetermined percentage or more, and so that the constraint conditions are as strict as possible. For example, with respect to a parameter indicating the radius of an area that the control target 900 must not enter, the constraint condition learning unit 391 may set, as the parameter value, the largest radius among the radii so that the constraint conditions are satisfied for time series selected as time series that account for a predetermined percentage or more.

[0118] However, the method by which the constraint condition learning unit 391 learns the constraint conditions is not limited to a specific method, and various methods can be used to obtain constraint conditions that are consistent with the time series of the operation of the control object 900 and the success or failure of the processing to be executed in that time series.

[0119] Fig. 14 is a diagram showing an example of the data flow when the control device 300 newly generates environment-associated constraint condition data. In the example of Fig. 14, the data flows in the real-environment characteristic information generation unit 191, the similarity calculation unit 192, and the constraint condition selection unit 193 are the same as those in Fig. 11.

[0120] Specifically, the real-environment characteristic information generating unit 191 generates real-environment characteristic information based on observation data of the real environment obtained by a sensor that observes the real environment. The similarity calculating unit 192 calculates the similarity between the real-environment characteristic information and the environment characteristic information included in each piece of environment-related constraint condition data stored in the environment-related constraint condition data storage unit 181.

[0121] The constraint condition selection unit 193 determines whether there is environment-related constraint condition data that indicates that the similarity calculated by the similarity calculation unit 192 is similar to or greater than a predetermined threshold. When the constraint condition selection unit 193 determines that there is corresponding environment-related constraint condition data, the data flow in the constraint condition selection unit 193 and the control execution unit 194 is the same as that described with reference to Figure 8, and therefore will not be illustrated or described here.

[0122] On the other hand, if it is determined that there is no environmentally-compatible constraint condition data that indicates that the similarity calculated by the similarity calculation unit 192 is similar to or greater than a predetermined threshold, the constraint condition selection unit 193 outputs information to the support request unit 291 indicating that there is no corresponding environmentally-compatible constraint condition data.

[0123] 11 , when the support request unit 291 receives input of information indicating that there is no relevant environmental constraint data, it requests support from the user. For example, the support request unit 291 may control the display unit 120 to display a message indicating that a control plan for the control target 900 cannot be formulated and that support from the user is requested.

[0124] In the third embodiment, a user who has received a request for assistance performs an operation on the control target 900. For example, the assistance request unit 291 may control the display unit 120 to display a request to perform an operation on the control target 900. Furthermore, the operation input unit 130 may accept a user operation on the control target 900. For example, the user may input a control command for the control target 900 using a keyboard provided in the operation input unit 130. Alternatively, the operation input unit 130 may be provided with an operating device for operating the control target 900, such as a joystick, and may accept a user operation.

[0125] The control execution unit 194 generates a control command for the control target 900 in accordance with a user operation on the control target 900, and transmits the command to the control target 900 via the communication unit 110. This causes the control execution unit 194 to operate in accordance with the user operation.

[0126] Furthermore, the constraint condition learning unit 391 acquires observation data of the real environment when the user is operating the control object 900. The observation data of the real environment includes data indicating the operation of the control object 900, such as the position of the control object 900. The constraint condition learning unit 391 learns constraint conditions in the real environment based on the observation data of the real environment, and outputs the obtained constraint conditions to the environment-associated constraint condition data generation unit 292.

[0127] The environment-adapted constraint condition data generation unit 292 generates environment-adapted constraint condition data by linking the real environment characteristic information generated by the real environment characteristic information generation unit 191 with the constraint conditions obtained through learning by the constraint condition learning unit 391. The environment-adapted constraint condition data generation unit 292 stores the generated environment-adapted constraint condition data in the environment-adapted constraint condition data storage unit 181. As a result, the environment-adapted constraint condition data generated by the environment-adapted constraint condition data generation unit 292 is added to the environment-adapted constraint condition data stored in the environment-adapted constraint condition data storage unit 181.

[0128] 15 is a diagram showing an example of a procedure of a process in which the control device 300 controls the control target 900. Steps S301 to S304 in FIG. 15 are the same as steps S201 to S204 in FIG.

[0129] In step S304, if the constraint condition selection unit 193 determines that there is corresponding environment-related constraint condition data (step S304: YES), the process proceeds to step S311. Steps S311 to S312 are the same as steps S211 to S212 in FIG.

[0130] On the other hand, if the constraint condition selection unit 193 determines in step S304 that there is no corresponding environment-associated constraint condition data (step S304: NO), the support request unit 291 requests support from the user (step S321). Then, the control device 100 controls the control target 900 based on the operation on the control target 900 performed by the user (step S322).

[0131] Next, the constraint condition learning unit 391 learns constraint conditions based on the time series of the operation of the control object 900 when the user is operating the control object 900 (step S323). The environment-responsive constraint condition data generating unit 292 associates the real environment characteristic information generated by the real environment characteristic information generating unit 191 with the constraint conditions obtained by learning by the constraint condition learning unit 391 to generate environment-responsive constraint condition data, and stores the generated environment-responsive constraint condition data in the environment-responsive constraint condition data storage unit 181 (step S324). After step S324, the control device 300 ends the processing of FIG. 15.

[0132] As described above, the constraint condition learning unit 391 learns constraint conditions based on user operations to operate the control target 900. The environment-adaptive constraint condition data generating unit 292 generates environment-adaptive constraint condition data in which the real-environment feature data and the constraint conditions obtained through learning are linked. Since the control device 300 learns constraint conditions based on user operations to operate the control target 900, it is expected that it will be able to learn constraint conditions that will result in successful execution of the target process.

[0133] <Fourth embodiment> Fig. 16 is a diagram showing an example of the configuration of a control device according to a fourth embodiment. In the configuration shown in Fig. 16, a control device 400 includes a communication unit 110, a display unit 120, an operation input unit 130, a storage unit 180, and a processing unit 490. The processing unit 490 includes a real environment characteristic information generation unit 191, a similarity calculation unit 192, a constraint condition selection unit 193, a control execution unit 194, an environment-associated constraint condition data generation unit 292, a constraint condition learning unit 391, and a simulation processing unit 491. In the fourth embodiment, the control device 400 is used instead of the control device 100 of Fig. 1.

[0134] 16, parts having the same functions as those in FIG. 13 are designated by the same reference numerals (110, 120, 130, 180, 181, 191, 192, 193, 194, 292, 391), and detailed description thereof will be omitted here. The control device 400 differs from the control device 300 in that the processing unit 490 does not include the support request unit 291 of the parts included in the processing unit 390, but instead includes a simulation processing unit 491. In other respects, the control device 400 is similar to the control device 300.

[0135] In the fourth embodiment, the constraint condition learning unit 391 learns constraint conditions based on the results of a simulation of the process to be executed in an environment that simulates the real environment, instead of the user's operation on the control target 900 .

[0136] The simulation processing unit 491 simulates the operation of the control target 900. In particular, the simulation processing unit 491 receives settings for an environment that simulates a real environment and performs a simulation of the control target 900 performing operations to perform processing to be executed in the set environment. The simulation in this case corresponds to a simulation of the processing to be executed in an environment that simulates a real environment. The control device 400 corresponds to an example of a data generation device.

[0137] Fig. 17 is a diagram showing an example of the data flow when the control device 400 newly generates environment-associated constraint condition data. In the example of Fig. 17, the data flow in the real-environment characteristic information generation unit 191, the similarity calculation unit 192, and the constraint condition selection unit 193 is the same as in Fig. 11.

[0138] Specifically, the real-environment characteristic information generating unit 191 generates real-environment characteristic information based on observation data of the real environment obtained by a sensor that observes the real environment. The similarity calculating unit 192 calculates the similarity between the real-environment characteristic information and the environment characteristic information included in each piece of environment-related constraint condition data stored in the environment-related constraint condition data storage unit 181.

[0139] The constraint condition selection unit 193 determines whether there is environment-related constraint condition data that indicates that the similarity calculated by the similarity calculation unit 192 is similar to or greater than a predetermined threshold. When the constraint condition selection unit 193 determines that there is corresponding environment-related constraint condition data, the data flow in the constraint condition selection unit 193 and the control execution unit 194 is the same as that described with reference to Figure 8, and therefore will not be illustrated or described here.

[0140] On the other hand, if it is determined that there is no environmentally-compatible constraint condition data that indicates that the similarity calculated by the similarity calculation unit 192 is similar to or greater than a predetermined threshold, the constraint condition selection unit 193 outputs information indicating that there is no corresponding environmentally-compatible constraint condition data to the simulation processing unit 491.

[0141] When the simulation processing unit 491 receives input information indicating that there is no relevant environmentally-compatible constraint condition data, it simulates the processing to be executed in an environment that simulates the real environment and outputs the simulation results to the constraint condition learning unit 391.

[0142] The constraint condition learning unit 391 learns the constraint conditions based on the simulation results by the simulation processing unit 491. The time series of the operation of the control object 900 due to user operations in the third embodiment is replaced with the time series of the operation of the control object 900 in the simulation performed by the simulation processing unit 491. The constraint condition learning unit 391 outputs the constraint conditions obtained by the learning to the environment-adapted constraint condition data generation unit 292.

[0143] The data flow in the environment-adapted constraint data generation unit 292 is the same as that in Fig. 14 . Specifically, the environment-adapted constraint data generation unit 292 generates environment-adapted constraint data by linking the real environment characteristic information generated by the real environment characteristic information generation unit 191 with the constraints obtained through learning by the constraint learning unit 391. The environment-adapted constraint data generation unit 292 stores the generated environment-adapted constraint data in the environment-adapted constraint data storage unit 181. As a result, the environment-adapted constraint data generated by the environment-adapted constraint data generation unit 292 is added to the environment-adapted constraint data stored in the environment-adapted constraint data storage unit 181.

[0144] In this case, it is expected that the similarity calculated by the similarity calculation unit 192 for the added environmentally-responsive constraint condition data will indicate that the environmental characteristic information included in the environmentally-responsive constraint condition data is similar to the actual environment characteristic information by a threshold value or more. The constraint condition selection unit 193 selects the constraint condition included in the added environmentally-responsive constraint condition data, and the control execution unit 194 controls the control target 900 based on the selected constraint condition, thereby resulting in successful execution of the target process.

[0145] 18 is a diagram showing an example of a procedure of a process in which the control device 400 controls the control target 900. Steps S401 to S404 in FIG. 18 are the same as steps S201 to S204 in FIG.

[0146] In step S404, if the constraint condition selection unit 193 determines that there is corresponding environment-related constraint condition data (step S404: YES), the process proceeds to step S411. Steps S411 to S412 are the same as steps S211 to S212 in FIG.

[0147] On the other hand, if the constraint condition selection unit 193 determines in step S404 that there is no relevant environmental constraint condition data (step S404: NO), the simulation processing unit 491 performs a simulation of the processing to be executed in an environment that simulates the real environment (step S421).

[0148] The constraint condition learning unit 391 then learns the constraint conditions based on the time series of the operation of the control target 900 in the simulation (step S422). The environment-adapted constraint condition data generating unit 292 associates the real environment characteristic information generated by the real environment characteristic information generating unit 191 with the constraint conditions obtained by learning by the constraint condition learning unit 391 to generate environment-adapted constraint condition data, and stores the generated environment-adapted constraint condition data in the environment-adapted constraint condition data storage unit 181 (step S423).

[0149] The constraint condition selection unit 193 selects the newly generated environment-related constraint condition data, and the control execution unit 194 controls the control target 900 based on the constraint conditions included in the newly generated environment-related constraint condition data (step S424). After step S424, the control device 400 ends the processing of FIG.

[0150] Fig. 19 is a diagram showing an example of processing performed by the control object 900 under the control of the control device 400. Fig. 19 shows an example of processing for transferring an item in a basket C11 to a basket C12. Here, it is assumed that constraint conditions including (1) not placing a heavy item on a soft item, (2) placing an item in a stable orientation, etc. have been obtained through learning by the constraint condition learning unit 391.

[0151] Line L11 indicates that bread is being moved out of basket C11. Line L12 indicates that candy is being transferred from basket C11 to basket C12. When the control target 900 moves the candy, the control target 900 places the candy sideways because if the candy is placed upright, the bottom area is small and it is easy for it to fall over.

[0152] Line L13 indicates that milk is being transferred from basket C11 to basket C12. When the control object 900 moves the milk, if the milk is placed sideways, part of the milk will rest on the candy, making it unstable, so the control object 900 places the milk upright. Line L14 indicates that bread is being moved into basket C12. The control object 900 places the soft bread on top of the hard candy.

[0153] In this way, by the control device 400 controlling the control object 900 based on the constraint conditions, it is expected that the control object 900 will be able to perform processing in accordance with the control of the control device 400 even if the type of item, the arrangement of the items, or the position of the basket changes. Furthermore, if the control object 900 fails in processing, it is expected that the control object 900 will be able to succeed in processing by the constraint condition learning unit 391 learning the constraint conditions based on the simulation results by the simulation processing unit 491. Furthermore, when the control system 1 is introduced, it is expected that the control device 400 will be able to learn the constraint conditions in advance through simulation, so that the control system 1 will be able to perform processing immediately after introduction.

[0154] As described above, the constraint condition learning unit 391 learns the constraint conditions based on the simulation results of the process to be executed in an environment that simulates the real environment. The environment-adapted constraint condition data generating unit 292 generates environment-adapted constraint condition data in which the real environment characteristic data and the constraint conditions obtained by learning are linked.

[0155] In the control device 400, it is expected that the constraint condition learning unit 391 will learn the constraint conditions, thereby enabling the target process that has failed to be executed to succeed. Furthermore, in the control device 400, the constraint condition learning unit 391 learns the constraint conditions based on the simulation results, so it is possible to learn the constraint conditions before the introduction of the control system 1, and it is expected that the process will be able to be performed immediately after the introduction of the control system 1.

[0156] Fifth Embodiment Fig. 20 is a diagram showing an example of the configuration of a control device according to a fifth embodiment. In the configuration shown in Fig. 20, a control device 610 includes a constraint condition selection unit 611 and a control execution unit 612.

[0157] In this configuration, the constraint condition selection unit 611 selects environmentally-adaptive constraint condition data corresponding to the actual environment, which is the execution environment of the process to be executed, from among the environmentally-adaptive constraint condition data in which environmental feature information, which is information about the execution environment of the process, is linked to constraint conditions for executing the process in that environment.The control execution unit 612 controls the control target to execute the process to be executed based on the constraint conditions indicated by the selected environmentally-adaptive constraint condition data.The constraint condition selection unit 611 is an example of a constraint condition selection means.The control execution unit 612 is an example of a control execution means.

[0158] By acquiring constraints according to the real environment, the control device 610 is expected to be able to deal with not only differences in the execution environment of the process but also differences in the process being executed. For example, a constraint that a controlled object must avoid certain obstacles when moving can be applied to various processes that involve movement of the controlled object.

[0159] The constraint condition selection unit 611 can be realized using, for example, the function of the constraint condition selection unit 193 shown in Fig. 2. The control execution unit 612 can be realized using, for example, the function of the control execution unit 194 shown in Fig. 2.

[0160] Sixth Embodiment Fig. 21 is a diagram showing an example of the configuration of a constraint condition selection device according to a sixth embodiment. In the configuration shown in Fig. 21, the constraint condition selection device 620 includes a constraint condition selection unit 621. In this configuration, the constraint condition selection unit 621 selects environment-related constraint condition data corresponding to the real environment, which is the execution environment of the process to be executed, from among environment-related constraint condition data in which environmental characteristic information, which is information about the execution environment of the process, is linked to constraint conditions for executing the process in that environment. The constraint condition selection unit 621 corresponds to an example of constraint condition selection means.

[0161] By using the constraints selected by the constraint selection device 620, it is expected that it will be possible to deal with not only differences in the execution environment of the process but also differences in the process to be executed. For example, a constraint that a controlled object must avoid certain obstacles when moving can be applied to various processes involving movement of the controlled object. The constraint selection unit 621 can be realized using, for example, the functions of the constraint selection unit 193 shown in FIG. 2 .

[0162] 22 is a diagram showing an example of the configuration of a data generation device according to the seventh embodiment. In the configuration shown in Fig. 22, a data generation device 630 includes a constraint condition learning unit 631 and an environment-adapted constraint condition data generation unit 632.

[0163] With this configuration, the constraint learning unit 631 learns constraints based on the results of the processing simulation. The environment-responsive constraint data generation unit 632 links information about the environment in the processing simulation with the constraints obtained by learning, thereby generating environment-responsive constraint data in which environmental feature information, which is information about the environment, is linked to the constraints. The constraint learning unit 631 is an example of constraint learning means. The environment-responsive constraint data generation unit 632 is an example of environment-responsive constraint data generation means.

[0164] By learning the constraints, the data generating device 630 is expected to be able to use the obtained constraints to deal with not only differences in the execution environment of the processing but also differences in the processing to be executed. For example, a constraint that a controlled object must avoid certain obstacles when moving can be applied to various processing involving movement of the controlled object.

[0165] The constraint condition learning unit 631 can be realized, for example, by using the functions of the constraint condition learning unit 391 in Fig. 16. The environment-adaptive constraint condition data generating unit 632 can be realized, for example, by using the functions of the environment-adaptive constraint condition data generating unit 292 in Fig. 16.

[0166] Eighth Embodiment Fig. 23 is a diagram showing an example of a processing procedure in a control method according to an eighth embodiment. The control method shown in Fig. 23 includes selecting a constraint condition (step S611) and executing control (step S612).

[0167] In selecting a constraint (step S611), the computer selects environmentally-adapted constraint data corresponding to the actual environment, which is the execution environment of the process to be executed, from among environmentally-adapted constraint data in which environmental feature information, which is information about the execution environment of the process, is linked to constraints on the execution of the process in that environment. In executing control (step S612), the computer controls the control target to execute the process to be executed based on the constraints indicated by the selected environmentally-adapted constraint data.

[0168] 23, by acquiring constraint conditions according to the actual environment, it is expected that the control method will be able to cope with not only differences in the execution environment of the processing but also differences in the processing to be executed. For example, a constraint condition that a predetermined obstacle must be avoided when a controlled object moves can be applied to various processing involving movement of the controlled object.

[0169] Ninth Embodiment Fig. 24 is a diagram showing an example of a processing procedure in a constraint condition selection method according to a ninth embodiment. The constraint condition selection method shown in Fig. 24 includes selecting constraint conditions (step S621).

[0170] In selecting constraint conditions (step S621), the computer selects environmentally-compatible constraint condition data that corresponds to the actual environment, which is the execution environment of the process to be executed, from among the environmentally-compatible constraint condition data that links environmental characteristic information, which is information about the execution environment of the process, with constraint conditions for executing the process in that environment.

[0171] 24, it is expected that the constraints selected can be used to accommodate not only differences in the execution environment of the process but also differences in the process to be executed. For example, a constraint that a controlled object must avoid certain obstacles when moving can be applied to various processes involving the movement of the controlled object.

[0172] Tenth Embodiment Fig. 25 is a diagram showing an example of a processing procedure in a data generation method according to a tenth embodiment. The data generation method shown in Fig. 25 includes learning constraint conditions (step S631) and generating environment-associated constraint condition data (step S632).

[0173] In learning constraints (step S631), the computer learns the constraints based on the results of the process simulation. In generating environment-related constraint data (step S632), information about the environment in the process simulation is linked to the constraints obtained by learning, thereby generating environment-related constraint data in which environmental feature information, which is information about the environment, is linked to the constraints.

[0174] 25, by learning the constraints, it is expected that the obtained constraints can be used to deal with not only differences in the execution environment of the processing but also differences in the processing to be executed. For example, a constraint that a controlled object must avoid certain obstacles when moving can be applied to various processing involving the movement of the controlled object.

[0175] 26 is a schematic block diagram illustrating the configuration of a computer according to at least one embodiment. In the configuration shown in FIG. 26, a computer 700 includes a CPU 710, a main memory device 720, an auxiliary memory device 730, an interface 740, and a non-volatile recording medium 750.

[0176] One or more of the above-described control devices 100, 200, 300, 400, 610, constraint selection device 620, and data generation device 630, or a portion thereof, may be implemented in a computer 700. In this case, the operation of each of the above-described processing units is stored in the auxiliary storage device 730 in the form of a program. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-described processing in accordance with the program. The CPU 710 also allocates storage areas in the main storage device 720 corresponding to each of the above-described storage units in accordance with the program. Communication between each device and other devices is performed by an interface 740 having a communication function and performing communication under the control of the CPU 710.

[0177] When the control device 100 is implemented in a computer 700, the operations of the processing unit 190 and each of its units are stored in the form of a program in an auxiliary storage device 730. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-described processing in accordance with the program.

[0178] Furthermore, the CPU 710 allocates storage areas for the storage unit 180 and each unit thereof in the main storage device 720 in accordance with the program. Communication with other devices by the communication unit 110 is achieved by the interface 740 having a communication function and operating under the control of the CPU 710. Display of various images by the display unit 120 is achieved by the interface 740 having a display device and displaying various images under the control of the CPU 710. Reception of user operations by the operation input unit 130 is achieved by the interface 740 having an input device and receiving user operations under the control of the CPU 710.

[0179] When the control device 200 is implemented in a computer 700, the operations of the processing unit 290 and each of its units are stored in the form of a program in an auxiliary storage device 730. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-described processing in accordance with the program.

[0180] Furthermore, the CPU 710 allocates storage areas for the storage unit 180 and each unit thereof in the main storage device 720 in accordance with the program. Communication with other devices by the communication unit 110 is achieved by the interface 740 having a communication function and operating under the control of the CPU 710. Display of various images by the display unit 120 is achieved by the interface 740 having a display device and displaying various images under the control of the CPU 710. Reception of user operations by the operation input unit 130 is achieved by the interface 740 having an input device and receiving user operations under the control of the CPU 710.

[0181] When the control device 300 is implemented in a computer 700, the operations of the processing unit 390 and each of its units are stored in the form of a program in an auxiliary storage device 730. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-described processing in accordance with the program.

[0182] Furthermore, the CPU 710 allocates storage areas for the storage unit 180 and each unit thereof in the main storage device 720 in accordance with the program. Communication with other devices by the communication unit 110 is achieved by the interface 740 having a communication function and operating under the control of the CPU 710. Display of various images by the display unit 120 is achieved by the interface 740 having a display device and displaying various images under the control of the CPU 710. Reception of user operations by the operation input unit 130 is achieved by the interface 740 having an input device and receiving user operations under the control of the CPU 710.

[0183] When the control device 400 is implemented in a computer 700, the operations of the processing unit 490 and each of its units are stored in the form of a program in an auxiliary storage device 730. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-described processing in accordance with the program.

[0184] Furthermore, the CPU 710 allocates storage areas for the storage unit 180 and each unit thereof in the main storage device 720 in accordance with the program. Communication with other devices by the communication unit 110 is achieved by the interface 740 having a communication function and operating under the control of the CPU 710. Display of various images by the display unit 120 is achieved by the interface 740 having a display device and displaying various images under the control of the CPU 710. Reception of user operations by the operation input unit 130 is achieved by the interface 740 having an input device and receiving user operations under the control of the CPU 710.

[0185] When the control device 610 is implemented in the computer 700, the operations of the constraint condition selection unit 611 and the control execution unit 612 are stored in the form of a program in the auxiliary storage device 730. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-mentioned processing in accordance with the program.

[0186] Furthermore, the CPU 710 allocates a storage area in the main storage device 720 for the control device 610 to perform processing in accordance with the program. Communication between the control device 610 and other devices is performed by the interface 740, which has a communication function and operates under the control of the CPU 710. Interaction between the control device 610 and a user is performed by the interface 740, which has a display device and an input device, displaying various images under the control of the CPU 710 and accepting user operations.

[0187] When the constraint condition selection device 620 is implemented in the computer 700, the operation of the constraint condition selection unit 621 is stored in the form of a program in the auxiliary storage device 730. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-mentioned processing in accordance with the program.

[0188] Furthermore, the CPU 710, in accordance with the program, allocates a storage area in the main storage device 720 for the constraint selection device 620 to perform processing. Communication between the constraint selection device 620 and other devices is achieved by the interface 740, which has a communication function and operates under the control of the CPU 710. Interaction between the constraint selection device 620 and a user is achieved by the interface 740, which has a display device and an input device, displaying various images under the control of the CPU 710 and accepting user operations.

[0189] When the data generating device 630 is implemented in the computer 700, the operations of the constraint condition learning unit 631 and the environment-associated constraint condition data generating unit 632 are stored in the form of a program in the auxiliary storage device 730. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-mentioned processing in accordance with the program.

[0190] Furthermore, the CPU 710 allocates a storage area in the main storage device 720 for the data generating device 630 to perform processing in accordance with the program. Communication between the data generating device 630 and other devices is performed by an interface 740 having a communication function and operating under the control of the CPU 710. Interaction between the data generating device 630 and a user is performed by the interface 740 having a display device and an input device, displaying various images under the control of the CPU 710, and accepting user operations.

[0191] One or more of the above-described programs may be recorded on nonvolatile recording medium 750. In this case, interface 740 may read the programs from nonvolatile recording medium 750. Then, CPU 710 may directly execute the programs read by interface 740, or may temporarily store the programs in main storage device 720 or auxiliary storage device 730 and then execute them.

[0192] In addition, a program for executing all or part of the processing performed by the control device 100, control device 200, control device 300, control device 400, control device 610, constraint condition selection device 620, and data generation device 630 may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be loaded into a computer system and executed to perform the processing of each unit. Note that the term "computer system" here includes hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs (Read Only Memory), and CD-ROMs (Compact Disc Read Only Memory), as well as storage devices such as hard disks built into the computer system. The program may be designed to implement part of the aforementioned functions, or may be capable of implementing the aforementioned functions in combination with a program already recorded on the computer system.

[0193] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention.

[0194] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0195] (Supplementary Note 1) A control device comprising: a constraint condition selection means for selecting environmentally-compatible constraint condition data corresponding to a real environment, which is the execution environment of a process to be executed, from environmentally-compatible constraint condition data in which environmental characteristic information, which is information about the execution environment of the process, is linked to constraint conditions for executing the process in that environment; and a control execution means for controlling a control object to execute the process to be executed based on the constraint conditions indicated by the selected environmentally-compatible constraint condition data.

[0196] (Supplementary Note 2) The control device described in Supplementary Note 1 further comprises: a real environment characteristic information generation means for generating real environment characteristic information, which is environmental characteristic information in the real environment, based on observation data of the real environment obtained by a sensor that observes the real environment; and a similarity calculation means for calculating the similarity of the environmental characteristic information, wherein the constraint condition selection means selects environmentally-adapted constraint condition data corresponding to the real environment based on the similarity between the real environment characteristic information and the environmental characteristic information included in the environmentally-adapted constraint condition data.

[0197] (Supplementary Note 3) The control device according to Supplementary Note 2, wherein the environmental feature information is expressed as a graph including nodes that indicate objects in an execution environment of the process and edges that indicate relationships between the nodes.

[0198] (Appendix 4) A control device described in any one of Appendices 1 to 3, further comprising: an environmentally-adapted constraint condition data generation means that generates environmentally-adapted constraint condition data that links the actual environment characteristic information with constraint conditions in the actual environment when there is no environmentally-adapted constraint condition data that includes environmental characteristic information whose similarity to the actual environment characteristic information is greater than a predetermined threshold.

[0199] (Appendix 5) The control device described in Appendix 4 further comprises an assistance request means for requesting user operation when there is no environment-adapted constraint condition data including environmental feature information whose similarity to real environment feature information is greater than a predetermined threshold, and the environment-adapted constraint condition data generation means generates environment-adapted constraint condition data that links the real environment feature information with constraint conditions in the real environment based on the user operation.

[0200] (Supplementary Note 6) The control device according to Supplementary Note 5, wherein the environment-adapted constraint condition data generating means generates environment-adapted constraint condition data in which real environment characteristic information and constraint conditions input by user operation are linked together.

[0201] (Supplementary Note 7) The control device according to Supplementary Note 5, further comprising: a constraint condition learning means for learning constraint conditions based on a user operation for operating a control object; and the environment-adaptive constraint condition data generating means for generating environment-adaptive constraint condition data in which the real environment characteristic data and the constraint conditions obtained by learning are linked.

[0202] (Supplementary Note 8) The control device according to Supplementary Note 4, further comprising a constraint condition learning means for learning constraint conditions based on simulation results of the processing to be executed in an environment simulating a real environment, wherein the environment-adapted constraint condition data generation means generates environment-adapted constraint condition data in which the real environment characteristic data and the constraint conditions obtained by learning are linked.

[0203] (Supplementary Note 9) A constraint condition selection device comprising: a constraint condition selection means for selecting environmentally-compatible constraint condition data corresponding to the actual environment, which is the execution environment of the process to be executed, from environmentally-compatible constraint condition data in which environmental feature information, which is information about the execution environment of the process, is linked to constraint conditions for executing the process in that environment.

[0204] (Supplementary Note 10) A data generation device comprising: a constraint condition learning means for learning constraint conditions based on the results of a simulation of a process; and an environment-responsive constraint condition data generation means for generating environment-responsive constraint condition data in which environmental feature information, which is information about the environment, is linked to the constraint conditions by linking information about the environment in the simulation of the process with the constraint conditions obtained by the learning.

[0205] (Supplementary Note 11) A control method including a computer selecting, from among environmental constraint condition data in which environmental feature information, which is information about the environment in which the processing is executed, is linked to constraint conditions for executing the processing in that environment, environmental constraint condition data that corresponds to the actual environment, which is the execution environment of the processing to be executed, and controlling the controlled object to execute the processing to be executed based on the constraint conditions indicated by the selected environmental constraint condition data.

[0206] (Supplementary Note 12) A constraint selection method including a computer selecting, from among environmental constraint data in which environmental feature information, which is information about the environment in which the processing is executed, is linked to constraints for executing the processing in that environment, environmental constraint data that corresponds to the actual environment, which is the execution environment of the processing to be executed.

[0207] (Supplementary Note 13) A data generation method including the steps of: a computer learning constraint conditions based on the results of a simulation of a process; and linking information about the environment in the simulation of the process with the constraint conditions obtained by the learning, thereby generating environment-responsive constraint condition data in which environmental feature information, which is information about the environment, is linked with the constraint conditions.

[0208] (Appendix 14) A recording medium having recorded thereon a program for causing a computer to perform the following steps: selecting environmentally-compatible constraint condition data corresponding to the actual environment, which is the execution environment of the process to be executed, from environmentally-compatible constraint condition data in which environmental feature information, which is information about the execution environment of the process, is linked to constraint conditions for executing the process in that environment; and controlling the controlled object to execute the process to be executed based on the constraint conditions indicated by the selected environmentally-compatible constraint condition data.

[0209] (Appendix 15) A recording medium having recorded thereon a program for causing a computer to execute the following: selecting environmentally-compatible constraint condition data corresponding to the actual environment, which is the execution environment of the process to be executed, from environmentally-compatible constraint condition data in which environmental characteristic information, which is information about the execution environment of the process, is linked to constraint conditions for executing the process in that environment.

[0210] (Appendix 16) A recording medium having recorded thereon a program for causing a computer to execute the following steps: learning constraint conditions based on the results of a simulation of a process; and linking information about the environment in the simulation of the process with the constraint conditions obtained by the learning, thereby generating environment-responsive constraint condition data in which environmental feature information, which is information about the environment, is linked with the constraint conditions.

[0211] The present invention may be applied to a control device, a constraint condition selection device, a data generation device, a control method, a constraint condition selection method, a data generation method, and a storage medium.

[0212] 1 Control system 100, 200, 300, 400, 610 Control device 110 Communication unit 120 Display unit 130 Operation input unit 180 Storage unit 181 Environment-associated constraint condition data storage unit 190, 290, 390, 490 Processing unit 191 Real environment characteristic information generation unit 192 Similarity calculation unit 193, 611, 621 Constraint condition selection unit 194, 612 Control execution unit 291 Support request unit 292, 632 Environment-associated constraint condition data generation unit 391, 631 Constraint condition learning unit 491 Simulation processing unit 620 Constraint condition selection device 630 Data generation device 900 Control object

Claims

1. Constraint condition selection means for selecting environment - corresponding constraint condition data corresponding to the actual environment, which is the execution environment of the process to be executed, from among the environment - corresponding constraint condition data in which environment characteristic information, which is information on the execution environment of the process, and constraint conditions for the execution of the process in that environment are associated; Control execution means for controlling the control target to execute the process to be executed based on the constraint conditions indicated by the selected environment - corresponding constraint condition data; A control device comprising the above.

2. Actual environment characteristic information generation means for generating actual environment characteristic information, which is environment characteristic information in the actual environment, based on observation data of the actual environment by a sensor that observes the actual environment; Similarity calculation means for calculating the similarity of the environment characteristic information; Further comprising: The constraint condition selection means selects the environment - corresponding constraint condition data corresponding to the actual environment based on the similarity between the actual environment characteristic information and the environment characteristic information included in the environment - corresponding constraint condition data. The control device according to claim 1.

3. The environment characteristic information is represented by a graph including nodes indicating objects in the execution environment of the process and edges indicating relationships between the nodes. The control device according to claim 2.

4. When there is no environment - corresponding constraint condition data including environment characteristic information that is similar to the actual environment characteristic information by a predetermined threshold or more, environment - corresponding constraint condition data generation means for generating environment - corresponding constraint condition data in which the actual environment characteristic information and the constraint conditions in the actual environment are associated; The control device according to any one of claims 1 to 3, further comprising the above.

5. When there is no environment - corresponding constraint condition data including environment characteristic information that is similar to the actual environment characteristic information by a predetermined threshold or more, further comprising assistance request means for requesting a user operation; The environment - corresponding constraint condition data generation means generates environment - corresponding constraint condition data in which the actual environment characteristic information and the constraint conditions in the actual environment are associated based on the user operation. The control device according to claim 4.

6. The environment - corresponding constraint condition data generation means generates environment - corresponding constraint condition data in which the actual environment characteristic information and the constraint conditions input by the user operation are associated. The control device according to claim 5.

7. Constraint condition learning means for learning constraint conditions based on a user operation for operating the control target; Further comprising: The environmental response constraint condition data generation means generates environmental response constraint condition data in which the actual environment feature data and the constraint conditions obtained by learning are associated. The control device according to claim 5.

8. Constraint condition learning means for learning constraint conditions based on the simulation results of the processing to be executed in an environment that simulates the actual environment. further comprising The environmental response constraint condition data generation means generates environmental response constraint condition data in which the actual environment feature data and the constraint conditions obtained by learning are associated. The control device according to claim 4.

9. A computer selects environmental response constraint condition data corresponding to the actual environment, which is the execution environment of the processing to be executed, from among the environmental response constraint condition data in which environmental feature information, which is information on the execution environment of the processing, and the constraint conditions for the execution of the processing in that environment are associated, and controls the control target to execute the processing to be executed based on the constraint conditions indicated by the selected environmental response constraint condition data. A control method including the above.

10. On a computer selecting environmental response constraint condition data corresponding to the actual environment, which is the execution environment of the processing to be executed, from among the environmental response constraint condition data in which environmental feature information, which is information on the execution environment of the processing, and the constraint conditions for the execution of the processing in that environment are associated, and controlling the control target to execute the processing to be executed based on the constraint conditions indicated by the selected environmental response constraint condition data. A program for causing the above to be executed.