Control device, control method, and program

Abstract

The present invention provides a control device comprising: a constraint condition selection means that selects environment-compliant constraint condition data which is of environment-compliant constraint condition data in which environment characteristic information, which is information pertaining to an execution environment of a process and a constraint condition for execution of the process in that environment are linked to each other and corresponds to an actual environment, which is an execution environment of a process to be executed; and a control execution means that controls a control object to execute the process to be executed on the basis of the constraint condition indicated by the selected environment-compliant constraint condition data.

Description

Technical Field

[0004]

[0001] The present invention relates to a control device , regulation control method and program and related thereto.

Background Art

[0002] Techniques for obtaining control commands for a robot according to the operating environment of the robot and controlling the robot have been proposed. For example, the learned model generation device described in Patent Document 1 generates a learning dataset in which the feature amount extracted by an autoencoder from sensor data related to the operating environment of the robot is associated with the operation to be performed by the robot. Then, the learned model generation device generates a learned model in which the relationship between the operating environment and the operation to be performed by the robot is defined by referring to the obtained learning dataset. Further, the robot control device generates a command corresponding to the operating environment by inputting the feature amount extracted from the sensor data related to the operating environment of the controlled robot into the learned model. The robot control device operates the controlled robot by transmitting the generated command to the controlled robot.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When the processes to be performed on a controlled object such as a robot are different, it is preferable that each process can be performed on the controlled object.

[0005] An example of the object of this disclosure is to provide a control device , regulation control method, constraint condition selection method and program that can solve the above problems. [Means for solving the problem]

[0006] According to a first aspect of the present invention, the control device is A real-environment feature information generation means generates real-environment feature information, which is information about the real environment, based on observation data of the real environment by a sensor that observes the real environment, which is the execution environment of the process to be executed; and a similarity calculation means calculates the similarity of the environmental feature information. Among the environmental constraint data, which links environmental characteristic information (information about the execution environment of the process) with constraints on the execution of the process in that environment, the environmental constraint data corresponding to the actual environment that is the execution environment of the process being executed is selected. Based on the similarity between the aforementioned real-world characteristic information and the environmental characteristic information included in the environmental constraint data The system comprises a constraint selection means for selecting constraint conditions, and a control execution means for controlling the controlled object to execute the process to be executed based on the constraint conditions indicated by the selected environment-responsive constraint data. According to a second aspect of the present invention, the control device includes constraint selection means for selecting environmental constraint data corresponding to the actual environment which is the execution environment of the process to be executed, from among environmental constraint data which is linked to environmental feature information which is information relating to the execution environment of the process and constraints for the execution of the process in that environment; control execution means for controlling the control target to execute the process to be executed based on the constraints indicated by the selected environmental constraint data; and environmental constraint data generation means for generating environmental constraint data which is linked to the actual environment feature information and constraints in the actual environment if there is no environmental constraint data which is similar to the actual environment feature information to a predetermined threshold or higher.

[0009] This invention three According to this embodiment, the control method includes a computer selecting environmental constraint data from among environmental constraint data, which is a collection of environmental characteristic information relating to the execution environment of a process and constraints on the execution of the process in that environment, that corresponds to the actual environment which is the execution environment of the process to be executed, and controlling the target to execute the process to be executed based on the constraints indicated by the selected environmental constraint data.

[0012] This invention four According to this embodiment, the program is a program that causes a computer to perform the following actions: select environmental constraint data that corresponds to the actual environment which is the execution environment of the process being executed, from among environmental constraint data which is linked to environmental characteristic information which is information about the execution environment of the process and constraints for the execution of the process in that environment; and control the target to execute the process being executed based on the constraints indicated by the selected environmental constraint data. [Effects of the Invention]

[0015] According to the present invention, it is expected that, in cases where the processes to be performed by a controlled object such as a robot are different, each process can be made to be performed by the controlled object. [Brief explanation of the drawing]

[0016] [Figure 1] This figure shows an example of the configuration of the control system according to the first embodiment. [Figure 2] This figure shows an example of the configuration of the control device according to the first embodiment. [Figure 3] This figure shows an example of an image captured in a real environment according to the first embodiment. [Figure 4] This figure shows a first example of an environmental graph in the first embodiment. [Figure 5] This figure shows a second example of the environmental graph in the first embodiment. [Figure 6] This figure shows a third example of the environmental graph in the first embodiment. [Figure 7] This figure shows a fourth example of the environmental graph in the first embodiment. [Figure 8] This figure shows an example of the data flow when the control device according to the first embodiment controls the controlled object. [Figure 9] This figure shows an example of the procedure for a process in which the control device according to the first embodiment controls the controlled object. [Figure 10] This figure shows an example of the configuration of a control device according to the second embodiment. [Figure 11] This figure shows an example of the data flow when the control device according to the second embodiment generates new environmental constraint data. [Figure 12] This figure shows an example of the procedure for a process in which the control device according to the second embodiment controls the controlled object. [Figure 13] This figure shows an example of the configuration of a control device according to the third embodiment. [Figure 14] This figure shows an example of the data flow when the control device according to the third embodiment generates new environmental constraint data. [Figure 15]It is a diagram showing an example of the procedure of the process in which the control device according to the third embodiment controls the control target. [Figure 16] It is a diagram showing an example of the configuration of the control device according to the fourth embodiment. [Figure 17] It is a diagram showing an example of the data flow when the control device according to the fourth embodiment newly generates environment adaptation constraint condition data. [Figure 18] It is a diagram showing an example of the procedure of the process in which the control device according to the fourth embodiment controls the control target. [Figure 19] It is a diagram showing an example of the process that the control target in the fourth embodiment performs according to the control by the control device. [Figure 20] It is a diagram showing an example of the configuration of the control device according to the fifth embodiment. [Figure 21] It is a diagram showing an example of the configuration of the constraint condition selection device according to the sixth embodiment. [Figure 22] It is a diagram showing an example of the configuration of the data generation device according to the seventh embodiment. [Figure 23] It is a diagram showing an example of the procedure of the process in the control method according to the eighth embodiment. [Figure 24] It is a diagram showing an example of the procedure of the process in the constraint condition selection method according to the ninth embodiment. [Figure 25] It is a diagram showing an example of the procedure of the process in the data generation method according to the tenth embodiment. [Figure 26] It is a schematic block diagram showing the configuration of a computer according to at least one embodiment.

Embodiments for Carrying Out the Invention

[0017] Hereinafter, embodiments of the present invention will be described. However, the following embodiments do not limit the invention according to the claims. Also, not all combinations of the features described in the embodiments are essential for the solution means of the invention.

[0018] <First Embodiment> Figure 1 is a diagram showing an example of the configuration of a control system according to the first embodiment. In the configuration shown in Figure 1, the control system 1 comprises a control device 100 and a controlled object 900. Control system 1 is a system in which the controlled object 900 operates according to the control of the control device 100.

[0019] The control device 100 controls the controlled object 900 so that it performs the process to be executed. In particular, the control device 100 determines the constraints on the controlled object 900 when it performs the process, according to the execution environment of the process. Then, the control device 100 controls the controlled object 900 based on the determined constraints.

[0020] Specifically, the control device 100 stores data that links information about the processing execution environment with constraints on the execution of the processing in that environment. Information about the processing execution environment is also called environmental characteristic information. Data that links information about the processing execution environment with constraints on the execution of the processing in that environment is also called environment-compatible constraint data.

[0021] The control device 100 then compares the environmental characteristic information contained in the environmental constraint data with the environmental characteristic information of the execution environment of the process being executed, and selects the environmental constraint data corresponding to the execution environment of the process being executed. The control device 100 then reads the constraints from the selected environmental constraint data to obtain the constraints corresponding to the execution environment of the process being executed. The control device 100 is an example of a constraint selection device. The process being executed is also called the execution target process. The execution environment of the execution target process is also called the actual environment. The environmental characteristic information in the actual environment is also called the actual environment characteristic information.

[0022] The control device 100 is expected to be able to handle not only differences in the execution environment of the process, but also differences in the target process, by acquiring constraints according to the actual environment. For example, a constraint that the controlled object 900 must avoid a predetermined obstacle when it moves can be applied to various processes that involve the movement of the controlled object 900.

[0023] Furthermore, the control device 100 may acquire constraints not only from the actual environment but also from the processing to be executed. For example, as will be described later, the environmental characteristic information may include not only information about the initial state of the processing execution environment but also information about the target state.

[0024] The control device 100 may be configured using a computer such as a personal computer (PC) or a workstation (WS).

[0025] The controlled object 900 operates according to the control of the control device 100. The controlled object 900 is not limited to a specific object, but can be various controllable objects. The processing performed by the controlled object 900 can be various processing depending on the controlled object 900.

[0026] For example, the controlled object 900 may be a multi-joint robot (manipulator). In this case, depending on the type of end effector, various processes can be performed on the controlled object, such as grasping and moving an object, or processing a manufactured product.

[0027] Furthermore, the controlled object 900 may be a mobile device such as an automated guided vehicle or a drone. In this case, various processes involving the movement of the controlled object 900, such as the transport of objects, can be performed as the processes to be executed. Furthermore, the controlled object 900 may be a single device, or it may be a system including multiple devices, such as a power plant or a chemical plant. In this case, various processes can be performed depending on the type of device or system.

[0028] The execution environment for processing is also referred to as the operating environment for the controlled object 900. However, it is not necessary for the execution environment for processing and the controlled object 900 to have a one-to-one correspondence. In a particular environment, one of several controlled objects 900 may be selectively operated, or multiple controlled objects 900 may be made to operate in a coordinated manner. For example, the constraint described above, that the controlled object 900 must avoid a predetermined obstacle when it moves, can be used when the controlled object 900 is a variety of devices that move.

[0029] The operating environment of the controlled object 900 may include the controlled object 900 itself. For example, the state of the operating environment of the controlled object 900 may include the state of the controlled object 900.

[0030] Figure 2 shows an example of the configuration of the control device 100. In the configuration shown in Figure 2, the control device 100 comprises a communication unit 110, a display unit 120, an operation input unit 130, a storage unit 180, and a processing unit 190. The storage unit 180 comprises an environment-responsive constraint data storage unit 181. The processing unit 190 comprises a real-world feature information generation unit 191, a similarity calculation unit 192, a constraint selection unit 193, and a control execution unit 194.

[0031] The communication unit 110 communicates with other devices. For example, the communication unit 110 may receive observation data of the real environment from a sensor that observes the real environment. Alternatively, the communication unit 110 may transmit control commands to the controlled object 900.

[0032] The display unit 120 has a display screen such as a liquid crystal panel or an LED (Light Emitting Diode) panel, and displays various images. For example, the display unit 120 may display various information related to the execution of processing by the controlled object 900, such as constraint conditions selected by the control device 100, or real-time images of the controlled object 900.

[0033] The operation input unit 130 is configured to include, for example, input devices such as a keyboard and a mouse, and accepts user operations. For example, the operation input unit 130 may be configured to accept user operations that specify the process to be executed by the control system 1 (the process to be executed).

[0034] The storage unit 180 stores various types of data. The storage unit 180 is configured using the storage devices provided by the control device 100. The environmental constraint data storage unit 181 stores environmental constraint data.

[0035] The processing unit 190 controls various parts of the control device 100 to perform various processes. The functions of the processing unit 190 are performed, for example, by the CPU (Central Processing Unit) of the control device 100 reading a program from the storage unit 180 and executing it.

[0036] The real-world feature information generation unit 191 generates real-world feature information based on observation data of the real environment from sensors that observe the real environment. As described above, real-world feature information is environmental feature information in the real environment. Environmental feature information is information about the execution environment of the processing. The real-world feature information generation unit 191 is an example of a real-world feature information generation means.

[0037] Environmental feature information is not limited to specific information, but can be various types of information for which similarity can be calculated. For example, the sensors used to generate real-world environmental feature information can be various sensors depending on the controlled object 900 or the processing of the executed object. A camera may be used as the sensor, and the real-world feature information generation unit 191 may acquire images of the real environment as observation data of the real environment. Alternatively, a sensor that outputs some value, such as the air temperature around the controlled object 900 or the flow rate of the raw material handled by the controlled object 900, may be used, and the real-world feature information generation unit 191 may acquire the sensor measurement value. The real-world feature information generation unit 191 may also acquire data from multiple sensors.

[0038] Furthermore, as real-world feature information, observational data of the real environment may be used as is, or processed data from the observational data may be used. For example, if an image of the real environment is obtained as observational data, the real-world feature information generation unit 191 may perform image recognition processing on the image of the real environment to extract objects that are captured in the image. The real-world feature information generation unit 191 may then generate information indicating the attributes or state of each extracted object, such as the position, size, color, material, or a part thereof, as real-world feature information. Alternatively, the real-world feature information generation unit 191 may use the captured images of the real environment directly as real-world feature information. Objects captured in images of real-world environments are examples of objects located in those environments.

[0039] Furthermore, the real-world feature information is not limited to any specific format. For example, if the real-world feature information uses information indicating the attributes or state of each object located in the real environment, the real-world feature information may be in the form of a graph (directed or undirected). A graph showing real-world feature information is also called an environment graph. Alternatively, the real-world characteristic information may be in tabular format, showing the attributes or state of each object. Or, the real-world information may be in vector format, listing the attributes or state of each object in order.

[0040] Figure 3 shows an example of an image taken in a real environment. In the example in Figure 3, the image of a real environment shows a tray, a rectangular box placed inside the tray, and a cylindrical can placed on top of the box. These tray, box, and can are examples of objects that exist in a real environment.

[0041] Figure 4 shows a first example of an environmental graph. Figure 4 shows an example of an environmental graph generated by the real-world feature information generation unit 191 from the captured images of the real environment shown in Figure 3. In the example in Figure 4, the environment graph consists of three nodes, "Tray 1," "Rectangular Parallel 1," and "Cylinder 1," and edges that show the relationships between these nodes.

[0042] In the example in Figure 4, the nodes in the environment graph represent objects that appear in images of the real environment. The node "Tray 1" represents a tray. The node "Cuboid 1" represents a rectangular box. The node "Cylinder 1" represents a cylindrical can. The shapes of these objects that appear in images of the real environment are used as identification information for the nodes in the environment graph. In the example in Figure 4, the shape of a tray is referred to as "Tray". The "1" in "Tray 1," etc., is a sequential number for each object shape used to identify each node when there are multiple objects of the same shape.

[0043] Furthermore, in the example in Figure 4, each node is associated with information that indicates the attributes or state of the object it represents, including the type of object ("tray", "box", "can"), the position of each object ("x=...", "y=...", "z=..."), and the color of each object ("white", "blue", "red").

[0044] Furthermore, in the example in Figure 4, each edge is associated with a relationship it represents. The relationship "in" indicates that the object represented by the starting node of a directed edge is located inside the object represented by the ending node. The relationship "on" indicates that the object represented by the starting node of a directed edge is located on top of the object represented by the ending node. As shown in the example in Figure 4, environmental feature information may be represented by an environment graph that includes nodes representing objects in the environment and edges representing the relationships between nodes.

[0045] Figure 5 shows a second example of an environment graph. In the example in Figure 5, in addition to each object captured in the image of the real environment, the attributes or state of each object are also shown as nodes. A node representing an object and a node representing the type of that object are connected by the edge "Type". A node representing an object and a node representing the position of that object are connected by the edge "Position". A node representing an object and a node representing the color of that object are connected by the edge "Color".

[0046] In Figure 5, for the sake of clarity, nodes and edges indicating the attributes or states of the tray indicated by node "Tray 1" and the can indicated by node "Cylinder 1" are shown, but detailed information about these nodes and edges is omitted. As shown in the example in Figure 5, environmental feature information may also be represented by an environment graph that includes nodes other than those representing objects in the environment.

[0047] Figure 6 shows a third example of an environment graph. In the example in Figure 6, the environment graphs for the initial state and the target state of the processing execution environment are shown. The environment graph showing the initial state shows that there are two trays, "Tray 1" and "Tray 2," and two objects, "Rectangular Parallel 1" and "Cylinder 1," are located inside "Tray 1." The environment graph showing the target state shows that the two objects, "Rectangular Parallel 1" and "Cylinder 1," are located inside "Tray 2," and "Cylinder 1" is located on top of "Rectangular Parallel 1."

[0048] In the example in Figure 6, the process of moving "Rectangular Parallel 1" and "Cylinder 1" to change from the initial state to the target state is shown, based on a combination of the initial state environment graph and the target state environment graph. As shown in the example in Figure 6, the environmental feature information may include an environmental graph of the initial state and an environmental graph of the target state.

[0049] Thus, the combination of the initial state environment graph and the target state environment graph provides information not only about the execution environment of the process but also about the process to be executed. It is expected that the control device 100 can obtain constraints that correspond not only to the actual environment but also to the process to be executed by selecting environment-responsive constraint data using environmental feature information including the initial state environment graph and the target state environment graph.

[0050] Figure 7 shows a fourth example of an environment graph. Figure 7 shows an example of an environment graph that combines the initial state environment graph and the target state environment graph from the example in Figure 6 into a single graph. The nodes "Tray 1 Initial," "Tray 2 Initial," "Cuboid 1 Initial," and "Cylinder 1 Initial" represent the objects in the initial state. The nodes "Tray 1 Target," "Tray 2 Target," "Cuboid 1 Target," and "Cylinder 1 Target" represent the objects in the target state. The "time" edge shows the time elapsed from the initial state to the target state.

[0051] As shown in the example in Figure 7, the environmental feature information may include an environment graph that combines the environment graph of the initial state of the processing execution environment and the environment graph of the target state into a single graph. It is expected that the control device 100 can select environment-responsive constraint condition data using environmental feature information that includes an environment graph that combines the environment graph of the initial state of the processing execution environment and the environment graph of the target state into a single graph, thereby obtaining constraint conditions that correspond not only to the actual environment but also to the processing to be executed, similar to the example in Figure 6.

[0052] The similarity calculation unit 192 calculates the similarity of the environmental feature information. Various indicators can be used as the similarity calculated by the similarity calculation unit 192, depending on the environmental feature information.

[0053] For example, if environmental feature information is represented as a graph (directed or undirected graph), the similarity calculation unit 192 may use the similarity of nodes or subgraphs representing objects, and the similarity of edges between nodes representing objects, as the similarity of the environmental feature information.

[0054] In calculating the similarity between two nodes or subgraphs representing objects, for example, the attributes or states of each object can be represented by vectors, and an index representing vector similarity, such as cosine similarity or norm, can be used. When using cosine similarity as the index representing vector similarity, a larger index value indicates that the two vectors are similar. On the other hand, when using norm as the index representing vector similarity, a smaller index value indicates that the two vectors are similar.

[0055] When representing the attributes or states of two objects using vectors, the similarity calculation unit 192 ensures that the order of items in the two vectors is the same, for example, by arranging the coordinate values ​​indicating position in the order of x, y, and z coordinates. The similarity calculation unit 192 may also exclude items for which data exists for only one of the two objects from the vector elements, thereby excluding them from the similarity determination.

[0056] Furthermore, for items that are not numerical in nature, such as the shape of an object, the similarity calculation unit 192 may quantify the similarity separately from the vector similarity. For example, if the values ​​of the items are the same for both objects, the similarity may be set to a predetermined value, and if they are different, the similarity may be set to 0. This quantification may then be added to the vector similarity. In cases where a norm is used as an index to represent vector similarity, and a smaller index value indicates greater vector similarity, the similarity may be set to a sufficiently large predetermined value when the values ​​of the same items are different for both objects, which is larger than the value when the values ​​of the same items are the same for both objects.

[0057] In calculating the similarity of nodes representing objects in two graphs, the similarity calculation unit 192 may, for example, associate the nodes representing objects between the two graphs in such a way that the average value of the similarity of the associated nodes across the entire graph is maximized. The similarity calculation unit 192 may then use the maximum value of the average value of the similarity of the associated nodes across the entire graph as the similarity of the nodes in the two graphs.

[0058] In calculating edge similarity, for example, two edges selected from each of the two graphs may be defined as the same edge if both of the nodes at both ends of the edge are matched using the node mapping described above. The similarity calculation unit 192 may then count the number of identical edges across the entire graph. To avoid the similarity of edges increasing with increasing graph scale, the similarity calculation unit 192 may divide the total number of identical edges across the entire graph by the number of nodes representing objects and use this value as the edge similarity between the two graphs.

[0059] Edges can be used to represent not only the connection relationships between nodes, but also various values ​​related to the nodes connected by those edges, such as the distance between nodes. Edges can be represented as real numbers or as vectors. When edges are represented as vectors, metrics representing vector similarity, such as cosine similarity or norm, can be used to express the similarity of the edges.

[0060] The similarity calculation unit 192 may also quantify the similarity of edges for items that are not numerical, separately from the vector similarity. For example, if the values ​​of the items are the same for two edges, the similarity is set to a predetermined value, and if they are different, the similarity is set to 0. This quantification is then added to the vector similarity. Furthermore, when using a norm as an index to represent vector similarity, for example, if a smaller index value indicates greater vector similarity, the similarity may be set to a sufficiently large predetermined value when the values ​​of the same items are different for two objects, which is larger than the value when the values ​​of the same items are the same for two objects.

[0061] Furthermore, the similarity calculation unit 192 may use a value obtained by weighting the similarity of nodes in the two graphs and the similarity of edges in the two graphs by a predetermined weight coefficient, as the similarity of the two graphs.

[0062] Alternatively, the similarity calculation unit 192 may be configured to include neural networks such as a graph neural network (GNN) and a graph convolutional network (GCN), and may be pre-trained to calculate the graph's features. The similarity calculation unit 192 may then use the neural network to calculate the features of the entire graph and compare (cluster) the similarity of the two graphs based on the calculated features. Alternatively, the similarity calculation unit 192 may calculate the similarity between the two graphs using a kernel function, such as a graph kernel or a kernel method.

[0063] When environmental feature information is represented by a real vector (a vector whose elements are real numbers), a known index for the similarity of real vectors, such as cosine similarity or norm, may be used as the similarity measure of the environmental feature information. When environmental feature information is represented by an image, a known method for calculating image similarity may be used as the method for calculating the similarity of the environmental feature information.

[0064] The constraint selection unit 193 selects environmental constraint data that corresponds to the actual environment from the environmental constraint data stored in the environmental constraint data storage unit 181. Specifically, the constraint selection unit 193 selects environmental constraint data that corresponds to the actual environment based on the similarity between the actual environment feature information calculated by the similarity calculation unit 192 and the environmental feature information included in the environmental constraint data. The constraint selection unit 193 is an example of a constraint selection means.

[0065] The similarity calculation unit 192 may calculate the similarity between the environmental feature information contained in each of the environmental constraint data stored in the environmental constraint data storage unit 181 and the actual environmental feature information. The constraint selection unit 193 may then select the environmental constraint data stored in the environmental constraint data storage unit 181 that has the highest similarity calculated by the similarity calculation unit 192.

[0066] The control execution unit 194 performs control on the controlled object 900. Specifically, the control execution unit 194 controls the controlled object 900 to execute the target process based on the constraints indicated by the environment-responsive constraint data selected by the constraint selection unit 193. The control execution unit 194 generates a control command for the controlled object 900 and transmits the generated control command to the controlled object 900 via the communication unit 110, thereby performing control on the controlled object 900. The control execution unit 194 is an example of a control execution means.

[0067] For example, if the execution target process is for the controlled object 900 to reach a predetermined target location, an objective function indicating whether or not the execution target process has been achieved may be set in advance, such as a function that indicates the distance between the controlled object 900 and the target location. The control execution unit 194 may then search for a time series of control commands to the controlled object 900 that satisfies the constraints and for the objective function to be a value that indicates the execution target process has been achieved. The control execution unit 194 may then send control commands to the controlled object 900 according to the time series of control commands obtained, thereby causing the controlled object 900 to perform the execution target process.

[0068] Alternatively, the constraint selection unit 193 may select constraints that include constraints indicating the target state in the execution target process. For example, the constraints may be expressed in temporal logic, and the constraint that the controlled object 900 is located at the target location after a predetermined time has elapsed may be included in the constraints selected by the constraint selection unit 193. The control execution unit 194 may then search for a time series of control commands to the controlled object 900 that satisfy the constraints. The control execution unit 194 may then send control commands to the controlled object 900 according to the obtained time series of control commands, thereby causing the controlled object 900 to perform the execution target process.

[0069] Figure 8 shows an example of the data flow when the control device 100 controls the controlled object 900. In the example shown in Figure 8, the real-world feature information generation unit 191 generates real-world feature information based on observation data of the real environment from sensors that observe the real environment.

[0070] The similarity calculation unit 192 calculates the similarity between the environmental feature information contained in each of the environmental constraint data stored in the environmental constraint data storage unit 181 and the actual environmental feature information. The constraint selection unit 193 selects the environmental constraint data with the highest similarity calculated by the similarity calculation unit 192 from among the environmental constraint data stored in the environmental constraint data storage unit 181.

[0071] The control execution unit 194 generates a control command for the controlled object 900 based on the constraints indicated by the environmental constraint data selected by the constraint selection unit 193. The control execution unit 194 performs control on the controlled object 900 by transmitting the generated control command to the controlled object 900 via the communication unit 110.

[0072] Figure 9 shows an example of the procedure for the process by which the control device 100 controls the controlled object 900. In the process shown in Figure 9, the real-world feature information generation unit 191 acquires observational data of the real environment (step S101).

[0073] Then, the real-world feature information generation unit 191 generates real-world feature information based on the acquired observation data (step S102). Next, the similarity calculation unit 192 calculates the similarity between the environmental feature information contained in each of the environmental constraint data stored in the environmental constraint data storage unit 181 and the actual environmental feature information (step S103).

[0074] Next, the constraint selection unit 193 selects the environmental constraint data with the highest similarity calculated by the similarity calculation unit 192 from among the environmental constraint data stored in the environmental constraint data storage unit 181 (step S104). Then, the control execution unit 194 controls the controlled object 900 based on the constraints indicated by the environmental constraint data selected by the constraint selection unit 193 (step S105). Specifically, the control execution unit 194 generates a control command for the controlled object 900 based on the constraints and transmits the generated control command to the controlled object 900 via the communication unit 110. After step S105, the control device 100 terminates the process shown in Figure 9.

[0075] As described above, the constraint selection unit 193 selects environmental constraint data that corresponds to the actual environment, which is the execution environment of the process being executed, from among the environmental constraint data, which is linked to environmental characteristic information, which is information about the execution environment of the process, and the constraints for executing the process in that environment. The control execution unit 194 controls the control target 900 to execute the process being executed based on the constraints indicated by the selected environmental constraint data.

[0076] The control device 100 is expected to be able to handle not only differences in the execution environment of the process, but also differences in the process being executed, by acquiring constraints according to the actual environment. For example, a constraint that the controlled object 900 must avoid a predetermined obstacle when it moves can be applied to various processes that involve the movement of the controlled object 900.

[0077] Furthermore, the real-world feature information generation unit 191 generates real-world feature information, which is environmental feature information in the real environment, based on observation data of the real environment from sensors that observe the real environment. The similarity calculation unit 192 calculates the similarity of the environmental feature information. The constraint condition selection unit 193 selects environmental constraint condition data appropriate to the real environment based on the similarity between the real-world feature information and the environmental feature information included in the environmental constraint condition data.

[0078] The control device 100 is expected to be able to obtain constraints suitable for the actual environment by selecting environmental constraint data based on the similarity of environmental feature information. In this respect, the control device 100 is expected to be able to control the controlled object 900 with high precision.

[0079] Furthermore, environmental feature information is represented as a graph that includes nodes representing objects in the processing execution environment and edges representing the relationships between nodes. According to the control device 100, the relationship between objects is indicated by environmental feature information, which is expected to enable high-precision determination of the similarity of the processing execution environment, and thus enable the acquisition of constraints suitable for the real environment.

[0080] <Second Embodiment> In the second to fourth embodiments, we will describe an example of how the control device generates environmental constraint data when it determines that there is no environmental constraint data suitable for the actual environment.

[0081] In the second embodiment, an example is described in which the control device generates environmental constraint data in response to user input of constraints corresponding to the actual environment. In the third embodiment, an example is described in which the control device generates environmental constraint data in response to user operations on the controlled object 900. In the fourth embodiment, an example is described in which the control device generates environmental constraint data based on a simulation of the operation of the controlled object 900.

[0082] Figure 10 is a diagram showing an example of the configuration of a control device according to the second embodiment. In the configuration shown in Figure 10, the control device 200 comprises a communication unit 110, a display unit 120, an operation input unit 130, a storage unit 180, and a processing unit 290. The processing unit 290 comprises a real-world feature information generation unit 191, a similarity calculation unit 192, a constraint condition selection unit 193, a control execution unit 194, a support request unit 291, and an environment-compatible constraint condition data generation unit 292. In the second embodiment, control device 200 is used instead of control device 100 in Figure 1.

[0083] In Figure 10, parts that have the same function as the parts in Figure 1 are denoted by the same reference numerals (110, 120, 130, 180, 181, 191, 192, 193, 194), and detailed explanations are omitted here. The control device 200 differs from the control device 100 in that its processing unit 290 includes, in addition to the components of the processing unit 190, a support request unit 291 and an environmental constraint data generation unit 292. In all other respects, the control device 200 is the same as the control device 100.

[0084] The support request unit 291 requests user action if there is no environmental constraint data stored in the environmental constraint data storage unit 181 that contains environmental feature information whose similarity to the actual environmental feature information is greater than or equal to a predetermined threshold. The threshold here may be set in advance by the user and stored in the storage unit 180. The support request unit 291 is an example of a support request means.

[0085] For example, the support request unit 291 may control the display unit 120 to display a message indicating that it is not possible to plan the control for the controlled object 900 and that it is requesting assistance from the user. Alternatively, in the second embodiment, the support request unit 291 may control the display unit 120 to display a message requesting input of constraints for the operation of the controlled object 900.

[0086] If the environmental constraint data generation unit 292 does not have any environmental constraint data stored in the environmental constraint data storage unit 181 that includes environmental feature information whose similarity to the actual environment feature information is greater than a predetermined threshold, the environmental constraint data generation unit 292 generates environmental constraint data that links the actual environment feature information with the constraint conditions in the actual environment. The environmental constraint data generation unit 292 is an example of an environmental constraint data generation means.

[0087] In the second embodiment, the environmental constraint data generation unit 292 generates environmental constraint data by linking the real-environment characteristic information generated by the real-environment characteristic information generation unit 191 with the constraint conditions entered by the user. The user may also enter constraint conditions that include constraint conditions indicating the target state in the execution target process. In this case, the environmental constraint data generation unit 292 generates environmental constraint data in which the real-environment characteristic information and constraint conditions that include constraint conditions indicating the target state in the execution target process are linked.

[0088] Figure 11 shows an example of the data flow when the control device 200 generates new environmental constraint data. In the example shown in Figure 11, the data flow in the real-world feature information generation unit 191 and the similarity calculation unit 192 is the same as in Figure 8.

[0089] Specifically, the real-environment feature information generation unit 191 generates real-environment feature information based on observation data of the real environment from sensors that observe the real environment. The similarity calculation unit 192 calculates the similarity between the environmental feature information contained in each of the environmental constraint data stored in the environmental constraint data storage unit 181 and the real-environment feature information.

[0090] The constraint selection unit 193 determines whether there is any environmental constraint data that indicates the similarity calculated by the similarity calculation unit 192 is above a predetermined threshold. When the constraint selection unit 193 determines that there is any relevant environmental constraint data, the data flow in the constraint selection unit 193 and the control execution unit 194 is the same as described with reference to Figure 8, and therefore the illustration and explanation are omitted here.

[0091] On the other hand, if the similarity calculation unit 192 determines that there is no environmental constraint data that indicates the similarity is above a predetermined threshold, the constraint selection unit 193 outputs information to the support request unit 291 indicating that there is no corresponding environmental constraint data.

[0092] When the support request unit 291 receives input indicating that there is no corresponding environmental constraint data, it requests assistance from the user. For example, as described above, the support request unit 291 may control the display unit 120 to display that it is not possible to plan the control for the controlled object 900 and that it is requesting assistance from the user.

[0093] In the second embodiment, the user who receives the request for assistance inputs constraints to the control device 200. The control execution unit 194 generates a control command for the controlled object 900 to satisfy the constraints input by the user, and transmits the generated control command to the controlled object 900 via the communication unit 110.

[0094] Furthermore, the environmental constraint data generation unit 292 links the real-world feature information generated by the real-world feature information generation unit 191 with the constraint conditions entered by the user to generate environmental constraint data. The environmental constraint data generation unit 292 stores the generated environmental constraint data in the environmental constraint data storage unit 181. As a result, the environmental constraint data generated by the environmental constraint data generation unit 292 is added to the environmental constraint data stored in the environmental constraint data storage unit 181.

[0095] Figure 12 shows an example of the procedure for the process by which the control device 200 controls the controlled object 900. Steps S201 to S203 in Figure 12 are the same as steps S101 to S103 in Figure 9.

[0096] After step S203, the constraint selection unit 193 determines whether or not there is environmental constraint data that indicates the similarity calculated by the similarity calculation unit 192 is similar to a predetermined threshold (step S204). If the constraint selection unit 193 determines that there is corresponding environmental constraint data (step S204: YES), the process proceeds to step S211. Steps S211 to S212 are the same as steps S104 to S105 in Figure 9. After step S212, the control device 200 terminates the process shown in Figure 12.

[0097] On the other hand, if the constraint selection unit 193 determines in step S204 that there is no corresponding environmental constraint data (step S204: NO), the support request unit 291 requests support from the user (step S221). Then, the control device 100 acquires the constraint conditions input by the user (step S222). For example, the operation input unit 130 receives a user operation to input constraint conditions.

[0098] Next, the control execution unit 194 controls the controlled object 900 to satisfy the constraints entered by the user (step S223). Furthermore, the environmental constraint data generation unit 292 links the real-world feature information generated by the real-world feature information generation unit 191 with the constraint conditions entered by the user to generate environmental constraint data, and stores the generated environmental constraint data in the environmental constraint data storage unit 181 (step S224). After step S224, the control device 200 terminates the process shown in Figure 12.

[0099] As described above, if there is no environmental constraint data that includes environmental feature information whose similarity to the actual environment feature information is greater than a predetermined threshold, the environmental constraint data generation unit 292 generates environmental constraint data that links the actual environment feature information with the constraints in the actual environment. According to the control device 200, it is expected that if there is no environmental constraint data suitable for the actual environment, it will be possible to add environmental constraint data suitable for the actual environment.

[0100] Furthermore, the support request unit 291 requests user input if there is no environmental constraint data containing environmental feature information whose similarity to the actual environment feature information is greater than a predetermined threshold. The environmental constraint data generation unit 292 generates environmental constraint data that links the actual environment feature information with the constraints in the actual environment based on the user input. According to the control device 200, by generating environmental constraint data based on user operations, it is expected that environmental constraint data suitable for the actual environment can be added.

[0101] Furthermore, the environmental constraint data generation unit 292 generates environmental constraint data that links real-world characteristic information with constraints entered by the user. According to the control device 200, it is expected that environmentally responsive constraint data suitable for the actual environment can be added by generating environmentally responsive constraint data using constraint conditions input by the user.

[0102] <Third Embodiment> Figure 13 is a diagram showing an example of the configuration of a control device according to the third embodiment. In the configuration shown in Figure 13, the control device 300 comprises a communication unit 110, a display unit 120, an operation input unit 130, a storage unit 180, and a processing unit 390. The processing unit 390 comprises a real-world feature information generation unit 191, a similarity calculation unit 192, a constraint condition selection unit 193, a control execution unit 194, a support request unit 291, an environment-compatible constraint condition data generation unit 292, and a constraint condition learning unit 391. In the third embodiment, control device 300 is used instead of control device 100 in Figure 1.

[0103] In Figure 13, parts that have the same function as those in Figure 10 are denoted by the same reference numerals (110, 120, 130, 180, 181, 191, 192, 193, 194, 291, 292), and detailed explanations are omitted here. The control device 300 differs from the control device 200 in that its processing unit 390 includes a constraint learning unit 391 in addition to the components of the processing unit 290. In all other respects, the control device 300 is the same as the control device 200.

[0104] The constraint learning unit 391 learns the constraints based on the time series of the operation of the controlled object 900. In the third embodiment, the constraint learning unit 391 learns the constraints based on the time series of the operation of the controlled object 900 in response to user operations. In the third embodiment, it can be said that the constraint learning unit 391 learns the control conditions based on user operations that operate the controlled object 900.

[0105] For example, the memory unit 180 stores constraint conditions, including parameters, in advance. Constraint conditions, including parameters, are also called constraint template conditions. Then, the constraint learning unit 391 determines the parameter values ​​of the constraint template based on the time series of the operation of the controlled object 900.

[0106] Furthermore, for example, the memory unit 180 stores multiple constraint template templates. Also, the control device 300 acquires multiple time series of the operation of the controlled object 900. The constraint learning unit 391 then adopts a constraint template that contains parameter values ​​such that the constraints are satisfied for all time series when the target process is successful. On the other hand, the constraint learning unit 391 rejects constraint templates that do not contain parameter values ​​such that the constraints are satisfied for all time series when the target process is successful. In this context, "successful execution" refers to the successful completion of the target process. "Failure to complete the target process" is also referred to as "failure of the target process."

[0107] Alternatively, the constraint learning unit 391 may decide whether to adopt a constraint template based on whether, in the time series of the operation of the controlled object 900 when the execution target process is successful, parameter values ​​that satisfy the constraint exist at a rate higher than a predetermined rate. For example, the constraint learning unit 391 may adopt a constraint template in which parameter values ​​satisfy the constraint for a predetermined rate or higher of the time series in the time series when the execution target process is successful. On the other hand, the constraint learning unit 391 may reject constraint templates that do not satisfy the condition that parameter values ​​satisfy the constraint exist for a predetermined rate or higher of the time series in the time series when the execution target process is successful.

[0108] Here, we consider the case where the control device 300 can obtain both the time series of the operation of the controlled object 900 when the execution target process is successful and the time series of the operation of the controlled object 900 when the execution target process fails. In the third embodiment, the case where the user performs both operations that result in the execution target process being successful and operations that result in the execution target process being unsuccessful corresponds to the case where the control device 300 can obtain both the time series of the operation of the controlled object 900 when the execution target process is successful and the time series of the operation of the controlled object 900 when the execution target process is unsuccessful.

[0109] In this case, the constraint learning unit 391 searches for and sets a combination of parameter values ​​for all combinations of parameters in the adopted constraint template such that the constraint is satisfied if the execution target process is successful, and the constraint is not satisfied if the execution target process fails.

[0110] In this context, "the constraint is satisfied if the execution target process is successful" means that, in the case of a successful execution target process, the constraint is satisfied for all constraint template values ​​in the time series of the operation of the controlled object 900. Conversely, "the constraint is not satisfied if the execution target process fails" means that, in the case of a failed execution target process, the constraint is not satisfied for at least one constraint template value in the time series of the operation of the controlled object 900.

[0111] If a combination of parameter values ​​cannot be obtained such that the constraint is satisfied if the execution target process is successful and unsatisfied if the execution target process fails, the constraint learning unit 391 may change one or more of the adopted constraint templates to unadopted.

[0112] Here too, the condition that "the constraint is satisfied if the execution target process is successful" may be relaxed to "the constraint is satisfied at a rate higher than a predetermined rate if the execution target process is successful." For example, the constraint learning unit 391 may search for and set a combination of parameter values ​​for all combinations of parameters of the adopted constraint template such that the constraint is satisfied for a predetermined percentage or more of the time series when the execution target process is successful, and the constraint is not satisfied for all time series when the execution target process fails. Furthermore, if such a combination of parameter values ​​cannot be obtained, the constraint learning unit 391 may change one or more of the adopted constraint templates to not be adopted.

[0113] Next, let's consider the case where the control device 300 can obtain a time series of the operation of the controlled object 900 when the execution target process is successful, but cannot obtain a time series of the operation of the controlled object 900 when the execution target process fails. In this case, the constraint learning unit 391 searches for and sets a combination of parameter values ​​for all parameter combinations of the adopted constraint template such that the constraints are satisfied for all constraint templates in the time series of the operation of the controlled object 900 when the execution target process is successful.

[0114] The constraint learning unit 391 may set combinations of parameter values ​​so that the constraints become as strict as possible. For example, with respect to the parameter indicating the radius of the area that the controlled object 900 is not allowed to enter, the constraint learning unit 391 may set the parameter value to the largest radius among those radii that satisfy the constraints for the time series of the controlled object 900's actions when the execution target process is successful.

[0115] Furthermore, if the execution target process is successful, and a combination of parameter values ​​is not obtained that satisfies the constraints for all constraint templates in the time series of the operation of the controlled object 900, the constraint learning unit 391 may change one or more of the adopted constraint templates to not be adopted.

[0116] In this case as well, the condition that all time series occur when the target process is successful may be relaxed to the condition that a certain percentage or higher of the time series occur when the target process is successful. For example, the constraint learning unit 391 may search for and set combinations of parameter values ​​for all combinations of parameters of the adopted constraint template such that the constraint is satisfied for a certain percentage or higher of the time series of the operation of the controlled object 900 when the target process is successful.

[0117] In this case, setting a combination of parameter values ​​to make the constraints as strict as possible means setting a combination of parameter values ​​such that the constraints are met for time series selected as a predetermined proportion or more, and that the constraints are made as strict as possible. For example, with respect to the parameter indicating the radius of the area that the controlled object 900 is not allowed to enter, the constraint learning unit 391 may set the parameter value to the largest radius among the radii that satisfy the constraints for time series selected as a predetermined proportion or more.

[0118] However, the method by which the constraint learning unit 391 learns the constraints is not limited to a specific method, and can be any of the various methods that can obtain constraints that are consistent with the time series of the operation of the controlled object 900 and the success or failure of the target processing to be executed in that time series.

[0119] Figure 14 shows an example of the data flow when the control device 300 generates new environmental constraint data. In the example shown in Figure 14, the data flow in the real-world feature information generation unit 191, the similarity calculation unit 192, and the constraint selection unit 193 is the same as in Figure 11.

[0120] Specifically, the real-environment feature information generation unit 191 generates real-environment feature information based on observation data of the real environment from sensors that observe the real environment. The similarity calculation unit 192 calculates the similarity between the environmental feature information contained in each of the environmental constraint data stored in the environmental constraint data storage unit 181 and the real-environment feature information.

[0121] The constraint selection unit 193 determines whether there is any environmental constraint data that indicates the similarity calculated by the similarity calculation unit 192 is above a predetermined threshold. When the constraint selection unit 193 determines that there is any relevant environmental constraint data, the data flow in the constraint selection unit 193 and the control execution unit 194 is the same as described with reference to Figure 8, and therefore the illustration and explanation are omitted here.

[0122] On the other hand, if the similarity calculation unit 192 determines that there is no environmental constraint data that indicates the similarity is above a predetermined threshold, the constraint selection unit 193 outputs information to the support request unit 291 indicating that there is no corresponding environmental constraint data.

[0123] Furthermore, as explained with reference to Figure 11, when the support request unit 291 receives input indicating that there is no corresponding environmental constraint data, it requests assistance from the user. For example, the support request unit 291 may control the display unit 120 to display that it is not possible to formulate a control plan for the controlled object 900 and that it requests assistance from the user.

[0124] In the third embodiment, the user who receives a request for assistance performs an operation on the controlled object 900. For example, the assistance request unit 291 may control the display unit 120 to display a message requesting that the user perform an operation on the controlled object 900. Alternatively, the operation input unit 130 may accept user operations on the controlled object 900. For example, the user may input control commands for the controlled object 900 using the keyboard provided by the operation input unit 130. Or, the operation input unit 130 may be equipped with an operating device for operating the controlled object 900, such as a joystick, to accept user operations.

[0125] The control execution unit 194 generates control commands for the controlled object 900 in accordance with user operations on the controlled object 900 and transmits them to the controlled object 900 via the communication unit 110. This causes the control execution unit 194 to operate according to user operations.

[0126] Furthermore, the constraint learning unit 391 acquires observational data of the real environment while the user is performing operations on the controlled object 900. The observational data of the real environment is assumed to include data indicating the operation of the controlled object 900, such as the position of the controlled object 900. The constraint learning unit 391 learns the constraints in the real environment based on observational data of the real environment and outputs the obtained constraints to the environment-responsive constraint data generation unit 292.

[0127] The environmental constraint data generation unit 292 generates environmental constraint data by linking the real-world feature information generated by the real-world feature information generation unit 191 with the constraints obtained through learning by the constraint learning unit 391. The environmental constraint data generation unit 292 stores the generated environmental constraint data in the environmental constraint data storage unit 181. As a result, the environmental constraint data generated by the environmental constraint data generation unit 292 is added to the environmental constraint data stored in the environmental constraint data storage unit 181.

[0128] Figure 15 shows an example of the procedure for the process by which the control device 300 controls the controlled object 900. Steps S301 to S304 in Figure 15 are the same as steps S201 to S204 in Figure 12.

[0129] If the constraint selection unit 193 determines in step S304 that there is corresponding environmental constraint data (step S304: YES), the process proceeds to step S311. Steps S311 to S312 are the same as steps S211 to S212 in Figure 12.

[0130] On the other hand, if the constraint selection unit 193 determines in step S304 that there is no corresponding environmental constraint data (step S304: NO), the support request unit 291 requests support from the user (step S321). Then, the control device 100 controls the controlled object 900 based on the operation performed by the user on the controlled object 900 (step S322).

[0131] Next, the constraint learning unit 391 learns the constraints based on the time series of the actions of the controlled object 900 while the user is performing operations on the controlled object 900 (step S323). The environmental constraint data generation unit 292 links the real-world feature information generated by the real-world feature information generation unit 191 with the constraints obtained through learning by the constraint learning unit 391 to generate environmental constraint data, and stores the generated environmental constraint data in the environmental constraint data storage unit 181 (step S324). After step S324, the control device 300 terminates the process shown in Figure 15.

[0132] As described above, the constraint learning unit 391 learns constraints based on user operations that control the controlled object 900. The environment-responsive constraint data generation unit 292 generates environment-responsive constraint data in which real-world feature data and the constraints obtained through learning are linked. According to the control device 300, by learning constraint conditions based on user operations that operate the controlled object 900, it is expected that the control device 300 can learn the constraint conditions for successful execution of the target process.

[0133] <Fourth Embodiment> Figure 16 is a diagram showing an example of the configuration of a control device according to the fourth embodiment. In the configuration shown in Figure 16, the control device 400 comprises a communication unit 110, a display unit 120, an operation input unit 130, a storage unit 180, and a processing unit 490. The processing unit 490 comprises a real-world feature information generation unit 191, a similarity calculation unit 192, a constraint condition selection unit 193, a control execution unit 194, an environment-compatible constraint condition data generation unit 292, a constraint condition learning unit 391, and a simulation processing unit 491. In the fourth embodiment, control device 400 is used instead of control device 100 in Figure 1.

[0134] Of the parts in Figure 16, those that have the same function as the parts in Figure 13 are denoted by the same reference numerals (110, 120, 130, 180, 181, 191, 192, 193, 194, 292, 391), and detailed explanations are omitted here. The control device 400 differs from the control device 300 in that the processing unit 490 does not include the support request unit 291, which is one of the components of the processing unit 390, but does include the simulation processing unit 491. In all other respects, the control device 400 is the same as the control device 300.

[0135] In the fourth embodiment, the constraint learning unit 391 learns the constraints based on the simulation results of the execution target process in an environment that simulates the actual environment, instead of the user's operation on the controlled object 900.

[0136] The simulation processing unit 491 performs a simulation of the operation of the controlled object 900. In particular, the simulation processing unit 491 receives a setting for an environment that simulates the actual environment and performs a simulation of the operation of the controlled object 900 in the set environment to perform the target processing. In this case, the simulation corresponds to a simulation of the processing to be executed in an environment that simulates the actual environment. The control device 400 is an example of a data generation device.

[0137] Figure 17 shows an example of the data flow when the control device 400 generates new environmental constraint data. In the example shown in Figure 17, the data flow in the real-world feature information generation unit 191, the similarity calculation unit 192, and the constraint selection unit 193 is the same as in Figure 11.

[0138] Specifically, the real-environment feature information generation unit 191 generates real-environment feature information based on observation data of the real environment from sensors that observe the real environment. The similarity calculation unit 192 calculates the similarity between the environmental feature information contained in each of the environmental constraint data stored in the environmental constraint data storage unit 181 and the real-environment feature information.

[0139] The constraint selection unit 193 determines whether there is any environmental constraint data that indicates the similarity calculated by the similarity calculation unit 192 is above a predetermined threshold. When the constraint selection unit 193 determines that there is any relevant environmental constraint data, the data flow in the constraint selection unit 193 and the control execution unit 194 is the same as described with reference to Figure 8, and therefore the illustration and explanation are omitted here.

[0140] On the other hand, if the similarity calculation unit 192 determines that there is no environmental constraint data that indicates the similarity is above a predetermined threshold, the constraint selection unit 193 outputs information to the simulation processing unit 491 indicating that there is no corresponding environmental constraint data.

[0141] When the simulation processing unit 491 receives information indicating that there is no corresponding environmental constraint data, it performs a simulation of the process to be executed in an environment that simulates the actual environment, and outputs the simulation results to the constraint learning unit 391.

[0142] The constraint learning unit 391 learns the constraints based on the simulation results from the simulation processing unit 491. In the third embodiment, the time series of the operation of the controlled object 900 by user operation is replaced with the time series of the operation of the controlled object 900 in the simulation performed by the simulation processing unit 491. The constraint learning unit 391 outputs the constraints obtained through learning to the environment-responsive constraint data generation unit 292.

[0143] The data flow in the environmental constraint data generation unit 292 is the same as in the case of Figure 14. Specifically, the environmental constraint data generation unit 292 generates environmental constraint data by linking the real-world feature information generated by the real-world feature information generation unit 191 with the constraints obtained through learning by the constraint learning unit 391. The environmental constraint data generation unit 292 stores the generated environmental constraint data in the environmental constraint data storage unit 181. As a result, the environmental constraint data generated by the environmental constraint data generation unit 292 is added to the environmental constraint data stored in the environmental constraint data storage unit 181.

[0144] In this case, the similarity calculated by the similarity calculation unit 192 for the added environmental constraint data is expected to indicate that the environmental feature information included in the environmental constraint data and the actual environmental feature information are similar to a threshold or higher. The constraint selection unit 193 selects the constraints included in the added environmental constraint data, and the control execution unit 194 performs control on the controlled object 900 based on the selected constraints, which is expected to result in the successful execution of the target process.

[0145] Figure 18 shows an example of the procedure for the process by which the control device 400 controls the controlled object 900. Steps S401 to S404 in Figure 18 are the same as steps S201 to S204 in Figure 12.

[0146] If the constraint selection unit 193 determines in step S404 that there is corresponding environmental constraint data (step S404: YES), the process proceeds to step S411. Steps S411 to S412 are the same as steps S211 to S212 in Figure 12.

[0147] On the other hand, if the constraint selection unit 193 determines in step S404 that there is no corresponding environmental constraint data (step S404: NO), the simulation processing unit 491 performs a simulation of the process to be executed in an environment that simulates the actual environment (step S421).

[0148] Then, the constraint learning unit 391 learns the constraints based on the time series of the operation of the controlled object 900 in the simulation (step S422). The environmental constraint data generation unit 292 links the real-world feature information generated by the real-world feature information generation unit 191 with the constraints obtained through learning by the constraint learning unit 391 to generate environmental constraint data, and stores the generated environmental constraint data in the environmental constraint data storage unit 181 (step S423).

[0149] Furthermore, the constraint selection unit 193 selects newly generated environmental constraint data, and the control execution unit 194 controls the controlled object 900 based on the constraints included in the newly generated environmental constraint data (step S424). After step S424, the control device 400 terminates the process shown in Figure 18.

[0150] Figure 19 shows an example of a process performed by the controlled object 900 in accordance with the control of the control device 400. Figure 19 shows an example of a process in which items in basket C11 are transferred to basket C12. Here, we assume that the constraint learning unit 391 has obtained constraints including (1) not to place heavy objects on soft objects, and (2) to place objects in a stable orientation.

[0151] Line L11 indicates moving the bread out of basket C11. Line L12 indicates moving the candy from basket C11 to basket C12. When the controlled object 900 moves the candy, if the candy is placed upright, its base area is small and it is likely to tip over, so the controlled object 900 places the candy on its side.

[0152] Line L13 indicates transferring the milk from basket C11 to basket C12. When the controlled object 900 moves the milk, if the milk is placed on its side, part of the milk will rest on top of the candy and become unstable, so the controlled object 900 places the milk upright. Line L14 indicates moving the bread into basket C12. The controlled object 900 is placing the soft bread on top of the hard candy.

[0153] In this way, by the control device 400 controlling the controlled object 900 based on constraints, it is expected that the controlled object 900 will be able to process according to the control of the control device 400 even if the type of item, the arrangement of the items, or the position of the basket changes. Furthermore, if the controlled object 900 fails to process, the constraint learning unit 391 is expected to learn the constraints based on the simulation results from the simulation processing unit 491, thereby enabling the controlled object 900 to succeed in processing. Furthermore, when introducing the control system 1, it is expected that the control device 400 will be able to perform processing immediately after the introduction of the control system 1 by learning the constraint conditions in advance through simulation.

[0154] As described above, the constraint learning unit 391 learns constraints based on the simulation results of the process to be executed in an environment that simulates the real environment. The environment-responsive constraint data generation unit 292 generates environment-responsive constraint data in which the real environment feature data and the constraints obtained through learning are linked.

[0155] In the control device 400, it is expected that the constraint learning unit 391 will learn the constraints, thereby enabling the previously failed execution target process to succeed. Furthermore, in the control device 400, the constraint learning unit 391 learns the constraints based on the simulation results, which means that the constraints can be learned before the introduction of the control system 1, and it is expected that processing can be performed immediately after the introduction of the control system 1.

[0156] <Fifth Embodiment> Figure 20 shows an example of the configuration of a control device according to the fifth embodiment. In the configuration shown in Figure 20, the control device 610 includes a constraint selection unit 611 and a control execution unit 612.

[0157] In this configuration, the constraint selection unit 611 selects environmental constraint data that corresponds to the actual environment, which is the execution environment of the process being executed, from among the environmental constraint data, which is linked to environmental characteristic information, which is information about the execution environment of the process, and the constraints for executing the process in that environment. The control execution unit 612 controls the controlled object to execute the process being executed based on the constraints indicated by the selected environmental constraint data. The constraint selection unit 611 is an example of a constraint selection means. The control execution unit 612 is an example of a control execution means.

[0158] The control device 610 is expected to be able to handle not only differences in the execution environment of the process, but also differences in the process being executed, by acquiring constraints according to the actual environment. For example, a constraint that the controlled object must avoid a predetermined obstacle when it moves can be applied to various processes that involve the movement of the controlled object.

[0159] The constraint selection unit 611 can be implemented using functions such as the constraint selection unit 193 shown in Figure 2. The control execution unit 612 can be implemented using functions such as the control execution unit 194 shown in Figure 2.

[0160] <Sixth Embodiment> Figure 21 is a diagram showing an example of the configuration of a constraint selection device according to the sixth embodiment. In the configuration shown in Figure 21, the constraint selection device 620 includes a constraint selection unit 621. In this configuration, the constraint selection unit 621 selects environmental constraint data that corresponds to the actual environment, which is the execution environment of the process being executed, from among the environmental constraint data, which is linked to environmental characteristic information, which is information about the execution environment of the process, and constraints for the execution of the process in that environment. The constraint selection unit 621 is an example of a constraint selection means.

[0161] By using the constraints selected by the constraint selection device 620, it is expected that not only differences in the execution environment of the process but also differences in the process being executed can be accommodated. For example, a constraint that the controlled object must avoid a predetermined obstacle when it moves can be applied to various processes that involve the movement of the controlled object. The constraint selection unit 621 can be implemented using functions such as the constraint selection unit 193 shown in Figure 2.

[0162] <Seventh Embodiment> Figure 22 shows an example of the configuration of a data generation device according to the seventh embodiment. In the configuration shown in Figure 22, the data generation device 630 includes a constraint learning unit 631 and an environment-responsive constraint data generation unit 632.

[0163] In this configuration, the constraint learning unit 631 learns constraints based on the simulation results of the process. The environment-responsive constraint data generation unit 632 generates environment-responsive constraint data in which environmental feature information, which is information about the environment, and constraints are linked by linking the environmental information in the simulation of the process with the constraints obtained through learning. The constraint learning unit 631 is an example of a constraint learning means. The environmental constraint data generation unit 632 is an example of an environmental constraint data generation means.

[0164] By having the data generation device 630 learn the constraints, it is expected that the obtained constraints can be used to handle not only differences in the execution environment of the process, but also differences in the process being executed. For example, a constraint that the controlled object must avoid a predetermined obstacle when it moves can be applied to various processes that involve the movement of the controlled object.

[0165] The constraint learning unit 631 can be implemented using, for example, the functions of the constraint learning unit 391 shown in Figure 16. The environmental constraint data generation unit 632 can be implemented using, for example, the functions of the environmental constraint data generation unit 292 shown in Figure 16.

[0166] <Eighth Embodiment> Figure 23 is a diagram showing an example of the processing procedure in the control method according to the eighth embodiment. The control method shown in Figure 23 includes selecting constraint conditions (step S611) and executing control (step S612).

[0167] In selecting constraints (step S611), the computer selects environmental constraint data that corresponds to the actual environment, which is the execution environment for the process being executed, from among the environmental constraint data, which is linked to environmental characteristic information, which is information about the execution environment of the process, and constraints for the execution of the process in that environment. In the control execution step (step S612), the computer controls the controlled object to execute the process to be executed based on the constraints indicated by the selected environment-aware constraint data.

[0168] According to the control method shown in Figure 23, by acquiring constraints that correspond to the actual environment, it is expected that the method can handle not only differences in the execution environment of the process but also differences in the process being executed. For example, a constraint that the controlled object must avoid a predetermined obstacle when it moves can be applied to various processes that involve the movement of the controlled object.

[0169] <Ninth Embodiment> Figure 24 shows an example of the processing procedure in the constraint selection method according to the ninth embodiment. The constraint selection method shown in Figure 24 includes selecting constraints (step S621).

[0170] In selecting constraints (step S621), the computer selects environmental constraint data that corresponds to the actual environment, which is the execution environment for the process being executed, from among the environmental constraint data, which is linked to environmental characteristic information, which is information about the execution environment of the process, and constraints for the execution of the process in that environment.

[0171] According to the constraint selection method shown in Figure 24, it is expected that by using the selected constraints, it will be possible to accommodate not only differences in the execution environment of the process but also differences in the process being executed. For example, a constraint that the controlled object must avoid a predetermined obstacle when it moves can be applied to various processes that involve the movement of the controlled object.

[0172] <Tenth Embodiment> Figure 25 shows an example of the processing steps in the data generation method according to the 10th embodiment. The data generation method shown in Figure 25 includes learning constraint conditions (step S631) and generating environmental constraint condition data (step S632).

[0173] In learning the constraints (step S631), the computer learns the constraints based on the simulation results of the process. In generating environmental constraint data (step S632), environmental constraint data is generated by linking environmental information in the simulation of the process with the constraints obtained through learning, thereby linking environmental feature information, which is information about the environment, with the constraints.

[0174] According to the data generation method shown in Figure 25, by learning the constraints, it is expected that the obtained constraints can be used to handle not only differences in the execution environment of the process, but also differences in the process being executed. For example, a constraint that the controlled object must avoid a predetermined obstacle when it moves can be applied to various processes that involve the movement of the controlled object.

[0175] Figure 26 is a schematic block diagram showing the configuration of a computer according to at least one embodiment. As shown in Figure 26, the computer 700 comprises a CPU 710, a main memory 720, an auxiliary memory 730, an interface 740, and a non-volatile recording medium 750.

[0176] One or more of the above-mentioned control devices 100, 200, 300, 400, 610, constraint selection device 620, and data generation device 630, or a part thereof, may be implemented in the computer 700. In that case, the operation of each of the above-mentioned processing units is stored in the auxiliary storage device 730 in the form of a program. The CPU 710 reads the program from the auxiliary storage device 730, expands it into the main memory 720, and executes the above-mentioned processing according to the program. The CPU 710 also reserves memory areas in the main memory 720 corresponding to each of the above-mentioned storage units according to the program. Communication between each device and other devices is performed by the interface 740 having a communication function and communicating according to the control of the CPU 710.

[0177] When the control device 100 is implemented in the computer 700, the operation of the processing unit 190 and each of its parts is stored in auxiliary storage device 730 in the form of a program. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main memory device 720, and executes the above processing according to the program.

[0178] Furthermore, the CPU 710 allocates the storage area of ​​the storage unit 180 and each of its parts in the main memory 720 according to the program. Communication with other devices by the communication unit 110 is performed by the interface 740 having a communication function and operating under the control of the CPU 710. Display of various images by the display unit 120 is performed by the interface 740 having a display device and displaying various images under the control of the CPU 710. Reception of user operations by the operation input unit 130 is performed by the interface 740 having an input device and receiving user operations under the control of the CPU 710.

[0179] When the control device 200 is implemented in the computer 700, the operation of the processing unit 290 and each of its parts is stored in auxiliary storage device 730 in the form of a program. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main memory device 720, and executes the above processing according to the program.

[0180] Furthermore, the CPU 710 allocates the storage area of ​​the storage unit 180 and each of its parts in the main memory 720 according to the program. Communication with other devices by the communication unit 110 is performed by the interface 740 having a communication function and operating under the control of the CPU 710. Display of various images by the display unit 120 is performed by the interface 740 having a display device and displaying various images under the control of the CPU 710. Reception of user operations by the operation input unit 130 is performed by the interface 740 having an input device and receiving user operations under the control of the CPU 710.

[0181] When the control device 300 is implemented in the computer 700, the operation of the processing unit 390 and each of its parts is stored in auxiliary storage device 730 in the form of a program. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main memory 720, and executes the above processing according to the program.

[0182] Furthermore, the CPU 710 allocates the storage area of ​​the storage unit 180 and each of its parts in the main memory 720 according to the program. Communication with other devices by the communication unit 110 is performed by the interface 740 having a communication function and operating under the control of the CPU 710. Display of various images by the display unit 120 is performed by the interface 740 having a display device and displaying various images under the control of the CPU 710. Reception of user operations by the operation input unit 130 is performed by the interface 740 having an input device and receiving user operations under the control of the CPU 710.

[0183] When the control device 400 is implemented in the computer 700, the operation of the processing unit 490 and each of its parts is stored in auxiliary storage device 730 in the form of a program. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main memory 720, and executes the above processing according to the program.

[0184] Furthermore, the CPU 710 allocates the storage area of ​​the storage unit 180 and each of its parts in the main memory 720 according to the program. Communication with other devices by the communication unit 110 is performed by the interface 740 having a communication function and operating under the control of the CPU 710. Display of various images by the display unit 120 is performed by the interface 740 having a display device and displaying various images under the control of the CPU 710. Reception of user operations by the operation input unit 130 is performed by the interface 740 having an input device and receiving user operations under the control of the CPU 710.

[0185] When the control device 610 is implemented in the computer 700, the operations of the constraint selection unit 611 and the control execution unit 612 are stored in the auxiliary storage device 730 in the form of a program. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main memory 720, and executes the above process according to the program.

[0186] Furthermore, the CPU 710 reserves memory in the main memory 720 for the control device 610 to process according to the program. Communication between the control device 610 and other devices is performed by the interface 740 having a communication function and operating under the control of the CPU 710. Interaction between the control device 610 and the user is performed by the interface 740 being equipped with a display device and an input device, displaying various images under the control of the CPU 710, and accepting user operations.

[0187] When the constraint selection device 620 is implemented in the computer 700, the operation of the constraint selection unit 621 is stored in auxiliary storage device 730 in the form of a program. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main memory 720, and executes the above process according to the program.

[0188] Furthermore, the CPU 710 reserves memory in the main memory 720 for the constraint selection device 620 to process according to the program. Communication between the constraint selection device 620 and other devices is performed by the interface 740 having a communication function and operating under the control of the CPU 710. Interaction between the constraint selection device 620 and the user is performed by the interface 740 equipped with a display device and an input device, displaying various images under the control of the CPU 710 and accepting user operations.

[0189] When the data generation device 630 is implemented in the computer 700, the operations of the constraint learning unit 631 and the environment-responsive constraint data generation unit 632 are stored in auxiliary storage device 730 in the form of a program. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main memory 720, and executes the above processing according to the program.

[0190] Furthermore, the CPU 710 reserves memory in the main memory 720 for processing by the data generation device 630 according to the program. Communication between the data generation device 630 and other devices is performed by the interface 740 having a communication function and operating under the control of the CPU 710. Interaction between the data generation device 630 and the user is performed by the interface 740 equipped with a display device and an input device, displaying various images under the control of the CPU 710 and accepting user operations.

[0191] One or more of the above-mentioned programs may be recorded on the non-volatile recording medium 750. In this case, the interface 740 may read the program from the non-volatile recording medium 750. The CPU 710 may then either directly execute the program read by the interface 740, or temporarily save it in the main memory 720 or auxiliary memory 730 before executing it.

[0192] Alternatively, a program for executing all or part of the processing performed by the control device 100, control device 200, control device 300, control device 400, control device 610, constraint selection device 620, and data generation device 630 may be recorded on a computer-readable recording medium, and the processing of each part may be performed by having the computer system read and execute the program recorded on this recording medium. The term "computer system" here includes hardware such as the operating system and peripheral devices. Furthermore, "computer-readable recording media" refers to portable media such as flexible disks, magneto-optical disks, ROMs (Read Only Memory), CD-ROMs (Compact Disc Read Only Memory), and storage devices such as hard disks built into computer systems. The above-mentioned program may be intended to implement only a part of the functions described above, and may also be able to implement the above-mentioned functions in combination with programs already recorded in the computer system.

[0193] Although embodiments of this invention have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments and includes designs and the like that do not depart from the spirit of this invention.

[0194] Some or all of the above embodiments may also be described as follows, but are not limited to the following:

[0195] (Note 1) A constraint selection means selects environment-compatible constraint data from environment-compatible constraint data, which is a collection of environment-compatible constraint data that links environmental characteristic information, which is information about the execution environment of the process, with constraints on the execution of the process in that environment, and which corresponds to the actual environment that is the execution environment of the process being executed. A control execution means that controls the controlled object to execute the process to be executed based on the constraints indicated by the selected environmental constraint data, A control device equipped with the following features.

[0196] (Note 2) A real-environment feature information generation means generates real-environment feature information, which is environmental feature information in the real environment, based on observation data of the real environment by a sensor that observes the real environment. A similarity calculation means for calculating the similarity of environmental feature information, Furthermore, The constraint selection means selects environmental constraint data appropriate to the actual environment based on the similarity between the actual environment feature information and the environmental feature information included in the environmental constraint data. The control device described in Appendix 1.

[0197] (Note 3) The aforementioned environmental characteristic information is represented in a graph that includes nodes representing objects in the processing execution environment and edges representing the relationships between nodes. The control device described in Appendix 2.

[0198] (Note 4) If there is no environmental constraint data containing environmental feature information whose similarity to the actual environmental feature information is greater than a predetermined threshold, an environmental constraint data generation means generates environmental constraint data that links the actual environmental feature information with the constraints in the actual environment. A control device further comprising any one of the appendices 1 to 3.

[0199] (Note 5) If there is no environmental constraint data containing environmental feature information whose similarity to the actual environmental feature information is greater than a predetermined threshold, the system further provides a support request means for requesting user action. The aforementioned environmental constraint data generation means generates environmental constraint data that links real-world characteristic information with constraints in the real environment, based on user operations. The control device described in Appendix 4.

[0200] (Note 6) The aforementioned environmental constraint data generation means generates environmental constraint data in which real-world characteristic information and constraint conditions entered by user operations are linked. The control device described in Appendix 5.

[0201] (Note 7) A constraint learning means that learns constraint conditions based on user operations that manipulate the controlled object. Furthermore, The aforementioned environmental constraint data generation means generates environmental constraint data in which real-world feature data and constraints obtained through learning are linked. The control device described in Appendix 5.

[0202] (Note 8) A constraint learning method that learns constraints based on the simulation results of the process being executed in an environment that simulates a real environment. Furthermore, The aforementioned environmental constraint data generation means generates environmental constraint data in which real-world feature data and constraints obtained through learning are linked. The control device described in Appendix 4.

[0203] (Note 9) A constraint selection means selects environmental constraint data that corresponds to the actual environment, which is the execution environment of the process being executed, from among environmental constraint data, which is a data linking environmental characteristic information, which is information about the execution environment of the process, with constraints on the execution of the process in that environment. A constraint selection device equipped with the following features.

[0204] (Note 10) A constraint learning means that learns constraint conditions based on the simulation results of the process, An environmental constraint data generation means generates environmental constraint data in which environmental feature information, which is information about the environment, and constraints are linked by linking environmental information in the simulation of the process with constraints obtained through learning. A data generation device equipped with the following features.

[0205] (Note 11) Computers From the environmental constraint data, which links environmental characteristic information (information about the execution environment of the process) with constraints on the execution of the process in that environment, select the environmental constraint data that corresponds to the actual environment, which is the execution environment of the process being executed. Based on the constraints indicated by the selected environmental constraint data, the controlled object is controlled to execute the target process. A control method that includes the following.

[0206] (Note 12) Computers From the environmental constraint data, which links environmental characteristic information (information about the execution environment of the process) with constraints on the execution of the process in that environment, select the environmental constraint data that corresponds to the actual environment in which the process is being executed. A method for selecting constraints that includes the following.

[0207] (Note 13) Computers Based on the simulation results of the process, the constraints are learned. By linking environmental information in the simulation of that process with constraints obtained through learning, environmental constraint data is generated in which environmental feature information (information about the environment) and constraints are linked. A data generation method that includes the following.

[0208] (Note 14) On the computer, This involves selecting environmental constraint data that corresponds to the actual environment, which is the execution environment of the process being executed, from among environmental characteristic information, which is information about the execution environment of the process, and the constraints on the execution of the process in that environment. Based on the constraints indicated by the selected environmental constraint data, the controlled object is controlled to execute the process that is to be executed. A recording medium that stores a program to execute.

[0209] (Note 15) On the computer, From the environmental constraint data, which links environmental characteristic information (information about the execution environment of the process) with constraints on the execution of the process in that environment, select the environmental constraint data that corresponds to the actual environment in which the process is being executed. A recording medium that stores a program to execute.

[0210] (Note 16) On the computer, Based on the simulation results of the process, the constraints are learned, By linking environmental information in the simulation of that process with the constraints obtained through learning, environmental constraint data is generated in which environmental feature information (information about the environment) and constraints are linked. A recording medium that stores a program to execute. [Industrial applicability]

[0211] The present invention may be applied to a control device, a constraint selection device, a data generation device, a control method, a constraint selection method, a data generation method, and a storage medium. [Explanation of Symbols]

[0212] 1. Control System 100, 200, 300, 400, 610 control devices 110 Communications Department 120 Display section 130 Operation Input Section 180 Storage section 181 Environmental Constraint Data Storage Unit 190, 290, 390, 490 Processing Unit 191 Real-world feature information generation unit 192 Similarity calculation unit 193, 611, 621 Constraint Selection Section 194, 612 Control Execution Unit 291 Support Request Department 292, 632 Environmental Constraint Data Generation Unit 391, 631 Constraint Learning Unit 491 Simulation Processing Unit 620 Constraint Selection Device 630 Data Generation Device 900 Controlled object

Claims

1. A real environment feature information generation means that generates real environment feature information, which is information about the real environment, based on observation data of the real environment by a sensor that observes the real environment, which is the execution environment of the process to be executed, A similarity calculation means for calculating the similarity of environmental feature information, A constraint selection means selects environmental constraint data that corresponds to the actual environment, based on the similarity between the actual environment characteristic information and the environmental characteristic information included in the environmental constraint data, from among environmental constraint data that links environmental characteristic information, which is information about the execution environment of the processing, with constraint conditions for the execution of the processing in that environment. A control execution means that controls the controlled object to execute the process to be executed based on the constraints indicated by the selected environmental constraint data, A control device equipped with the following features.

2. The aforementioned environmental characteristic information is represented in a graph that includes nodes representing objects in the processing execution environment and edges representing the relationships between nodes. The control device according to claim 1.

3. If there is no environmental constraint data containing environmental feature information whose similarity to the actual environmental feature information is greater than a predetermined threshold, an environmental constraint data generation means generates environmental constraint data that links the actual environmental feature information with the constraints in the actual environment. The control device according to claim 1 or claim 2, further comprising:

4. If there is no environmental constraint data containing environmental feature information whose similarity to the actual environmental feature information is greater than a predetermined threshold, the system further provides a support request means for requesting user action. The aforementioned environmental constraint data generation means generates environmental constraint data that links real-world characteristic information with constraints in the real environment, based on user operations. The control device according to claim 3.

5. The aforementioned environmental constraint data generation means generates environmental constraint data in which real-world characteristic information and constraint conditions entered by user operations are linked. The control device according to claim 4.

6. A constraint learning means that learns constraint conditions based on user operations that manipulate the controlled object. Furthermore, The aforementioned environmental constraint data generation means generates environmental constraint data in which real-world feature data and constraints obtained through learning are linked. The control device according to claim 4.

7. A constraint learning method that learns constraints based on the simulation results of the process being executed in an environment that simulates a real environment. Furthermore, The aforementioned environmental constraint data generation means generates environmental constraint data in which real-world feature data and constraints obtained through learning are linked. The control device according to claim 3.

8. A constraint selection means for selecting environment-compatible constraint data that corresponds to the actual environment which is the execution environment of the process being executed, from among environment-compatible constraint data which is linked to environment characteristic information which is information about the execution environment of the process and constraint conditions for the execution of the process in that environment, A control execution means that controls the controlled object to execute the process to be executed based on the constraints indicated by the selected environmental constraint data, If there is no environmental constraint data containing environmental feature information whose similarity to the actual environmental feature information is greater than a predetermined threshold, the environmental constraint data generation means generates environmental constraint data that links the actual environmental feature information with the constraints in the actual environment. A control device equipped with the following features.

9. Computers Based on observation data of the real environment, which is the execution environment for the process being executed, obtained from sensors that observe the real environment, real environment characteristic information, which is information about the real environment, is generated. From among the environmental constraint data, which is a collection of environmental characteristic information that is information about the execution environment of a process and constraints on the execution of the process in that environment, environmental constraint data corresponding to the actual environment is selected based on the similarity between the actual environmental characteristic information and the environmental characteristic information included in the environmental constraint data. Based on the constraints indicated by the selected environmental constraint data, the controlled object is controlled to execute the target process. A control method that includes the following.

10. On the computer, Based on observation data of the real environment, which is the execution environment for the process being executed, obtained from sensors that observe the real environment, real environment feature information, which is information about the real environment, Among the environmental constraint data, which is a collection of environmental characteristic information that is information about the execution environment of a process and constraints on the execution of the process in that environment, the environmental constraint data corresponding to the actual environment is selected based on the similarity between the actual environmental characteristic information and the environmental characteristic information included in the environmental constraint data. Based on the constraints indicated by the selected environmental constraint data, the controlled object is controlled to execute the process that is to be executed. A program to execute.

Images