System, method, and program for automatically operating operation target object

Abstract

To provide a system and the like for automatically operating an operation target object.SOLUTION: A system of the present invention comprises: acquisition means for acquiring initial information indicating an initial state of an operation target object and further acquiring a plurality of pieces of information indicating a state of the operation target object for each of a plurality of states of the operation target object when a series of operations on the operation target object is being performed; simulation means for simulating an action of the operation target object based on the initial information; and generation means for generating an operation signal for the operation target object, in which the generation means repeats generating the operation signal based on one of a plurality of pieces of information indicating a state of the operation target object and the operation scenario until the series of operations is completed.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a system, a method, and a program for automatically operating an object to be operated. The present invention also relates to a system for generating an operation scenario used for automatically operating an object to be operated. [Background technology]

[0002] The automation of robots and machines (construction machines) is generally carried out using imitation learning and offline reinforcement learning (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 6-226665 Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present invention is to provide a system for automatically manipulating an object to be operated so as to achieve smooth movement of the object to be operated. [Means for solving the problem]

[0005] The present invention provides a system for automatically operating an operation object, and the like, which generates an operation scenario and operates the operation object in accordance with the operation scenario.

[0006] The present invention provides, for example, the following items. (Item 1) A system for automatically operating an operation object, comprising: an acquisition means for acquiring initial information indicating an initial state of the operation object, the acquisition means further acquiring a plurality of pieces of information indicating states of the operation object for each of a plurality of states of the operation object when a series of operations of the operation object is being performed; a simulation means for simulating an action of the operation object based on the initial information, the simulation means being configured to generate an operation scenario, the operation scenario associating information indicating a plurality of states that the operation object may take when a series of operations of the operation object are being performed with an operation signal for causing the operation object in each state to perform a next action; a generating means for generating an operation signal for the operation target object, the generating means repeating the generation of the operation signal based on one of a plurality of pieces of information indicating a state of the operation target object and the operation scenario until the series of operations is completed; A system comprising: (Item 2) the series of operations includes at least a first operation performed in a first time segment and a second operation performed in a second time segment; the receiving means receives first initial information indicating an initial state of the operation object in the first time segment; The simulation means generates a first operation scenario for the first operation based on first initial information; the generation means repeats generating a first operation signal for the first operation based on one of a plurality of pieces of information indicating a state of the operation target object acquired in the first time segment and the first operation scenario until the first time segment ends; the receiving means receives second initial information indicating an initial state of the operation object in the second time segment; The simulation means generates a second operation scenario for the second operation based on second initial information; The system described in the above item, wherein the generation means repeats generating a second operation signal for the second operation based on one of a plurality of pieces of information indicating the state of the object to be operated acquired in the second time segment and the second operation scenario until the second time segment ends. (Item 3) 2. The system of claim 1, wherein the first time period and the second time period partially overlap. (Item 4) The simulation means includes: (a) estimating an operation signal for causing the operation target object in a state indicated by the initial information to perform a next action; (b) estimating a state of the operation target object after being operated with the estimated operation signal; and (c) estimating an operation signal for causing the operation target object in the estimated state to perform a next action; and (d) repeating (b) to (c) until the series of operations is completed; The system according to any one of the preceding items, wherein the system is configured to generate the operation scenario by performing the following: (Item 5) The simulation means includes a machine learning model and a simulator of the operation object, the machine learning model performs the steps (a) and (c), and the machine learning model learns a relationship between the information indicating a state of the operation object and an operation signal so that, when information indicating a state of the operation object is input to the machine learning model, an operation signal for causing the operation object in the state to perform a next action can be output; The system according to any one of the preceding claims, wherein the simulator performs (b). (Item 6) the simulation means comprises a machine learning model; The system described in any one of the above items, wherein the machine learning model learns the relationship between the information indicating the state of the object to be operated and the operation signal so that, when information indicating the state of the object to be operated is inputted into the machine learning model, the machine learning model can output an operation signal for causing the object to be operated in the said state to perform a next action and information indicating the state of the object to be operated after the next action is performed. (Item 7) The system described in any one of the above items further comprises a modification means for modifying the operation scenario, the modification means modifying the operation scenario based on input by a user or in response to the operation scenario violating a predetermined rule. (Item 8) The generating means includes: determining a difference between a state indicated by the information acquired by the acquisition means and a state defined in the operational scenario; determining whether the difference is greater than or equal to a threshold; The system according to any one of the preceding claims, (Item 9) The system according to any one of the preceding claims, wherein the information indicating the state includes at least one of a joint angle of the object to be operated, position information of the object to be operated, Lidar information of the object to be operated, and image information of the object to be operated. (Item 10) The system according to any one of the preceding claims, wherein the operation object is a construction machine. (Item 11) A method for automatically manipulating an object to be manipulated, comprising: receiving initial information indicating an initial state of the operation object; generating an operation scenario by simulating an action of the operation object based on the initial information, the operation scenario associating information indicating a plurality of states that the operation object may take when a series of operations of the operation object are performed with an operation signal for causing the operation object in each state to perform a next action; When a series of operations of the operation object is being performed, (i) receiving information indicating a state of the operation object; (ii) generating an operation signal for the operation target object based on the acquired information and the operation scenario; Repeating the above series of operations until completion. The method includes: (Item 11A) 12. The method according to claim 11, comprising the features according to any one of the preceding claims. (Item 12) A program for automatically operating an operation object, the program being executed on a computer having a processor, the program comprising: receiving initial information indicating an initial state of the operation object; generating an operation scenario by simulating an action of the operation object based on the initial information, the operation scenario associating information indicating a plurality of states that the operation object may take when a series of operations of the operation object are performed with an operation signal for causing the operation object in each state to perform a next action; When a series of operations of the operation object is being performed, (i) receiving information indicating a state of the operation object; (ii) generating an operation signal for the operation target object based on the acquired information and the operation scenario; Repeating the above series of operations until completion. A program causing the processor to perform a process including the steps of: (Item 12A) 13. The program according to item 12, comprising any one of the features according to the preceding items. (Item 13) A system for generating an operation scenario used to automatically operate an operation target object, comprising: A receiving means for receiving initial information indicating an initial state of the operation object; a simulation means for simulating an action of the operation object based on the initial information, the simulation means being configured to generate an operation scenario, the operation scenario associating information indicating a plurality of states that the operation object can take when a series of operations of the operation object are performed with an operation signal for causing the operation object in each state to perform a next action; wherein the operation scenario is used to generate an operation signal for the operation object based on information indicating a state of the operation object. (Item 13A) Item 14. A system according to item 13, comprising the features according to any one of the preceding items. (Item 14) A method for generating an operation scenario to be used for automatically operating an operation object, comprising the steps of: receiving initial information indicating an initial state of the operation object; generating an operation scenario by simulating an action of the operation object based on the initial information, the operation scenario associating information indicating a plurality of states that the operation object may take when a series of operations of the operation object are performed with an operation signal for causing the operation object in each state to perform a next action; wherein the operation scenario is used to generate an operation signal for the operation object based on information indicating a state of the operation object. (Item 14A) 15. The method according to claim 14, comprising the features according to any one of the preceding claims. (Item 15) A program for generating an operation scenario used for automatically operating an operation target object, the program being executed on a computer having a processor, receiving initial information indicating an initial state of the operation object; generating an operation scenario by simulating an action of the operation object based on the initial information, the operation scenario associating information indicating a plurality of states that the operation object may take when a series of operations of the operation object are performed with an operation signal for causing the operation object in each state to perform a next action; wherein the operation scenario is used to generate an operation signal for the operation object based on information indicating a state of the operation object. (Item 15A) Item 16. A program according to item 15, comprising the features according to any one of the preceding items. Effect of the Invention

[0007] According to the present invention, a system for automatically operating an operation object is provided, which can automatically operate an operation object with smooth movements. [Brief description of the drawings]

[0008] [Figure 1] A diagram showing an example of a flow for automatically operating construction machinery using an operation scenario. [Figure 2A] FIG. 1 is a diagram showing an example of the configuration of a system 100 for automatically manipulating an object to be manipulated. [Figure 2B] FIG. 1 illustrates an example of a configuration for system 100', which is an alternative embodiment of system 100. [Diagram 3] FIG. 13 is a diagram showing an example of a data configuration of an operation scenario generated by the simulation means 120. [Figure 4A] 1 is a block diagram of a system including a simulation means 120 according to a first embodiment. [Figure 4B] 1 is a block diagram of a system including a simulation means 120 according to a second embodiment. [Diagram 5] FIG. 1 is a diagram showing an example of the configuration of an information processing device 200 that implements the system 100. [Figure 6]6 is a flowchart showing an example of a process 600 in the system 100 for automatically manipulating an object to be manipulated. [Figure 7] Conceptual diagram showing how a sequence of operations includes multiple operations [Figure 8] 8 is a flowchart illustrating an example of a process 800 in the system 100 when the sequence of operations includes at least a first operation and a second operation. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] (definition) In this specification, the term "operation object" refers to an object that is the object to be operated, and the "operation object" is, for example, a robot or a construction machine. The "operation object" is preferably an object that can be operated unmanned, without a human being riding on it.

[0010] In this specification, a "series of operations" refers to a plurality of operations performed to make an object to be operated perform a certain action. For example, when making an object to be operated lift a member, (1) the arm of the object to be operated is moved to the member, (2) the arm of the object to be operated grasps the member, and (3) the arm of the object to be operated is raised up. The series of operations to make an object to be operated lift the member includes operations for (1) to (3). The series of operations preferably includes a first operation performed in a first time segment and a second operation performed in a second time segment.

[0011] In this specification, "action" refers to any action that changes a state, and includes not only movements but also actions that are not visible from the outside, such as changing modes.

[0012] In this specification, "initial information" refers to information indicating the initial state of an operation target object, and "initial state" refers to the state immediately before a corresponding operation scenario is generated. For example, when multiple operation scenarios are generated for a series of operations, there are at least a first initial state immediately before a first operation scenario of the multiple operation scenarios is generated, and a second initial state immediately before a second operation scenario of the multiple operation scenarios is generated.

[0013] 1.Automated operation of construction machinery The inventor of the present invention is working on the development of a mechanism for automatically operating construction machinery using AI. Here, AI is an abbreviation for "Artificial Intelligence," and refers to technology for artificially implementing human intelligence (or intelligence that surpasses human intelligence) on a computer system, or a computer system on which such technology is implemented.

[0014] The inventor found that if AI were to grasp the status of a construction machine from sensor signals indicating the status of the machine and output operation signals in real time, time lags and noise would easily occur in the operation signals, making it difficult to move the construction machine smoothly. As a result of intensive research, the inventor developed a mechanism that can address this issue.

[0015] In this system, instead of the AI ​​outputting operation signals in real time from sensor signals indicating the status of the construction machinery, the AI ​​generates an operation scenario from the initial state of the construction machinery and outputs operation signals for the construction machinery according to the generated operation scenario. An operation scenario specifies the ideal operation to be performed when making the construction machinery perform a specific action, and by controlling the construction machinery according to the operation scenario, it is possible to realize the smooth movement of the construction machinery.

[0016] FIG. 1 shows an example of a flow for automatically operating a construction machine using an operation scenario. In this example, a backhoe 10 is used as an example of a construction machine, and an operation for making the backhoe 10 excavate the ground is used as an example of a series of operations. The backhoe 10 is equipped with a sensor for detecting the state of the backhoe 10, or the sensor is disposed around the backhoe 10. The sensor may be, for example, a sensor for detecting the joint angle of the backhoe 10, a sensor for detecting the position information of the whole or part of the backhoe 10, a camera for photographing the posture of the backhoe 10, a Lidar for detecting the posture of the backhoe 10, or the like.

[0017] First, the sensor detects an initial state of the backhoe 10 and obtains initial information indicating the detected initial state.

[0018] Next, in step S1, the acquired initial information is input to the system 100. As a result, in the system 100, an operation scenario is generated based on the initial information.

[0019] Specifically, the initial information is input to the simulation means 120 of the system 100. The simulation means 120 has a function of generating an operation scenario by simulating the action of the backhoe 10. In this way, the simulation means 120 generates an operation scenario that specifies an operation for moving the backhoe 10 from an initial state to a final state where the ground is excavated, based on the initial information.

[0020] If the first state to the final state of excavating the ground are passed through the nth state from the initial state, this operation scenario, for example, associates a first operation signal for moving the backhoe 10 from the initial state to the first state with the first state, associates a second operation signal for moving the backhoe 10 from the first state to the second state with the second state, and associates an nth signal for moving the backhoe 10 from the n-1th state to the nth state with the nth state. In this way, if the current state of the backhoe 10 is known, it is possible to output an operation signal for moving the backhoe 10 from the current state to the next state by referring to the operation scenario. Here, the next state is the state that the backhoe 10 takes next to the current state of the backhoe 10 among the states that the backhoe 10 can take in the action up to the final state.

[0021] The operation scenario generated by the simulation means 120 is passed to the generation means 130, and the generation means 130 generates an operation signal for moving the backhoe 10 from the current state to the next state by referring to the operation scenario.

[0022] In step S2, the generated operation signal is provided to the backhoe 10, and the backhoe 10 automatically moves in accordance with the operation signal.

[0023] The sensor then detects the condition after the backhoe 10 has automatically moved and obtains information indicative of the detected condition.

[0024] Next, in step S3, the acquired information is input to the system 100 and provided to the generating means 130 of the system 100. The generating means 130 generates an operation signal for moving the backhoe 10 from the current state to the next state by referring to the operation scenario.

[0025] In step S4, the generated operation signal is provided to the backhoe 10, and the backhoe 10 moves automatically in accordance with the operation signal.

[0026] By repeating steps S3 and S4 until the backhoe 10 reaches the final state, the backhoe 10 automatically excavates the ground.

[0027] In this way, by operating the backhoe 10 with reference to the operation scenario, smooth movement of the object to be operated can be achieved without delays due to calculations, compared to when the AI ​​outputs operation signals in real time.

[0028] In the above example, the backhoe 10 is automatically operated, but the object to be automatically operated is not limited to a construction machine such as a backhoe. Any object can be automatically operated as long as its state can be detected and its operation can be controlled based on the state.

[0029] Automatically manipulating an operation object can be implemented, for example, by a system 100 for automatically manipulating an operation object, which will be described later.

[0030] 2. Configuration of a system for automatically operating an object to be operated FIG. 2A shows an example of the configuration of a system 100 for automatically manipulating an object to be manipulated.

[0031] The system 100 comprises an obtaining means 110, a simulating means 120, and a generating means 130.

[0032] The acquiring means 110 is configured to acquire information indicating a state of the operation object. The acquiring means 110 can acquire initial information indicating an initial state of the operation object before the simulation means 120 generates an operation scenario. Furthermore, the acquiring means 110 can acquire information indicating a state of the operation object after the simulation means 120 generates an operation scenario. The information indicating the state of the operation object is acquired for each of a plurality of states of the operation object when a series of operations of the operation object are being performed. For example, the acquiring means 110 can acquire, after generating an operation scenario, information indicating a state of the operation object when a series of operations is started, information indicating a state after a first operation of the series of operations is performed, information indicating a state after a second operation of the series of operations is performed, and so on.

[0033] The acquisition means 110 can acquire at least one of the joint angle of the operation object, the position information of the operation object, the Lidar information of the operation object, and the image information of the operation object as information indicating the state of the operation object. The joint angle of the operation object may be, for example, the angle of all joints of the joints of the operation object, or may be the angle of a part of the joints, and preferably may be the angle of a joint related to a part that operates in a series of operations. The position information of the operation object may be, for example, the position information of the entire operation object, or may be the position information of a part, and preferably may be the position information of a part that operates in a series of operations. The Lidar information of the operation object may be point cloud data, and may be, for example, the Lidar information of the entire operation object, or may be the Lidar information of a part, or may be the Lidar information of the periphery of the operation object, and preferably may be the Lidar information of a part that operates in a series of operations. The image information of the operation object may be, for example, the image information of the entire operation object, or may be the image information of a part, and preferably may be the image information of a part that operates in a series of operations.

[0034] In a preferred embodiment, the acquisition means 110 acquires a combination of angle information of each joint of the operation object and point cloud data of the surroundings of the operation object as information indicating the state of the operation object. This is preferable in that it makes it possible to accurately acquire the surrounding state in addition to the state of the operation object. This can be achieved, for example, by an angle sensor and Lidar.

[0035] The initial information acquired by the acquiring means 110 is passed to the simulation means 120. In addition, the information acquired by the acquiring means 110 after the operational scenario is generated is passed to the generating means .

[0036] The simulation means 120 is configured to generate an operation scenario by simulating the action of the operation object based on the initial information acquired by the acquisition means 110. The operation scenario associates information indicating a plurality of states that the operation object can take when a series of operations of the operation object are performed with an operation signal for making the operation object in each state perform a next action, and in order to generate this operation scenario, the simulation means 120 estimates a plurality of states in a series of operations and operation signals corresponding to each state by repeating the process of estimating an operation signal for making the operation object in a certain state perform a next action and estimating the state of the operation object after the action.

[0037] Specifically, the simulation means 120 can generate an operation scenario by performing the steps of (a) estimating an operation signal for making the operation object in the initial state indicated by the initial information perform the next action, (b) estimating a state of the operation object after being operated by the estimated operation signal, (c) estimating an operation signal for making the operation object in the estimated state perform the next action, and (d) repeating (b) to (c) until a series of operations is completed. In the operation scenario generated in this way, the initial state is associated with a first operation signal for operating the operation object in the initial state, the second state after being operated by the first operation signal is associated with a second operation signal for operating the operation object in the second state, the third state after being operated by the second operation signal is associated with a third operation signal for operating the operation object in the third state, and the nth state after being operated by the (n-1)th operation signal is associated with the nth operation signal for operating the operation object in the nth state.

[0038] In the first embodiment, the simulation means 120 may include a machine learning model and a simulator of the operation object.

[0039] In this embodiment, the machine learning model is a model that has learned the relationship between information indicating the state of the operation object and the operation signal so that, when information indicating the state of the operation object is input, an operation signal for making the operation object in the state perform the next action can be output. The machine learning model may be any model, and the learning method may also be any method, as long as it is possible to output an operation signal for making the operation object in the state perform the next action when information indicating the state of the operation object is input. For example, the machine learning model may be a model based on a neural network, or a model based on other algorithms. It is preferable to use a model suitable for information used for input (for example, a model suitable for numerical input such as joint angles, a model suitable for image input, etc.). For example, the learning method may be imitation learning, reinforcement learning, offline reinforcement learning, etc. Typically, the learned model may be a model constructed by acquiring data (state, operation signal) when a human operates the operation object in time series, and using the data, learning a neural network by imitation learning with (state) as input learning data and (operation signal) as output learning data.

[0040] When an operation signal is input, the simulator can simulate the action of the operation object according to the operation signal, and outputs information indicating the state of the operation object after the action. The simulator outputs the same type of information as the information acquired by the acquisition means 110. The simulator may be, for example, a simulator that simulates the minimum action of the operation object, or a simulator that simulates more accurate and complicated actions. In the method of the present invention, since the generation of an operation scenario using the simulator is performed before the operation of the operation object, there is an advantage that even if the calculation speed of the simulator is slow, it does not affect the real-time execution of the operation scenario. Therefore, it is possible to use a complicated simulator.

[0041] In this embodiment, the machine learning model performs the above steps (a) and (c), and the simulator performs the above step (b). That is, when initial information indicating an initial state is input to the machine learning model, the machine learning model estimates and outputs a first operation signal for making the operation target in the initial state perform the next action. When the output first operation signal is input to the simulator, the simulator simulates the action of the operation target according to the first operation signal, and outputs information indicating the state (second state) of the operation target after the action. When the output information indicating the second state is input to the machine learning model, the machine learning model estimates and outputs a second operation signal for making the operation target in the second state perform the next action. When the output second operation signal is input to the simulator, the simulator simulates the action of the operation target according to the second operation signal, and outputs information indicating the state (third state) of the operation target after the action. By repeating these steps until a series of operations is completed, an operation scenario for a series of operations can be generated.

[0042] In a second embodiment, the simulation means 120 may comprise a machine learning model.

[0043] In this embodiment, the machine learning model is a model that has learned the relationship between information indicating the state of the operation object and the operation signal so that, when information indicating the state of the operation object is input, an operation signal for making the operation object in that state perform the next action and information indicating the state of the operation object after the next action are output. The machine learning model may be any model, and the learning method may also be any method, as long as it is possible to output, when information indicating the state of the operation object is input, an operation signal for making the operation object in that state perform the next action and information indicating the state of the operation object after the next action is output. For example, the machine learning model may be a model based on a neural network, a model based on a convolutional neural network, or a model based on other algorithms. It is preferable to use a model suitable for information used for input (for example, a model suitable for numerical input such as joint angles, a model suitable for image input, etc.). For example, the learning method may be imitation learning, reinforcement learning, offline reinforcement learning, etc. Typically, the trained model may be a model constructed by acquiring data (state, operation signal) in time series when a human operates an operation target, organizing the data into pairs of (state, (operation signal, next state)), and using the organized data to train a neural network by imitation learning, with (state) as input learning data and (operation signal, next state) as output learning data. Data (state, operation signal) when a human operates an operation target can be used in both this embodiment and the above-mentioned embodiment, and there is no need to acquire data separately for the two embodiments.

[0044] In the method of the present invention, since the generation of an operation scenario using a machine learning model is performed before the operation of the operation target object, even if the calculation speed of the machine learning model is slow, there is an advantage that it does not affect the real-time execution of the operation scenario. Therefore, it is possible to use a complex machine learning model.

[0045] In this embodiment, the machine learning model performs the above steps (a) to (c). That is, when initial information indicating an initial state is input to the machine learning model, the machine learning model estimates a first operation signal for making the operation target in the initial state perform the next action, estimates information indicating a second state of the operation target after the action of the operation target according to the first operation signal, and outputs the first operation signal and information indicating the second state. When the output information indicating the second state is input to the machine learning model, the machine learning model estimates a second operation signal for making the operation target in the second state perform the next action, estimates information indicating a third state of the operation target after the action of the operation target according to the second operation signal, and outputs the second operation signal and information indicating the third state. By repeating these steps until a series of operations is completed, an operation scenario for a series of operations can be generated.

[0046] 3 shows an example of a data configuration of an operation scenario generated by the simulation means 120. In this example, an example of an operation scenario for a series of operations for making an operation target object perform an operation of digging the ground is shown.

[0047] In the operation scenario, the state of the operation object is associated with an operation signal for making the operation object in that state perform the next action. For example, if the operation object is a backhoe equipped with a boom extending from a base, an arm extending from the boom, and a bucket at the tip of the arm, the action may be an action to move the backhoe around each joint, and may be an action that can be achieved by one or a combination of, for example, raising and lowering the boom (changing the joint angle between the base and the boom), raising and lowering the arm (changing the joint angle between the boom and the arm), and rotating the bucket (changing the joint angle between the arm and the bucket). For example, in an operation scenario for a backhoe, a first state (i.e., a state in which the joint angle between the base and the boom is α1, the joint angle between the boom and the arm is β1, and the joint angle between the arm and the bucket is γ1) is associated with a first operation signal for causing the backhoe in the first state to perform a first action for excavating the ground (i.e., to set the joint angle between the base and the boom to α2 in order to lower the bucket so that the bucket contacts the ground), and a second state (i.e., a state in which the joint angle between the base and the boom is α2, the joint angle between the boom and the arm is β1, and the joint angle between the arm and the bucket is γ1) resulting from the first action is associated with a second operation signal for causing the backhoe in the first state to perform a first action for excavating the ground (i.e., to set the joint angle between the base and the boom to α2 in order to lower the bucket so that the bucket contacts the ground), and a second operation signal for causing the backhoe in the third state to perform a second action to excavate the ground (i.e., setting the joint angle between the boom and the arm to β2 so that the arm moves so that the bucket gouges the ground), and a third state resulting from the second action (i.e., a state in which the joint angle between the base and the boom is α2, the joint angle between the boom and the arm is β2, and the joint angle between the arm and the bucket is γ1) is associated with a third operation signal for causing the backhoe in the third state to perform a third action to excavate the ground (i.e., setting the joint angle between the arm and the bucket to γ2 so that the bucket moves so that the bucket scoops up soil).

[0048] In both the first and second embodiments, the generated operation scenario is passed to a generating means 130 .

[0049] 2A again, the generating means 130 is configured to generate an operation signal based on the information indicating the state of the operation target object acquired by the acquiring means 110 and the operation scenario generated by the simulation means 120. The generating means 130 can refer to the operation scenario and generate an operation signal corresponding to the state indicated by the information acquired by the acquiring means 110 in the operation scenario.

[0050] The generating means 130 can generate an operation signal for each of the plurality of pieces of information acquired by the acquiring means 110, and can repeat this until a series of operations is completed. The operation target object can be automatically operated by the series of operation signals generated by the generating means 130.

[0051] For example, when there is a difference between the state of the operation target object indicated by the information acquired by the acquisition means 110 and the state defined in the operation scenario, the generation means 130 can perform the following processing.

[0052] If the difference is smaller than a predetermined threshold, the generating means 130 determines that the difference can be ignored and can generate an operation signal corresponding to the state defined in the operation scenario. In this case, the cause of the difference is determined to be noise or a slight time lag, and if it is noise, it can be considered that the operation target object is being operated according to the operation scenario, and if it is a slight time lag, the time lag can be absorbed by the next action.

[0053] If the difference is equal to or greater than a predetermined threshold, the generating means 130 can determine that the realization of the operation scenario has failed and can not generate an operation signal. The generating means 130 can emergency stop the operation of the operation target object without generating an operation signal, or can regenerate the operation scenario. Specifically, the simulation means 120 can generate an operation scenario using a state in which the difference is equal to or greater than the predetermined threshold as an initial state, and the generating means 130 can generate an operation signal according to the newly generated operation scenario. It is preferable that the user sets whether to perform an emergency stop or regenerate the operation scenario when the difference is equal to or greater than the predetermined threshold before the operation of the system 100.

[0054] In this way, even if the operation target object cannot be operated according to the operation scenario, the system 100 can appropriately deal with the situation. The predetermined threshold value can be appropriately set by the system 100 or the user, and if the predetermined threshold value is too large, the accuracy of execution of the operation scenario may deteriorate, whereas if the predetermined threshold value is too small, the completion rate of the operation scenario may deteriorate.

[0055] 2B shows an example of the configuration of a system 100' which is an alternative embodiment of the system 100. The system 100' has the same configuration as the system 100 except that the system 100' includes a correction means 140. Therefore, detailed description of the same configuration as the system 100 will be omitted here.

[0056] The modifying means 140 is configured to modify the operation scenario generated by the simulation means 120 .

[0057] The modifying means 140 can modify the operation scenario based on, for example, an input by a user. The modifying means 140 can modify the operation scenario as, for example, indicated by an input by a user. For example, the operation scenario generated by the simulation means 120 is presented to the user before being provided to the generating means 130, and the user can check the operation scenario and input an input for modification to the system 100 if necessary.

[0058] Alternatively, the modifying means 140 can determine whether the operation scenario generated by the simulation means 120 violates a predetermined rule, and can modify the operation scenario in response to determining that the operation scenario violates the predetermined rule. The predetermined rule can be, for example, that the movable range of a part of the operation target object is within a predetermined range, that the operation target object does not enter a predetermined prohibited area, etc. If the predetermined rule is violated, the modifying means 140 can modify the operation scenario so as not to violate the predetermined rule.

[0059] The modified operational scenario is provided to the generating means 130, which is capable of generating an operational signal based on the operational scenario, as described above.

[0060] FIG. 4A shows a block diagram of a system including a simulation means 120 in the first embodiment.

[0061] First, the acquisition unit 110 acquires initial information indicating an initial state of the operation target object 10. The acquisition unit 110 may be, for example, an angle sensor, Lidar, etc. The initial information acquired by the acquisition unit 110 is provided to the simulation unit 120.

[0062] Next, the simulation means 120 generates an operation scenario. The initial information is first provided to the trained model. When the trained model receives the initial information, it outputs an operation signal for making the operation object in the state indicated by the initial information perform the next action. The operation signal for making the operation object perform the next action is then provided to the simulator. The simulator can simulate the action of the operation object, and simulates the operation object performing the action according to the input operation signal. This outputs an estimate of the state of the operation object after the action is performed, and the estimate of the state is provided to the trained model. When the estimate of the state is input, the trained model outputs an operation signal for making the operation object in that state perform the next action. By repeating this, the estimate of the state and the corresponding operation signal are accumulated, and an operation scenario is generated by associating them.

[0063] The generated operation scenario is provided to the generating means 130.

[0064] When the operation scenario is generated, operation of the operation target object 10 is started. First, the acquisition means 110 acquires information indicating the current state of the operation target object 10. Next, the acquisition means 110 provides the information to the generation means 130.

[0065] The generating means 130 refers to the operation scenario and generates an operation signal for causing the operation target object 10 in the current state to execute the next action. The generated operation signal is provided to the operation target object 10, and the operation target object 10 executes the action in accordance with the operation signal.

[0066] After the action is executed, the state of the operation target object 10 changes, so the acquisition means 110 again acquires information indicating the current state of the operation target object 10 and provides the information to the generation means 130. The generation means 130 generates an operation signal based on the information and the operation scenario, and the operation target object 10 executes the action according to the operation signal.

[0067] By repeating this process until the series of operations is completed, the operation target object 10 can complete the action according to the series of operations.

[0068] FIG. 4B shows a block diagram of a system including a simulation means 120 in the second embodiment.

[0069] First, the acquisition unit 110 acquires initial information indicating an initial state of the operation target object 10. The acquisition unit 110 may be, for example, an angle sensor, Lidar, etc. The initial information acquired by the acquisition unit 110 is provided to the simulation unit 120.

[0070] Next, the simulation means 120 generates an operation scenario. The initial information is provided to the trained model. When the initial information is input, the trained model outputs an operation signal for causing the operation object in the state indicated by the initial information to perform the next action, and an estimated value of the state of the operation object after the action is executed. The output estimated value of the state is input to the trained model. When the estimated value of the state is input, the trained model outputs an operation signal for causing the operation object in that state to perform the next action, and an estimated value of the state of the operation object after the action is executed. By repeating this, the estimated value of the state and the corresponding operation signal are accumulated, and an operation scenario is generated by associating them.

[0071] The generated operation scenario is provided to the generating means 130. The subsequent process is the same as that described above with reference to Fig. 4A.

[0072] The above-described systems 100 and 100' can be implemented in, for example, an information processing device 200 that communicates with an operation target object. In the following, the system 100 will be described, but the same description also applies to the system 100'.

[0073] FIG. 5 shows an example of the configuration of an information processing device 200 in which the system 100 is implemented.

[0074] The information processing device 200 includes a communication interface unit 210, a memory unit 220, and a processor unit 230. The information processing device 200 is connected to a database unit 300. The information processing device 200 is communicatively connected to an operation target object (in this example, a backhoe 10).

[0075] A sensor may be attached to the operation object 10 as the acquisition means 110, or a sensor may be arranged around the operation object 10 as the acquisition means 110. The sensor may be, for example, a sensor that detects a joint angle of the operation object 10, a sensor that detects position information of the whole or part of the operation object 10, a camera that captures an image of the attitude of the operation object 10, a LIDAR that detects the attitude of the operation object 10, or the like.

[0076] The information indicating the state of the operation target object acquired by the acquisition means 110 is provided to the information processing device 200.

[0077] The communication interface unit 210 controls communication between the information processing device 200 and the operation target 10. The communication interface unit 210 also controls communication with the database unit 300. The processor unit 230 of the information processing device 200 can receive information from outside the information processing device 200 via the communication interface unit 210, and can transmit information to outside the information processing device 200. The processor unit 230 of the information processing device 200 can receive information indicating a state of the operation target 10 from the operation target 10 via the communication interface unit 210, and can transmit an operation signal to the operation target 10. The communication interface unit 210 can control communication in any manner.

[0078] The memory unit 220 stores a program required for executing the processing of the information processing device 200, data required for executing the program, and the like. For example, a part or all of a program for automatically operating an operation object (for example, a program for realizing the processing shown in FIG. 6 and FIG. 8 described later) or a program for generating an operation scenario used for automatically operating an operation object (for example, a program for realizing step S602 of the processing shown in FIG. 6 and FIG. 8 described later) is stored. Here, how the program is stored in the memory unit 220 does not matter. For example, the program may be pre-installed in the memory unit 220. Alternatively, the program may be installed in the memory unit 220 by being downloaded via a network. The program may be stored on a computer-readable tangible storage medium. The memory unit 220 may be implemented by any storage means.

[0079] The processor unit 230 controls the overall operation of the information processing device 200. The processor unit 230 reads out a program stored in the memory unit 220 and executes the program. This makes it possible for the information processing device 200 to function as a device that executes desired steps. The processor unit 230 may be implemented by a single processor or by multiple processors.

[0080] The database unit 300 can store the generated operation scenario.

[0081] The simulation means 120 and the generation means 130 of the system 100 of the present invention may be implemented by a processor unit 230 .

[0082] In the example shown in FIG. 5, each component of the information processing device 200 is provided in the server device 200, but the present invention is not limited to this. Any of the components of the information processing device 200 can also be provided outside the information processing device 200. For example, when the memory unit 220 and the processor unit 230 are each composed of separate hardware components, the hardware components may be connected via any network. In this case, the type of network does not matter. The hardware components may be connected, for example, via a LAN, wirelessly, or wired.

[0083] In the example shown in Fig. 5, the database unit 300 is provided outside the information processing device 200, but the present invention is not limited to this. The database unit 300 can also be provided inside the information processing device 200. In this case, the database unit 300 may be implemented by the same storage means as the storage means implementing the memory unit 220, or may be implemented by a storage means different from the storage means implementing the memory unit 220. In any case, the database unit 300 is configured as a storage unit for the information processing device 200. The configuration of the database unit 300 is not limited to a specific hardware configuration.

[0084] The information processing device 200 is not limited to a specific hardware configuration. For example, it is within the scope of the present invention to configure the processor unit 230 with an analog circuit rather than a digital circuit. The configuration of the information processing device 200 is not limited to the above-mentioned one as long as it can realize its functions.

[0085] In the above example, the system 100 is described as including the acquisition means 110, the simulation means 120, and the generation means 130, but the simulation means 120 alone can be the subject of the present invention. That is, the simulation means 120 can be regarded as a system for generating an operation scenario used to automatically operate an operation object. In this case, the simulation means 120 includes a receiving means for receiving initial information indicating an initial state of the operation object, and generates an operation scenario by simulating an action of the operation object based on the information received by the receiving means. As described above, this operation scenario is used by the generation means 130 to generate an operation signal for the operation object based on the information indicating the state of the operation object. The generation means 130 may be in the same information processing device as the simulation means 120, or may be in a different information processing device.

[0086] 3. Processing in the system 100 for automatically manipulating an object to be manipulated 6 is a flowchart showing an example of a process 600 in the system 100 for automatically operating an operation target object. The process 600 is executed in the processor unit 230 of the information processing device 200, for example.

[0087] First, the acquisition means 110 acquires initial information indicating an initial state of the operation object 10. The acquisition means 110 can acquire at least one of the joint angles of the operation object, the position information of the operation object, the Lidar information of the operation object, and the image information of the operation object as information indicating the state of the operation object. The information from the acquisition means 110 is passed to the processor unit 230 via the communication interface unit 210.

[0088] In step S601, the processor unit 230 receives the initial information acquired by the acquisition means 110.

[0089] In step S602, the processor unit 230 generates an operation scenario by simulating an action of the operation target object based on the initial information. Specifically, the processor unit 230 can generate an operation scenario by performing the steps of (a) estimating an operation signal for making the operation target object in the initial state indicated by the initial information received in step S601 perform the next action, (b) estimating a state of the operation target object after being operated by the estimated operation signal, (c) estimating an operation signal for making the operation target object in the estimated state perform the next action, and (d) repeating (b) to (c) until a series of operations is completed.

[0090] In step S603, the processor unit 230 performs a series of operations on the operation target object by referring to the operation scenario generated in step S602. Specifically, first, the acquisition means 110 acquires information indicating the state of the operation target object before or when the series of operations is started, and the processor unit 230 receives the information indicating the state of the operation target object from the acquisition means 110. Next, the processor unit 230 generates an operation signal for the operation target object based on the acquired information and the operation scenario. This is repeated until the series of operations is completed.

[0091] When generating an operation signal for the operation target object based on the acquired information and the operation scenario, the processor unit 230 compares the state indicated by the acquired information with the state specified in the operation scenario, and identifies the same or most similar state among the states specified in the operation scenario. Next, the processor unit 230 determines whether or not the difference between the state indicated by the acquired information and the state specified in the operation scenario exceeds a predetermined threshold. If the difference does not exceed the predetermined threshold, the processor unit 230 identifies the operation signal associated with the state specified in the operation scenario and outputs it. This causes the operation target object to take an action according to the operation signal. If the difference is equal to or greater than the predetermined threshold, the processor unit 230 can emergency stop the series of operations or return to step S602 and regenerate the operation scenario. This is because it is determined that the deviation between the actual state of the operation target object and the state specified in the operation scenario is large, and the current operation scenario cannot be used. When regenerating the operation scenario, step S602 is performed using the acquired information as initial information.

[0092] By operating the operation object by referring to the operation scenario, smooth movement of the operation object can be achieved without delay due to calculations, compared to when AI outputs operation signals in real time. For example, when operating a backhoe as an operation object, if AI outputs operation signals in real time, the boom, arm, and / or bucket will move in a jerky manner, repeating movement and stopping, due to delays in calculations and deviations due to noise. In contrast, when outputting operation signals according to the operation scenario of the present invention, the boom, arm, and / or bucket will move according to the operation scenario, so that continuous smooth movement of the boom, arm, and / or bucket can be achieved.

[0093] For example, each operation in a series of operations may typically be performed once per second, but may also be performed several times per second, or once every few seconds.

[0094] The length of the generated operation scenario can be changed depending on the operation object and / or the environment the operation object is in. If the operation scenario can be long, multiple operation scenarios can be used for a series of operations.

[0095] For example, the series of operations may include at least a first operation performed in a first time segment and a second operation performed in a second time segment. In the example described below, the first operation and the second operation will be described as an example, but it should be noted that any number of operations may be included in the series of operations, such as a third operation performed in a third time segment, a fourth operation performed in a fourth time segment, etc. The time segments may be, for example, segmented by a predetermined time interval (e.g., every 10 seconds), or segmented by meaningful subtasks (e.g., subtask A: moving the bucket to the excavation site, subtask B: performing excavation, subtask C: lifting the bucket, etc.).

[0096] An operation scenario can be generated for each of a plurality of operations included in a series of operations, and each operation can be performed by referring to the corresponding operation scenario. Each operation scenario for the plurality of operations can be generated using information indicating the state of the operation object when each operation is started as initial information. For example, when a first operation from 0 to 10 seconds and a second operation from 10 to 20 seconds are performed, the first operation scenario is generated based on information indicating the state of the operation object at the start of the first operation (0 seconds), the first operation is performed according to the first operation scenario, and a second operation scenario is generated based on information indicating the state of the operation object at the start of the second operation (10 seconds), and the second operation is performed according to the second operation scenario.

[0097] However, in the above example, the smoothness of the movement of the operation object may be lost due to the influence of the calculation time for generating the operation scenario at 10 seconds. If the calculation time is sufficiently short, there is no problem, but if the calculation time is long, the movement may become intermittent. Therefore, it is preferable to make the time section of the first operation partially overlap with the time section of the second operation. For example, the first operation is set to 0 to 10 seconds, the second operation is set to 9 to 20 seconds, and the second operation scenario can be generated based on information indicating the state of the operation object at the start of the second operation (9 seconds). Then, at the end of the first operation (10 seconds), the operation can be switched to the second operation and performed according to the second operation scenario. This makes it possible to reduce or eliminate the influence of the calculation time, and ensure smooth movement of the operation object.

[0098] FIG. 7 is a diagram conceptually showing that a series of operations includes a plurality of operations.

[0099] For example, a series of operations for operating a backhoe and bringing the bucket into contact with the ground in order to excavate the ground will be described as an example. This series of operations is performed from time t0 to time tn.

[0100] For example, the operation from the initial state at time t0 to the state where the joint angle of the arm is increased at time t2 is defined as the first operation, and the operation from time t1 before time t2 to time t4 is defined as the second operation. In this way, the time segment for the first operation (t0 to t2) and the time segment for the second operation (t1 to t4) partially overlap. By generating the second operation scenario based on the information at time t1, the influence of the time for calculating the second operation scenario can be reduced or eliminated, and the smooth movement of the operation target object can be ensured.

[0101] 8 is a flowchart showing an example of a process 800 in the system 100 when the series of operations includes at least a first operation and a second operation. The process 800 is executed in the processor unit 230 of the information processing device 200, for example.

[0102] First, the acquiring means 110 acquires first initial information indicating the state of the operation object at the start of the first operation.

[0103] In step S801, the processor unit 230 receives first initial information.

[0104] In step S802, the processor unit 230 generates a first operation scenario for the first operation based on the first initial information. Specifically, the processor unit 230 can generate the first operation scenario by performing the steps of (a) estimating an operation signal for causing the operation target object in the initial state indicated by the first initial information received in step S801 to perform the next action, (b) estimating a state of the operation target object after being operated by the estimated operation signal, (c) estimating an operation signal for causing the operation target object in the estimated state to perform the next action, and (d) repeating (b) to (c) until the first operation is completed.

[0105] In step S803, the processor unit 230 performs a first operation of the operation object by referring to the first operation scenario generated in step S802. Specifically, first, the acquisition means 110 acquires information indicating the state of the operation object when the first operation is started, and the processor unit 230 receives the information indicating the state of the operation object from the acquisition means 110. Next, the processor unit 230 generates an operation signal for the operation object based on the acquired information and the first operation scenario. This is repeated until the first operation is completed.

[0106] Next, second initial information indicating a state of the operation object at the start of the second operation is acquired by the acquiring means 110. When the time segment of the first operation and the time segment of the second operation partially overlap, the second initial information is acquired during the first operation.

[0107] In step S804, the processor unit 230 receives the second initial information.

[0108] In step S805, the processor unit 230 generates a second operation scenario for the second operation based on the second initial information. Specifically, the processor unit 230 can generate the second operation scenario by performing the steps of (a) estimating an operation signal for causing the operation target object in the initial state indicated by the second initial information received in step S804 to perform the next action, (b) estimating a state of the operation target object after being operated by the estimated operation signal, (c) estimating an operation signal for causing the operation target object in the estimated state to perform the next action, and (d) repeating (b) to (c) until the second operation is completed.

[0109] In step S806, the processor unit 230 performs the second operation of the operation object by referring to the second operation scenario generated in step S805. Specifically, first, the acquisition means 110 acquires information indicating the state of the operation object when the second operation is started, and the processor unit 230 receives the information indicating the state of the operation object from the acquisition means 110. Next, the processor unit 230 generates an operation signal for the operation object based on the acquired information and the second operation scenario. This is repeated until the second operation is completed.

[0110] In this way, a series of operations can be performed according to multiple operation scenarios.

[0111] In the above example, the steps are performed in a specific order, but the order in which the steps are performed is not limited to that shown. The steps can be performed in any order that is logically possible. For example, step S804 or step S805 may be performed before step S803 or during step S803.

[0112] In the examples described above with reference to Figures 6 and 8, the processes of the steps shown in Figures 6 and 8 are implemented by the programs stored in the processor unit 230 and the memory unit 220, or the programs stored in the processor unit 330 and the memory unit 320, but the present invention is not limited to this. At least one of the processes of the steps shown in Figures 6 and 8 may be implemented by a hardware configuration such as a control circuit.

[0113] The present invention is not limited to the above-mentioned embodiment. It is understood that the scope of the present invention should be interpreted only by the claims. It is understood that a person skilled in the art can implement an equivalent scope based on the description of the present invention and technical common sense from the description of the specific preferred embodiment of the present invention. [Industrial Applicability]

[0114] INDUSTRIAL APPLICABILITY The present invention is useful for providing a system or the like for automatically manipulating an object to be operated so as to achieve smooth movement of the object to be operated. [Explanation of symbols]

[0115] 10 Backhoe 100, 100' system 110 Acquisition means 120 Simulation Methods 130 Generation means 140 Remedies

Claims

1. A system for automatically operating an operation object, comprising: an acquisition means for acquiring initial information indicating an initial state of the operation object, the acquisition means further acquiring a plurality of pieces of information indicating states of the operation object for each of a plurality of states of the operation object when a series of operations of the operation object is being performed; a simulation means for simulating an action of the operation object based on the initial information, the simulation means being configured to generate an operation scenario, the operation scenario associating information indicating a plurality of states that the operation object may take when a series of operations of the operation object are being performed with an operation signal for causing the operation object in each state to perform a next action; a generating means for generating an operation signal for the operation target object, the generating means repeating the generation of the operation signal based on one of a plurality of pieces of information indicating a state of the operation target object and the operation scenario until the series of operations is completed; A system comprising:

2. the series of operations includes at least a first operation performed in a first time segment and a second operation performed in a second time segment; the receiving means receives first initial information indicating an initial state of the operation object in the first time segment; The simulation means generates a first operation scenario for the first operation based on first initial information; the generation means repeats generating a first operation signal for the first operation based on one of a plurality of pieces of information indicating a state of the operation target object acquired in the first time segment and the first operation scenario until the first time segment ends; the receiving means receives second initial information indicating an initial state of the operation object in the second time segment; The simulation means generates a second operation scenario for the second operation based on second initial information; The system described in claim 1, wherein the generation means repeats generating a second operation signal for the second operation based on one of a plurality of pieces of information indicating the state of the object to be operated acquired in the second time section and the second operation scenario until the second time section ends.

3. The system of claim 2 , wherein the first time interval and the second time interval partially overlap.

4. The simulation means includes: (a) estimating an operation signal for causing the operation target object in a state indicated by the initial information to perform a next action; (b) estimating a state of the operation target object after being operated with the estimated operation signal; and (c) estimating an operation signal for causing the operation target object in the estimated state to perform a next action; and (d) repeating (b) to (c) until the series of operations is completed; The system of claim 1 , configured to generate the operational scenario by:

5. The simulation means includes a machine learning model and a simulator of the operation object, the machine learning model performs the steps (a) and (c), and the machine learning model learns a relationship between information indicating a state of the operation object and an operation signal so that, when information indicating a state of the operation object is input to the machine learning model, an operation signal for causing the operation object in the state to perform a next action can be output; The system of claim 4 , wherein the simulator performs (b).

6. the simulation means comprises a machine learning model; The system of claim 4, wherein the machine learning model learns a relationship between the information indicating the state of the object to be operated and the operation signal so that, when information indicating the state of the object to be operated is input to the machine learning model, the machine learning model can output an operation signal for causing the object to be operated in the state to perform a next action and information indicating the state of the object to be operated after the next action is performed.

7. 2. The system of claim 1, further comprising a modifying means for modifying the operational scenario, the modifying means modifying the operational scenario based on an input by a user or in response to the operational scenario violating a predetermined rule.

8. The generating means includes: determining a difference between a state indicated by the information acquired by the acquisition means and a state defined in the operational scenario; determining whether the difference is greater than or equal to a threshold; The system of claim 1 , further comprising:

9. The system according to claim 1 , wherein the information indicating the state includes at least one of a joint angle of the object to be operated, position information of the object to be operated, Lidar information of the object to be operated, and image information of the object to be operated.

10. The system according to claim 1 , wherein the operation object is a construction machine.

11. A method for automatically manipulating an object to be manipulated, comprising: receiving initial information indicating an initial state of the operation object; generating an operation scenario by simulating an action of the operation object based on the initial information, the operation scenario associating information indicating a plurality of states that the operation object may take when a series of operations of the operation object are performed with an operation signal for causing the operation object in each state to perform a next action; When a series of operations of the operation object is being performed, (i) receiving information indicating a state of the operation object; (ii) generating an operation signal for the operation target object based on the acquired information and the operation scenario; Repeating the above series of operations until completion. The method includes:

12. A program for automatically operating an operation object, the program being executed on a computer having a processor, the program comprising: receiving initial information indicating an initial state of the operation object; generating an operation scenario by simulating an action of the operation object based on the initial information, the operation scenario associating information indicating a plurality of states that the operation object may take when a series of operations of the operation object are performed with an operation signal for causing the operation object in each state to perform a next action; When a series of operations of the operation object is being performed, (i) receiving information indicating a state of the operation object; (ii) generating an operation signal for the operation target object based on the acquired information and the operation scenario; Repeating the above series of operations until completion. A program causing the processor to perform a process including the steps of:

13. A system for generating an operation scenario used to automatically operate an operation target object, comprising: A receiving means for receiving initial information indicating an initial state of the operation object; a simulation means for simulating an action of the operation object based on the initial information, the simulation means being configured to generate an operation scenario, the operation scenario associating information indicating a plurality of states that the operation object can take when a series of operations of the operation object are performed with an operation signal for causing the operation object in each state to perform a next action; wherein the operation scenario is used to generate an operation signal for the operation object based on information indicating a state of the operation object.

14. A method for generating an operation scenario to be used for automatically operating an operation object, comprising the steps of: receiving initial information indicating an initial state of the operation object; generating an operation scenario by simulating an action of the operation object based on the initial information, the operation scenario associating information indicating a plurality of states that the operation object may take when a series of operations of the operation object are performed with an operation signal for causing the operation object in each state to perform a next action; wherein the operation scenario is used to generate an operation signal for the operation object based on information indicating a state of the operation object.

15. A program for generating an operation scenario used for automatically operating an operation target object, the program being executed on a computer having a processor, receiving initial information indicating an initial state of the operation object; generating an operation scenario by simulating an action of the operation object based on the initial information, the operation scenario associating information indicating a plurality of states that the operation object may take when a series of operations of the operation object are performed with an operation signal for causing the operation object in each state to perform a next action; wherein the operation scenario is used to generate an operation signal for the operation object based on information indicating a state of the operation object.

Images