Control system and control method

Abstract

This control system (2000a) is for assisting with a task that uses a device, and comprises a learning model unit (200) that outputs second information, which is information for operating a target device (2), on the basis of output data obtained by inputting, to a trained learning model, input data based on first information indicating a request related to the operation of the target device (2).

Description

Technical Field

[0004] , , , , , , ,

[0005] , , ,

[0001] The present disclosure relates to a control system and a control method.

Background Art

[0002] In recent years, the utilization of AI (Artificial Intelligence) has been progressing. In particular, AI called generative Artificial Intelligence that can generate various contents has also started to spread, and it is expected that the application areas of AI will expand. The application areas of AI are not limited to tasks within the home, but also include tasks in various facilities such as buildings, factories, stations, schools, hospitals, or commercial facilities, as well as outdoor areas such as roads, outdoor facilities, the sky, or the sea.

[0003] For example, Patent Document 1 describes a machining program generation device that generates a program for controlling a machine using a large language model.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0007] Examples of tasks performed by humans include the following: • Work performed by one person on behalf of another person or other living creatures. • Tasks performed by people on various machines This work may include tasks such as conversation, viewing, verification, operation, monitoring, instruction, mediation, and interpretation.

[0008] The examples mentioned above are just a few examples, and the work covered by this disclosure is not limited to these.

[0009] When using some kind of learning model to have an information processing device perform some or all of the tasks involved in human or object work, the validity of the model's output may become an issue. Furthermore, the validity of the model's input, which affects the model's output, may also become an issue.

[0010] Furthermore, depending on the equipment in question, appropriate control may not be possible without understanding the current situation. In such cases, how to perform situational awareness becomes a problem. In this situation, it may be necessary to recognize the situation in a continuous manner, including not only the current situation but also past situations. For example, when deciding on the next control based on the content of past control actions, the accuracy of situational awareness may be a problem in order to ensure the continuity of control.

[0011] Furthermore, in situations where immediate control of equipment is required, the response time from giving instructions to the learning model to obtaining results can become a problem.

[0012] Furthermore, model maintainability can be an issue, as the model may need to be retrained every time equipment is changed or added.

[0013] Thus, various problems still plague the use of machine learning models. Depending on the severity of the problem, the attempt to improve the efficiency or performance of a task by using a machine learning model may actually end up decreasing its efficiency or performance.

[0014] These problems when using learning models will become particularly pronounced as the task being supported becomes more complex and sophisticated.

[0015] Therefore, this disclosure aims to further improve the efficiency or performance of tasks performed by people or objects by utilizing a learning model. [Means for solving the problem]

[0016] The control system described herein is a control system for supporting work using equipment, and is a learning model that can output second information, which is information for operating the equipment, based on output data obtained by inputting input data based on first information indicating requirements for the operation of the equipment into a trained learning model. Department, The learning model unit, based on the user's credentials, determines whether the user is authorized to perform the task corresponding to the second piece of information. If the user is authorized to perform the task corresponding to the second piece of information, it outputs the second piece of information based on the output data. If the user is not authorized to perform the task corresponding to the second piece of information, it outputs the second piece of information. The learning model unit does not output output data, but rather outputs second information using the output data and equipment information, which is information about the equipment. The equipment information includes at least one of the computer-aided design information of the equipment and the computer-aided design information of the object being worked on. It is characterized by the following.

[0017] The control method according to the present disclosure is a control method for assisting work using a device. A learning model unit determines whether work performed in response to first information indicating a request regarding the operation of the device is permitted for the user based on the user's qualification information. When the work is permitted for the user, second information, which is information for operating the device, is output based on output data obtained by inputting input data based on the first information into a learned learning model. When the work is not permitted for the user, the second information is not output. Using the output data and equipment information, which is information about the equipment, a second piece of information is output, and the equipment information includes at least one of the computer-aided design information of the equipment and the computer-aided design information of the object being worked on. , which is characterized by the above.

Effects of the Invention

[0018] According to the present disclosure, by using a learning model, it is possible to further improve the efficiency or performance of work by a person or an object.

Brief Description of the Drawings

[0019] [Figure 1] It is a configuration diagram showing an example of a control system according to Embodiment 1. [Figure 2] It is an explanatory diagram showing a configuration example of a learning model unit. [Figure 3] [[ID= 23]]It is an explanatory diagram showing another configuration example of a learning model unit. [Figure 4] It is an explanatory diagram showing another configuration example of a learning model unit. [Figure 5] It is a configuration diagram showing another example of an information processing apparatus that is an operating environment of a control unit including a learning model unit. [Figure 6] It is an explanatory diagram showing an example of model learning in a model generation unit. [Figure 7] It is a flowchart showing an operation example of a control system according to Embodiment 1. [Figure 8] It is a configuration diagram showing another example of a control system according to Embodiment 1. [Figure 9] It is a configuration diagram showing another example of a control system according to Embodiment 1. [Figure 10] It is a configuration diagram showing another example of a control system according to Embodiment 1. [Figure 11] This is a configuration diagram showing an example of a control system according to Embodiment 2. [Figure 12] This is a configuration diagram showing another example of the control system according to Embodiment 2. [Figure 13] This flowchart shows an example of the operation of the control system according to Embodiment 2. [Figure 14] This is a configuration diagram showing an example of a control system according to Embodiment 3. [Figure 15] This flowchart shows an example of the operation of the control system according to Embodiment 3. [Figure 16] This is a configuration diagram showing another example of the control system according to Embodiment 3. [Figure 17] This flowchart shows an example of operation of a modified example according to Embodiment 3. [Figure 18] This is a configuration diagram showing another example of the control system according to Embodiment 3. [Figure 19] This is a configuration diagram showing another example of the control system according to Embodiment 3. [Figure 20] This is a configuration diagram showing another example of the control system according to Embodiment 3. [Figure 21] This is a configuration diagram showing another example of the control system according to Embodiment 3. [Figure 22] This flowchart shows an example of operation of a modified example according to Embodiment 3. [Figure 23] This is a configuration diagram showing an example of a control system according to Embodiment 4. [Figure 24] This flowchart shows an example of the operation of the control system according to Embodiment 4. [Figure 25] This is a configuration diagram showing another example of the control system according to Embodiment 4. [Figure 26] This flowchart shows an example of operation of a modified example according to Embodiment 4. [Figure 27] This is a configuration diagram showing an example of a control system according to Embodiment 5. [Figure 28]This flowchart shows an example of the operation of the control system according to Embodiment 5. [Figure 29] This is a configuration diagram showing another example of the control system according to Embodiment 5. [Figure 30] This is a configuration diagram showing another example of the control system according to Embodiment 5. [Figure 31] This is a configuration diagram showing another example of the control system according to Embodiment 5. [Figure 32] This is a configuration diagram showing another example of the control system according to Embodiment 5. [Figure 33] This is a configuration diagram showing an example of a control system according to Embodiment 6. [Figure 34] This is a configuration diagram showing an example of a control system according to Embodiment 6, which has a function for generating models. [Figure 35] This is a configuration diagram showing an example of a control system according to Embodiment 6, where device information is input during learning. [Figure 36] This is a configuration diagram showing an example of a control system according to Embodiment 6 when retraining is performed. [Figure 37] This is a configuration diagram showing an example of a control system according to Embodiment 6, which includes a learning device. [Figure 38] This flowchart shows an example of the operation of the control system according to Embodiment 6. [Figure 39] This figure shows an example configuration of a computer system that implements the control system of Embodiment 6. [Figure 40] This is a diagram illustrating the autoencoder of Embodiment 6. [Modes for carrying out the invention]

[0020] Hereinafter, in order to explain this disclosure in more detail, the forms for implementing this disclosure will be described with reference to the attached drawings. In the following, identical elements will be denoted by the same reference numerals and their descriptions will be omitted.

[0021] Embodiment 1. This embodiment describes an example of using a learning model to support the process of generating code for a target device.

[0022] Figure 1 is a configuration diagram showing an example of a control system 1000 according to Embodiment 1. The control system 1000 shown in Figure 1 is a control system for controlling equipment using a learning model, and comprises a learning model unit 100, an equipment information storage unit 110 (referred to as equipment information DB in the figure), and an execution code generation unit 120.

[0023] Although Figure 1 shows User 1 and Target Device 2, these may also be included as part of the control system 1000. In that case, "User 1" may be read as "User Terminal 1". The same applies to other embodiments.

[0024] When the learning model unit 100 receives input information D11, it outputs a control description D12. When the learning model unit 100 receives input information D11, it outputs a control description D12 based on the model information D102 described later.

[0025] In this embodiment, the learning model unit 100 is a model and its operating environment configured to output a control description D12 corresponding to input information D11 when input information D11 is received. Alternatively, the learning model unit 100 may be a model and its operating environment configured to generate and output a control description D12 based on input information D11, device information D13, and / or other information that can be referenced in the learning model unit 100 (such as model reference information D104 described later) when input information D11 is received.

[0026] In this embodiment, input information D11 includes information indicating the control content required for the target device 2. Input information D11 may be, for example, text, images, audio, or a combination thereof indicating the control content for the target device 2. Input information D11 may also be, for example, text, images, audio, or a combination thereof indicating multiple control contents for the target device 2. Furthermore, input information D11 may include information indicating the content of controls performed continuously over time, in which case it may be time-series data of a predetermined data structure including text, images, audio, or a combination thereof indicating the control content as described above. The method of indicating the control content is assumed to conform to the input format of the model used by the learning model unit 100, however, this does not apply if error processing, correction processing, or conversion processing is included in the preceding stage of the learning model unit 100.

[0027] An example of how to indicate the control content in input information D11 is to specify the control to be performed on target device 2, and then specify the parameter values ​​for that control and the state after the control. In this case, input information D11 may include, for example, information that identifies the control and information that indicates the parameter values ​​for that control or the state after the control. The parameter values ​​for the control may include, for example, values ​​related to the type of control (ON / OFF, etc.), direction, quantity, and time. Examples of control content include "turn on function X" for a PLC (programmable logic controller, also called a sequencer), "move the tip to point A" for a robot arm, and "lower the set temperature by 1 degree" for an air conditioner. Another example of how to indicate the control content in input information D11 is to use various types of information such as document strings (docstrings) that explain the specifications of functions, specifications, or specifications, design documents, operation commands, control codes, and source code applied to other devices such as other models.

[0028] Furthermore, in addition to the explicit method described above, the control content in input information D11 can also be indicated by showing the operation details if there is a control performed by that operation. Alternatively, it can be implicitly indicated by using, for example, the user's actions associated with a specific control, the operating results of the target device 2, or similar control commands for other models. In other words, input information D11 may include not only information that directly indicates the control content for the target device 2, but also information that indirectly indicates the control content using the corresponding operation details, the user's actions, or images of the target device 2. As an example, to indicate the control content related to the temperature control of an air conditioner, the user's words such as "It's hot," or the user's actions such as wiping sweat, rolling up sleeves, or fanning oneself, can be used. In this case, input information D11 can include text, audio, or images of the user's statements, or images (videos) of the user's actions. As another example, information can be used to indicate the control content related to the control of a robotic device's arm, such as the posture of the robotic device after control, information specifying the destination of a predetermined part, or information indicating the imitation of robotic movement or instruction movement to the robot (movement instruction by gestures such as pointing) by a person or other object (a simulator that simulates the robot's movements, including objects on a screen).

[0029] Here, the format of input information D11 is not particularly limited. For example, the information may be text, images, audio, data written in a predetermined design language, control descriptions (including source code and information written in a predetermined programming platform language), information written in other platform languages, control commands (including control instructions, control signals, control codes and controller commands), or executable code. These pieces of information may be combined as appropriate. In this disclosure, when "text" is used without particular distinction, it may include not only natural language expressed in text, but also data written in a predetermined design language that is indistinguishable from human beings, control descriptions (including source code and information written in a predetermined programming platform language), information written in other platform languages, control commands (including control instructions, control signals, control codes and controller commands), and executable code expressed in text.

[0030] The control description D12 includes control information written in a predetermined format that can be recognized by the subsequent executable code generation unit 120. The control description D12 is, for example, source code written in a predetermined programming language. Alternatively, the control description D12 may be, for example, a set of commands written in a format (platform language) handled by a predetermined programming platform. Here, the predetermined programming platform may include no-code programming platforms and low-code programming platforms.

[0031] The device information storage unit 110 stores device information D13, which is information relating to the target device 2. Device information D13 may include, for example, information indicating the function, performance, structure, dimensions, operation, and / or control method of the target device 2. Device information D13 may also include, for example, information relating to the program used to control the target device 2. Device information D13 may also be, for example, a digitized version of the manual or instruction manual for the target device 2. Digitization here includes text conversion, image data conversion, digitized audio conversion, and combinations thereof. Device information D13 is used, for example, as additional information when the learning model unit 100 outputs the control description D12.

[0032] Furthermore, device information D13 may include information indicating the status of target device 2. This information may include not only the current status of target device 2 but also information indicating its past status. For example, device information D13 may include time-series data of a predetermined data structure indicating the status of target device 2. This information may be, for example, information output from target device 2, or information input by user 1 or other devices. This information may include various types of information output from target device 2 (e.g., error information, log information, notification information, etc.). In this embodiment, the information indicating the status of target device 2 may be referred to as status information D15.

[0033] When the execution code generation unit 120 receives the control description D12 as input, it generates and outputs execution code D14, which is code that the target device 2 can execute, based on the control description D12. The execution code D14 may be, for example, a set of codes written in machine language. The execution code D14 may include, for example, information used when the target device 2 is actually controlled. The execution code D14 may output the amount of movement for each predetermined control cycle for each control cycle. The execution code D14 may be, for example, control information written in a format that the target device 2 can recognize. The execution code generation unit 120 may be, for example, a compiler that converts the control description D12 into execution code D14.

[0034] The execution code D14 output from the execution code generation unit 120 is input to the target device 2. As a result, the target device 2 operates according to the execution code D14 output from the execution code generation unit 120. The execution code D14 may be input to the target device 2 directly from the execution code generation unit 120, or it may be input indirectly through a communication network, other devices (servers, various conversion devices, etc.), or by human intervention.

[0035] Target device 2 is not particularly limited. It is assumed that target device 2 is a device capable of receiving and actually executing the executable code D14; however, this does not apply if there is an interface between target device 2 and the other device, such as a programming device, that allows target device 2 to read the executable code.

[0036] Target device 2 may be, for example, a PLC, processing machine, robot, radar, sensor, camera, projector, or communication equipment. Alternatively, target device 2 may be, for example, an air conditioner, refrigerator, television, lighting, or washing machine. Furthermore, target device 2 may be, for example, an elevator, mobility device, conveying equipment, other machinery, or a control device that controls such machinery. Also, target device 2 may be equipment operating in power generation / substation / storage plants, water treatment plants, etc., or a control device that controls other equipment. Additionally, if the control description D12 is in an interpreter language and target device 2 is capable of receiving and directly executing the control description D12, the execution code generation unit 120 is omitted.

[0037] Figure 2 is an explanatory diagram showing an example configuration of the learning model unit 100. As shown in Figure 2, the learning model unit 100 may include a model control unit 101 operating on the information processing device 10 and a model information storage unit 11 (referred to as the model information DB in the figure) that stores model information D102. Here, the model information storage unit 11 may be composed of multiple databases connected via a network.

[0038] Model information D102 includes model information. Model information D102 may include, for example, information showing the correlation between model input data D101 and model output data D103. Model information D102 may also include, for example, information showing candidates for model output data D103. Furthermore, model information D102 may include, as model information, information showing candidates for model output data D103 and information showing the relationships between those candidates. Model information D102 may also include model parameters, which are information that defines the behavior of the learning model, such as constraints, weighting variables, and evaluation functions.

[0039] The model may be a model that has been trained using supervised learning, reinforcement learning, or unsupervised learning. The model may also be a model obtained by performing training according to deep learning, genetic programming, functional logic programming, or other known algorithms and methods. The model may also be a model known as an NN (Neural Network) model, a CNN (Convolutional Neural Network) model, an RNN (Recurrent Neural Network), a VAE (Variational Autoencoder), a GAN (Generative Adversarial Networks), a diffusion model, a Transformer model, an LLM (Large Language Model), a VLM (Visual Language Model), a BERT (Bidirectional Encoder Representations from Transformers), a GPT (Generative Pre-trained Transformer), or a CLIP (Contrastive Language Image Pre-training). Furthermore, the model may be a rule-based model that obtains output results by referring to a predetermined table or making decisions based on predetermined conditions. The models mentioned above are not mutually exclusive; for example, LLM, VLM, BERT, and GPT are all included in Transformer models. Also, for example, Transformer models are included in NN models. Furthermore, learning algorithms and models may be combinations of multiple types. Models also include what are called multimodal models, which are trained using a combination of multiple different types of data.

[0040] When the model control unit 101 receives model input data D101, it outputs model output data D103 corresponding to the model input data D101 based on the model input data D101 and model information D102. When the model control unit 101 receives model input data D101, it outputs model output data D103 corresponding to the model input data D101 using, for example, the model indicated in model information D102.

[0041] The model control unit 101 is implemented, for example, by a CPU that operates according to a program provided by the information processing device 10. Hereinafter, the learning model unit 100 may be referred to as the artificial intelligence unit. Here, the artificial intelligence unit refers to an AI equipped with intelligent functions such as reasoning and judgment, and its operating environment. Therefore, the model control unit 101 may include an AI equipped with intelligent functions such as reasoning and judgment, and its operating environment. The model control unit 101 may also be, for example, an AI equipped with the learning model described above, and its operating environment. The model control unit 101 may also be an element (module) of the control unit 104 provided by the information processing device 10.

[0042] Furthermore, the learning model unit 100 may also include a reference information storage unit 12 (referred to as reference information DB12 in the figure) that stores model reference information D104, as shown in Figure 3. Here, the reference information storage unit 12 may consist of multiple databases connected via a network. The same applies to other storage units (e.g., device information storage unit, etc.) described later.

[0043] Model reference information D104 is information that the model control unit 101 refers to in order to output model output data. Model reference information D104 may include a history of previously input model data and / or a history of previously output model data. Model reference information D104 may also include information that associates features contained in past inputs with features contained in outputs performed for those inputs. Furthermore, model reference information D104 may include information that evaluates the results output for past inputs.

[0044] Furthermore, model reference information D104 may include information related to expressions or concepts contained in model input data D101. For example, model reference information D104 may include information that associates a specific expression or concept that may be contained in model input data D101 with other expressions or concepts related to that expression or concept. Here, other expressions or concepts related to a certain expression or concept include expressions or concepts that are more concrete than the expression or concept, or other expressions or concepts that are recalled based on the expression or concept. For example, model reference information D104 may include information that associates a specific expression or concept that may be contained in model input data D101 with expressions or concepts related to that expression or concept. For example, model reference information D104 may include information that associates a specific expression or concept that may be contained in model input data D101 with information related to that expression or concept. For example, model reference information D104 may include information that associates search keys and values ​​extracted from expressions or concepts that may be contained in model input data D101. Model reference information D104 may include information for so-called grounding. Model reference information D104 may also include a so-called knowledge graph that describes real-world entities and the relationships between them. In a knowledge graph, various pieces of information are systematically linked and represented in a graph structure.

[0045] Furthermore, the model reference information D104 may include information for so-called attention. For example, the model reference information D104 may include information showing the correlation between expressions or concepts that may be included in the model input data D101 and other expressions or concepts. The model reference information D104 may also include a feature map in which key information extracted from expressions or concepts that may be included in the model output data D103, which is linked to expressions or concepts that may be included in the model input data D101, is used as a feature. The model reference information D104 may also include information that associates queries extracted from expressions or concepts that may be included in the model input data D101 with key information for searching that corresponds to those queries.

[0046] In the example shown in Figure 3, when the model control unit 101 receives model input data D101, it outputs model output data D103 based on the model input data D101, model information D102, and model reference information D104.

[0047] The learning model unit 100 may also include a search engine for searching the model reference information D104 or an interface to said search engine, instead of the reference information storage unit 12. In such a case, the search range of said search engine may be an external network or a specific network. Here, as one of the external network or specific networks, a database (e.g., an equipment information DB) provided by the control system of this disclosure can be used.

[0048] The term "learning model" can refer to a computer algorithm that produces some output based on learned information in response to input information, or to the learned information itself. However, in the context of an operating environment, "learning model" often refers to the actual program that runs such a computer algorithm and its operating environment. In this disclosure, the latter is adopted, and in order to distinguish it from a mere algorithm or a set of learned information, the model that actually operates based on the information stored in model information D102 is referred to as the "learning model." The control system according to this disclosure includes a learning model unit (particularly the model control unit 101) that corresponds to such a learning model. Therefore, in the following description of the control system, "learning model" refers to the learning model unit or, in particular, the model control unit 101.

[0049] Figure 4 is an explanatory diagram showing another configuration example of the learning model unit 100. As shown in Figure 4, the learning model unit 100 may include an input unit 102, an output unit 103, and a control unit 104.

[0050] The input unit 102 accepts model input data D101. The input unit 102 may accept model input data D101 entered by user 1, etc. The input unit 102 may accept model input data D101 that constitutes time series data. In this case, the input unit 102 may accept model input data D101 that constitutes time series data sequentially, or it may accept model input data D101 that has been buffered to some extent. In addition, the input unit 102 may accept model input data D101 entered from multiple input sources. In this case, the input unit 102 may accept model input data D101 with information of the input source (e.g., user identifier, user attribute information, etc.) attached, or the input unit 102 may determine the input source and attach the information of the input source to the model input data D101 before accepting it, or it may accept it without doing anything in particular. The input unit 102 is implemented by various input devices provided by the information processing device 10 (for example, a pointing device, a keyboard, a voice input device, an image input device, a data reading device, a data input device compatible with various communication interfaces, etc.). Alternatively, the input unit 102 may be implemented by external equipment of the information processing device 10. In that case, the information processing device 10 only needs to include an interface with the input unit 102.

[0051] The output unit 103 outputs an object generated by the control unit 104. Here, the object includes model output data D103 or data generated from the model output data D103. Furthermore, if the object generated by the control unit 104 contains information for multiple output destinations, the output unit 103 may output the object to multiple output destinations. In this case, the output unit 103 may output the same data to multiple output destinations, or it may output different data to each output destination. The output unit 103 is implemented by various output devices provided by the information processing device 10 (e.g., display device, audio output device, image output device, data writing device, data output device compatible with various communication interfaces, etc.). Note that the output unit 103 may also be implemented by an external device of the information processing device 10. In that case, the information processing device 10 only needs to include an interface with the output unit 103.

[0052] The control unit 104 operates on the information processing device 10 and includes, in addition to the model control unit 101 described above, a pre-processing unit 105 and a post-processing unit 106.

[0053] The preprocessing unit 105 performs processing to improve the accuracy of the object generated by the control unit 104. For example, the preprocessing unit 105 may add, change or delete elements to the model input data D101, or perform data transformation (including processing).

[0054] The preprocessor 105 may, for example, when the input unit 102 receives the model input data D101, modify elements of the model input data D101 (including additions and deletions) or transform the data (including processing). Modifying elements or transforming the data (including processing) includes not only changing the data format but also changing the representation or concept that the data represents. The subsequent model control unit 101 receives the modified data from the preprocessor 105 as the model input data D101. The processing performed by the preprocessor 105 includes so-called prompt formatting for the model control unit 101.

[0055] The preprocessing unit 105 may, for example, decompose the model input data D101 into predetermined unit data. Alternatively, the preprocessing unit 105 may, for example, integrate multiple model input data D101. Furthermore, the preprocessing unit 105 may decompose the model input data D101 into predetermined unit data and then modify the elements or convert the data, or it may integrate multiple model input data D101 and then modify the elements or convert the data.

[0056] The post-processing unit 106 corrects the object if, for example, there is a problem with the object generated by the control unit 104 (particularly the model control unit 101). The post-processing unit 106 may, for example, use the knowledge graph described above to determine whether or not there is a problem with the object. For example, it may compare the similarity between the relationships shown by the knowledge graph and the relationships between the representations or concepts included in the model input data and the representations or concepts included in the model output data, and / or the relationships between the representations or concepts included in the model output data, and determine that there is a problem with the object if it is more than a predetermined distance away from the relationships shown by the knowledge graph.

[0057] Of the components mentioned above, all components except the model control unit 101 are not mandatory, and their implementation can be selected as appropriate.

[0058] Furthermore, the model information D102 and other information used by the learning model may be pre-prepared, or they may be obtained via a communication network as needed.

[0059] Figure 5 is a configuration diagram showing another example of an information processing device 10 as an operating environment, including a control unit 104 that includes a learning model unit 100. The information processing device 10 shown in Figure 5 may include a control unit 104a that includes a learning model unit 100 (particularly a model control unit 101), an input processing unit 201, an output confirmation unit 202, and a correction confirmation unit 203.

[0060] The input processing unit 201 receives input information D11 from an input source 1a, such as user 1. The input processing unit 201 also outputs the received input information D11 as model input data D101 to the learning model unit 100.

[0061] In this case, the input processing unit 201 may, for example, output model input data D101 after modifying elements or transforming data in the input information D11. The input processing unit 201 may, for example, remove noise from the input information D11. The input processing unit 201 may also, for example, convert qualitative information in the input information D11 into quantitative information. The input processing unit 201 may also, for example, correct the quantity in the input information D11 according to the equipment and its operating environment that are the target of the input information D11's requirements. The input processing unit 201 may also, for example, perform so-called grounding processing, that is, change the expression or concept shown in the input information D11 to a more concrete expression or concept.

[0062] Furthermore, the input processing unit 201 may return a query to the input source if the input information D11 contains unclear or uncertain information. The input processing unit 201 may output a message as a query, for example, a message to confirm the input content, a message to propose a correction to the input information D11, or a message requesting re-input of the input information D11 with a different state or expression. The proposed correction to the input information D11 may also be generated by the correction confirmation unit 203, which will be described later. In the following, information indicating correction, addition, or cancellation of content in the input and output data of the learning model after input and output may be referred to as supplementary information D18. The proposed correction is an example of supplementary information D18.

[0063] The output verification unit 202 performs a simulation that simulates the control and state of the target device 2 based on the model output data D103 output from the learning model unit 100. The output verification unit 202 may perform the simulation after converting the model output data D103 into control information that matches a predetermined simulator (not shown) capable of simulating the control and state of the target device 2. The output verification unit 202 may also have the functionality of a simulator. When performing the simulation, the output verification unit 202 may use information obtained from the output destination 2a of the model output data D103. Here, the output destination 2a includes the output destination of information generated from the model output data D103. The information obtained from the output destination 2a may include, for example, state information D15 and / or feedback information D16, which will be described later.

[0064] The output verification unit 202 may, for example, verify the status of the target device 2, the status of the system including the target device 2, and / or the status of the workpieces held by the target device 2. Furthermore, before performing operational verification, the output verification unit 202 may generate and display human-readable intermediate products for the model output data D103 or the information generated therefrom. Examples of intermediate products include source code for the control program and operation images of the controller of the target device 2 for operation commands to the target device 2. The output verification unit 202 may also display the simulation results along with a reliability index of the learned model.

[0065] The following is an example of a reliability index for a learning model. For example, during pre-training, the results of human evaluations of the input to the learning model could be accumulated, and an evaluation network that has learned inputs and evaluation results could be established. When using the learning model, the input to the learning model could also be input to the aforementioned evaluation network, and its output could be used as a reliability index.

[0066] Alternatively, for example, a learner may be provided to cluster the output of the trained model during pre-training, and when using the trained model, the output of the trained model may also be input to the aforementioned learner, and the result of that clustering may be used as a reliability indicator.

[0067] Alternatively, for example, during pre-training, the results of human evaluations of inputs to the learning model could be accumulated, and an evaluation network could be equipped with features that have received high evaluations. When using the learning model, the inputs to the learning model could also be input to the aforementioned evaluation network, and the similarity between the output features and the learned features could be used as a reliability indicator.

[0068] Alternatively, for example, during pre-training, the system could be equipped with a learner that accumulates the results of human evaluations each time an input is received into the learning model, and clusters the inputs of the learning model that receive high evaluation results. When using the learning model, the inputs of the learning model could also be input into the aforementioned learner, and the results of that clustering could be used as a reliability index.

[0069] The correction verification unit 203 uses the results of the simulation performed by the output verification unit 202 to determine the validity of the model output data D103 and / or model input data D101. The correction verification unit 203 may, for example, determine the validity of the model output data D103 and / or model input data D101 by comparing the state of the target device 2 shown in the simulation results with the state of the target device 2 identified by the input information D11, model output data D103 and / or model input data D101, and determining whether correct control is being performed. The correction verification unit 203 may also determine that correct control is being performed if the state of the target device 2 shown in the simulation results matches the state of the target device 2 identified by the model output data D103 and / or model input data D101. The state of the target device 2 to be compared here is not limited to one.

[0070] Furthermore, the correction verification unit 203 may determine the validity of the model output data D103 and / or model input data D101 by, for example, checking whether the state or control trajectory of the target device 2 shown in the simulation results matches the control shown in the input information D11, or whether it does not contain any content that has been prohibited in advance.

[0071] Alternatively, the correction verification unit 203 may determine the validity of the model output data D103 and / or model input data D101 by presenting the simulation results to the input source 1a of the input information D11 and receiving a response indicating whether the desired control is being performed.

[0072] If the correction verification unit 203 determines that the model output data D103 and / or the model input data D101 are incorrect, it may correct the model input data D101. Alternatively, instead of correcting the model input data D101, the correction verification unit 203 may generate supplementary information D18 for the input information D11 and output it to the input source 1a.

[0073] The control system 1000 may, for example, have a configuration as shown in any of Figures 1 to 5 as the operating environment for the learning model unit 100. In this case as well, some or all of the configuration may be an internal or external configuration of the control system 1000.

[0074] It should be noted that the configuration of the learning model unit 100 and its surrounding components described above is merely an example, and not all components are mandatory. You can choose whether or not to implement them as appropriate depending on the desired functionality.

[0075] Figure 6 is an explanatory diagram illustrating an example of model learning. As shown in Figure 6, the model information D102 may be generated, for example, by the model generation unit 107 performing machine learning using the model learning data D105.

[0076] The model generation unit 107 is a processing unit that generates or updates model information D102 based on the input model training data D105 according to a predetermined algorithm. The model generation unit 107 is implemented, for example, by a CPU that operates according to a program provided by the information processing device 20. Here, the algorithm followed by the model generation unit 107 may be a machine learning algorithm corresponding to the training model, such as supervised learning, reinforcement learning, or unsupervised learning, or it may be deep learning, genetic programming, functional logic programming, or other known algorithms.

[0077] Furthermore, the model generation unit 107 may generate or update model information D102 based on the input model training data D105 and model reference information D104. Alternatively, the model generation unit 107 may generate or update model information D102 based on the input model training data D105 and model output data D103 from the model control unit 101.

[0078] The model training data D105 is not particularly limited. For example, when supervised learning is used as the training algorithm, the model training data D105 may include candidate model input data D101 and corresponding candidate model output data D103. The model training data D105 may also include the model input data D101 that has actually been input and / or the model output data D103 that has actually been output. Feedback control can be performed by appropriately using the actual model input data D101 and / or model output data D103. The model training data D105 may also include information obtained from equipment or processing units of the system in which the trained model actually operates.

[0079] The model information D102 generated or updated by the model generation unit 107 is stored in the model information storage unit 11 and provided to the model control unit 101. Alternatively, the model generation unit 107 can directly output the model information D102 to the model control unit 101.

[0080] The model generation unit 107 may, for example, generate model information D102 using the input model training data D105 through pre-training before the model control unit 101 uses the model information D102, and store it in the model information storage unit 11.

[0081] The update of model information D102 by the model generation unit 107 may be a process known as Fine-Tuning (or FineTune).

[0082] The model generation unit 107 may be included in the control system 1000, or it may be included in a system separate from the control system 1000.

[0083] Furthermore, although Figure 1 shows the learning model unit 100 and the device information storage unit 110 and device information D13 separately, the device information storage unit 110 and device information D13 may be part of the learning model unit 100. That is, the learning model unit 100 may include the device information storage unit 110 and device information D13. For example, the learning model unit 100 may include the device information storage unit 110 as one of the reference information storage units 12 described later. Also, by using the device information D13 in the model learning phase in which the model used by the learning model unit 100 is learned, the model may be pre-loaded with the device information D13. In that case, the device information storage unit 110 may be omitted.

[0084] Furthermore, part or all of the learning model unit 100 may be an internal component of the control system 1000, or an external component of the control system 1000. If it is an external component of the control system 1000, the control system 1000 may have an interface that allows it to exchange information with an external system that includes part or all of the learning model unit 100, instead of part or all of the learning model unit 100. For example, the control system 1000 may have an external model information storage unit 11, called the core of the learning model. Alternatively, for example, the control system 1000 may have an external model information storage unit 11, called the core of the learning model, and a model control unit 101 that handles the model's algorithm.

[0085] Hereinafter, in the control system 1000, in order to distinguish between the model control unit 101, which is responsible for the algorithm of the learning model, and the part that processes requests to the model control unit 101 and obtains responses, the latter part may be referred to as the "model processing unit." More specifically, the model processing unit corresponds to the part of the information processing device 10, control unit 104, or control unit 104a other than the model control unit 101. The model processing unit may be implemented, for example, by an OS (Operating System), a prompt application (and its operating environment), which calls the learning model application, running on the information processing device 10, if the model control unit 101 is located in an internal environment. The model processing unit may be implemented, for example, by a browser, a client application (and its operating environment), which runs on the information processing device 10, if the model control unit 101 is located in an external environment.

[0086] Furthermore, the configuration of the learning model and its operating environment as an information processing device, as well as the relationship between the learning model and the control system equipped therewith, are the same in other embodiments.

[0087] In this embodiment, input information D11 corresponds to model input data D101. Also, control description D12 corresponds to model output data D103. The learning model unit 100 (in particular, the model control unit 101) may be configured to, for example, receive input information D11 and output a control description D12 corresponding to the input information D11 based on model information D102 and, if necessary, model reference information D104.

[0088] In such cases, the model generation unit 107, which is provided in correspondence with the learning model unit 100, may perform machine learning using model learning data D105, which includes candidate input information D11 that can be input to the model control unit 101, to generate or update model information D102. Alternatively, the model generation unit 107 may perform machine learning using model learning data D105, which includes candidate input information D11 that can be input to the model control unit 101 and candidate control descriptions D12 that correspond to them, to generate or update model information D102.

[0089] In this embodiment, the learning model unit 100 may be, for example, a language learning model such as an LLM that takes natural language as input and obtains an output result, and its operating environment. Alternatively, the learning model unit 100 may be, for example, an image learning model such as a VLM that takes an image as input and obtains an output result, and its operating environment. Alternatively, the learning model unit 100 may be, for example, a multimodal model that takes natural language and an image as input and obtains an output result, and its operating environment. In this case, the input information D11 may be input as text data, image data, a combination of text data and image data, or a data format that can be converted to these (such as audio data, or a video which is a combination of audio data and image data). Note that the learning model used in this embodiment is not limited to the models described above.

[0090] In this embodiment, the input information D11 received by the control system 1000 can be said to be information relating to the requirements in the work environment, in this case the environment in which the target device 2 operates (in this case, the control content required for the target device 2). Therefore, the input information D11 received by the control system 1000 can be said to be an example of first information indicating the requirements in the work environment. Furthermore, the control description D12 and execution code D14 can be said to be information used for the work (work relating to the control of the target device 2) in correspondence with such input information D11. Hereinafter, the control description D12 output to a predetermined output destination from the operating environment of the learning model to which the model input data based on the input information D11 is input may be referred to as second information.

[0091] Here, in the relationship between input information D11 and model input data, when we refer to model input data based on input information D11, it may include input information D11 itself, input information D11 converted into a format that matches the input of the learning model, or input information D11 supplemented by other information. Similarly, in the relationship between model output data and second information, when we refer to second information based on model output data, it may include model output data itself, model output data converted into a format that matches the input of the output destination, or model output data supplemented by other information. The same applies to the relationship between input / output information and model input / output data in other embodiments.

[0092] Next, the operation of the control system 1000 of this embodiment will be described. Figure 7 is a flowchart showing an example of the operation of the control system 1000.

[0093] In the example shown in Figure 7, the control system 1000 first receives input information D11 (step S110). For example, the input unit 102 or input processing unit 201 described above may receive the input information D11. The received input information D11 is input to the learning model unit 100 as model input data D101.

[0094] In step S110, the control system 1000 may accept multiple input information D11. Alternatively, the control system 1000 may interact with user 1, that is, repeatedly input and output information related to input information D11 with user 1, to accept input information D11 that better matches user 1's requirements.

[0095] Next, the control system 1000 performs a control description D12 generation process using the learning model unit 100 (step S111). In step S111, the learning model unit 100 generates and outputs a control description D12 corresponding to the input information D11. For example, the learning model unit 100 (more specifically, the model control unit 101) outputs a control description D12 corresponding to the input information D11 based on model reference information D104, which includes model information D102, the input information D11, and, if necessary, equipment information D13. The learning model unit 100 may, for example, use a learning model capable of generating text data to generate a control description D12 of text data from the input information D11.

[0096] In step S111, the learning model unit 100 (more specifically, the pre-processing unit 105 or the input processing unit 201) may further add, change, or delete elements, or convert (including processing) data, to the input information D11 in order to improve the accuracy of the control description D12 before processing by the model control unit 101. Alternatively, in step S111, the learning model unit 100 (more specifically, the post-processing unit 106) may further determine whether there is a problem with the control description D12 after processing by the model control unit 101, and if it determines that there is a problem, it may perform a process to correct the control description D12.

[0097] The control description D12 output from the learning model unit 100 is input to the execution code generation unit 120. Upon receiving the control description D12, the execution code generation unit 120 generates execution code D14 based on the input control description D12 (step S112).

[0098] Next, the execution code D14 generated by the execution code generation unit 120 is input to the target device 2 (step S113). As already explained, the input of the execution code D14 to the target device 2 may be directly from the control system 1000 (more specifically from the execution code generation unit 120), or it may be input indirectly via a communication network, other devices (servers, various conversion devices, etc.) or by human intervention.

[0099] As a result, target device 2 operates according to the input execution code D14.

[0100] If the control system 1000 outputs execution code D14 and there is a change in the state of the target device 2, such as when the target device 2 is controlled, it may acquire state information D15 (step S114). The acquired state information D15 is stored in the device information storage unit 110, for example, as part of the device information D13. The control system 1000 may, for example, use the acquired state information D15 to update the device information D13 stored in the device information storage unit 110. The control system 1000 may also, for example, output the acquired state information D15 as information indicating the control result to user 1, the learning model unit 100, or other devices not shown. If the control system 1000 does not use the state information D15, the processing in step S114 may be omitted.

[0101] The control system 1000 may repeat the process of steps S110 to S114 multiple times (for example, until the desired control is completed for the target device 2).

[0102] The control system 1000 may output the control description D12 to the user 1's terminal or the like, so that user 1 can confirm the contents and then perform subsequent processing (such as code generation by the execution code generation unit 120) based on user 1's actions.

[0103] The state information D15 input to the learning model unit 100 is used, for example, for additional learning of the learning model unit 100. The learning model unit 100 may update the model information D102 and / or model reference information D104 based on the input state information D15.

[0104] As described above, according to this embodiment, execution code D14 can be generated from input information D11 input by user 1 without user 1 having to create a control description D12, thus improving the efficiency of the work of controlling the target device 2.

[0105] Furthermore, in this embodiment, the input information D11 may be text, images, audio, or a combination thereof that explicitly or implicitly indicates the control content for the target device 2. Therefore, the effort required to input the input information D11 can be further reduced while improving the efficiency of the work of controlling the target device 2.

[0106] Furthermore, according to this embodiment, since a control description D12 can be generated from input information D11 using a learning model, even if user 1 does not know the detailed specifications of the target device 2 or the specifications of the control description D12, or any other information necessary to control the target device 2, a control description D12 corresponding to the input information D11 can be generated, thereby improving the performance of the work of controlling the target device 2. Here, improving the performance of the work of controlling the target device 2 also includes improving the accuracy of the control of the target device 2.

[0107] Furthermore, in this embodiment, the state information D15 obtained after controlling the target device 2 based on the input information D11 can be used to generate the next control description D12, etc., thereby further improving the performance of the operation to control the target device 2.

[0108] In the example described above, only one target device 2 is shown, but there may be multiple target devices 2 that the control system 1000 controls. In such cases, for example, the input information D11 may include information that allows for the identification of the target device 2, or the input side to the learning model unit 100 (input unit 102, pre-processing unit 105, input processing unit 201) may perform processing to identify the target device 2 based on the input information D11, or the learning model unit 100 may output control content in which the target device 2 has been identified as a result of learning.

[0109] Variation 1-1. Next, a modified version of the control system 1000 will be described. Figure 8 is a configuration diagram showing an example of control system 1000a, which is a modified version of control system 1000 according to this embodiment. Elements identical to those in control system 1000 are denoted by the same reference numerals and their descriptions are omitted.

[0110] In the control system 1000a shown in Figure 8, the user 1 confirms the output from the learning model unit 100 and then inputs it to the subsequent execution code generation unit 120.

[0111] In this embodiment, User 1 can check the control description D12 output from the learning model unit 100 and input input information D11 based on the check result. In addition to the control description D12 output from the learning model unit 100, User 1 may also check the feedback information D16 from the execution code generation unit 120 and / or the target device 2 and input input information D11 based on the check results. In this case, User 1 may input new input information D11, or input input information D11 indicating modification, addition, or cancellation of previously input content. At this time, the input information D11 may include commands for the learning model unit 100. For example, User 1 may input commands to remove defects included in the input information D11 or defects included in the output control description D12, along with the feedback information D16, as input information D11. Here, commands to remove defects may also include input to search for the cause of the defect or a method to solve the defect.

[0112] The feedback information D16 may include the response to a request for control from a processing unit downstream of the learning model unit 100, which is returned by the processing unit. The feedback information D16 may also include information obtained from the processing unit after a request for control has been made to the processing unit downstream of the learning model unit 100. For example, the feedback information D16 may include the response to a request returned by the execution code generation unit 120 when the control description D12 is input to the execution code generation unit 120 and the generation of the control description D12 is requested. The feedback information D16 may also include the response returned by the target device 2 when the execution code D14 is input to the target device 2 and the execution of the code is requested. The feedback information D16 may include state information D15. The feedback information D16 may be output directly to the user 1, or it may be output to the user 1 via an output device (not shown) provided by the execution code generation unit 120 or the control system 1000.

[0113] Furthermore, the feedback information D16 may include information for determining whether the control requested from the processing unit downstream of the learning model unit 100 is correctly executed by that processing unit. This information is not limited to information directly obtained from that processing unit. For example, it may be information obtained from another person, device, network, or AI (none of which are shown). The feedback information D16 may also include analytical information for determining whether the execution code D14 can correctly execute the intended control, such as execution time or control trajectory information. User 1 can, for example, instruct the learning model unit 100 to control the timing of the flow in the control description D12 or adjust the lead time based on such information contained in the feedback information D16.

[0114] Furthermore, User 1 may, for example, use the feedback information D16 to interact with the learning model unit 100 multiple times and determine the validity (whether there are any problems) of the output control description D12 each time. If User 1 determines that there are no problems with the control description D12, User 1 may output the control description D12 to the execution code generation unit 120.

[0115] The feedback information D16 can be obtained, for example, in step S114 described above.

[0116] In Figure 8, an example is shown in which User 1 inputs the control description D12 to the execution code generation unit 120. However, the input of the control description D12 to the execution code generation unit 120 can also be performed by the learning model unit 100 upon receiving instructions from User 1.

[0117] In this example, the control description D12 may include, for example, a description that corresponds to low-code or no-code.

[0118] In this example, the exchange of information between User 1 and the learning model unit 100 may be carried out, for example, via a terminal provided by User 1, or via a user interface (for example, an input unit 102) provided by the information processing device 10 on which the learning model unit 100 operates.

[0119] Furthermore, the update of input information D11 in this example may be configured to be performed by the control system 1000a (for example, the correction confirmation unit 203) rather than by user 1.

[0120] Furthermore, the feedback information D16 may be input to the learning model unit 100. The feedback information D16 input to the learning model unit 100 may be used, for example, for additional learning of the learning model unit 100. The learning model unit 100 may update the model information D102 and / or model reference information D104 based on the input feedback information D16.

[0121] Other aspects may be the same as other control systems according to this embodiment.

[0122] As described above, in this modified example, user 1 can modify the input information D11 by checking the control description D12 output from the learning model unit 100 and exchanging additional instructions and bug reports with the learning model unit 100, thereby improving the accuracy of the output control description D12. As a result, the efficiency and performance of the work of controlling the target device 2 can be improved.

[0123] Variation 1-2. Next, a second modified example of the control system 1000 will be described. Figure 9 is a configuration diagram showing an example of control system 1000b, which is a modified example of control system 1000. Elements identical to those in control system 1000 and control system 1000a are denoted by the same reference numerals and their descriptions are omitted.

[0124] In the control system 1000b shown in Figure 9, the difference is that the learning model unit 100 returns query D17 to user 1. Examples of query D17 include, for example, questioning unclear or uncertain input information D11, asking for a solution, or requesting re-input of a state or representation that has been changed. As a questioning of unclear or uncertain input information D11, the learning model unit 100 may output query D17 to user 1, along with the presentation of a reference location, requesting input of more specific information. Alternatively, as a questioning of a solution, the learning model unit 100 may output query D17 to user 1, along with the presentation of a reference location, providing candidate solutions as options. Furthermore, as a questioning of a solution, the learning model unit 100 may output query D17 to user 1, along with the presentation of a reference location, providing information on the most likely solution and asking whether it is correct or not. Alternatively, the learning model unit 100 may first generate an intermediate control description, which is an intermediate control description that is easy for humans to understand, and output the generated intermediate control description along with a query D17 asking for its correctness to the user 1.

[0125] The output of query D17 may be generated, for example, after step S110 described above.

[0126] When the learning model unit 100 receives a response from user 1 to inquiry D17, it may update the input information D11 or finalize the interpretation (meaning) of the input information D11.

[0127] The processing of the learning model unit 100 described above in this example can also be implemented, for example, as part of the input unit 102 or preprocessing unit 105 of the learning model unit 100, or as part of the input processing unit 201 (not shown) of the information processing device 10.

[0128] As described above, in this modified example, a query D17 is output to user 1 regarding the input information D11, and the input information D11 is updated or its interpretation is finalized based on the response, thereby eliminating uncertainty in the input information D11. As a result, the accuracy of the output control description D12 can be improved, and consequently, the efficiency and performance of the work of controlling the target device 2 can be improved.

[0129] Variant examples 1-3. Next, a third modified example of the control system 1000 will be described. Figure 10 is a configuration diagram showing an example of control system 1000c, which is a modified example of control system 1000. Elements identical to those in control systems 1000, 1000a, and 1000b are denoted by the same reference numerals and their descriptions are omitted.

[0130] The control system 1000c shown in Figure 10 further includes a state acquisition unit 130. The state acquisition unit 130 acquires feedback information D16 indicating the processing result or state information D15 indicating the state of the device after processing from the processing destination of the control description D12 output from the learning model unit 100 and the execution code D14 generated therefrom. Here, the feedback information D16 or state information D15 may include information for determining whether the execution code D14 was able to correctly execute the intended control, such as execution time or control trajectory information.

[0131] The state acquisition unit 130 may, for example, input the acquired information to the learning model unit 100. The state acquisition unit 130 may also, for example, update the device information D13 based on the acquired information. The state acquisition unit 130 may also, for example, generate information to supplement (including addition, modification, and cancellation) the input information D11 based on the acquired information and input it to the learning model unit 100 as supplementary information D18. The state acquisition unit 130 may also, for example, generate information to supplement (including addition, modification, and cancellation) the control description D12 based on the acquired information and input it to the learning model unit 100 as supplementary information D18.

[0132] The state acquisition unit 130 may, for example, generate a control command with new content, or a command indicating the addition, modification, or cancellation of content already entered in the input information D11, as supplementary information D18 and input it to the learning model unit 100. Alternatively, the state acquisition unit 130 may, for example, input a command to remove a defect included in the input information D11 or a defect included in the output control description D12, along with the acquired information, as supplementary information D18 to the learning model unit 100.

[0133] The status acquisition unit 130 may, for example, determine whether the acquired information indicates normal processing or a normal state at the processing destination, and if not, it may input supplementary information D18 indicating a modification, addition, or cancellation of the content indicated by the input information D11 already entered, along with the acquired information, to the learning model unit 100.

[0134] The learning model unit 100 may update the model information D102 and / or model reference information D104 based on the input information (state information D15, feedback information D16, supplementary information D18, etc.).

[0135] The generation of supplementary information D18 may be performed, for example, in step S115 described above. Furthermore, the output destinations of supplementary information D18 may include devices other than the learning model unit 100. For example, the control system 1000 may output the supplementary information D18 generated by the state acquisition unit 130 to user 1 or other devices (not shown).

[0136] Furthermore, the state acquisition unit 130 may acquire the operation results from a simulator (not shown) of the target device 2 or the operation results of the target device 2 in debug mode without actually operating the target device 2. The debug mode of the target device 2 is a mode in which the execution code is executed on the control board of the target device 2, but no actual device control is performed, and only the internal state is updated, and is also called the idle operation mode. By using the debug mode, the execution code D14 can be safely tested on the target device 2 in a state close to actual control.

[0137] Not limited to this modified example, as a method for determining the validity of the control description D12 output from the learning model unit 100 and the execution code D14 generated from the control description D12 without actually operating the target device 2, the execution code generation unit 120 may be connected to the target device 2 and a simulator in a way that allows switching between them as the output destination for the execution code D14. The simulator may include one that operates the icon of the target device 2 in an augmented reality space. Furthermore, when outputting the execution code D14 to the target device 2, the execution code generation unit 120 may include information indicating whether to execute in normal mode or debug mode.

[0138] The processing of the state acquisition unit 130 in this example can also be implemented as part of the input unit 102, pre-processing unit 105 and post-processing unit 106 of the learning model unit 100, or as part of the input processing unit 201, output confirmation unit 202 and correction confirmation unit 203 (none of which are shown) of the information processing device 10.

[0139] Other aspects may be the same as other control systems according to this embodiment.

[0140] As described above, in this modified example, the state acquisition unit 130 acquires feedback information D16 indicating the processing result or state information D15 indicating the state of the device after processing from the target device 2 or the execution code generation unit 120, which is the output destination for the model output data D103 and / or information generated therefrom, in response to the input information D11. Based on the acquired information, it issues supplementary information D18 to the learning model unit 100 as appropriate. This makes it possible to improve the accuracy of the control description D12, and consequently improve the efficiency and performance of the work of controlling the target device 2.

[0141] Furthermore, in this modified example, for example, a person and a machine (state acquisition unit 130) can cooperate to improve the accuracy of input to the learning model unit 100, thus contributing to reducing the workload of user 1.

[0142] Embodiment 2. Next, we will describe Embodiment 2. In this embodiment, we will describe an example of using a learning model to support tasks related to the control of a target device.

[0143] The following discussion considers the control of various control devices, such as PLCs, processing machines, robots, sensors, conveying equipment, and other machine control devices, within a factory. While skilled workers may be familiar with the control methods of a wide variety of complex control devices, reassignments or other changes may necessitate that less experienced workers perform these operations. Furthermore, the introduction of new control devices (including version upgrades) necessitates informing all workers of the appropriate control methods for the new devices; insufficient dissemination of this information could lead to errors.

[0144] In such cases, it is desirable to be able to reliably perform the desired control even without knowing the specific control method, such as control commands, control signals, control codes, or commands to the controller corresponding to the control device, as this leads to increased work efficiency and performance.

[0145] Furthermore, the situations in which the equipment is controlled are not limited to within a factory, nor are the applications of this embodiment limited to within a factory.

[0146] Figure 11 is a configuration diagram showing an example of a control system 2000 according to Embodiment 2. The control system 2000 shown in Figure 11 is a control system for controlling equipment using a learning model, and comprises a learning model unit 200 and an equipment information storage unit 210 (referred to as equipment information DB in the figure).

[0147] The learning model unit 200 outputs a control command D22 when input information D21 is received. For example, when input information D21 is received, the learning model unit 200 outputs a control command D22 based on model information D102. The configuration of the learning model unit 200 may be basically the same as that of the learning model unit 100 in Embodiment 1.

[0148] In this embodiment, the learning model unit 200 is a model and its operating environment configured to output a control command D22 corresponding to input information D21 when input information D21 is received. Alternatively, the learning model unit 200 may be a model and its operating environment configured to generate and output a control command D22 based on input information D21, device information D23, and other information that can be referenced by the learning model unit 200 when input information D21 is received.

[0149] In this embodiment, input information D21 includes information indicating the control content for the target device 2. Input information D21 may be, for example, text, images, audio, or a combination thereof indicating the control content for the target device 2. Input information D21 may be, for example, text, images, audio, or a combination thereof indicating multiple control contents for the target device 2. Furthermore, input information D21 may include information indicating control contents that are performed continuously over time, in which case it may be time-series data of a predetermined data structure including text, images, audio, or a combination thereof indicating the control contents as described above. The method of indicating the control contents is assumed to conform to the input format of the model used by the learning model unit 200, however, this does not apply if error processing, correction processing, or conversion processing is included in the preceding stage of the learning model unit 200.

[0150] The way in which the control content is shown in input information D21 may be the same as in Embodiment 1, for example. For example, the control to be performed on the target device 2 may be specified, and then the parameter values ​​for performing that control or the state after the control may be specified. In that case, input information D21 may include, for example, information that specifies the control and information that indicates the parameter values ​​for performing that control or the state after the control. Furthermore, input information D21 may include not only information that directly shows the control content for the target device 2, but also information that indirectly shows it using the operation content corresponding to that control content, the words and actions of user 1, an image of the target device 2, or similar control commands for other models, etc.

[0151] Control command D22 includes information regarding the control of target device 2, expressed in a predetermined format that allows identification of the target device 2 or the interface requesting control from target device 2. Control command D22 may also include information indicating a control request to target device 2. Control command D22 may be, for example, a control command, control signal, or control code for target device 2. Alternatively, control command D22 may be, for example, a command written in a format handled by a predetermined controller corresponding to target device 2.

[0152] The device information storage unit 210 stores device information D23, which is information relating to the target device 2. The handling of the device information storage unit 210 and device information D23 is basically the same as that of the device information storage unit 110 and device information D13 in Embodiment 1. In this embodiment, device information D23 may include, for example, information used for controlling the target device 2. Device information D23 is used, for example, as additional information when the learning model unit 200 outputs a control command D22. Hereinafter, in this embodiment, information indicating the state of the target device 2 may be referred to as state information D25.

[0153] In this embodiment, the learning model unit 200 may be, for example, a language learning model such as an LLM that takes natural language as input and obtains an output result, and its operating environment. Alternatively, the learning model unit 200 may be, for example, an image learning model such as a VLM that takes an image as input and obtains an output result, and its operating environment. Alternatively, the learning model unit 200 may be, for example, a multimodal model that takes natural language and an image as input and obtains an output result, and its operating environment. In this case, the input information D21 may be input as text data, image data, a combination of text data and image data, or a data format that can be converted to these (such as audio data, or a video which is a combination of audio data and image data). Note that the learning model used in this embodiment is not limited to the models described above.

[0154] In this embodiment, for the sake of simplicity, the reference numerals of the components corresponding to the learning model unit 100 may be used to describe the components corresponding to the learning model unit 200. However, it should be noted that these components are provided specifically in relation to the learning model unit 200. The same applies to other embodiments.

[0155] In this embodiment, input information D21 corresponds to model input data D101. Control command D22 corresponds to model output data D103. The learning model unit 200 (in particular, the model control unit 101) may be configured to, for example, receive input information D21 and output a control command D22 corresponding to the input information D21 based on model information D102 and, if necessary, model reference information D104.

[0156] In such cases, the model generation unit 107, which is provided in correspondence with the learning model unit 200, may, for example, perform machine learning using model learning data D105 that includes candidate input information D21 that can be input to the model control unit 101, to generate or update model information D102. Alternatively, the model generation unit 107 may, for example, perform machine learning using model learning data D105 that includes candidate input information D21 that can be input to the model control unit 101 and candidate control commands D22 that correspond to them, to generate or update model information D102.

[0157] Symbol D26 is feedback information indicating the control result in the target device 2. In this embodiment as well, state information D25 and / or feedback information D26 may be obtained from the output destination of model output data D103 and / or information generated therefrom. The control system 2000 may, for example, output the obtained state information D25 and / or feedback information D26 as information indicating the control result to user 1, the learning model unit 200, or other devices not shown. The control system 2000 can also generate supplementary information D28 for the input / output data of the learning model unit 200 based on the obtained state information D25 and / or feedback information D26 and issue it to user 1, the learning model unit 200, or other devices not shown. Furthermore, the control system 2000 may be configured to return a query D27 to user 1 if the input information D21 contains unclear or uncertain information. The handling of query D27 is the same as query D17 in Embodiment 1.

[0158] Figure 12 is a configuration diagram showing another example of the control system 2000. As shown in Figure 12, the control system 2000 may further include a state acquisition unit 230 that acquires state information D25 and / or feedback information D26 and issues supplementary information D28. The state acquisition unit 230 is similar to the state acquisition unit 130 of Embodiment 1.

[0159] In this embodiment as well, the target device 2 is not particularly limited. It is assumed that the target device 2 is a device that can actually be controlled upon receiving the control command D22, but this does not apply if a conversion device that converts various signals, such as a controller or converter, is included between the target device 2 and the other device. In that case, the conversion device can receive the control command D22 and control the target device 2.

[0160] In this embodiment, the input information D21 received by the control system 2000 can be said to be information relating to the requirements in the work environment, in this case the environment in which the target device 2 operates (in this case, the control content required for the target device 2). Therefore, the input information D21 received by the control system 2000 can be said to be an example of first information indicating the requirements in the work environment. Furthermore, the control command D22 can be said to be information used for the work (work relating to the control of the target device 2) in correspondence with such input information D21. Hereinafter, the control command D22 output to a predetermined output destination from the operating environment of the learning model to which model input data based on the input information D21 is input may be referred to as second information.

[0161] Next, the operation of the control system 2000 of this embodiment will be described. Figure 13 is a flowchart showing an example of the operation of the control system 2000.

[0162] In the example shown in Figure 13, the control system 2000 first receives input information D21 (step S210). For example, the input unit 102 or input processing unit 201 described above may receive the input information D21. The received input information D21 is input to the learning model unit 200 as model input data D101.

[0163] Next, the control system 2000 performs a control command D22 generation process using the learning model unit 200 (step S211). In step S211, the learning model unit 200 (more specifically, the model control unit 101) outputs a control command D22 corresponding to the input information D21 based on the model information D102, the input information D21, and model reference information D104 which includes equipment information D23 as needed.

[0164] In step S211, the learning model unit 200 may, for example, use a learning model capable of generating binary data to generate a control command D22 for binary data from the input information D21. Alternatively, the learning model unit 200 may, for example, use a learning model capable of generating text data to generate a control command D22 for text data from the input information D21. Alternatively, the learning model unit 200 may, for example, use a learning model capable of generating image data to generate a control command D22 for image data from the input information D21. Alternatively, the learning model unit 200 may, for example, use a learning model capable of generating audio data to generate a control command D22 for audio data from the input information D21.

[0165] In step S211, the preprocessing unit 105 and / or postprocessing unit 106 of the learning model unit 200 may further perform the processing described above.

[0166] The control command D22 output from the learning model unit 200 is input to the target device 2, for example (step S212). The input of the control command D22 to the target device 2 may be directly from the control system 2000 (more specifically, the learning model unit 200 or the information processing device 10 which is its operating environment), or it may be input indirectly via a communication network or other devices (server, various conversion devices, etc.).

[0167] As a result, the target device 2 operates according to the input control command D22.

[0168] If the control system 2000 outputs a control command D22 and as a result there is a change in the state of target device 2, such as when the target device 2 is controlled, or if there is feedback from target device 2, it may acquire state information D25 and feedback information D26 (step S213). Note that the processing in step S213 is not mandatory and may be omitted as appropriate.

[0169] The control system 2000 may repeat the process from steps S210 to S213 multiple times (for example, until the desired control is completed for the target device 2).

[0170] As described above, according to this embodiment, even if user 1 does not know the specific control method for the target device 2, a control command D22 can be generated from the input information D21 input from user 1, and the target device 2 can be controlled based on the generated control command D22. Therefore, the efficiency and sophistication of the work related to controlling the target device 2 can be improved.

[0171] Furthermore, according to this embodiment, the device can be controlled to an appropriate state even with ambiguous information.

[0172] Embodiment 3. Next, Embodiment 3 will be described. In this embodiment, an example will be described in which a learning model is used to support tasks related to the operation of a target device.

[0173] In the following, we consider the operation of various devices in homes and buildings, such as air conditioners, refrigerators, televisions, lighting, washing machines, projectors, various sensors, and communication devices. In recent years, even these consumer-grade devices have become more sophisticated in their functions, and their control has become more complex. Although controllers such as user interfaces and remote controllers have been designed to simplify complex control, it is still difficult to remember all the operations, and even when a desired function is provided, it can be difficult to easily access that function.

[0174] Furthermore, even for similar functions, the names of the functions offered often differed depending on the model, there were subtle differences in the functions themselves, and the control methods were often different. This meant that when introducing a different model, such as through a replacement, users had to learn all these differences from scratch, which was cumbersome.

[0175] Furthermore, while some devices can automatically control themselves to an appropriate state by remembering past operation history or understanding the operating environment, accurate control can be difficult in situations where the appropriate state differs from person to person, such as when multiple people are gathered together or when the appropriate state changes due to changes in a person's physical condition.

[0176] In such cases, it is desirable that the desired state can be easily achieved even without knowing the specific operating procedures or the operator being aware of the appropriate state, as this leads to increased work efficiency and performance.

[0177] Furthermore, the situations in which the device is operated are not limited to within a home or building, nor are the applications of this embodiment limited to within a home or building.

[0178] Figure 14 is a configuration diagram showing an example of a control system 3000 according to Embodiment 3. The control system 3000 shown in Figure 14 is a control system for operating equipment using a learning model, and comprises a learning model unit 300, an equipment information storage unit 310 (referred to as equipment information DB in the figure), an input interface 311 (referred to as input IF in the figure), and an output interface 312 (referred to as output IF in the figure).

[0179] When input information D31 is received, the learning model unit 300 outputs an operation command D32. For example, when input information D31 is received, the learning model unit 300 outputs an operation command D32 based on model information D102. The configuration of the learning model unit 300 may be basically the same as that of the learning model unit 100 in Embodiment 1.

[0180] In this embodiment, the learning model unit 300 is a model and its operating environment configured to output an operation command D32 corresponding to the input information D31 when input information D31 is received. Alternatively, the learning model unit 300 may be a model and its operating environment configured to generate and output an operation command D32 based on the input information D31, device information D33, and other information that can be referenced by the learning model unit 300 when input information D31 is received.

[0181] In this embodiment, input information D31 includes information indicating the operation content required for the target device 2. Input information D31 may be, for example, text, images, audio, or a combination thereof indicating the operation content for the target device 2. Input information D31 may be, for example, text, images, audio, or a combination thereof indicating multiple operation content for the target device 2. Furthermore, input information D31 may include information indicating operation content that is performed sequentially over time, in which case it may be time-series data of a predetermined data structure including text, images, audio, or a combination thereof indicating the operation content as described above. The method of indicating the operation content is assumed to conform to the input format of the model used by the learning model unit 300, however, this is not limited to cases where error processing, correction processing, or conversion processing is included before the learning model unit 300.

[0182] As an example of how the operation details in input information D31 can be indicated, the operation to be performed on target device 2 may be specified, and then the values ​​of the parameters for performing that operation and the state after the operation may be specified. In that case, input information D31 may include, for example, information that identifies the operation and information that indicates the values ​​of the parameters for performing that operation or the state after the operation. The values ​​of the parameters for performing the operation may include, for example, values ​​related to the type of operation (ON / OFF, etc.), direction, quantity, and time. Furthermore, input information D31 may include not only information that directly indicates the operation details on target device 2, but also information that indirectly indicates them using control details corresponding to the operation details, the words and actions of user 1, an image of target device 2, or similar operation commands for other models.

[0183] Operation command D32 includes information regarding the operation of target device 2, presented in a predetermined format that allows the target device 2 or an interface (including a person) requesting control of target device 2 to be identified. Operation command D32 may also include information indicating an operation request or control request to target device 2. Operation command D32 may be, for example, an operation command, operation signal, operation code, control command, control signal, or control code for target device 2. Operation command D32 may also be, for example, a command written in a format handled by a predetermined controller corresponding to target device 2. Operation command D32 can be considered a concept that adds information regarding operation to the control command D22 described above. Furthermore, if the interface is a person, i.e., control is requested to target device 2 via a person, operation command D32 may also be information indicating how to operate target device 2, presented in a format that is identifiable to the person.

[0184] The device information storage unit 310 stores device information D33, which is information relating to the target device 2. The handling of the device information storage unit 310 and device information D33 is basically the same as that of the device information storage unit 110 and device information D13 in Embodiment 1. In this embodiment, device information D33 may include, for example, information used for operating the target device 2. Device information D33 may include, for example, information indicating the procedure of operations actually performed on the target device 2 in response to the operation content. Also, device information D33 may include, for example, commands, signals, codes, etc. issued to the target device 2. Device information D33 is used, for example, as additional information when the learning model unit 300 outputs an operation command D32. Hereinafter, in this embodiment, information indicating the state of the target device 2 in particular may be referred to as state information D35.

[0185] The input interface 311 is an interface that receives input information D31 from user 1 and inputs it to the learning model unit 300. The input interface 311 may also be an interface that converts the input information D31 input from user 1 into data that matches the input of the learning model unit 300 and outputs it. The input interface 311 may be provided as, for example, an example of the input unit 102 described above.

[0186] The output interface 312 is an interface that receives operation commands D32 from the learning model unit 300 and outputs them to a predetermined output destination. The output interface 312 may be provided, for example, as an example of the output unit 103 described above. The output interface 312 may also be an interface that converts the operation commands D32 output from the learning model unit 300 into data that matches the predetermined output destination and outputs it. In this embodiment, the output destinations of the output interface 312 may include the target device 2, the controller 4 (not shown), a predetermined display 7 (not shown), and the user 1's operation terminal (not shown).

[0187] In this embodiment, the learning model unit 300 may be, for example, a language learning model such as an LLM that takes natural language as input and obtains an output result, and its operating environment. Alternatively, the learning model unit 300 may be, for example, an image learning model such as a VLM that takes an image as input and obtains an output result, and its operating environment. Alternatively, the learning model unit 300 may be, for example, a multimodal model that takes natural language and an image as input and obtains an output result, and its operating environment. In this case, the input information D31 may be input as text data, image data, a combination of text data and image data, or a data format that can be converted to these (such as audio data, or a video which is a combination of audio data and image data). Note that the learning model used in this embodiment is not limited to the models described above.

[0188] In this embodiment, input information D31 corresponds to model input data D101. Operation command D32 corresponds to model output data D103. The learning model unit 300 (in particular, the model control unit 101) may be configured to, for example, receive input information D31 and output an operation command D32 corresponding to the input information D31 based on model information D102 and, if necessary, model reference information D104.

[0189] In such cases, the model generation unit 107, which is provided in correspondence with the learning model unit 300, may perform machine learning using model learning data D105, which includes candidate input information D31 that can be input to the model control unit 101, to generate or update model information D102. Alternatively, the model generation unit 107 may perform machine learning using model learning data D105, which includes candidate input information D31 that can be input to the model control unit 101 and candidate operation commands D32 that correspond to them, to generate or update model information D102.

[0190] Although not shown in the illustration, in this embodiment as well, state information D35 and / or feedback information D36 may be obtained from the output destination of the model output data D103 of the learning model unit 300 and / or the information generated therefrom. The control system 3000 may, for example, output the obtained state information D35 and / or feedback information D36 as information indicating the response result to user 1, the learning model unit 300, or other devices not shown. The control system 3000 may also be configured to return a query D37 to user 1 if the input information D31 contains unclear or uncertain information. Furthermore, the control system 3000 can generate supplementary information D38 for the input and output data of the learning model unit 300 based on the obtained state information D35 and / or feedback information D36 and issue it to user 1, the learning model unit 300, or other devices not shown. The handling of state information D35, feedback information D36, query D37, and supplementary information D38 may be basically the same as in Embodiment 1.

[0191] The control system 3000 may also further include a state acquisition unit 330 (not shown) that acquires state information D35 and / or feedback information D36 and issues supplementary information D38 as necessary. The state acquisition unit 330 is the same as the state acquisition unit 130 in Embodiment 1.

[0192] In this embodiment as well, the target device 2 is not particularly limited. It is assumed that the target device 2 is a device capable of receiving the operation command D32 and performing control corresponding to the operation content indicated by the operation command D32. However, this does not apply if the target device 2 includes a controller 4 or a converter or other signal-converting device or operator. In that case, the converter or operator can receive the operation command D32 and operate the target device 2.

[0193] In this embodiment, the input information D31 received by the control system 3000 can be said to be information relating to the requirements in the work environment, in this case the environment in which the target device 2 operates (in this case, the operations required for the target device). Therefore, the input information D31 received by the control system 3000 can be said to be an example of first information indicating the requirements in the work environment. Furthermore, the operation command D32 can be said to be information used for the operation (operation on the target device 2) in response to such input information D31. Hereinafter, the operation command D32 output to a predetermined output destination from the operating environment of the learning model to which model input data based on the input information D31 is input may be referred to as second information.

[0194] Next, the operation of the control system 3000 of this embodiment will be described. Figure 15 is a flowchart showing an example of the operation of the control system 3000.

[0195] In the example shown in Figure 15, first, the input interface 311 of the control system 3000 receives input information D31 (step S310). The received input information D31 is input to the learning model unit 300 as model input data D101.

[0196] Next, the control system 3000 performs the process of generating an operation command D32 using the learning model unit 300 (step S311). In step S311, the learning model unit 300 (more specifically, the model control unit 101) generates and outputs an operation command D32 corresponding to the input information D31 based on the model information D102, the input information D31, and model reference information D104 which includes equipment information D33 as needed.

[0197] In step S311, the learning model unit 300 may, for example, use a learning model capable of generating binary data to generate a binary data manipulation command D32 from the input information D31. Alternatively, the learning model unit 300 may, for example, use a learning model capable of generating text data to generate a text data manipulation command D32 from the input information D31. Alternatively, the learning model unit 300 may, for example, use a learning model capable of generating image data to generate an image data manipulation command D32 from the input information D31. Alternatively, the learning model unit 300 may, for example, use a learning model capable of generating audio data to generate an audio data manipulation command D32 from the input information D31.

[0198] In step S311, the preprocessing unit 105 and / or postprocessing unit 106 of the learning model unit 300 may further perform the processing described above.

[0199] The operation command D32 output from the learning model unit 300 is output to a predetermined output destination, for example, via the output interface 312. Here, the predetermined output destination may be the target device 2, the controller 4, a predetermined display 7, or the user 1's operation terminal (not shown). When the operation command D32 is input to the predetermined output destination, the target device 2 is operated according to the input operation command D32 (step S312).

[0200] For example, the output interface 312 may output the operation command D32 to the target device 2. In this case, the target device 2, upon receiving the operation command D32 (e.g., an operation command, operation signal, operation code, control command, control signal, or control code), may perform the actual control according to the operation command D32. Alternatively, the output interface 312 may output the operation command D32 to the controller 4 corresponding to the target device 2. In this case, the controller 4, upon receiving the operation command D32 (e.g., indirect control information for the target device 2, such as a command to the controller 4, an operation command, an operation signal, or an operation code), may operate the target device 2 according to the operation command D32. The controller 4 may also operate the target device 2 by outputting direct control information, such as a control code, to the target device 2 based on the control information indicated by the received operation command D32. Here, the controller 4 may be, for example, an operation panel installed on the target device 2, or a remote controller corresponding to the target device 2 that is directly operated by user 1. The controller 4 may include controllers specific to the target device 2 and general-purpose controllers. Furthermore, the output interface 312 may output the operation command D32 to the user 1's operating terminal or a designated display. In this case, the user 1's operating terminal or display, upon receiving the operation command D32 (for example, information indicating the operation method), will display the operation command D32. The user 1 may then operate the target device 2 or controller 4 by referring to the displayed operation command D32.

[0201] The input of the operation command D32 to the output destination may be directly input from the control system 3000 (more specifically, the learning model unit 300 or the information processing device 10 which is its operating environment), or it may be input indirectly via a communication network or other devices (server, various conversion devices, etc.).

[0202] As a result, the target device 2 operates according to the operation command D32.

[0203] If the control system 3000 outputs an operation command D32 and as a result there is a change in the state of the target device 2, such as when the target device 2 is operated, or if there is feedback from the target device 2, it may acquire state information D35 and feedback information D36 (step S313). Note that the processing in step S313 is not mandatory and may be omitted as appropriate.

[0204] The control system 3000 may repeat the process from steps S310 to S313 multiple times (for example, until the desired operation is completed for the target device 2).

[0205] As described above, according to this embodiment, even if user 1 does not know the specific operation method for the target device 2, an operation command D32 can be generated from the input information D31 input by user 1, and the target device 2 can be operated based on the generated operation command D32. Therefore, the work related to operating the target device 2 can be made more efficient and sophisticated.

[0206] Furthermore, according to this embodiment, it is possible to operate the device to the appropriate state even with ambiguous information. Also, according to this embodiment, it is possible to operate the device to the appropriate state without being dependent on the device or learning how to operate it.

[0207] Variation 3-1. Next, a modified example of the control system 3000 will be described. Figure 16 is a configuration diagram showing an example of control system 3000a, which is a modified example of the control system 3000 according to this embodiment. Elements identical to those in control system 3000 are denoted by the same reference numerals and their descriptions are omitted.

[0208] The control system 3000a shown in Figure 16 further includes an input determination unit 31. The input determination unit 31, upon receiving input information D31, analyzes the input information D31 and is a means for switching the control destination for the input information D31. In this modified example, the input determination unit 31 switches the control destination for the input information D31 between the learning model unit 300 and the output interface 312.

[0209] The input determination unit 31 may, for example, switch the control destination for input information D31 depending on whether or not the input information D31 matches the command rules for operation on the target device 2. If the input determination unit 31 matches the command rules for operation on the target device 2, it may input the input information D31 directly to the output interface 312. On the other hand, if the input determination unit 31 does not match the command rules for operation on the target device 2, it may input the input information D31 to the learning model unit 300.

[0210] Whether or not the operation conforms to the command rules may be determined, for example, using a rule-based model. Here, the input determination unit 31 may be a relatively lightweight learning model compared to the learning model unit 300.

[0211] Hereinafter, in order to distinguish the information input to the output interface 312, the operation command D32 output from the learning model unit 300 may be referred to as operation command D32a, and the input information D31 output to the output interface 312 may be referred to as operation command D32b.

[0212] In this example, the output interface 312 can be any interface that receives operation command D32a or operation command D32b and outputs it to a predetermined output destination.

[0213] Next, the operation of the control system 3000a in this modified example will be explained. Figure 17 is a flowchart showing an example of the operation of the control system 3000a.

[0214] In the example shown in Figure 17, first, the input interface 311 of the control system 3000a receives input information D31 (step S310). The received input information D31 is input to the input determination unit 31.

[0215] Next, the input determination unit 31 determines whether the input information D31 matches the command rules for operating the target device 2 (step S321). If it is determined that the input information D31 matches the command rules for operating the target device 2 (Yes in step S321), the input information D31 is input to the output interface 312 (proceed to step S322). On the other hand, if it is determined that the input information D31 does not match the command rules for operating the target device 2 (No in step S321), the input information D31 is input to the learning model unit 300 (proceed to step S311).

[0216] The processing in steps S311 to S313 is the same as in the example shown in Figure 15.

[0217] In step S322, the output interface 312 outputs the input information D31 as an operation command D32b to a predetermined output destination. As a result, the target device 2 operates according to the operation command D32b.

[0218] Other aspects may be the same as other control systems according to this embodiment.

[0219] As described above, this modified version allows the system to operate the target device 2 according to the command rules for operating the target device 2 if the input from user 1 matches those rules, while operating the target device 2 using a learned model if the input does not match. Therefore, the efficiency of the work involved in operating the target device 2 can be further improved.

[0220] Variation 3-2. Next, another variation of the control system 3000 will be described. In this variation, a learning model is used to generate operation commands that include mediation of multiple inputs.

[0221] Figure 18 is a configuration diagram showing an example of control system 3000b, which is a modified version of control system 3000 according to this embodiment. Elements identical to those in control system 3000 are denoted by the same reference numerals and their descriptions are omitted.

[0222] In the control system 3000b shown in Figure 18, the input interface 311 receives input information D31 from multiple users 1.

[0223] The input interface 311 receives input information D31 from multiple users 1 and inputs it to the learning model unit 300. At this time, the input interface 311 may receive input information D31 with information of the inputting user 1 attached, or the input interface 311 may identify the inputting user 1 and attach the inputting user information before receiving the input information D31, or it may receive it without doing anything in particular.

[0224] The learning model unit 300 may be any model and operating environment configured to output an operation command D32 corresponding to the input information D31 group when the input interface 311 receives the input information D31 group. Alternatively, the learning model unit 300 may be any model and operating environment configured to generate and output an operation command D32 based on the input information D31 group, device information D33, and other information that can be referenced by the learning model unit 300 when the input information D31 group is received.

[0225] For example, the learning model unit 300 may use a language learning model such as an LLM that takes natural language as input and obtains output results, and perform a process to extract a suitable solution in the language space (more specifically, on a feature vector space that has information in the language space), thereby generating and outputting an operation command D32 that is a compromise for the different operation contents indicated by the input information group D31. In this case, the learning model unit 300 may refer to the history of input information D31 for each user 1 that inputs and / or the history of operation commands D32 for each user 1 that inputs.

[0226] Other aspects may be the same as other control systems according to this embodiment.

[0227] As described above, according to this modified version, even when information regarding different operation content is input from multiple users, the learning model unit 300 can be used to generate a more appropriate operation command D32 by mediating that information, thereby further enhancing the functionality of the operation of the target device 2.

[0228] Modification example 3-3. Next, another variation of the control system 3000 will be described. In this variation, a learning model is used to generate the operation screen user interface.

[0229] Figure 19 is a configuration diagram showing an example of a control system 3000c, which is a modified version of the control system 3000 according to this embodiment. Elements identical to those in the control system 3000 are denoted by the same reference numerals and their descriptions are omitted.

[0230] The control system 3000c shown in Figure 19 is further equipped with an operation screen user interface 3 (referred to as the operation screen UI in the figure). The learning model unit 300 generates an operation screen as an operation command D32, which actually performs the operation on the target device 2 corresponding to the operation content of the input information D31.

[0231] The operation screen generated by the learning model unit 300 may be, for example, a screen API (Application Programming interface) equipped with a function to receive operation input from the user along with an explanation of the operation content, and output control commands D34 such as control codes according to the received operation input. Here, the output of control codes etc. according to the operation input includes a mode in which multiple control commands D34 are output sequentially in response to a single operation input. Furthermore, the operation screen may be a screen API equipped with operation explanations, operation input reception, and control command output corresponding to two or more different operation contents. For example, the learning model unit 300 may extract operation information indicating two or more different operation contents as operation commands D32 corresponding to input information D31, and generate a screen API equipped with operation input reception and control command output corresponding to each piece of operation information.

[0232] Furthermore, the operation screen generated by the learning model unit 300 may be a modified version of an existing operation screen, in which the operation area corresponding to the operation content is highlighted, the operation functions are restricted, and the position and form (shape, size, color, etc.) of UI components on the screen are changed.

[0233] The operation screen user interface 3 is an interface that displays an operation screen for the target device 2 and accepts user input on the operation screen. The operation screen user interface 3 may be implemented, for example, by a touch panel display or a controller equipped with operation buttons and a display unit. Alternatively, the operation screen user interface 3 may be implemented by a display device such as a display that works in conjunction with an operation input device such as a mouse.

[0234] Furthermore, in this modified example, the output interface 312 outputs the operation command D32 (operation screen) output from the learning model unit 300 to the operation screen user interface 3.

[0235] Furthermore, in this modified example, the learning model unit 300 may have a function to interactively confirm the operation expected by user 1. In such a case, for example, after presenting an operation screen as operation command D32, if the learning model unit 300 receives information requesting the reacquisition of operation command D32, it may reacquire operation command D32 after changing some of the input information, some of the model parameters, or the reference destination of the reference information.

[0236] As described above, in this modified version, an operation screen that has been modified (such as constructing a screen API or changing the display mode) to allow the desired operation to be performed easily or in an easy-to-understand manner can be generated using the learning model unit 300, thereby further improving the efficiency of the work involved in operating the target device 2. Furthermore, according to this modified version, the user 1 can perform the actual operation while confirming the explanation of the operation commands generated by the learning model unit 300, so that the operation can be performed without errors.

[0237] Variation 3-4. Next, another variation of the control system 3000 will be described. In this variation, the learning model further uses environmental information to generate operation commands.

[0238] Figure 20 is a configuration diagram showing an example of a control system 3000d, which is a modified version of the control system 3000 according to this embodiment. Elements identical to those in the control system 3000 are denoted by the same reference numerals and their descriptions are omitted.

[0239] The control system 3000d shown in Figure 20 further includes an environmental information storage unit 313 (referred to as the environmental information DB in the figure).

[0240] The environmental information storage unit 313 stores environmental information D33a, which is information about the environment in which the target device 2 operates. Environmental information D33a may include information about the space in which the target device 2 operates. In this modified example, information about objects or people present in the space in which the target device 2 operates, as well as user 1, who is the operator of the target device 2, are also considered part of the environment. Therefore, environmental information D33a may include information about such objects or people or user 1.

[0241] Environmental information D33a may include, for example, information about a person, such as their attributes, temperature, location, posture, and heart rate. Environmental information D33a may also include, for example, information about a space, such as its location, temperature, humidity, and brightness. Furthermore, environmental information D33a may retain information showing the changes in such space or person information. Here, information showing changes is also called time-series data or historical information. Environmental information D33a may be configured, for example, as part of the model reference information D104 of a learning model.

[0242] Environmental information D33a may be acquired, for example, by a sensor (not shown).

[0243] The learning model unit 300 is a model and its operating environment configured to generate and output an operation command D32 based on input information D31, device information D33, environment information D33a, and other information that can be referenced in the learning model unit 300, for example, when input information D31 is input.

[0244] As described above, this modified version allows the learning model to generate operation commands D32 using environmental information D33a related to the space in which the target device 2 is operating, thereby further enhancing the functionality of the operations related to the operation of the target device 2.

[0245] Variation 3-5. Next, other variations of the control system 3000 will be described. In this variation, two learning models are combined to generate operation commands. Figure 21 is a configuration diagram showing an example of control system 3000e, which is a variation of the control system 3000 according to this embodiment. Elements that are the same as those in control system 3000 are denoted by the same reference numerals and their descriptions are omitted.

[0246] In the control system 3000e shown in Figure 21, instead of the learning model unit 300 shown in Figure 20, it includes a learning model unit 300a as the first learning model unit 300 and a learning model unit 300b as the second learning model unit 300.

[0247] The learning model unit 300a outputs operation information D320 when input information D31 is received. The learning model unit 300a may also be a model and its operating environment configured to generate and output operation information D320 based at least on input information D31 and environment information D33a when input information D31 is received.

[0248] Here, operation information D320 includes information regarding the operation of the target device 2, presented in a predetermined format that can be recognized by the subsequent learning model unit 330b. Here, operation information D320 may be information obtained by supplementing (including adding, modifying, and canceling) the operation content shown in input information D31 according to the environment information D33a. Operation information D320 may also be information obtained by changing the operation content or its representation shown in input information D31 according to the conditions of the space in which the target device 2 is driven. The learning model unit 300a may mainly be a model that performs grounding on input information D31.

[0249] For example, even if the desired operation is the same, differences in linguistic expression and perception of events may arise depending on the environment in which the target device 2 is operating. For instance, differences in dialect, idiomatic expressions, the use of company or household jargon, and differences in perception such as heat / cold may cause the operation content that input information D31 signifies to differ.

[0250] The learning model unit 300a plays a role in, for example, absorbing differences in such linguistic expressions and / or differences in perception of events, and modifying them into more generalized or specific content. The learning model unit 300a may also be a local learning model that obtains output results based on local information, such as having limited reference databases.

[0251] The operation information D320 generated by the learning model unit 300a is input to the learning model unit 300b.

[0252] The learning model unit 300b can be basically the same as the learning model unit 300 described above. However, instead of input information D31, operation information D320 generated by the learning model unit 300a is input.

[0253] The learning model unit 300b outputs an operation command D32 when operation information D320 is input. The learning model unit 300b may be a model and its operating environment configured to generate and output an operation command D32 based on operation information D320, device information D33, and other information that can be referenced by the learning model unit 300b when operation information D320 is input. The learning model unit 300b may be a global learning model that obtains output results based on global information, such as having free access to an external network.

[0254] Figure 22 is a flowchart showing an example of operation of this modified example. In the example shown in Figure 22, when the input interface 311 of the control system 3000e receives input information D31 in step S310, the input information D31 is input to the learning model unit 300a.

[0255] Next, the control system 3000e performs the process of generating operation information D320 using the learning model unit 300a (step S331). In step S331, the learning model unit 300a (more specifically, the model control unit 101) generates and outputs operation information D320 corresponding to the input information D31 based on the model information D102, the input information D31, and model reference information D104 including environment information D33a as needed. The operation information D320 output from the learning model unit 300a is input to the learning model unit 300b.

[0256] Next, the control system 3000e performs the process of generating an operation command D32 using the learning model unit 300b (step S332). In step S332, the learning model unit 300b (more specifically, the model control unit 101) generates and outputs an operation command D32 corresponding to the operation information D320 based on the model information D102, the input operation information D320, and model reference information D104 which includes equipment information D33 as needed.

[0257] Subsequent processing may be the same as that of other control systems according to this embodiment.

[0258] As described above, according to this modification example, with respect to the input information D31 input from the user 1, after absorbing differences in language expressions and / or differences in recognition of events and changing it to more generalized or specific content, the operation command D32 can be generated. Therefore, it is possible to further enhance the functionality of the work related to the operation of the target device 2.

[0259] Note that even in the configuration shown in Modification Example 3-4, it is possible for the learning model unit 300 to generate an operation command D32 that equalizes differences in language expressions and / or differences in recognition of events based on model reference information D104 including environmental information D33a, device information D33, and past operation histories. However, according to this modification example, the role of the learning model can be clearly divided, such as absorbing differences in expressions and converting them into operation commands. Therefore, the learning model can be specialized and learned for this purpose, and a compact design can be achieved, such as suppressing the scale of learning.

[0260] Embodiment 4. Next, Embodiment 4 will be described. In this embodiment, an example of assisting work related to monitoring a certain work situation using a learning model will be described.

[0261] For example, consider monitoring for abnormalities in a FA (Factory Automation) system including control devices such as robots and PLCs in a factory. For example, when there is an obvious installation error in the target work of the control device, existing monitoring algorithms based on rules can handle it. However, it is also conceivable that a minor installation error triggers an abnormality discovered in a subsequent process. In such a case, even if analysis etc. is performed triggered by, for example, abnormality detection, it has been difficult to accurately grasp the situation and obtain a method for improvement.

[0262] In this embodiment, by assisting work related to monitoring a work environment where cases where the occurrence situation does not match existing rules and it is difficult to investigate the cause, such as a relatively small defect spreading into a major abnormality, can be assumed, the efficiency and performance of the monitoring work are improved.

[0263] Figure 23 is a configuration diagram showing an example of a control system 4000 according to Embodiment 4. The control system 4000 shown in Figure 23 is a control system for monitoring specific work conditions using a learning model, and comprises a sensor 5, a learning model unit 400a, a learning model unit 400b, a device information storage unit 410 (referred to as device information DB in the figure), a model interface 6 (referred to as model IF in the figure), and a display unit 7.

[0264] Sensor 5 acquires data indicating the status of the work being monitored. Hereinafter, the data acquired by Sensor 5 will be referred to as sensor data. Sensor data may be, for example, image data of the work being monitored. Sensor data may also be, for example, audio data of the work being monitored. Sensor data may also be, for example, measurement data of the position or state of the person or object performing the work being monitored.

[0265] The acquisition of sensor data by sensor 5 is assumed to be performed at all times, but may also be performed based on a trigger provided by, for example, a person or another monitoring system. The sensor data acquired by sensor 5 is input to the learning model unit 400a as input information D41. Alternatively, the sensor data itself, which is designated as input information D41, may be provided by a person or another monitoring system. In such cases, sensor 5 can be omitted.

[0266] The learning model unit 400a outputs the analysis result D42a when input information D41 is received. For example, when input information D41 is received, the learning model unit 400a outputs the analysis result D42a based on model information D102. The configuration of the learning model unit 400a may be basically the same as that of the learning model unit 100 in Embodiment 1.

[0267] In this embodiment, the learning model unit 400a is a model and its operating environment configured to output an analysis result D42a corresponding to input information D41 when input information D41 is received. Alternatively, the learning model unit 400a may be a model and its operating environment configured to generate and output an analysis result D42a based on input information D41, equipment information D43, and other information that can be referenced by the learning model unit 400a (such as model reference information D104) when input information D41 is received. Here, the learning model unit 400a may refer to and use information related to the work to be monitored as model reference information D104. Information related to the work to be monitored may be, for example, information indicating the location, person, object, procedure, conditions, etc., of the work to be performed. The learning model unit 400a may use, for example, a data version of a manual that describes the conditions of the equipment used for the work, the installation environment, the operating procedure, etc., as model reference information D104.

[0268] In this embodiment, input information D41 includes information indicating the status of the work to be monitored. Here, the work to be monitored includes one or more tasks performed by a person or machine. Input information D41 may be, for example, measured values, images, audio, or a combination thereof indicating the status of the work to be monitored. Input information D41 may be, for example, measured values, images, audio, or a combination thereof indicating the status of multiple tasks to be monitored. Furthermore, input information D41 may include information indicating the status of tasks performed continuously over time, in which case it may be time-series data of a predetermined data structure including measured values, images, audio, or a combination thereof indicating the status as described above. The method of indicating the work status is assumed to conform to the input format of the model used by the learning model unit 400a, however, this does not apply if error processing, correction processing, or conversion processing is included before the learning model unit 400a.

[0269] The analysis result D42a includes information indicating the situation analysis result obtained by analyzing the work situation shown in input information D41. The information indicating the situation analysis result may also be information indicating objects (environment) and / or events occurring in the work situation shown in input information D41. The information indicating the situation analysis result can be said to be information indicating an interpretation of the work situation shown in input information D41. For example, the analysis result D42a may be text indicating an interpretation of the work situation shown in input information D41. Alternatively, the analysis result D42a may be text indicating an interpretation of a part of the work situation shown in input information D41 that differs from the normal situation, by focusing on that part. Note that the format of the analysis result D42a may be other than text. The format of the analysis result D42a is not particularly limited as long as it is written in a predetermined format that can be recognized by the subsequent learning model unit 400b, and may be, for example, text, image, audio, or a combination thereof.

[0270] Examples of interpreting the work situation include describing the objects present in the work situation using their attributes, describing the events that occurred in the work situation using a predetermined syntactic format such as 5W1H or 7W1H, or summarizing such a concrete expression. Other examples include decomposing the work being performed in the work situation into multiple perspectives and expressing them from each perspective, or, if the work being performed in the work situation includes multiple sub-tasks or steps, decomposing the target work into sub-task units or step units and explaining each sub-task or step. Analysis result D42a can be said to be the result of concretizing, subdividing, and / or extracting singularities from the work situation shown in input information D11, with the addition of representation in a predetermined format. In this way, analysis result D42a presents the work situation in an easy-to-understand and organized manner.

[0271] When the learning model unit 400b receives the analysis result D42a as input, it outputs the analysis result D42b. For example, when the learning model unit 400b receives the analysis result D42a as input, it outputs the analysis result D42b based on the model information D102. The configuration of the learning model unit 400b may be basically the same as that of the learning model unit 100 in Embodiment 1.

[0272] In this embodiment, the learning model unit 400b is a model and its operating environment configured to output an analysis result D42b corresponding to the analysis result D42a when the analysis result D42a is input. Alternatively, the learning model unit 400b may be a model and its operating environment configured to generate and output an analysis result D42b based on the analysis result D42a, equipment information D43, and / or information that can be referenced in the learning model unit (such as model reference information D104) when the analysis result D42a is input.

[0273] Analysis result D42b includes information indicating methods for improving the work situation, derived from the analysis results of the work situation by the learning model unit 400a. The information indicating methods for improving the work situation may be information indicating recovery methods to restore an abnormal state to normal, or it may be information indicating solutions to problems that occur in the environment (work environment) where the monitored work is being performed, such as when a person is in trouble or when equipment has stopped working.

[0274] Information indicating an improvement method may be, for example, text, images, or audio illustrating the method; or it may be control commands (e.g., instructions, control signals, control codes, etc.) for the device (target device 2) on which the method is to be implemented; a procedure document describing the method; a sequence diagram; source code; executable code; or a controller command for causing the controller to execute the method. Information indicating an improvement method may also be text, images, audio illustrating the method; data written in a predetermined design language; control descriptions (including source code and information written in a predetermined programming platform language); information written in other platform languages; control commands (including control instructions, control signals, control codes, and controller commands); executable code; or a combination of two or more of these elements. An example of a predetermined design language is UML (Unified Modeling Language).

[0275] In the following, the learning model unit 400a may be referred to as the first learning model unit 400, and the analysis result D42a may be referred to as the first analysis result D42. Furthermore, the learning model unit 400b may be referred to as the second learning model unit 400, and the analysis result D42b may be referred to as the second analysis result D42.

[0276] As already explained, the analysis result D42a includes information indicating the result of the situation analysis of the work situation shown in the input information D41. Therefore, the learning model unit 400b may be a model configured to output an analysis result D42b corresponding to the situation analysis result shown in analysis result D42a, and its operating environment. Here, if the information indicating the situation analysis result is text explaining the work situation shown in the input information D41, the learning model unit 400b may be a model configured to output an analysis result D42b corresponding to the text explaining the work situation, and its operating environment.

[0277] The handling of the equipment information storage unit 410 and equipment information D43 is basically the same as that of the equipment information storage unit 110 and equipment information D13 in Embodiment 1. In this embodiment, the equipment information storage unit 410 stores equipment information D43, which is information about equipment related to the work to be monitored, as the target equipment 2. Here, equipment related to the work broadly includes equipment necessary for deriving the situation analysis and improvement method described above. More specifically, it includes not only equipment used in the work but also equipment that affects the person or equipment performing the work. More specifically, equipment that affects the person or equipment performing the work may be equipment that directly or indirectly brings about changes to the person or equipment performing the work. Examples include equipment directly used in the work (including various machines such as processing machines and conveyors, as well as tools such as workbenches and tools), equipment that controls equipment directly used in the work (power supply, relay, switch, controller, etc.), and equipment that brings about changes in the work environment (lighting equipment, air conditioning equipment, vacuum cleaner, air purifier, etc.).

[0278] The device information D43 is used, for example, as additional information when the learning model unit 400a and / or the learning model unit 400b output model output data D103 (analysis result D42a, analysis result D42b). Hereinafter, in this embodiment, information indicating the state of the target device 2 may be referred to as state information D45.

[0279] In this embodiment, the learning model unit 400a may be an image learning model such as VLM that takes an image as input and obtains an output result, and its operating environment. Alternatively, the learning model unit 400a may be a multimodal model that takes natural language and an image as input and obtains an output result, and its operating environment. Similarly, the learning model unit 400b may be a language learning model such as LLM that takes natural language as input and obtains an output result, and its operating environment. In this case, the input information D41 may be input as text data, image data, a combination of text data and image data, or a data format that can be converted to these (such as audio data or a video which is a combination of audio data and image data). Note that the learning model used in this embodiment is not limited to the models described above.

[0280] The model interface 6 is an interface that receives model output data (analysis results D42a and D42b) from the learning model unit 400a and the learning model unit 400b and outputs it to a predetermined output destination. The model interface 6 may also be an interface that converts the model output data output from the learning model unit 400a and the learning model unit 400b into data that matches the predetermined output destination and outputs it. The model interface 6 may be provided, for example, as an example of the output unit 103 described above. In this embodiment, the output destinations of the model interface 6 include the target device 2 and the display unit 7.

[0281] The model interface 6 may, for example, output result information D44a to the display unit 7, which shows the situation analysis results included in analysis result D42a and the improvement methods included in analysis result D42b, and may also output result information D44b, which shows the improvement methods included in analysis result D42b, to the target device 2. In this case, the model interface 6 may extract some data from analysis result D42a and / or analysis result D42b, convert it to a data format that matches the output destination, and then output it as result information D44a and result information D44b.

[0282] In the example shown in FIG. 23, the target device 2 and the display 7 are shown as the output destinations of the model interface 6. However, the output destination of the model output data is not limited to the above. For example, when the improvement method for the situation indicated by the model output data to be output includes information indicating control for the target device 2, the model interface 6 may directly output the model output data or the information indicating the method to the target device 2 as the execution destination of the method, or output the model output data or the information indicating the method to a conversion device (not shown) that converts it into information that can be received by the target device 2. The conversion device may be, for example, the control system 1000 of Embodiment 1 that converts the input information into a control description or an execution code that can be discriminated by the target device 2.

[0283] Also, the model interface 6 itself may have the function of a conversion device. For example, the model interface 6 may not only control the output of the model output data, but also convert the improvement method output by the learning model unit 400b into code that can be executed by an interpreter, and have a function of outputting the converted code or controlling a device based on the code. Further, the model interface 6 may have a function of controlling a processing flow, such as immediately executing the process when the improvement method includes a highly urgent process. Also, the model interface 6 may have a function of transmitting a prompt input through the display of the display 7, such as an answer to the proposed method displayed on the display 7, to the learning model unit 400b.

[0284] Also, the model interface 6 may have the functions of the above-described output confirmation unit 202 and correction confirmation unit 203. For example, the model interface 6 determines the urgency of the analyzed situation, and when it determines that the urgency is not high, it confirms the validity of the improvement method by inquiring a monitor or using a simulator or the like, and if the improvement method is not valid, it transmits that fact to the learning model unit 400b and prompts the output of the improvement method again. In that case, the model interface 6 may issue supplementary information D48 for the model input data of the target learning model unit.

[0285] In this embodiment, input information D41 corresponds to model input data D101 of the learning model unit 400a. Analysis result D42a corresponds to model output data D103 of the learning model unit 400a. Analysis result D42a corresponds to model input data D101 of the learning model unit 400b. Analysis result D42b corresponds to model output data D103 of the learning model unit 400b. The learning model unit 400a (in particular the model control unit 101) may be configured to output analysis result D42a corresponding to input information D41, for example, upon receiving input information D41, based on model information D102 and, if necessary, model reference information D104. The learning model unit 400b (in particular the model control unit 101) may be configured to output analysis result D42b corresponding to analysis result D42a, for example, upon receiving analysis result D42a, based on model information D102 and, if necessary, model reference information D104.

[0286] Furthermore, in such cases, the model generation unit 107, provided in correspondence with the learning model unit 400a, may, for example, perform machine learning using model learning data D105 that includes candidate input information D41 that can be input to the model control unit 101, to generate or update model information D102, or it may perform machine learning using model learning data D105 that includes candidate input information D41 that can be input to the model control unit 101 and candidate corresponding analysis result D42a, to generate or update model information D102. Furthermore, the model generation unit 107, provided in correspondence with the learning model unit 400b, may, for example, perform machine learning using model learning data D105 that includes candidate analysis result D42a that can be input to the model control unit 101, to generate or update model information D102, or it may perform machine learning using model learning data D105 that includes candidate analysis result D42a that can be input to the model control unit 101 and candidate corresponding analysis result D42b, to generate or update model information D102.

[0287] Although not shown in the illustration, in this embodiment as well, state information D45 and / or feedback information D46 may be obtained from the output destination of the model output data D103 of the learning model unit 400a and the learning model unit 400b and / or the information generated therefrom. The control system 4000 may, for example, output the obtained state information D45 and / or feedback information D46 as information indicating the control result to the user, the learning model unit 400a, the learning model unit 400b, or other devices not shown. The control system 4000 may also be configured to return a query D47 to the user if the input information D41 contains unclear or uncertain information. Furthermore, the control system 4000 can generate supplementary information D48 for the input and output data of the learning model unit 400a and the learning model unit 400b based on the obtained state information D45 and / or feedback information D46, and issue it to the user, the learning model unit 400a, the learning model unit 400b, or other devices not shown. The handling of status information D45, feedback information D46, inquiry D47, and supplementary information D48 may be basically the same as in Embodiment 1. Here, the output of information to the user may be performed, for example, via an input / output interface provided by the display unit 7 or an information processing device 10 (not shown).

[0288] The control system 4000 may also further include a state acquisition unit 430 (not shown) that acquires state information D45 and / or feedback information D46 and issues supplementary information D48 as necessary. The state acquisition unit 430 is the same as the state acquisition unit 130 in Embodiment 1.

[0289] In this embodiment as well, the target device 2 is not particularly limited. It is assumed that the target device 2 is a device that can actually be controlled upon receiving the analysis result D42b, but this does not apply if the above-mentioned conversion device is included between the target device 2 and the system.

[0290] In this embodiment, the input information D41 received by the control system 4000 can be said to be information relating to the conditions in the work environment (in this case, the conditions in the environment in which monitoring work is performed). Therefore, the input information D41 received by the control system 4000 can be said to be an example of first information indicating the conditions in the work environment. Furthermore, the analysis results D42a and D42b can be said to be information used for the work (monitoring work) in correspondence with such input information D41. Hereinafter, the analysis results D42a and / or D42b, which are output to a predetermined output destination from the operating environment of the learning model to which model input data based on the input information D41 is input, may be referred to as second information.

[0291] Next, the operation of the control system 4000 of this embodiment will be described. Figure 24 is a flowchart showing an example of the operation of the control system 4000.

[0292] In the example shown in Figure 24, the control system 4000 first receives input information D41 (step S410). For example, the input unit 102 or input processing unit 201 described above may receive the input information D41. The received input information D41 is input to the learning model unit 400a as model input data D101.

[0293] Next, the control system 4000 performs the process of generating the analysis result D42a using the learning model unit 400a (step S411). In step S411, the learning model unit 400a (more specifically, the model control unit 101) outputs the analysis result D42a corresponding to the input information D41 based on the model information D102, the input information D41, and model reference information D104 which includes equipment information D43 as needed. The learning model unit 400a may, for example, use a learning model capable of generating text data to generate the analysis result D42a of text data from the input information D41.

[0294] In step S411, the preprocessing unit 105 and / or postprocessing unit 106 of the learning model unit 400a may further perform the processing described above.

[0295] The analysis result D42a output from the learning model unit 400a is input to the learning model unit 400b. The analysis result D42a output from the learning model unit 400a is also input to the learning model unit 400b and the model interface 6. Alternatively, the analysis result D42a output from the learning model unit 400a may be input to the model interface 6 via the learning model unit 400b. In that case, the learning model unit 400b may output model output data D103 containing both the analysis result D42a and the analysis result D42b.

[0296] Next, the control system 4000 performs the process of generating the analysis result D42b using the learning model unit 400b (step S412). In step S412, the learning model unit 400b (more specifically, the model control unit 101) outputs the analysis result D42b corresponding to the analysis result D42a based on the model information D102, the input analysis result D42a, and model reference information D104 including equipment information D43 as needed. The learning model unit 400b may, for example, use a learning model capable of generating text data to generate the binary data analysis result D42b from the input analysis result D42a. Alternatively, the learning model unit 400b may, for example, use a learning model capable of generating both text data and binary data to generate the text data and binary data analysis result D42b from the input analysis result D42a.

[0297] In step S412, the preprocessing unit 105 and / or postprocessing unit 106 of the learning model unit 400b may further perform the processing described above.

[0298] The analysis result D42b output from the learning model unit 400b is input to, for example, the model interface 6.

[0299] The model interface 6 controls the target device 2 and / or displays information on the display unit 7 based on the analysis results from the learning model unit 400a and the learning model unit 400b (step S413). In step S413, for example, the model interface 6 outputs information based on the analysis results D42a and D42b to a predetermined output destination. For example, based on the analysis results D42a and D42b, the model interface 6 outputs result information D44a showing the situation analysis results and improvement methods to the display unit 7, and also outputs result information D44b showing improvement methods based on the analysis result D42b to the target device 2.

[0300] Result information D44a may, for example, indicate the situation that occurred in the work environment and the method of improvement, in text and audio. Similarly, result information D44b may, for example, indicate the method of improvement, in text or control signals.

[0301] As a result, the display unit 7 displays the situation analysis results shown in analysis result D42a and the improvement method shown in analysis result D42b, based on the result information D44a, and the target device 2 implements the improvement method shown in analysis result D42b, based on the result information D44b. Information input to the display unit 7 and the target device 2 may be directly input from the control system 4000 (more specifically, the model interface 6), or it may be input indirectly via a communication network, other devices (servers, various conversion devices, etc.) or by human intervention.

[0302] The control system 4000 may acquire status information D45 and feedback information D46 if there is a change in the state of the target device 2, such as when the target device 2 is controlled, or if there is feedback from the output destination (step S414). Note that the processing in step S414 is not mandatory and may be omitted as appropriate.

[0303] The control system 4000 may repeat the process of steps S410 to S414 multiple times (for example, until a desired state is reached in the target work environment).

[0304] As described above, according to this embodiment, understanding the situation and acquiring improvement methods are performed in two stages using different learning models, so the accuracy of the final output can be improved, and as a result, the efficiency of the work related to monitoring the work situation can be increased.

[0305] For example, when assessing a situation, it's important to broadly detect abnormal conditions in the work environment, such as "something unusual has happened." On the other hand, when acquiring corrective measures, specific information is important, such as "stop this machine, move the workpiece to point A, return the machine to state B, and then restart it."

[0306] When the level of abstraction of the information to be extracted, i.e., the target information, differs, attempting to train and extract it together using a single learning model may lead to a decrease in the accuracy of the output. In particular, when acquiring improvement methods, specific methods are required based on knowledge and information of the work environment. In such cases, separating the learning models and providing them with appropriate domain knowledge (environmental information) can more reliably improve the accuracy of the output.

[0307] Furthermore, when attempting to obtain solutions for two different tasks—understanding the situation and acquiring methods for improvement—with a single learning model, the problem of hallucination is likely to become significant. This is because a function may implicitly operate within the model algorithm to adjust the solution for one task (understanding the situation) so that the solution for the other task (acquiring methods for improvement) appears more plausible. This embodiment also has an effect on this problem of hallucination. That is, by separating the learning model to correspond to the two tasks of understanding the situation and acquiring methods for improvement, modalities entering each learning model can be suppressed, and as a result, the magnitude of hallucination can be reduced, thereby improving the accuracy of the final output.

[0308] Furthermore, in this embodiment, the analysis results D42a and D42b, which are the output results of the learning model unit 400a and the learning model unit 400b, can be verbalized and displayed on the display unit 7. By having a person confirm the contents, hallucination can be suppressed, and methods for improving the situation can be implemented more reliably.

[0309] Furthermore, the control system 4000 of this embodiment can be applied not only to monitoring the control systems of equipment within a factory as described above, but also, for example, to monitoring logistics targets in a logistics system.

[0310] Variation 4-1. Next, a modified example of the control system 4000 will be described. Figure 25 is a configuration diagram showing an example of control system 4000a, which is a modified example of the control system 4000 according to this embodiment. Elements identical to those in control system 4000 are denoted by the same reference numerals and their descriptions are omitted.

[0311] The control system 4000a shown in Figure 25 differs from the control system 4000 in that it has two analysis means for analyzing and improving the situation in different ways, and switches between the analysis means used as appropriate depending on the situation that occurs.

[0312] The control system 4000a shown in Figure 25 includes a first analysis unit 41-1 which uses the learning model units 400a and 400b described above to analyze the situation and acquire improvement methods, as well as a second analysis unit 41-2, a switching unit 42, and an output switching switch 43.

[0313] The second analysis unit 41-2 is not limited in any way as long as it performs a means of analyzing the situation and obtaining improvement methods for the input information D41 in a manner different from that of the first analysis unit 41-1. For example, the second analysis unit 41-2 may be a means of performing a rule-based analysis of the situation and obtaining improvement methods. For example, when the input information D41 is input, the second analysis unit 41-2 may determine whether the input information D41 matches a predetermined pattern of abnormality, and if it matches any of the abnormality patterns, it may obtain an improvement method corresponding to that abnormality pattern. The second analysis unit 41-2 outputs an analysis result D42c which includes at least a method of improving the situation.

[0314] The analysis result D42c may include, for example, information corresponding to the result information D44a described above and information corresponding to the result information D44b. In this modified example, the analysis result D42c includes at least result information D44b indicating an improvement method obtained by the second analysis unit 41-2.

[0315] In this modified example, the second analysis unit 41-2 may be implemented as an internal execution module, for example, by being mounted in a PLC, information processing device, etc., located within the work environment.

[0316] The switching unit 42 is a means for switching the control destination for input information D41 according to predetermined conditions. In this modified example, the switching unit 42 switches the control destination for input information D41 between the first analysis unit 41-1 and the second analysis unit 41-2. The switching unit 42 may, for example, switch the control destination for input information D41 depending on whether or not the input information D41 conforms to an existing rule. In this case, the switching unit 42 may switch the control destination by switching the output destination of input information D41 to the second analysis unit 41-2 if the input information D41 conforms to an existing rule, and by switching the output destination of input information D41 to the first analysis unit 41-1 if it does not conform to an existing rule.

[0317] The switching unit 42 may, for example, switch the control destination for input information D41 in accordance with instructions from a supervisor. The switching unit 42 may also switch the control destination for input information D41 based on, for example, time, work content, or the presence or absence of a supervisor. The switching unit 42 may also switch the control destination for input information D41 based on, for example, whether or not an abnormality has occurred in the work environment. Here, the presence or absence of an abnormality in the work environment may be determined, for example, by whether or not an abnormality signal has been generated. For example, in the event of an abnormality, the switching unit 42 may switch the control destination for input information D41 to the first analysis unit 41-1. The switching unit 42 may also switch the control destination for input information D41 based on, for example, the degree or urgency of the abnormality occurring in the work environment.

[0318] Furthermore, the switching unit 42 may control an output switching switch 43 that switches the connection path (circuit or communication path, etc.) connecting the output of the first analysis unit 41-1 or the output of the second analysis unit 41-2 to the target device 2 and display unit 7 which are to be the output destinations for the analysis results, in accordance with the switching of the control destination for the input information D41.

[0319] The switching unit 42 may, for example, when switching the control destination for input information D41 to the first analysis unit 41-1, control the output switching switch 43 to turn on the connection path connecting the output of the first analysis unit 41-1 to the target device 2 and the display unit 7, and turn off the connection path connecting the output of the second analysis unit 41-2 to the target device 2 and the display unit 7. Similarly, the switching unit 42 may, for example, when switching the control destination for input information D41 to the second analysis unit 41-2, control the output switching switch 43 to turn on the connection path connecting the output of the second analysis unit 41-2 to the target device 2 and the display unit 7, and turn off the connection path connecting the output of the first analysis unit 41-1 to the target device 2 and the display unit 7.

[0320] Figure 26 is a flowchart showing an example of operation of this modified example. In the example shown in Figure 26, when the control system 4000a receives input information D41 in step S410, the switching unit 42 switches the control destination for input information D41 according to predetermined conditions (step S421). In the example shown in Figure 26, the switching unit 42 determines whether or not input information D41 matches an existing rule, and if it determines that it does not match (No in step S421), it proceeds to the first analysis process (step S422). On the other hand, if it determines that input information D41 matches an existing rule (Yes in step S421), it proceeds to the second analysis process (step S423).

[0321] In the first analysis process of step S422, the learning model unit 400a and the learning model unit 400b, which are the first analysis unit 41-1, perform an analysis of the situation and acquire methods for improvement. As a result of the first analysis process, the learning model unit 400a and the learning model unit 400b output an analysis result D42a including the analysis results of the situation and an analysis result D42b including methods for improving the situation.

[0322] In the second analysis process of step S423, the second analysis unit 41-2 analyzes the situation and obtains improvement methods according to existing rules. The second analysis unit 41-2 outputs, for example, an analysis result D42c that includes at least a method for improving the situation as a result of the first analysis process.

[0323] When the results of the analysis processing by the first analysis unit 41-1 or the second analysis unit 41-2 are output, the target device 2 is controlled and / or information is displayed on the display unit 7 based on the results of either analysis processing, depending on the state of the output switching switch 43 (step S424).

[0324] In this example, when the first analysis unit 41-1 is performing analysis processing, the connection path between the output of the first analysis unit 41-1 and the target device 2 and the display unit 7 is turned on. In this case, the model interface 6 may, for example, output result information D44a indicating the situation analysis results and improvement methods to the display unit 7 based on analysis results D42a and D42b, and also output result information D44b indicating improvement methods based on analysis result D42b to the target device 2. On the other hand, when the second analysis unit 41-2 is performing analysis processing, the connection path between the output of the second analysis unit 41-2 and the target device 2 and the display unit 7 is turned on. In this case, result information D44a indicating the situation analysis results and improvement methods may be output to the display unit 7 based on analysis result D42c output from the second analysis unit 41-2, and / or result information D44b indicating improvement methods may be output to the target device 2.

[0325] The display unit 7 may, for example, display result information D44b in a manner that can be confirmed by the worker. In that case, the worker may refer to the result information D44b displayed on the display unit 7, confirm the improvement method indicated by the result information D44b, and perform the work related to that method. Alternatively, the worker may confirm the improvement method indicated by the result information D44b and judge its validity. At this time, if the improvement method indicated by the result information D44b is not valid, the worker may prompt the learning model unit 400b to acquire an alternative improvement method (re-acquisition of model output data). When the learning model unit 400b receives information requesting the re-acquisition of model output data, for example, it may change some of the input information, some of the model parameters, or the reference destination of the reference information, and then re-acquire the model output data.

[0326] Subsequent processing may be the same as that of other control systems according to this embodiment.

[0327] As described above, this modified version is equipped with multiple analysis units that analyze the situation and acquire improvement methods in different ways, and is configured to switch between them depending on the situation. This makes it possible to control the situation in a way that is more appropriate to the situation. For example, for problems with a clear cause, the second analysis unit, which has a high processing load, can immediately analyze the situation and propose and implement improvement methods, while for problems with an unclear cause, the first analysis unit, which uses a learning model, can analyze the complex situation and propose and implement better improvement methods.

[0328] In the example described above, the first analysis unit 41-1 uses two learning models to perform situation analysis and acquire improvement methods. However, the configuration of the first analysis unit 41-1 is not limited to the example described above. For example, if situation analysis is not required, the learning model unit 400a can be omitted. Similarly, if acquiring improvement methods is not required, the learning model unit 400b can be omitted. Furthermore, it is possible to perform both situation analysis and acquisition of improvement methods with a single learning model unit.

[0329] For example, if the first analysis unit 41-1 includes a learning model unit 400b that acquires methods for improving the situation based on input information D41, the switching unit 42 may switch the control destination for input information D41 to the first analysis unit 41-1 in the event of an abnormality. In that case, the learning model unit 400b of the first analysis unit 41-1 should be configured to output information indicating improvement methods corresponding to the abnormality situation indicated by input information D41 when input information D41 is received. At this time, the learning model unit 400b may refer to the device information storage unit 410 accessible by the control system and output information indicating improvement methods corresponding to the situation.

[0330] Embodiment 5. Next, Embodiment 5 will be described. This embodiment describes an example of using a learning model to support response work in a call center or product website, where a response is provided to information sent by a user. Here, information sent by a user may include inquiries or opinions about a certain service, information, event, or product.

[0331] Figure 27 is a configuration diagram showing an example of a control system 5000 according to Embodiment 5. The control system 5000 shown in Figure 27 comprises a learning model unit 500, a reference information storage unit 12 (referred to as the reference information DB in the figure), a database search unit 511 (referred to as the DB search unit in the figure), a control generation unit 512, a speech recognition unit 513v, and a speech synthesis unit 514v. Here, the reference information storage unit 12, the database search unit 511, and the control generation unit 512 may be provided as part of the learning model unit 500.

[0332] When input information D51 is received, the learning model unit 500 outputs response information D52 indicating the response content. For example, when input information D51 is received, the learning model unit 500 outputs response information D52 based on model information D102. The configuration of the learning model unit 500 may be basically the same as that of the learning model unit 100 in Embodiment 1.

[0333] In this embodiment, the learning model unit 500 is a model and its operating environment configured to output response information D52 corresponding to input information D51 when input information D51 is received. Alternatively, the learning model unit 500 may be a model and its operating environment configured to generate and output response information D52 based on input information D51 and other information that can be referenced by the learning model unit 500 when input information D51 is received.

[0334] In this embodiment, input information D51 includes information indicating content transmitted by user 1, etc. Input information D51 may also include information indicating content for which a response is required in the work environment. Input information D51 may be, for example, text, images, audio, or a combination thereof indicating an inquiry or opinion regarding a certain service, information, event, or object. Input information D51 may be, for example, text, images, audio, or a combination thereof indicating multiple inquiries or opinions regarding a certain service, information, event, or object. Furthermore, input information D51 may include information indicating temporally continuous transmitted content, in which case it may be time-series data of a predetermined data structure including text, images, audio, or a combination thereof indicating the transmitted content as described above. The method of indicating the transmitted content is assumed to conform to the input format of the model used by the learning model unit 500, however, this is not limited to cases where error processing, correction processing, or conversion processing is provided before the learning model unit 500.

[0335] Response information D52 includes information indicating a response to the content of the transmission included in input information D51. Response information D52 may, for example, be information indicating a response to an inquiry or opinion regarding a service, information, event, or object indicated by the content of the transmission included in input information D51.

[0336] The reference information storage unit 12 stores model reference information D104, which the model control unit 101 of the learning model unit 500 refers to in order to output response information D52. Model reference information D104 includes, for example, information relating to a service, information, event, or thing that may be included in the input information D51. Here, the reference information storage unit 12 may specifically store information relating to a particular service, information, event, or thing as model reference information D104. Model reference information D104 may include, for example, a digitized version of a response manual. Also, model reference information D104 may include, for example, a history of previously input information D51 or the content of messages contained therein. In this case, the reference information storage unit 12 may store, along with information of the originating user 1 (e.g., user identifier, user attribute information, etc.), history information indicating previously input information D51 or the content of messages contained therein as model reference information D104. Hereinafter, in this embodiment, information indicating the state of the originating user 1 may be referred to as state information D55.

[0337] The database search unit 511 is a search engine for the reference information storage unit 12 and other databases. In response to a request from the model control unit 101 of the learning model unit 500, the database search unit 511 searches the databases to which it is connected that it can access and outputs the search results. At this time, the database search unit 511 may be restricted in which databases it can access.

[0338] The control generation unit 512 is an interface for setting preconditions when the learning model unit 500 (in particular, the model control unit 101) generates model output data. The control generation unit 512 may also be an interface used, for example, to recognize information to be controlled by the learning model unit 500 and / or to set the output trend. Here, the information to be controlled is information indicating the object to which the control of the model control unit 101 is focused. The model control unit 101 may be configured to generate model output data D103 from model input data D101 based on the information to be controlled indicated by the control generation unit 512. The control generation unit 512 may, for example, recognize a portion of the model input data entered by the user as information to be controlled, recognize information generated by the model control unit 101 as information to be controlled, or recognize information generated by the model control unit 101 and modified by other control units as information to be controlled. The settings for the information to be controlled and / or the output trend may be specified by the user, specified by an external processing unit, or specified by the control generation unit 512 according to a predetermined algorithm.

[0339] The speech recognition unit 513v recognizes the speech indicated by the input information D51v when the input from user 1 includes input information D51v in voice format, converts it to a format that matches the data format of the learning model unit 500, and outputs it. The speech recognition unit 513v may, for example, convert the input information D51v in voice format to input information D51 in text format.

[0340] The speech synthesis unit 514v converts the content indicated in response information D52 into speech format and outputs it. For example, if the response information D52, which is output from the learning model unit 500, includes a data format other than speech, the speech synthesis unit 514v converts the content of that portion indicated in response information D52 into speech format and outputs it. For example, if the response information D52 is a data structure that includes a specification of a data format, the speech synthesis unit 514v may convert the data element in which a speech format is specified into a speech format and output it. For example, the speech synthesis unit 514v may convert response information D52 in text format into response information D52v in speech format.

[0341] In the example described above, audio data is used for input and output with User 1, but the data format used for input and output with User 1 is not limited to audio format. In that case, instead of the speech recognition unit 513v and the speech synthesis unit 514v, a processing unit that converts the data format used for input from User 1 to the data format used for input to the learning model unit 500 and a processing unit that converts the data format used for output from the learning model unit 500 to the data format used for input to User 1 may be provided.

[0342] Furthermore, if the learning model unit 500 can accept the data format used for input from user 1, the speech recognition unit 513v can be omitted. Also, if user 1 can accept the data format used for output from the learning model unit 500, the speech synthesis unit 514v can be omitted.

[0343] In this embodiment, input information D51 corresponds to model input data D101. Response information D52 corresponds to model output data D103. The learning model unit 500 (in particular, the model control unit 101) may be configured, for example, to receive input information D51 and output response information D52 corresponding to the input information D51 based on model information D102 and, if necessary, model reference information D104.

[0344] In such cases, the model generation unit 107, which is provided in correspondence with the learning model unit 500, may, for example, perform machine learning using model learning data D105 that includes candidate input information D51 that can be input to the model control unit 101, to generate or update model information D102. Alternatively, the model generation unit 107 may, for example, perform machine learning using model learning data D105 that includes candidate input information D51 that can be input to the model control unit 101 and candidate response information D52 that corresponds to them, to generate or update model information D102.

[0345] Although not shown in the illustration, in this embodiment as well, state information D55 and / or feedback information D56 may be obtained from the output destination of the model output data D103 of the learning model unit 500 and / or the information generated therefrom. The control system 5000 may, for example, output the obtained state information D55 and / or feedback information D56 as information indicating the response result to a predetermined supervisor, the learning model unit 500, or other devices not shown. The control system 5000 may also be configured to return a query D57 to user 1 if the input information D51 contains unclear or uncertain information. Furthermore, the control system 5000 can generate supplementary information D58 for the input and output data of the learning model unit 500 based on the obtained state information D55 and / or feedback information D56, and issue it to user 1, a predetermined supervisor, the learning model unit 500, or other devices not shown. The handling of state information D55, feedback information D56, query D57, and supplementary information D58 may be basically the same as in Embodiment 1.

[0346] The control system 5000 may also further include a state acquisition unit 530 (not shown) that acquires state information D55 and / or feedback information D56 and issues supplementary information D58 as necessary. The state acquisition unit 530 is the same as the state acquisition unit 130 in Embodiment 1.

[0347] In this embodiment, the input information D51 received by the control system 5000 can be said to be information relating to a request in the work environment (here, the content of a transmission requesting a response in an environment where a response operation is performed in response to an inquiry). Therefore, the input information D51 received by the control system 5000 can be said to be an example of first information indicating a request in the work environment. Furthermore, the response information D52 can be said to be information used for the operation (response operation) in correspondence with such input information D51. Hereinafter, the response information D52 output to a predetermined output destination from the operating environment of the learning model to which model input data based on the input information D51 is input may be referred to as second information.

[0348] Next, the operation of the control system 5000 of this embodiment will be described. Figure 28 is a flowchart showing an example of the operation of the control system 5000.

[0349] In the example shown in Figure 28, the control system 5000 first receives input information D51v (step S510). For example, the input unit 102 or input processing unit 201 described above may receive the input information D51v. The received input information D51v is input to the speech recognition unit 513v.

[0350] When the speech recognition unit 513v receives the input information D51v, it recognizes the speech contained in the input information D51v and converts it into input information D51 that matches the input data format of the learning model unit 500 (step S511). The converted input information D51 is input to the learning model unit 500 as model input data D101.

[0351] If the speech recognition unit 513v is omitted, the received input information D51v may be input to the learning model unit 500 as model input data D101.

[0352] Next, the control system 5000 performs a process to generate response information D52 using the learning model unit 500 (step S512). In step S512, the learning model unit 500 (more specifically, the model control unit 101) outputs response information D52 corresponding to the input information D51 based on the model information D102, the input information D51, and, if necessary, the model reference information D104.

[0353] In step S512, the preprocessing unit 105 and / or postprocessing unit 106 of the learning model unit 500 may further perform the processing described above.

[0354] The response information D52 output from the learning model unit 500 is input to the speech synthesis unit 514v, for example (step S513). The response information D52 to the speech synthesis unit 514v may be input directly from the control system 5000 (more specifically, the learning model unit 500 or the information processing device 10 as its operating environment, etc.), or it may be input indirectly via a communication network, other devices (servers, various conversion devices, etc.) or by human intervention.

[0355] Next, the speech synthesis unit 514v converts the input response information D52 into speech-format response information D52v and outputs it (step S514). The speech synthesis unit 514v may generate the response information D52v by, for example, synthesizing speech that speaks the response content indicated by the response information D52, which is in a data format other than speech. The response information D52v is output to user 1, the source of the input information D51v (step S515).

[0356] If the speech synthesis unit 514v is omitted, the response information D52 output from the learning model unit 500 may be output to user 1, which is the source of the input information D51v.

[0357] As described above, in this embodiment, in response to information sent from user 1, the learning model unit 500 can dynamically generate response information D52 and send it back to the user who sent the information, without the need to prepare an operator or a site with pre-embedded response content. This improves the efficiency and performance of the response process.

[0358] Variation 5-1. Next, a modified example of the control system 5000 will be described. Figure 29 is a configuration diagram showing an example of the control system 5000a, which is a modified example of the control system 5000 according to this embodiment. Elements identical to those in the control system 5000 are denoted by the same reference numerals and their descriptions are omitted.

[0359] The control system 5000a shown in Figure 29 differs from the control system 5000 in that it includes a correct / incorrect judgment unit 515.

[0360] The correctness determination unit 515 determines whether the content indicated by the response information D52, which is an output from the learning model unit 500, is correct or not. The correctness determination unit 515 may, for example, output the response information D52 to user 1 or update the contents of the reference information storage unit 12 only if it determines that the content indicated by the response information D52 is correct.

[0361] Furthermore, if the correctness determination unit 515 determines, for example, that the content indicated by response information D52 is incorrect, it may prompt the learning model unit 500 to acquire different response information D52 (re-acquisition of model output data). The correctness determination unit 515 may be provided, for example, as an example of the post-processing unit 106 described above.

[0362] Other aspects may be the same as other control systems according to this embodiment.

[0363] As described above, this modified version determines whether the content of the response information output from the learning model unit 500 is correct, and based on the result, it determines whether or not to output to the user, whether to reacquire the response information, and whether to update the reference information, thereby further improving the performance of the response process.

[0364] Variation 5-2. Next, a second modified example of the control system 5000 will be described. Figure 30 is a configuration diagram showing an example of control system 5000b, which is a modified example of the control system 5000 according to this embodiment. Elements that are the same as those in control system 5000 and control system 5000a are denoted by the same reference numerals and their descriptions are omitted.

[0365] As shown in Figure 30, the control system 5000b may further include an emotion determination unit 516.

[0366] The emotion determination unit 516 uses the input information D51 and other information to determine the emotion of the originating user 1. Alternatively, the emotion determination unit 516 may determine the emotion of user 1 after the response information D52 from the learning model unit 500 has been output to user 1.

[0367] The emotion of user 1 determined by the emotion determination unit 516 may be input to the learning model unit 500 as state information D55 included in the model reference information D104, or it may be recorded in the reference information storage unit 12 as history along with the model's input and output data.

[0368] As a method for recording in the reference information storage unit 12, for example, the control system 5000b may further include a registration determination unit 518, and the registration determination unit 518 may determine whether or not to record in the reference information storage unit 12 based on the result of the emotion determination unit 516's determination of the user 1's emotion.

[0369] The registration determination unit 518 may, for example, if the determined emotion of user 1 is positive, record the model's input and output data as historical information in the reference information storage unit 12 as a good case. In this case, if the registration determination unit 518 has a determination result of user 1's emotion before the output of response information D52 from the learning model unit 500, it may also record the model's input and output data, including the emotion information before and after the response, as historical information in the reference information storage unit 12.

[0370] Furthermore, if the registration determination unit 518 determines, for example, that the environment of the determined user 1 is negative, it may cause the reference information storage unit 12 to record the model's input and output data as historical information, treating it as a bad case. In this case, if the registration determination unit 518 has a determination result of user 1's emotions before the output of response information D52 from the learning model unit 500, it may cause the reference information storage unit 12 to record the model's input and output data, including the emotion information before and after the response, as historical information.

[0371] Furthermore, the control system 5000b may also include an additional learning unit 519, which, when updating the contents of the reference information storage unit 12, reconstruct (additionally learn) the model reference information D104 and other information referenced by the model control unit 101 stored in the reference information storage unit 12 based on the updated information.

[0372] Furthermore, the control system 5000b may include an evaluation acquisition unit 517 in place of or in addition to the emotion determination unit 516.

[0373] The evaluation acquisition unit 517 queries user 1 for an evaluation of response information D52 and obtains evaluation information D59 as the response. Evaluation information D59 can be used, for example, to update information referenced by the model, to perform additional learning, etc., similar to the sentiment of user 1 described above.

[0374] Furthermore, the control system 5000b may also include a control determination unit 520.

[0375] The control determination unit 520 specifies the information to be controlled and / or the output tendency settings to the control generation unit 512 based on the speech recognition results for the input information from user 1, the emotion determination results and / or the evaluation results of response information D52, instructions from the operator (not shown), etc. Here, the speech recognition results for the input information from user 1 may include information such as user 1's attributes, emotions, region, language, past usage history, and usage frequency. The control determination unit 520 may also set the synthesized speech to the speech synthesis unit 514v based on the speech recognition results for the input information from user 1, the emotion determination results and / or the evaluation results of response information D52, instructions from the operator (not shown), etc.

[0376] The control determination unit 520 can, for example, specify the difficulty level of the explanation in the response, the way of speaking (tone and intonation), the level of language and grammar, the level of politeness, the speaker's position, and the conclusion of the conversation, as an example of setting the tendency of the output. It can also specify the gender, tone, and intonation of the synthesized voice. Furthermore, the control determination unit 520 can, for example, specify the gender, way of speaking, the level of language and grammar, and the level of politeness of the synthesized voice, as an example of setting the synthesized voice. The control determination unit 520 may, for example, make these settings based on predetermined setting rules.

[0377] The elements of the control system 5000b shown in Figure 30 can be appropriately selected according to the desired function.

[0378] Other aspects may be the same as other control systems according to this embodiment.

[0379] As described above, according to this modified configuration, the control determination unit 520 specifies the information to be controlled and / or the output tendency setting based on information obtainable from the control system 5000b, so that response information that is more likely to match the request of the sender can be generated. Therefore, the performance of the response work to the user can be further improved.

[0380] Modification 5-3. Next, a third modified example of the control system 5000 will be described. Figure 31 is a configuration diagram showing an example of control system 5000c, which is a modified example of the control system 5000 according to this embodiment. Elements that are the same as those in control system 5000, control system 5000a, and control system 5000b are denoted by the same reference numerals and their descriptions are omitted.

[0381] As shown in Figure 31, the control system 5000c may further include an image analysis unit 513i, an image generation unit 514i, and a program generation unit 514p.

[0382] The image analysis unit 513i analyzes the image indicated by the input information D51i when the input from user 1 includes input information D51i in image format, converts it to a format that matches the data format of the learning model unit 500, and outputs it. The image analysis unit 513i may, for example, convert the input information D51i in image format to input information D51 in text format.

[0383] The image analysis unit 513i may, for example, analyze an image captured from the operation screen of a product owned by user 1, identify which product and operation screen it is, what operation state it is in, and output it as descriptive text. Alternatively, the image analysis unit 513i may, for example, analyze an image captured from a shopping site that user 1 is viewing, identify which site and operation screen it is, what operation state it is in, and output it as descriptive text.

[0384] The image generation unit 514i generates and outputs an image based on the response information D52. For example, if the response information D52, which is the output from the learning model unit 500, includes a data format other than an image, the image generation unit 514i may generate and output an image showing the content of the part indicated in the response information D52. For example, if the response information D52 is a data structure that includes a specification of a data format, the image generation unit 514i may convert the data element in which an image format is specified in the specification into an image format and output it. For example, the image generation unit 514i may generate image format response information D52i based on text format response information D52. For example, the image generation unit 514i may perform a synthesis process to add the content indicated in text format response information D52 as an annotation to the image included in the input information D51. Alternatively, based on text format response information D52, the image generation unit 514i may perform a process to highlight a part of the image included in the input information D51. The image generation unit 514i may generate an image from input information (response information D52 and, if necessary, input information D51) using a learned model.

[0385] The program generation unit 514p converts the content indicated in response information D52 into a data format for a predetermined program and outputs it. For example, if the response information D52, which is output from the learning model unit 500, includes a data format other than the data format for a predetermined program, the program generation unit 514p converts the content of that portion indicated in response information D52 into the data format for a predetermined program and outputs it. For example, if response information D52 is a data structure that includes a specification of a data format, the program generation unit 514p may convert the data element in which the data format for a predetermined program is specified into the data format for a predetermined program and output it. For example, the program generation unit 514p may convert response information D52 in text format into response information D52p in the data format for a predetermined program. The program generation unit 514p may generate a predetermined program from input information using a learning model.

[0386] The image analysis processing by the image analysis unit 513i is performed, for example, in step S511 described above. The image generation processing by the image generation unit 514i and the program generation processing by the program generation unit 514p are performed, for example, in step S514 described above.

[0387] Other aspects may be the same as other control systems according to this embodiment.

[0388] As described above, this modified version allows for inquiries and responses to be made not only through voice but also through both voice and images, enabling more effective responses to inquiries, such as those regarding the operation screen. Furthermore, this modified version allows for the provision of program information to the source in addition to voice and images, enabling more effective responses to inquiries such as those regarding bug fixes.

[0389] Modification 5-4. Next, a fourth modified example of the control system 5000 will be described. Figure 32 is a configuration diagram showing an example of a control system 5000d, which is a modified example of the control system 5000 according to this embodiment. Elements that are the same as those in control systems 5000 to 5000c are denoted by the same reference numerals and their descriptions are omitted.

[0390] In this modified version, the system has a function that switches between a response by operator 8 or a response by a different learning model, based on the content of the inquiry from user 1 and / or the output result from the learning model.

[0391] As shown in Figure 32, the control system 5000d may further include a call confirmation unit 531 and an output selection unit 532.

[0392] Here, the control system 5000d is provided with a learning model unit 500a as a first response function, and with an operator 8 and a communication channel with operator 8 as a second response function. Furthermore, the control system 5000d may also provide a third response function, which is another learning model unit 500b with a different algorithm or data used from the learning model unit 500a. In this case, the third response function may also include an operator 8 and a communication channel with operator 8. The type and number of response functions are not particularly limited. For example, the response function to be switched over may be a response system that does not use a learning model.

[0393] In this example, we will explain the case where the learning model unit 500a, which is considered the first response function, is the learning model unit 500 described above, the second response function is the operator 8 and the communication channel with the operator 8, and the third response function is another learning model unit 500b that uses a different algorithm or data from the learning model unit 500a.

[0394] Here, the learning model unit 500a may be a local learning model that obtains output results based on local information, such as having restrictions on the database it references, while the learning model unit 500b may be a global learning model that obtains output results based on global information, such as having free access to an external network.

[0395] The call confirmation unit 531 switches the processing destination for response processing based on the content of the inquiry from user 1 and / or the output result from the learning model.

[0396] The call confirmation unit 531 may, based on the content of the inquiry from user 1 and / or the output result from the learning model, for example, if it determines that the accuracy of the output by the first response function is not expected to be sufficient, call operator 8 as a second response function. The call confirmation unit 531 may, for example, use a communication channel with operator 8 to call operator 8 and input input information D51 to operator 8's operating device. Alternatively, the call confirmation unit 531 may use a communication channel with operator 8 to call operator 8 and input input information D51 to operator 8's operating terminal (not shown).

[0397] Furthermore, if the call confirmation unit 531 determines that it is impossible to call the second response function or that the accuracy of the output cannot be expected, it may call the learning model unit 500b as a third response function. The call confirmation unit 531 may, for example, call the learning model unit 400b by inputting input information D51 to the learning model unit 500b using the interface with the learning model unit 400b.

[0398] Here, the output accuracy can be determined, for example, by using evaluation values ​​or likelihoods output by the response function itself, or by using the reliability evaluation described above. Furthermore, if the response function itself outputs a message indicating that it does not know the answer or requests the use of another function, the accuracy can also be determined by the presence or absence of such a message.

[0399] The output selection unit 532 selects response information D52 to output to user 1 based on the result of the response processing switch by the call confirmation unit 531. If the response processing switch by the call confirmation unit 531 determines that the execution entity of the response processing is the first response function, the output selection unit 532 outputs response information D52a, which is output from the first response function, to user 1. If the response processing switch by the call confirmation unit 531 determines that the execution entity of the response processing is the second response function, the output selection unit 532 outputs response information D52b, which is output from the second response function, to user 1. If the response processing switch by the call confirmation unit 531 determines that the execution entity of the response processing is the third response function, the output selection unit 532 outputs response information D52c, which is output from the third response function, to user 1.

[0400] The output selection unit 532 may output the output from the selected response function toward user 1 by controlling an output selection switch (not shown) that switches the connection path (circuit or communication path, etc.) connecting the response function designated as the execution entity and user 1 designated as the output destination.

[0401] Here, the connection path between the response function and user 1 may include various conversion devices such as the aforementioned speech synthesis unit, image generation unit, and program generation unit, as well as predetermined interfaces, as needed.

[0402] For example, if the text input by operator 8 using the operation terminal is output as response information D52b, the connection path between the response function and user 1 may include a speech synthesis unit that converts text to speech. The output selection unit 532 can also accept a modified version of the response information D52a output by the first response function as an output of a second response function, etc. In this case, operator 8's operation terminal includes a text display unit and a text input unit, and the control system 5000d may accept, for example, response information D52b which is a modified version of the response information D52a output from operator 8's operation terminal.

[0403] Other aspects may be the same as other control systems according to this embodiment.

[0404] As described above, this modified version allows for the generation of responses not only using the learning model unit 500 described above, but also, for example, by an operator or by using other learning models (including, for example, a tandem structure model connecting multiple models, a multimodal model, or a model specifically trained for a particular device or service). This further improves the performance of the response work to the user.

[0405] In the embodiments described above, examples of system configurations corresponding to the tasks to be focused on were illustrated and explained, but the control system according to this disclosure is not limited to the examples described above. For example, the control system according to this disclosure can be created by appropriately combining one or more of the embodiments described above.

[0406] As an example, the control system according to this disclosure can combine the configuration of Embodiment 1 and the configuration of Embodiment 4, and by utilizing the functions of Embodiment 4, information indicating a solution obtained from sensor data can be input to the control system of Embodiment 1, converted into a program, and directly control the target device 2.

[0407] Embodiment 6. Next, Embodiment 6 will be described. In this embodiment, an example of supporting the control of the target device 2, as described in Embodiment 2, will be explained in more detail. Note that the configuration and operation in this embodiment may be applied to any of Embodiments 1, 3-5. Below, the differences from Embodiment 2 will be mainly described, and explanations that overlap with Embodiment 2 will be omitted.

[0408] Figure 33 is a configuration diagram showing an example of a control system 2000a according to Embodiment 6. The target equipment 2 in this embodiment is, for example, industrial equipment used in a factory, etc., similar to Embodiment 2. Specifically, it is a PLC (also called a sequencer), servo system, motion controller, NC (Numerical Control) control device (numerical control device), display, sensor, processing machine, robot, conveying device, assembly device, control device for other machines, inverter, etc., but is not limited to industrial equipment. In the example shown in Figure 33, the control system 2000a comprises a learning model unit 200, an equipment information storage unit 210, a verification unit 220, an input processing unit 240, and a proposal unit 250. The target equipment 2 controlled by the control system 2000a may be one or more. When a system including multiple target equipment 2 is the controlled object, the system will hereinafter also be referred to as the target system. The multiple target equipment 2 constituting the target system may be of the same type or may be a mixture of different types of equipment. Also, the target equipment 2 may include sensors.

[0409] The input processing unit 240 receives input information D21 from user 1. The input processing unit 240 also processes the received input information D21 as preprocessing and outputs it to the learning model unit 200 as model input data D21a. As described in Embodiments 1 and 2, the input information D21 may be input in the form of audio data in which natural language is expressed by voice, text data expressed in natural language, text data containing codes in a defined format other than natural language, image data, a combination of text data and image data, or a data format that can be converted to these (such as audio data or video, which is a combination of audio data and image data). In addition, text data may be input in chat format, and in this case, the inquiry D27 described later may also be made via chat.

[0410] As described in Embodiment 2, the input information D21 in this embodiment is an example of first information indicating requirements in the work environment, and more specifically, it is information regarding requirements for the operation of the target device 2. Furthermore, the control command D22 can be said to be information used for the operation (requirements regarding the operation of the target device 2) in correspondence with such input information D21. Hereafter, as in Embodiment 2, the control command D22 may be referred to as second information.

[0411] Specifically, for example, the following instruction may be entered as input information D21. (1) Instructions for the movement of the target workpiece, which is the object being worked on by target equipment 2 (for example, "Cut workpiece X 10 cm from the right") (2) Instructions for the movement of target device 2 or the target device system (3) Specify the target position of the target device 2, the components of the target device 2, the target device system, or the target workpiece in terms of absolute position or relative displacement. (4) Specify the target device 2, the components of target device 2, and the operating speed of the target device system. (5) Instruct the task to be performed (for example, "Take all the parts out of the box," "Pack the fried chicken into the lunch box," etc.) (6) ON / OFF instructions for target device 2, components of target device 2, target device system, sensors, etc. (7) Instructions on which sensor to use (8) Specify the timing for acquiring sensor information. (9) Specify the layout configuration and issue instructions based on the specified layout (for example, specify a layout that shows the position of the container with the fried chicken and the position of the lunch boxes, and then issue instructions such as "Take two pieces of fried chicken out of the container and put them in the lunch box").

[0412] Alternatively, input information D21 may be entered using the user interface of the device operation screen. For example, the input processing unit 240 may segment the actions that the target device 2 can perform, display a device operation screen that shows the segmented phrases as options, and complete the sentence describing the action by receiving input of the selection result from user 1. Alternatively, the part to be selected from the options and the part to be freely entered by the user may be combined.

[0413] The input processing unit 240 may be the same as or different from the input processing unit 201 described in Embodiment 1. That is, the model input data D21a may be the same as or different from the model input data D101 in Embodiment 1. For example, if the target device 2 is used in a factory, production site, construction site, etc., noise may be present. Therefore, if the input information D21 is audio data, noise reduction processing to remove the noise can increase the likelihood of correctly recognizing the voice spoken by user 1. In addition, the input processing unit 240 may perform grounding processing as described in Embodiment 1.

[0414] Furthermore, if there are any unclear points in the input information D21, the input processing unit 240 may make an inquiry D27 to the user 1 to clarify the unclear parts. For example, if the input information D21 contains an instruction word and the object or action indicated by the instruction word is unclear, the input processing unit 240 may make an inquiry D27 to confirm its content by voice or screen display. Also, if the input processing unit 240 determines that the input information D21 is an instruction to move the entire target device 2 or a movable part of the target device 2, and that there is insufficient information regarding the location, such as where to move it or how to move it, the input processing unit 240 may make an inquiry D27 regarding the location.

[0415] For example, the input processing unit 240 deletes or modifies typical examples where the response efficiency decreases when the input information D21 described above is one of instructions (1) or (2) is input to the learning model unit 200, so that the response efficiency improves through learning. Response efficiency is an indicator used to reduce the number of repetitions when generating the final operation command to the machine by repeatedly re-questioning. Specifically, the input processing unit 240 may re-question if the target workpiece or manufacturing equipment is ambiguous. In this case, the input processing unit 240 may perform inquiry D27 by suggesting target workpieces or manufacturing equipment that are likely to be used based on past performance, etc. It may also re-question if the target position is ambiguous. For example, it may provide several candidates and allow user 1 to select the target position. Furthermore, the input processing unit 240 may re-question the path if the path for moving the target workpiece or manufacturing equipment is ambiguous. For example, the input processing unit 240 may, considering the acceleration and deceleration constraints of the movement of the target workpiece or manufacturing equipment, re-examine the input by presenting several correction amount candidates if it is preferable to take a shortcut using a curve rather than a right angle, or if it is easier to move along a curve. The input processing unit 240 may also re-examine the input by presenting candidates when a relative speed or relative movement amount is specified. In this case as well, the input processing unit 240 may re-examine the input by presenting candidates. Furthermore, the input processing unit 240 may present correction candidates using previously corrected results.

[0416] Furthermore, the input processing unit 240 may perform a query D27 for the part it determines has been omitted. For example, suppose input information D21 contains an object (A) indicating a part of the target device 2 and an instruction to perform an action, such as "move A". If there are multiple possible locations for moving A, the input processing unit 240 may perform a query D27 such as "Where do you want to move A?" or a query D27 that presents options such as "Do you want to move A to X or to Y?". Alternatively, if, for example, past experience shows that there are many instructions to move A to X, the input processing unit 240 may perform a query D27 that presents options in order of likelihood, such as "Do you want to move A to X?", so that the answer can be given as either correct or incorrect.

[0417] Furthermore, if the input processing unit 240 receives the instruction in (5) above, and the task is unclear, or if the order of tasks, the time interval between tasks, the quantity, etc., is unknown, it may ask for clarification and confirmation of the unclear points. For example, if the instruction is "Pack fried chicken into a lunch box," the input processing unit 240 may ask for the quantity, such as "How many pieces of fried chicken should be packed into the lunch box?", using inquiry D27.

[0418] Furthermore, when instructions (6), (7), and (8) are input, the input processing unit 240 may request clarification if the designation of the target device or sensor is unclear. For example, the input processing unit 240 may request clarification if the time to execute the instruction, the ON / OFF interval, or the ON / OFF conditions are unclear.

[0419] Furthermore, if the input processing unit 240 determines that demonstrative pronouns such as "that," "that over there," or "it" are used and that the meaning of the demonstrative pronoun is unclear, it may ask for clarification of the specific content of the demonstrative pronoun. In addition, it may automatically translate instructions entered in Japanese into English and add "a," "the," etc. in English if doing so would improve response efficiency. The format of instructions that improve response efficiency or improve the response rate (constraints, rules, etc. regarding instructions that improve the response rate) may be determined based on past performance. For example, the verification unit 220 may determine this using evaluation results obtained from evaluating control commands D22 obtained by inputting instruction sentences etc. to the learning model unit 200 in the past. Alternatively, user 1 may determine this based on their past experience using the control system 2000a. Conversely, the format of instructions that decrease response efficiency or improve the response rate may be determined, and the input processing unit 240 may determine that clarification is necessary and ask for clarification of the instruction content if an instruction in the format of an instruction that decreases response efficiency or improves the response rate is entered. Furthermore, if the verification unit 220 uses its evaluation results to propose modifications to user 1, the proposal unit 250 may present the modified content to user 1 via voice, display, etc., and the input processing unit 240 may accept input from user 1 regarding the presented content. Hereinafter, the information presented by the proposal unit 250 to user 1 will also be referred to as proposal information D29. In this case, if user 1 inputs a response indicating that the modifications proposed by the proposal unit 250 are acceptable, the modifications proposed by the proposal unit 250 will be input to the input processing unit 240 as input information D21.

[0420] Furthermore, if non-linguistic instructions such as photographs or videos are input, the input processing unit 240 may prompt for linguistic supplementation. For example, if a video showing the task to be performed by the target device 2 is input, and the type of target workpiece is difficult to determine from the video, the input processing unit 240 may ask for clarification of the type of target workpiece.

[0421] The input processing unit 240 may also present the input information D21 interpreted by the input processing unit 240 via voice, display, etc., and obtain permission from User 1 by receiving input from User 1 indicating that the presented content is acceptable, before outputting the model input data D21a to the learning model unit 200. Furthermore, if the input information D21 is modified by a re-examination as exemplified above, the input processing unit 240 may present the modified instructions, i.e., model input data D21a (or information expressing the content of model input data D21a in a format easily understood by User 1), to User 1 via voice, display, etc., and obtain permission from User 1 by receiving input from User 1 indicating that the presented content is acceptable, before outputting the model input data D21a to the learning model unit 200. In addition, when making an inquiry D27 to confirm unclear points, the input processing unit 240 may use information such as past performance and equipment constraints, as described above, and this information may be included in the equipment information D23 stored in the equipment information storage unit 210. Alternatively, although not shown in the diagram, an information storage unit for storing information used by the input processing unit 240 when querying D27 may be provided separately from the device information storage unit 210.

[0422] As described in Embodiment 2, when an operator who is not a skilled worker performs the control, or when introducing new control equipment (including version upgrades), using the control system 2000 described in Embodiment 2, even without knowing the control commands for the target equipment 2, leads to increased work efficiency and performance. Furthermore, during trial runs, startup adjustments, robot teaching, and troubleshooting, the appropriate control commands themselves may not be determined, and operators may have to repeatedly try and fail, regardless of whether they are skilled or not. Also, if the target equipment 2 is a manufacturing device or production system used in individual production, adjustments specific to the target equipment 2 are required. In such cases as well, by using the control system 2000 in Embodiment 2, operators only need to input natural language, images, and text data as input information D21, rather than the control commands themselves, thus improving work efficiency and performance. Similarly, in this embodiment as well, operators only need to input natural language, images, and text data as input information D21, rather than the control commands themselves, thus improving work efficiency and performance.

[0423] The device information storage unit 210 stores device information D23. Similar to Embodiment 2, device information D23 may also include status information D25 indicating the state of the target device 2. In addition, in this embodiment, device information D23 may further include at least one of the following: manuals (manuals for the target device 2, the devices constituting the target device 2, the target system, etc.), specifications of the target device 2 or the devices constituting the target device 2, CAD (Computer Aided Design) information for the target device 2 or the target system, CAD information for the object of work (target workpiece), layout information of the site where the target device 2 operates, user (operator) qualification information, user (operator) proficiency (skill level), user (operator) native language information, and recommended prompts. Recommended prompts can be used, for example, to present to the user to encourage input in accordance with the recommendation, or to convert the user's input based on the information in the recommended prompt, and display the converted result to the user for confirmation.

[0424] When the learning model unit 200 receives model input data D21a from the input processing unit 240, it outputs a control command D22 using the device information D23 stored in the device information storage unit 210. In this embodiment as well, the learning model unit 200 is a model and its operating environment configured to output a control command D22 corresponding to the model input data D21a when the model input data D21a is received, similar to Embodiment 2. The configuration of the learning model unit 200 may be basically the same as that of the learning model unit 100 in Embodiment 1.

[0425] The learning model unit 200 outputs a control command D22 using, for example, a model generated by machine learning, with the equipment information D23 as a constraint. For example, the learning model unit 200 may use the manual, the target equipment 2, or the specifications of the devices constituting the target equipment 2 to determine the format of the control command D22 that the target equipment 2 can accept, and generate and output the control command D22 according to the determined format. In addition, the learning model unit 200 may use the CAD information of the target equipment 2 or the target system, the CAD information of the target object, or the layout information of the site where the target equipment 2 operates to notify the confirmation unit 220 if the control command D22 output from the model (learning model) is an operation that exceeds the acceptable range or an operation that requires a path correction.

[0426] Furthermore, the learning model unit 200 may determine, based on the user's credentials, whether the task corresponding to control command D22 is permitted for user 1 who entered input information D21. If the task is not permitted for user 1, it may not generate the corresponding control command D22 and may notify the confirmation unit 220 that the task is not permitted. When the confirmation unit 220 is notified by the learning model unit 200 that the task is not permitted, it may notify the proposal unit 240 of this fact. As a result, the proposal unit 250 will present the task to user 1 as not permitted.

[0427] Furthermore, as described in Embodiment 1, at least a portion of the device information D23 may be used as input in machine learning along with the model input data D21a. In this case, at least a portion of the device information D23 is also used when the model is generated. That is, in this case, the learning model unit 200 is a model and its operating environment configured to output control commands D22 corresponding to the model input data D21a and the device information D23 when they are input. Note that information in the device information D23 that does not change may be incorporated into the model during machine learning and may not be input to the learning model unit 200.

[0428] The verification unit 220 performs a verification process to confirm the status of the equipment corresponding to the input information D21. The status of the equipment includes the status of at least some of the target equipment 2, the target system, and the target workpiece. The status of the equipment corresponding to the input information D21 is the status of the equipment with respect to the control command D22 generated based on the input information D21, and if the input information D21 is modified, it includes the status of the equipment with respect to the control command D22 generated based on the modified information. The information verification process may also be a process to allow user 1 to confirm the status of at least some of the target equipment 2, the target system, and the target workpiece.

[0429] Based on the verification results, the verification unit 220 determines that the model input data D21a or control command D22 needs to be modified, and outputs the modified model input data D21a or control command D22 to the proposal unit 250. The proposal unit 250 may also change the method of presenting the proposal information D29 based on the user's proficiency level and user credentials in the equipment information D23. For example, a highly proficient user may be presented with the control command D22 itself and asked to confirm it, while a less proficient user may be presented with information in a format that is easy to understand even for those unfamiliar with controlling the target equipment 2, such as a video or natural language.

[0430] The verification unit 220 may also input the corrected model input data D21a or control command D22 to the learning model unit 200. When the learning model unit 200 receives the corrected model input data D21a from the verification unit 220, it outputs the control command D22 using the corrected model input data D21a as input. When the learning model unit 200 receives the corrected control command D22, it outputs the corrected control command D22 to the target device 2.

[0431] Specifically, the verification unit 220 performs verification processing as follows: For example, the verification unit 220 receives a control command D22 from the learning model unit 200, performs a simulation simulating the target device 2 or target system based on the received control command D22, and outputs the simulation results to the proposal unit 250 along with the control command D22. Alternatively, the verification unit 220 may output information in which the control command D22 has been converted into a format easily understood by user 1, or input information D21 (or model input data D21a), along with the simulation results, instead of the control command D22. When performing the simulation, the verification unit 220 may use the device information D23 stored in the device information storage unit 210 to perform the simulation. Furthermore, the verification unit 220 may generate data for displaying the simulation results in AR (Augmented Reality) and output this data to the proposal unit 250. The data for AR display may be generated by the proposal unit 250. The verification unit 220 or the proposal unit 250 may generate video using a machine learning model that generates video from information other than images.

[0432] The verification unit 220 may, based on the simulation results, confirm whether any malfunctions occur during the operation of the target device 2, such as interference between the target device 2 and surrounding objects, interference between the target device 2 and the target workpiece, or the target device 2 performing an operation that deviates from the operation constraints. If the verification unit 220 determines that a malfunction will occur, it may perform a correction, such as by automatically generating an avoidance path to avoid interference. If a correction is performed, the verification unit 220 may output the correction amount along with information indicating the operation of the target device 2 after the correction to the proposal unit 250. The proposal unit 250 may present the correction amount and information indicating the operation of the target device 2 after the correction to the user 1. The user 1 may re-enter the input information D21 based on the correction amount, or may input an indication that they approve the correction amount. When the input processing unit 240 receives a response indicating approval of the correction amount, it transmits to the verification unit 220, either via the learning model unit 200 or directly, that the correction amount has been approved, and the verification unit 220 outputs a control command D22 reflecting the correction amount to the learning model unit 200. Furthermore, the proposal unit 250 may receive a response indicating approval of the correction amount and transmit the response to the confirmation unit 220. In addition, user 1 may modify the correction amount presented by the proposal unit 250, and if modified, the modified correction amount will be transmitted to the confirmation unit 220.

[0433] Furthermore, the confirmation unit 220 may output an intermediate language to the proposal unit 250 for display and request confirmation from the user 1. For example, if the control command D22 output by the learning model unit 200 is robot language, machine language, ladder language, binary data, etc., the intermediate language may be natural language that specifically describes the operation of the target device 2, or a programming language that is easy for the user 1 to understand, such as Python or C language, or something else. The confirmation unit 220 converts the control command D22 into an intermediate language and displays the code written in the intermediate language on the proposal unit 250. The user 1's response regarding the content proposed by the proposal unit 250 is received by the input processing unit 240 or the proposal unit 250 and transmitted to the confirmation unit 220, as described above. If the user 1 approves, the confirmation unit 220 notifies the learning model unit 200 that the control command D22 has been approved. Alternatively, the verification unit 220 may output the approved control command D22 to the learning model unit 200. If the code written in the intermediate language by user 1 is modified, the verification unit 220 modifies the control command D22 based on the modification and outputs the modified control command D22 to the learning model unit 200. When the learning model unit 200 is notified by the verification unit 220 that it has been approved, it outputs the control command D22 to the target device 2. Also, when the learning model unit 200 receives the control command D22 (including the modified control command D22) from the verification unit 220, it outputs the received control command D22 to the target device 2.

[0434] The learning model unit 200 may also have a function for generating an intermediate language. For example, the learning model unit 200 may operate using both a first learning model that outputs an intermediate command, which is an intermediate language command, from the model input data D21a, and a second learning model that generates a control command D22 from the intermediate language. In this case, the learning model unit 200 inputs the model input data D21a into the first learning model, outputs the obtained intermediate command to the confirmation unit 220, and outputs the control command D22 to the target device 2 by inputting the intermediate language approved by user 1 into the second learning model. Alternatively, the learning model unit 200 may output the intermediate language to the input processing unit 240, and the input processing unit 240 may present the intermediate language to user 1. When the input processing unit 240 receives input from user 1 indicating approval of the intermediate language, it may notify the learning model unit 200 of this, and upon receiving the notification of approval, the learning model unit 200 may generate the control command D22 using the intermediate language.

[0435] Alternatively, the confirmation unit 220 may generate an audio message indicating an action corresponding to the control command D22, or an audio message indicating an action corresponding to the simulation result, and output it to the proposal unit 250, causing the proposal unit 250 to present the audio message to the user 1. The proposal unit 250 may also generate the audio message. The user 1's response to the content proposed by the proposal unit 250 is received by the input processing unit 240 or the proposal unit 250 and transmitted to the confirmation unit 220, as described above.

[0436] Furthermore, the confirmation unit 220 may have the proposal unit 250 present the work content step by step. For example, the confirmation unit 220 may use one instruction unit of the control command D22, or other defined units of operation, as a step, and output information indicating the operation corresponding to the control command D22, or simulation results, to the proposal unit 250 for each step. The information indicating the operation may be the control command D22 itself, or it may be written in the intermediate language described above, or it may be video or audio, or it may be text data. User 1's response to the content proposed by the proposal unit 250 is received by the input processing unit 240 or the proposal unit 250 for each step and transmitted to the confirmation unit 220, as described above.

[0437] Alternatively, the verification unit 220 may generate a control command to run the target device 2 without a target workpiece based on the control command D22 output by the learning model unit 200, and output this control command to the learning model unit 200, thereby running the target device 2 without a target workpiece. Examples of run-within the target device 2 include, but are not limited to, operating the target device 2 without a target workpiece, or operating the target device 2 using a dummy workpiece instead of a target workpiece. The verification unit 220 may also obtain an operation result from the target device 2 indicating the state of the target device 2 during run-within the target device 2, and output the obtained operation result to the proposal unit 250, causing the proposal unit 250 to present the operation result to the user 1. This allows the user 1 to evaluate the operation of the target device 2, and if there are no problems, to input an indication that the operation is approved. If corrections are necessary, the user 1 inputs corrected input information D21. The input processing unit 240 receives the corrected input information D21, and performs the same operation as described above, and runs the device without a target workpiece again. Alternatively, the user 1 may evaluate the operation of the target device 2 by directly checking the target device 2 during run-within the target device 2.

[0438] Furthermore, the verification unit 220 may calculate a reliability index or information equivalent to a reliability index and have the proposal unit 250 present the calculated reliability index or information equivalent to a reliability index. Details of the reliability index will be described later. User 1 refers to the reliability index or information equivalent to a reliability index presented to the proposal unit 250, and if there are no problems, approves the execution of the operation, or if corrections are necessary, inputs the corrected input information D21.

[0439] Alternatively, the verification unit 220 may perform a general language-level error check on the control command D22 output from the learning model unit 200 based on the language of the control command D22, and if an error is found, it may automatically correct it and output the correction result to the learning model unit 200. Or, if an error is found, the verification unit 220 may cause the proposal unit 250 to present the error, and the input processing unit 240 or the proposal unit 250 may receive input of the error correction result from user 1 and transmit the correction result to the verification unit 220. Or, the verification unit 220 may cause the proposal unit 250 to present a proposed error correction and obtain approval from user 1.

[0440] Furthermore, the verification unit 220 may cause the proposal unit 250 to present at least one of the above-mentioned pieces of information, such as simulation results, correction amounts, and reliability indicators, so that User 1 can perform an evaluation based on the presented information, modify the input information D21 based on the evaluation result, and input the modified input information D21. The input processing unit 240 receives the modified input information D21 and performs the same operation as described above, and again performs a verification corresponding to the modified input information D21, and the proposal unit 250 similarly presents the information. User 1 checks the presented information again, performs an evaluation, and if correction is necessary, modifies the input information D21 and inputs the modified input information D21. These processes may be repeated until User 1 no longer makes corrections, that is, until User 1 approves. Furthermore, the verification unit 220 may calculate an evaluation function using at least one of the above-mentioned pieces of information and provide feedback based on the evaluation function. The evaluation function is, for example, at least one of the following: the operating time of the target device 2, the power consumption, the travel distance of the tip of the target device 2, the number of commands used, etc., but is not limited to these.

[0441] Furthermore, a function for generating models may be added to the control system 2000a. Figure 34 is a configuration diagram showing an example of a control system 2000b according to Embodiment 6, which has a function for generating models. The control system 2000b shown in Figure 34 is the same as the control system 2000a shown in Figure 33, except that a learning model generation unit 260 for generating models is added. The learning model generation unit 260 is the same as the model generation unit 107 described in Embodiment 1. The model learning data D200 input to the model generation unit 107 is the same as the model learning data D105 described in Embodiment 1, and the model information D201 is the same as the model information D102 of Embodiment 1.

[0442] Furthermore, as described above, device information D23 may be input during learning. Figure 35 is a configuration diagram showing an example of a control system 2000c according to Embodiment 6 when device information D23 is input during learning. The control system 2000c shown in Figure 35 is the same as the control system 2000a shown in Figure 33, except that a learning model generation unit 260a for generating a model is added. The learning model generation unit 260a performs learning using model learning data D200 and device information D23 as input. In the example shown in Figure 35, device information D23 is also input to the model when using the learning model unit 200.

[0443] Furthermore, retraining may be performed using the information obtained when using the learning model unit 200. Figure 36 is a configuration diagram showing an example of the control system 2000d according to Embodiment 6 when retraining is performed. The control system 2000d shown in Figure 36 is the same as the control system 2000a shown in Figure 33, except that a learning model generation unit 260 for generating a model is added, and model learning data D200a for retraining is input to the learning model generation unit 260 from the verification unit 220. The learning model generation unit 260 performs learning using the model learning data D200 as input. Furthermore, the learning model generation unit 260 performs retraining using the model learning data D200a as input. The model learning data D200a includes the information input to the learning model unit 200 and the output of the learning model unit 200 when the result corresponding to the input is good.

[0444] Figure 37 is a configuration diagram showing an example of a control system 2000e according to Embodiment 6, which includes a learning device 2001. In the example shown in Figure 37, the system includes a learning device 2001 and a control device 2002. The control device 2002 includes a learning model unit 200, an equipment information storage unit 210, a verification unit 220, a proposal unit 250, and an input processing unit 240, similar to the example shown in Figure 34. The learning device 2001 includes a learning model generation unit 260, similar to the example shown in Figure 34. Thus, the learning device 2001 and the control device 2002 may be separate devices. The learning device 2001 may also be provided separately from the control system 2000e. Furthermore, in the examples described in Figures 34 to 36, the learning device and the control device may be separated into different configurations. In addition, the learning model unit 200 may be an independent device and may be provided within or outside the control systems 2000a to 2000e.

[0445] Figure 38 is a flowchart illustrating an example of the operation of control system 2000a according to Embodiment 6. While control system 2000a will be used as an example in the following explanation, similar operations will be performed in control systems 2000b to 2000e.

[0446] Step S210 is the same as in Embodiment 2. After step S210, the control system 2000a performs preprocessing (step S214). Specifically, the input processing unit 240 preprocesses the input information D21 and outputs the preprocessed input information D21 (model input data D21a) to the learning model unit 200.

[0447] The control system 2000a generates a control command D22 corresponding to the pre-processed input information D21 using the learning model unit 200 (step S211a). Step S211a is the same as step S211 of Embodiment 2, except that the input information D21 is the pre-processed input information D21.

[0448] Step S212 is the same as in Embodiment 2. Note that in Figure 38, an example is shown in which verification is performed by dry running, so step S212 is performed after step S211a. However, if the verification process is performed without operating the target device 2, steps S212 and S213a are not performed after step S211a, and steps S212 and S213a are performed after user 1 approves the operation after steps S215 and S216, which will be described later.

[0449] In step S213a, the control system 2000a acquires state information D25. The acquisition of state information D25 in step S213a is the same as the acquisition of state information D25 in Embodiment 2. As with Embodiment 2, the processing in step S213a is not mandatory and may be omitted as appropriate. Furthermore, if feedback information can be obtained from the target device 2, the control system 2000a may also acquire the feedback information in step S213a.

[0450] The control system 2000a performs a confirmation process (step S215). Specifically, the confirmation unit 220 performs the confirmation process. The control system 2000a performs a proposal process (step S216). Specifically, the proposal unit 250 presents information to user 1 based on the results of the confirmation process by the confirmation unit 220. After presenting the information in step S216, if user 1 gives input indicating approval of the operation, the control system 2000a confirms the control command D22 output in step S211a and outputs the control command D22 to the target device 2, thereby performing the official operation by the target device 2. Furthermore, after presenting the information in step S216, if user 1 modifies the input information D21 and the control system 2000a accepts the modified input information, the process from step S210 is repeated.

[0451] Through the above processing, when User 1 inputs instructions regarding the operation of Target Device 2 as input information D21 using natural language, images, etc., a control command D22 in a format that Target Device 2 can accept is generated, and Target Device 2 operates. This allows for efficient generation of control commands D22 even when User 1 lacks sufficient knowledge of creating control commands D22, or during trial runs, startup adjustments, robot teaching, and troubleshooting. Furthermore, in this embodiment, a verification process is performed to confirm whether an appropriate control command D22 has been generated, and if it is inappropriate, the input is repeated, thus preventing Target Device 2 from operating with an inappropriate control command D22. In addition, pre-processing by the input processing unit 240 allows for correction before input to the learning model unit 200 in cases where the input is unclear, thus enabling efficient generation of control commands D22. Note that the input processing unit 240, verification unit 220, and proposal unit 250 are not mandatory, and at least some of them may be omitted. For example, the input processing unit 240 may be omitted.

[0452] Next, the hardware configuration of the control system 2000a will be described. In this embodiment, the control system 2000a functions as the computer system when a program (computer program) describing the processing in the control system 2000a is executed on the computer system. Figure 39 is a diagram showing an example of the configuration of a computer system that realizes the control system 2000a in this embodiment. As shown in Figure 39, this computer system comprises a processor 901, a memory 902, an input unit 903, a display unit 904, and a communication unit 905, which are connected via a system bus. The processor 901 and the memory 902 constitute a processing circuit.

[0453] In Figure 39, the processor 901 is, for example, a processor such as a CPU or GPU (Graphics Processing Unit), and executes a program that describes the processing in the control system 2000a of this embodiment. The input unit 903 consists of, for example, a keyboard, mouse, microphone, etc., and is used by the user of the computer system to input various information. The memory 902 includes various types of memory such as RAM (Random Access Memory) and ROM (Read Only Memory), and storage devices such as a hard disk, and stores the program that the processor 901 should execute, necessary data obtained during the processing, etc. The memory 902 is also used as a temporary storage area for the program. The display unit 904 consists of a display, LCD (Liquid Crystal Display), etc., and displays various screens to the user of the computer system. The input unit 903 and the display unit 904 may be integrated and implemented as a touch panel. The communication unit 905 is a receiver and transmitter that perform communication processing. Note that Figure 39 is an example, and the configuration of the computer system is not limited to the example in Figure 39. For example, the computer system that implements the control system 2000a does not need to have a communication unit 905. Also, the computer system may include a speaker, which is not shown in the diagram.

[0454] Here, an example of the operation of the computer system until the program of this embodiment becomes executable will be described. In a computer system with the above configuration, for example, the program is installed from a CD-ROM or DVD-ROM set in a CD (Compact Disc)-ROM drive or DVD (Digital Versatile Disc)-ROM drive (not shown) into an auxiliary storage device which is part of memory 902. Then, when the program is executed, the program read from the auxiliary storage device of memory 902 is stored in the main memory area of ​​memory 902. In this state, the processor 901 executes processing as the control system 2000a of this embodiment according to the program stored in memory 902.

[0455] In the above explanation, a program describing the processing in the control system 2000a is provided on a CD-ROM or DVD-ROM as the recording medium. However, the explanation is not limited to this, and depending on the configuration of the computer system, the capacity of the program to be provided, a program provided via a transmission medium such as the Internet via the communication unit 905 may also be used.

[0456] The learning model unit 200 and verification unit 220 shown in Figure 33 are realized by executing a program stored in the memory 902 shown in Figure 39 by the processor 901 shown in Figure 39. The memory 902 is also used to realize the learning model unit 200 and verification unit 220. The device information storage unit 210 shown in Figure 33 is part of the memory 902 shown in Figure 39. The input processing unit 240 shown in Figure 33 is realized by the input unit 903 and processor 901 shown in Figure 39. The memory 902 may also be used to realize the input processing unit 240, or a speaker (not shown) may be used. The proposal unit 250 shown in Figure 33 is realized by the display unit 904 and processor 901 shown in Figure 39. The memory 902 may also be used to realize the proposal unit 250. In addition, the input unit 903 may be used, or a speaker (not shown) may be used to realize the proposal unit 250. Furthermore, if input from user 1 is provided via a device such as a user terminal, the communication unit 905 is used to implement the input processing unit 240. Also, if the proposal unit 250 presents information via the user terminal, the communication unit 905 is used to implement the proposal unit 250.

[0457] Control systems 2000b to 2000e are similarly implemented by the computer system illustrated in Figure 39. Each of the control systems 2000a to 2000e may be implemented by multiple computer systems. At least a portion of the control systems 2000a to 2000e may be implemented by a cloud computing system.

[0458] Next, specific examples of reliability indices calculated by the verification unit 220 will be described. The first to fourth methods described below are illustrative examples, and the reliability indices calculated by the verification unit 220 are not limited to these. Note that the prior training time in the first to fourth methods may be the training time of the model used by the learning model unit 200, or it may include the time when the learning model unit 200 uses the model, i.e., the time when inference is performed by the learning model unit 200.

[0459] First, let's explain the first method. For example, during pre-training, each time training data is input to the model, a person evaluates the result corresponding to the training data. Specifically, during evaluation before actual use, the model's output is generated under multiple conditions, and the evaluation results evaluated by the person are recorded. The evaluation result can be defined as, for example, 1 if the operation of target device 2 is successful and 0 if it fails, but the definition is not limited to this. In this case, the evaluation result may also include correction information indicating what should be corrected. The result corresponding to the training data may be the operation result of target device 2 or a simulation result. Multiple datasets are accumulated, each consisting of model input data D101 in the training data during pre-training and the (human) evaluation result corresponding to the model input data D101. Using these multiple datasets, a training model for evaluation is generated by machine learning. This machine learning can be, for example, supervised learning using neural networks or support vector machines, but is not limited to these. Furthermore, the evaluation learning model may undergo additional training not only during pre-training, but also by using a dataset consisting of model input data D21a obtained when input information D21 is input from user 1, and evaluation results for said model input data D21a, obtained when using the learning model unit 200. The evaluation results can be obtained, for example, by a person evaluating the control command D22 obtained by inputting the model input data D21a into the learning model unit 200, or the operation results of the target device 2 based on the control command D22.

[0460] The verification unit 220 obtains an evaluation result by inputting model input data D21a into the evaluation learning model. The verification unit 220 can use this evaluation result as a reliability index. The better the evaluation result, the higher the reliability. The verification unit 220 outputs the evaluation result to the proposal unit 250, which presents the evaluation result to user 1 as a reliability index. The proposal unit 250 may also present points that need correction based on the correction information in the evaluation result and prompt the user to correct the points that need correction, or it may prompt the user to correct the points that need correction by voice. The input processing unit 240 may also function as the verification unit 220, and the input processing unit 240 may obtain an evaluation result by inputting input information D21 into the evaluation learning model and perform re-questioning according to the evaluation result. The evaluation model generation unit that generates the evaluation learning model may be provided in the verification unit 220, or it may be provided separately in the control system 2000a, or it may be provided in a learning device outside the control system 2000a.

[0461] Next, the second method will be explained. The verification unit 220 uses a learner that clusters the control commands D22 (combinations of trajectories and operation commands of the target device 2), which are the output of the learning model unit 200. The learner learns thresholds and other parameters for clustering using the control commands D22 obtained during pre-training. When the learning model unit 200 is used, the verification unit 220 performs clustering by inputting the control commands D22 output from the learning model unit 200 into the learner. Examples of clustering methods include the k-means method and the x-means method, but are not limited to these. When the learning model unit 200 is used, the verification unit 220 obtains the clustering result by inputting the control commands D22 output from the learning model unit 200 into the learner, outputs the clustering result to the proposal unit 250, and the proposal unit 250 presents the clustering result to the user 1 as information equivalent to a reliability index. The user 1 refers to the clustering result and checks whether it has been classified into a normal cluster or not. If it has been classified into a normal cluster, it will be considered to have high reliability. A normal cluster may be determined by a human based on the clustering results, or a normal control command D22 may be input, and the cluster resulting from that control command D22 may be designated as a normal cluster.

[0462] Next, the third method will be explained. During pre-training, each time training data is input to the model, a person evaluates the results corresponding to the training data, and the features of the inputs with high evaluation results (model input data D101 in the training data) are trained. Specifically, during evaluation before actual use, the model outputs are generated under multiple conditions, the generated outputs are evaluated by a person, and input data with high (good) evaluation results are extracted. For example, the features extracted from the input data with high evaluation results are accumulated, and when the training model unit 200 is used, the verification unit 220 extracts features from the model input data D21a input to the training model unit 200 and compares the extracted features with the accumulated features. The verification unit 220 calculates a reliability index according to the closeness between the extracted features and the accumulated features. For example, the verification unit 220 may calculate the distance between the extracted features and the accumulated features, and determine a reliability index such that reliability is higher when the distance is close. The verification unit 220 outputs the reliability index to the proposal unit 250, and the proposal unit 250 presents the reliability index to the user 1.

[0463] Specifically, for example, the features extracted from the model input data D21a are clustered, evaluation results corresponding to each cluster are obtained, and normal clusters are determined. For example, the verification unit 220 may calculate the average value of the evaluation results corresponding to the inputs belonging to each cluster, or it may calculate the lowest value (worst value) of the evaluation results. If the evaluation results indicate that a larger value indicates a better result, the verification unit 220 may designate clusters with evaluation results above a threshold as normal clusters and store the features belonging to the normal clusters as features extracted from input data with high evaluation results. If the evaluation results indicate that a smaller value indicates a better result, the verification unit 220 designates clusters with evaluation results below a threshold as normal clusters.

[0464] Next, the fourth method will be explained. During pre-training, each time training data is input to the model, a person evaluates the result corresponding to the training data, and a learner is used that performs learning to cluster the input (model input data D101 in the training data) based on the evaluation result. For example, the learner calculates thresholds for each cluster using the input training data and calculates the evaluation result for each cluster. For example, the learner may calculate the average value of the evaluation results corresponding to the inputs belonging to each cluster, or it may calculate the lowest value (worst value) of the evaluation result. Based on the evaluation result for each cluster, the normal clusters are determined. When using the training model unit 200, the verification unit 220 inputs the model input data D21a to the learner and performs clustering, and if the model input data D21a is classified into a normal cluster, it may be considered to have high reliability. The verification unit 220 outputs the result of whether or not it was classified into a normal cluster to the proposal unit 250, and the proposal unit 250 presents the result to the user 1 as information equivalent to a reliability index.

[0465] Regarding the determination of features in the third method, for example, the methods of Proposal 1 or Proposal 2 below can be used, but are not limited to these. Since using the input information itself, such as the model input data D21a, as features generally results in a large amount of information, the computational load can be reduced by using features obtained by reducing the dimensionality of the input information, for example, as in Proposal 1 and Proposal 2 below. Alternatively, the features may be calculated using the method of Proposal 3 below.

[0466] Proposal 1 is a method for determining features using an autoencoder. For example, during pre-training, the autoencoder learns features using the input data for training the model. Figure 40 is a diagram showing an overview of the autoencoder in this embodiment. The autoencoder is made to perform training using the input information to be input to the model as the input to the encoder and the output of the decoder. The output of the hidden layer of the autoencoder is then determined as the features. During pre-training, the input information to the model is input to the autoencoder and the output of the hidden layer is obtained as features, and these features are used to find and store features belonging to normal clusters. When the learning model unit 200 is used, the verification unit 220 inputs the model input data D21a to be input to the learning model unit 200 to the autoencoder and obtains the output of the hidden layer as features. The verification unit 220 calculates a reliability index by comparing the features obtained as described above with the stored features.

[0467] Proposal 2 is a method in which, during pre-training, the input data for the model is used to reduce the dimensionality of the input information, and this information is used as features. Examples of dimensionality reduction methods include PCA (Principal Component Analysis) and Kernel PCA, but are not limited to these. When the learning model unit 200 is used, the verification unit 220 calculates features by reducing the dimensionality of the model input data D21a input to the learning model unit 200 using the dimensionality reduction method.

[0468] Proposal 3 is a method that uses the frequency of command usage in the control command D22 output from the model as a feature. For example, the verification unit 220 calculates the type of operation command output from the model as control command D22 and the number of times that command has been used as features.

[0469] In the first method described above, supervised learning may be performed using the features extracted from the input information (input data) to the model and the evaluation results as a dataset. The first to fourth methods described above are examples, and the reliability index may be calculated by machine learning for evaluation as described above. The method for generating the reliability index using machine learning for evaluation is not limited to the above examples. For example, the reliability index may be calculated by supervised learning using features extracted from the input data to the learning model unit 200 or features extracted from the output of the learning model unit 200 and the evaluation results, which are the correct data. The features may be the input data itself or the output data itself. The reliability index may also be calculated by unsupervised clustering or supervised clustering using features extracted from the input data to the learning model unit 200 or features extracted from the output of the learning model unit 200. In this case as well, the features may be the input data itself or the output data itself. In summary, for example, a reliability index can be calculated by training a clustering or supervised regression model using either the input data to the model or features extracted from the output data from the model. The features may be the input data itself, the output data itself, calculated by any of the above-mentioned methods 1, 2, and 3, or calculated by other methods.

[0470] Furthermore, since the model initially used by the learning model unit 200 may be a general-purpose model, additional learning may be performed based on the evaluation results by the verification unit 220 so that appropriate control commands D22 can be generated according to the target object such as the workpiece. This enables updating the model to one that is appropriate for the target object and target equipment 2. Alternatively, the model may be distilled before additional learning is performed.

[0471] Furthermore, as described above, the learning model unit 200 may use two learning models: one that outputs model input data D21a or intermediate results described in an intermediate language, and another that outputs control commands D22 from the intermediate results. Learning may also be performed using a program database. In addition, the control systems 2000a to 2000e may be equipped with a module that supports prompt engineering. Furthermore, the optimal prompt may be determined in advance by conducting trials and evaluations, such as through reinforcement learning.

[0472] Furthermore, the embodiments and variations described above are not limited to the examples described above and can be modified as appropriate within the scope of the disclosure.

[0473] Furthermore, the control systems and control methods relating to this disclosure include the control systems and control methods described in the following appendix.

[0474] (Note 1) A control system for supporting work performed by a person or object using equipment, An input interface that accepts input of first information indicating the situation or requirements in the work environment, which is the environment in which the aforementioned work is performed, A model processing unit provided to be accessible to a predetermined learning model, The system includes an output interface that outputs second information to support the task based on the output from the learning model, The model processing unit inputs model input data based on the first information into the learning model, and receives model output data corresponding to the model input data from the learning model. The aforementioned model output data includes information used in the aforementioned operation, The output interface outputs the second information based on the model output data. A control system characterized by the following: (Note 2) The first information includes information indicating the control content or operation content required for the device, The model input data is data in which the control content or operation content indicated by the first information matches the input of the learning model. The model output data includes information used for controlling or operating the device, corresponding to the control content or operation content indicated in the model input data. The second information includes information used for controlling or operating the device, which is included in the model output data, described in a predetermined format that is discriminable at the output destination of the output interface. The control system described in Appendix 1. (Note 3) The output destination of the output interface is the device or an interface that requests control from the device. As a result of the output of the second information to the aforementioned device or the interface that requests control from the aforementioned device, the aforementioned device is controlled. The control system described in Appendix 2. (Note 4) The aforementioned device further comprises an executable code generation unit that generates and outputs executable code, which is executable code. The output destination of the output interface is the executable code generation unit, As a result of the output of the second information to the execution code generation unit, the device is controlled by the generated execution code. The control system described in Appendix 2. (Note 5) The output destination of the aforementioned output interface is a terminal operated by the user. As a result of the output of the second information to the terminal, the device is controlled. The control system described in Appendix 2. (Note 6) The first information includes information indicating the conditions in the work environment, The aforementioned model input data is data in which the conditions of the work environment indicated by the first information are shown in a format that matches the input of the learning model. The model output data includes the results of an analysis of the working environment conditions indicated by the model input data and / or information on methods for improving those conditions. The second information includes information described in a predetermined format that is discernible at the output destination of the output interface, which includes the results of an analysis of the conditions in the work environment and / or information on methods for improving said conditions. The control system described in Appendix 1. (Note 7) The model processing unit is provided with access to the first learning model and the second learning model, The model processing unit inputs first model input data based on the first information to the first learning model, and receives first model output data corresponding to the first model input data from the first learning model. The model processing unit inputs second model input data based on the first model output data to the second learning model, and receives second model output data corresponding to the second model input data from the second learning model. The output interface outputs the second information based on the second model output data. The control system described in Appendix 1. (Note 8) The first information includes information indicating the control content or operation content required for the device, The first model input data is data in which the control content or operation content indicated by the first information is shown in a format that matches the input of the first learning model. The first model output data includes information that more generalizes or specifies the control content or operation content shown in the first model input data. The second model input data is data in which the control content or operation content shown in the first model output data is shown in a format that matches the input of the second learning model. The second model output data includes information used for controlling or operating the device, corresponding to the control content or operation content indicated in the second model input data. The second information includes information used for controlling or operating the device, which is included in the second model output data, described in a predetermined format that is discriminable at the output destination of the output interface. The control system described in Appendix 7. (Note 9) The first information includes information indicating the conditions in the work environment, The first model input data is data in which the conditions of the work environment indicated by the first information are shown in a format that matches the input of the first learning model. The first model output data includes the results of the analysis of the working environment conditions shown in the model input data, The second model input data is data in which the analysis results of the working environment conditions shown in the first model output data are presented in a format that matches the input of the second learning model. The second model output data includes information on methods for improving the conditions in the work environment, corresponding to the analysis results of the work environment shown in the second model input data. The second information includes information, which is included in the second model output data, that describes at least information on how to improve the conditions in the working environment in a predetermined format that is discernible at the output destination of the output interface. The control system described in Appendix 7. (Note 10) The aforementioned operation is a response by a person or object using equipment, The first information includes information indicating the content of the reaction request, which is the content of the reaction required in the work environment. The model input data is data in which the response request content indicated by the first information matches the input of the learning model. The model output data includes information used for the response that corresponds to the response request content indicated in the model input data, The second information includes information used in the response included in the model output data, described in a predetermined format that is discriminable at the output destination of the output interface. The control system described in Appendix 1. (Note 11) The output destination of the output interface is a screen operation interface that requests control from the device via an operation screen. The model output data includes an operation screen for actually performing an operation on the device corresponding to the control content or operation content indicated in the model input data, and includes information on the operation screen described in a predetermined format that can be identified at the output destination of the output interface. The control system described in Appendix 3. (Note 12) The input interface receives input of the first information indicating the requirements for the work environment from multiple users. The model processing unit inputs the model input data, which includes the first information input from the multiple users, into the learning model, and receives the model output data corresponding to the model input data from the learning model. A control system as described in any of the appendices 1 through 11. (Note 13) The learning models include a language learning model that takes natural language as input and produces an output result, an image learning model that takes an image as input and produces an output result, and a multimodal model that takes natural language and an image as input and produces an output result. A control system as described in any of the appendices 1 to 12. (Note 14) The model processing unit is provided with access to the first learning model and the second learning model, One of the first and second learning models is a local learning model in which the referenced database is limited to its internal information. The other of the first and second learning models is a global learning model in which the referenced database is not limited to its internal information. A control system as described in any of the appendices 1 through 13. (Note 15) The model processing unit is provided with access to the first learning model and the second learning model, One of the first learning model and the second learning model is a learning model that can access information specifically defined in the work environment, The other of the first and second learning models is a learning model in which information specifically defined in the work environment is not accessible. A control system as described in any of the appendices 1 through 13. (Note 16) The system includes an output verification unit that performs a simulation to mimic the control and state of the device based on the model output data output from the learning model. A control system as described in any of the appendices 1 through 15. (Note 17) Based on the information collected from the output destination of the output interface, the learning model is further trained, or the correctness of the output information is determined, and the flow of the output information is controlled. A control system as described in any of the appendices 1 through 16. (Note 18) The system includes an input processing unit that queries the input source if the first information contains unclear or uncertain information. A control system as described in any of the appendices 1 through 17. (Note 19) The aforementioned query includes information indicating modifications, additions, or deletions to the input and output data of the learning model. A control system as described in any of the appendices 1 through 18. (Note 20) The first piece of information is time-series data that shows the situation or requirements in the work environment, which is the environment in which the work is performed, along with time information. A control system as described in any of the appendices 1 through 19. (Note 21) The execution environment for the learning model includes a model information storage unit that stores model information, and a model control unit that receives the model input data and outputs the model output data based on the model input data and the information stored in the model information storage unit. A control system as described in any of the appendices 1 through 19. (Note 22) A control method for supporting work performed by a person or object using equipment, The input interface accepts input of first information indicating the situation or requirements in the work environment, which is the environment in which the aforementioned work is performed. A model processing unit, which is provided to be accessible to a predetermined learning model, inputs model input data based on the first information into the learning model, and receives model output data from the learning model that corresponds to the model input data and includes information used in the operation. The output interface outputs second information based on the model output data, which is used to support the operation, based on the output from the learning model. A control method characterized by the following: [Industrial applicability]

[0475] The control system described herein is suitably applicable as part of a work support system that assists in work performed by people or objects. Furthermore, the control system described herein is suitably applicable as a control system for controlling equipment when such equipment is used to perform some kind of control or work. Here, the control system is also suitably applicable as a control system for controlling FA equipment, a control system in homes or buildings, and a control system for controlling information processing devices such as server devices that perform information processing on a network. [Explanation of Symbols]

[0476] 1000, 1000a, 1000b, 1000c, 2000, 2000a, 2000b, 2000c, 2000d, 2000e, 3000, 3000a, 3000b, 3000c, 3000d, 3000e, 4000, 4000a, 5000, 5000a, 5000b, 5000c, 5000d Control System 100,200,300,300a,300b,400,400a,400b,500,500a,500b Learning Model Section 10,20 Information Processing Devices 11 Model Information Storage Unit 12 Reference information storage section 101 Model Control Unit 102 Input section 103 Output section 105 Pre-processing section 106 Post-processing unit 107 Model Generation Unit 104,104a Control Unit 201 Input Processing Unit 202 Output Verification Section 203 Correction confirmation section 1 user 1a Input source 2. Target devices 2a Output destination 3. User Interface 4 controllers 31 Input determination unit 41-1,41-2 Analysis section 42 Switching section 43 Output selector switch 5 sensors 6 Model Interfaces 7 Display 8 Operators 110,210,310,410 Device information storage section 120 Executable Code Generation Unit 130, 230, 330, 430, 530 Status acquisition unit 220 Verification section 240 Input Processing Unit 250 Proposal Department 260,260a Learning Model Generation Unit 311 Input Interface 312 Output Interface 313 Environmental information storage unit 511 Database Search Section 512 Control Generation Unit 513v Voice Recognition Unit 513i Image Analysis Unit 514i Image generation section 514v Speech Synthesis Unit 514p Program generation unit 515 Correct / Incorrect Judgment Unit 516 Emotion Judgment Department 517 Evaluation Acquisition Department 518 Registration Determination Unit 519 Additional Learning Section 520 Control Determination Unit 531 Call Confirmation Unit 532 Output Selection Section 533 Output switching section 2001 Learning device 2002 Control Unit D101, D21a Model Input Data D102, D201 Model Information D103 Model Output Data D104 Model Reference Information D105, D200, D200a Model training data D11, D21, D31, D41, D51, D51v, D51i Input Information D12 Control Description D22, D34 Control Commands D29 Proposal Information D42a,D42b analysis results D52, D52v, D52i, D52p, D52a, D52b, D52c Response Information D32,D32a,D32b operation command D320 Operation Information D13, D23, D33, D43 Equipment Information D33a environmental information D14 Execution Code D44a, D44b Result Information D15, D25, D35, D45 Status Information D16, D26, D36, D46 Feedback Information Inquiries regarding D17, D27, D37, D47, and D57. D18, D28, D38, D48 Supplementary Information D59 Evaluation Information

Claims

1. A control system for supporting work using equipment, A learning model unit that outputs second information, which is information for operating the device, based on output data obtained by inputting input data based on first information indicating requirements for the operation of the device into a trained learning model, Equipped with, The learning model unit determines, based on the user's credentials, whether the user is permitted to perform the task corresponding to the second information, outputs the second information based on the output data if the user is permitted to perform the task corresponding to the second information, and does not output the second information if the user is not permitted to perform the task corresponding to the second information. The learning model unit outputs the second information using the output data and the device information which is information about the device. The equipment information includes at least one of the computer-aided design information of the equipment and the computer-aided design information of the object that is the subject of the work. A control system characterized by the following:

2. A confirmation unit performs confirmation processing to confirm the operation of the device corresponding to the first information. The control system according to claim 1, characterized by comprising:

3. Based on the aforementioned verification process, a proposal unit presents proposal information, which is information for the user to determine whether or not the first information needs to be modified. The control system according to claim 2, characterized by comprising:

4. The verification unit performs a simulation of the operation of the device using the second information, The control system according to claim 3, characterized in that the proposed unit presents the results of the simulation.

5. The control system according to claim 4, characterized in that the proposed unit presents the results of the simulation by displaying the simulation results in augmented reality.

6. The control system according to claim 2, characterized in that the verification unit outputs a reliability index corresponding to the first information.

7. The control system according to claim 2, characterized in that the verification unit outputs a reliability index corresponding to the second information.

8. The control system according to claim 6, characterized in that the verification unit outputs the reliability index by inputting the features extracted from the input data based on the first information into an evaluation learning model generated by supervised machine learning using features extracted from the input data to the learning model and a reliability index of the operation of the device which is the correct answer data corresponding to the input data.

9. The control system according to claim 7, characterized in that the verification unit outputs the reliability index by inputting the features extracted from the second information corresponding to the first information into an evaluation learning model generated by supervised machine learning using features extracted from second information based on output data from the learning model and a reliability index of the operation of the device which is the correct answer data corresponding to the second information.

10. The control system according to claim 6, characterized in that the verification unit outputs the reliability index based on the clustering result obtained by inputting the features extracted from the input data based on the first information into a trained learner that clusters the features extracted from the input data to the learning model.

11. The control system according to claim 7, characterized in that the verification unit outputs the reliability index based on the clustering result obtained by inputting the features extracted from the output data based on the first information to a trained learner that clusters the features extracted from the second information based on the output data from the learning model.

12. The control system according to any one of 8 to 10, characterized in that the aforementioned feature quantities are extracted using an autoencoder.

13. The control system according to any one of 8 to 10, characterized in that the aforementioned feature quantities are extracted using a dimensionality reduction means.

14. The second piece of information includes control commands to the device, The control system according to any one of 8 to 10, characterized in that the feature quantity includes the frequency of use of the command used as the control command in the second information.

15. The control system according to any one of claims 1 to 11, characterized in that the first information is information representing the operation of the device or the operation in natural language or images.

16. The control system according to any one of claims 1 to 11, characterized in that the second information is a control command for controlling the operation of the device and is in a format that the device can interpret.

17. The control system according to any one of claims 1 to 11, characterized in that the learning model includes a first learning model that takes the first information as input and outputs an intermediate result, and a second learning model that takes the intermediate result as input and outputs the second information.

18. The control system according to any one of claims 1 to 11, characterized in that the aforementioned equipment information includes the user's proficiency level.

19. The control system according to any one of claims 1 to 11, characterized in that the equipment includes industrial equipment.

20. A learning model generation unit that generates the aforementioned learning model, Equipped with, The control system according to claim 2, characterized in that the learning model generation unit performs retraining based on the model learning data generated by the verification unit by utilizing the learning model.

21. The control system according to any one of claims 1 to 11, further comprising an input processing unit that queries the input source when the first information contains unclear or uncertain information.

22. A control method for supporting work using equipment, The learning model unit determines, based on the user's credentials, whether the user is authorized to perform the task in accordance with first information indicating a request regarding the operation of the device. If the user is authorized to perform the task, it inputs input data based on the first information into the trained learning model and outputs second information, which is information for operating the device, based on the output data obtained. If the user is not authorized to perform the task, it does not output the second information. Using the output data and the device information which is information about the device, the second information is output. The equipment information includes at least one of the computer-aided design information of the equipment and the computer-aided design information of the object that is the subject of the work. A control method characterized by the following:

Images