Control system and control method

Abstract

This control system is provided with a model interface (6) that controls a target device (2) according to an improvement method generated by a learning model unit (400), which generates an improvement method when the status of work to be monitored is abnormal.

Description

[Technical Field]

[0001] The present disclosure relates to a control system and a control method. [Background technology]

[0002] In recent years, the use of AI (artificial intelligence) has been increasing. In particular, generative artificial intelligence, which is capable of generating a wide variety of content, is becoming more widespread, and it is expected that the applications of AI will expand. AI will not only be used in the home, but also in various places and situations, such as in buildings, factories, stations, schools, hospitals, and commercial facilities, as well as outdoors, such as on roads, outdoor facilities, and in the sky or at sea.

[0003] For example, Patent Document 1 describes a machining program generation device that generates a program for controlling a machine by utilizing a large-scale language model. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-060806 Summary of the Invention [Problem to be solved by the invention]

[0005] To support work by people or objects, we consider using a learning model to handle some or all of the tasks involved in the work, not just the generation of control programs for devices. Note that "work by people or objects" includes not only work in the real world performed by people or machines, but also work in the data space, such as information processing performed by processors such as CPUs (central processing units).

[0006] Examples of work with objects include the following: -Work using various devices such as robots, machines, equipment, and sensors Work using various types of mobility, including cars, trains, buses, aircraft, and ships Such operations may include, for example, operations referred to as controlling, processing, machining, instructing, calculating, inputting, outputting, displaying, communicating, testing, manufacturing, converting, generating, measuring, irradiating, emitting, inhaling, radiating heat, heating, cooling, recording, reading, shaping, driving, moving, transporting, flying, investigating, monitoring, measuring, extracting, and the like.

[0007] Examples of work performed by humans include the following: Work done by a person on another person or other living creature Work performed by people on various equipment Such tasks may include, for example, tasks known as conversation, listening, checking, operating, monitoring, instructing, mediating, and interpreting.

[0008] Note that the above examples are merely examples, and the tasks that can be supported by the present disclosure are not limited to these.

[0009] When using a learning model to have an information processing device execute some or all of the tasks involved in work by a person or an object, the validity of the model's output may become an issue, as may the validity of the model's input, which affects the model's output.

[0010] Furthermore, depending on the target device, appropriate control may not be possible without understanding the current situation. In such cases, how to perform situation recognition can be an issue. In such cases, it may be necessary to recognize a continuous situation that includes not only the current situation but also past situations. For example, when determining the next control based on the content of past control, the accuracy of situation recognition may be an issue in order to ensure the continuity of control.

[0011] Furthermore, in cases where immediate control of equipment is required, the response time between giving instructions to the learning model and obtaining the results can be an issue.

[0012] Furthermore, the maintainability of the model may be an issue, as the model needs to be retrained every time a device is changed or added.

[0013] As such, the use of learning models still involves various problems. Depending on the magnitude of the problem, even if we try to use learning models to improve work efficiency or performance, it may actually decrease work efficiency or performance.

[0014] These problems when using learning models will become more pronounced, particularly as the task to be supported becomes more complex and the task becomes more advanced.

[0015] Therefore, the present disclosure aims to use a learning model to further improve the efficiency or performance of work by people or objects. [Means for solving the problem]

[0016] The control system according to the present disclosure includes a model interface that controls equipment in accordance with an improvement method generated by a learning model unit that generates an improvement method when the status of a work to be monitored is abnormal, and the learning model unit has a first learning model unit that interprets the status of the work based on first data that indicates the status of the work and outputs second data that indicates the interpretation result, and a second learning model unit that outputs third data that indicates an improvement method for the status of the work based on the second data. The second data includes information that expresses, in an attribute or a predetermined syntax format, at least one of the instantiation, subdivision, and specific point extraction for the work situation indicated by the first data. It is characterized by:

[0017] In the control method according to the present disclosure, a model interface controls equipment in accordance with an improvement method generated by a learning model unit that generates an improvement method when the status of a work to be monitored is abnormal, and in the learning model unit, a first learning model unit interprets the status of the work based on first data indicating the status of the work and outputs second data indicating the interpretation result, and a second learning model unit outputs third data indicating an improvement method for the status of the work based on the second data. The second data includes information that expresses, in an attribute or a predetermined syntax format, at least one of the instantiation, subdivision, and specific point extraction for the work situation indicated by the first data. It is characterized by: [Effects of the Invention]

[0018] According to the present disclosure, learning models can be used to further improve the efficiency or performance of work by people or objects. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a configuration diagram illustrating an example of a control system according to a first embodiment. [Figure 2] FIG. 2 is an explanatory diagram illustrating an example of the configuration of a learning model unit. [Figure 3] FIG. 10 is an explanatory diagram showing another example of the configuration of the learning model unit. [Figure 4] FIG. 10 is an explanatory diagram showing another example of the configuration of the learning model unit. [Figure 5] FIG. 1 is a configuration diagram illustrating an example of an information processing device that is the operating environment of a control unit including a learning model unit. [Figure 6] FIG. 10 is an explanatory diagram illustrating an example of model learning in a model generation unit. [Figure 7] 4 is a flowchart illustrating an example of the operation of the control system according to the first embodiment. [Figure 8] FIG. 4 is a configuration diagram showing another example of the control system according to the first embodiment. [Figure 9] FIG. 4 is a configuration diagram showing another example of the control system according to the first embodiment. [Figure 10] FIG. 4 is a configuration diagram showing another example of the control system according to the first embodiment. [Figure 11]FIG. 10 is a configuration diagram illustrating an example of a control system according to a second embodiment. [Figure 12] FIG. 10 is a configuration diagram showing another example of the control system according to the second embodiment. [Figure 13] 10 is a flowchart illustrating an example of the operation of the control system according to the second embodiment. [Figure 14] FIG. 10 is a configuration diagram illustrating an example of a control system according to a third embodiment. [Figure 15] 11 is a flowchart illustrating an example of the operation of the control system according to the third embodiment. [Figure 16] FIG. 10 is a configuration diagram showing another example of the control system according to the third embodiment. [Figure 17] 13 is a flowchart showing an example of operation of a modified example according to the third embodiment. [Figure 18] FIG. 10 is a configuration diagram showing another example of the control system according to the third embodiment. [Figure 19] FIG. 10 is a configuration diagram showing another example of the control system according to the third embodiment. [Figure 20] FIG. 10 is a configuration diagram showing another example of the control system according to the third embodiment. [Figure 21] FIG. 10 is a configuration diagram showing another example of the control system according to the third embodiment. [Figure 22] 13 is a flowchart showing an example of operation of a modified example according to the third embodiment. [Figure 23] FIG. 10 is a configuration diagram illustrating an example of a control system according to a fourth embodiment. [Figure 24] 10 is a flowchart showing an example of the operation of the control system according to the fourth embodiment. [Figure 25] FIG. 10 is a configuration diagram showing another example of the control system according to the fourth embodiment. [Figure 26] 13 is a flowchart showing an example of operation of a modified example according to the fourth embodiment. [Figure 27] FIG. 10 is a configuration diagram illustrating an example of a control system according to a fifth embodiment. [Figure 28] 13 is a flowchart showing an example of the operation of the control system according to the fifth embodiment. [Figure 29] FIG. 13 is a configuration diagram showing another example of the control system according to the fifth embodiment. [Figure 30] FIG. 13 is a configuration diagram showing another example of the control system according to the fifth embodiment. [Figure 31] FIG. 13 is a configuration diagram showing another example of the control system according to the fifth embodiment. [Figure 32] FIG. 13 is a configuration diagram showing another example of the control system according to the fifth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0020] In order to explain the present disclosure in more detail, embodiments of the present disclosure will be described below with reference to the accompanying drawings. In the following, identical elements are designated by the same reference numerals and will not be described again.

[0021] Embodiment 1 In this embodiment, an example will be described in which a learning model is used to support the work involved in generating code for the target device 2. The term "learning model" used here is not limited to a model to be learned, but also includes a model that has already been trained. This also applies to the other embodiments. Furthermore, the "target device" may also be simply called the "device."

[0022] Fig. 1 is a configuration diagram illustrating an example of a control system 1000 according to the first embodiment. The control system 1000 illustrated in Fig. 1 is a control system for controlling equipment using a learning model, and includes a learning model unit 100, an equipment information storage unit 110 (referred to as equipment information DB in the figure), and an execution code generation unit 120.

[0023] 1 shows a user 1 and a target device 2, but these may also be included in the control system 1000. In that case, "user 1" may be read as "user terminal 1." This also applies to the other embodiments.

[0024] When input information D11 is input, the learning model unit 100 outputs a control description D12. When input information D11 is input, the learning model unit 100 outputs a control description D12 based on model information D102, which will be described later.

[0025] In this embodiment, the learning model unit 100 is a model and its operating environment configured to output a control description D12 corresponding to input information D11 when input information D11 is received. Alternatively, the learning model unit 100 may be a model and its operating environment configured to generate and output a control description D12 when input information D11 is received, based on the input information D11, device information D13, and / or other information that can be referenced by the learning model unit 100 (such as model reference information D104 described below).

[0026] In this embodiment, the input information D11 includes information indicating control content requested of the target device 2. The input information D11 may be, for example, text, images, audio, or a combination thereof indicating the control content for the target device 2. The input information D11 may be, for example, text, images, audio, or a combination thereof indicating multiple control content for the target device 2. The input information D11 may also include information indicating control content that is performed consecutively over time. In this case, the input information D11 may be time-series data with a predetermined data structure including text, images, audio, or a combination thereof indicating the control content as described above. It is assumed that the way the control content is represented matches the input format of the model used by the learning model unit 100. However, this does not apply if error processing, correction processing, or conversion processing is included in the upstream stage of the learning model unit 100.

[0027] An example of a method for indicating control content in the input information D11 is to identify a control to be performed on the target device 2 and then specify parameter values ​​for that control or a state after the control. In this case, the input information D11 may include, for example, information for identifying the control and information indicating parameter values ​​for that control or a state after the control. The parameter values ​​for the control may include values ​​related to the type of control (ON / OFF, etc.), direction, amount, and time. Examples of control content include "turn on function X" for a programmable logic controller (PLC), "move the tip to point A" for a robot arm, and "lower the set temperature by 1 degree" for an air conditioner. Another example of a method for indicating control content in the input information D11 is to use a document string (docstring) describing specifications such as a function, specifications, or various information such as specifications, design documents, operation commands, control codes, and source codes applied to other devices, such as other models.

[0028] In addition to the explicit method described above, the input information D11 may indicate control content by indicating the operation content when a control is performed by a certain operation. Another example is implicit indication of the control content by, for example, using the user 1's behavior associated with a specific control, the operation result of the target device 2, or a similar control command for another model. In other words, the input information D11 may include not only information directly indicating the control content for the target device 2, but also information indirectly indicating the operation content corresponding to the control content, the user 1's behavior, or an image of the target device 2. As an example, the control content related to the temperature control of an air conditioner may be indicated by the user 1's words such as "It's hot," or by the user 1's actions such as wiping sweat, rolling up sleeves, or raising their hands. In this case, the input information D11 may include information such as text, audio, or images indicating the user 1's statements, or images (videos) indicating the user 1's actions. As another example, the control content for controlling the arm of a robotic device can be indicated by the posture of the robotic device after control, information specifying the destination point of a specified part, or information indicating an imitation of the robot's movements by a person or other object (a simulator that simulates the movements of the robot, including objects on a screen) or an instruction to the robot (an instruction to an action by a gesture such as pointing).

[0029] Here, the format of the input information D11 is not particularly limited. For example, the information may be text, images, audio, data written in a predetermined design language, control descriptions (including source code and information written in a predetermined programming platform language), information written in other platform languages, control instructions (including control commands, control signals, control codes, and controller commands), or execution code. These types of information may be combined as appropriate. In this disclosure, when the term "text" is used without any particular distinction, it may include not only natural language expressed in text, but also data that can be interpreted by machines, such as data written in a predetermined design language that is indistinguishable from humans, control descriptions (including source code and information written in a predetermined programming platform language), information written in other platform languages, control instructions (including control commands, control signals, control codes, and controller commands), and execution codes, expressed in text.

[0030] The control description D12 includes information about control written in a predetermined format that can be interpreted by the subsequent execution code generation unit 120. The control description D12 is, for example, source code written in a predetermined programming language. The control description D12 may also be, for example, a group of commands written in a format (platform language) handled by a predetermined programming platform. Here, the predetermined programming platform may include a no-code programming platform and a low-code programming platform.

[0031] The device information storage unit 110 stores device information D13, which is information about the target devices 2. The device information D13 may include, for example, information indicating the functions, performance, structure, dimensions, operation, and / or control method of the target devices 2. The device information D13 may also include, for example, information about the positional relationships and / or signals between the target devices 2. That is, it may be difficult to generate an effective program using only information about individual target devices 2, and information indicating the positional relationships between multiple target devices 2 and the types of interactions between the multiple target devices 2 may be required. Note that information about the positional relationships between the target devices 2 can be automatically generated, for example, from a computer-aided design (CAD) or a simulator. Information about signals between the target devices 2 can be automatically generated, for example, from a PLC development environment. The device information D13 may also include, for example, information about a program used to control the target devices 2. The device information D13 may also be, for example, a digital version of a manual or instruction manual for the target devices 2. Here, the conversion to data includes conversion to text, image data, data by voice reading, and combinations thereof. The device information D13 is used, for example, as additional information when the learning model unit 100 outputs the control description D12.

[0032] Furthermore, the device information D13 may include information indicating the state of the target device 2. The information indicating the state of the target device 2 may include not only the current state of the target device 2 but also information indicating past states. For example, the device information D13 may include time-series data with a predetermined data structure indicating the state of the target device 2. The information indicating the state of the target device 2 may be, for example, information output from the target device 2 or information input by the user 1 or another device. The information indicating the state of the target device 2 may include various types of information output from the target device 2 (for example, error information, log information, notification information, etc.). Hereinafter, in this embodiment, information indicating the state of the target device 2 may be referred to as state information D15.

[0033] When the control description D12 is input, the execution code generation unit 120 generates and outputs execution code D14, which is code executable by the target device 2, based on the control description D12. The execution code D14 may be, for example, a group of codes written in machine language. The execution code D14 includes, for example, information used when the target device 2 actually performs control. The execution code D14 may be, for example, information related to control written in a format that can be recognized by the target device 2. The execution code generation unit 120 may be, for example, a compiler that converts the control description D12 into the execution code D14.

[0034] The execution code D14 output from the execution code generation unit 120 is input to the target device 2. As a result, the target device 2 operates in accordance with the execution code D14 output from the execution code generation unit 120. The execution code D14 may be input to the target device 2 directly from the execution code generation unit 120, or may be input indirectly via a communication network, another device (such as a server or various conversion devices), or manually.

[0035] There are no particular limitations on the target device 2. Note that it is assumed that the target device 2 is a device that can receive and actually execute the execution code D14, but this does not necessarily apply if an interface, such as a writing device, that allows the target device 2 to read the execution code is included between the target device 2 and the target device 2.

[0036] The target device 2 is, for example, a PLC, a processing machine, a robot, a radar, a sensor, a camera, a projector, or a communication device. The target device 2 may also be, for example, an air conditioner, a refrigerator, a television, a light, or a washing machine. The target device 2 may also be, for example, an elevator, a mobility device, a conveyance device, or other machine, or a control device that controls such machine. The target device 2 may also be a device operating in a power generation / transformation / storage plant, a water treatment plant, or the like, or a control device that controls other equipment. If the control description D12 is in an interpreter language and the target device 2 is a device that can receive the control description D12 and execute it as is, the execution code generation unit 120 is omitted.

[0037] Fig. 2 is an explanatory diagram showing an example of the configuration of the learning model unit 100. As shown in Fig. 2, the learning model unit 100 may include a model control unit 101 that operates on the information processing device 10, and a model information storage unit 11 (referred to as model information DB in the figure) that stores model information D102. Here, the model information storage unit 11 may be composed of multiple databases connected via a network.

[0038] The model information D102 includes model information. The model information D102 may include, as the model information, information indicating the correlation between the model input data D101 and the model output data D103, for example. The model information D102 may also include, as the model information, information indicating candidates for the model output data D103. The model information D102 may also include, as the model information, information indicating candidates for the model output data D103 and information indicating the relationships between these candidates. The model information D102 may also include model parameters, which are information that defines the behavior of the learning model, such as constraints, weighting variables, and evaluation functions.

[0039] The model may be a model developed by machine learning using, for example, supervised learning, reinforcement learning, or unsupervised learning. The model may be a model obtained by executing learning according to, for example, deep learning, genetic programming, functional logic programming, or other known algorithms or methods. The model may also be a model called a neural network (NN) model, a convolutional neural network (CNN) model, a recurrent neural network (RNN), a variational autoencoder (VAE), a generative adversarial network (GAN), a diffusion model, a transformer model, a large language model (LLM), a visual language model (VLM), a bidirectional encoder representations from transformers (BERT), a generative pre-trained transformer (GPT), or a contrastive language image pre-training (CLIP). The model may also be a rule-based model that obtains output results by referencing a predetermined table or making a decision based on predetermined conditions. The above-mentioned models are not exclusive; for example, LLM, VLM, BERT, and GPT are included in the Transformer model. Furthermore, for example, the Transformer model is included in the NN model. Furthermore, the learning algorithm and model may be a combination of multiple types. The model also includes a so-called multimodal model that is trained by combining multiple different types of data.

[0040] When the model control unit 101 receives model input data D101, it outputs model output data D103 corresponding to the model input data D101 based on the model input data D101 and model information D102. When the model control unit 101 receives the model input data D101, it outputs model output data D103 corresponding to the model input data D101, for example, by using a model indicated by the model information D102.

[0041] The model control unit 101 is realized by, for example, a CPU that operates according to a program provided in the information processing device 10. Hereinafter, the learning model unit 100 may be referred to as an artificial intelligence unit. Hereinafter, the artificial intelligence unit refers to an AI having intelligent functions such as inference and judgment and its operating environment. Therefore, the model control unit 101 may include an AI having intelligent functions such as inference and judgment and its operating environment. The model control unit 101 may be, for example, an AI having the above-mentioned learning model and its operating environment. The model control unit 101 may be one element (module) of the control unit 104 provided in the information processing device 10.

[0042] 3, the learning model unit 100 may further include a reference information storage unit 12 (referred to as a reference information DB12 in the figure) that stores model reference information D104. Here, the reference information storage unit 12 may be configured with multiple databases connected via a network. The same applies to other storage units (e.g., a device information storage unit, etc.) described later.

[0043] The model reference information D104 is information that the model control unit 101 references to output model output data. The model reference information D104 may include a history of model input data that was previously input and / or a history of model output data that was previously output. The model reference information D104 may also include information that associates feature amounts included in past inputs with feature amounts included in outputs performed for those inputs. The model reference information D104 may also include information on evaluations of results output for past inputs.

[0044] The model reference information D104 may also include information related to expressions or concepts included in the model input data D101. The model reference information D104 may include, for example, information associating a specific expression or concept that may be included in the model input data D101 with other expressions or concepts related to that expression or concept. Here, other expressions or concepts related to a certain expression or concept include expressions or concepts that more specifically embody the certain expression or concept, and other expressions or concepts that are evoked based on the certain expression or concept. The model reference information D104 may also include, for example, information associating a specific expression or concept that may be included in the model input data D101 with expressions or concepts related to that expression or concept. As an example, the model reference information D104 may include, for example, information associating a specific expression or concept that may be included in the model input data D101 with information related to that expression or concept. The model reference information D104 may also include, for example, information associating search keys and values ​​extracted from expressions or concepts that may be included in the model input data D101. The model reference information D104 may include information for so-called grounding. The model reference information D104 may also include a so-called knowledge graph that describes real-world entities and the relationships between them. In a knowledge graph, various pieces of information are systematically linked and represented in a graph structure.

[0045] The model reference information D104 may also include information for so-called attention. For example, the model reference information D104 may include information indicating the correlation between an expression or concept that may be included in the model input data D101 and another expression or concept. The model reference information D104 may also include a feature map in which key information extracted from an expression or concept that may be included in the model output data D103 linked to an expression or concept that may be included in the model input data D101 is used as a feature. The model reference information D104 may also include information that associates a query extracted from an expression or concept that may be included in the model input data D101 with key information for search corresponding to the query.

[0046] In the example shown in FIG. 3, when the model control unit 101 receives model input data D101, it outputs model output data D103 based on the model input data D101, model information D102, and model reference information D104.

[0047] The learning model unit 100 may include a search engine for searching the model reference information D104 or an interface with the search engine, instead of the reference information storage unit 12. In such a case, the search range of the search engine may be an external network or a specific network. Here, a database (e.g., a device information DB) included in the control system of the present disclosure may be used as one of the external networks or specific networks.

[0048] The term "learning model" may refer to a computer algorithm that generates some output based on learned information in response to input information, or the learned information itself. However, when referring to a "learning model" in an operating environment, it often refers to an actual program that runs such a computer algorithm and its operating environment. In this disclosure, the latter term is adopted, and a model that actually runs based on information stored in model information D102 is called a "learning model" to distinguish it from a simple algorithm or a group of learned information. The control system according to this disclosure includes a learning model unit (particularly, model control unit 101) that corresponds to such a learning model. Therefore, hereinafter, when referring to a "learning model" in the description of the control system, it refers to the learning model unit or, in particular, the model control unit 101.

[0049] 4 is an explanatory diagram showing another example of the configuration of the learning model unit 100. As shown in FIG. 4, the learning model unit 100 may include an input unit 102, an output unit 103, and a control unit 104.

[0050] The input unit 102 accepts model input data D101. The input unit 102 may accept model input data D101 input by user 1 or the like. The input unit 102 may accept model input data D101 constituting time-series data. In this case, the input unit 102 may sequentially accept the model input data D101 constituting the time-series data, or may accept model input data D101 that has been buffered to a certain extent. Furthermore, the input unit 102 may accept model input data D101 input from a plurality of input sources. In this case, the input unit 102 may accept model input data D101 to which information about the input source (e.g., a user identifier, attribute information of user 1, etc.) has been added, or the input unit 102 may determine the input source and add the information about the input source to the model input data D101 before accepting the data, or may accept the data without doing anything in particular. The input unit 102 is realized, for example, by various input devices (for example, a pointing device, a keyboard, a voice input device, an image input device, a data reading device, a data input device compatible with various communication interfaces, etc.) provided in the information processing device 10. Note that the input unit 102 may also be realized by an external device of the information processing device 10. In that case, it is sufficient for the information processing device 10 to include an interface with the input unit 102.

[0051] The output unit 103 outputs an object generated by the control unit 104. Here, the object includes model output data D103 or data generated from the model output data D103. Furthermore, if the object generated by the control unit 104 includes information for multiple output destinations, the output unit 103 may output the object to multiple output destinations. In this case, the output unit 103 may output the same data to multiple output destinations, or may output different data for each output destination. The output unit 103 is realized, for example, by various output devices included in the information processing device 10 (e.g., a display device, an audio output device, an image output device, a data writing device, a data output device compatible with various communication interfaces, etc.). Note that the output unit 103 may also be realized by an external device of the information processing device 10. In this case, the information processing device 10 only needs to include an interface with the output unit 103.

[0052] The control unit 104 operates on the information processing device 10, and includes a pre-processing unit 105 and a post-processing unit 106 in addition to the model control unit 101 described above.

[0053] The preprocessing unit 105 performs processing to increase the accuracy of the object generated by the control unit 104. The preprocessing unit 105 may, for example, add, change, or delete elements from the model input data D101, or convert (including process) the data.

[0054] For example, when the input unit 102 receives model input data D101, the preprocessing unit 105 may change elements (including addition and deletion) or convert data (including processing) for the model input data D101. Changing elements or converting data (including processing) includes not only changing the data format but also changing the expression or concept represented by the data. The data changed by the preprocessing unit 105 is input as model input data D101 to the model control unit 101 at the subsequent stage. The processing performed by the preprocessing unit 105 includes so-called prompt shaping for the model control unit 101.

[0055] The preprocessing unit 105 may, for example, perform a process of decomposing the model input data D101 into predetermined unit data. The preprocessing unit 105 may also, for example, perform a process of integrating a plurality of model input data D101. Furthermore, the preprocessing unit 105 may change elements or convert data after decomposing the model input data D101 into predetermined unit data, or may integrate a plurality of model input data D101 and then change elements or convert data.

[0056] The post-processing unit 106 corrects the object, for example, if there is a problem with the object generated by the control unit 104 (particularly the model control unit 101). The post-processing unit 106 may determine whether there is a problem with the object, for example, by using the above-mentioned knowledge graph. For example, the post-processing unit 106 may compare the similarity between the relationship indicated by the knowledge graph and the relationship between the expression or concept included in the model input data and the expression or concept included in the model output data, and / or the relationship between the expressions or concepts included in the model output data, and determine that there is a problem with the object if the relationship is more than a predetermined distance away from the relationship indicated by the knowledge graph.

[0057] Of the above-mentioned components, components other than the model control unit 101 are not essential, and it is possible to select whether or not to implement them as appropriate.

[0058] Furthermore, the model information D102 and other information used by the learning model may be prepared in advance, or may be acquired via a communication network as necessary.

[0059] 5 is a configuration diagram showing another example of an information processing device 10 as an operating environment for a control unit 104 including a learning model unit 100. The information processing device 10 shown in FIG. 5 may include a control unit 104a including the learning model unit 100 (particularly, a model control unit 101), an input processing unit 201, an output confirmation unit 202, and a correction confirmation unit 203.

[0060] The input processing unit 201 receives input information D11 from an input source 1a such as a user 1. The input processing unit 201 also outputs the received input information D11 to the learning model unit 100 as model input data D101.

[0061] In this case, the input processing unit 201 may, for example, change elements or convert data of the input information D11 and output the result as the model input data D101. The input processing unit 201 may, for example, remove noise from the input information D11. Furthermore, when the input information D11 contains qualitative information, the input processing unit 201 may convert the information into quantitative information. Furthermore, when the input information D11 contains quantitative information, the input processing unit 201 may correct the amount of quantitative information according to the device that is the target of the request for the input information D11 and its operating environment. Furthermore, the input processing unit 201 may, for example, perform so-called grounding processing, i.e., change the expression or concept indicated in the input information D11 into a more specific expression or concept.

[0062] Furthermore, the input processing unit 201 may return a query to the input source when the input information D11 contains unclear or uncertain information. The input processing unit 201 may output, as a query, a message to confirm the input content, a message proposing a correction to the input information D11, or a message requesting re-input of the input information D11 with a different state or expression. The correction to the input information D11 may be generated by the correction confirmation unit 203 described later. Hereinafter, information indicating correction, addition, or cancellation of content to the input / output data of the learning model after input / output may be referred to as supplemental information D18. The correction is an example of supplemental information D18.

[0063] The output confirmation unit 202 performs a simulation that simulates the control and state of the target device 2 based on the model output data D103 output from the learning model unit 100. The output confirmation unit 202 may perform the simulation after converting the model output data D103 into control information that matches a predetermined simulator (not shown) that can simulate the control and state of the target device 2. The output confirmation unit 202 may have the function of a simulator. When performing the simulation, the output confirmation unit 202 may use information acquired from an output destination 2a of the model output data D103. Here, the output destination 2a includes an output destination of information generated from the model output data D103. The information acquired from the output destination 2a may include, for example, state information D15 and / or feedback information D16, which will be described later.

[0064] The output confirmation unit 202 may, for example, confirm the state of the target device 2, the state of the system including the target device 2, and / or the state of the work included in the target device 2. Furthermore, before checking the operation, the output confirmation unit 202 may generate and display a human-understandable intermediate product for the model output data D103 or information generated based on the model output data D103. Examples of intermediate products include source code for a control program and an operation image of the controller of the target device 2 for an operation command to the target device 2. Furthermore, the output confirmation unit 202 may display the results of the simulation together with a reliability index for the learning model.

[0065] The following are examples of reliability indices for learning models. For example, during pre-learning, for example, a system can be configured in which human evaluations of the results of inputs to a learning model are accumulated, an evaluation network is provided that has learned the inputs and evaluation results, and when the learning model is used, the inputs to the learning model are also input to the evaluation network, and the output results are used as the reliability index.

[0066] Furthermore, for example, a learning device may be provided that clusters the output of the learning model during pre-learning, and when the learning model is used, the output of the learning model may also be input to the above-mentioned learning device, and the result of this clustering may be used as a reliability index.

[0067] Furthermore, for example, during pre-learning, a human evaluation of the results of each input to the learning model can be accumulated, and an evaluation network can be provided that has learned the features of inputs with high evaluation results.When the learning model is used, the input of the learning model can also be input to the above-mentioned evaluation network, and the similarity between the features that are the output result and the features of the learning result can be used as a reliability index.

[0068] Furthermore, for example, during pre-learning, a learning device may be provided that accumulates the results of human evaluation of each input to the learning model, clusters the inputs of the learning model with high evaluation results, and when the learning model is used, the inputs of the learning model may also be input to the above-mentioned learning device, and the results of this clustering may be used as a reliability index.

[0069] The correction confirmation unit 203 determines the validity of the model output data D103 and / or the model input data D101 using the results of the simulation performed by the output confirmation unit 202. The correction confirmation unit 203 may, for example, compare the state of the target device 2 indicated by the simulation results with the state of the target device 2 specified by the input information D11, the model output data D103, and / or the model input data D101 to determine whether correct control is being performed, thereby determining the validity of the model output data D103 and / or the model input data D101. The correction confirmation unit 203 may determine that correct control is being performed when the state of the target device 2 indicated by the simulation results matches the state of the target device 2 specified by the model output data D103 and / or the model input data D101. The state of the target device 2 compared here is not limited to one.

[0070] Furthermore, the correction confirmation unit 203 may determine the validity of the model output data D103 and / or the model input data D101 by, for example, confirming whether the state or control trajectory of the target device 2 shown in the simulation result matches the control shown in the input information D11, or whether it does not include any content that has been prohibited in advance.

[0071] In addition, the correction confirmation unit 203 may present the simulation results to the input source 1a of the input information D11 and ask the input source 1a to respond as to whether the desired control is being performed, thereby determining the validity of the model output data D103 and / or the model input data D101.

[0072] If the correction confirmation unit 203 determines that the model output data D103 and / or the model input data D101 are incorrect, it may correct the model input data D101. Alternatively, instead of correcting the model input data D101, the correction confirmation unit 203 may generate supplemental information D18 for the input information D11 and output it to the input source 1 a.

[0073] The control system 1000 may have, for example, any of the configurations shown in Figures 1 to 5 as the operating environment of the learning model unit 100. As with the learning model unit 100, in this case, some or all of the configuration may be internal to the control system 1000 or may be external.

[0074] The above-described configuration of the learning model unit 100 and its surroundings is merely an example, and not all components are essential components; implementation may be selected as appropriate depending on the desired function.

[0075] Fig. 6 is an explanatory diagram showing an example of model learning. As shown in Fig. 6, model information D102 may be generated by, for example, the model generation unit 107 performing machine learning using model learning data D105.

[0076] The model generation unit 107 is a processing unit that generates or updates model information D102 based on input model learning data D105 in accordance with a predetermined algorithm. The model generation unit 107 is realized, for example, by a CPU that operates in accordance with a program provided in the information processing device 20. Here, the algorithm followed by the model generation unit 107 may be a machine learning algorithm corresponding to the learning model, such as supervised learning, reinforcement learning, or unsupervised learning, or may be deep learning, a genetic program, a functional logic program, or any other known algorithm.

[0077] Furthermore, the model generation unit 107 may generate or update model information D102 for the input model learning data D105, further based on model reference information D104. Furthermore, the model generation unit 107 may generate or update model information D102 for the input model learning data D105, further based on model output data D103 from the model control unit 101.

[0078] The model training data D105 is not particularly limited. For example, when supervised learning is used as the learning algorithm, the model training data D105 may include candidates for model input data D101 that can be input and candidates for model output data D103 corresponding thereto. The model training data D105 may also include actually input model input data D101 and / or actually output model output data D103. Feedback control can be performed by appropriately using the actual model input data D101 and / or model output data D103. The model training data D105 may also include information acquired from a device or processing unit included in a system in which the learning model actually operates.

[0079] The model information D102 generated or updated by the model generation unit 107 is stored in the model information storage unit 11 and is then provided to the model control unit 101. Alternatively, the model generation unit 107 can output the model information D102 directly to the model control unit 101.

[0080] For example, the model generation unit 107 may generate model information D102 by pre-learning using the input model learning data D105 before the model control unit 101 uses the model information D102, and store the model information D102 in the model information storage unit 11.

[0081] The update of the model information D102 by the model generation unit 107 may be a process called FineTune.

[0082] The model generation unit 107 may be included in the control system 1000 or may be included in a system separate from the control system 1000.

[0083] 1, the learning model unit 100 is shown separately from the device information storage unit 110 and the device information D13, but the device information storage unit 110 and the device information D13 may be part of the learning model unit 100. That is, the learning model unit 100 may include the device information storage unit 110 and the device information D13. For example, the learning model unit 100 may include the device information storage unit 110 as one of the reference information storage units 12 described below. Furthermore, the device information D13 may be used in a model learning phase in which the model used by the learning model unit 100 learns, so that the device information D13 is pre-integrated into the model. In this case, the device information storage unit 110 may be omitted.

[0084] Furthermore, part or all of the learning model unit 100 may be an internal configuration of the control system 1000, or may be an external configuration of the control system 1000. When part or all of the learning model unit 1000 is configured externally, the control system 1000 only needs to be provided with an interface that can exchange information with an external system that includes part or all of the learning model unit 100, instead of part or all of the learning model unit 100. For example, the control system 1000 may have the model information storage unit 11, which is called the core of the learning model, as an external configuration. Furthermore, for example, the control system 1000 may have the model information storage unit 11, which is called the core of the learning model, and the model control unit 101, which handles the model algorithm, as external configurations.

[0085] Hereinafter, in the control system 1000, in order to distinguish between the model control unit 101 that handles the learning model algorithm and the unit that performs processing to send a request to the model control unit 101 and obtain a response, the unit that performs the latter processing may be referred to as the "model processing unit." More specifically, the model processing unit corresponds to the above-mentioned information processing device 10, the control unit 104, or the control unit 104a, other than the model control unit 101. Note that, for example, when the model control unit 101 exists in an internal environment, the model processing unit may be realized by an OS (Operating System) or a prompt application (and its operating environment) that runs on the information processing device 10 and calls a learning model application. Note that, for example, when the model control unit 101 exists in an external environment, the model processing unit may be realized by a browser or a client application (and its operating environment) that runs on the information processing device 10.

[0086] The above-described learning model and the configuration of the information processing device as its operating environment, as well as the relationship between the learning model and the control system including it, are similar to those in other embodiments.

[0087] In this embodiment, the input information D11 corresponds to the model input data D101. The control description D12 corresponds to the model output data D103. For example, the learning model unit 100 (particularly the model control unit 101) may be configured to, upon receiving the input information D11, output the control description D12 corresponding to the input information D11 based on the model information D102 and, if necessary, the model reference information D104.

[0088] In such a case, the model generation unit 107 provided corresponding to the learning model unit 100 may perform machine learning using, for example, model learning data D105 including candidates for input information D11 that may be input to the model control unit 101, to generate or update the model information D102. In addition, the model generation unit 107 may perform machine learning using, for example, model learning data D105 including candidates for input information D11 that may be input to the model control unit 101 and candidates for the corresponding control description D12, to generate or update the model information D102.

[0089] In this embodiment, the learning model unit 100 may be, for example, a language learning model such as LLM, which receives input of natural language and obtains output results, and its operating environment. The learning model unit 100 may also be, for example, an image learning model such as VLM, which receives input of images and obtains output results, and its operating environment. The learning model unit 100 may also be, for example, a multimodal model, which receives input of natural language and images and obtains output results, and its operating environment. In this case, the input information D11 may be input in the form of text data, image data, a combination of text data and image data, or a data format convertible thereto (such as audio data or a video that is a combination of audio data and image data). The learning model used in this embodiment is not limited to the above-mentioned models.

[0090] In this embodiment, the input information D11 received by the control system 1000 can be said to be information about a request (here, the control content required of the target device 2) in the work environment, in this case the environment in which the target device 2 operates. Therefore, the input information D11 received by the control system 1000 can be said to be an example of first information indicating a request in the work environment. Furthermore, the control description D12 and the execution code D14 can be said to be information used for the work (the work related to controlling the target device 2) corresponding to such input information D11. Hereinafter, the control description D12 output to a predetermined output destination from the operating environment of the learning model to which model input data based on the input information D11 is input may be referred to as second information.

[0091] Here, in the relationship between input information D11 and model input data, model input data based on input information D11 may include the input information D11 itself, the input information D11 converted into a format that matches the input of the learning model, and supplemented input information D11. Also, in the relationship between model output data and second information, second information based on model output data may include the model output data itself, the model output data converted into a format that matches the input of the output destination, and supplemented model output data. The same applies to the relationship between input / output information and model input / output data in other embodiments.

[0092] Next, a description will be given of the operation of the control system 1000 of this embodiment.

[0093] 7, first, the control system 1000 receives input information D11 (step S110). For example, the input unit 102 or the input processing unit 201 described above may receive the input information D11. The received input information D11 is input to the learning model unit 100 as model input data D101.

[0094] In step S110, the control system 1000 may accept a plurality of pieces of input information D11. Alternatively, the control system 1000 may accept the input information D11 that more closely matches the desires of the user 1 interactively with the user 1, that is, by repeatedly inputting and outputting information related to the input information D11 between the control system 1000 and the user 1.

[0095] Next, the control system 1000 performs a process of generating a control description D12 using the learning model unit 100 (step S111). In step S111, the learning model unit 100 generates and outputs a control description D12 corresponding to the input information D11 that has been input. For example, the learning model unit 100 (more specifically, the model control unit 101) outputs the control description D12 corresponding to the input information D11 based on model information D102, the input input information D11, and model reference information D104 including device information D13 as needed. The learning model unit 100 may generate the control description D12 of text data from the input information D11 that has been input, for example, using a learning model capable of generating text data.

[0096] In step S111, the learning model unit 100 (more specifically, the preprocessing unit 105 or the input processing unit 201) may further add, change, or delete elements from the input information D11, or convert (including processing) data, in order to improve the accuracy of the control description D12 before the processing by the model control unit 101. Also, in step S111, the learning model unit 100 (more specifically, the postprocessing unit 106) may further determine whether or not there is a problem with the control description D12 after the processing by the model control unit 101, and may perform processing to correct the control description D12 if it is determined that there is a problem.

[0097] The control description D12 output from the learning model unit 100 is input to the execution code generation unit 120. When the control description D12 is input, the execution code generation unit 120 generates execution code D14 based on the input control description D12 (step S112).

[0098] Next, the execution code D14 generated by the execution code generation unit 120 is input to the target device 2 (step S113). As already described, the execution code D14 may be input to the target device 2 directly from the control system 1000 (more specifically, the execution code generation unit 120), or may be input indirectly via a communication network, another device (a server, various conversion devices, etc.), or manually.

[0099] As a result, the target device 2 operates in accordance with the input execution code D14.

[0100] If the state of the target device 2 changes as a result of outputting the execution code D14, for example, by controlling the target device 2, the control system 1000 may acquire state information D15 (step S114). The acquired state information D15 is stored in the device information storage unit 110, for example, as part of the device information D13. The control system 1000 may update the device information D13 stored in the device information storage unit 110 using the acquired state information D15. The control system 1000 may also output the acquired state information D15 as information indicating the control result to the user 1, the learning model unit 100, or another device (not shown). Note that if the control system 1000 does not use the state information D15, the processing of step S114 may be omitted.

[0101] The control system 1000 may repeat the processes of steps S110 to S114 multiple times (for example, until the desired control of the target device 2 is completed).

[0102] In addition, the control system 1000 may output the control description D12 to an operation terminal of the user 1, etc., so that the user 1 can confirm the contents, and then subsequent processing (such as code generation in the execution code generation unit 120) can be executed by the operation of the user 1.

[0103] The state information D15 input to the learning model unit 100 is used, for example, for additional learning of the learning model unit 100. The learning model unit 100 may, for example, update the model information D102 and / or the model reference information D104 based on the input state information D15.

[0104] As described above, according to this embodiment, the execution code D14 can be generated from the input information D11 entered by the user 1 without the user 1 having to create the control description D12, thereby improving the efficiency of the work of controlling the target device 2.

[0105] Furthermore, in this embodiment, the input information D11 may be text, an image, audio, or a combination thereof that explicitly or implicitly indicates the control content for the target device 2, thereby further reducing the effort required to input the input information D11 while improving the efficiency of the work of controlling the target device 2.

[0106] Furthermore, according to this embodiment, the control description D12 can be generated from the input information D11 using a learning model, so that the control description D12 corresponding to the input information D11 can be generated even if the user 1 does not know information for controlling the target device 2, such as detailed specifications of the target device 2 or specifications of the control description D12, thereby improving the performance of the task of controlling the target device 2. Here, improving the performance of the task of controlling the target device 2 also includes high precision in controlling the target device 2.

[0107] Furthermore, in this embodiment, the status information D15 obtained after controlling the target device 2 based on the input information D11 can be used to generate the next control description D12, etc., thereby further improving the performance of the task of controlling the target device 2.

[0108] Although only one target device 2 is shown in the above example, there may be multiple target devices 2 that are controlled by the control system 1000. In such a case, for example, the input information D11 may include information that can identify the target device 2, or the input side to the learning model unit 100 (the input unit 102, the preprocessing unit 105, and the input processing unit 201) may perform processing to identify the target device 2 based on the input information D11, or the learning model unit 100 may output control content in which the target device 2 has been identified as a result of learning.

[0109] Variation 1-1. Next, a modified example of the control system 1000 will be described. Fig. 8 is a configuration diagram showing an example of a control system 1000a which is a modified example of the control system 1000 according to the present embodiment. Note that the same elements as those in the control system 1000 are given the same reference numerals and description thereof will be omitted.

[0110] In the control system 1000a shown in FIG. 8, the output from the learning model unit 100 is confirmed by the user 1 and then input to the execution code generation unit 120 at the subsequent stage.

[0111] In this embodiment, the user 1 can check the control description D12 output from the learning model unit 100 and input the input information D11 based on the check results. Furthermore, the user 1 may check feedback information D16 from the execution code generation unit 120 and / or the target device 2 in addition to the control description D12 output from the learning model unit 100 and input the input information D11 based on the check results. In this case, the user 1 may input the input information D11 containing new content, or may input the input information D11 indicating correction, addition, or cancellation of content that has already been input. In this case, the input information D11 may include a command for the learning model unit 100. For example, the user 1 may input, together with the feedback information D16, a command for removing a defect contained in the input information D11 or a defect contained in the output control description D12 as the input information D11. Here, the command for removing the defect may include an input for finding the cause of the defect or a solution to the defect.

[0112] The feedback information D16 may include a response to a request returned from a processing unit downstream of the learning model unit 100 when a control request is made to the processing unit. The feedback information D16 may also include information obtained from a processing unit downstream of the learning model unit 100 after a control request is made to the processing unit. For example, the feedback information D16 may include a response to a request returned from the execution code generation unit 120 when a control description D12 is input to the execution code generation unit 120 to request the execution of the execution code D14. The feedback information D16 may also include a response to a request returned from the target device 2 when the execution code D14 is input to the target device 2 to request the execution of the code. The feedback information D16 may include status information D15. The feedback information D16 may be output directly to the user 1 or may be output to the user 1 via an output device (not shown) included in the execution code generation unit 120 or the control system 1000.

[0113] The feedback information D16 may also include information for determining whether the control requested of a processing unit downstream of the learning model unit 100 is correctly executed in that processing unit. The information is not limited to information obtained directly from the processing unit. For example, the information may be information obtained from another person, a device, a network, or an AI (none of which are shown). The feedback information D16 may include analysis information, such as execution time or control trajectory information, for determining whether the execution code D14 can correctly execute the target control. For example, based on such information included in the feedback information D16, the user 1 may instruct the learning model unit 100 to control the timing of the flow in the control description D12 or adjust the lead time.

[0114] Furthermore, the user 1 may use, for example, feedback information D16 to exchange information with the learning model unit 100 multiple times, and each time determine the validity (presence or absence of a problem) of the output control description D12. If the user 1 determines that there is no problem with the control description D12, the user 1 may output the control description D12 to the execution code generation unit 120.

[0115] The feedback information D16 can be obtained, for example, in step S114 described above.

[0116] Note that Figure 8 shows an example in which user 1 inputs control description D12 to the execution code generation unit 120, but the input of control description D12 to the execution code generation unit 120 can also be performed by the learning model unit 100 upon receiving instructions from user 1.

[0117] In this example, the control description D12 may include, for example, a description corresponding to low code or no code.

[0118] In this example, the exchange of information between user 1 and the learning model unit 100 may be carried out, for example, via a terminal provided by user 1, or via a user interface (e.g., input unit 102) provided by the information processing device 10 on which the learning model unit 100 operates.

[0119] Furthermore, the input information D11 in this example may be updated not by the user 1 but by the control system 1000 (for example, the correction confirmation unit 203, etc.).

[0120] Furthermore, the feedback information D16 may be input to the learning model unit 100. The feedback information D16 input to the learning model unit 100 is used, for example, for additional learning of the learning model unit 100. The learning model unit 100 may update the model information D102 and / or the model reference information D104 based on the input feedback information D16.

[0121] Other points may be the same as those of the other control systems according to this embodiment.

[0122] As described above, in this modification, the user 1 can modify the input information D11 while checking the control description D12 output from the learning model unit 100 and exchanging additional instructions, bug consultations, and the like with the learning model unit 100, thereby improving the accuracy of the control description D12 to be output. As a result, the efficiency and performance of the work of controlling the target device 2 can be improved.

[0123] Variation 1-2. Next, a second modified example of the control system 1000 will be described. Fig. 9 is a configuration diagram showing an example of a control system 1000b, which is a modified example of the control system 1000. Note that the same elements as those in the control systems 1000 and 1000a are given the same reference numerals and description thereof will be omitted.

[0124] The control system 1000b shown in FIG. 9 differs in that the learning model unit 100 returns a query D17 to the user 1. Examples of the query D17 include a query that queries unclear or uncertain input information D11, a query that queries for a solution, and a query that requests re-input of a changed state or expression. The learning model unit 100 may output a query D17 to the user 1 as a query for unclear or uncertain input information D11, which queries the user 1 by presenting a reference section and requesting input of more specific information. Alternatively, the learning model unit 100 may output a query D17 to the user 1 as a query for a solution, which queries the user 1 by presenting a reference section and providing candidate solutions as options. Alternatively, the learning model unit 100 may output a query D17 to the user 1 as a query for a solution, which queries the user 1 by presenting a reference section and providing information on the most likely solution and asking whether the solution is correct or incorrect. In addition, the learning model unit 100 may first generate an intermediate control description, which is an intermediate control description that is easy for humans to understand, and then output the generated intermediate control description along with a query D17 asking whether it is correct or not to the user 1.

[0125] The output of the inquiry D17 may be performed, for example, after the above-mentioned step S110.

[0126] When the learning model unit 100 receives a response from the user 1 to the query D17, the learning model unit 100 may update the input information D11 or determine the interpretation (meaning) of the input information D11.

[0127] The processing of the learning model unit 100 described above in this example can also be implemented, for example, as part of the function of the input unit 102 or preprocessing unit 105 of the learning model unit 100, or the input processing unit 201 (not shown) provided in the information processing device 10.

[0128] As described above, in this modification, the query D17 is output to the user 1 in response to the input information D11, and the input information D11 is updated or its interpretation is confirmed based on the response, thereby eliminating uncertainty in the input information D11. As a result, the precision of the output control description D12 can be improved, and furthermore, the efficiency and performance of the work of controlling the target device 2 can be improved.

[0129] Variation 1-3. Next, a third modified example of the control system 1000 will be described. Fig. 10 is a configuration diagram showing an example of a control system 1000c, which is a modified example of the control system 1000. Note that the same elements as those in the control systems 1000, 1000a, and 1000b are given the same reference numerals and descriptions thereof will be omitted.

[0130] The control system 1000c shown in Fig. 10 further includes a state acquisition unit 130. The state acquisition unit 130 acquires feedback information D16 indicating the processing result or state information D15 indicating the state of the device after processing from the control description D12 output from the learning model unit 100 and the processing destination of the execution code D14 generated from it. Here, the feedback information D16 or state information D15 can include information for determining whether the execution code D14 has correctly executed the target control, such as execution time or control trajectory information.

[0131] The state acquisition unit 130 may, for example, input the acquired information to the learning model unit 100. Furthermore, the state acquisition unit 130 may, for example, update the device information D13 based on the acquired information. Furthermore, the state acquisition unit 130 may, for example, generate information that supplements (including addition, correction, and cancellation) the input information D11 based on the acquired information, and input the information to the learning model unit 100 as supplemental information D18. Furthermore, the state acquisition unit 130 may, for example, generate information that supplements (including addition, correction, and cancellation) the control description D12 based on the acquired information, and input the information to the learning model unit 100 as supplemental information D18.

[0132] The state acquisition unit 130 may generate, for example, a control command of new content or a command indicating addition, modification, or cancellation of content indicated in the already input input information D11 as supplementary information D18 and input it to the learning model unit 100. In addition, the state acquisition unit 130 may input, for example, a command to remove a defect included in the input information D11 or a defect included in the output control description D12 together with the acquired information to the learning model unit 100 as supplementary information D18.

[0133] The status acquisition unit 130 may, for example, determine whether the acquired information indicates normal processing or a normal state at the processing destination, and if not, input supplementary information D18 indicating correction, addition, or cancellation of the content indicated in the input information D11 already input to the learning model unit 100 together with the acquired information.

[0134] The learning model unit 100 may update the model information D102 and / or the model reference information D104 based on, for example, input information (state information D15, feedback information D16, supplementary information D18, etc.).

[0135] The supplemental information D18 may be generated, for example, in step S115 described above. The output destination of the supplemental information D18 may include a destination other than the learning model unit 100. For example, the control system 1000 may output the supplemental information D18 generated by the state acquisition unit 130 to the user 1 or another device (not shown).

[0136] Alternatively, the state acquisition unit 130 may acquire the operation results of a simulator (not shown) of the target device 2 or the operation results of the target device 2 in debug mode, without actually operating the target device 2. The debug mode of the target device 2 refers to a mode in which execution code is executed on the control board of the target device 2 but actual device control is not performed and only the internal state is updated, and is also called an idle operation mode. By using the debug mode, the execution code D14 can be safely tested on the target device 2 in a state close to actual control.

[0137] As a method for determining the validity of the control description D12 output from the learning model unit 100 and the execution code D14 generated from the control description D12 without actually operating the target device 2, the execution code generation unit 120 may be connected so as to be switchable between the target device 2 and a simulator as the output destination of the execution code D14. The simulator may operate an icon of the target device 2 in an augmented reality space. Furthermore, when outputting the execution code D14 to the target device 2, the execution code generation unit 120 may add information instructing whether the execution is to be in normal mode or debug mode.

[0138] The processing of the state acquisition unit 130 in this example can also be implemented as part of the functions of, for example, the input unit 102, pre-processing unit 105 and post-processing unit 106 of the learning model unit 100, or the input processing unit 201, output confirmation unit 202 and correction confirmation unit 203 (all not shown) provided in the information processing device 10.

[0139] Other points may be the same as those of the other control systems according to this embodiment.

[0140] As described above, in this modification, in response to input information D11, the state acquisition unit 130 acquires feedback information D16 indicating the processing result or state information D15 indicating the state of the device after processing from the target device 2 or the execution code generation unit 120, which is the output destination of the model output data D103 and / or information generated based on the model output data, and issues supplemental information D18 to the learning model unit 100 as appropriate based on the acquired information. This makes it possible to improve the accuracy of the control description D12, and ultimately to improve the efficiency and performance of the work of controlling the target device 2.

[0141] Furthermore, in this modified example, for example, a person and a machine (state acquisition unit 130) can cooperate to improve the accuracy of input to the learning model unit 100, which can also contribute to reducing the workload of user 1.

[0142] Embodiment 2 Next, a description will be given of a second embodiment of the present invention. In this embodiment, an example will be described in which a learning model is used to support work related to the control of a target device.

[0143] In the following, we consider the control of various control devices such as PLCs, processing machines, robots, sensors, transport devices, and other machine control devices in a factory. Experienced workers may be familiar with the control methods for a wide variety of control devices and complex control devices, but due to reassignments and other reasons, less experienced workers may be required to control the control devices. Furthermore, when new control devices (including upgrades) are introduced, it is necessary to inform all workers about the control methods for the new control devices, and insufficient awareness could lead to mistakes.

[0144] In such cases, it is preferable to be able to reliably perform the desired control without knowing the specific control method, such as the control instructions, control signals, control codes, or commands to the controller corresponding to the control device, as this will lead to improved work efficiency and performance.

[0145] The situation in which devices are controlled is not limited to within a factory, and the situation in which this embodiment is used is not limited to within a factory.

[0146] Fig. 11 is a configuration diagram illustrating an example of a control system 2000 according to the second embodiment. The control system 2000 illustrated in Fig. 11 is a control system for controlling equipment using a learning model, and includes a learning model unit 200 and an equipment information storage unit 210 (referred to as equipment information DB in the figure).

[0147] When input information D21 is input, the learning model unit 200 outputs a control command D22. When input information D21 is input, for example, the learning model unit 200 outputs a control command D22 based on model information D102. The configuration of the learning model unit 200 may be basically the same as that of the learning model unit 100 in the first embodiment.

[0148] In this embodiment, the learning model unit 200 is a model and its operating environment configured to output a control command D22 corresponding to input information D21 when input information D21 is received. Alternatively, the learning model unit 200 may be a model and its operating environment configured to generate and output a control command D22 based on the input information D21, device information D23, and other information that can be referenced in the learning model unit 200 when input information D21 is received.

[0149] In this embodiment, the input information D21 includes information indicating control details for the target device 2. The input information D21 may be, for example, text, images, audio, or a combination thereof indicating the control details for the target device 2. The input information D21 may be, for example, text, images, audio, or a combination thereof indicating multiple control details for the target device 2. The input information D21 may also include information indicating control details that are performed consecutively over time. In this case, the input information D21 may be time-series data with a predetermined data structure including text, images, audio, or a combination thereof indicating the control details, as described above. It is assumed that the way the control details are represented matches the input format of the model used by the learning model unit 200. However, this does not apply if error processing, correction processing, or conversion processing is included upstream of the learning model unit 200.

[0150] The control content in the input information D21 may be indicated in the same manner as in the first embodiment, for example. For example, after specifying the control to be performed on the target device 2, the value of a parameter for performing the control or the state after the control may be specified. In this case, the input information D21 may include, for example, information for specifying the control and information indicating the value of a parameter for performing the control or the state after the control. Furthermore, the input information D21 may include not only information directly indicating the control content to the target device 2, but also information indirectly indicating the control content using operation content corresponding to the control content, words and actions of the user 1, an image of the target device 2, or a similar control command for another model, etc.

[0151] The control command D22 includes information regarding the control of the target device 2, which is indicated in a predetermined format that enables identification of the target device 2 or an interface requesting control of the target device 2. The control command D22 may include information indicating a control request to the target device 2. The control command D22 is, for example, a control command, a control signal, or a control code for the target device 2. Furthermore, the control command D22 may be, for example, a command written in a format that can be handled by a predetermined controller corresponding to the target device 2.

[0152] The device information storage unit 210 stores device information D23, which is information related to the target device 2. The device information storage unit 210 and the device information D23 are handled basically in the same way as the device information storage unit 110 and the device information D13 in the first embodiment. Note that the device information D23 in this embodiment may include, for example, information used to control the target device 2. The device information D23 is used, for example, as additional information when the learning model unit 200 outputs the control command D22. Hereinafter, in this embodiment, information particularly indicating the state of the target device 2 may be referred to as state information D25.

[0153] In this embodiment, the learning model unit 200 may be, for example, a language learning model such as LLM, which receives input of natural language and obtains output results, and its operating environment. The learning model unit 200 may also be, for example, an image learning model such as VLM, which receives input of images and obtains output results, and its operating environment. The learning model unit 200 may also be, for example, a multimodal model, which receives input of natural language and images and obtains output results, and its operating environment. In this case, the input information D21 may be input in the form of text data, image data, a combination of text data and image data, or a data format convertible thereto (such as audio data or a video that is a combination of audio data and image data). The learning model used in this embodiment is not limited to the above-mentioned models.

[0154] In this embodiment, for the sake of simplicity, the components provided in correspondence with the learning model unit 200 may be described using the same reference numerals as those of the components provided in correspondence with the learning model unit 100, but it should be noted that these components are provided in correspondence with the learning model unit 200. This also applies to the other embodiments.

[0155] In this embodiment, the input information D21 corresponds to the model input data D101. The control command D22 corresponds to the model output data D103. For example, the learning model unit 200 (particularly the model control unit 101) may be configured to, upon receiving the input information D21, output the control command D22 corresponding to the input information D21 based on the model information D102 and, if necessary, the model reference information D104.

[0156] In such a case, the model generation unit 107 provided corresponding to the learning model unit 200 may perform machine learning using, for example, model learning data D105 including candidates for input information D21 that may be input to the model control unit 101, to generate or update the model information D102. In addition, the model generation unit 107 may perform machine learning using, for example, model learning data D105 including candidates for input information D21 that may be input to the model control unit 101 and candidates for the corresponding control commands D22, to generate or update the model information D102.

[0157] Reference symbol D26 denotes feedback information indicating the control result of the target device 2. In this embodiment, too, the state information D25 and / or feedback information D26 may be acquired from the output destination of the model output data D103 and / or information generated based on the model output data. The control system 2000 may output the acquired state information D25 and / or feedback information D26 as information indicating the control result to the user 1, the learning model unit 200, or another device (not shown). The control system 2000 may also generate supplemental information D28 for the input / output data of the learning model unit 200 based on the acquired state information D25 and / or feedback information D26, and issue the supplemental information D28 to the user 1, the learning model unit 200, or another device (not shown). The control system 2000 may also be configured to return a query D27 to the user 1 when the input information D21 contains unclear or uncertain information. The query D27 is handled in the same manner as the query D17 in the first embodiment.

[0158] Fig. 12 is a configuration diagram showing another example of the control system 2000. As shown in Fig. 12, the control system 2000 may further include a status acquisition unit 230 that acquires status information D25 and / or feedback information D26 and issues supplemental information D28. The status acquisition unit 230 is similar to the status acquisition unit 130 in the first embodiment.

[0159] In this embodiment, the target device 2 is not particularly limited. It is assumed that the target device 2 is a device that can actually be controlled by receiving the control command D22, but this does not necessarily apply if a conversion device that converts various signals, such as a controller or converter, is included between the target device 2 and the target device 2. In that case, it is sufficient that the conversion device receives the control command D22 and controls the target device 2.

[0160] In this embodiment, the input information D21 received by the control system 2000 can be said to be information related to a demand in the work environment, in this case, the environment in which the target device 2 operates (here, the control content required of the target device 2). Therefore, the input information D21 received by the control system 2000 can be said to be an example of first information indicating a demand in the work environment. Furthermore, the control command D22 can be said to be information used for the work (the work related to the control of the target device 2) corresponding to such input information D21. Hereinafter, the control command D22 output to a predetermined output destination from the operating environment of the learning model to which model input data based on the input information D21 is input will be referred to as second information.

[0161] Next, a description will be given of the operation of the control system 2000 according to this embodiment.

[0162] 13, first, the control system 2000 receives input information D21 (step S210). For example, the input unit 102 or the input processing unit 201 described above may receive the input information D21. The received input information D21 is input to the learning model unit 200 as model input data D101.

[0163] Next, the control system 2000 performs a process of generating a control command D22 using the learning model unit 200 (step S211). In step S211, the learning model unit 200 (more specifically, the model control unit 101) outputs a control command D22 corresponding to the input information D21 based on the model information D102, the input information D21 that has been input, and model reference information D104 including device information D23 as needed.

[0164] In step S211, the learning model unit 200 may generate a binary data control command D22 from the input information D21 using, for example, a learning model capable of generating binary data. Alternatively, the learning model unit 200 may generate a text data control command D22 from the input information D21 using, for example, a learning model capable of generating text data. Alternatively, the learning model unit 200 may generate an image data control command D22 from the input information D21 using, for example, a learning model capable of generating image data. Alternatively, the learning model unit 200 may generate an audio data control command D22 from the input information D21 using, for example, a learning model capable of generating audio data.

[0165] In step S211, the pre-processing unit 105 and / or the post-processing unit 106 of the learning model unit 200 may further perform the above-mentioned processing.

[0166] The control command D22 output from the learning model unit 200 is input to, for example, the target device 2 (step ST212). The control command D22 may be input to the target device 2 directly from the control system 2000 (more specifically, the learning model unit 200 or the information processing device 10 that is its operating environment), or may be input indirectly via a communication network or other devices (servers, various conversion devices, etc.).

[0167] As a result, the target device 2 operates in accordance with the input control command D22.

[0168] When the state of the target device 2 changes due to the target device 2 being controlled as a result of outputting the control command D22, or when there is feedback from the target device 2, the control system 2000 may acquire state information D25 and feedback information D26 (step S213). Note that the processing of step S213 is not essential and may be omitted as appropriate.

[0169] The control system 2000 may repeat the processes of steps S210 to S213 multiple times (for example, until the desired control of the target device 2 is completed).

[0170] As described above, according to this embodiment, even if the user 1 does not know the specific control method for the target device 2, the control command D22 can be generated from the input information D21 input by the user 1, and the target device 2 can be controlled based on the generated control command D22, thereby making it possible to improve the efficiency and sophistication of the work involved in controlling the target device 2.

[0171] Furthermore, according to this embodiment, even with ambiguous information, the device can be controlled to an appropriate state.

[0172] Embodiment 3 Next, a description will be given of a third embodiment of the present invention. In this embodiment, an example will be described in which a learning model is used to assist in a task related to operating a target device.

[0173] In the following, we consider the operation of various devices in a home or building, such as air conditioners, refrigerators, televisions, lighting, washing machines, projectors, various sensors, and communication devices. In recent years, even these consumer devices have become more sophisticated and their control has become more complex. Although improvements have been made to operation screens and controllers such as remote controllers to make complex controls easier, it is still difficult to remember all the operations, and even if a desired function is available, it may not be easy to reach that function.

[0174] Furthermore, even when the functions are similar, the names of the functions provided vary depending on the model, there are often differences in the detailed functions, and the control methods are different. Therefore, when introducing a different model, such as by replacing it, it is necessary to learn all the differences from scratch, which is cumbersome.

[0175] Furthermore, some devices automatically control themselves to an appropriate state by remembering past operation history or understanding the operating environment, but accurate control can be difficult in situations where multiple people gather and the appropriate state varies depending on the person, or in situations where the appropriate state varies even for one person due to changes in their physical condition, etc.

[0176] In such cases, it is desirable to be able to easily perform operations to achieve the desired state, even if the operator does not know the specific operating method or does not understand the appropriate state, as this will lead to improved work efficiency and performance.

[0177] The situation where the device is operated is not limited to within a home or a building, and the situation where the present embodiment is used is not limited to within a home or a building.

[0178] Fig. 14 is a configuration diagram showing an example of a control system 3000 according to the third embodiment. The control system 3000 shown in Fig. 14 is a control system for operating equipment using a learning model, and includes a learning model unit 300, an equipment information storage unit 310 (referred to as equipment information DB in the figure), an input interface 311 (referred to as input IF in the figure), and an output interface 312 (referred to as output IF in the figure).

[0179] When input information D31 is input, the learning model unit 300 outputs an operation command D32. For example, when input information D31 is input, the learning model unit 300 outputs an operation command D32 based on model information D102. The configuration of the learning model unit 300 may be basically the same as that of the learning model unit 100 in the first embodiment.

[0180] In this embodiment, the learning model unit 300 is a model and its operating environment configured to output an operation command D32 corresponding to input information D31 when input information D31 is received. Alternatively, the learning model unit 300 may be a model and its operating environment configured to generate and output an operation command D32 based on the input information D31, device information D33, and other information that can be referred to in the learning model unit 300 when input information D31 is received.

[0181] In this embodiment, the input information D31 includes information indicating operation details requested of the target device 2. The input information D31 may be, for example, text, images, audio, or a combination thereof indicating operation details for the target device 2. The input information D31 may be, for example, text, images, audio, or a combination thereof indicating multiple operation details for the target device 2. The input information D31 may also include information indicating operation details performed consecutively over time. In this case, the input information D31 may be time-series data with a predetermined data structure including text, images, audio, or a combination thereof indicating the operation details, as described above. It is assumed that the way the operation details are indicated matches the input format of the model used by the learning model unit 300. However, this does not apply when error processing, correction processing, or conversion processing is included upstream of the learning model unit 300.

[0182] As an example of how the input information D31 indicates the operation content, the input information D31 may first identify an operation to be performed on the target device 2 and then specify the parameter values ​​for performing the operation or the state after the operation. In this case, the input information D31 may include, for example, information identifying the operation and information indicating the parameter values ​​for performing the operation or the state after the operation. The parameter values ​​for performing the operation may include, for example, values ​​related to the type of operation (ON / OFF, etc.), direction, amount, and time. Furthermore, the input information D31 may include not only information directly indicating the operation content on the target device 2, but also information indirectly indicating the control content corresponding to the operation content using, for example, the words and actions of the user 1, an image of the target device 2, or a similar operation command for another model.

[0183] The operation command D32 includes information regarding the operation of the target device 2, which is displayed in a predetermined format that enables the target device 2 or an interface (including a person) requesting control of the target device 2 to be identified. The operation command D32 may include information indicating an operation request or control request to the target device 2. The operation command D32 is, for example, an operation command, an operation signal, an operation code, a control command, a control signal, or a control code for the target device 2. The operation command D32 may also be, for example, a command written in a format that can be handled by a predetermined controller corresponding to the target device 2. The operation command D32 can be considered a concept that adds information regarding operation to the above-mentioned control command D22. Furthermore, for example, when the interface is a person, that is, when control of the target device 2 is requested via a person, the operation command D32 may be information indicating a method of operating the target device 2, which is displayed in a format that can be identified by a person.

[0184] The device information storage unit 310 stores device information D33, which is information related to the target device 2. The device information storage unit 310 and the device information D33 are handled basically in the same way as the device information storage unit 110 and the device information D13 in the first embodiment. Note that the device information D33 in this embodiment may include, for example, information used to operate the target device 2. The device information D33 may include, for example, information indicating the procedure of operations actually performed on the target device 2 in response to operation content. The device information D33 may also include, for example, commands, signals, codes, etc. issued to the target device 2. The device information D33 is used, for example, as additional information when the learning model unit 300 outputs the operation command D32. Hereinafter, in this embodiment, information indicating the state of the target device 2 may be referred to as state information D35.

[0185] The input interface 311 is an interface that receives input information D31 from the user 1 and inputs it to the learning model unit 300. The input interface 311 may be, for example, an interface that converts the input information D31 input from the user 1 into data that matches the input to the learning model unit 300 and outputs the data. The input interface 311 may be provided, for example, as an example of the input unit 102 described above.

[0186] The output interface 312 is an interface that receives the operation command D32 from the learning model unit 300 and outputs it to a predetermined output destination. The output interface 312 may be provided, for example, as an example of the output unit 103 described above. The output interface 312 may be, for example, an interface that converts the operation command D32 output from the learning model unit 300 into data that matches the predetermined output destination and outputs it. In this embodiment, the output destinations of the output interface 312 may include the target device 2, the controller 4 (not shown), a predetermined display 7 (not shown), and the operation terminal (not shown) of the user 1.

[0187] In this embodiment, the learning model unit 300 may be, for example, a language learning model such as LLM, which receives natural language as input and obtains an output result, and its operating environment. The learning model unit 300 may also be, for example, an image learning model such as VLM, which receives images as input and obtains an output result, and its operating environment. The learning model unit 300 may also be, for example, a multimodal model, which receives natural language and images as input and obtains an output result, and its operating environment. In this case, the input information D31 may be input in the form of text data, image data, a combination of text data and image data, or a data format convertible thereto (such as audio data or a video that is a combination of audio data and image data). The learning model used in this embodiment is not limited to the above-mentioned models.

[0188] In this embodiment, the input information D31 corresponds to the model input data D101. The operation command D32 corresponds to the model output data D103. For example, the learning model unit 300 (particularly the model control unit 101) may be configured to, upon receiving the input information D31, output the operation command D32 corresponding to the input information D31 based on the model information D102 and, if necessary, the model reference information D104.

[0189] In such a case, the model generation unit 107 provided corresponding to the learning model unit 300 may perform machine learning using, for example, model learning data D105 including candidates for input information D31 that may be input to the model control unit 101, to generate or update the model information D102. In addition, the model generation unit 107 may perform machine learning using, for example, model learning data D105 including candidates for input information D31 that may be input to the model control unit 101 and candidates for the corresponding operation commands D32, to generate or update the model information D102.

[0190] Although not shown in the figure, in this embodiment, state information D35 and / or feedback information D36 may be acquired from the model output data D103 of the learning model unit 300 and / or the output destination of information generated based on the model output data. For example, the control system 3000 may output the acquired state information D35 and / or feedback information D36 as information indicating a response result to the user 1, the learning model unit 300, or another device (not shown). The control system 3000 may also be configured to return a query D37 to the user 1 when the input information D31 contains unclear or uncertain information. The control system 3000 may also generate supplemental information D38 for the input / output data of the learning model unit 300 based on the acquired state information D35 and / or feedback information D36 and issue the supplemental information D38 to the user 1, the learning model unit 300, or another device (not shown). The handling of the state information D35, feedback information D36, query D37, and supplemental information D38 may be basically the same as in the first embodiment.

[0191] The control system 3000 may further include a status acquisition unit 330 (not shown) that acquires the status information D35 and / or the feedback information D36 and issues supplemental information D38 as necessary. The status acquisition unit 330 is similar to the status acquisition unit 130 in the first embodiment.

[0192] In this embodiment, the target device 2 is not particularly limited. It is assumed that the target device 2 is a device that can receive the operation command D32 and perform control corresponding to the operation content indicated by the operation command D32, but this does not apply if a conversion device that converts various signals, such as a controller 4 or a converter, or an operator is included between the target device 2 and the target device 2. In that case, the conversion device or the operator can receive the operation command D32 and operate the target device 2.

[0193] In this embodiment, the input information D31 received by the control system 3000 can be said to be information related to a request (here, operation content requested of the target device) in the work environment, in this case, the environment in which the target device 2 operates. Therefore, the input information D31 received by the control system 3000 can be said to be an example of first information indicating a request in the work environment. Furthermore, the operation command D32 can be said to be information used for the work (work related to operating the target device 2) corresponding to such input information D31. Hereinafter, the operation command D32 output to a predetermined output destination from the operating environment of the learning model to which model input data based on the input information D31 is input may be referred to as second information.

[0194] Next, a description will be given of the operation of the control system 3000 according to this embodiment.

[0195] 15, first, the input interface 311 of the control system 3000 receives input information D31 (step S310). The received input information D31 is input to the learning model unit 300 as model input data D101.

[0196] Next, the control system 3000 performs a process of generating an operation command D32 using the learning model unit 300 (step S311). In step S311, the learning model unit 300 (more specifically, the model control unit 101) generates and outputs an operation command D32 corresponding to the input information D31 based on the model information D102, the input information D31 that has been input, and model reference information D104 including device information D33 as needed.

[0197] In step S311, the learning model unit 300 may generate a binary data operation command D32 from the input information D31 using, for example, a learning model capable of generating binary data. Alternatively, the learning model unit 300 may generate a text data operation command D32 from the input information D31 using, for example, a learning model capable of generating text data. Alternatively, the learning model unit 300 may generate an image data operation command D32 from the input information D31 using, for example, a learning model capable of generating image data. Alternatively, the learning model unit 300 may generate an audio data operation command D32 from the input information D31 using, for example, a learning model capable of generating audio data.

[0198] In step S311, the pre-processing unit 105 and / or the post-processing unit 106 of the learning model unit 300 may further perform the above-mentioned processing.

[0199] The operation command D32 output from the learning model unit 300 is output to a predetermined output destination via the output interface 312, for example. Here, the predetermined output destination may be the target device 2, the controller 4, a predetermined display 7, or an operation terminal (not shown) of the user 1. When the operation command D32 is input to the predetermined output destination, the target device 2 is operated in accordance with the input operation command D32 (step S312).

[0200] For example, the output interface 312 may output an operation command D32 to the target device 2. In this case, the target device 2 that has received the operation command D32 (e.g., an operation command, an operation signal, an operation code, a control command, a control signal, or a control code) may perform actual control in accordance with the operation command D32. The output interface 312 may also output the operation command D32 to a controller 4 corresponding to the target device 2. In this case, the controller 4 that has received the operation command D32 (e.g., indirect control information for the target device 2, such as a command, an operation command, an operation signal, or an operation code for the controller 4) may operate the target device 2 in accordance with the operation command D32. The controller 4 may operate the target device 2 by outputting direct control information, such as a control code, to the target device 2 based on the control information indicated in the received operation command D32. Here, the controller 4 may be, for example, an operation panel provided on the target device 2, or a remote controller corresponding to the target device 2 that is directly operated by the user 1. The controller 4 may be a controller specific to the target device 2 or a general-purpose controller. Furthermore, the output interface 312 may output the operation command D32 to an operation terminal or a predetermined display device of the user 1. In this case, the operation terminal or display device of the user 1 that has received the operation command D32 (e.g., information indicating an operation method) displays the operation command D32. The user 1 may then operate the target device 2 or the controller 4 by referring to the displayed operation command D32.

[0201] The operation command D32 may be input to the output destination directly from the control system 3000 (more specifically, the learning model unit 300 or the information processing device 10 which is its operating environment), or may be input indirectly via a communication network or other equipment (servers, various conversion devices, etc.).

[0202] As a result, the target device 2 operates in accordance with the operation command D32.

[0203] If, as a result of outputting the operation command D32, the state of the target device 2 is changed due to the target device 2 being operated or there is feedback from the target device 2, the control system 3000 may acquire state information D35 and feedback information D36 (step S313). Note that the processing of step S313 is not essential and can be omitted as appropriate.

[0204] The control system 3000 may repeat the processes of steps S310 to S313 multiple times (for example, until the desired operation on the target device 2 is completed).

[0205] As described above, according to this embodiment, even if the user 1 does not know the specific operation method for the target device 2, the operation command D32 can be generated from the input information D31 input by the user 1, and the target device 2 can be operated based on the generated operation command D32, thereby making it possible to improve the efficiency and sophistication of the work involved in operating the target device 2.

[0206] Furthermore, according to this embodiment, even if ambiguous information is used, it is possible to operate the device in an appropriate state. Furthermore, according to this embodiment, it is possible to operate the device in an appropriate state independently of the device and without having to learn how to operate the device.

[0207] Variation 3-1. Next, a modified example of the control system 3000 will be described. Fig. 16 is a configuration diagram showing an example of a control system 3000a which is a modified example of the control system 3000 according to the present embodiment. Note that the same elements as those in the control system 3000 are given the same reference numerals and description thereof will be omitted.

[0208] 16 further includes an input determination unit 31. Upon receiving input information D31, the input determination unit 31 analyzes the input information D31 and switches the control destination for the input information D31. In this modification, the input determination unit 31 switches the control destination for the input information D31 between the learning model unit 300 and the output interface 312.

[0209] The input determination unit 31 may switch the control destination for the input information D31 depending on, for example, whether or not the input information D31 conforms to a command rule for an operation on the target device 2. If the input information D31 conforms to a command rule for an operation on the target device 2, the input determination unit 31 may input the input information D31 as is to the output interface 312. On the other hand, if the input information D31 does not conform to a command rule for an operation on the target device 2, the input determination unit 31 may input the input information D31 to the learning model unit 300.

[0210] Whether or not the input conforms to the command rules of the operation may be determined using, for example, a model described in a rule base. Here, the input determination unit 31 may be a learning model that is relatively lightweight compared to the learning model unit 300.

[0211] Hereinafter, in order to distinguish the information input at the output interface 312, the operation command D32 output from the learning model unit 300 may be referred to as the operation command D32a, and the input information D31 output to the output interface 312 may be referred to as the operation command D32b.

[0212] In this example, the output interface 312 may be any interface that receives the operation command D32a or D32b and outputs it to a predetermined output destination.

[0213] Next, the operation of the control system 3000a of this modified example will be described below. Fig. 17 is a flowchart showing an example of the operation of the control system 3000a.

[0214] 17, first, the input interface 311 of the control system 3000a receives input information D31 (step S310). The received input information D31 is input to the input determining unit 31.

[0215] Next, the input determination unit 31 determines whether the input information D31 matches the command rule for the operation on the target device 2 (step S321). If it is determined that the input information D31 matches the command rule for the operation on the target device 2 (Yes in step S321), the input information D31 is input to the output interface 312 (proceed to step S322). On the other hand, if it is determined that the input information D31 does not match the command rule for the operation on the target device 2 (No in step S321), the input information D31 is input to the learning model unit 300 (proceed to step S311).

[0216] The processing in steps S311 to S313 is the same as the example shown in FIG.

[0217] In step S322, the output interface 312 outputs the input information D31 to a predetermined output destination as an operation command D32b, causing the target device 2 to operate in accordance with the operation command D32b.

[0218] Other points may be the same as those of the other control systems according to this embodiment.

[0219] As described above, according to this modification, when the input from the user 1 matches the command rule for operating the target device 2, the target device 2 can be operated in accordance with the input, while when the input does not match, the learning model can be used to operate the target device 2. This makes it possible to further improve the efficiency of the work involved in operating the target device 2.

[0220] Variation 3-2. Next, we will explain another modified example of the control system 3000. In this modified example, a learning model is used to generate an operation command that includes arbitration of multiple inputs.

[0221] 18 is a configuration diagram showing an example of a control system 3000b which is a modified example of the control system 3000 according to the present embodiment. Note that the same elements as those in the control system 3000 are given the same reference numerals and description thereof will be omitted.

[0222] In a control system 3000b shown in FIG. 18, an input interface 311 receives input information D31 from a plurality of users 1.

[0223] The input interface 311 accepts input information D31 from multiple users 1 and inputs it to the learning model unit 300. At this time, the input interface 311 may accept the input information D31 to which information about the user 1 who is the input source is attached, or the input interface 311 may identify the user 1 who is the input source and attach the information about the input source before accepting the input information D31, or may accept the input information without doing anything in particular.

[0224] The learning model unit 300 may be a model and its operating environment configured to output an operation command D32 corresponding to the input information D31 group when the input information D31 group is received by the input interface 311. The learning model unit 300 may be a model and its operating environment configured to generate and output an operation command D32 based on the input information D31 group, device information D33, and other information that can be referred to in the learning model unit 300 when the input information D31 group is received.

[0225] For example, the learning model unit 300 may use a language learning model such as LLM, which inputs natural language and obtains an output result, to perform processing to extract a preferred solution in a language space (more specifically, in a feature vector space having information on the language space), thereby generating and outputting an operation command D32 that is a compromise between different operation contents indicated by a group of input information D31. At this time, the learning model unit 300 may refer to the history of the input information D31 for each input source user 1 and / or the history of the operation command D32 for each input source user 1.

[0226] Other points may be the same as those of the other control systems according to this embodiment.

[0227] As described above, according to this modified example, even if information regarding different operation contents is input from multiple users 1, the learning model unit 300 can be used to generate a more appropriate operation command D32 by reconciling the contents, thereby further improving the functionality of the work related to operating the target device 2.

[0228] Variation 3-3. Next, we will explain another modified example of the control system 3000. In this modified example, a learning model is used to generate an operation screen user interface.

[0229] 19 is a configuration diagram showing an example of a control system 3000c which is a modified example of the control system 3000 according to the present embodiment. Note that the same elements as those in the control system 3000 are given the same reference numerals and description thereof will be omitted.

[0230] 19 further includes an operation screen user interface 3 (referred to as operation screen UI in the figure). The learning model unit 300 generates an operation screen for actually performing an operation on the target device 2 according to the operation content corresponding to the input information D31 as an operation command D32.

[0231] The operation screen generated by the learning model unit 300 may be, for example, a screen API (Application Programming Interface) having a function to receive operation input from the user 1 along with an explanation of the operation content and output a control command D34 such as a control code corresponding to the received operation input. Here, the output of a control code or the like corresponding to the operation input includes a mode in which multiple control commands D34 are output sequentially in response to one operation input. The operation screen may also be a screen API having operation instructions, operation input acceptance, and control command output corresponding to two or more different operation contents. For example, the learning model unit 300 may extract operation information indicating two or more different operation contents as operation commands D32 corresponding to the input information D31, and generate a screen API having operation input acceptance and control command output corresponding to each operation information.

[0232] In addition, the operation screen generated by the learning model unit 300 may be an existing operation screen whose display mode is modified so that the operation points corresponding to the corresponding operation content are highlighted, the operation functions are displayed with limited functionality, or the position and form (shape, size, color, etc.) of UI components on the screen are changed and displayed.

[0233] The operation screen user interface 3 is an interface that displays an operation screen for the target device 2 and accepts input related to user operations on the operation screen. The operation screen user interface 3 may be realized, for example, by a touch panel display or a controller equipped with operation buttons and a display unit. The operation screen user interface 3 may also be realized by a display device such as a display that cooperates with an operation input device such as a mouse.

[0234] Moreover, the output interface 312 in this modification outputs the operation command D32 (operation screen) output from the learning model unit 300 to the operation screen user interface 3.

[0235] In this modification, the learning model unit 300 may have a function of interactively confirming the operation expected by the user 1. In such a case, for example, after presenting an operation screen as the operation command D32, when the learning model unit 300 receives information requesting reacquisition of the operation command D32, the learning model unit 300 may change part of the input information, part of the model parameters, or the reference destination of the reference information, and then reacquire the operation command D32.

[0236] As described above, in this modification, the learning model unit 300 can be used to generate an operation screen on which measures (such as constructing a screen API or changing the display mode) have been taken to enable desired operations to be performed simply or clearly, thereby further improving the efficiency of work involved in operating the target device 2. Furthermore, according to this modification, the user 1 can perform actual operations while checking explanations of the operation commands generated by the learning model unit 300, allowing for error-free operation.

[0237] Variation 3-4. Next, we will explain another modified example of the control system 3000. In this modified example, the learning model further uses environmental information to generate operation commands.

[0238] 20 is a configuration diagram showing an example of a control system 3000d which is a modified example of the control system 3000 according to the present embodiment. Note that the same elements as those in the control system 3000 are given the same reference numerals and description thereof will be omitted.

[0239] The control system 3000d shown in FIG. 20 further includes an environment information storage unit 313 (referred to as environment information DB in the drawing).

[0240] The environment information storage unit 313 stores environment information D33a, which is information about the environment in which the target device 2 operates. The environment information D33a may include information about the space in which the target device 2 operates. In this modification, information about objects or people present in the space in which the target device 2 operates and the user 1 who is the operator of the target device 2 are also considered to be part of the environment. Therefore, the environment information D33a may include information about the objects or people or user 1.

[0241] The environmental information D33a may include, for example, information about the person, such as the person's attributes, temperature, position, posture, and heart rate. The environmental information D33a may also include, for example, information about the space, such as the location, temperature, humidity, and brightness of the space. The environmental information D33a may also hold information indicating a transition in the information about the space or person when that information changes. Here, the information indicating the transition is also referred to as time-series data or history information. The environmental information D33a may be configured, for example, as part of the model reference information D104 of the learning model.

[0242] The environmental information D33a may be acquired by, for example, a sensor (not shown).

[0243] The learning model unit 300 is a model and its operating environment configured to, for example, when input information D31 is input, generate and output operation commands D32 based on the input information D31, device information D33, environmental information D33a, and other information that can be referenced in the learning model unit 300.

[0244] As described above, according to this modified example, the learning model can generate operation commands D32 by also using environmental information D33a related to the space in which the target device 2 is operating, thereby further improving the functionality of the work involved in operating the target device 2.

[0245] Variation 3-5. Next, another modified example of the control system 3000 will be described. In this modified example, two learning models are combined to generate an operation command. FIG. 21 is a configuration diagram showing an example of a control system 3000e, which is a modified example of the control system 3000 according to the present embodiment. Note that the same elements as those in the control system 3000 are given the same reference numerals and their description will be omitted.

[0246] A control system 3000e shown in FIG. 21 includes a learning model unit 300a as the first learning model unit 300 and a learning model unit 300b as the second learning model unit 300, instead of the learning model unit 300 shown in FIG.

[0247] When input information D31 is input, the learning model unit 300a outputs operation information D320. The learning model unit 300a may be a model and its operating environment configured to generate and output operation information D320 based at least on the input information D31 and environmental information D33a when input information D31 is input.

[0248] Here, the operation information D320 includes information about the operation of the target device 2, which is displayed in a predetermined format that can be interpreted by the subsequent learning model unit 330b. Here, the operation information D320 may be information in which the operation content indicated in the input information D31 is supplemented (including addition, correction, or cancellation) in accordance with the environment information D33a. The operation information D320 may be information in which the operation content indicated in the input information D31 or its expression is changed in accordance with the situation of the space in which the target device 2 is operated. The learning model unit 300a may mainly be a model that performs grounding on the input information D31.

[0249] For example, even if the desired operation is the same, differences in linguistic expression and perception of events may occur depending on the environment in which the target device 2 is running. For example, the operation content implied by the input information D31 may differ depending on dialects, idiosyncratic expressions, the use of company terminology or household terminology, differences in perception of hot / cold, etc.

[0250] The learning model unit 300a, for example, absorbs such differences in linguistic expression and / or differences in perception of events and modifies the content to be more general or specific. The learning model unit 300a may be a local learning model that obtains output results based on local information, such as limited reference databases.

[0251] The operation information D320 generated by the learning model unit 300a is input to the learning model unit 300b.

[0252] The learning model unit 300b may basically be the same as the above-described learning model unit 300. However, instead of the input information D31, the operation information D320 generated by the learning model unit 300a is input.

[0253] The learning model unit 300b outputs an operation command D32 when operation information D320 is input. The learning model unit 300b may be a model and its operating environment configured to generate and output the operation command D32 when operation information D320 is input, based on the operation information D320, device information D33, and other information that can be referenced by the learning model unit 300b. The learning model unit 300b may be a global learning model that obtains output results based on global information, such as information that can freely access an external network.

[0254] Fig. 22 is a flowchart showing an example of the operation of this modified example. In the example shown in Fig. 22, when the input interface 311 of the control system 3000e receives input information D31 in step S310, the input information D31 is input to the learning model unit 300a.

[0255] Next, the control system 3000e performs a process of generating operation information D320 using the learning model unit 300a (step S331). In step S331, the learning model unit 300a (more specifically, the model control unit 101) generates and outputs operation information D320 corresponding to the input information D31 based on the model information D102, the input input information D31, and model reference information D104 including environment information D33a as needed. The operation information D320 output from the learning model unit 300a is input to the learning model unit 300b.

[0256] Next, the control system 3000e performs a process of generating an operation command D32 using the learning model unit 300b (step S332). In step S332, the learning model unit 300b (more specifically, the model control unit 101) generates and outputs the operation command D32 corresponding to the operation information D320 based on the model information D102, the input operation information D320, and the model reference information D104 including the device information D33 as needed.

[0257] The subsequent processing may be the same as that of the other control systems according to this embodiment.

[0258] As described above, according to this modified example, the input information D31 input by the user 1 can be modified to more general or specific content by absorbing differences in linguistic expression and / or differences in perception of events, and then an operation command D32 can be generated, thereby further improving the functionality of the tasks related to operating the target device 2.

[0259] Note that even in the configuration shown in Modification 3-4, the learning model unit 300 can generate operation commands D32 that average out differences in linguistic expressions and / or differences in recognition of events, based on the environment information D33a, the device information D33, and the model reference information D104 including past operation history, etc. However, according to this modification, the roles of the learning model can be clearly divided into absorbing differences in expression and converting them into operation commands, so that the learning model can be specialized for learning, and a compact design can be achieved, such as by suppressing the scale of learning.

[0260] Embodiment 4 Next, a fourth embodiment will be described. In this embodiment, an example will be described in which a learning model is used to support work related to monitoring a certain work situation.

[0261] For example, consider monitoring anomalies in a factory automation (FA) system that includes control devices such as robots and PLCs. For example, if there is an obvious installation error in the workpiece that the control device targets, existing rule-based monitoring algorithms can handle the error. However, there are also cases where a slight installation error triggers an anomaly to be discovered in a later process. In such cases, even if an analysis is performed using the anomaly detection as a trigger, it is difficult to accurately grasp the situation and identify improvement methods. In this way, it is possible to imagine cases where a relatively small defect spreads to become a major anomaly, and the circumstances surrounding the occurrence do not match existing rules, making it difficult to determine the cause.

[0262] Also, for example, consider monitoring distribution items such as cardboard cases in a distribution center. In this case, the logistics system manages the individual distribution objects by reading the barcode attached to the distribution objects with a barcode reader while the distribution objects are transported on a belt conveyor. In this type of individual management, the orientation of the distribution object may change during transport, making it impossible to read the barcode, and the existence of the distribution object may disappear. In this case, conventional logistics systems can trigger an anomaly detection, such as the disappearance of a logistics target, and record video footage of the events before and after the incident. However, simply recording the video footage makes it difficult for User 1 to accurately and quickly determine the situation and how to improve it from the video footage.

[0263] Therefore, by using a learning model to support the work involved in monitoring these types of work environments, we aim to improve the efficiency and performance of the monitoring work.

[0264] Fig. 23 is a configuration diagram illustrating an example of a control system 4000 according to the fourth embodiment. The control system 4000 illustrated in Fig. 23 is a control system for monitoring a specific work situation using a learning model, and includes a sensor 5, a learning model unit (first learning model unit) 400a, a learning model unit (second learning model unit) 400b, a device information storage unit 410 (referred to as device information DB in the figure), a model interface 6 (referred to as model IF in the figure), and a display (output unit) 7.

[0265] 23 shows a case where the sensor 5, the learning model unit 400a, the learning model unit 400b, the device information storage unit 410, the model interface 6, and the display 7 are provided inside the control system 4000. However, this is not limiting, and the components other than the model interface 6 may be provided outside the control system 4000.

[0266] The sensor 5 acquires data indicating the status of the work to be monitored. Hereinafter, the data acquired by the sensor 5 will be referred to as sensor data. The sensor data may be, for example, image (including video) data capturing the status of the work to be monitored. The sensor data may also be, for example, audio data recording the status of the work to be monitored. The sensor data may also be, for example, measurement data capturing the status of the position of a person or object performing the work to be monitored.

[0267] It is assumed that the sensor 5 always acquires sensor data, but may also acquire sensor data based on a trigger provided by a person or another monitoring system. The sensor data acquired by the sensor 5 is input to the learning model unit 400a as input information D41. The sensor data itself, which is the input information D41, may also be provided by a person or another monitoring system. In such a case, the sensor 5 can be omitted.

[0268] For example, when the control system 4000 is applied to a logistics system, the sensor 5 may be a camera or other device. Examples of the camera include a camera (line sensor) installed on the line along which the logistics object is transported, or a camera (entrance / exit sensor) installed at the entrance / exit gate of the logistics center. Examples of other devices include a barcode reader that reads a barcode attached to the logistics object, a sensor that can acquire electrical information (current, voltage, etc.) of a device installed on the line or in its vicinity, or the target device 2 itself. Examples of devices installed on the line or in its vicinity include various sensors or a motor that drives a belt conveyor.

[0269] When the sensor 5 is a camera, the sensor data is image data. Hereinafter, this sensor data may also be referred to as first sensor data. If the sensor 5 is another device, the sensor data is, for example, data read by a barcode reader, electrical information acquired by a sensor capable of acquiring electrical information, or output data from the target device 2. Hereinafter, this sensor data may also be referred to as second sensor data. In other words, the second sensor data is sensor data other than image data. Furthermore, the sensor 5 may output a combination of the first sensor data and the second sensor data.

[0270] When input information D41 is input, the learning model unit 400a outputs an analysis result D42a. When input information D41 is input, for example, the learning model unit 400a outputs an analysis result D42a based on model information D102. The configuration of the learning model unit 400a may be basically the same as that of the learning model unit 100 in the first embodiment.

[0271] In this embodiment, the learning model unit 400a is a model and its operating environment configured to output an analysis result D42a corresponding to input information D41 when input information D41 is received. Alternatively, the learning model unit 400a may be a model and its operating environment configured to generate and output an analysis result D42a when input information D41 is received, based on the input information D41, equipment information D43, and other information that can be referenced by the learning model unit 400a (such as model reference information D104). Here, the learning model unit 400a may refer to and use information related to the monitored work as the model reference information D104. The information related to the monitored work may be information indicating, for example, the location, person, object, procedure, conditions, etc., where the work is to be performed. For example, the learning model unit 400a may use, as the model reference information D104, a digitized version of a manual describing the conditions, installation environment, operating procedures, etc., of the equipment used in the work.

[0272] In this embodiment, the input information D41 includes information indicating the status of the work to be monitored. The work to be monitored includes one or more tasks performed by humans or machines. The input information D41 may be, for example, a measurement value, an image, a sound, or a combination thereof indicating the status of the work to be monitored. The input information D41 may be, for example, a measurement value, an image, a sound, or a combination thereof indicating the status of multiple tasks to be monitored. The input information D41 may also include information indicating the status of tasks performed consecutively over time. In this case, the input information D41 may be time-series data with a predetermined data structure including the measurement value, image, sound, or a combination thereof indicating the status as described above. The way in which the work status is represented is assumed to match the input format of the model used by the learning model unit 400a. However, this does not apply if error processing, correction processing, or conversion processing is included in the upstream stage of the learning model unit 400a.

[0273] The analysis result D42a includes information indicating a situation analysis result obtained by analyzing the work situation indicated by the input information D41. The information indicating the situation analysis result may be information indicating objects (environment) present in the work situation indicated by the input information D41 and / or events occurring. The information indicating the situation analysis result may be information indicating an interpretation of the work situation indicated by the input information D41. The analysis result D42a may be, for example, text indicating an interpretation of the work situation indicated by the input information D41. Furthermore, the analysis result D42a may be, for example, text indicating an interpretation of a part of the work situation indicated by the input information D41 that differs from the normal situation, focusing on the part. The analysis result D42a may be in a format other than text. The analysis result D42a may be in any format as long as it is described in a predetermined format that can be interpreted by the subsequent learning model unit 400b. For example, the analysis result D42a may be in a format such as text, image, audio, or a combination thereof.

[0274] Examples of interpreting a work situation include expressing objects present in the work situation using their attributes, expressing events occurring in the work situation in a predetermined syntactic format such as 5W1H or 7W1H, or further summarizing them after concretizing them in such a manner. Other examples include breaking down the work being performed in the work situation into multiple perspectives and expressing each perspective, or, if the work being performed in the work situation includes multiple subtasks or steps, breaking down the target work into subtasks or steps and explaining each subtask or step. The analysis result D42a can be said to be the result of concretizing, subdividing, and / or extracting specific points from the work situation represented by the input information D11, and then expressing it in a predetermined format. In this way, the analysis result D42a expresses the work situation in an easy-to-understand and organized manner.

[0275] When the analysis result D42a is input, the learning model unit 400b outputs the analysis result D42b. For example, when the analysis result D42a is input, the learning model unit 400b outputs the analysis result D42b based on the model information D102. The configuration of the learning model unit 400b may be basically the same as that of the learning model unit 100 in the first embodiment.

[0276] In this embodiment, the learning model unit 400b is a model and its operating environment configured to output an analysis result D42b corresponding to the analysis result D42a when the analysis result D42a is input. Also, the learning model unit 400b may be a model and its operating environment configured to generate and output the analysis result D42b based on the analysis result D42a, device information D43, and / or information that can be referenced in the learning model unit 400b (such as model reference information D104) when the analysis result D42a is input.

[0277] The analysis result D42b includes information indicating a method for improving the work situation derived from the analysis result of the work situation by the learning model unit 400a. The information indicating a method for improving the work situation may be information indicating a recovery method for restoring an abnormal state to normal, or may be information indicating a method for solving a problem when a problem occurs in the environment (work environment) where the work to be monitored is being performed, such as when a person is in trouble or a device is stopped.

[0278] Furthermore, when the improvement method to be generated includes multiple processes, the learning model unit 400b may add information indicating the order of the multiple processes to the improvement method.

[0279] The information indicating the improvement method may be, for example, text, images, or audio indicating the method, or may be a control command (e.g., command, control signal, control code, etc.) for the device (target device 2) on which the method is to be implemented, a procedure manual describing the method, a sequence diagram, source code, execution code, or a controller command for causing a controller to execute the method. The information indicating the improvement method may be text, images, audio, data written in a predetermined design language, control description (including source code and information written in a predetermined programming platform language), information written in other platform languages, control commands (including control commands, control signals, control codes, and controller commands), execution code, or a combination of two or more of these elements. Examples of predetermined design languages ​​include, but are not limited to, UML (Unified Modeling Language).

[0280] Hereinafter, the learning model unit 400a may be referred to as the first learning model unit 400, and the analysis result D42a may be referred to as the first analysis result D42. Also, the learning model unit 400b may be referred to as the second learning model unit 400, and the analysis result D42b may be referred to as the second analysis result D42.

[0281] As already explained, the analysis result D42a includes information indicating the situation analysis result of the work situation indicated by the input information D41. Therefore, the learning model unit 400b may be a model configured to output the analysis result D42b corresponding to the situation analysis result indicated by the analysis result D42a, and its operating environment. Here, if the information indicating the situation analysis result is text explaining the work situation indicated by the input information D41, the learning model unit 400b may be a model configured to output the analysis result D42b corresponding to the text explaining the work situation, and its operating environment.

[0282] The learning model unit 400b may determine whether the work status is abnormal or normal from the analysis result D42a, and output the analysis result D42b when it is determined that the work status is abnormal. Alternatively, the learning model unit 400a may determine whether the work situation is abnormal or normal from the analysis result D42a, and output the analysis result D42a when it determines that the work situation is abnormal. That is, in this case, the learning model unit 400b may output the analysis result D42b when the analysis result D42a is output from the learning model unit 400a. Alternatively, the learning model unit 400b may include information indicating that normal control is to be performed on the target device 2 as the analysis result D42b when the work situation is normal.

[0283] When the sensor data is only the first sensor data (image data), for example, the learning model unit 400a or the learning model unit 400b may determine whether the data is normal or abnormal using a trained CNN model. Furthermore, when the sensor data is only the second sensor data (sensor data other than image data), for example, the learning model unit 400a or the learning model unit 400b may determine whether the sensor data is normal or abnormal using a preset table. The table is a table that can determine whether the sensor data is normal or abnormal based on a combination of predetermined parameters. Furthermore, when the sensor data is the first sensor data and the second sensor data, for example, the learning model unit 400a or the learning model unit 400b may determine whether the data is normal or abnormal using both of the above methods.

[0284] Furthermore, the learning model unit 400a may include information indicating whether the work situation is normal or abnormal in the analysis result D42a. Furthermore, the learning model unit 400b may include information indicating whether the work situation is normal or abnormal in the analysis result D42b.

[0285] Furthermore, when data indicating a prompt is input from the display device 7 via the model interface 6, the learning model unit 400b may regenerate the analysis result D42b based on this prompt.

[0286] In Figure 23, the learning model unit 400a and the learning model unit 400b are shown separately, but this is not limited to this, and the learning model unit 400a and the learning model unit 400b may be configured as a single unit, and the same effect as above will be achieved. On the other hand, as will be described later, it is preferable in terms of accuracy that the learning model unit 400a that interprets the situation and the learning model unit 400b that generates the improvement method are separate units.

[0287] The device information storage unit 410 and the device information D43 are handled in basically the same way as the device information storage unit 110 and the device information D13 in the first embodiment. In this embodiment, the device information storage unit 410 stores device information D43, which is information about devices related to the work to be monitored as the target device 2. Here, the devices related to the work broadly include devices required for the situation analysis and deriving the improvement method described above. More specifically, the devices include not only devices used in the work but also devices that affect the person or device performing the work. More specifically, the devices that affect the person or device performing the work may be devices that directly or indirectly bring about changes to the person or device performing the work. Examples of devices that affect the work include devices directly used in the work (including various machines such as processing machines and conveyors, as well as tools such as workbenches and tools), devices that control the devices directly used in the work (such as power supplies, relays, switches, and controllers), and devices that bring about changes in the work environment (such as lighting equipment, air conditioning equipment, vacuum cleaners, and purifiers).

[0288] The device information D43 is used, for example, as additional information when the learning model unit 400a and / or the learning model unit 400b outputs the model output data D103 (the analysis result D42a, the analysis result D42b). Hereinafter, in this embodiment, information particularly indicating the state of the target device 2 may be referred to as state information D45.

[0289] In this embodiment, the learning model unit 400a may be an image learning model such as a VLM that receives an image as an input and obtains an output result, and its operating environment. The learning model unit 400a may also be, for example, a multimodal model that receives a natural language and an image as an input and obtains an output result, and its operating environment. The learning model unit 400b may also be, for example, a language learning model such as an LLM that receives a natural language as an input and obtains an output result, and its operating environment. In this case, the input information D41 may be input in the form of text data, image data, a combination of text data and image data, or a data format convertible thereto (such as audio data or a video that is a combination of audio data and image data). The learning model used in this embodiment is not limited to the above-mentioned models.

[0290] The model interface 6 is an interface that receives model output data (analysis results D42a and D42b) from the learning model unit 400a and the learning model unit 400b and outputs the data to a predetermined output destination. The model interface 6 may be, for example, an interface that converts the model output data output from the learning model unit 400a and the learning model unit 400b into data that matches a predetermined output destination and outputs the data. The model interface 6 may be provided, for example, as an example of the output unit 103 described above. In this embodiment, output destinations of the model interface 6 include the target device 2 and the display 7.

[0291] For example, the model interface 6 may output result information D44a indicating the situation analysis result included in the analysis result D42a and the improvement method included in the analysis result D42b to the display device 7, and may also output result information D44b indicating the improvement method included in the analysis result D42b to the target device 2. At this time, the model interface 6 may extract some data from the analysis result D42a and / or the analysis result D42b, convert it into a data format that matches the output destination, and then output it as result information D44a and result information D44b.

[0292] 23, the target device 2 and the display device 7 are shown as output destinations of the model interface 6, but the output destinations of the model output data are not limited to these. For example, if a method for improving the situation indicated by the model output data to be output includes information indicating control over the target device 2, the model interface 6 can output the model output data or information indicating the method directly to the target device 2 as the target for implementing the method, or it can also output the model output data or information indicating the method to a conversion device (not shown) that converts the model output data or information indicating the method into information acceptable to the target device 2. The conversion device may be, for example, the control system 1000 of the first embodiment that converts input information into a control description or execution code that can be recognized by the target device 2.

[0293] Furthermore, the model interface 6 itself may have the function of a conversion device. For example, the model interface 6 may have the function not only of output control of model output data, but also of converting the improvement method output by the learning model unit 400b into code executable by an interpreter, and outputting the converted code or controlling equipment based on the code. The model interface 6 may also have the function of controlling the processing flow, such as immediately executing a process that is highly urgent when the improvement method includes such a process. The model interface 6 may also have the function of transmitting prompts input via the display 7, such as responses to the proposed method displayed on the display 7, to the learning model unit 400b.

[0294] The model interface 6 may also have the functions of the output confirmation unit 202 and the correction confirmation unit 203 described above. For example, the model interface 6 may determine the urgency of the analyzed situation, and if it determines that the urgency is not high, it may confirm the appropriateness of the improvement method by inquiring of a supervisor or using a simulator or the like, and if the improvement method is not appropriate, it may communicate this to the learning model unit 400b and prompt it to output the improvement method again. In this case, the model interface 6 may issue supplemental information D48 for the model input data of the target learning model unit 400b.

[0295] The model interface 6 may perform normal control on the target device 2 when the working situation is normal. Alternatively, the control system 4000 may be provided with an execution module separate from the model interface 6. In this case, the execution module performs normal control on the target device 2 when the work situation is normal. The execution module may determine whether the work situation is normal or not based on the input information D41. On the other hand, when the work situation is abnormal, the model interface 6 takes over from the execution module and performs control on the target device 2 based on the analysis result D42b.

[0296] In this embodiment, the input information D41 corresponds to the model input data D101 of the learning model unit 400a. The analysis result D42a corresponds to the model output data D103 of the learning model unit 400a. The analysis result D42a corresponds to the model input data D101 of the learning model unit 400b. The analysis result D42b corresponds to the model output data D103 of the learning model unit 400b. The learning model unit 400a (particularly, the model control unit 101) may be configured, for example, to receive the input information D41, and output the analysis result D42a corresponding to the input information D41 based on the model information D102 and, if necessary, the model reference information D104. The learning model unit 400b (particularly, the model control unit 101) may be configured, for example, to receive the analysis result D42a, and output the analysis result D42b corresponding to the analysis result D42a based on the model information D102 and, if necessary, the model reference information D104.

[0297] In such a case, the model generation unit 107 provided corresponding to the learning model unit 400a may perform machine learning using model learning data D105 including candidates for input information D41 that may be input to the model control unit 101 to generate or update the model information D102, or may perform machine learning using model learning data D105 including candidates for input information D41 that may be input to the model control unit 101 and candidates for corresponding analysis results D42a to generate or update the model information D102. The model generation unit 107 provided corresponding to the learning model unit 400b may perform machine learning using model learning data D105 including candidates for analysis results D42a that may be input to the model control unit 101 to generate or update the model information D102, or may perform machine learning using model learning data D105 including candidates for analysis results D42a that may be input to the model control unit 101 and candidates for corresponding analysis results D42b to generate or update the model information D102.

[0298] Although not shown, in this embodiment, status information D45 and / or feedback information D46 may be acquired from the model output data D103 of the learning model unit 400a and the learning model unit 400b and / or the output destination of information generated based on the model output data D103. The control system 4000 may output the acquired status information D45 and / or feedback information D46 to the user 1, the learning model unit 400a, the learning model unit 400b, or another device (not shown) as information indicating the control result. The control system 4000 may also be configured to return a query D47 to the user 1 when the input information D41 contains unclear or uncertain information. The control system 4000 may also generate supplemental information D48 for the input / output data of the learning model unit 400a and the learning model unit 400b based on the acquired status information D45 and / or feedback information D46, and issue the supplemental information D48 to the user 1, the learning model unit 400a, the learning model unit 400b, or another device (not shown). The handling of the status information D45, feedback information D46, inquiry D47, and supplemental information D48 may be basically the same as in embodiment 1. Here, information may be output to the user 1 via, for example, the display device 7 or an input / output interface (not shown) included in the information processing device 10.

[0299] The control system 4000 may further include a status acquisition unit 430 (not shown) that acquires the status information D45 and / or the feedback information D46 and issues supplemental information D48 as necessary. The status acquisition unit 430 is similar to the status acquisition unit 130 in the first embodiment.

[0300] In this embodiment, there is also no particular limitation on the target device 2. Note that, although it is assumed that the target device 2 is a device that can actually be controlled by receiving the analysis result D42b, this does not necessarily apply when the above-mentioned conversion device is included between the target device 2 and the target device 2.

[0301] The display 7 may output sound in addition to the display. Also, in FIG. 23, the output unit 7 is shown as a display device 7 that displays or displays and outputs audio, but is not limited to this, and the output unit 7 may be an audio output device that outputs audio.

[0302] In this embodiment, the input information D41 received by the control system 4000 can be considered to be information relating to the situation in the work environment (here, the situation in the environment where the monitoring work is performed). Therefore, the input information D41 received by the control system 4000 can be considered to be an example of first information indicating the situation in the work environment. Furthermore, the analysis result D42a and the analysis result D42b can be considered to be information used for the work (monitoring work) in response to such input information D41. Hereinafter, the analysis result D42a and / or the analysis result D42b output to a predetermined output destination from the operating environment of the learning model to which model input data based on the input information D41 is input will be referred to as second information.

[0303] Next, a description will be given of the operation of the control system 4000 of this embodiment. Fig. 24 is a flowchart showing an example of the operation of the control system 4000. In addition, the specific example shown below shows a case where the control system 4000 is applied to a logistics system.

[0304] 24, first, the control system 4000 receives input information D41 (step S410). For example, the input unit 102 or the input processing unit 201 described above may receive the input information D41. The received input information D41 is input to the learning model unit 400a as model input data D101.

[0305] Next, the control system 4000 performs a process of generating an analysis result D42a using the learning model unit 400a (step S411). In step S411, the learning model unit 400a (more specifically, the model control unit 101) outputs an analysis result D42a corresponding to the input information D41 based on the model information D102, the input information D41, and model reference information D104 including device information D43 as needed. The learning model unit 400a may generate the analysis result D42a of text data from the input information D41 using, for example, a learning model capable of generating text data.

[0306] For example, if the data input to the learning model unit 400b is only text data, the learning model unit 400a needs to generate text data as the analysis result D42a.

[0307] Here, when the sensor data input to the learning model unit 400a is only the second sensor data (sensor data other than image data), the learning model unit 400a may generate text data indicating the second sensor data itself as the analysis result D42a. Note that the learning model unit 400a may include text data indicating whether the work situation is normal or abnormal in this analysis result D42a. Alternatively, the learning model unit 400a may generate, as the analysis result D42a, text data indicating only the data determined to be abnormal among the second sensor data. Note that the learning model unit 400a may include, in the analysis result D42a, text data indicating that the work situation is abnormal.

[0308] Furthermore, the learning model unit 400a may generate text data by changing the sentence structure or order of description for the information determined to be abnormal based on the importance of terms, correlation in chronological order, correlation in word order, etc. For example, if the learning model unit 400a obtains three abnormalities as work conditions, namely, "the current of the belt conveyor has increased," "the feed speed of the belt conveyor is 0," and "there is no output from the entry / exit sensor," it may generate text data such as "there is no output from the entry / exit sensor, the feed speed of the belt conveyor has become 0, and the current has increased," taking into consideration the chronological order, etc.

[0309] In addition, if the sensor data input to the learning model unit 400a is only the first sensor data (image data), the learning model unit 400a may generate text data after recognizing what is present mainly from images taken by a camera using object recognition technology such as CNN. For example, if the learning model unit 400a recognizes a "suspicious person" from an entry / exit sensor and recognizes a "suspicious person," "in front of the barcode reading position," and "numerous cardboard cases" from a line sensor, it may generate text data such as "suspicious person, in front of the barcode reading position, numerous cardboard cases."

[0310] Furthermore, when the sensor data input to the learning model unit 400a is the first sensor data and the second sensor data, the text data may be generated by both of the above methods. For example, in the above example, the learning model unit 400a may generate text data such as "there was no output from the entry / exit sensor, a suspicious person entered, the suspicious person placed a large number of cardboard boxes in front of the barcode reading position, the conveyor belt speed became 0, and the current increased."

[0311] Furthermore, when the analysis result D42a input to the learning model unit 400b is text data and images, the learning model unit 400a may include, as the analysis result D42a, not only the text data shown above, but also images such as, for example, "a video of a suspicious person breaking in" or "a video or clipped image of a suspicious person placing a large number of cardboard boxes at XX coordinates."

[0312] In step S411, the pre-processing unit 105 and / or the post-processing unit 106 of the learning model unit 400a may further perform the above-mentioned processing.

[0313] The analysis result D42a output from the learning model unit 400a is input to the learning model unit 400b. In addition, the analysis result D42a output from the learning model unit 400a is input to the learning model unit 400b and the model interface 6. The analysis result D42a output from the learning model unit 400a may be input to the model interface 6 via the learning model unit 400b. In this case, the learning model unit 400b may output model output data D103 including the analysis result D42a and the analysis result D42b.

[0314] Next, the control system 4000 performs a process of generating an analysis result D42b using the learning model unit 400b (step S412). In step S412, the learning model unit 400b (more specifically, the model control unit 101) outputs an analysis result D42b corresponding to the analysis result D42a based on the model information D102, the input analysis result D42a, and model reference information D104 including device information D43 as needed. The learning model unit 400b may generate a binary data analysis result D42b from the input analysis result D42a using, for example, a learning model capable of generating text data. Furthermore, the learning model unit 400b may generate a text data and binary data analysis result D42b from the input analysis result D42a using, for example, a learning model capable of generating text data and binary data.

[0315] Here, if the analysis result D42a input to the learning model unit 400b is only text data containing multiple words or numeric data, etc., the learning model unit 400b extracts text data of an improvement method with a high probability of improvement that corresponds to the words or numeric data, etc. that are determined to be abnormal.

[0316] If the extracted text data of the improvement method includes multiple processes, the learning model unit 400b may extract the parts related to each process and add information indicating the order of the processes. For example, if the extracted text data of the improvement method includes three processes, such as "stop the belt conveyor," "move the cardboard case in front of the barcode reader," and "restart the belt conveyor," the learning model unit 400b extracts the portion related to each process. Then, based on the portion related to each process, the learning model unit 400b takes into consideration the order of the improvement measures using chronological order or correlation between words, etc., and generates text data with the order of the processes added, such as "1. stop the belt conveyor," "2. move the cardboard case in front of the barcode reader," and "3. restart the belt conveyor."

[0317] Furthermore, the learning model unit 400b may convert the generated text data into an image and use it as the analysis result D42b. In the above example, the learning model unit 400b may convert the generated text data into images such as frame-by-frame images or videos corresponding to "1. Belt conveyor stops," "2. Cardboard case is moved in front of the barcode reader," and "3. Belt conveyor is restarted," and use these as the analysis result D42b. Furthermore, the learning model unit 400b may use both the text data and the image converted from the text data as the analysis result D42b.

[0318] The above-mentioned process of converting text data into images does not necessarily have to be performed by the learning model unit 400b, but may be performed by providing a separate learning model unit in the control system 4000 that is different from the learning model unit 400b.

[0319] Furthermore, the learning model unit 400b may convert the generated text data into a program corresponding to the programming language of the model interface 6, and use the converted program as the analysis result D42b. In the above example, the learning model unit 400b may convert the generated text data into a program corresponding to the programming language of the model interface 6, which corresponds to "1. Stop the belt conveyor," "2. Move the cardboard case in front of the barcode reader," and "3. Restart the belt conveyor," and use this as the analysis result D42b.

[0320] Furthermore, the learning model unit 400b may convert the generated text data into a program that conforms to the communication protocol for controlling the target device 2 synchronously from the model interface 6 for highly urgent processes such as "1. Stop the conveyor belt" in order to improve the processing speed and execute the process immediately. In this case, the model interface 6 controls the target device 2 to immediately execute "1. Stop the belt conveyor."

[0321] Furthermore, when the generated text data includes multiple processes, the learning model unit 400b may include in the analysis result D42b an instruction or code for displaying the text data indicating the multiple processes in chronological order on the display 7. Note that since this is not as urgent as synchronous communication, it may be output to the display 7, for example, by asynchronous communication within the communication protocol for controlling the target device 2 from the model interface 6 by synchronous control. This process may be performed by the model interface 6 instead of the learning model unit 400b.

[0322] Furthermore, when the learning model unit 400b generates a program, it may use a simulator to verify whether the program has any problems. This process may be performed by the model interface 6 instead of the learning model unit 400b.

[0323] In addition, if the information input to the learning model unit 400b is text data and images, the learning model unit 400b may include, for example, instructions or code for displaying an image indicating an abnormality on the display device 7 in the analysis result D42b.

[0324] Furthermore, in the program of the above example, if the model interface 6 is notified that the cardboard case has been moved in front of the barcode reader, the belt conveyor can be restarted. However, there may be cases where the analysis by the learning model unit 400b is incomplete, such as when the fact that a suspicious person was dealt with in the factory has not been resolved. To deal with such a situation, for example, if the analysis result D42a indicates that an abnormal condition such as the intrusion of a suspicious person has not been eliminated, the learning model unit 400b may continue to output the analysis result D42b including "1. Stop the belt conveyor" in accordance with the communication protocol for controlling by synchronous control. This makes it possible to stop the operation of the release program, such as restarting the belt conveyor, until the abnormal condition such as the intrusion of a suspicious person is eliminated.

[0325] In step S412, the pre-processing unit 105 and / or the post-processing unit 106 of the learning model unit 400b may further perform the above-mentioned processing.

[0326] The analysis result D42b output from the learning model unit 400b is input to the model interface 6, for example.

[0327] The model interface 6 controls the target device 2 and / or displays information on the display device 7 based on the analysis results by the learning model unit 400a and the learning model unit 400b (step S413). In step S413, for example, the model interface 6 outputs information based on the analysis results D42a and D42b to a predetermined output destination. For example, the model interface 6 outputs result information D44a indicating the situation analysis result and an improvement method to the display device 7 based on the analysis results D42a and D42b, and outputs result information D44b indicating the improvement method based on the analysis results D42b to the target device 2.

[0328] The result information D44a may be, for example, text and audio indicating the situation that occurred in the work environment and how to improve it, while the result information D44b may be, for example, text or a control signal indicating how to improve it.

[0329] As a result, the display device 7 displays the situation analysis result indicated by the analysis result D42a and the improvement method indicated by the analysis result D42b based on the result information D44a, and the target device 2 implements the improvement method indicated by the analysis result D42b based on the result information D44b. Information input to the display device 7 and the target device 2 may be input directly from the control system 4000 (more specifically, the model interface 6), or may be input indirectly via a communication network, other devices (servers, various conversion devices, etc.), or manually.

[0330] If there is a process with a high degree of urgency among the improvement methods indicated by the analysis result D42b, the model interface 6 may control the target device 2 to immediately execute that process. For example, the model interface 6 controls the target device 2 to immediately execute "1. Stop belt conveyor" because it is a highly urgent process. At this time, for example, the model interface 6 outputs a belt conveyor stop signal to the target device 2. In this case, the target device 2 stops the belt conveyor in response to the belt conveyor stop signal.

[0331] Furthermore, the model interface 6 may perform display control on the display device 7 to allow the user 1 to confirm the validity of the improvement method indicated in the analysis result D42b. This confirmation is considered to be particularly effective when the improvement method includes processes other than those with high urgency. For example, the model interface 6 may control the display of the display 7 to check the appropriateness of "2. Move the cardboard case in front of the barcode reader" and "3. Restart the belt conveyor" in light of the situation. This allows the user 1 to determine the appropriateness of the improvement method displayed on the display 7. If the displayed improvement method is not appropriate and the user 1 inputs data indicating a prompt via the display 7, the model interface 6 may output data indicating the prompt to the learning model unit 400b to prompt the user 1 to regenerate the improvement method.

[0332] When there is a change in the state of the target device 2 due to the target device 2 being controlled or when there is feedback from the output destination, the control system 4000 may acquire state information D45 and feedback information D46 (step S414). Note that the processing of step S414 is not essential and may be omitted as appropriate.

[0333] The control system 4000 may repeat the processes of steps S410 to S414 multiple times (for example, until a desired state is achieved in the target work environment).

[0334] As described above, according to this embodiment, understanding the situation and acquiring improvement methods are carried out in two stages using different learning models, thereby improving the accuracy of the final product and, as a result, making it possible to improve the efficiency of work related to monitoring the work situation.

[0335] For example, when assessing a situation, it is important to broadly detect abnormalities in the work environment, such as "something strange has occurred." On the other hand, when determining how to improve something, specific information such as "stop this machine, move the workpiece to point A, return the machine to state B, and then restart it" is important.

[0336] When the information to be extracted, i.e., the target information, has different levels of abstraction, there is a concern that if a single learning model is used to learn and extract both types of information, the accuracy of the output results will decrease. In particular, when acquiring improvement methods, it is necessary to present specific methods based on knowledge and information about the work environment. In such cases, by separating the learning models and providing appropriate domain knowledge (environmental information), it is possible to more reliably improve the output accuracy.

[0337] Furthermore, when attempting to obtain solutions for the different tasks of grasping the situation and acquiring improvement methods using a single learning model, the problem of hallucination is likely to become more pronounced. This is because a function that adjusts the solution for one task (grasping the situation) so that the solution for the other task (acquiring improvement methods) appears more plausible may implicitly operate within the model algorithm. This embodiment is also effective in addressing such hallucination problems. That is, by separating the learning models corresponding to the two tasks of grasping the situation and acquiring improvement methods, the modalities entering each learning model can be suppressed, thereby reducing the magnitude of hallucination, thereby improving the accuracy of the final product.

[0338] Furthermore, in this embodiment, the analysis results D42a and D42b, which are the output results of the learning model unit 400a and the learning model unit 400b, can be put into words and displayed on the display 7, so that a person can check the contents to suppress hallucination and more reliably realize a method for improving the situation.

[0339] The control system 4000 of this embodiment can be applied to, for example, monitoring of a control system for equipment in a factory, and monitoring of distribution objects in a distribution system.

[0340] In the control system 4000 according to the fourth embodiment, for example, the following can be realized.

[0341] When starting up the line first thing in the morning, some target devices 2 take a long time to start up. In such a case, for example, the learning model unit 400a detects that all or some of the target devices 2 are not powered on, and transmits text data indicating this to the learning model unit 400b as the analysis result D42a. For example, the learning model unit 400b can learn information about start-up times from the device information storage unit 410, etc., and can display the start-up order of all or some of the target devices 2 as the analysis result D42b from the start-up times of each target device 2 calculated backward from the start of operation. Note that in some cases, the start-up order of each target device 2 is specified separately from the start-up times, and so the learning model unit 400b can learn information about the start-up order from the device information storage unit 410, etc., and can display the start-up order taking this into consideration as the analysis result D42b.

[0342] The shutdown sequence after the line has finished operating is also the same as the startup sequence.

[0343] In addition to scheduled line stoppages, unexpected line stoppages, known as "choco-tei," can occur during line operation due to a variety of factors, including misalignment of objects on the line, processing errors, identification errors in barcodes or colors, inspection errors to measure quality, worker mistakes, work delays, or emergency stoppages due to breakdowns in manufacturing or inspection equipment. In response to this, the use of IoT in logistics systems has made it possible to visualize the situation to a certain extent, and it is now possible to instantly display the location of the line where the error occurred, the cause of the momentary stop of the target device 2, whether it was an emergency stop caused by a human being or an emergency stop caused by the manufacturing equipment, etc. However, it can be difficult to instantly identify the cause unless you are familiar with the line.

[0344] Regarding this momentary stop, if it is merely a case of "the conveyor belt speed (becoming) 0 and the current (increasing)," the situation can be understood from the abnormal information in the list display of the operation status, even with the display function of IoT in the logistics system. However, in this state, it is not possible to identify the cause, and it is not possible to understand whether the cause is a blockage of something on the line or a problem with the conveyor belt motor. In addition, in complex cases such as when "a large number of cardboard cases are placed in front of the barcode reading position, the conveyor belt speed drops to 0, and the current increases," the display functions of IoT-enabled logistics systems do not convert images into text, so an experienced worker must look at the images and understand the situation from a list of operating conditions. In contrast, the learning model unit 400a can convert information determined to be abnormal into sentences or change the order of description using the importance of terms, correlations in chronological order, or correlations in word order, and generate the analysis result D42a. This has the advantage of allowing the situation and factors to be grasped instantly. Furthermore, this effect is even greater when the learning model unit 400a generates the information from information obtained by multiple measuring instruments and cameras.

[0345] Furthermore, the learning model unit 400a has the advantage of being able to aggregate to some extent factors that can cause short stops, such as misalignment of objects on the line, processing errors, identification errors such as barcodes or colors, inspection errors to measure quality, worker mistakes, work delays, or emergency stops due to failures of manufacturing equipment or inspection equipment. For example, misalignment of an object on the line and identification errors such as barcode or color errors can generally be summarized and instantly displayed as factors related to the misalignment of the object. Furthermore, processing errors or inspection errors that measure quality can be summarized and instantly displayed as factors related to the processing of the object. Furthermore, an emergency stop due to a malfunction of manufacturing equipment or inspection equipment can be summarized and instantly displayed as factors related to the malfunction of the manufacturing equipment or inspection equipment. Furthermore, a worker's work error or work delay can be summarized and instantly displayed as factors related to the work delay. In other words, after a line stop, the analysis result D42a generated by the learning model unit 400a can instantly distinguish between factors that make it easy to restore the line to its original state and factors that make it difficult to restore the line to its original state.

[0346] Furthermore, the learning model unit 400b can extract text data of an improvement method with a high probability of improvement corresponding to each of the words or numeric data determined to be abnormal as described above. The learning model unit 400b can also output a program that matches the programming language of the model interface 6 together with the text data. The learning model unit 400b can also issue instructions to the target device 2 from the corresponding model interface 6. Furthermore, the learning model unit 400b can generate a program for restoring the machine to its original state or issue instructions to the target device 2 depending on whether the cause is easy to restore. In particular, if the cause is related to the processing of the item or a failure of the manufacturing or inspection equipment, the learning model unit 400b determines that it is not appropriate to restore the machine to its original state after issuing an emergency stop command, and it is not appropriate to output a program or instructions for restoring the machine to its original state. On the other hand, if the cause is easy to restore the machine to its original state, the learning model unit 400b generates a program or instructions for restoring the machine to its original state and outputs it to the model interface 6, or executes it if the conditions allow it to operate on the line. This reduces the labor required for operators to operate the machine to restore it to its original state, and shortens the time required to restore the machine to its original state. Furthermore, the learning model unit 400b may use a simulator to verify whether the generated program or command has any problems, and if the verification results show that there are no problems, it may execute it or display the results to indicate the final execution decision to the operator.

[0347] The same is true for information that is not a momentary stop, but is at a warning level: for example, if the case is something like "the conveyor belt speed has decreased and the current has increased," the situation can be understood from the abnormal information in the list of operation status displayed by the display function of IoT in the logistics system. However, in this state, it is not possible to identify the cause, and it is not possible to understand whether the cause is a blockage on the line or a problem with the conveyor belt motor. In addition, in complex cases such as when "a large number of cardboard cases were placed in front of the barcode reading position, causing the conveyor belt speed to decrease and the current to increase," the display functions of IoT-based logistics systems do not convert images into text, so an experienced worker must look at the images and understand the situation from a list of operating conditions. In contrast, the learning model unit 400a can convert information determined to be at the caution level into sentences or change the order of description using the importance of terms, correlations in chronological order, or correlations in word order, and generate the analysis result D42a. This has the advantage of allowing the situation and factors to be grasped instantly. Furthermore, this effect is even greater when the learning model unit 400a generates information from information obtained by multiple measuring instruments and cameras.

[0348] Furthermore, the learning model unit 400a has the advantage of being able to aggregate to some extent factors that warrant caution, such as misalignment of items on the line, increased variation in processing, inspection errors to measure quality, slight delays in work, and wear and tear on manufacturing or inspection equipment.

[0349] Furthermore, the learning model unit 400b can extract text data of an improvement method with a high probability of improvement corresponding to each of the words or number data determined to be at the caution level as described above. Furthermore, the learning model unit 400b can output a program adapted to the programming language of the model interface 6 together with the text data. Furthermore, the learning model unit 400b can issue instructions to the target device 2 from the corresponding model interface 6. In the control system 4000, such measures have the effect of avoiding chocolate stops.

[0350] Variation 4-1. Next, a description will be given of a modified example of the control system 4000. Fig. 25 is a configuration diagram showing an example of a control system 4000a which is a modified example of the control system 4000 according to the present embodiment. Note that the same elements as those in the control system 4000 are given the same reference numerals and description thereof will be omitted.

[0351] Control system 4000a shown in FIG. 25 differs from control system 4000 in that it has two analysis means that analyze and improve the situation using different methods, and switches between the analysis means to be used as appropriate depending on the situation that occurs.

[0352] The control system 4000a shown in Figure 25 includes a first analysis unit 41-1, which uses the above-mentioned learning model unit 400a and learning model unit 400b to analyze the situation and obtain improvement methods, as well as a second analysis unit 41-2, a switching unit 42, and an output switching switch 43.

[0353] The second analysis unit 41-2 may be any means for analyzing the situation and acquiring an improvement method for the input information D41 using a method different from that used by the first analysis unit 41-1. As an example, the second analysis unit 41-2 may be a means for analyzing the situation and acquiring an improvement method based on rules. For example, when the input information D41 is input, the second analysis unit 41-2 may determine whether the input information D41 matches a predetermined abnormality pattern, and if the input information D41 matches any of the abnormality patterns, acquire an improvement method corresponding to the abnormality pattern. The second analysis unit 41-2 outputs an analysis result D42c including at least an improvement method for the situation.

[0354] The analysis result D42c may include, for example, information corresponding to the result information D44a and information corresponding to the result information D44b described above. In this modification, the analysis result D42c includes at least the result information D44b indicating the improvement method obtained by the second analysis unit 41-2.

[0355] In this modification, the second analysis unit 41-2 may be realized as an internal execution module by being implemented in, for example, a PLC, an information processing device, or the like arranged in the work environment.

[0356] The switching unit 42 is means for switching the control destination for the input information D41 in accordance with a predetermined condition. In this modification, the switching unit 42 switches the control destination for the input information D41 between the first analysis unit 41-1 and the second analysis unit 41-2. The switching unit 42 may switch the control destination for the input information D41, for example, depending on whether the input information D41 matches an existing rule. In this case, the switching unit 42 may switch the control destination by switching the output destination of the input information D41 to the second analysis unit 41-2 when the input information D41 matches the existing rule, and by switching the output destination of the input information D41 to the first analysis unit 41-1 when the input information D41 does not match the existing rule.

[0357] The switching unit 42 may switch the control destination for the input information D41 in accordance with, for example, an instruction from a supervisor. The switching unit 42 may also switch the control destination for the input information D41 based on, for example, time, the work content, or the presence or absence of a supervisor. The switching unit 42 may also switch the control destination for the input information D41 based on, for example, whether an abnormality has occurred in the work environment. The presence or absence of an abnormality in the work environment may be determined based on, for example, whether an abnormality signal has been generated. For example, the switching unit 42 may switch the control destination for the input information D41 to the first analysis unit 41-1 when an abnormality has occurred. The switching unit 42 may also switch the control destination for the input information D41 based on, for example, the degree or urgency of the abnormality occurring in the work environment.

[0358] For example, the switching unit 42 may normally set the control destination for the input information D41 to the second analysis unit 41-2 such as a PLC. If the second analysis unit 41-2 detects an abnormal state during processing, the second analysis unit 41-2 may stop the target device 2 in the abnormal state, and then the switching unit 42 may switch the control destination for the input information D41 to the first analysis unit 41-1. For example, if something gets caught in a gear and the load increases, the second analysis unit 41-2, such as a PLC, can detect the overload and stop it. However, only the first analysis unit 41-1, which analyzes camera images, can identify the cause and develop a countermeasure. Therefore, when the second analysis unit 41-2 detects an overload, the control system 4000a may deal with the problem by switching the control destination for the input information D41 to the first analysis unit 41-1.

[0359] In addition, the switching unit 42 may control an output switching switch 43 that switches the connection path (such as a circuit or communication path) connecting the output of the first analysis unit 41-1 or the output of the second analysis unit 41-2 to the target device 2 and display 7 that are to be the output destinations of the analysis results, in conjunction with switching the control destination for the input information D41.

[0360] For example, when the switching unit 42 switches the control destination for the input information D41 to the first analysis unit 41-1, the switching unit 42 may control the output changeover switch 43 to turn on the connection path connecting the output of the first analysis unit 41-1 with the target device 2 and the display 7, and turn off the connection path connecting the output of the second analysis unit 41-2 with the target device 2 and the display 7. Similarly, when the switching unit 42 switches the control destination for the input information D41 to the second analysis unit 41-2, the switching unit 42 may control the output changeover switch 43 to turn on the connection path connecting the output of the second analysis unit 41-2 with the target device 2 and the display 7, and turn off the connection path connecting the output of the first analysis unit 41-1 with the target device 2 and the display 7.

[0361] Fig. 26 is a flowchart showing an example of operation of this modified example. In the example shown in Fig. 26, when the control system 4000a receives input information D41 in step S410, the switching unit 42 switches the control destination for the input information D41 in accordance with a predetermined condition (step S421). In the example shown in Fig. 26, the switching unit 42 determines whether the input information D41 matches an existing rule, and if it determines that the input information D41 does not match (No in step S421), the process proceeds to a first analysis process (step S422). On the other hand, if it determines that the input information D41 matches an existing rule (Yes in step S421), the process proceeds to a second analysis process (step S423).

[0362] In the first analysis process of step S422, the learning model unit 400a and the learning model unit 400b, which serve as the first analysis unit 41-1, analyze the situation and acquire an improvement method. As a result of the first analysis process, the learning model unit 400a and the learning model unit 400b output an analysis result D42a including the analysis result of the situation and an analysis result D42b including an improvement method for the situation.

[0363] In the second analysis process of step S423, the second analysis unit 41-2 analyzes the situation and acquires an improvement method according to the existing rules. The second analysis unit 41-2 outputs, for example, an analysis result D42c including at least an improvement method for the situation as a result of the first analysis process.

[0364] When the results of the analysis process by the first analysis unit 41-1 or the second analysis unit 41-2 are output, the target device 2 is controlled and / or information is displayed on the display 7 based on the results of either analysis process, depending on the state of the output changeover switch 43 (step S424).

[0365] In this example, when the first analysis unit 41-1 is performing an analysis process, the connection paths connecting the output of the first analysis unit 41-1 with the target device 2 and the display 7 are turned on. In this case, the model interface 6 may output, for example, result information D44a indicating a situation analysis result and an improvement method to the display 7 based on the analysis results D42a and D42b, and may output result information D44b indicating the improvement method based on the analysis results D42b to the target device 2. On the other hand, when the second analysis unit 41-2 is performing an analysis process, the connection paths connecting the output of the second analysis unit 41-2 with the target device 2 and the display 7 are turned on. In this case, based on the analysis result D42c output from the second analysis unit 41-2, result information D44a indicating a situation analysis result and an improvement method may be output to the display 7 and / or result information D44b indicating the improvement method may be output to the target device 2.

[0366] The display device 7 may display the result information D44b in a manner that can be confirmed by a worker, for example. In this case, the worker may refer to the result information D44b displayed on the display device 7, confirm the improvement method indicated by the result information D44b, and perform work related to that method. The worker may also confirm the improvement method indicated by the result information D44b and determine its appropriateness. At this time, if the improvement method indicated by the result information D44b is not appropriate, the worker may prompt the learning model unit 400b to acquire another improvement method (reacquire the model output data). For example, when the learning model unit 400b receives information requesting reacquisition of the model output data, the learning model unit 400b may reacquire the model output data after changing part of the input information, part of the model parameters, or the reference destination of the reference information.

[0367] The subsequent processing may be the same as that of the other control systems according to this embodiment.

[0368] As described above, this modified example is configured to have multiple analysis units that analyze situations and obtain improvement methods using different methods, and to switch between them depending on the situation. This enables control that is more suited to the situation. For example, for a problem with a clear cause, the second analysis unit with a high processing load can immediately analyze the situation and propose and execute an improvement method, while for a problem with an unclear cause, the first analysis unit using a learning model can analyze the complex situation and propose and execute a better improvement method.

[0369] In the above example, the first analysis unit 41-1 analyzes the situation and acquires an improvement method using two learning models, but the configuration of the first analysis unit 41-1 is not limited to the above example. For example, if analysis of the situation is not necessary, the learning model unit 400a can be omitted. Also, if acquisition of an improvement method is not necessary, the learning model unit 400b can be omitted. Also, analysis of the situation and acquisition of an improvement method can be performed by a single learning model unit.

[0370] For example, if the first analysis unit 41-1 includes a learning model unit 400b that acquires a method for improving the situation based on the input information D41, the switching unit 42 may switch the control destination for the input information D41 to the first analysis unit 41-1 when an abnormality occurs. In this case, the learning model unit 400b of the first analysis unit 41-1 may be configured to output, when the input information D41 is input, information indicating an improvement method corresponding to the occurrence situation of the abnormality indicated by the input information D41. In this case, the learning model unit 400b may refer to the device information storage unit 410 accessible by the control system and output information indicating an improvement method corresponding to the situation.

[0371] As described above, according to this embodiment 4, the control system 4000 is provided with a model interface 6 that controls the target equipment 2 in accordance with the improvement method generated by the learning model unit 400, which generates an improvement method when the status of the work being monitored is abnormal. Furthermore, according to this embodiment 4, the learning model unit 400 interprets the situation based on data acquired by the sensor 5 that acquires data indicating the status of the work being monitored, and generates an improvement method if it determines that the situation is abnormal. Moreover, according to the fourth embodiment, the learning model unit 400 generates an improvement method based on the information stored in the device information storage unit 410 that stores information about the target device 2. As a result, the control system 4000 according to the fourth embodiment can immediately respond to an abnormality in the work situation, thereby reducing downtime.

[0372] Furthermore, according to the fourth embodiment, when the improvement method to be generated includes a plurality of processes, the learning model unit 400 adds information indicating the order of the plurality of processes to the improvement method. As a result, the control system 4000 according to the fourth embodiment can immediately respond to abnormal work conditions, thereby reducing downtime. Also, in the control system 4000, it is easier to achieve consistency when the control program used in the model interface 6 is a ladder program.

[0373] Furthermore, according to this embodiment 4, the learning model unit 400 has a learning model unit 400a that interprets the status of the work being monitored based on data acquired by the sensor 5, and a learning model unit 400b that generates an improvement method when the learning model unit 400a determines that the status is abnormal. As a result, the control system 4000 of embodiment 4 improves the accuracy of situation interpretation and improvement methods by separating the learning model unit 400a that interprets the situation from the learning model unit 400b that generates the improvement method.

[0374] Furthermore, according to the fourth embodiment, the model interface 6 causes the output unit 7 to output the improvement method generated by the learning model unit 400 by at least one of display and voice. As a result, the control system 4000 according to the fourth embodiment allows the user 1 to check the improvement method, and also allows the user 1 to determine the appropriateness of the improvement method.

[0375] Furthermore, according to this embodiment 4, the output unit 7 accepts input of a prompt from the user 1, the model interface 6 outputs data indicating the prompt accepted by the output unit 7 to the learning model unit 400, and the learning model unit 400 regenerates an improvement method based on the data indicating the prompt output by the model interface 6. This allows the control system 4000 according to the fourth embodiment to regenerate an improvement method when the improvement method is not appropriate.

[0376] Furthermore, according to this embodiment 4, the control method is such that the model interface 6 controls the target equipment 2 in accordance with the improvement method generated by the learning model unit 400, which generates an improvement method in the event that the status of the work being monitored is abnormal. As a result, the control method according to the fourth embodiment can immediately respond to an abnormality in the work situation, thereby reducing downtime.

[0377] Embodiment 5 Next, a fifth embodiment will be described. In this embodiment, an example will be described in which a learning model is used to support a response task in a call center, a product site, or the like, in which a response is given to information transmitted from a user 1. Here, the information transmitted from the user 1 may include an inquiry or opinion regarding a certain service, information, event, or object.

[0378] 27 is a configuration diagram illustrating an example of a control system 5000 according to the fifth embodiment. The control system 5000 illustrated in FIG. 27 includes a learning model unit 500, a reference information storage unit 12, a database search unit 511 (referred to as a DB search unit in the figure), a control generation unit 512, a voice recognition unit 513v, and a voice synthesis unit 514v. Here, the reference information storage unit 12, the database search unit 511, and the control generation unit 512 may be provided as parts of the learning model unit 500.

[0379] When input information D51 is input, the learning model unit 500 outputs response information D52 indicating the content of the response. For example, when input information D51 is input, the learning model unit 500 outputs response information D52 based on model information D102. The configuration of the learning model unit 500 may be basically the same as that of the learning model unit 100 in the first embodiment.

[0380] In this embodiment, the learning model unit 500 is a model configured to output response information D52 corresponding to input information D51 when input information D51 is received, and its operating environment. Alternatively, the learning model unit 500 may be a model configured to generate and output response information D52 based on the input information D51 and other information that can be referenced in the learning model unit 500 when input information D51 is received, and its operating environment.

[0381] In this embodiment, the input information D51 includes information indicating content transmitted from the user 1, etc. The input information D51 may also include information indicating content requiring a response in the work environment. The input information D51 may be, for example, text, images, audio, or a combination thereof indicating an inquiry or opinion regarding a certain service, information, event, or object. The input information D51 may be, for example, text, images, audio, or a combination thereof indicating multiple inquiries or opinions regarding a certain service, information, event, or object. The input information D51 may also include information indicating temporally consecutive transmitted content. In this case, the input information D51 may be time-series data with a predetermined data structure including text, images, audio, or a combination thereof indicating the transmitted content, as described above. It is assumed that the way the transmitted content is displayed matches the input format of the model used by the learning model unit 500. However, this does not necessarily apply if error processing, correction processing, or conversion processing is provided upstream of the learning model unit 500.

[0382] The response information D52 includes information indicating a response to the transmission content included in the input information D51. The response information D52 may be, for example, information indicating a response to an inquiry or opinion regarding the service, information, event, or thing indicated by the transmission content included in the input information D51.

[0383] The reference information storage unit 12 stores model reference information D104 that the model control unit 101 of the learning model unit 500 references to output response information D52. The model reference information D104 includes, for example, information related to services, information, events, or objects that may be included in the input information D51. Herein, the reference information storage unit 12 may particularly store information related to a specific service, information, event, or object as the model reference information D104. The model reference information D104 may include, for example, a response manual converted into digital form. The model reference information D104 may also include, for example, a history of previously input input information D51 or the transmission content included therein. In this case, the reference information storage unit 12 may store, as the model reference information D104, history information indicating previously input information D51 or the transmission content included therein, along with information about the sender user 1 (e.g., a user identifier, attribute information of user 1, etc.). Hereinafter, in this embodiment, information indicating the status of the sender user 1 may be referred to as status information D55.

[0384] The database search unit 511 is a search engine for the reference information storage unit 12 and other databases. In response to a request from the model control unit 101 of the learning model unit 500, the database search unit 511 searches a database that is accessible to the database search unit 511 and outputs the search results. At this time, the database that the database search unit 511 accesses may be limited.

[0385] The control generation unit 512 is an interface for setting preconditions for the learning model unit 500 (particularly, the model control unit 101) to generate model output data. The control generation unit 512 may be, for example, an interface used to recognize information to be controlled by the learning model unit 500 and / or to set output trends. Here, the control target information is information indicating an object to be focused on in control by the model control unit 101. The model control unit 101 may be configured, for example, to generate model output data D103 from model input data D101 based on the control target information indicated by the control generation unit 512. The control generation unit 512 may, for example, recognize part of the model input data input by the user 1 as control target information, recognize information generated by the model control unit 101 as control target information, or recognize information generated by the model control unit 101 and modified by another control unit as control target information. The control target information and / or output trend settings may be specified by the user 1, an external processing unit, or the control generation unit 512 according to a predetermined algorithm.

[0386] When the input from the user 1 includes input information D51v in speech format, the speech recognition unit 513v recognizes the speech indicated by the input information D51v, converts it into a format that matches the data format of the learning model unit 500, and outputs it. The speech recognition unit 513v may, for example, convert the input information D51v in speech format into input information D51 in text format.

[0387] The speech synthesis unit 514v converts the content indicated by the response information D52 into a speech format and outputs it. For example, when the response information D52 output from the learning model unit 500 includes a data format other than speech, the speech synthesis unit 514v converts the content of that part indicated by the response information D52 into a speech format and outputs it. For example, when the response information D52 has a data structure including a data format specification, the speech synthesis unit 514v may convert a data element for which a speech format is specified in the specification into a speech format and output it. For example, the speech synthesis unit 514v may convert the response information D52 in text format into response information D52v in speech format.

[0388] In the above example, voice-format data is used for input and output to and from user 1, but the data format used for input and output to and from user 1 is not limited to voice format. In this case, instead of the voice recognition unit 513v and the voice synthesis unit 514v, a processing unit that converts the data format used for input from user 1 into the data format used for input to the learning model unit 500 and a processing unit that converts the data format used for output from the learning model unit 500 into the data format used for input to user 1 may be provided.

[0389] Furthermore, if the learning model unit 500 can accept the data format used for input from the user 1, the speech recognition unit 513v can be omitted. Furthermore, if the user 1 can accept the data format used for output from the learning model unit 500, the speech synthesis unit 514v can be omitted.

[0390] In this embodiment, the input information D51 corresponds to the model input data D101. The response information D52 corresponds to the model output data D103. The learning model unit 500 (particularly the model control unit 101) may be configured to, for example, upon receiving the input information D51, output the response information D52 corresponding to the input information D51 based on the model information D102 and, if necessary, the model reference information D104.

[0391] In such a case, the model generation unit 107 provided corresponding to the learning model unit 500 may perform machine learning using, for example, model learning data D105 including candidates for input information D51 that may be input to the model control unit 101, to generate or update the model information D102. In addition, the model generation unit 107 may perform machine learning using, for example, model learning data D105 including candidates for input information D51 that may be input to the model control unit 101 and candidates for response information D52 corresponding thereto, to generate or update the model information D102.

[0392] Although not shown in the figure, in this embodiment, status information D55 and / or feedback information D56 may be acquired from the output destination of the model output data D103 of the learning model unit 500 and / or information generated based on the model output data. For example, the control system 5000 may output the acquired status information D55 and / or feedback information D56 as information indicating a response result to a predetermined supervisor, the learning model unit 500, or another device (not shown). The control system 5000 may also be configured to return a query D57 to the user 1 when the input information D51 contains unclear or uncertain information. The control system 5000 may also generate supplemental information D58 for the input / output data of the learning model unit 500 based on the acquired status information D55 and / or feedback information D56 and issue the supplemental information D58 to the user 1, a predetermined supervisor, the learning model unit 500, or another device (not shown). The handling of the status information D55, feedback information D56, query D57, and supplemental information D58 may be basically the same as in the first embodiment.

[0393] The control system 5000 may further include a status acquisition unit 530 (not shown) that acquires the status information D55 and / or the feedback information D56 and issues supplemental information D58 as necessary. The status acquisition unit 530 is similar to the status acquisition unit 130 in the first embodiment.

[0394] In this embodiment, the input information D51 received by the control system 5000 can be considered to be information regarding a request in the work environment (here, transmission content that requires a response in an environment where a response task is performed in response to an inquiry). Therefore, the input information D51 received by the control system 5000 can be considered to be an example of first information indicating a request in the work environment. Furthermore, the response information D52 can be considered to be information used in the task (response task) in response to such input information D51. Hereinafter, the response information D52 output to a predetermined output destination from the operating environment of the learning model to which model input data based on the input information D51 is input will be referred to as second information.

[0395] Next, a description will be given of the operation of the control system 5000 of this embodiment.

[0396] 28, first, the control system 5000 receives input information D51v (step S510). For example, the input unit 102 or the input processing unit 201 described above may receive the input information D51v. The received input information D51v is input to the voice recognition unit 513v.

[0397] When the speech recognition unit 513v receives the input information D51v, it recognizes the speech included in the input information D51v and converts it into input information D51 that matches the data format of the input to the learning model unit 500 (step S511). The converted input information D51 is input to the learning model unit 500 as model input data D101.

[0398] When the speech recognition unit 513v is omitted, the received input information D51v may be input to the learning model unit 500 as the model input data D101.

[0399] Next, the control system 5000 performs a process of generating response information D52 using the learning model unit 500 (step S512). In step S512, the learning model unit 500 (more specifically, the model control unit 101) outputs response information D52 corresponding to the input information D51 based on the model information D102 and the input information D51 that has been input, and, if necessary, the model reference information D104.

[0400] In step S512, the pre-processing unit 105 and / or the post-processing unit 106 of the learning model unit 500 may further perform the above-mentioned processing.

[0401] The response information D52 output from the learning model unit 500 is input to, for example, the speech synthesis unit 514v (step S513). The response information D52 may be input to the speech synthesis unit 514v directly from the control system 5000 (more specifically, the learning model unit 500 or the information processing device 10 as its operating environment), or indirectly via a communication network, other devices (servers, various conversion devices, etc.), or human hands.

[0402] Next, the voice synthesis unit 514v converts the input response information D52 into voice-format response information D52v and outputs it (step S514). The voice synthesis unit 514v may generate the response information D52v, for example, by synthesizing a voice that speaks the response content indicated by the response information D52, which is in a data format other than voice. The response information D52v is output to the user 1 who is the sender of the input information D51v (step ST515).

[0403] When the voice synthesis unit 514v is omitted, the response information D52 output from the learning model unit 500 may be output to the user 1 who is the sender of the input information D51v.

[0404] As described above, in this embodiment, response information D52 can be dynamically generated using the learning model unit 500 and sent back to the sending user 1 without the need to prepare an operator or a site with pre-embedded response content in response to information sent from user 1, thereby making it possible to improve the efficiency and performance of response work.

[0405] Variation 5-1. Next, a modified example of the control system 5000 will be described. Fig. 29 is a configuration diagram showing an example of a control system 5000a which is a modified example of the control system 5000 according to the present embodiment. Note that the same elements as those in the control system 5000 are given the same reference numerals and description thereof will be omitted.

[0406] A control system 5000a shown in FIG. 29 differs from the control system 5000 in that a true / false determination unit 515 is provided.

[0407] The correctness determination unit 515 determines whether the content indicated by the response information D52 output from the learning model unit 500 is correct. For example, the correctness determination unit 515 may output the response information D52 to the user 1 or update the content of the reference information storage unit 12 only when it determines that the content indicated by the response information D52 is correct.

[0408] Furthermore, for example, when the correctness determination unit 515 determines that the content indicated by the response information D52 is not correct, the correctness determination unit 515 may prompt the learning model unit 500 to acquire another response information D52 (reacquire model output data). The correctness determination unit 515 may be provided as, for example, an example of the post-processing unit 106 described above.

[0409] Other points may be the same as those of the other control systems according to this embodiment.

[0410] As described above, according to this modified example, it is determined whether the content indicated by the response information output from the learning model unit 500 is correct, and based on the result, it is determined whether to output to user 1, re-acquire the response information, and update the reference information, thereby achieving higher performance in the response process.

[0411] Variation 5-2. Next, a second modified example of the control system 5000 will be described. Fig. 30 is a configuration diagram showing an example of a control system 5000b, which is a modified example of the control system 5000 according to the present embodiment. Note that the same elements as those in the control systems 5000 and 5000a are given the same reference numerals and will not be described again.

[0412] As shown in FIG. 30, the control system 5000b may further include an emotion determination unit 516.

[0413] The emotion determination unit 516 uses the input information D51 and other information to determine the emotion of the sender, user 1. The emotion determination unit 516 may also determine the emotion of user 1 after response information D52 from the learning model unit 500 is output to user 1.

[0414] The emotion of user 1 determined by the emotion determination unit 516 may be input to the learning model unit 500 as state information D55 included in the model reference information D104, or may be recorded as history in the reference information storage unit 12 together with the input and output data of the model.

[0415] As a method of recording in the reference information storage unit 12, for example, the control system 5000b may further include a registration determination unit 518, which may determine whether or not to record in the reference information storage unit 12 based on the determination result of the emotion of user 1 by the emotion determination unit 516.

[0416] For example, if the determined emotion of user 1 is positive, the registration determination unit 518 may record the input / output data of the model as history information in the reference information storage unit 12 as a good example. At this time, if there is a determination result of the emotion of user 1 before the output of response information D52 from the learning model unit 500, the registration determination unit 518 may record the input / output data of the model including emotion information before and after the response as history information in the reference information storage unit 12.

[0417] Furthermore, for example, if the determined emotion of user 1 is negative, the registration determination unit 518 may treat it as a bad case and record the input / output data of the model as history information in the reference information storage unit 12. At this time, if there is a determination result of the emotion of user 1 before the output of response information D52 from the learning model unit 500, the registration determination unit 518 may record the input / output data of the model including emotion information before and after the response as history information in the reference information storage unit 12.

[0418] In addition, the control system 5000b may further include an additional learning unit 519, and when updating the contents of the reference information storage unit 12, may reconstruct (additionally learn) the model reference information D104 stored in the reference information storage unit 12 and other information referenced by the model control unit 101 based on the update information.

[0419] Furthermore, the control system 5000b may include an evaluation acquisition unit 517 instead of the emotion determination unit 516 or in addition to the emotion determination unit 516.

[0420] The evaluation acquisition unit 517 inquires of the user 1 about the evaluation of the response information D52, and acquires evaluation information D59 as a response. The evaluation information D59 can be used, for example, for updating information referenced by the model, additional learning, and the like, similar to the emotion of the user 1 described above.

[0421] The control system 5000b may further include a control determination unit 520.

[0422] The control determination unit 520 designates information to be controlled and / or designates output tendency settings for the control generation unit 512 based on the results of speech recognition for the input information from the user 1, the emotion determination result and / or the evaluation result of the response information D52, instructions from an operator (not shown), etc. Here, the results of speech recognition for the input information from the user 1 may include information such as the attributes, emotions, region, language, whether or not the user has used the service in the past, and frequency of use. Furthermore, the control determination unit 520 may set synthetic speech for the speech synthesis unit 514v based on the results of speech recognition for the input information from the user 1, the emotion determination result and / or the evaluation result of the response information D52, instructions from an operator (not shown), etc.

[0423] The control determination unit 520 can specify, for example, the difficulty of the explanation in the response, the speaking style (tone and voice), the language, the level of grammar, the politeness, the speaker's position, the destination of the conversation, etc. as examples of output tendency settings. The control determination unit 520 can specify, for example, the gender, speaking style, tone, etc. of the synthetic voice. Furthermore, the control determination unit 520 can specify, for example, the gender, speaking style, the language, the level of grammar, the politeness, etc. of the synthetic voice as examples of synthetic voice settings. The control determination unit 520 may make these settings based on, for example, predetermined setting rules.

[0424] The elements of the control system 5000b shown in FIG. 30 can be selected appropriately depending on the desired functions.

[0425] Other points may be the same as those of the other control systems according to this embodiment.

[0426] As described above, according to this modification, the control determination unit 520 designates the information to be controlled and / or designates the output tendency setting based on information obtainable from the control system 5000b, and therefore it is possible to generate response information that is likely to match the request of the sender. This makes it possible to further improve the performance of the response work to the user 1.

[0427] Variation 5-3. Next, a third modified example of the control system 5000 will be described. Fig. 31 is a configuration diagram showing an example of a control system 5000c, which is a modified example of the control system 5000 according to the present embodiment. Note that the same elements as those in the control systems 5000, 5000a, and 5000b are given the same reference numerals, and description thereof will be omitted.

[0428] As shown in FIG. 31, the control system 5000c may further include an image analysis unit 513i, an image generation unit 514i, and a program generation unit 514p.

[0429] When the input from the user 1 includes input information D51i in image format, the image analysis unit 513i analyzes the image indicated by the input information D51i, converts it into a format that matches the data format of the learning model unit 500, and outputs it. The image analysis unit 513i may, for example, convert the input information D51i in image format into input information D51 in text format.

[0430] For example, if the input from user 1 includes an image capturing an operation screen of a product owned by user 1, the image analysis unit 513i may analyze the image, identify which operation screen of which product it is, and what operation state it is in, and convert it into explanatory text and output it. Also, for example, if the input from user 1 includes an image capturing a purchasing site that user 1 is browsing, the image analysis unit 513i may analyze the image, identify which operation screen of which site it is, and what operation state it is in, and convert it into explanatory text and output it.

[0431] The image generation unit 514i generates and outputs an image based on the response information D52. For example, when the response information D52 output from the learning model unit 500 includes a data format other than an image, the image generation unit 514i may generate and output an image showing the content of the portion indicated by the response information D52. For example, when the response information D52 has a data structure including a data format specification, the image generation unit 514i may convert a data element for which an image format is specified in the specification into an image format and output the image. For example, the image generation unit 514i may generate image-format response information D52v based on text-format response information D52. For example, the image generation unit 514i may perform a synthesis process to add the content indicated by the text-format response information D52 as an annotation to an image included in the input information D51. Furthermore, the image generation unit 514i may perform a process to highlight a portion of the image included in the input information D51 based on the text-format response information D52. The image generating unit 514i may generate an image from input information (the response information D52 and, if necessary, the input information D51) using a learning model.

[0432] The program generation unit 514p converts the content indicated by the response information D52 into the data format of a predetermined program and outputs the converted content. For example, when the response information D52 output from the learning model unit 500 includes a data format other than that of the predetermined program, the program generation unit 514p converts the content of the portion indicated by the response information D52 into the data format of the predetermined program and outputs the converted content. For example, when the response information D52 has a data structure including a data format specification, the program generation unit 514p may convert data elements whose data format of the predetermined program is specified in the specification into the data format of the predetermined program and output the converted content. For example, the program generation unit 514p may convert the response information D52 in text format into response information D52p in the data format of the predetermined program. The program generation unit 514p may generate a predetermined program from input information using a learning model.

[0433] The image analysis process by the image analysis unit 513i is performed, for example, in the above-mentioned step S511. The image generation process by the image generation unit 514i and the program generation process by the program generation unit 514p are performed, for example, in the above-mentioned step S514.

[0434] Other points may be the same as those of the other control systems according to this embodiment.

[0435] As described above, according to this modification, inquiries and responses can be made not only by voice but also by voice and images, so that it is possible to more effectively respond to inquiries about the operation screen, for example. Furthermore, according to this modification, it is possible to provide the sender with a program as response information in addition to voice and images, so that it is possible to more effectively respond to inquiries about troubleshooting, etc.

[0436] Variation 5-4. Next, a fourth modified example of the control system 5000 will be described. Fig. 32 is a configuration diagram showing an example of a control system 5000d, which is a modified example of the control system 5000 according to the present embodiment. Note that the same elements as those in the control systems 5000 to 5000c are given the same reference numerals and description thereof will be omitted.

[0437] This modified example has a function of switching between a response by the operator 8 or a response by another learning model based on the content of the inquiry from the user 1 and / or the output result from the learning model.

[0438] As shown in FIG. 32, the control system 5000d may further include a call confirmation unit 531 and an output selection unit 532.

[0439] Here, the control system 5000d is assumed to include a learning model unit 500a as a first response function, and an operator 8 and a communication channel with the operator 8 as a second response function. The control system 5000d may further include another learning model unit 500b as a third response function, which has a different algorithm or data used from the learning model unit 500a. Note that the second response function may include another learning model unit 500b that has a different algorithm or data used from the learning model unit 500a. In this case, the third response function may further include an operator 8 and a communication channel with the operator 8. Note that the type and number of response functions are not particularly limited. For example, the response function to be switched to may be a response system that does not use a learning model.

[0440] In this example, we will explain the case where the learning model unit 500a, which is the first response function, is the above-mentioned learning model unit 500, the second response function is the operator 8 and a communication channel with the operator 8, and the third response function is another learning model unit 500b that has a different algorithm or data used than the learning model unit 500a.

[0441] Here, the learning model unit 500a may be a local learning model that obtains output results based on local information, such as having limited reference databases, and the learning model unit 500b may be a global learning model that obtains output results based on global information, such as having free access to external networks.

[0442] The call confirmation unit 531 switches the processing destination for performing response processing based on the content of the inquiry from the user 1 and / or the output result from the learning model.

[0443] The call confirmation unit 531 may call an operator 8 as a second response function, for example, when it is determined that the accuracy of the output by the first response function is not expected based on the content of the inquiry from the user 1 and / or the output result from the learning model. The call confirmation unit 531 may, for example, call the operator 8 using a communication channel with the operator 8 and input the input information D51 into an operation device of the operator 8. Furthermore, the call confirmation unit 531 may call the operator 8 using a communication channel with the operator 8 and input the input information D51 into an operation terminal (not shown) of the operator 8.

[0444] Furthermore, if it is determined that a call to the second response function is impossible or that output accuracy is not expected, the call confirmation unit 531 may call the learning model unit 500b as a third response function. The call confirmation unit 531 may call the learning model unit 500b, for example, by inputting input information D51 to the learning model unit 500b using an interface with the learning model unit 500b.

[0445] Here, the output accuracy may be determined using, for example, an evaluation value or likelihood output by the response function itself, or the reliability evaluation described above. In addition, if the response function itself outputs a message indicating that it does not understand or requesting the calling of another function, it is also possible to determine the output accuracy based on the presence or absence of such a message.

[0446] The output selection unit 532 selects response information D52 to be output to the user 1 based on the result of switching the response process by the call confirmation unit 531. When the result of switching the response process by the call confirmation unit 531 shows that the execution subject of the response process is the first response function, the output selection unit 532 outputs response information D52a to be output from the first response function to the user 1. Furthermore, when the result of switching the response process by the call confirmation unit 531 shows that the execution subject of the response process is the second response function, the output selection unit 532 outputs response information D52b to be output from the second response function to the user 1. Furthermore, when the result of switching the response process by the call confirmation unit 531 shows that the execution subject of the response process is the third response function, the output selection unit 532 outputs response information D52c to be output from the third response function to the user 1.

[0447] The output selection unit 532 may output the output from the selected response function to user 1 by controlling an output switching switch (not shown) that switches the connection path (such as a circuit or communication path) connecting the response function that is the execution subject and user 1 that is the output destination.

[0448] Here, the connection path between the response function and the user 1 may include various conversion devices such as the above-mentioned voice synthesis unit, image generation unit, program generation unit, and predetermined interfaces as needed.

[0449] For example, if the character input by the operator 8 using an operation terminal is output as response information D52b, the connection path between the response function and the user 1 may include a speech synthesis unit that converts text into speech. The output selection unit 532 may also accept, as the output of the second response function, response information D52a that has been modified from response information D52a output by the first response function. In this case, the operation terminal of the operator 8 includes a text display unit and a text input unit, and the control system 5000d may accept, for example, response information D52b that has been partially modified from response information D52b output from the operation terminal of the operator 8.

[0450] Other points may be the same as those of the other control systems according to this embodiment.

[0451] As described above, according to this modified example, in addition to generating responses using the above-mentioned learning model unit 500, responses can also be generated by an operator or using other learning models (including, for example, a tandem structure model in which multiple models are connected, a multimodal model, or a model trained specifically for a specific device or service), thereby further improving the performance of the response work to user 1.

[0452] In the above-described embodiments, an example of a system configuration corresponding to a task of interest has been described, but the control system according to the present disclosure is not limited to the above-described examples. For example, the control system according to the present disclosure may be implemented by appropriately combining one or more of the above-described embodiments.

[0453] As an example, the control system according to the present disclosure can combine the configuration of the first embodiment with the configuration of the fourth embodiment, and utilize the functions of the fourth embodiment to input information indicating a solution obtained from sensor data into the control system of the first embodiment, convert it into a program, and directly control the target device 2.

[0454] Furthermore, the embodiments and modifications are not limited to the examples described above, and can be modified as appropriate within the scope of the disclosure. [Industrial Applicability]

[0455] The control system according to the present disclosure can be suitably applied as part of a work support system that supports work by people or objects. The control system according to the present disclosure can also be suitably applied as a control system that controls equipment when performing some control or work using the equipment. Here, the control system can also be suitably applied as a control system that controls factory automation equipment, a control system in a home or building, or a control system that controls an information processing device such as a server device that processes information on a network. [Explanation of symbols]

[0456] 1000, 1000a, 1000b, 1000c, 2000, 3000, 3000a, 3000b, 3000c, 3000d, 3000e, 4000, 4000a, 5000, 5000a, 5000b, 5000d Control System 100, 200, 300, 300a, 300b, 400, 400a, 400b, 500, 500a, 500b Learning model section 10, 20 Information processing device 11 Model information storage unit 12 Reference information storage unit 101 Model control section 102 Input section 103 Output section 105 Pretreatment section 106 Post-processing section 107 Model Generation Unit 104, 104a control section 201 Input processing section 202 Output confirmation section 203 Correction confirmation section 1 user 1a Input source 2. Target devices 2a Output destination 3 Operation screen user interface 4 Controller 41-1, 41-2 Analysis section 42 Switching section 43 Output selector switch 5 sensors 6 Model Interface 7 Display 8 Operators 110, 210, 310, 410 Device information storage section 120 Execution code generation unit 230 Status acquisition unit 311 Input Interface 312 Output Interface 313 Environmental information storage unit 511 Database Search Department 512 Control Generation Unit 513v Voice Recognition Unit 513i Image Analysis Unit 514v Image synthesis unit 514p Program Generation Section 515 Correctness Division 516 Emotion Judgment Department 517 Evaluation Acquisition Department 518 Registration Judgment Department 519 Additional Learning Department 531 Call Confirmation Unit 532 Output selection unit 533 Output switching unit D101 Model Input Data D102 Model Information D103 Model Output Data D104 Model Reference Information D11, D21, D31, D41, D51, D51v, D51i Input information D12 Control Description D22, D34 control commands D42a, D42b analysis results D52, D52v, D52i, D52p, D52a, D52b, D52c Response Information D32, D32a, D32b operation command D320 Operation Information D13, D23, D33, D43 Device Information D33a environmental information D14 Execution Code D44a, D44b Results Information D15, D25, D35, D45 Status Information D16, D26, D36, D46 Feedback Information D17, D27, D37, D47, D57 Inquiry D18, D28, D38, D48 Supplementary Information D59 Evaluation Information

Claims

1. a model interface that controls the equipment in accordance with an improvement method generated by a learning model unit that generates an improvement method when the status of the work to be monitored is abnormal; The learning model unit a first learning model unit that interprets the work situation based on first data indicating the work situation and outputs second data indicating the interpretation result; a second learning model unit that outputs third data indicating an improvement method for the work situation based on the second data, The second data includes information that expresses, in an attribute or a predetermined syntax format, at least one of instantiation, subdivision, and specific point extraction for the work situation indicated by the first data. A control system comprising:

2. The second data includes information indicating at least one of an object present and an event occurring in the work situation indicated by the first data.

2. The control system of claim 1.

3. The second data includes at least one of information expressed using attributes of an object that exists and an event that occurs in the work situation indicated by the first data, and information expressed in a predetermined syntax format about the event.

2. The control system of claim 1.

4. The second data includes at least one of summary information expressed using attributes of an object that exists and an event that occurs in the work situation indicated by the first data, and summary information expressed in a predetermined syntax format of the event.

2. The control system of claim 1.

5. the first data is image data acquired by a plurality of cameras; The second data includes information on the interpretation results for each small task or step when the task status indicated by the first data is decomposed into small task units or step units.

2. The control system of claim 1.

6. The second data is data indicating an interpretation of a part of the work situation indicated by the first data that differs from a normal situation.

2. The control system of claim 1.

7. The third data includes information indicating a method for improving the status of the work, which is derived from the second data.

2. The control system of claim 1.

8. The third data includes data written in a predetermined design language, a control description, information written in a predetermined platform language, a control command, an execution code, and a combination of two or more of these elements.

2. The control system of claim 1.

9. The first learning model unit interprets the status of the work based on the first data and information related to the work, and outputs second data indicating the interpretation result.

2. The control system of claim 1.

10. The information about the work includes information indicating the location, person, object, procedure, or condition for performing the work.

10. The control system of claim 9.

11. The information about the work includes information about equipment related to the work.

10. The control system of claim 9.

12. The equipment related to the work includes equipment used in the work and equipment that affects the person or equipment performing the work.

12. The control system of claim 11.

13. The equipment that affects the person or equipment performing the work includes at least one of equipment that controls the equipment used in the work and equipment that brings about changes in the environment of the work.

13. The control system of claim 12.

14. The information on the equipment used in the work includes manual data for the equipment.

13. The control system of claim 12.

15. The second learning model unit determines whether the state of the work is abnormal or normal from the second data, and outputs the third data when it is determined to be abnormal.

2. The control system of claim 1.

16. The first learning model unit determines whether the state of the work is abnormal or normal from the generated second data, and outputs the second data when determining that the state is abnormal.

2. The control system of claim 1.

17. the first learning model unit receives sensor data and outputs the second data expressed in a text format; The second learning model unit receives the second data expressed in text format and outputs the third data.

2. The control system of claim 1.

18. The first learning model unit receives sensor data including image data, recognizes an object in the image data using an object recognition technology, and generates the second data expressed in a text format.

2. The control system of claim 1.

19. The first learning model unit receives sensor data as input and outputs text data indicating only data determined to be abnormal among the sensor data as the second data.

2. The control system of claim 1.

20. The first learning model unit outputs, as the second data, text data in which the text data is written or the order of description is changed so that the situation and factors can be easily understood based on the importance of terms, correlation in chronological order, or correlation in word order in the text data generated from the data determined to be abnormal.

20. The control system of claim 19.

21. The second learning model unit inputs the second data expressed in text format, generates the third data expressed in text format, and converts a process with a high degree of urgency included in the third data into a program conforming to a communication protocol for controlling the device from the model interface and outputs the program.

21. The control system of claim 20.

22. When the third data includes a program, the second learning model unit or the model interface verifies whether the program has any problems using a simulator.

2. The control system of claim 1.

23. When the model interface receives the second data and the third data from the first learning model unit and the second learning model unit, it converts the data into data that matches a predetermined output destination including the device and outputs the data to the output destination.

2. The control system of claim 1.

24. The predetermined output destination includes the device and the display.

24. The control system of claim 23.

25. The model interface extracts a portion of data from at least one of the second data and the third data, converts the extracted portion of data into a data format that matches the predetermined output destination, and outputs the converted data.

24. The control system of claim 23.

26. the model interface controls the device to immediately execute a process when the third data includes a process with a high urgency; The model interface performs a process of confirming with a user the validity of an improvement method indicated by the third data when the third data includes a process other than a process with a high degree of urgency.

2. The control system of claim 1.

27. The validation is performed by verbalizing the second data and the third data and displaying them on a display device.

27. The control system of claim 26.

28. If the improvement method is not valid as a result of the validity check, the model interface outputs the information input by the user to the second learning model unit to prompt the regeneration of the improvement method.

27. The control system of claim 26.

29. The status of the work to be monitored is the status of work related to the distribution object in the distribution system.

2. The control system of claim 1.

30. The status of the work to be monitored is the status of work related to the control system of equipment in the factory.

2. The control system of claim 1.

31. When the improvement method to be generated includes a plurality of processes, the second learning model unit adds information indicating the order of the plurality of processes to the improvement method.

2. The control system of claim 1.

32. The model interface causes an output unit to output the improvement method generated by the second learning model unit at least by displaying or audibly.

32. A control system according to any one of claims 1 to 31.

33. the output unit accepts a prompt input from a user; the model interface outputs data indicating the prompt received by the output unit to the second learning model unit; The second learning model unit regenerates an improvement method based on data indicating the prompt output by the model interface.

33. The control system of claim 32.

34. The model interface controls the equipment in accordance with the improvement method generated by the learning model unit that generates an improvement method when the status of the work to be monitored is abnormal; The learning model unit a first learning model unit interpreting the work situation based on first data indicating the work situation and outputting second data indicating the interpretation result; a second learning model unit outputs third data indicating an improvement method for the work situation based on the second data; The second data includes information that expresses, in an attribute or a predetermined syntax format, at least one of instantiation, subdivision, and specific point extraction for the work situation indicated by the first data. A control method comprising:

Images